Amplitude Self-Calibration and Flux Levels

If you want to get really high dynamic range, self-calibration is a necessary. (See this post for the basics of self-calibrating in AIPS.) However, getting the fluxes right (especially for weak extended emission) is tough. See below for more details.



Usually you start off with a phase calibration boxing more and more sources as you improve the model. I usually start boxing fairly conservatively and then box more freely in later phase self-calibration interations. The amplitude self-calibration, however, is slightly different in that you need to include all the emission that's real and none that's not, i.e., you need to make the best possible image of your source. If you don't include all the emission (say the diffuse stuff), then you are essentially saying that there's zero flux in that part of the image and the fluxes in the resulting image will be forced downward closer to that value.

There are a couple things to check to see how well you're doing with the flux estimates. One is to plot the visibility amplitudes vs. uv distance. They should be more or less the same. If you have amplitudes in the amplitude self-calibration data that are significantly below that in the unself-calibrated data you've got problems. Another thing to check is how much total flux you're recovering. The total clean components you use (given in the output of IMAGR) should be close to the total flux in the image and the fluxes of sources shouldn't go down after amplitude self-calibration.

Thanks to Crystal Brogan and Bill Cotton for their help with this issue.

Read more!

Why is imaging quick sometimes and slloooow others?

I have 6 sources, each with 2-3 hours of continuum L-band data on them. For five of them, imaging and self-calibration are really quite quick, but one of the sources takes a factor of 5-6 longer to image/calibrate than the others! Its correlator setup is not different, it was taken only one day previous to the other sources, and it has a similar amount of time on source. Any ideas about why imagr seems to choke for this source? Thanks!

Read more!

How does CALIB make use of clean components?

A question from Sam Tun....
I've been working on some self-cal programs for the Owens Valley Solar Array, and I am having similar problems as these. I would want a self-caled map that gives me a good approximation to my CLEAN map fluxes, but no go on that. So, my question is, does anyone know what is done to the clean components to get them to return appropriate visibilities to feed into self-cal? I believe that in AIPS you feed the CLEAN map as a model into calib, but does anyone know the details of what goes on in there (or where to find them)? Gracias.

Read more!

How to Know When You Can Self-Cal and What Solint To Use

Have you ever done a self-calibration run to find out the self cal is actually making your images worse, not better? Have you ever guessed at what SOLINT to use while self calibrating? I know I have!

People always say you should evaluate the signal-to-noise of your data before self-calibrating, but I never understood what this meant until today! There is a simple equation to find out if you can self-cal and if the SOLINT you are considering might be too short....

First image your data and clean it pretty good. Afterwards you can look in the image header and not the total cleaned flux. This is your 'Signal'.

Second of all, you want to calculate the noise in your data, per baseline per SOLINT. First, measure the rms noise in your image, in Jy/beam (sig_image). Next, calculate the number of baselines in your data (N_base where N_base = ((N_ant * N_ant-1) / 2) and N_ant is the number of operational antennae). Finally, figure out how much time-on-source went into making your image, in minutes (TOS). The noise of your data per baseline for a given SOLINT (in minutes) is then:
Noise = sig_image * sqrt(N_base) * sqrt(TOS/SOLINT)

Now, compare the 'Signal' with the 'Noise'. For 'P' self cal, you want the Signal to be at least 5 times greater than the Noise. If it's not, then increase your SOLINT. For A&P self cal, you probably want a signal-to-noise of 10-20, at least.

Note: If you are doing multi-facet imaging (at lower frequencies), you want to use the total flux in your data-- that is the sum from all facets, The image headers tell you this as 'CCTOTAL'.

Read more!

Self Calibration and DBCON

Let's say you have two data sets that you want to DBCON together. Should you self calibrate them individually? or after DBCONing?

The tip I got is you should self-cal each data set individually as best you can, then DBCON. However, after DBCONing, there might be some small gain offsets between the two data sets. Do a final A&P self cal on the DBCONed data set with a really long SOLINT, so that you basically have one SN solution per data set. This will ensure the original data sets are as consistent with one another as possible.

Read more!

Flagging After A&P Self Calibration?

I've been wondering lately-- when you run an Amplitude & Phase Self-Cal run, often times this can make certain parts of your data look anomalously high or low. Usually it's pretty mild, but a couple of times, it has been rather dramatic. A bit distressing, as I don't really understand why CALIB chose to make my amplitudes worse...

But anyway, does anyone out there have opinions on whether you should flag your data again after an A&P self cal? If you do flag, do you then copy the flag table over to the un-A&P-self-calibrated data set, and the re-self-calibrate using this new flag table? And iterate so on till no more "bad" data pops up on the A&P self cal?

Read more!

Help Miriad Selfcal Tip

IGNORE THE FOLLOWING POST. If Miriad is rescaling to large values during the self-calibration process, this is a sign that something is wrong with your self-calibration and you need to increase the self-calibration interval or need more clean components.


This is a bit off the beaten path, but I'd thought I'd post it anyway. If you're self-calibrating in Miriad, it helps to set the noscale option.



The noscale option prevents Miriad from rescaling the gains during amplitude+phase self-calibrating so that the rms gain is 1.0. If you have a lot of outliers, not setting this option can play havoc with the derived amplitudes. A sign that this may be a problem for your data would be unrealistically large amplitudes (much greater than 1.0) after amplitude+phase self-calibration.

Read more!

Beam Squint and Self Calibration (relevant even if you don't care about polarization!)

Here's another tip from Juan Uson about a self-cal trick to get the very best dynamic range in your images.

The VLA has beam squint in Stokes V. What this means is that away from the center of your field, the RR and LL polarizations separate a bit (I don't really understand the details of this). This leads to an image artifact-- if you have a bright source off the center of your field, it'll seem like you can't clean it very well. Even if you've done your very best job at self calibration and cleaned very deeply, there will still remain rings around the source (which of course drives up your noise).

One easy fix for this problem is to do a normal Phase self calibration, but when it's time for your A&P self calibration, average your RR and LL. I believe the relevant parameter for this in CALIB is APARM(3) > 1. This should give you a self cal that is not affected by the beam squint, and models your bright sources better!

Read more!

Clean Components and Self Calibration

Well, I just talked to Juan Uson from NRAO-Charlottesville, and he delivered some useful information for us burgeoning self-cal experts.

He says that you should ALWAYS use all your clean components for self calibration. Don't cut at the first negative! He says that when they were first learning about self cal, they were very timid and therefore advised people to cut at the first negative, but now they know better. You want to use your very best clean component model for self cal, which includes all components!

Here are some reasons why--

a bright source that is centered on the edge between two pixels will actually take an infinite number of clean components to model, because basically the clean algorithm keeps exchanging flux back and forth between the surrounding pixels. so, to model such a source, there are lots of negative clean components required.

if a source has any complex structure at all, you can't really effectively model it without using some negative components.

I told him that occasionally, the word on the street is "Just self cal off of one bright source that you know the structure of, and leave the rest of the flux alone." When I tried to do that once, the self cal created fictional ghost sources in my data and it scared me! Juan was not surprised; he said its a terrible idea to self cal on only a select few sources. Basically what self cal does is divide your real fluxes by your model fluxes, and try to make the residuals look like noise. If there is still real flux in your residuals (because your clean model is based off only a select few sources), then it will try to turn that real flux into noise and crazy things will happen.

So, clean out all your flux, and use all your clean components!

Read more!

self-calibration subtleties : 2 questions

• i have a dataset which shows significantly different structure depending on the weighting (robust value) used when i image it. should i self-cal using maps with a natural weighting that show the diffuse emission or maps with a more uniform weighting which emphasize the point sources? i'm inclined to use a robust value around 0 because i feel like the point sources are more reliable self-cal targets (and contribute more flux) even though what i really care about is the diffuse stuff and the maps i make for the analysis will use a robust > 0.



• do i always need to self-cal using clean components from the entire primary beam even if the source i care about is only in the inner 15%? since it does seem true that "the best self-cal model is the one that contains the most flux" i think i am obligated to image the entire primary beam during self-calibration.

Read more!

CPARM(2) and Self Cal

Emily read in the GMRT documentation that, when doing A&P self calibration, you should set CPARM(2) = 1, which means you would 'normalize gains'. However, we didn't really understand what this meant, and didn't understand why we were doing it. Today I asked Miller Goss, who's visiting UW from NRAO, and he said...


You should set CPARM(2) = 1 only if you don't trust your amplitudes initially! If you think your original amplitude calibration is not so good, say in the case of VLBA data or certain frequencies at the GMRT, you might want to set it.

However, at the VLA, you should set CPARM(2) = 0 because your original amplitude calibration was really quite decent. (At least at L and C band, not sure about crazy other frequencies!)

Read more!

Self-Cal Wackiness

UPDATE (4/18/07): I think I've tracked the errors down to very bad amplitude solutions for antenna 1 for about an hour of one day. Still working on a solution. I'm tempted to flag antenna 1 for that time frame since it seems so self-contained, but that makes me a bit nervous.

UPDATE (4/23/07): I ended up flagging antenna 1 for the time where the solutions weren't converging and one scan on antenna 28. Now the data look fabulous. These antennas had phase errors in the phase calibrators during the relevant periods of time, so I'm feeling better about flagging them.



This isn't completely relevant to this blog because I'm doing it in Miriad, but I'm getting some funny artifacts when I amplitude+phase self-cal in Miriad. The phase self-cal goes fine, but I get these cheap carpet patterns when I amplitude+phase self-cal. The first image below is no cleaning, phase self-cal only. I can get rid of the swirls with cleaning. The second image is no cleaning, amplitude+phase self-cal. Weird! Any idea of the origin of this pattern? Has anyone else seen something like this?





I'm wondering if I can ignore the amplitude self-calibration and just do the phase self-calibration.....

Read more!

Flux Conservation in Self Calibration?

Any one know what self calibration really does to your flux? I need fluxes of sources that are mostly point-ish, and I like self calibrating because it makes them look more like circles and less like amorphous blobs. But I wonder if I can trust my fluxes after self cal? If I do only phase self cal, should I be ok?

I found in someone's Ph.D. thesis that your fluxes are no longer "absolute" after self calibration. I guess because you're no longer comparing everything with 3C286, you're comparing your target source with itself.

Anyway, when i just do an imean on my images, the total flux density doesn't change a huge amount. Maybe 20% (well, sometimes I get factors of 2, but I avoid those cases). I don't really want to be introducing 20% errors into my flux measurements, though....
Maybe I shouldn't be self caling if I want robust fluxes?

Anyone really understand what cparm(2) does in calib? Is it only important for A&P self cal, or P self cal too?
Squawk!

Read more!

Self Cal according to Claudia (as inspired by Crystal Brogan)

I'm pasting Claudia's notes below, about how she self calibrated her SMA data. I always like to see how people do their self cals! I'll probably post later some more details about how I personally do it.



CLEANING/SELF-CAL
Clean in IMAGR
To get a circle (as opposed to a square):
tvbox
"a"
"c"
to get another one, keep hitting c until you get a circle (may take a few times)
When cleaning interactively, in order not to have to redraw the boxes every time:

to create a boxfile, set a filename in oboxfile parameter in imagr (with the logical name ahead of it, same as when you're reading in a fits file)

Then, to use it once you've created in, set boxfile=oboxfile. This will read in the file, allow you to modify it, then spit it out again (with any modifications). Can leave this set in imagr through all the steps that follow

REBOX _not_ TVBOX to add boxes when you're working with a file--otherwise it will delete everything you already have

Stop cleaning when the image looks about uniform. A good way to decide when to stop is when the min and max (see pasted in example below) are about equal.

IMAGR2: I Polarization model processed
IMAGR2: Field 1 min = -15.4 MilliJy,max = 16.1 MilliJy

In a self-cal cycle, can tell that your image is improving if it includes more visibilities than the last iteration.

NB: The best self-cal model (the model is the image you're making of the UV dataset that comes out of a calib run) is the one that contains the most flux--but beware of boxing dubious sources, since anything you put into the model will create a source around it. If a dubious source gets weaker when you box another, more probably real source, that's a good indication
that it's not real.

Herringbone pattern can be due to amplitude errors, poor UV coverage, the beam shape being poorly defined.

HOW TO DO A PROPER SELF-CAL:
i) To get started, make an image of your original UV database (say BASFIT from UVLSF)and clean it as described above--probably only with on box.

ii) Task 'calib'
Do phase-only first: CALIB SOLMODE ='P '/Solution mode
Feed in as your model (get2n) the map made in (i)
Pick a short solution interval for phase, since phase varies quickly as a function of the atmosphere (say 2
minutes)
CALIB SOLINT = 2.00 /Soln. inter. (min)
CALIB APARM(1) = 4 /Min. no antennas
CALIB APARM(7) = 3.0 /SNR cutoff

Can use a lower S/N cutoff than the default b/c phase is constrained by phase closure.
set OUTCLASS for calib to 'calib' to distinguish the UV files that will come outalso add a _P to the filename to
mark it as phase-cal only

WHAT COMES OUT OF CALIB IS UV DATA WITH THE SN TABLE ALREADY APPLIED

iii) image your _P.CALIB UV file that 'calib' has produced, using imagr

iv) Run calib again. Input is your ORIGINAL UV data. The map (in2name) is the map you made in (iii). see
examples below.
112 CALIB INNAME='S255N_USB ' INCLASS='BASFIT'
113 CALIB INSEQ= 1 INDISK= 1
114 CALIB OUTNAME='S255N_USB_P1' OUTCLASS='CALIB '
115 CALIB OUTSEQ= 2 OUTDISK= 1
116 CALIB /TIMERANG = beginning to end
117 CALIB IN2NAME='S255N_USB ' IN2CLASS='ICL001'
118 CALIB IN2SEQ= 2 IN2DISK= 1
**this is very important. This makes the self-calibration insensitive to errors early on--eg boxing sources that it later becomes apparent you shouldn't have boxed. and it retains flexibility. And means that your original UV data file will accumulate SN tables, and you can correctly copy the last version over to the line data and apply it there. DON'T do iterative self-cal. This is most important with low S/N and poor UV coverage data--certainly the case for SMA.

Repeat steps (iii) and (iv) until the phase is as good as it can get.

Once it is:

v) do an amplitude calibration
NB: amplitude calibration is inherently much less constrained than phase calibration. Never do an amplitude calibration until the phases are fixed as best they can be.
This time, use your last, most correct, .CALIB UV file as an input to task CALIB
set CALIB APARM(7) = /SNR cutoff parameter back to =5 (since amplitude is much less constrained)

Unlike phase, which varies quickly as a function of atmosphere, amplitude should vary slowly, and the variation should be primarily due to instrumental effects
Want to solve for amplitude only once per scan (scan=length of time on source)
(for s255n data, 20 minutes)
set SOLINT=scan length

CALIB SOLMODE ='A&P '/Solution mode

vi) After amplitude calibration, do uvplt as a reality check. If you have crazy amplitude errors, specific baselines may jump way up. (Remember that you're actually changing the amplitudes with the amplitude calibration)

vii) SMA weights are based on tsys. Will (particularly with low S/N, poor UV coverage) get a better image if you apply the weights, rather than pretending all antennas have same tsys (which at SMA they emphatically don't).

So, make an image by running imagr, with your last calib phase file as an input, and applying the SN table (which will be the SN table from the amplitude calibration)
To do this:
142 IMAGR INNAME ='S255N_USB_P1'
143 IMAGR INCLASS ='CALIB'
144 IMAGR INSEQ = 4
(for example, if this is your most last phase cal iteration)
149 IMAGR DOCALIB = 2.00000E+00
150 IMAGR GAINUSE = 1

Read more!

Speeding Up Self Cal (a tiny bit)

When doing self calibration, you have to do lots of iterations of calib and then imagr. IMAGR has been taking forever for me!

Just realized that imaging goes a lot faster if you use the calibrated UV files which are outputted from CALIB, instead of using the orignal uv file and applying the SN table in IMAGR. Every little bit of speed helps!

Read more!