In this third post explaining my PixInsight workflow, we’ll be taking a look at dealing with Dark Current by creating and applying dark frames using a Canon EOS 500D. I think it is fair to say that creating a good master dark in order to calibrate flat and light frames is one of the more challenging parts of the image acquisition process when using a DSLR. There are many mysteries, a fair few myths and indeed some question as to whether DSLR users should bother with dark frames at all.

Firstly though, it is worth reminding ourselves of what “Dark Current” is. Camera sensors create an image by measuring electrons that are displaced in to the sensor’s electron wells as they gain energy from incoming photons of light. Unfortunately, those electrons can also gain energy from heat in the sensor. Some of this heat may be come from the local environment and some will be generated by the sensor electronics. This process occurs in all exposures, even if the sensor is completely shielded from external light, hence the name “Dark Current”.

Dark current increases both with the length of an exposure (as there is more time for it to accumulate) and with the temperature of the camera sensor (as there is more energy available to displace electrons). The net result is that all pixels in an image will get brighter over time and with increasing temperature. In theory we can create a master dark frame to measure the dark current and quite literally subtract this extra brightness from the image, but the practice for DSLR imagers is somewhat more complicated.

Read on to find out about my method of tackling the problem. (If you’ve arrived part way through, start the series here with Bias Frames).

Purpose of Dark Frames

If we had a perfect sensor that accumulated dark current in every sensor element at the same rate, then we would not need to bother with dark frame subtraction at all. It would be sufficient to move the black point of the image histogram further to the right in order to subtract the same value from every pixel and thus eliminate the unwanted brightening of the image caused by the dark current.

In the real world however, the majority of pixels will accumulate dark current at slightly differing rates. Furthermore, a few cold pixels will accumulate dark current much more slowly than the majority and a few hot pixels will accumulate it much more rapidly. Thus each individual sub-exposure contains a fixed pattern of dark current which cannot be removed by manipulating histogram to subtract a single value from every pixel.

So the first objective of calibration using a dark frame is to accurately reproduce the fixed pattern created by the dark current and subtract it from our image. The standard process when using a set-point cooled astronomical CCD is to cap the camera with a light-proof cover, cool it to the same temperature as used to take the light frames and shoot a series of dark frame exposures of the same length as the lights.

Since no light has hit the camera’s sensor, any values in the resulting dark frames must be there as a result of the dark current (plus the fixed pattern and read-out noise which we covered in Bias Frames). By matching the sensor temperature and exposure length between dark frames and light frames, we can accurately measure the dark current and subtract it from the light frames, thus achieving our first objective.

Our second and equally important objective is to introduce the least amount of additional noise whilst performing dark frame calibration.

Dark current accumulates in a statistically random manner which creates dark current noise. Dark current noise increases in proportion to the square root of the dark current. Therefore one obvious way to minimise dark current noise is to minimise dark current itself!

Again a cooled camera offers significant benefits in this regard; we know that lower temperatures bring less dark current and so there must also be correspondingly less dark current noise, i.e. half the dark current = a quarter of the dark current noise. Since we’re talking about an uncooled DSLR that point may seem irrelevant, but I’ll explain shortly why it does matter.

Note: If you have a cooled sensor and a good quality camera, the dark current might be sufficiently low that you can get away without using dark subtraction at all. Unfortunately this is usually not the case with unmodified DSLRs.

The other means by which we can minimise the effect of dark current noise is to do exactly as we did when creating our bias frames. By taking a large number of dark frames and integrating (stacking) them in to a master dark we can improve its signal to noise ratio (SNR). In this case the “signal” is the fixed pattern element of the dark current and the noise is dark current noise. It is the ratio of dark current fixed pattern to dark current noise that we are interested in increasing.

Integrating multiple dark frames in to a master dark improves the SNR (ΔSNR) for each additional frame stacked as ΔSNR = √ n – √ (n -1) where n is the number of dark frames. You will notice that this is the same relationship as in previous post on Bias Frames. (Note that this says nothing about the absolute SNR of a given master dark, simply how much better one master dark is relative to others built from fewer frames.)

The conclusion is the same as for a bias frame; a master dark constructed from more dark frames has a better SNR, i.e. it better represents the true dark current. After a certain point the improvement is not worth the extra effort required. A stack of 100 frames has a SNR that is 10 times better than a single frame, but to make it 20 times better you would have to stack 400 frames, 30 times better needs 900 frames and so on.

In absolute terms, if we start with less noisy dark frames we can stack fewer of them to reach the same SNR compared with stacking noisier dark frames. For DSLR users, that means the higher the temperature of the dark frames, the more of them we are going to need, so anything you can do to keep the camera cool should make life easier for you.

Okay so that’s the theory but unlike bias frames, for DSLR users there are some practical problems which make it much harder to create a sufficiently large stack of darks.

Taking Dark Frames

The first of those problems is the need to create lots of dark frames with similar temperatures. I needed a reasonably large set of matching darks for the various experiments that I had in mind. As previously noted, this is easier said than done due to the lack of cooling on a DSLR.

The method I used was as follows:

• Firstly I needed to make sure no light could get in to the camera as I planned to take the darks during the day when I had time available. It is worth noting that light can leak in to the camera through the viewfinder or even gaps in the body, and also that many plastic lens/camera caps transmit a fair amount of infra-red light (some of which will be detected by the sensor despite the IR blocking filter in a standard Canon DSLR).
• I also needed to cool the camera to something approaching the temperatures in which I had been imaging. At the time of my experiments, I was imaging in freezing (or nearly freezing) temperatures and the sensor for the light frames was reported to be in the region of 9° to 13°C.
• My solution was to use a mini-fridge (a portable camping type fridge with a small thermoelectric cooler in the base). I evicted my stash of beer and placed the camera inside with cables to the laptop trailing out of the door, then sealed the whole thing up with black duct tape to keep as much light and warm air out as possible. After an hour or two the walls of the fridge were cooler than the reported probe temperatures inside the camera, thus (I hoped) reducing IR leakage through the camera cap as a potential source of problems.
• When I started taking dark frames, I found that the probe temperature was still higher than that for the light frames, so for my second attempt I put freezer ice packs around the camera to supplement the fridge’s rather puny cooler. This enabled me to take darks at similar temperatures to the lights, but I did have to take more care to avoid condensation dripping in to the camera’s electronics.
• Even so I found that after a few ten minute dark frames, the probe temperature would start to creep inexorably upwards. By adding a 20 to 30 minute pause between each dark exposure so the camera could cool I was able to get the probe temperature stay in the region of 9 – 13°C, thus matching the typical probe temperatures of my light frames. (Your own mileage will undoubtedly vary).
• I used APT to run the camera on a programmed sequence of dark frame exposures and pauses, so I could happily set up in the morning and come back at the end of the day having bagged perhaps 15 or so useful darks. It took me several weekends to build up sufficient numbers of exposures at each temperature to continue with the experiment.

Is the Camera’s Temperature Probe Reliable?

So with my entire library of darks in hand, I turned to the first problem. Is it possible to ensure that dark frames are taken at the same temperature as the light frames using an uncooled DSLR?

Canon cameras do have a temperature probe somewhere inside the camera body, and the temperature registered by the probe is recorded in the EXIF data in the RAW image file. Your camera software may contain a utility that allows you to inspect the EXIF data, but if not you could try something like ExifTool. Most usefully, when using an imaging package like APT or BackyardEOS it is also possible to have the probe temperature form part of the image’s file name when it is created.

So the first test was to determine whether the camera’s temperature probe accurately reflects the temperature of the sensor over the duration of an exposure. If so, it would be simple to sort and match dark and light frames by reference to their recorded probe temperatures.

In theory, the median brightness of a dark frame should be proportional to both the sensor temperature and the exposure duration, and in practice this is usually the case for well-behaved CCD cameras.

Note: We use the Median brightness as this is less susceptible to outliers in the image like hot pixels or cosmic ray hits than the Mean (average) brightness would be.

If the temperature probe is reliable, two dark frames with the same median brightness should also have the same EXIF temperature assuming their exposure durations and ISO settings are also equal. To test whether this was applicable to my Canon 500D I measured the median brightness of each of my entire library of 213 dark frames (all of which are ten minute exposures taken at ISO400).

I needed a simple way to measure the statistics of a large set of images. Originally I used ImageJ, available from: http://imagej.nih.gov/ij/index.html

• Using the ‘Plugins -> Analyze -> Batch Measure’ option in ImageJ you can obtain a fair few different statistics from a set of images and save them to a text file for analysis.

Unfortunately I later found that I wanted additional statistics that are not available in ImageJ but are available in PixInsight’s Statistics process. The latter tool can only be used on a single image at a time, and all the cutting and pasting of numbers soon became rather tiresome. The ‘simple’ answer was to take a year out of this project and write the PixInsight BatchStatistics script (details here).

Fortunately my script is now part of the official PixInsight distribution, so you can skip ahead a year in your own experiments! I ran the BatchStatistics script with the following settings:

• Use the ‘Add Files..’ button to add all the dark frames that you want to measure to the file list.
• Uncheck the ‘Normalize’ and ‘Unclipped’ check boxes, and set the ‘Number Format’ to something appropriate. By default PixInsight converts the integer ADU values to its Normalized Real format. If you want the values in camera ADUs instead, then you’d select the appropriate bit depth from the ‘Number Format’ drop-down. Canon DSLRs normally use 12 bit or 14 bit format depending on the model, and most modern CCD cameras use 16 bit format.
• Check ‘Full File Name’, ‘Median’ and ‘Median Absolute Deviation (MAD)’ as the statistics to output.
• Use the ‘Output File..’ button to set a file name and location for the statistics, and choose ‘CSV’ as the ‘File Format’.
• Use the ‘Analyse’ button to generate the statistics. Note that the more statistics options you choose, the longer this process will take, so only use the ones you really need.
• You can then open the resulting output file in Microsoft Excel (though any spread sheet will do, e.g. OpenOffice, Google Drive, etc.) Excel has a wizard which will guide you through importing the CSV formatted data. A sample of the results after importing is shown below:

• Note that the columns headed ‘Median_0’ and ‘MAD_0’ are the median and median absolute deviation of the first channel in each image. These greyscale dark frames only have a single channel (channel 0).
• Next I manually added a column ‘Temp’ and entered the EXIF temperature of each dark frame, which I obtained from the file name as recorded by APT.
• I also created a ‘Date-Time’ column and entered the date and time of capture of each dark frame, again from the file name.
• Finally I sorted the entire list in ascending order of capture date and time (using Excel’s ‘Sort’ function).
• A small section of the final set of data in 14 bit ADU format is shown below:

Checking for Outliers

Next I plotted the average brightness (Median) of each image against the temperature reported by the camera sensor (Temp) on a line graph to obtain the following:

So what does this tell us?

• The EXIF temperature (orange line) fluctuates quite markedly over time. Bear in mind that these dark frames were captured in multiple sessions and not in a continuous sequence, so we’d expect some big temperature jumps between the end of a session and the start of the next one. Nonetheless, on examining the data, it was quite clear that the reported temperature fluctuated a lot over the course of individual sessions despite my best efforts to control it.
• The median brightness (blue line) stayed relatively stable, except for a small spike in one of the early sessions at the left of the graph and a really big spike in another session at the middle of the graph.

These two spikes in the medians clearly warranted attention, so I looked more closely at the dates and times of the relevant frames.

• It was clear that frames in the big spike were taken from shortly before sunrise until a few hours after it. I wracked my brain and recalled that I had left my scope and camera set up taking darks at the end of an imaging session, but had clearly not made it out of bed early enough to stop daylight leaking in to the camera.
• Similarly the frames in the small spike were a few taken during the middle of the day at a time before I started my camera-cooling experiments in the light-proof fridge.

Notice that the EXIF temperature increased significantly for these ‘outlier’ dark frames. This is not surprising as I was not making any effort to keep the camera cool. What is important are the equally large spikes in the temperature at other times during dark frame capture.

The first lesson we have learned is to be wary of relying on the camera’s temperature sensor alone to judge the quality of dark frames. By measuring the images statistically, it was relatively easy for me to identify and weed out any rogue frames cause by poor technique or other transient issues.

Analysing the Dark Frames

Next I pruned my set of results to remove those outlier frames that had clearly suffered from external light leaking in to the camera. Plotting the only the good ‘inlier’ frames, I got the following:

This graph shows that the median brightness (blue line / left axis) does fluctuate with temperature, but on a relatively small scale. Generally speaking, as the temperature increases the median brightness decreases.

The graph below shows this relationship between temperature and median brightness much more clearly (note that the darker orange data points are where many frames have the same values):

As you can see, the brightness of a dark frame reduces as the temperature increases. This is exactly the opposite of what we would expect if we assume that the EXIF temperature corresponds to the true temperature of the sensor.

In order to test this further, I next plotted the Median Absolute Deviation (MAD) of each frame against the EXIF temperature:

The MAD is a measure of variability of the individual pixel values in an image. Given that my dark frames are a set of (nearly) identical images, any change in the MAD between images should be proportional to the amount of noise that each contains. Given that dark current and dark current noise both increase as temperature increases, the MAD enables us to double-check the strange result above. You can see from the graph above that the MAD (generally) increases with temperature as we’d expect.

Note: We use the MAD in preference to the Standard Deviation or Variance as it is a more robust estimator of scale, i.e. outliers like hot pixels and cosmic ray hits won’t affect our results as much.

Again, the graph below shows this relationship more clearly:

So we now have contradictory findings; the MAD increases with temperature, but the median unexpectedly decreases. Why should that be?

On-Camera Processing

To be perfectly honest, I wasn’t that surprised. As noted by Craig Stark in his article Profiling the Long-Exposure Performance of a Canon DSLR, the RAW image files produced by Canon DSLRs are not the same as the ‘pure’ output files created by an astronomical CCD camera. The Canon pre-processes the image on-camera before creating the RAW file, whereas a CCD will typically give you an unadulterated measurement direct from each sensor element.

You can read the details in Craig’s article, but in short, the Canon DSLR seems to try to minimise the appearance of dark current. In his tests, Craig found that dark frames get darker as the exposure length increases, at least for shorter exposures. He also found that the trend reversed for longer exposures with the dark frames brightening again. My own tests reproduce the first part of this result, showing that the appearance of dark current decreases as sensor temperature increases. In both types of test, measuring the dark current noise shows that more dark current has accumulated (as required by the laws of physics!), but that the camera processes the image to make it appear darker instead of brighter.

On a more positive note, it is clear that for my Canon 500D, the temperature sensor does bear some relationship to the imaging sensor, as demonstrated by the “EXIF Temperature vs MAD” graph above. Subject to the caveat about eliminating outliers noted above, in the case of this particular camera it would be reasonable to use the reported temperature to match dark frames together in order to create a master dark.

That said, there are several reasons why the temperature sensor in other camera models may not always be  an accurate guide to the mean temperature of the sensor during the whole exposure:

• The probe may not be located sufficiently close the sensor to give a useful reading.
• The probe may not heat up and cool down at the same rate as the sensor, e.g. it may be attached to something that has a large thermal mass that initially slows the temperature rise of the probe and subsequently reduces the rate at which it cools vs the rate of heating/cooling of the sensor itself. Alternatively it may be near another component that generates its own heat (processing chips, the battery, etc.)
• There have been some reports that the main purpose of the temperature sensor in Canon DSLRs is to monitor the chips used to process live view and movie modes (in order to shut the camera down should they overheat due the intensive processing needed).
• For Canon cameras, the probe temperature is recorded at the start of each exposure. Since the camera is not cooled, the sensor may heat up during a long exposure. Thus the temperature measured at the start doesn’t necessarily reflect the temperature at the end of the exposure. (Though we might look at the temperature at the start of the next exposure to get a rough idea of the temperature at the end of the previous one provided we don’t wait too long between exposures.)
• We don’t know if the sensor heats up linearly during the exposure. If it does then we could average the start and (rough) end temperatures of each exposure to match darks and lights which have the same averages. If the sensor heats non-linearly or unpredictably, then we may be wasting our time. Bear in mind that it is usually the case that much more heat is generated when the image is being read out of the sensor at the end of an exposure, suggesting a non-linear heating/cooling cycle.
• The overriding problem is that we have no practical way to measure any of this, short of disassembling the camera and installing our own temperature probe attached to the imaging sensor.

If you perform the same sort of tests that I have and you discover a clear correlation between the MAD of your dark frames and the EXIF temperature, then you can use the temperature to match dark frames together in order to create a good master dark.

Alternatives to Temperature

If you don’t find such a correlation, then try plotting the medians against the MADs as in the graph below:

Unsurprisingly, you can see that there is a clear correlation between the median brightness and the MAD (albeit the line goes down instead of up due to the Canon’s on-camera processing).

• If your own graph shows a similar correlation, then you can try to match dark frames using the MAD instead of the EXIF temperature, as I’ll show in the next part of this series.
• If there is no clear correlation between these two values then you’re out of luck. It might be that your DSLR is also performing some kind of noise reduction on the image before you get hold of it. All you can do is take your best guess at matching dark frames to make a master and test their effect on your images. Again I’ll show you how to do this later in the series.

The key conclusion is that you need to use some simple statistics to check the dark frames produced by your camera. Be wary of relying on recipes provided by others and perform your own tests for your own camera.

In the next part of this series, we’ll look at creating and testing master darks, as well as methods for matching DSLR dark frames to light frames for best results.