Sunday, January 18, 2009

Computational Imaging - experiments to date

I've been experimenting with doing things with a stack of images, some of it is virtual focus, some of it synthetic aperture, and some of it involving multiple image layering (aka "multisync"). This seems to have generated an audience who wishes to know more, so I'm writing this in response. Please feel free to add questions in the comments, and I'll answer as precisely as I can.


The first thing to discuss is what I call "multisync" for lack of a better term. Suggestions for a better term for this would be greatly appreciated. A "multisync" photo is simply a set of photos layered to show (or hide) a subject across multiple moments in time. A tripod makes it relatively trival to do so. Here's one of my personal favorites:

Virginia walks with ball and flower

Here you see Virginia at 2 years old, clearly walking towards a goal. At the time I had something like this in mind, but when the opprotunity presented itself, it was a quck rush to get into position and holding down the shutter to get a continous stream of photos. The results were amazing to me. It's not often that something truely original results from an experiment, but this time I feel like it did.

I didn't have a tripod at the time, so the images were all taken with slightly different angles. I used Hugin to align them, because that is the tool I feel most comfortable with. It's a matter of trading time post-exposure for pre-exposure setup. I'm very happy with this particular trade.

Synthetic aperture

Once I learned about synthetic aperture, I was hooked. This is the thing that got me started on this thread of experimentation. I still tip my hat to Marc Levoy for his work and demos that got me interested.

The basic idea is to trade your time and effort to replace a very large lens to create your own short depth of field. Here's an example:

Marshall Fields Clock

The basic process is can be reduced to these basic steps:
  • take a lot of almost identical photos from slightly different positions
  • spend minutes or hours manually adding control points using Hugin
  • output the remapped images to a series of TIFF formattted images
  • average the results using a python script written for the purpose
I took this set of photos while standing near the Marshall Fields clock at State and Randolph in Chicago, Illinois. I took the pictures while moving left and right and forwards and backwards covering an area of a circle approximately 1 meter in diameter if I recall correctly.

Again, here I use Hugin to trade off pre-exposure alignment for time post exposure. However, in the case of virtual focus, it's pretty much impossible to align things pre-exposure. I imported the photos into Hugin, and chose 4 points on the face as alignment points and allowed Hugin to optimize for Roll, Pitch, Yaw, Zoom and X/Y offset. I'm pleased with the results, though I do wonder how many photos it would take to get a creamy bokeh.

Virtual Focus

Another way to combine photos is to take them from widely varying locations, to create images that would otherwise be impossible to capture using film, because it combines photos taken non-continous locations.

Here is a good example of an otherwise impossible shot using multiple exposures taken while receeding from the subject at 30+ miles per hour:

Railroad Signals

The compression of distance as the magnification becomes greater to keep the relative size shows so very interesting artifacts in the photo, and in fact I use this shot to help explain the process to others who commute with me.

Hugin - The Process

The process is pretty simple, if tedious. Import all of your exposures into Hugin, using the Images tab to avoid the auto-alignment process. Then manually enter control points between image pairs that are in your desired plane of focus.

Once you've gotten enough points entered and you've managed to optimize the error to an acceptable level (I try to get below 1 pixel of error), you then use the following option on the Stitcher tab:

Output : Remapped images

In remapper options I turn off the "saved cropped images" because my script can't handle them cropped. You then tell it to stitch now, and it will ask for a prefix, I always use the underscore character _ because it's easy to remember.

I usually then run the resulting image through one of the scripts I've written to average the frames. Lately, I use a lot, because it shows all of the intermediate steps. Here is a listing of the script:

import os, sys
import fnmatch
import Image

mask = sys.argv[1]
count = 0

# Program to average a set of photos, producing merge.jpg as a result
# version 0.01
# Michael Warot

print mask
for file in os.listdir('.'):
if fnmatch.fnmatch(file, mask):
print file
count = count + 1
in2 =
if count == 1:
in1 = in2
in1 = Image.blend(in1,in2,(1.0/count))"trail"+str(count)+".jpg","JPEG")"merge.jpg","JPEG")

You need to be warned that this script overwrites the output files without asking, so be careful. It also requires the python image library, which might not be installed by default.

Once done, you'll have a set of jpeg images which show the intermediate results, along with merge.jpg which is the average of all frames.

So, I hope this has been of help. Please feel free to ask additional questions, or point out errors or omissions. Thanks for your time and attention.


Raz said...

I recently saw your Virtual Focus - Howto on Flickr which was a link someone posted in the autopano pro forum and I was amazed. I have been reading through your whole blog and going through your Flickr page trying to understand the process. I am so glad you finally went into more detail. I am not familiar with python. What image types can your script handle? is there a specific naming convention I should follow? I have more questions, is this where I should ask them?

Mike Warot said...

The python script accepts one input, which is the file name mask, I usually use _*.tif myself, but it should work with jpegs as well.

It's a quick and dirty tool to get an average of a set of pictures and has some pretty severe limitations, including one that all of the pictures must be the same dimensions.

It outputs images trail00.jpg through whatever count it reaches, along with merge.jpg. It overwrites these without asking, so please be careful.

Raz said...

oh ok, I understand know... how about actually capturing the images, do you change the yaw at all as you move the tripod?

Anonymous said...

how does one go about using your script?

Mike Warot said...

None of the examples above involved a tripod. They were all shot handheld, the last while looking out the back of a moving train. I use Hugin to make up for the lack of a tripod in some instances.
The clock photos involved moving the viewpoint to change the yaw angle slightly by moving left and right... It's easier for me to move forward and backward than up or down to change the pitch (with an small change in zoom as a result).

As far as the script goes, my usual command line is

\photos\source\2008\ _*.tif

This is because the script lives in my \photos\source\2008 folder (I haven't moved it yet to this year) and I want to create the trail of images sourced by _*.tif which is the set of files that Hugin outputs.

The resulting set of files includes:

To get used to it, the script and a copy of your photos in a folder, and give it a spin. Be careful with your filenames because it could eat it's tail if you manage to include it's own output. 8)

I hope this helps.

Raz said...

ok, so you don't rotate the camera at all... you pretty much hold it and move your body around slightly, is that right. I thought maybe there was some rotation happening with the camera. I want to thank you for taking the time to answer my questions, and for pretty much introducing me to this type of photography.

Sonus Loci said...

Excuse me Mike, why the script stop at first image? If I have a set of image like DSC1.jpg DSC2.jpg DSCn.jpg the script works only on the first (DSC1) image?

Mike Warot said...

Excuse me Mike, why the script stop at first image? If I have a set of image like DSC1.jpg DSC2.jpg DSCn.jpg the script works only on the first (DSC1) image?

The script expects a wildcard parameter, thus you would tell it DSC*.jpg, and it would find all of them.