Computational Imaging - experiments to date

I've been experimenting with doing things with a stack of images, some of it is virtual focus, some of it synthetic aperture, and some of it involving multiple image layering (aka "multisync"). This seems to have generated an audience who wishes to know more, so I'm writing this in response. Please feel free to add questions in the comments, and I'll answer as precisely as I can.

Multisync

The first thing to discuss is what I call "multisync" for lack of a better term. Suggestions for a better term for this would be greatly appreciated. A "multisync" photo is simply a set of photos layered to show (or hide) a subject across multiple moments in time. A tripod makes it relatively trival to do so. Here's one of my personal favorites:




Here you see Virginia at 2 years old, clearly walking towards a goal. At the time I had something like this in mind, but when the opprotunity presented itself, it was a quck rush to get into position and holding down the shutter to get a continous stream of photos. The results were amazing to me. It's not often that something truely original results from an experiment, but this time I feel like it did.

I didn't have a tripod at the time, so the images were all taken with slightly different angles. I used Hugin to align them, because that is the tool I feel most comfortable with. It's a matter of trading time post-exposure for pre-exposure setup. I'm very happy with this particular trade.

Synthetic aperture

Once I learned about synthetic aperture, I was hooked. This is the thing that got me started on this thread of experimentation. I still tip my hat to Marc Levoy for his work and demos that got me interested.

The basic idea is to trade your time and effort to replace a very large lens to create your own short depth of field. Here's an example:



The basic process is can be reduced to these basic steps:

• take a lot of almost identical photos from slightly different positions
• spend minutes or hours manually adding control points using Hugin
• output the remapped images to a series of TIFF formattted images
• average the results using a python script written for the purpose

I took this set of photos while standing near the Marshall Fields clock at State and Randolph in Chicago, Illinois. I took the pictures while moving left and right and forwards and backwards covering an area of a circle approximately 1 meter in diameter if I recall correctly.

Again, here I use Hugin to trade off pre-exposure alignment for time post exposure. However, in the case of virtual focus, it's pretty much impossible to align things pre-exposure. I imported the photos into Hugin, and chose 4 points on the face as alignment points and allowed Hugin to optimize for Roll, Pitch, Yaw, Zoom and X/Y offset. I'm pleased with the results, though I do wonder how many photos it would take to get a creamy bokeh.

Virtual Focus

Another way to combine photos is to take them from widely varying locations, to create images that would otherwise be impossible to capture using film, because it combines photos taken non-continous locations.

Here is a good example of an otherwise impossible shot using multiple exposures taken while receeding from the subject at 30+ miles per hour:



The compression of distance as the magnification becomes greater to keep the relative size shows so very interesting artifacts in the photo, and in fact I use this shot to help explain the process to others who commute with me.

Hugin - The Process

The process is pretty simple, if tedious. Import all of your exposures into Hugin, using the Images tab to avoid the auto-alignment process. Then manually enter control points between image pairs that are in your desired plane of focus.

Once you've gotten enough points entered and you've managed to optimize the error to an acceptable level (I try to get below 1 pixel of error), you then use the following option on the Stitcher tab:

Output : Remapped images

In remapper options I turn off the "saved cropped images" because my script can't handle them cropped. You then tell it to stitch now, and it will ask for a prefix, I always use the underscore character _ because it's easy to remember.

I usually then run the resulting image through one of the scripts I've written to average the frames. Lately, I use trail.py a lot, because it shows all of the intermediate steps. Here is a listing of the script:

import os, sys
import fnmatch
import Image

mask = sys.argv[1]
count = 0

# Program to average a set of photos, producing merge.jpg as a result
#
# version 0.01
# Michael Warot
#

print mask  
for file in os.listdir('.'):
  if fnmatch.fnmatch(file, mask):
      print file
      count = count + 1
      in2 = Image.open(file)
      if count == 1:
          in1 = in2
      in1 = Image.blend(in1,in2,(1.0/count))
      in1.save("trail"+str(count)+".jpg","JPEG")
in1.save("merge.jpg","JPEG")      

You need to be warned that this script overwrites the output files without asking, so be careful. It also requires the python image library, which might not be installed by default.

Once done, you'll have a set of jpeg images which show the intermediate results, along with merge.jpg which is the average of all frames.

Summary
So, I hope this has been of help. Please feel free to ask additional questions, or point out errors or omissions. Thanks for your time and attention.