A paper from this year’s SIGGRAPH by James Hays and Alexei Efros (Carnegie Mellon) takes a new approach to digital image “healing” or hole filling. This is a problem faced in many situations, whether it’s restoring a damaged picture, eliminating flaws like a bug on the lens, or just replacing unwanted areas of an image – picture, for example, a plate for a motion picture backdrop in which the makers of a period film want to eliminate modern details.
What makes this technique unusual is that it doesn’t look for other areas in the image for texture and detail to insert. The software that the authors developed uses a database of 2.3 million unique photographs and – without human intervention – looks for semanitic matches for the area that they chose to replace.
No human tagging or hints were used. The program looked over the entire database for broad matches, then looked more carefully for the best fit. When finished (after about an hour on a single CPU, or five minutes on a cluster of fifteen), the authors were presented with several results to choose from. Their paper shows how completely different those results often were, which is one of the most interesting aspects of the algorithm. They also show us examples of bad matches, in which existing methods provided better results than theirs.
Overall the results are impressive, especially considering that the software has to figure out a lot about the images it’s “looking” at in order for this to work. The larger the database, the better it does.
I think that there are some inherent problems with actually using this sort of technique. Their huge database was acquired from Flickr accounts and so there would obviously be copyright issues; also, because some of the solutions include people – even crowds of them – a commercial use could produce photographs of people without model releases, or even any way of knowing who the heck ended up in the picture. And because it’s hard to figure how else you could assemble a library of millions of source photographs, I think that’s a pretty large hurdle to jump. Still, very interesting.
