I ran through the questions here and compared them to our notes to see what else I can share. Most of this first part is more on the theoretical side, but I thought it might still be interesting.
Photogrammetry is the most common scanning method nowadays for the following reasons:
- it's very cheap, especially for static subjects (a single camera and the software is enough, and a hobbyist Agisoft license is like $180)
- it's the most flexible and scalable; the number of cameras and their placement is basically the only factor in what size and level of detail you can scan, and you can also freely re-arrange everything at any time
- it works with widely available commodity equipment so you can build everything on your own; DSLR cameras and related elements (cables, flashes etc) are usually enough, but our rig is also using raspberry pi minicomputers, for example
- it's also easily portable, for example our guys regularly make trips to Korda Studio here in Budapest, which is where movies like The Martian where shot
- one minor drawback is that it's not super accurate; but that is only really important for industry utilization, stuff like analyzing structural integrity on the used Falcon rockets at SpaceX and so
Other methods are:
- digitizing armatures were available from early on; it's basically a pen at the end of a robotic arm, where the computer can calculate the 3D coordinates from the pose
- laser scanners have several main variations: static ones have a big turntable that rotates the subject (live or object) and the laser is directed by mirrors and such; these tend to be really big and unmovable
- there are also handheld laser scanners, but they work really badly with subjects that can move (and even the simple act of breathing will ruin live subjects)
- there are also various kinds of LIDAR, which basically substitutes the radio waves in radar with a laser; there are ones that look like a camera on a stand and there are ones that you can put on a helicopter to scan entire cities
- structured light scanners project various patterns (usually bands with varying width) and then capture photographs of the subject to analyze; the upside here is that even a single 'camera' can capture a lot of 3D detail
But these systems are usually less flexible, less portable (except the helicopter fitted LIDAR
and much more expensive
- and of course the Lightstage systems, which are mostly a combination of photogrammetry, structured light, and computer controlled lighting
The theory behind stereo photogrammetry isn't really new; it's basically about using at least two cameras to capture imagery, then running optical flow software to identify tracking points on both of the images and triangulating their position to generate a point cloud. Today's software is a breakthrough mostly because it can process hundreds of images and it can work with data from common DSLR cameras (or even mobile phone captured images).
But scanning isn't an actual production tool - all it can do (but really really well) is to provide source data for the rest of the asset building pipeline. The raw data has to be translated into usable geometry and various textures like color, normal and displacement data. It is possible to write software to automate a lot (or even practically all) of the processing, and the various content creation apps available also have a lot of tools for that (like automatic mesh building).
It's also worth noting that many things are impossible (or at least very hard) to scan with photogrammetry:
- shiny or transparent objects, or objects without any kind of surface patterns (although if you spray something on them, it can still work)
- anything that does not exist in real life (fantasy creatures for example)
Also, the fact that photogrammetry rigs are super flexible means that there are no clear blueprints to build them, you have to experiment a lot. Our team for example decided to contact FilmFX, a local practical effects studio. They're pretty amazing guys with credits on stuff like Hellboy 2 sets or Matt Damon prosthetics and the Pathfinder in The Martian. We've got some super realistic human head props, because we knew they'd stay completely still and so the guys can experiment with the camera placement with a single DSLR, before buying all the equipment. The funny story was that we've gotten a severed head prop from the BBC Robin hood show, and of course the poor cleaning lady found it in a cupboard one night, nearly getting a heart attack
It's also like an early Christmas for the team to unpack the equipment, like getting dozens of cameras at once And it took a LOT of hard work to build the rig, lots of people spent a lot of time soldering and rigging cables and such.
And you also have to build the studio itself - a large enough room that can fit the subject and the camera rig, with lighting as even as possible, using white painted walls and floors, and camera synced flash lights.
I forgot to mention in the previous post - with the dentist chair stuff - that it's also quite complicated to scan live animals. There are obviously cases where you can get a stuffed one, but for stuff like cats and dogs, or rare birds, or (although we haven't tried it yet) for things like horses, it can get really interesting and funny.
Another thing worth mentioning is that processing the source data requires a LOT of computing power. Most single scans take hours to generate the point cloud mesh, so dozens of FACS scans can take days to go through. Fortunately it can be automated to a high level, but it's still time consuming. And even with a LOT of cameras, it's still not good enough without some manual work, and you're gonna need relatively good artists for that.
Photogrammetry is the most common scanning method nowadays for the following reasons:
- it's very cheap, especially for static subjects (a single camera and the software is enough, and a hobbyist Agisoft license is like $180)
- it's the most flexible and scalable; the number of cameras and their placement is basically the only factor in what size and level of detail you can scan, and you can also freely re-arrange everything at any time
- it works with widely available commodity equipment so you can build everything on your own; DSLR cameras and related elements (cables, flashes etc) are usually enough, but our rig is also using raspberry pi minicomputers, for example
- it's also easily portable, for example our guys regularly make trips to Korda Studio here in Budapest, which is where movies like The Martian where shot
- one minor drawback is that it's not super accurate; but that is only really important for industry utilization, stuff like analyzing structural integrity on the used Falcon rockets at SpaceX and so
Other methods are:
- digitizing armatures were available from early on; it's basically a pen at the end of a robotic arm, where the computer can calculate the 3D coordinates from the pose
- laser scanners have several main variations: static ones have a big turntable that rotates the subject (live or object) and the laser is directed by mirrors and such; these tend to be really big and unmovable
- there are also handheld laser scanners, but they work really badly with subjects that can move (and even the simple act of breathing will ruin live subjects)
- there are also various kinds of LIDAR, which basically substitutes the radio waves in radar with a laser; there are ones that look like a camera on a stand and there are ones that you can put on a helicopter to scan entire cities
- structured light scanners project various patterns (usually bands with varying width) and then capture photographs of the subject to analyze; the upside here is that even a single 'camera' can capture a lot of 3D detail
But these systems are usually less flexible, less portable (except the helicopter fitted LIDAR
and much more expensive- and of course the Lightstage systems, which are mostly a combination of photogrammetry, structured light, and computer controlled lighting
The theory behind stereo photogrammetry isn't really new; it's basically about using at least two cameras to capture imagery, then running optical flow software to identify tracking points on both of the images and triangulating their position to generate a point cloud. Today's software is a breakthrough mostly because it can process hundreds of images and it can work with data from common DSLR cameras (or even mobile phone captured images).
But scanning isn't an actual production tool - all it can do (but really really well) is to provide source data for the rest of the asset building pipeline. The raw data has to be translated into usable geometry and various textures like color, normal and displacement data. It is possible to write software to automate a lot (or even practically all) of the processing, and the various content creation apps available also have a lot of tools for that (like automatic mesh building).
It's also worth noting that many things are impossible (or at least very hard) to scan with photogrammetry:
- shiny or transparent objects, or objects without any kind of surface patterns (although if you spray something on them, it can still work)
- anything that does not exist in real life (fantasy creatures for example)
Also, the fact that photogrammetry rigs are super flexible means that there are no clear blueprints to build them, you have to experiment a lot. Our team for example decided to contact FilmFX, a local practical effects studio. They're pretty amazing guys with credits on stuff like Hellboy 2 sets or Matt Damon prosthetics and the Pathfinder in The Martian. We've got some super realistic human head props, because we knew they'd stay completely still and so the guys can experiment with the camera placement with a single DSLR, before buying all the equipment. The funny story was that we've gotten a severed head prop from the BBC Robin hood show, and of course the poor cleaning lady found it in a cupboard one night, nearly getting a heart attack
It's also like an early Christmas for the team to unpack the equipment, like getting dozens of cameras at once
And it took a LOT of hard work to build the rig, lots of people spent a lot of time soldering and rigging cables and such.And you also have to build the studio itself - a large enough room that can fit the subject and the camera rig, with lighting as even as possible, using white painted walls and floors, and camera synced flash lights.
I forgot to mention in the previous post - with the dentist chair stuff - that it's also quite complicated to scan live animals. There are obviously cases where you can get a stuffed one, but for stuff like cats and dogs, or rare birds, or (although we haven't tried it yet) for things like horses, it can get really interesting and funny.
Another thing worth mentioning is that processing the source data requires a LOT of computing power. Most single scans take hours to generate the point cloud mesh, so dozens of FACS scans can take days to go through. Fortunately it can be automated to a high level, but it's still time consuming. And even with a LOT of cameras, it's still not good enough without some manual work, and you're gonna need relatively good artists for that.
