3D Motion Capture Business

[FeatureSpace]

(I apologize for my first post being so long)

I’m interested in starting a 3D motion capture (mocap) technology business and wanted to see if any members here are involved in similar businesses.

In the last few years I have developed various image correspondence and photometric based methods for geometry and motion capture. Some of it is truly novel (and might be patented). Some just refinements of existing concepts. These systems operate at speeds of 1 million to 50 million vertices per second, with CPU or GPU processing. No single mocap method is best in every regard. Success seems to come from combining the right sensor hardware and algorithms. Sometimes its best to avoiding algorithm dependence. I’ve even started experimenting with millimeter wave radar and hope to demonstrate a pure radar-based mocap system that would have advantages over optical. All of this is valuable for motion picture, game, security, military and industrial applications.

Given the trends in computing and sensor hardware I know that mocap will someday be capable of capturing billions of vertices per second. It will support features like true normal vector and reflectivity distribution function capture, procedural geometry fitting and appropriate level of detail (like the actor face camera system used by James Cameron for Avatar). The consensus between my colleagues and customers is that 3D sensor hardware throughput is rapidly outstripping the ability of algorithms to process it into a useful output. Therefore my vision is to streamline the CGI pipeline so any customer or filmmaker can execute their creative vision or application faster and at lower cost. Executing this vision mostly requires interfacing the right mocap methods with the right algorithms, on a GPU cluster with the right user interface.

But the near-term barriers to this business venture are:

1) I’m currently in the wrong part of the USA for such business because of my family and my wife’s family. I used to think geography didn’t matter as much. Now I see this has been a huge barrier to networking with potential associates and customers. This can be resolved of course.

2) I am the top software engineer for my employer, overworked, well paid and my employer wants to keep me as much as possible. They know I want to pursue this vision and they want to be involved. But this is potentially a complication to an outside investor because there would then be too many things going on and the investor would want to spin off a company anyway. Not sure if this is the way to go.

3) Although I have connections to investment capital, the best investors want to see a strong team, each member with a strong but different background, not just one guy. This is the way business should be run. I know of one high-flying venture where its all one super brilliant guy doing 90% of the work because they can't find anyone willing or capable to do the work. I don't want to be in that situation.

So the next step is to form a good team. Unfortunately building a team is like a chicken-or-the-egg problem. I need the team and technology to finance the operation. But I might need to finance it first to be able to pay for a team. Or the team works for equity initially. Not sure yet.

I could go to Microsoft, Google, a CGI house or some other company but that has typical large company disadvantages. Shareholders and management often don't appreciate the value of a new revenue stream, especially one that's complex or founded on algorithms. Usually its only the engineers that "get it" well enough to execute the vision properly.

I've concluded its better build up some key technologies, establish the vision can work, build a team, find some revenue channels then raise more capital.

Sorry again for the long post. Does anyone have any suggestions?

 

[Davros]

Dragons Den, or something similar

 

[imtrobin]

It depends on the kind of sum you are raising. If it is not too big, there might be some R&D/entrepreuner grants (depends on your place) that you can apply.

Either way, you need to artifically create the chicken or egg. Remember, investors don't want to invest in the cool technology, they want to invest in technology that there are consumers willing to pay. So what you can also is to find with a production team to do some work to showcase the technology. Getting good reviews and press will help your fund raising.

Either way, good luck on it!

 

[Laa-Yosh]

Avatar's facial capture (as pretty much every other, too) works with an extra layer... they're evaluating the actor's face and translate it into metadata using the Facial Action Coding System (which is basically a library of about 50 elementary facial movements). Then they're using this metadata to drive their actual facial rig. This really is the only way to work with faces different from that of the actor, even if they share a lot of characteristic features (just as with Gollum).

There is no direct transformation data applied and probably never going to be, unless they would try to recreate some 100% - which usually does not make much sense (why not film the actor instead?). Even Digital Domain has changed Brad Pitt's age which had a considerable effect on the actual shapes of the facial expressions. Same will be true for their work on Tron Legacy, where they're de-aging Bridges and Boxleitner for the old Tron and Clu characters - no direct connection, only through a layer of metadata.
The only exception is digital doubles but you usually don't get to see them up close, so there the extra effort is not worth it.

 

[FeatureSpace]

It depends on the kind of sum you are raising. If it is not too big, there might be some R&D/entrepreuner grants (depends on your place) that you can apply.

Finding capital is never really the problem. I'm a co-founder of a 200M EUR hedge fund that is doing very well right now. I have connections to various VCs and angels. But this doesn't actually help.

The problem with any kind of loss-leading business endeavor is proving a revenue channel early enough so that the business does not inadvertently lose such investment, worry partners/employees, take too long to find revenue or otherwise lose the trust of its backing and everyone else. Trust is everything to me. I have seen too many business environments that were financed by some kind of high risk deception or ignorance, sometimes subtle, sometimes not.

So unless I find a good team, put together some kind of demonstration and get a solid endorsement or contract with a filmmaker or game company, I'd rather stay in the hedge fund business (because we have solid revenues) until such point that I personally have enough disposable funds. That's how the fund got started. Had to put together a stellar team so the backers would take it seriously.

Avatar's facial capture (as pretty much every other, too) works with an extra layer... they're evaluating the actor's face and translate it into metadata using the Facial Action Coding System (which is basically a library of about 50 elementary facial movements). Then they're using this metadata to drive their actual facial rig. This really is the only way to work with faces different from that of the actor, even if they share a lot of characteristic features (just as with Gollum).

Yes, the fixed anatomical (bone structure, pigmentation, etc) and pose/movement/deformation (mostly actor independent) components must be separated into two data sets and handled differently before being recombined into a rendered output. It takes a good geometry sensor, some actors, a good bit of software and proper algorithms.

In my opinion, the best way to this requires a solid anatomical model consisting of actor-specific bone geometry overlayed with muscle, adipose and skin. Yes it is extremely computationally complex. Movements are simulated as actual coordinated muscle contractions, not some deformation of a mesh handled by a heuristic algorithms that took a team of animators, lots of money and time to tweak. If done right, actor-specific anatomy and movement can be truly plug-and-play with a minimum of animator work or other tweaking. Plus this has enormous application to medical imaging, especially for a wide range of anatomical segmentation and measurements (for which there is NIH funding and significant revenue).

 

[Laa-Yosh]

Actually, there are two rather important issues with the super-scientific, extremely precise muscle simulation approach.

First is that it'll never be fast enough for an animator to use it in his Maya viewport in real time while working on the animation. And no, there is no way to use any kind of mocap without animator polish and correction, it always has to be touched up in some way.

Second, art directing muscle sims (or any other kind) is extremely complicated and involves way too long feedback cycles. In the case of Avatar, Weta decided to go for blendshapes because that was the best way to adjust the shapes of lips in various expressions to look exactly the way Cameron wanted them
It's one thing if your cloth sim is just 99%, or your smoke is just 90%, but faces are way too important to let the computer decide the final result, and it'll be the same in 2020 as well, IMHO. We may have far better and more streamlined tools but human input is still going to be the decisive factor, so the implementation has to facilitate it.

 

[FeatureSpace]

First is that it'll never be fast enough for an animator to use it in his Maya viewport in real time while working on the animation. And no, there is no way to use any kind of mocap without animator polish and correction, it always has to be touched up in some way.

I look at this in terms of Moores law and the difference between cost of animator labor vs cost per terra-flop. I write general purpose algorithms in CUDA that fully utilize a GTX 285 GPU. The performance/$ is a true game changer. There is no reason to prevent the development of a GPGPU based real time solid modeler that feeds surface geometry to Maya.

Maybe we're talking about different types of mocap? I've done fully dense 20 to 50 million vertex/second face mocap. So has Paul Debevec. My mocap could capture geometry, surface normals and diffuse/specular reflectivity maps. It also captured fine detail like pores and eyebrow hair. One challenge is getting enough camera pixels on the face of a moving actor. Cameron's head mounted camera is an interesting solution to this. I'm not sure if this head video data was used for sparse mocap, dense mocap, to help fit/direct the blendshapes or some combination.

Second, art directing muscle sims (or any other kind) is extremely complicated and involves way too long feedback cycles In the case of Avatar, Weta decided to go for blendshapes because that was the best way to adjust the shapes of lips in various expressions to look exactly the way Cameron wanted them

I'm suggesting the use of high quality and dense mocap to either:

Optimize or fit a solid anatomical model to mocap data. Extract out the muscle driving data. Apply to different or modified anatomy.

Or rely on face feature correspondences to anchor a series of procedural modifications. This is just virtual makeup and prostheses.

Since you mention blendshapes, such a mocap could also determine the blendshape basis shapes, fit weights to the mocap data, but still allow animators to modify the weights.

 

[imtrobin]

Finding capital is never really the problem. I'm a co-founder of a 200M EUR hedge fund that is doing very well right now. I have connections to various VCs and angels. But this doesn't actually help.

Depends on where you are. In my area, investment into games or media related is practically zero. In the OP case where he needs investment to get up to speed before he can get customers, he gets stuck. The "best" solution is to get some startup grants or angel if he can find to get things running.