Tracking Pose in Three Dimensions
People can easily interpret the 3D motions of a dancer from a 2D video clip, even though multiple 3D motions could give rise to a particular 2D projection. The secret is to resolve the ambiguity by applying background knowledge about how people move in the everyday world. By giving the computer experience with typical human motions, one can create an automated system that will track a person's motions in 3D from a single camera input.
My approach to pose tracking draws heavily on my experience in image retrieval: each frame of a video is treated as a query image, used to locate similar synthetic images whose pose is known. By stringing together a plausible series of such retrieved poses over time, the entire motion sequence can be reconstructed.
One of the students working on this project has created a page with pictures of Smith's Gypsy 3 motion capture suit in action.
Silhouette Lookup for Monocular 3D Pose Tracking, N. Howe. Image and Vision Computing, special issue on human motion (to appear). Preprint: [PDF] [PS.GZ] [BibTeX].
Boundary Fragment Matching and Articulated Pose Under Occlusion, N. Howe. IV Conference on Articulated Motion and Deformable Objects, July 2006. [PDF] [PS.GZ] [BibTeX] [Video clips] [Talk].
Flow Lookup and Biological Motion Perception, N. Howe. International Conference on Image Processing, September 2005. [PDF] [PS.GZ] [BibTeX] [Video clips].
Silhouette Lookup for Automatic Pose Tracking, N. Howe. IEEE Workshop on Articulated and Nonrigid Motion, June 2004. [PDF] [PS.GZ] [BibTeX].
Bayesian Reconstruction of 3D Human Motion from Single-Camera Video, N. Howe, M. Leventon and W. Freeman. Advances in Neural Information Processing Systems 12, 2000. [PDF] [PS.GZ] [BibTeX] [PowerPoint].
This material is based upon work supported by the National Science Foundation under Grant No. 0328741. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Research links: Overview | Handwriting | Segmentation | Retrieval