Real-Time Hand Tracking
Recovering the full articulation of human hands from sensor data is a challenging problem due to the hand’s high number of degrees of freedom, the complexity of its motions, and artifacts in the input sensor data. We investigated various different approaches for hand tracking and developed systems that are capable of real-time posture estimation and tracking of human hands based on RGBD input data.
Model-based approaches to hand tracking recover hand movements from RGBD sensors by fitting a virtual hand model to the sensor’s 3D point cloud data [3,5]. Low-cost RBGD devices are easily deployable, but can exhibit strong sensor artifacts (Figure 2). This can lead to errors while computing correspondences between the model and the sensor data. To ensure plausible hand posture reconstructions despite such data flaws, the geometric registration process can be made robust by employing various regularization priors.
Using robust articulated-ICP , hand tracking is formulated as a regularized articulated registration process, in which geometrical model fitting is combined with statistical, kinematic and temporal regularization priors. In this process, an energy encapsulating these combined quantities is minimized in a single linear solve (Figure 3). To account for occlusions and visibility constraints, a registration concept is employed that combines 2D and 3D alignment between the model and the data. 3D point correspondences between the point cloud and the model are computed only with the front-facing parts of the model, which significantly improves the alignment when occlusions occur (Figure 4, Figure 5). In addition to 3D point correspondences, which bring the model close to the data, 2D correspondences between the rendered and sensor silhouettes ensure that the model stays within the visible segment of the hand (Figure 6).
Beyond performing robust correspondence computations and enforcing kinematic constraints, like joint angle limits, temporal coherence and collision detection, statistical analysis of hand motions provides a valuable means of regularizing posture estimations. Performing PCA on a varied dataset of motion captured hand movements exposes the correlations and redundancies present within hand articulations [2,3]. This can be used to derive subspace representations of hand articulations (Figure 7), which can be used to improve the realism of the hand posture estimations in real-time systems when faced with imperfect sensor data [3,5]. The loss of information and flexibility caused by PCA dimension reduction can be compensated for by using an adaptive PCA model  that is adjusted during real-time tracking to account for observed hand articulations that are not covered by the initial hand posture parameter subspace.
Subspace representations of hand articulations allow for the reconstruction of hand motions from sparse sensor data [3,5]. Conversely, they can also be used to infer the minimal amount of input data necessary for robust hand posture reconstruction in the context of optical motion capture [6,7]. We addressed the problem of determining the optimal placement of a reduced number of markers in order to facilitate accurate posture reconstruction from sparse mocap marker data in a method that automatically generates functional layouts by optimizing for their numerical stability and geometric feasibility (Figure 8).
In contrast to model-based approaches, which continuously optimize for the kinematic parameters of the hand, appearance-based hand tracking methods produce hand posture estimations from a single frame based on a database of known configurations. We built a system for hand tracking using a color glove , which is based on matching the glove-wearing hand’s appearance in the sensor data with an image database (Figure 9), and used it for interactive robot teleoperation (Figure 10).
Real-Time Hand Tracking with a Color Glove for the Actuation of Anthropomorphic Robot Hands
Proceedings of IEEE-RAS International Conference on Humanoid Robots, 2012, pp. 262-269
Analysis of Hand Synergies for Inverse Kinematics Hand Tracking
Proceedings of IEEE International Conference on Robotics and Automation (ICRA), Workshop on Hand Synergies, 2013
Real-Time Hand Tracking using Synergistic Inverse Kinematics
IEEE International Conference on Robotics and Automation (ICRA), 2014, pp. 5447-5454.
Online Adaptive PCA for Inverse Kinematics Hand Tracking
Proceedings of Vision, Modeling and Visualization, 2014, pp. 111-118.
Robust Articulated-ICP for Real-Time Hand Tracking
Computer Graphics Forum 34(5), Proc. Symp. on Geometry Processing, 2015, pp. 101-114 (Best Paper Award).
Reduced Marker Layouts for Optical Motion Capture of Hands
Proceedings of ACM Motion in Games, 2015, pp. 7-16.
Fully Automatic Optical Motion Tracking using an Inverse Kinematics Approach
Proceedings of IEEE-RAS International Conference on Humanoid Robots, 2015, pp. 461-466.
Design and Evaluation of Reduced Marker Layouts for Hand Motion Capture
Computer Animation and Virtual Worlds, 29(6), 2018.