Hand Tracking Using Flock of Features Approach

In this approach pyramid-based KLT optical flow based feature tracking is used to track the hand region.

In the present application a initial rectangular box is provided.The hand is placed in the rectangular region and hand color model is learned.This process initializes skin color histogram parameters for the tracking process.The process of initialization learning needs to be triggered manually .


Histogram based approach is used to learn the hand color model.YCrCb color space is used to store the color information.
Once histogram is constructed,given a input image we can obtain a probability mask indicating the likely-hood that the pixel belongs to the hand region.


Local Edge information is used to identify important feature locations.Typically points having strong edges in all directions are chosen and feature points.This is accompanied by choosing points such that minimum pairwise distance is maintained between different feature points.

The skin probability map to determine if the feature point has high probability of being a skin pixel.If the obtained probability at the specified location is greater than specified threshold we retain the feature at location for further analysis.Else we randomly select pixel within the bounding box which satisfy the following constrain.

  1. The selected feature point must have skin probability higher than specified threshold.
  2. The selected feature point must satisfy pairwise minimum distance criteria.

The obtained feature must be located at distance greater than the specified minimum distance from all other feature points

The points in rectangular box are selected randomly and check if  the above criteria are satisfied.This process is carried out a fixed number of times.If the criteria is satisfied then the feature points is replaced with the new feature point.If no suitable match is found then feature points is retained as it is.

At the end of the feature initialization stage we obtain a feature vector consisting of feature location that are required to be tracked.


Once the feature have been initialized we can proceed with feature tracking.At each successive frame the initialized features are tracked using KLT pyramidal optical flow based feature tracked.

The optical flow based tracked provides a prediction of location of feature points in the current frame given the feature location in the previous frame.

To determine if feature initialization is to be carried out or not we calculate the optical flow between successive frames continuously.Some feature may lost in tracking process or provide inconsistent results due to noise.After tracking using optical flow based tracked,the features are filtered using a predefined criteria and inconsistent features are removed from the feature list.If the number of feature fall below specified threshold we perform the feature initialization again.


The primary criteria used to remove the inconsistent feature points is to use a distance criteria.This is a simple way to
identify the feature that have been lost in tracking process or errors occurring in tracking process due to noise and other factors.

If the feature points in successive frames are nearer than minimum distance threshold the point is removed from the feature vector list.Such point’s may corresponds to a stationary feature point however a slight optical flow may be detected due to noise .

If corresponding feature points in the successive frames are farther away than maximum distance threshold then they are ignored.Large distance indicate large movement and we assume that such large values are encountered due to loss of tracking
or noise.

The feature points which satisfy both the minimum and maximum distance criteria are feature points
that have been successfully tracked and are retained for future analysis.


Once a feature vector which have been successfully tracked have been obtained,the first step is to find the location of the median.This location corresponds to feature vector that has a minimum distance from all other feature vectors .This can be easily determined by obtaining the cumulative distance of the each of feature vector to all other feature vectors in the list.The list providing the minimum cumulative distance corresponds to the centroid or the median.


After determining the centroid the bounding box parameters are updated such that the centroid lies at the center of the bounding box.Now feature modification if performed.For each  feature in the list two conditions are imposed
No two features must be closer than the minimum threshold distance and no feature must be farther away from the feature median than maximum threshold distance.If these conditions are violated then the current feature is replaced with a new feature.The new feature is selected such that it has high skin color probability and lies within the bounding box and satisfies the minimum and maximum distance threshold criteria.

Since some feature may be lost during the tracking process they are also replenished .New features are randomly selected from withing the bounding box which have high skin probability.


The above approach was implemented using
JAVACV,OPENCV libraries on Ubuntu Linux and Android Platform


M. Kölsch and M. Turk. Hand Tracking with Flocks of Features. In Video Proc. CVPR IEEE Conference on Computer Vision and Pattern Recognition, 2005.

Mathias Kölsch and Matthew Turk. Flocks of Features for Tracking Articulated Objects. In B. Kisacanin, V. Pavlovic, and T. Huang, editors, Real-Time Vision for Human-Computer Interaction, chapter 9. Springer-Verlag, 2005.


Attached is  a small demo of the implementation