Conditional Random Field Model and Its Applications In Action/Gesture Recognition

Introduction/Tutorial to CRF

An Introduction to Conditional Random Fields for Relational Learning, Charles Sutton and Andrew McCallum, In "Introduction to Statistical Relational Learning". Edited by Lise Getoor and Ben Taskar. MIT Press. (2006) (PDF)
Classical Probabilistic Models and Conditional Random Fields, Roman Klinger and Katrin Tomanek, Algorithm Engineering Report, TR07-2-013, Dortmund University of Technology, December 2007 (PDF)
- Patrick Nguyen's comments (summarized by xiaodong):
  - Both tutorials are good enough for beginners, but the latter is better because it clearly illustrates the connection between Maximum Entropy Model and Logistic Regression Model.
  - He also emphasizes that the fundamental difference between HMM and CRF is the former models P(X, Y) while the latter models P(Y|X).
  - In literature including these two tutorials, another difference between HMM and CRF is also mentioned, i.e., the topology of the graph models: HMM assume independence of X_i given Y_i while this assumption is removed from CRF so that CRF can use any X to construct features for Y_i. But Patrick said this is difference is not fundamental, since we can also extend HMM to remove this assumption as well.
- Xiaodong's comments: in action/gesture/sequence recognition problems, we are interested in linear-chain CRF models, where each node represents a class label. This type of CRF is good at model the transition among difference class labels but can not model the internal structure of a sequence of the same class. To do this, we need hidden CRF, as follows.

Applications In Action/Gesture/Sequence Recognition

Hidden CRF (hCRF)

Hidden Conditional Random Fields for Gesture Recognition, Sy Bor Wang Ariadna Quattoni Louis-Philippe Morency David Demirdjian, and Trevor Darrell, CVPR 2006 (PDF)
- Xiaodong's comments: hCRF is good to model the internal structure (hidden states) of a sequence, e.g., sub-actions in the action recognition problem. But since hCRF assumes that the whole sequence has a single class label, we have to segment the input sequence such that each sequence includes only one action, before applying hCRF. With this limitation, not only we can not use it for un-segmented data, but also we can not incorporate the inter-class dependencies among difference (action) classes. To model both the internal structure (hidden states) and external dynamics (class labels), we need dynamic CRF, connected hierarchical CRF and hidden dynamic CRF, as follows.

Dynamic CRF

Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data, Charles Sutton, Khashayar Rohanimanesh and Andrew McCallum, ICML (2004) (PDF)
- Xiaodong's comments:
  - A special case of dynamic CRF is explained, namely, factorial CRF (fCRF). I have not read through the whole paper, but just happen to know this paper because of Hedvig's paper in ECCV 2008.
  - In action recognition problem, fCRF models the correlation between action and object at the frame basis, i.e., it assumes that each frame has an action label and an object label, both of which are modeled by a seperate hCRF, and there is an edge between the action and object nodes at each frame. This assumption is somewhat too tight in my opinion. See the comments on connect hierarchical CRF for more information.
Joint Recognition of Multiple Concurrent Activities using Factorial Conditional Random Fields, Tsu-yu Wu and Chia-chun Lian and Jane Yung-jen Hsu, 2007 AAAI Workshop on Plan, Activity, and Intent Recognition (PDF)
Spatio-Temporal Event Detection Using Dynamic Conditional Random Fields, Jie Yin,
Derek Hao Hu and Qiang Yang, IJCAI 2009 (PDF)

Connected Hierarchical CRF (chCRF)

Simultaneous Visual Recognition of Manipulation Actions and Manipulated Objects, Hedvig Kjellstrom, Javier Romero, David Martınez, and Danica Kragic, ECCV 2008 (PDF)
- Xiaodong's comments:
  - One of the work most related to our project, since it takes both the action and objects in to account in defining an activity.
  - In this paper, the authors compared four graph models for action recognition: CRF, hCRF, fCRF, and the proposed chCRF (connected hierarchical CRF). The authors clearly explain the difference between CRF, hCRF, fCRF and chCRF in Section 4.
  - In contrary to the "early fusion" scheme assumed in fCRF, chCRF adopts a "late fusion" assumption, which models the action and object in separate hCRF and there is only one edge to connect the action and object node over the whole sequence. In my opinion, this assumption is somewhat too loose. I think an "intermediate fusion" assumption should be the best, i.e., add connection between hidden states of actions and objects. See the comments of hdCRF for more discussion.

Hidden/Latent Dynamic CRF (hdCRF)

Latent-Dynamic Discriminative Models for Continuous Gesture Recognition, Louis-Philippe Morency Ariadna Quattoni Trevor Darrell, CVPR 2007 (PDF)
Hidden Dynamic Probabilistic Models for Labeling Sequence Data, Xiaofeng YU and Wai LAM, AAAI 2008 (PDF)
- Xiaodong's comments:
  - This is another extension of CRF to model both the external and internal dynamics of sequences. See comments in CRF and hCRF for reasons why we need to model both.
  - hdCRF is may be considered as the "intermediate fusion" scheme in my discussion on chCRF, when we augment it with multiple types of class labels and observations.
  - Both paper discuss the same thing, even the formulas are similar to each other. But why the later one did not cite the former one? Though the application domains are different, the underlying mathematics are the same!

Skip-Chain CRF (scCRF)

An Introduction to Conditional Random Fields for Relational Learning, Charles Sutton and Andrew McCallum, In "Introduction to Statistical Relational Learning". Edited by Lise Getoor and Ben Taskar. MIT Press. (2006) (PDF)
Real World Activity Recognition with Multiple Goals, Derek Hao Hu, Sinno Jialin Pan, Vincent Wenchen Zheng, Nathan Nan Liu, Qiang Yang, Proceedings of the 10th international conference on Ubiquitous computing 2008 (PDF)
CIGAR: Concurrent and Interleaving Goal and Activity Recogn, Derek Hao Hu and Qiang Yang, AAAI 2008 (PDF)
- Xiaodong's comments: This model is interesting in that it models the dynamic of goal (a super/high-level action class) over time in an action sequence. I have not go through the technical details so far, so I do not its relationship to the other CRF models mentioned above.

Paper Related but not Classified

Conditional Random Fields for Activity Recognition, Douglas L. Vail, Manuela M. Veloso and John D. Lafferty, In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, 2007. (PDF)
Conditional models for contextual human motion recognition, Cristian Sminchisescu, Atul Kanaujia and Dimitris Metaxas, CVIU, Vol. 104, pp. 210–220 2006 (PDF)

Software