Detecting flow in games using facial expressions

Many games use dynamic difficulty adjustment (DDA) to promote the achievement of flow and consequent positive affective states. However, performance based DDA assume a specific ludic attitude: that of the hard-core gamer. An alternative approach is to apply affective computing techniques to monitor players adjust difficulty to achieve the desired affective state directly. Such an emotion-controlled dynamic difficulty adjustment (EC-DDA) system might be more flexible and achieve better outcomes for a wider variety of players. Current approaches to monitoring affective state such as ECGs or EEGs can be very intrusive. However, monitoring affective state using facial expressions is non-intrusive, and can be done with minimal, generally existing hardware. This paper presents a simple arcade styled game incorporating a webcam and COTS facial expression analytical software. It presents the results of a set of experiments investigating the issues involved in collecting and analyzing facial expressions to determine player affect. Results demonstrate the feasibility of using facial expressions as a mechanism for determining player affect, but also illustrate some of the difficulties inherent in the EC- DDA approach. Specifically, the affects observed are not consistent with a standard interpretation of flow as characterized by high arousal and positive affect. Instead, even in a state of flow, the affect expressed may be flat. In other instances, affect may be highly variable, expressing a range of transitory basic emotions. Preliminary findings support the notion of flow as a complex cognitive state resulting from a cycle of transitions between simple affective states such as frustration and joy.