Sponsored by

Workshop Papers Downloads

Work in progress:



Multimodal, as well as unimodal, interactive data from behavioural signals such as speech, facial expression, mouse and pen input has recently been shown to provide indications as to the cognitive and emotional state of the user. Similarly, research into physiological signals such as electroencephalograph (EEG), galvanic skin response (GSR), electromyograph (EMG), heart rate, respiration and eye-movement detected by a camera-based tracker has demonstrated differences for contrasting cognitive states, e.g. the completion of complex or simple tasks. By studying individual physiological, interactive and other behavioural features, and also their fusion/integration, new holistic insights can be provided into the physical manifestations of various cognitive and emotional states - including affect, stress and workload factors.

Scope and Expected Impact

The use of multiple sensors in human-computer interaction (HCI) systems allows the collection of a wide variety of datasets. This, together with an array of machine learning techniques can shed light on implicit behavioural measures, and the kinds of cognitive and emotional states that can give rise to these signal patterns. The amount of data that is being collected now makes this challenge a very topical one. There are no standards by which multimodal data can be collected and labelled (interactive or otherwise), and there are no standards by which multimodal data can be induced and therefore interpreted. This full-day workshop will provide a unique opportunity to define and shape the future research agenda in unimodal/multimodal cognitive and emotional inference, as a community, acting as a focal point and stimulus for significant new research and collaboration. The development of systems based on these research insights can be expected to have a dramatic effect on future interfaces, opening up a new world of interaction and joint human-computer collaboration that is optimised to exploit the unique capabilities of each. Other related avenues of investigation include exploring the relative merits of different sensors, and correlations between features extracted from different sensor signals.


Guest Speakers

Prof. Sharon Oviatt (Incaa Designs, USA)
Sharon Oviatt currently is President of Incaa Designs, a nonprofit 501(c)3 organization that researches, designs, and evaluates new educational interfaces. Her research focuses on human-centered design and cognitive modeling, communication technologies, spoken language, pen-based and multimodal interfaces, mobile and educational interfaces. Examples of recent work involve the development of novel design concepts for educational interfaces (science and math), robust interfaces for real-world mobile and field environments, adaptive conversational interfaces with animated software characters, digital paper and pen-based interfaces for individuals and groups, and lifespan interfaces for children through the elderly. She has published over 140 scientific articles in a wide range of venues, including work featured in recent special issues of Communications of the ACM, Human Computer Interaction, Transactions on Human Computer Interaction, IEEE Multimedia, Proceedings of IEEE, and IEEE Transactions on Neural Networks. She was General Chair of the International Conference on Multimodal Interfaces (ICMI) in 2003, and is Founding Chair of ICMI's Advisory Board. In 2000, she received an NSF Special Extension for Creativity Award for pioneering work on mobile multimodal interfaces.
Prof. Peter Gerjets (KMRC, Germany)
Prof. Gerjets is currently head of the Hypermedia Lab, at the Knowledge Media Research Center in Tübingen. Since 2002 Peter Gerjets has been working there as principal research scientist, and as full professor for research on learning and instruction at the University of Tuebingen. Before taking over his current position at the University of Tuebingen, Peter Gerjets has been working as Assistant Professor at the Saarland University in Saarbruecken. He was honored with the Young Scientist Award of the German Cognitive Science Society in 1999. From 1997 to 1998 Peter Gerjets was granted a postdoctoral scholarship of the DFG-Graduate College for Cognitive Science in Saarbruecken. From 1991 to 1995 he was a Research Associate at the University of Goettingen. Peter Gerjets received his PD (Habilitation) from the Saarland University in 2002 with a thesis on "Cognitive control in hypertext-based learning and problem solving: Goal management, strategy selection, and adaptive information utilization". In 1994 he received his Ph.D. (Dr. rer. nat.) from the University of Goettingen where he also received his diploma in psychology in 1991. His research interests are related to earner control, cognitive load, adaptive user strategies, learning with dynamic visualizations, designing multimedia learning environments and supporting different learners' competences to use such environments.

Topics of Interest


Paper Submission

The workshop solicits original and unpublished papers that address a wide range of issues concerning, but not limiting to the list of topics. Submissions must be sent in PDF to the following email address: MMCogEmS2011@nicta.com.au. Authors should submit papers not exceeding 4 pages in total and in the ACM format (http://www.acm.org/sigs/pubs/proceed/template.html).

Accepted papers will be presented at the workshop in either oral or poster format and will appear in the ICMI conference proceedings. Note that accepted papers will not be automatically published in the ACM Digital Library, but selected authors will be invited to submit an extended version of their papers to a journal special issue. At least one author of each paper must register the ICMI conference and attend the workshop to present the paper. Please refer to the ICMI 2011 website for more information about registration.




Program Committee



We thank the Air Force Office of Scientific Research for its support of this workshop.