Beyond Language: Multimodal Semantic Representations


IWCS 2021


Important Dates



Beyond Language: Multimodal Semantic Representations

Welcome to MMSR I, the first workshop on Multimodal Semantic Representations!

The demand for more sophisticated natural human-computer and human-robot interactions is rapidly increasing as users become more accustomed to conversation-like interactions with AI and NLP systems. Such interactions require not only the robust recognition and generation of expressions through multiple modalities (language, gesture, vision, action, etc.), but also the encoding of situated meaning.

When communications become multimodal, each modality in operation provides an orthogonal angle through which to probe the computational model of the other modalities, including the behaviors and communicative capabilities afforded by each. Multimodal interactions thus require a unified framework and control language through which systems interpret inputs and behaviors and generate informative outputs. This is vital for intelligent and often embodied systems to understand the situation and context that they inhabit, whether in the real world or in a mixed-reality environment shared with humans.


This workshop intends to bring together researchers who aim to capture elements of multimodal interaction such as language, gesture, gaze, and facial expression with formal semantic representations. We provide a space for both theoretical and practical discussion of how linguistic co-modalities support, inform, and align with “meaning” found in the linguistic signal alone. In so doing, the MMSR workshop has several goals:

  1. To provide an opportunity for computational semanticists to critically examine existing NLP semantic frameworks for their validity to express multimodal elements;
  2. To explore and identify challenges in the semantic representation of co-modalities cross-linguistically and cross-culturally;
  3. To gain understanding of domains and tasks where certain semantic frameworks (multimodal or not) are most effective and why.


MMSR I is being held online in conjuction with IWCS 2021! Please check out our Call for Papers.

Contact email