Multimodal Communication on Social Platforms

Building machine learning models to detect sarcasm and satire across text, images, and video, and studying how people ask questions and seek information on social platforms.

← All research projects

Outcome

ICCBD 18, ICMI 18, HCC 19, TransAI 19, TransAI 19, APSEC 18, OzCHI 18, and MS Thesis

Overview

My earlier research addressed the technical challenge of building machine learning models that can recognize the nuance and intent behind complex human communication, such as sarcasm and satire, and studied how people ask questions and seek information on social platforms. This work shaped a question that runs through my current research: meaning depends on context, and systems that ignore context misread people.

Approach

We used supervised learning to develop multimodal models that go beyond text alone, integrating visual cues from images, patterns of user interaction (such as comments and reactions), and narrative trajectories through neural networks. We grounded this computational work in psycholinguistic theory to define the structural characteristics of sarcasm and humor in social media settings.

Key Contributions

We showed why sarcasm cannot be reliably detected from text alone and often depends on visual and contextual cues, offering an early demonstration of multimodal sarcasm detection across text, images, and videos. Our datasets and models showed that user interactions (e.g., reactions, comments) provide important signals for understanding sarcastic intent, and connected sarcasm and satire to broader questions of misinformation, interpretation, and meaning in digital content through tone and storytelling style. Related work modeled question difficulty on Stack Overflow and examined whom people tag, and why, when asking questions on social media.