Let me start by reading through the abstract and the sections provided.
First, the abstract mentions that embodied conversational agents (ECAs) benefit from non-verbal behavior, especially gestures. The paper presents a framework to evaluate modern gesture generation models in real-time interactions. The framework has three components: a 3D agent, a chatbot backend, and a gesticulating system. It's open-source and extensible, allowing replacement of each component for different studies. The code and video are available on their project page.
In the introduction, they talk about non-verbal communication in humans and how ECAs can be more engaging with appropriate gestures. Current ECAs use pre-recorded animations or handcrafted motion specs, but recent data-driven methods can generate realistic gestures. The challenge is evaluating these models in real-time, which this framework aims to address.
The system description details the three components. The Unity 3D agent allows interaction via voice or text. The chatbot backend includes speech recognition, a neural model, and text-to-speech. They demonstrate two configurations: one using DialogFlow and another with BlenderBot and Glow-TTS. The gesture generation model uses Gesticulator, which takes acoustic and semantic inputs to generate joint angles. They adjusted the model by using Geneva's acoustic parameters, approximating word timing with syllable counts, and replacing BERT with Fast-Text for efficiency.
The limitations section is cut off, but I should note that the current framework has some constraints. The authors mention that the system is modular and can be adapted for various studies, but there might be issues with real-time performance or specific model integrations.
I need to make sure the summary is in English, concise, and covers all key points without extra details. Start with the main purpose of the framework, the three components, the technologies used (Unity, DialogFlow, BlenderBot, Gesticulator), the adjustments made, and the availability of the code. Also, mention the evaluation aspect and the potential for future research. Avoid any markdown and keep it straightforward.
A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents presents an open-source, extensible system designed to evaluate data-driven gesture generation models in real-time interactions. The framework comprises three core components: a 3D interactive agent (built in Unity), a chatbot backend (supporting text-to-speech and speech recognition), and a gesticulating system that generates motion based on speech. Each component is modular and replaceable, enabling experimentation with different gesture models, chatbot architectures, or agent appearances. The system leverages Apache ActiveMQ and STOMP protocols for inter-component communication. Key innovations include adapting the Gesticulator model to use extended acoustic parameters, approximating word timing with syllable-based speech duration, and replacing BERT with Fast-Text to improve efficiency. The framework demonstrates flexibility through two chatbot implementations: DialogFlow for voice-based interaction and BlenderBot with Glow-TTS for free-form dialogue. The code and video are publicly available at https://nagyrajmund.github.io/project/gesturebot. The work addresses the challenge of assessing gesture generation in dynamic, real-time settings, accelerating development of embodied agents with integrated non-verbal behavior. Limitations include current constraints in real-time performance and specific model adaptations.
