DIALOG

Problem Context

Natural language is the default way for humans to communicate with each other and for one person to delegate tasks to another. Yet (tele)operation of robotic systems is mostly done through physical remote controls and graphical user interfaces that not only require expertise to learn how to use, but also take away an operator’s mobility. Although natural language interfaces have been developed and used in the past, many of them require specific verbiage to effectively function and lack a degree of flexibility.

Solution

The recent advent of Large Language Models (LLMs) and automatic speech recognition improvements offers opportunities for natural language interfaces that can enhance the interaction and communication of operators with (semi)autonomous robotic systems. Such LLM-based interfaces enable a more natural and fluid interaction that is expected to reduce training time, increase (calibrated) trust and observability, and consequently, increase meaningful human control over those systems. As such, this technology can function as an accelerator to support moving from stand-alone operations to collaborating in a team of humans and intelligent (robotic) systems.

Results

DIALOG has developed a proof of concept of an LLM-based agent that supports a human operator in tasking autonomous systems. DIALOG can not only interpret the verbal intentions given by the human operator, but also provide a suggestion for the most suitable (set of) tasks based on these intentions.