One day we may have AI-equipped robot butlers, but first we need them to be able to hand us the right pen off the counter. This task is harder than it sounds and requires vast amounts of math and coding to achieve. But researchers at Brown University are creating robotics that can identify and retrieve objects based on verbal input.

Researchers at Brown University have been working on solving a task that we encounter every day: object retrieval. These types of tasks are something that we humans go through daily. "Hey, can you grab me that pen?" If there are multiple pens on the table, someone might respond "Sure, which one?" You can then direct the person to the one you're asking for.

However, asking a robot the same question is a more complicated issue. When tasked to a robot, the scenario above is extremely difficult in terms programming and mathematics. 

Iorek is a robotics project led by computer scientist professor Stefanie Tellex at Brown's Humans to Robots Lab. Her work focuses on incorporating robots in homes as well as workspaces to help provide assistance in any way possible.


A Matter of Communication

A great place to begin is incorporating a robot's mechanics with learning algorithms. Learning algorithms are pivotal when dealing with something as complex as language. An object retrieval robot must understand our language.

Another layer to this issue is the fact that such a robot would need the capability to respond intelligently so we can understand it in return. For example, imagine that a mechanic repairing a car asks a robot to fetch a specific tool. There will be times, of course, where the robot will not quite understand due to lack of communication or internal errors—just like with humans. A fail-safe to this might be allowing the robot to tell the human that it doesn't understand.

The problem that Tellex and her team face lies with natural language and gestures. Their research states that speech-to-text software frequently introduces transcription errors, and human body trackers perform far worse than a human observer could record. These type of problems leave the robot questioning itself. Clearly, we would like for the robot to ask and clarify rather than stall and do nothing when it is uncertain; to correct for this, the team incorporated a Partially Observable Markov Decision Process (POMDP), which is a framework that is general enough to model a wide variety of real-world sequential decision processes. They call their particular model Feedback to Collaborative Hand-Off POMDP (or "Fetch POMDP") and are tailoring it specifically to handle ambiguous interactions without needing to ask an entire series of questions.


The Importance of Machine Learning for Robotics

The Humans to Robots lab has previously created an algorithm that equips robots with the ability to receive speech commands as well as various information from human gestures. This robot is on the path for passing the Turing Test; a test that measures how good a machine is at imitating human behavior. If the robot is able to pass the test then it's expected that the difference between machine and human interactions will imperceptible.


Iorek's concentration face. Image courtesy of Humans to Robots Lab, Brown University


In order to change the behavior of most robots, a new program must be written onto its computer. With machine learning algorithms, however, a robot may "learn" from its interactions the same way a human might. These interactions can feed into a neural network from which an AI can draw information in a way that is intuitive (at least, intuitive to humans). Iorek is an important step in that direction.

To that end, Iorek will help us achieve robots that can pass us the salt at the dinner table—but its true value is far beyond that. Iorek is also helping us achieve robots that can communicate effectively with humans by "thinking" the way that we do.

From a certain perspective, the future of robotics is looking more like it belongs in a neuroscience lab every day.