Can a household robot fetch any household item for you in the shortest time possible? LLMs can help as a prior store of world knowledge.

Imaging you have a household robot and a floorplan of your house. The robot during setup, roams around the house and remembers the permanent & less frequent fixtures in the house, e.g. Cabinets, KitchenTop, Dining tables. We call these receptacles and they are grounded onto the floorplan map. Now during your daily life, the robot is tasked to fetch any objects in the household which can exists in one of these receptacles, in the shortest time possible. However, these objects can be from a very large countable set.

If the robot naively visits all the receptacles in a linear order it might be searching very inefficiently. Large language models, which is a store of prior world information, can be used to guide the search. It can help to generate a probability distribution of the existence of the object over the receptacles, which can then be used to search for an optimal ordering of receptacles in which the agent should visit to minimize expected distance traveled before finding the object.

This is the topic of my current research for my Master’s Thesis at NUS, under Prof. David Hsu. This article will updated towards the end of my research around Dec 2023.