Novel object detection by a hierarchy of known parts

Imagine a robot that can help with search and rescue tasks, to find victims, in situations that are dangerous to humans. In the SNOW project, we aim to endow the SPOT robot dog with such a capability. Key is to recognize victims, even if they are only partially visible. For instance, a person may have fallen behind a bed. This perception is done by a SNOW algorithm, called HITCH (Hierarchical Task-specific Characterization).

We have tested HITCH in multiple rooms, with smoke, and victims in various positions. The results are promising: victims can be found even when they are partially visible or covered by smoke.

HITCH works by a taxonomy of human parts. The illustration shows that it is most confident about the head, torso, arm and hand (orange/red nodes). That is a sensible prediction and explanation.

In this video, you can see HITCH in action. It slows down when it finds a person.

This research is part of the SNOW project, within TNO’s Appl.AI research program. It was published at the International Workshop on Cognitive Robotics, 2020 (see Publications).