Ask a robot “What color is the couch?” after it has moved through a house, and it first has to find the right image among everything it saw. Giving a vision-language model more frames doesn’t reliably help. It also makes the system slower and harder to interpret.

In our paper we focus on selecting the right frames for Embodied Question Answering. We compare three strategies:

  • Scene-centric (global cues): match the whole image to the question.
  • Object-centric with NLP (local cues): pull out the objects the question names and detect them with OWLv2.
  • Object-centric with an LLM: also infer objects the question implies, e.g. “What can I eat?” → fruit, snacks.

The strategies complement each other. Combining them and keeping only a few frames gives an LLM-Match score of 70.8 on OpenEQA EM-EQA. That is 15.5 points above the previous best, a GPT baseline that uses 50 frames. Fewer frames also means less computation and answers you can trace back to a specific image.