Arxiv.pdf

Autonomous Visual Information Seeking (AVIS) is a framework that uses a Large Language Model (LLM) to dynamically strategize the utilization of external tools and investigate their outputs to answer visual questions. AVIS consists of three components: an LLM-powered planner, an LLM-powered reasoner, and a working memory component. The planner determines which tool to use next, the reasoner analyzes and extracts key information from tool outputs, and the working memory retains acquired information. AVIS achieves state-of-the-art results on knowledge-intensive visual question answering benchmarks such as Infoseek and OK-VQA. The framework uses a dynamic decision-making approach, making decisions at each step based on information acquired from previous API calls, and is informed by a user study that collected instances of human decision-making when using APIs to answer visual information seeking questions.