ICML 2024:
InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks

In this section, we furnish a comprehensive evaluation of both close-source LLMs such as GPT-4 and GPT-3.5, as well as widely-utilized open-source LLMs. In addition, we test our DAAgent, which is an agent for data analysis with instruction-tuning.

The advent of Large Language Models (LLMs) has spurred the development of LLM-augmented Autonomous Agents (LAAs). These agents are capable of generating and executing code through ongoing interactions between their core LLM and the code execution environment. In this project, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents in data analysis tasks. This benchmark contains DAEval, a dataset consisting of data analysis questions derived from CSV files, and an agent framework to evaluate LLMs as data analysis agents. This page describes the details of InfiAgent-DABench framework, including features such as dataset construction, evaluation metrics, analytical assessment, and the procedural details about pipeline onboarding.

We build data analysis query and response given existing csv files. Here is the construction pipeline. We split the dataset into a validation set and a test set. The validation dataset contains 311 questions with 55 csv files. We only public the validation set to avoid data leakage. Here're some examples: We categorize CSV files within the dataset into nine distinct categories, determined by their respective domains:

• Finance and Economics
• Health and Medical
• Demographics and Social Science
• Marketing and Consumer Behavior
• Energy and Environmental Monitoring
• Transportation, Logistics, and Tourism
• Culture, Entertainment, and Media
• Scientific Research and Technology
• Other Categories

Below is the pie chart depicting the categorical distribution:

We conduct statistical analyses on the individual concepts associated with each question, accounting for scenarios where a question encompasses multiple concepts:

For closed-form questions, we prompt LLMs with question description. Considering that most models hardly follow the format requirements, we add a reformat step for all models by using gpt-3.5-turbo-16k to format the responses given the format requirements. Here's a figure illustrating this process:

For closed-form questions, we have defined the following metrics: Proportional Accuracy by Subquestions (PASQ): $$ \text{PSAQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{M_i} \sum_{j=1}^{M_i} I_{ij} \right) $$ Here, $N$ is the total number of questions, $M_i$ is the number of subquestions for the i-th question, and $I_{ij}$ is the indicator function for the j-th subquestion of the i-th question. Accuracy by Questions (ABQ): $$ \text{ABQ} = \frac{1}{N} \sum_{i=1}^{N} \left( \prod_{j=1}^{M_i} I_{ij} \right) $$ In this expression, the product $\prod_{j=1}^{M_i} I_{ij}$ equals 1 if all subquestions of the \(i\)-th question are answered correctly, and 0 otherwise. Uniform Accuracy by Subquestions (UASQ): $$ \text{UASQ} = \frac{1}{\sum_{i=1}^{N} M_i} \sum_{i=1}^{N} \sum_{j=1}^{M_i} I_{ij} $$ Here, the total accuracy is the sum of the values of the indicator function across all subquestions, normalized by the total number of subquestions in the dataset.

Setup

1. Please initialize the environment using the following code:

conda create -n agent python==3.9.12
pip3 install -r requirements.txt

2. We also build a Python code sandbox in our pipeline based on docker, you can use the following code to build your docker image:

docker build -t myimg .

Example 1: Demo Usage

1. You can easily use the following command to start a demo using APIs.

# initialize poetry
poetry init
# Supported LLM: OPEN_AI, AZURE_OPEN_AI
# api_key is required for API-based models
bash run_demo.sh --llm AZURE_OPEN_AI --api_key 123

2. After running the above code, an interactive frontend interface will be displayed.

3. You can enter prompts in the dialogue box, and if you need to upload a file, you can select the file for upload in the "browse files" section.
4. Click the "run code interpreter" button, and the backend will execute our pipeline. The agent will generate code and execute it in the sandbox, and the results will be returned on the interactive page upon completion.

Example 2: Running With Local Models

1. Our local LLM service is developed on vLLM. First, you can start a vLLM model serving by running this command:

# Take llama-2-7b as an example
python3 src/activities/vllm_api_server.py --model "meta-llama/Llama-2-7b-hf"  --served_model_name "meta-llama/Llama-2-7b-hf"

At this point, we only support Linux environment.

2. You can try this command if the serving is successfully starting:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-2-7b-hf",
        "prompt": "San Francisco is a",
        "max_tokens": 7,
        "temperature": 0
    }'

3. Then you can run the demo following this command:

bash run_demo.sh --llm "meta-llama/Llama-2-7b-hf"

Example 3: Running Without Front-end

Our demo is designed to default to the use of the front-end. If you prefer not to use the front-end and instead perform command-line operations or process large amounts of data, you can refer to the following commands:

# Run with API.
python3 ./src/activities/eval.py --llm AZURE_OPEN_AI --api_key 123
# Run with local model.Take llama-2-7b as an example.
python3 ./src/activities/eval.py --llm "meta-llama/Llama-2-7b-hf"