# Social Agents: Collective Intelligence Improve LLM Predictions

This repository contains the source code for the project "Social Agents." It includes scripts for inference, evaluation, and ablation studies associated with the project.

## Table of Contents

- [Project Overview](#project-overview)
- [Repository Structure](#repository-structure)
- [Setup and Installation](#setup-and-installation)
- [Usage](#usage)
  - [Running Inference Scripts](#running-inference-scripts)
  - [Running Evaluation Scripts](#running-evaluation-scripts)
  - [Running Ablation Studies](#running-ablation-studies)
- [Citation](#citation)

## Project Overview

-   **Behavioral Attribution Classification**: Identifying actions, emotions, persuasion, reasons, and topics from video advertisements.
-   **Ad CTR and ROAS Prediction**: Predicting Click-Through Rates and Return on Ad Spend.
-   **Long-Term Video Memorability Prediction**: Assessing the long-term memorability of video content.
-   **Tweet Content Generation**: Generating engaging content for tweets.
-   **Tweet Engagement Prediction**: Predicting user engagement with tweets.
-   **Website Likability Prediction**: Evaluating the aesthetic appeal/likability rating of websites.


## Repository Structure

The repository is organized into three main directories:

-   `inference_scripts/`: Contains Python scripts to run inference with different LLMs (GPT-4o, LLaMA 3.3 70B Instruct, Qwen3 32B, LLaMA 3.2 90B Vision Instruct, Qwen2.5 VL 72B) on various datasets. Each subdirectory corresponds to a specific task.
-   `evaluation/`: Includes scripts to evaluate the performance of the models on the tasks defined in the inference scripts. The metrics are tailored to each specific task.
-   `ablations/`: Contains scripts for conducting ablation studies, such as analyzing the impact of personas, budget, and other factors on model performance.

```
/
├── ablations/
│   ├── correlation_w_human/
│   ├── impact_of_budget/
│   ├── inter_persona_correlation/
│   ├── persona_vs_nopersona_divergence/
│   └── prompting_methods/
├── evaluation/
│   ├── behavioural_attribution_detection/
│   ├── ctr_and_roas_prediction/
│   ├── long-term_video_memorability_prediction/
│   ├── tweet_content_generation/
│   ├── tweet_engagement_prediction/
│   └── website_likability_prediction/
└── inference_scripts/
    ├── behavioural_attribution_detection/
    ├── ctr_and_roas_prediction/
    ├── long-term_video_memorability_prediction/
    ├── tweet_content_generation/
    ├── tweet_engagement_prediction/
    └── website_likability_prediction/
```

## Setup and Installation

1.  **Clone the repository:**
    ```bash
    git clone https://github.com/your-username/codebase_socialagents.git
    cd codebase_socialagents
    ```

2.  **Create a virtual environment (recommended):**
    ```bash
    python3 -m venv venv
    source venv/bin/activate
    ```

3.  **Install the required dependencies:**
    ```bash
    pip install -r requirements.txt
    ```

4.  **Set up environment variables:**
    The scripts require API keys and other configuration to be set as environment variables. Specifically, you need to set your `OPENAI_API_KEY` and `AZURE_OPENAI_ENDPOINT`.
    ```bash
    export OPENAI_API_KEY="your_api_key"
    export AZURE_OPENAI_ENDPOINT="your_azure_endpoint"
    ```

## Usage

The primary way to run the experiments is through the `run.sh` scripts located in the task-specific subdirectories. These scripts handle the setup, execution, and result aggregation.

### Running Inference Scripts

To run an inference script, navigate to the desired task directory within `inference_scripts/` and execute the `run.sh` script.

```bash
cd inference_scripts/<task_category>/<specific_task>/
bash run.sh
```

The `run.sh` script is configurable. You can modify variables within the script to change the input dataset, output directories, and execution parameters (e.g., using GPUs for open source models). The script processes a CSV file, runs the corresponding Python script in parallel, and merges the final results into a JSON file.

### Running Evaluation Scripts

After running the inference scripts and generating the output JSON files, you can use the evaluation scripts to assess the model's performance. Navigate to the corresponding task directory within `evaluation/` and run the `metric.py` script.

```bash
cd evaluation/<task_category>/<specific_task>/
python metric.py --prediction_path /path/to/your/predictions.json --ground_truth_path /path/to/ground_truth.json
```
*Note: The arguments for the evaluation scripts might vary. Please refer to the specific `metric.py` file for details on the required arguments.*

### Running Ablation Studies

The `ablations/` directory contains scripts for more detailed analysis. To run an ablation study, navigate to the desired directory and execute the Python script.

```bash
cd ablations/<study_name>/
python <script_name>.py --data_path /path/to/your/data.csv
```
*Note: The arguments for the ablation scripts might vary. Please refer to the specific script for details on the required arguments.*
