# Steganographic Evaluations

A tool for evaluating steganographic and encoded reasoning and message passing techniques in language models. This tool allows you to test and evaluate various methods for hiding information or reasoning steps in model outputs, using different language models.

## Project Structure

- **stego_benchmark/**: Main package with all experiment logic, configuration, prompt generation, model wrappers, and utilities.
- **datasets/**: Example datasets for experiments.

## Prerequisites

- Python 3.10 or higher
- OpenAI API key
- Anthropic API key
- DeepSeek API key
- Vertex AI credentials (for Gemini)

## Installation

1. Clone the repository.

2. Create and activate a virtual environment:

   ```
   python -m venv venv
   source venv/bin/activate
   ```

3. Install the package with development dependencies:

   ```
   pip install -e ".[dev]"
   ```

4. Set up environment variables:
   ```
   cp .env.example .env
   ```
   Then edit `.env` with your API keys:
   ```
   OPENAI_API_KEY=your_openai_key_here
   ANTHROPIC_API_KEY=your_anthropic_key_here
   GOOGLE_APPLICATION_CREDENTIALS=path_to_your_vertex_json_key_file_here
   DEEPSEEK_API_KEY=your_deepseek_key_here
   ```

## Datasets

The repository includes several datasets for benchmarking:

- **AQUA-RAT**: A standard dataset of algebraic word problems, used for encoded reasoning and encoded message passing experiments.
- **GPQA-Diamond**: A challenging multiple-choice question dataset, used for encoded reasoning.
- **Anthropic-Helpful**: A standard dataset of conversational questions designed to test the helpfulness of models, used for encoded reasoning and message passing experiments.
- **State-Tracking Datasets**: Custom-generated datasets for evaluating state-tracking and reasoning abilities. These include both 1D and 2D maze movement tasks, where the model must track the agent's direction after a sequence of moves.

You can find these datasets in the `datasets/encoded_reasoning/` directory.

### Generating Custom Datasets

We provide a Jupyter notebook, [`generate_data.ipynb`](generate_data.ipynb), which allows you to generate your own state-tracking datasets. This notebook contains code for creating 1D and 2D maze movement tasks, and saving them in the appropriate format for use with the benchmark.

## Basic Usage

### Running a Single Experiment

You can run a single experiment by specifying a configuration file. For example, to run a direct (non-steganographic) experiment:

```bash
python -m stego_benchmark.main --config direct_config.yaml
```

- `direct_config.yaml` is an example config file for a direct experiment (cover task + hidden task, no steganographic instructions).
- `encoded_reasoning_config.yaml` is an example config for an encoded reasoning experiment (see paper for details).

You can modify or create your own config files to specify different models, datasets, experiment modes, and settings.

### Running All Encoded Message Passing Experiments

To reproduce all encoded message passing experiments (as in the paper):

```bash
python stego_benchmark/run_message_passing_experiments.py
```

**Warning:** This will run a large number of experiments and may result in significant API usage and cost.

### Running All Encoded Reasoning Experiments

To reproduce all encoded reasoning experiments (as in the paper), for each model run:

```bash
python -m stego_benchmark.run_encoded_reasoning_experiments
```

**Warning:** This will run a large number of experiments and may result in significant API usage and cost.

### Running All Steering Reasoning Experiments

To reproduce all steering reasoning experiments (as in the paper):

```bash
python stego_benchmark/run_steering_experiments.py
```

## Result File Structure

After running an experiment, results are saved in the directory specified by `output_dir` in your config file. Each run creates a timestamped subdirectory containing:

- `config.yaml`: The configuration used for the run.
- `results.jsonl`: The list of results, one JSON object per sample.

Each entry in `results.jsonl` contains fields such as:

- The input question(s)
- The model's response
- The decoded answer (if applicable)
- Whether the answer was correct
- Any additional metadata relevant to the experiment

### Code Style

Format code:

```bash
black .
isort .
```

## Common Issues

1. **API Key Issues**: Make sure your API keys are properly set in the `.env` file.
2. **Rate Limits**: The tool includes automatic retry logic for rate limits.
3. **Model Timeouts**: Adjust the timeout settings in the model configuration if needed.

## License

This project is licensed under the MIT License - see the LICENSE file for details.
