## Installation

```bash
conda create -n vtr_env python=3.11
conda activate vtr_env
pip install -r requirements.txt
```

## Prepare Datasets

Download datasets from huggingface:

- LVBench : https://huggingface.co/datasets/zai-org/LVBench
- VideoEval-Pro : https://huggingface.co/datasets/TIGER-Lab/VideoEval-Pro
- MLVU : https://huggingface.co/datasets/sy1998/MLVU_dev
- LongVideoBench : https://huggingface.co/datasets/longvideobench/LongVideoBench
- VideoMME : https://huggingface.co/datasets/lmms-lab/Video-MME

Then, modify "video_dir" and "anno_path" in "./configs/benchmarks".

## Prepare Models

Download models from huggingface:

- LLaVA-Video-7B : https://huggingface.co/lmms-lab/LLaVA-Video-7B-Qwen2
- Qwen2.5-VL-7B : https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
- LLaVA-Video-72B : https://huggingface.co/lmms-lab/LLaVA-Video-72B-Qwen2
- Qwen2.5-VL-72B : https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct

- CLIP-B/32 : https://huggingface.co/openai/clip-vit-base-patch32
- PE-L/14 : https://huggingface.co/facebook/PE-Core-L14-336
- PE-G/14 : https://huggingface.co/facebook/PE-Core-G14-448

Then, modify "model_path" in "./configs/vlm_models" and "./configs/vtr_models".

## Embedding Extraction

Modify "host" and "port" in "./configs/multinode/default.yaml" to set the distributer's ipv6 address and port.

Modify "./configs/default_emb.yaml".

Start the following code on main node:
```python
python eval/emb_distributer.py
```

Start the following code on main node or other nodes:
```python
python eval/emb_runner.py
```

Embedding results are saved in "./logs/emb/XXXX".

## Evaluation

Modify "./configs/default.yaml".

Start the following code on main node:
```python
python eval/vlm_distributer.py
```

Start the following code on main node or other nodes:
```python
python eval/vlm_runner.py
```

Evaluation results are saved in "./logs/vlm/XXXX".





