# Why Ask One When You Can Ask $k$? Learning-to-Defer to the Top-$k$ Experts

This folder contains scripts for running, evaluating, and logging Learning‑to‑Defer (L2D) and cardinality‑aware selector experiments on various datasets.

##  Dependencies

Install via:

```bash
pip install -r requirements.txt
```
---

## Top-$k$ Learning-to-Defer

### `main.py`
Orchestrates training and evaluation of the Top-$k$ L2D framework.  
- **Loads** pre-trained agents  
- **Evaluate** random and optimal baselines
- **Trains** the classifier with surrogate deferral loss  
- **Logs** metrics & checkpoints via Weights & Biases

**Usage**  
```bash
sbatch cost_lr_L2D.slurm
```

---
## Top-$k(x)$ Learning-to-Defer

### `cardinality_aware.py`
Trains a cardinality‑aware selector network given a fixed policy; 
uses the cardinality surrogate that balances prediction accuracy and average 
number of consulted experts. Logs tables & metrics with W&B.

One should select accordingly from:
```bash
--decision=("indicator" "majority" "weighted")
--lambda=("1e-9" "1e-2" "5e-2" "1e-1" "0.25" "0.5" "1" "1.5" "2" \
             "2.5" "3" "3.5" "4" "4.5" "5" "5.5" "6" "6.5")   
```
and load the correct classifier trained previously with Top-$k$ L2D: 
```bash
--load_classifier "svhn_l2d_top_k_one_stage_l2d_top_k_one_stage_svhn4_lr_0.01_seed_42_dropout_0.0_scheduler_cosine_optimizer_Adam_cost_[0.055, 0.05, 0.045, 0.04, 0.035, 0.03]/best_model_deferral_loss_top.pth" \
```

**Usage**  
```bash
sbatch cardinality_decision_lr_L2D.slurm
```


---
## Scripts

### `train.py`
Provides shared training and evaluation loops for:  
- **Main** classifier model (`train_model`)  
- **Cardinality** selector (`train_cardinality`)  
- Metric computations (Top‑k accuracy, deferral loss, MRR, etc.)  

_No direct invocation; imported by other scripts._

---

### `loss.py`
Defines custom loss modules for L2D:  
- `L2D_loss` with surrogate 0–1 deferral and cardinality‑aware terms  
- Utilities for computing true deferral loss, querying costs, voting losses  

_No direct invocation; imported by other scripts._

---

### `model.py`
Neural network architectures for predictors on different tasks:  
- Simple CNNs  
- ResNet variants  
- MLPs and linear regressors  

_No direct invocation; imported by other scripts._

---

### `preprocess.py`
Data loading & preprocessing utilities:  
- Datasets with agent annotations for CIFAR‑10, SVHN
- Agent‑knowledge generation (fixed/random experts)  
- Train/test `DataLoader` setup  

_No direct invocation; imported by other scripts._

---

## Quick Start

1. **Prepare** your environment and clone this repo.  
3. **Run** any of the top‑level scripts:  
   - `python main.py ...`  
   - `python cardinality_aware.py ...`  
4. **Monitor** training & evaluation metrics on W&B.  
5. **Inspect** saved checkpoints in `./checkpoints/{name_run}/`.
