# cLUT Benchmarks and Measurement Results

## Important Contents

* Our C reference implementation (`src/algorithms/cLUT.c`)
* Our measurements (`data/*.csv`)
* Scripts generating figures (`figures/*.py`)
* Scripts generating tables (`tables/*.py`)
* Related Work for C experiments (`aldr-experiments`)
* Input distributions and their generation file (`dists.tar.gz` and `generate_distributions.py`)
* Makefile for experiments and setup (`Makefile`)
* FFT wrapper for Python (`lut_c_wrapper.py`)
* Cython implementation of cLUT (`clut_sample.pyx`)

## Dependencies

* Recent Linux with C compiler (for energy measurements in the C experiment)
* Python3
* `Anonymous Energy Benchmark Tool` (`AEBT`) for energy measurements of TrueSkill (*blinded for review*)
* [cpupower](https://linux.die.net/man/1/cpupower) tool installed
* [taskset](https://linux.die.net/man/1/taskset) tool installed

## How to build and run the experiments

To compile all C files and generate the benchmark tool `lut_benchmark`:

```bash
$ make all
```

For setting up a python virtual environment and installing dependencies:

```bash
$ make venv
```

### Sampling speed/energy/bit efficiency compared to SOTA implementations in C

```bash
$ make c_experiment
# or
$ make c_experiment OUTNAME=yourname.log EXPERIMENT_LIST='cLUT alias.c'
# this runs `lut_benchmark <sampler> <distribution>` for provided samplers and files from distributions/*
```

### Sampling memory footprint compared to SOTA implementations in C

```bash
$ make c_memory_usage # [...] Optional flags as above from speed/energy/bit efficiency
```


### Sampling speed in Python

```bash
$ make python_experiment
```

### Running the TrueSkill Application

Relevant files:
- ```trueskill.py``` stores TrueSkill implementation.
- ```trueskill_benchmark.py``` benchmarking script.
- ```libclut32.pyx``` CLUT cython implementation. (works not with all distributions)
- ```pyclut.py``` CLUT python utilities.
- ```setup_libclut32.py``` CLUT cython setup script to install libclut32 library.
- ```clut_sample.pyx``` cython implementation for sampling function from CLUT.
- ```setup_libclut32.py``` CLUT cython setup script to install clut_sample library.

To run experiment:
```bash
$ make liblut.so # Python interface for C code (used for preprocessing)
$ make cython # Cython implementation for cLUT sampling stage
$ make setup_experiment # Fix CPU frequncy
# Run the below command once for a warmup without AEBT, then do the actual experiment
$ $AEBT <path_to_venv>/python trueskill_benchmark.py <baseline | clut> --distribution <bimodal | lognormal> --init <1 | 0> --plot <1 | 0> --samples <n_samples> --loops <n_repititions>

# example
python trueskill_benchmark.py clut --distribution bimodal --precision 32 --step 5 --init 1 --plot 1 --samples 1000000 --sampler lut_c_wrapper
python trueskill_benchmark.py baseline --distribution bimodal --precision 32 --step 5 --init 1 --plot 1 --samples 1000000 --sampler discrete
python trueskill_benchmark.py baseline --distribution bimodal --precision 32 --step 5 --init 1 --plot 1 --samples 1000000 --sampler continuous
```


### Other useful Makefile commands
```bash
$ make rebuild # Delete all files and recompile (e.g. if CFLAGS changed)
$ make clean # Cleanup temporary files
```
