This file briefly outlines the code used to generate our experimental results.


Our experiments were run using Python 3.11.9.
To use the code, please ensure that all packages listed in requirements.txt are installed.


Figure 1:

The data in Figure 1 (reference distance vs split distance) is generated by running rule_comparison.py

Results will be saved to "results/experiment-ground_truth_vs_split_distance-testing-nsplits=10-iclr.csv"

To plot the resulting data, use plotting.py:plot_kt_distance_vs_ground_truth(). Ensure that file names are matched.



Table 1:

Table 1 uses the dataset described by Kerzendorf et. al in "Distributed peer review enhanced with natural language processing and machine learning" (2020).

To reproduce the data in the table run scoring_analysis.py which will print results to the console.


Figure 2:

Data for this plot is generated by running preference_data_analysis.py which saves to "preflib/analysis_results-iclr.csv".

2a. City Election Data

To plot the generated data, run preflib/preflib_plots.py
This should be run as a module by running (from the base project directory):
python -m preflib.preflib_plots
The scatter plot shown in Figure 2a will be saved as preflib/plots/preflib_scatter.png


2b. F1 Data

To plot the generated data, run preflib/preflib_plots.py

This should be run as a module by running (from the base project directory):
python -m preflib.preflib_plots

The bar chart shown in Figure 2b will be saved as preflib/plots/f1_bar.png


Figure 3. SGD and Simulated Annealing Learning Curves

To generate this data, enable the gradient ranking rule (uncomment preference_data_analysis.py:line 268). 

Run preference_data_analysis.py:evaluate_preflib_data() to generate learning history files: "score_history-F1-method=sgd.csv" and "score_history-F1-method=annealing.csv"

Run plotting.py:plot_mean_best_scor() to generate the figure.



Figure 4 (appendix - expanded F1 plot):

Generate F1 scoring data as for Figure 2b.

Run preflib.preflib_plots.py:bar_plot_f1() to generate the plot.


Figure 5 (appendix - ALMA data):

This figure is based on data which cannot be shared due to privacy agreements.



Figure 6 (appendix):

Data for this plot is generated by running preference_data_analysis.py which saves to "results/olympic_data-iclr.csv"

To plot the generated data, run preflib/preflib_plots.py
The bar chart shown in Figure 3a will be saved as preflib/plots/olympics_bar.png



Figure 7 (appendix - axiom violations):

Axiom violation rates are calculated in two parts.

For generated preferences, run axioms.py:run_axiom_experiment() to generate violation data.

We omit the raw Preflib data required for this figure due to reduce the submission file size. Instead we include filtered and reformatted preflib elections in ./preflib/preflib-n_profiles=1392-min_alternatives=5-max_alternatives=20-min_voters=4-max_voters=1000.csv

To compute violations on preflib data, run axioms.py:test_axioms_on_saved_profiles()

To generate the figure itself run plotting.py:plot_axiom_evaluation_results()


========

Note that in addition to the packages that must be installed, we also include in our code the sdopt-tearing library, found at https://github.com/baharev/sdopt-tearing.
