Testing and profiling

Before committing code changes, run the tests/checks from a package built from your modified source.

This page collects the common checks used during AlphaPeel development: the regular pytest suite, accuracy benchmark, and profiling and coverage checks.

pytest suite

The pytest suite is ran automatically on the GitHub Actions CI for every pull request, but it is recommended to run the tests locally before committing code changes.

Install test dependencies

Install pytest and pytest-benchmark before running the pytest suite. For more background, see the pytest documentation and the pytest-benchmark documentation.

pip install pytest
pip install pytest-benchmark

Build and reinstall the modified package

Run tests against the package built from your current source tree. This is required for the tests/accuracy_tests/run_accu_test.py as it runs the tests on the built package rather than the source code.

python -m build
python -m pip uninstall AlphaPeel -y
python -m pip install dist/*.whl

For more detail on building a local distribution, see Install from the git repository.

Run the pytest suite

From the repository root, run:

pytest

If the tests pass, the beginning of the output should look similar to this:

============================================================================================ test session starts =============================================================================================
platform darwin -- Python 3.11.11, pytest-9.0.1, pluggy-1.6.0
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/xtang3/AlphaPeel
configfile: pyproject.toml
plugins: benchmark-5.2.3, memray-1.9.0
collected 16 items

tests/accuracy_tests/run_accu_test.py ...                                                                                                                                                              [ 18%]
tests/functional_tests/run_func_test.py .............                                                                                                                                                  [100%]
...

Run a targeted functional test

To run one functional test, provide the pytest node id. For example, to run test_files:

pytest tests/functional_tests/run_func_test.py::TestClass::test_files

Inspect failed tests and printed output

When a test fails and you need to inspect printed output, add -s:

pytest -s

Run accuracy benchmarks

In addition to running pytest, you can run the full accuracy benchmark suite. This is useful when you want to compare the impact of a new change on accuracy and runtime across the configured benchmark cases in detail.

From the repository root, run:

from src.accuracy_runner import run_full_accuracy_suite

run_full_accuracy_suite(run_name="benchmark")

This writes outputs under tests/accuracy_tests/outputs_benchmark and writes the accuracy report to tests/accuracy_tests/reports_benchmark/accu_report.txt. The report is a comma-separated text file with records in the form file_name,label,metric_name,value. For most accuracy metrics, value contains the population metric followed by the metrics for generations 1 to 5. Runtime is written separately as runtime,<label>,elapsed_seconds,<seconds>.

Here is an example of the first 3 lines of the accuracy report:

runtime,single,elapsed_seconds,10.043625167018035
dosage,single,marker_corr,['0.6792', '0.5175', '0.8658', '0.8722', '0.869', '0.7148']
dosage,single,ind_corr,['0.861', '0.6016', '0.9377', '0.9416', '0.9401', '0.8841']

The accuracy report would have a few hundred lines and could be difficult to read in a text editor. You can visualise the report with the create_accuracy_report_visualizations function described below.

It is recommended to save a copy of the accuracy report before the change and after the change, and compare them with the compare_reports function described below.

Visualise accuracy reports

You can create plots from an accuracy report (matplotlib.pyplot is required):

from src.accuracy_visualization import create_accuracy_report_visualizations

create_accuracy_report_visualizations(
    "tests/accuracy_tests/reports_benchmark/accu_report.txt",
    "tests/accuracy_tests/reports_benchmark/plots",
)

By default, this creates plots for marker_corr, abs_diff, and correct_rate, together with a separate runtime plot. To focus on a different set of metrics or methods, pass explicit selections:

create_accuracy_report_visualizations(
    "tests/accuracy_tests/reports_benchmark/accu_report.txt",
    "tests/accuracy_tests/reports_benchmark/plots",
    metric_names=("marker_corr", "switch_error_rate", "phase_error_rate"),
    labels=("single", "multi", "hybrid"),
)

These plots are intended as a development aid before opening a pull request.

Compare accuracy reports

After you have a baseline report and a report from your modified code, you can use the compare_reports function to compare them. The comparison highlights changes in runtime and accuracy, and can show whether differences are concentrated in particular generations.

You can compare two accuracy reports with the compare_reports function by running the following code from the repository root:

from src.accuracy_report_comparison import compare_reports
compare_reports()

It by default compares the two reports accu_report.txt (at root of the repository) and tests/accuracy_tests/reports_benchmark/accu_report.txt, but you can pass explicit paths to the two reports to compare with the arguments baseline_report_path and current_report_path.

An example output of the comparison is:

metric_name,matched_records,compared_values,mean_delta,mean_abs_delta,max_abs_delta,max_abs_delta_location
abs_diff,98,572,5.48548139e-06,5.664273926e-06,0.002599332529,hap_0.5:multi_est_alt_allele_prob:generation_1
ind_corr,98,572,-6.468531469e-06,6.468531469e-06,0.0029,hap_0.5:multi_est_alt_allele_prob:generation_1
marker_corr,98,572,-1.590909091e-05,1.835664336e-05,0.0081,hap_0.5:multi_est_alt_allele_prob:generation_1
correct_heterozygote_rate,15,90,1.333333333e-07,2.666666667e-07,7.5e-06,hap_0.5:multi_est_alt_allele_prob:generation_2
heterozygote_count,15,90,0,0,0,
homo_to_hetero_ratio,15,90,0,0,0,
homozygote_count,15,90,0,0,0,
phase_error_rate,15,90,1.73e-05,1.736666667e-05,0.0013,hap_0.5:multi_est_alt_allele_prob:generation_1
switch_error_rate,15,90,-1.900950475e-06,1.900950475e-06,0.0001425712856,hap_0.5:multi_est_alt_allele_prob:generation_1
uncalled_rate,15,90,2e-07,5.222222222e-07,1.75e-05,hap_0.5:multi_est_start_alt_allele_prob_est_geno_error_prob_est_seq_error_prob_seq_file:generation_1
wrong_homozygote_rate,15,90,-3.666666667e-07,4.888888889e-07,1.75e-05,hap_0.5:multi_est_start_alt_allele_prob_est_geno_error_prob_est_seq_error_prob_seq_file:generation_1
correct_rate,8,32,7.063383194e-09,7.89150818e-09,1.850000031e-08,seg_prob:multi_est_start_alt_allele_prob_est_geno_error_prob_est_seq_error_prob:generation_5

You can pass the output_path argument to save the comparison results to a CSV file.

Profiling and coverage

Profiling is useful when you need to identify performance bottlenecks.

Memory profiling

For memory profiling, create a small temporary Python driver benchmark_memray.py that runs the workflow you want to inspect. For example:

from src.accuracy_runner import run_full_accuracy_suite

run_full_accuracy_suite(run_name="benchmark")

Save the driver outside version control, or remove it after profiling. Then run it with memray:

pip install memray
memray run benchmark_memray.py

This creates a .bin memory profile file. Inspect it by generating a flamegraph:

memray flamegraph memray-*.bin

This generates an HTML file that you can open in a browser to inspect memory usage.

Runtime profiling

For line-wise runtime profiling, create a small temporary Python driver with line_profiler. Add the functions you want to inspect, then run the accuracy case inside the profiler:

from line_profiler import LineProfiler

from src.accuracy_core import get_params, sim_path
from src.accuracy_runner import run_accuracy_case
from src.tinypeel.tinypeel import main, peeling_cycle, run_peeling_cycles

lp = LineProfiler()

for fn in [
    run_peeling_cycles,
    peeling_cycle,
    main,
]:
    lp.add_function(fn)

# This is a single accuracy case from the benchmark suite. You can change it to
# any other case you want to profile.
case = (
    "multi",  # method
    False,    # est_start_alt_allele_prob
    True,     # est_geno_error_prob
    True,     # est_seq_error_prob
    True,     # seq_file
    False,    # alt_allele_prob_file
    False,    # est_alt_allele_prob
    False,    # metafounder
    False,    # x_chr
)

with lp:
    run_accuracy_case(
        get_params(),
        sim_path(),
        *case,
        benchmark=None,
        run_name="profile_line",
        TINYPEEL_DIRECT_IS_WARMED_UP=True,
    )

lp.print_stats()

Save this driver outside version control, or remove it after profiling. If the temporary file is named line_profile.py, run:

pip install line_profiler
python line_profile.py

The example above skips a warmup run by setting TINYPEEL_DIRECT_IS_WARMED_UP=True. In a fresh Python process, this means the reported line timings include JIT compilation time. If you have TINYPEEL_DIRECT_IS_WARMED_UP=False, then the reported line timings include one warmup run, and the actual runtime of the function, which is equivalent to the compilation time plus two times the runtime.

To keep a reference profile for later comparison, redirect the output to a text file with a descriptive name:

python line_profile.py > before_change_line_profile.txt

Multi-threading benchmarking

Use src.thread_profiler to benchmark the multi method across n_thread_fam and n_thread_loci combinations. The profiler runs each grid cell in a fresh Python subprocess, records raw timings, writes a summary CSV, and creates 3D runtime surface plots.

From the repository root, run:

from src.thread_profiler import run_thread_profiler

run_thread_profiler()

By default, this benchmarks thread counts 1, 2, 4, 8 with one replicate per grid cell and writes artifacts under tests/accuracy_tests/outputs_thread_grid. To use a smaller or larger grid, pass explicit arguments:

run_thread_profiler(
    thread_counts=(1, 2, 4, 8, 16),
    replicates=3,
    n_cycle=5,
    skip_existing=True,
)

The raw timings are written to thread_grid_raw.csv, the aggregated timings to thread_grid_summary.csv, and the runtime surface plot to thread_grid_runtime_surface_inner.png in the output directory. Keep the CSV files from a previous run if you want a reference for later comparisons. Because each grid cell is run in a fresh Python subprocess, timings include process startup and first-run JIT compilation time. Use the same settings when comparing results across code changes.

Test coverage

Coverage can be collected with coverage:

pip install coverage
coverage run -m pytest tests/functional_tests
coverage report

To generate an HTML coverage report, run:

coverage html

Note

Coverage reports cannot identify calls to JIT-compiled functions.