Reproducibility

How to reproduce these results

Every page on this site is generated from a deterministic harness. The same Docker invocation that we run weekly in CI will reproduce the numbers in Benchmark and Security on any machine with Docker and roughly 4 GB of RAM.

§1. Install GitHacker itself

To use GitHacker against a remote target — separate from the benchmark harness — either run the Docker image or install from PyPI:

Docker

docker run \
  -v $(pwd)/results:/tmp/githacker/results \
  wangyihang/githacker \
  --output-folder /tmp/githacker/results \
  --url http://target/.git/

pip

pip install GitHacker

githacker \
  --url http://target/.git/ \
  --output-folder result

Add --brute to brute-force branch and tag names when directory listing is disabled on the target.

§2. Re-run the benchmark

The full performance suite (7 tools × 5 web-server scenarios) runs from this repo:

git clone https://github.com/WangYihang/GitHacker
cd GitHacker
pip install gitpython semver coloredlogs
python -m benchmark run

Output writes to docs/public/data/benchmark.json; rebuild this site to view. The test repository is generated from a fixed random seed (seed=0), so all numbers are deterministic. Full configuration: benchmark/.

§3. Re-run the security suite

The adversarial scenarios (15 tests × 7 tools, malicious server vs. pillager) ship as a sibling command:

python -m benchmark security

Results land in docs/public/data/security.json. The runner enforces a 90-second per-tool timeout and tears down each Docker container between cases.

§4. Pinned environment

Docker 24+ (BuildKit enabled)
Python 3.10+ for the runner
~4 GB RAM and one CPU core per concurrent tool slot
Internet access for the initial tool-image builds (none required afterwards)

Tool versions and the harness commit are recorded in the metadata block of each JSON output, so any plot you see on this site can be traced back to an exact reproducible configuration.