Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Clean application corpus

The required test:clean-corpus job downloads the public vendor ZIP archives in scripts/clean-corpus/manifest.json. The manifest pins the version, official HTTPS source, SHA-256, license file, and exact regular-file count. Archives are cached under their checksum, so editing metadata or adding a source downloads only new pins. Each archive is verified before extraction into a new private directory. Corrupt cache entries are replaced with a verified download; unverified downloads never enter the reusable cache. Failed downloads, missing licenses, empty or incomplete inventories, unsafe archive entries, and read failures stop the job. No downloaded PHP or JavaScript is executed.

CSM_CORPUS_ARCHIVE_CACHE selects the cache directory. The tool reserves its .csm-corpus-v1 subdirectory for checksum-named archives, temporary downloads, and a persistent lock. Do not put other files in that subdirectory. Cleanup only touches generated archive and temporary names there, never ZIPs elsewhere in the configured directory or symlink targets. A symlinked managed directory is refused. Concurrent preparations sharing a cache fail with a busy error; retry after the active run finishes.

Valid flat cache entries from older versions are verified and copied into the managed cache without downloading. Their originals are left untouched because names alone cannot establish ownership; operators may remove those old inputs once no older tool uses them. Managed archives no longer pinned and interrupted downloads are pruned before fetching, even if preparation later fails. GitLab saves verified downloads and cleanup on failed jobs too, so retries retain progress without accumulating retired pins.

Every source names the CMS it belongs to in a cms field, and the manifest lists each supported CMS that has no pinned source yet under pending, with the roadmap item that blocks it. The manifest is version 2; the validator refuses older versions, a source without a CMS, a supported CMS that is neither sourced nor pending, and one that is both, so a database scanner for a new CMS cannot ship without a corpus decision. A pending entry records absent evidence only: no scanner, admission rule, hit budget or result filter reads it, and files from that CMS are treated exactly as before on a server. When a source for a pending CMS lands, its pending entry is removed in the same change. The manifest, not this page, is the record of which CMS has a false-positive gate.

Ordinary unit tests validate the manifest offline, including with go test -trimpath. Invalid metadata is rejected before cache, extraction or report files are created or changed. Command tests check the full archived manifest and every inventory record; these checks do not replace the detector gate.

The initial corpus contains 10,178 files: WordPress 6.8.2 (GPL-2.0-or-later), WooCommerce 9.9.5 (GPL-3.0-or-later), and Elementor 3.29.2 (GPL-3.0-only). The archives retain their license notices, including notices for bundled components. These are fixed test versions, not installation recommendations. Source URLs and license file locations are in the manifest.

Run the same gate locally with the Go toolchain from go.mod and the matching YARA-X C library installed:

bash scripts/clean-corpus-test.sh

The script compares the installed C library version with both the Go binding and production builder pins. It runs YARA, YAML signatures, PHP taint, and JavaScript taint gates. Ordinary unit tests may omit external corpus inputs; the required job sets CSM_CORPUS_REQUIRED=1, so missing input is fatal. Package publication and GitHub releases explicitly depend on the job.

corpus-results/ is retained for one year, including on failures:

  • manifest.json identifies the authenticated archives and licenses.
  • inventory.json lists every extracted file, its size and SHA-256.
  • engines.txt records the commit, toolchain, YARA-X and parser versions.
  • Engine JSON reports include counts, per-rule hits including zeroes, thresholds, and taint status counts. tests.jsonl records execution.

YARA and both taint engines require zero findings. YAML retains the existing reviewed false-positive budgets in internal/signatures/corpus_fp_gate_test.go; new rules default to zero. The pinned corpus currently produces 10 YAML hits across five rules, all within those existing budgets. This is a regression gate, not a claim that every engine has zero false positives. A second YAML check fails if any regex would be skipped on a corpus file it matches.

The signature engines apply the production default 16 MiB bound and archive admission policy; 10,157 inputs reach each engine. PHP taint reads 10,171 inputs within its source limit, including 163 that complete analysis. JavaScript admission receives all 10,178 files and completes analysis on 289; seven are oversized and twelve have parse errors. These JavaScript gaps have explicit ceilings in the required gate; canceled, resource-limited and panic outcomes have a zero ceiling. Neither oversized nor unparsed inputs count as proof of successful analysis. The required taint gates each need at least 100 completed analyses as well as their corpus inventory checks.

To update the corpus, download a versioned archive from its official source, review its license and file inventory, and commit the new manifest pin with the measurements. Investigate new matches and narrow detectors using content semantics. Do not add scan-path exclusions or raise budgets to pass a run. The gate also compiles deliberately overbroad YARA and YAML rules and verifies that their clean-file matches are rejected.