Attack replay corpus
The required test:attack-corpus job plants known attack samples and
near-miss controls in a temporary cPanel account and runs them through the
three production paths that read files: realtime analysis, the scheduled deep
tier and the operator full account scan. Each path applies its own production
responders. The job fails when a path stops catching a sample, catches it at a
lower severity, flags or changes a control, remediates differently, or starts
catching a recorded gap. Package publication and GitHub releases depend on it.
Unit tests prove that a detector works on the input it was written for. The corpus proves that the whole path still works: admission, path scoping, severity, quarantine eligibility and the responders. A change that breaks a detector end to end fails here even when every unit test passes.
The corpus measures file content only. Log and event replay and expected incidents are later stages.
The three paths
Every sample runs alone, with its own account tree, cPanel domain map, state
store, quarantine directory and monitor. The platform is pinned to cPanel, the
layout the samples use. The config is the shipped configs/csm.yaml template
with auto-response and file quarantine turned on and observe mode off, and
nothing else changed. Both signature engines load the shipped rules from
configs/. A partial rule load, an empty rule set or a scan error fails the
gate.
- Realtime. The driver builds the file monitor without fanotify, opens
each planted file and derives its event path the way the event loop does.
The file then goes through the production admission filter and, when
admitted, through
analyzeFile. A file the filter drops counts as not detected. The driver drains the dropper tracker and the staged package queue at once instead of waiting for their timers. Remediation is the inline quarantine thatanalyzeFileperforms. - Deep tier. The driver runs the file checks of the deep tier through the
production runner,
runParallelWithContext, so the runner’s responders act on the findings as they do in production. Every deep check is classified as a file check or a host check (network, database, mail, packages). Host checks do not run. A test fails when a new deep check has no classification, so it cannot stay out of the gate unnoticed. Before planting, the driver runs the tier once on the empty account tree with the same state store. Findings that name no planted file and also appeared in that empty run are dropped as host noise. No check or path is allowlisted. - Full scan. The driver runs the account scan with the options that
csm scan <account> --full --quarantineuses, then the scan job’s quarantine loop. That loop ignores the auto-response settings, as it does in production.
The tier and the full scan run PHP taint analysis in-process. Production runs it in a separate worker, which has its own tests.
Offline
The run never reaches the network. Requests through the default HTTP
transport are refused. On the two scan paths, database queries get an empty,
successful answer for a synthetic site, and host commands fail as a missing
program would. The realtime WordPress checksum fetcher is stopped. Refused
requests, database queries and wp runs are counted per sample and appear in
the report as network_attempts. A sample whose verdict depends on such a
lookup says so in its description.
Sample format
Samples live in internal/attackcorpus/testdata/corpus/<category>/<name>/.
Each directory holds a sample.json and the base64 content files it names.
Content is base64 at rest and decoded only in memory or under a test’s
temporary directory, because endpoint antivirus deletes decoded web shells
from a checkout.
sample.json fields:
name: unique across both halves of the corpus, public and private.category: one ofwebshell,dropper,malicious_plugin,credential_stealer,injected_js,phishing_kit,spam_script,persistence,binary,archive.kind:maliciousorcontrol.source:synthetic,fixtureorreal.description: one line on what the sample is. It also names any policy, shortfall or offline lookup that explains its expectations.provenance: where the content came from. For a fixture, the test it was taken from and how the planted shape differs; for a synthetic sample,written for the corpus.files: each withpath, account-relative (public_html/...) or@crontabfor the account’s crontab; exactly one ofb64(a file beside the manifest, ending in.b64) orselftest(a sample from the self-test bundle); and an optionalmode. JSON has no octal, so0755is written493.expect: one entry for each ofrealtime,tierandfullscan:findings: each withcheck,file(a planted path) andmin_severity(warning,highorcritical). Every listed finding must arrive at that severity or above. Extra findings are reported, not failed.gap: why this path misses the sample. Exclusive withfindings.remediation: per planted file,quarantined,cleanedoruntouched. A file left out is expected untouched.cleanedneeds the SHA-256 of the cleaned bytes incleaned_sha256. The deep tier and the full scan clean a WordPress core, plugin or theme file in place, instead of moving it, when a Critical quarantine-eligible finding names it. A plugin or theme file with a known web shell name is quarantined instead, and so is a file that cleaning would leave empty with no page output, or that its core or plugin release does not ship. Whole-file malware expectsquarantined; an injected site file expectscleaned, oruntouchedwhen the cleaner refuses the removal.
A missing file counts as quarantined only when a quarantine entry names its
path, or a parent directory’s, and the retained copy matches the original
bytes. A damaged entry is an error. A file that disappears with no entry is
reported as removed, which no manifest may expect. A file whose bytes
changed is cleaned.
All files of a sample are planted before the first one is replayed. Manifest order is not write order, so a sample whose verdict depends on the order of writes is judged on the final tree.
Every self-test bundle sample is referenced exactly once, so the bundle and the corpus cannot drift apart.
Adding a sample
- Write or decode content only in a temporary directory that your
antivirus excludes, never in the checkout. Commit only the
.b64file. - Add files by explicit path. Never run
git add -Aorgit add .: an antivirus that deletes a decoded file leaves a deletion for the next commit to pick up. - Public samples follow the public repository rules: documentation IP
addresses only (
192.0.2.0/24,198.51.100.0/24,203.0.113.0/24,2001:db8::/32),example.com,example.netorexample.orghosts, and no real names, accounts, domains, mailboxes or credentials. This applies to names, descriptions and provenance as well as content. make check-fixturesdecodes every corpus data file and scans it for addresses and private names. It decodes one layer only. A payload nested inside the sample, such as base64 inside PHP, a compressed blob, an archive member or aString.fromCharCodelist, needs a manual check of the same rules before commit.- Write the expectations from what the sample is (see below), then run the
gate.
TestPublicCorpusLoadsin the default test job loads the public corpus and checks category coverage, so a malformed manifest fails there first.
Expectation rules
Judge each expectation from what the sample does. A run may draft the findings, but every expected finding and severity must match what the attack deserves, not what the detector happens to report today, except a recorded severity floor (below).
- Gaps. A malicious sample that a path misses records a gap with the
specific reason: which check misses which shape. A gap is a measurement, not
an exemption. When a detector starts catching a recorded gap, the gate fails
with
gap closed:until the manifest replaces the gap with the finding. Only a finding on a planted file, or one that names no file but quotes an admitted planted file in its text, closes a gap; a finding on a directory or other path is reported as an extra finding. Files rejected by admission cannot close a gap. Closing a gap is a corpus edit in the same change as the detector fix. - Never weaken an expectation. Do not delete a sample, lower a
min_severity, turn a finding into a gap, or change an expected remediation to make the gate pass. A regression is fixed in the detector or the responder. Never allowlist a path or name to make a sample pass. - Explicit policy. Where production deliberately leaves a detected file in
place, because the check is not on an explicit quarantine list or the
responder is off in the shipped config, the path expects
untouchedand the description names the policy. When the policy changes, the expectation becomesquarantinedin the same change. - Severity floors. When a path detects a sample below the severity the
attack deserves,
min_severityrecords the observed level and the description names the shortfall. The gate then stops the severity from dropping further, but does not raise it. - Remediation shortfalls. When a detector heuristic, not an explicit policy, leaves a file in place that should move, the expectation records the observed remediation and the description names the shortfall.
Controls
Every category needs at least one malicious sample and one control, or the run
fails. A control is the legitimate file a detector for that category is most
likely to confuse: a cPanel MultiPHP handler in .htaccess, an inline base64
image, a vendored phar, a capability-checked upload handler. A control expects
no findings and untouched on every path, and cannot carry a gap.
Any finding on a control’s file fails the gate, including a file the realtime admission filter dropped. A finding that names no planted file but quotes a control’s path is a false positive. Any other unattributed finding on a control is an error.
Failures
Each failure in the report and the test log starts with its kind:
| Prefix | Meaning |
|---|---|
missed: | An expected finding did not arrive. |
severity: | An expected finding arrived below its min_severity. |
false positive: | A control drew a finding. |
remediation: | A planted file ended quarantined, cleaned, untouched or removed other than expected. |
gap closed: | A path caught a sample recorded as a gap on it. |
error: | The driver did not complete for this sample, or a control drew a finding that names no planted file. |
A run also fails before its first sample when a corpus cannot load, a required private half is not set, a private sample reuses a public sample’s name, a category lacks a malicious sample or a control, a self-test sample is not referenced exactly once, or an engine did not load. A test filter that excludes a sample, a driver that stops early, a check that panics, times out or reports partial coverage, and a failed quarantine all fail the sample.
Driver errors appear in the test log by type only, unless
CSM_ATTACK_CORPUS_SHOW_ERRORS=1 is set (see below). Reports carry check
names, severities, planted paths and verdicts, never sample content or finding
text, because either can quote decoded bytes.
Measurement limits
What the corpus does not measure yet:
- No driver replays program execution, so YARA scans of programs started from
a document root (
docroot_binary_exec) are not measured. - No driver replays the critical tier, so the crontab check is not measured on the scheduled path.
- The daemon’s batch responders (permission fix, database malware response,
companion sweep queueing, WP-Cron fix) are not driven. Realtime remediation
is the inline quarantine only, and the tier stops at the runner’s
responders. In production, realtime findings also reach the batch process
kill and file quarantine. Verdicts agree today because the only realtime
check on the batch quarantine list,
signature_match_realtime, passes the same high-confidence gate there as inline. A realtime check added to that list would change production remediation without the gate seeing it. supply_chaincannot fire: a fresh install has no advisory file.group_writable_phpcannot fire: planted files belong to the test user’s group, not a web server group.cleanedis not exercised: the.htaccesscleaner is off by default, the full scan skips.htaccessand crontab findings, the crontab fix is operator-only, and no sample draws a Critical quarantine-eligible finding on a file both scan paths clean in place (seeremediationabove).- The throttled performance file checks run once per sample, in the second tier run only. A tier run that skips any check fails the sample.
- The full scan quarantine step is a copy of the scan job’s loop. A change to that loop must be mirrored in the driver.
- Realtime does not admit
.jsor.pharfiles, so those controls are not exercised on that path.
Private half
Real attack samples, scrubbed of customer data, live in a private GitLab
project in the same format, with <category>/<name>/ at the project root.
The project comes from the CSM_ATTACK_CORPUS_PROJECT CI variable, and it
must allow this project’s CI job token to read it.
This repository pins the private commit in
internal/attackcorpus/private.commit: one line holding the full
40-character lowercase commit hash. Moving the pin is a reviewed public
commit.
In CI the script does the following before any test runs. Any failure stops the job.
- It fails when
CSM_ATTACK_CORPUS_PROJECTis unset or the pin file is missing or malformed. The pin accepts one full lowercase hash, with or without a single final newline, before any clone or checkout runs. - It clones the project with the CI job token into a new temporary directory, open only to its owner and outside the checkout and its artifacts. A credential helper reads the token from the environment, so the token never appears in a command line, a URL or the clone’s git config, and no configured helper can store it. The helper serves only the exact project on the CI server over HTTPS, and Git refuses redirects. A temporary directory inside the checkout or reports is rejected. The temporary clone is removed on success or failure, including a failed clone.
- It checks out the pinned commit and fails unless
HEADis exactly that commit, so a short hash or a ref name fails. - It runs
scripts/fixturecheck -corpus -require-termsover the checkout. Every file is scanned for IPv4 addresses outside the documentation ranges and for the private terms in theCSM_PRIVATE_TERMSfile variable. Sample data is decoded first, one layer deep, as in the public check. - It sets
CSM_ATTACK_CORPUS_PRIVATEto the checkout andCSM_ATTACK_CORPUS_PRIVATE_REQUIRED=1, so the tests fail rather than run without the private half.
engines.txt records the pin as private corpus: <commit>.
Outside CI nothing is cloned and the private half is not required. A local
run uses the public half alone unless CSM_ATTACK_CORPUS_PRIVATE names a
checkout. engines.txt then says private corpus: local, unpinned, and
otherwise private corpus: none.
The coverage rules apply to the public half alone. The private half must load without validation errors and hold at least one sample. Sample names are unique across both halves. Reports and test logs print sample names and planted paths, so private samples follow the public naming rules for both: no real names, accounts, domains or mailboxes.
Running it
Run the gate in the release builder image, the one CSM_BUILDER_IMAGE and
CSM_BUILDER_TAG name in .gitlab-ci.yml:
GO_LINUX_RUNTIME=docker GO_LINUX_IMAGE=<release-builder-image> \
scripts/go-linux.sh bash scripts/attack-corpus-test.sh
The script checks that the YARA-X C library matches the Go module and the
active builder pin exactly, then runs every TestAttackCorpus test in the
harness, daemon and checks packages with -tags yara. It writes to
attack-corpus-results/, which the job keeps for one year, including on failure:
realtime.json,tier.json,fullscan.json: per sample results and a per-category summary (detected, gaps, missed, false positives, closed gaps, remediation mismatches, errors). The script fails if any is empty.engines.txt: Go version, YARA-X version, commit, the private corpus line (see Private half) and run time.tests.jsonl: thego test -jsonevent stream.
The output directory and its report files are open only to their owner, including when the script reuses an existing directory. The script does not change the umask, so under the usual 022 the drivers create files with the modes the daemon gives them. CI artifact access still follows the project’s artifact permissions.
Environment:
CSM_ATTACK_CORPUS_REPORT_DIRrequests a run and names the report directory. Without it the corpus tests skip.CSM_ATTACK_CORPUS_REQUIRED=1turns that skip into a failure. The script sets both.CSM_ATTACK_CORPUS_PRIVATEadds the private half, a corpus checkout in the same format. It must load and hold at least one sample.CSM_ATTACK_CORPUS_PRIVATE_REQUIRED=1fails the run whenCSM_ATTACK_CORPUS_PRIVATEorCSM_ATTACK_CORPUS_REPORT_DIRis not set, instead of skipping. The script sets it in CI.CSM_ATTACK_CORPUS_SHOW_ERRORS=1logs the full reason of a driver error instead of its type. Local runs only: a reason can quote decoded sample bytes, and CI keeps test logs as artifacts. The report script unsets it, including when inherited from the caller.
The other TestAttackCorpus tests check the drivers themselves, on their own
samples, and need no report directory.