Recorded finding streams
Calibrating correlation (the coordinated-attack threshold, corroboration
grading, sequence rules) needs what a real host produced over days: every
finding with its check, severity, timestamp, owner and text, including the
false-positive floods and the long-lived rows that never clear. CSM already
writes that stream: every dispatched finding lands in
/var/log/csm/audit.jsonl (rotated copies are gzip files beside it). A
recorded stream is a copy of those files with every identity removed.
scripts/finding-stream turns the raw files into an anonymized stream:
# On the operator machine, after copying the files read-only from the host:
go run ./scripts/finding-stream anonymize \
--salt-file ~/.local/share/csm/finding-streams/salt \
--out ~/.local/share/csm/finding-streams/host-a.jsonl.gz \
raw/audit.jsonl raw/audit.jsonl-*.gz
What the tool replaces, in every structured field and in free text:
- Host names become
host-<id>, account namesacct-<id>, domainsdom-<id>.example, mailboxesuser-<id>@dom-<id>.example(a mailbox truncated after the@keeps the sameuser-<id>). The<id>is derived from an HMAC of the value under a private salt, so the same salt maps one account to one pseudonym on every host and streams can be joined without knowing who is who. Names are replaced wherever they sit: in/home/<account>/paths,Account:lines, process context, LiteSpeed vhost tokens, and inside longer tokens such asexample.com-ssl_logorweb1.log. A name holding an underscore or starting with a dot, such as a bare mail login, is replaced as a whole wherever it is not glued to a letter or digit, ignoring case. Where identities overlap, the leftmost complete one wins, and a full mailbox or home path keeps its usual mapping. A replacement is never rewritten again. A learned name made only of punctuation is not replaced, so it cannot turn separators into identities; the final check refuses the run if one appears. The host’s short name (its first label, when it carries a digit) maps to the same pseudonym as the full name. Any other token that looks like a domain (two or more labels, alphabetic last label that is not a file extension) is mapped too, even if no field named it; that over-reaches on a few dotted names likeoptions.optionand is accepted. Domain-shaped names inside filenames are replaced even when no structured field named them. Only pseudonyms actually emitted by this run are exempt; a raw name beginning withhost-oracct-is still scrubbed. - IPv4 addresses map into 198.18.0.0/15 and IPv6 addresses into
2001:db8::/32, both reserved and never routed; a raw value always maps to
the same pseudonym. The IPv4 range holds 131,072 addresses, so distinct
addresses can share a pseudonym. A manifest counts distinct addresses and
distinct pseudonyms per family, and a replay cannot tell merged addresses
apart.
Loopback addresses, system users such as
rootornobody, and bare numeric uids are kept: they identify nobody and carry meaning. Equivalent IPv6 spellings share one pseudonym across all streams; IPv4 addresses written in IPv6 form use the IPv4 pseudonym. Addresses inside filenames and before numeric rotation suffixes are also replaced. This can replace address-shaped version numbers; privacy takes priority over preserving ambiguous numeric text. New audit rows can carry acidrslist. Those addresses use the same pseudonyms as addresses in text, keep their prefix lengths, and participate in the output leak check. Older audit rows remain readable without the list. - The details of a credential-leak finding are dropped entirely, and generic
password=,secret:andtoken=material is blanked anywhere, including quoted keys and values containing spaces. - Finding ids become
fid-<id>, a salted id of at least 128 bits that keeps case, so an action row can name its finding without exposing the raw id. - Timestamps, check names, severities, path structure below the account, plugin and file names, and process names are kept unless they contain an identity: they are what calibration reads. Paths and mailboxes in process command lines and parent processes also teach the scrubber which identities to remove elsewhere.
Before writing, the tool scans its own output for every identity it learned from structured fields, paths and mail addresses, for domain-shaped names, for raw ids (including inside longer tokens), and for any mailbox or address outside the reserved ranges. It refuses to write if it finds one. The raw-id scan includes check names and severity labels; these fields otherwise keep their original vocabulary. Structured finding ids must be ids this run emitted. A name glued to underscores or file extensions is still found. The summary prints row counts, the number of distinct checks, replacement counts, the time span and a salt fingerprint, never identities, check names, paths or the salt itself. An error names only the stream, the file’s position on the command line, the line number and a fixed reason.
Every row must parse as exactly one JSON object of the known schema, with a supported version and a timestamp field. Rows that older CSM versions wrote with a zero time are kept, counted in the manifest and left out of time spans. Unknown or repeated fields, nulls, data after the object and values over the size limits refuse the whole run. Malformed Unicode, non-JSON whitespace and timestamps that cannot be written back as JSON also refuse the run before a salt is created. Typed rows accept only the exact pseudonym spellings emitted during that run.
Handling rules:
- Copy the audit files read-only (
taroverssh, orscp) into a local directory with mode 0700, run the tool, then delete the raw copies. Nothing runs on the monitored host. - The salt file is created exclusively on first use with mode 0600. Existing salts must be regular files without group or other access; symlinks are refused. A concurrent run that reads an unfinished salt fails without replacing it; retry after the creator finishes. Keep it private and reuse it for every host whose stream should be joinable with the others; losing it makes new recordings unjoinable with old ones.
- Outputs cannot replace an input, the salt, the input manifest or each
other, whether named by path, through a symlinked directory or as a hard
link, and an existing output must be a regular file. These checks run
before the salt is created, including for missing directories and paths
containing
... Case-only and Unicode-equivalent path variants are refused on every platform, as are paths that would need a file to also serve as a directory. Every output is written to a private temporary file beside its destination, and all are complete before any is renamed into place; if publishing fails partway, the outputs already replaced are restored. New directories have mode 0700 and outputs mode 0600, even when replacing a less restricted file. - Filesystem cleanup failures are reported without claiming success. If rollback fails, remaining recovery copies are retained. If every output was published but backup cleanup fails, the error explicitly says the outputs were published; check the manifest and remove the leftover backups before sharing the directory. Other cleanup errors report unchanged outputs with temporary files left to remove.
- Recorded streams stay outside the repository entirely, in a private
directory such as
~/.local/share/csm/finding-streams/with mode 0700. A pseudonymized stream still describes real incidents on a real host, and this repository is public. Do not keep them in an ignored directory inside the checkout:git clean -fdxremoves ignored files too, and a recording that took a host weeks to accumulate is not reproducible from anywhere else.
Joining actions and firewall entries
The action log (actions.jsonl) and the firewall audit log
(<state>/firewall/audit.jsonl) say what CSM did about the findings. The same
run can anonymize both next to the findings, so the three streams share
pseudonyms. A joined run needs a manifest, and a manifest needs a build of a
known commit without local changes, so build the tool from a clean checkout
first:
go build -o /tmp/finding-stream ./scripts/finding-stream
/tmp/finding-stream anonymize \
--salt-file ~/.local/share/csm/finding-streams/salt \
--out host-a/findings.jsonl.gz \
--actions raw/actions.jsonl --actions-out host-a/actions.jsonl.gz \
--firewall-audit raw/firewall-audit.jsonl --firewall-out host-a/firewall.jsonl.gz \
--manifest host-a/manifest.json \
raw/audit.jsonl raw/audit.jsonl-*.gz
Action and firewall rows are rebuilt from a closed list of fields rather than scrubbed:
- An action row keeps its time, operation and action, actor kind, result, a
reason category, whether an error occurred, whether the file existed before
and after, a block lease, and salted ids for the finding, incident, action
and the action it undoes. Addresses map as in findings, networks keep their
prefix length and endpoints their port and protocol. File paths, process ids
and other targets become
tid-<id>. Command lines, error and reason text, undo commands, recovery paths, digests, sizes, modes and owners are dropped. Account names are always mapped, system users included. - A firewall row keeps its time, action, target, reason category, source and lease. Firewall entries carry no ids, so nothing joins them to actions or findings; the manifest says so rather than guessing from times or addresses.
- Admission rows keep their kind and step, including the retirement of an
unknown outcome (
expired_unproven). An audit row’s target maps like a firewall target, so it joins the legacy row for the same address, and its reason category names the lane. An hourly summary, which setscount, keeps its check, entry, decision, refusal reason and attribution cause, dry-run prediction, count andepochand names no target; a check that matches a learned account, host or domain refuses the run. An attribution cause belongs only on a refusal summary; action inspection rows carrying one refuse the run. Anepochbelongs only on a summary; any other row carrying one refuses the run. - An operation, action, actor, result or source the tool has not been reviewed against refuses the run instead of passing through.
The manifest is written last and is the bundle’s completion marker. It lists every input and output with its stream kind, position, SHA-256 of the exact bytes, record count and time span, the tool’s source revision, the salt fingerprint, join counts, counts of discarded fields, action results by value (admission steps and decisions included, at zero when absent), and coverage for each stream. Check every output’s digest against it; a bundle without a manifest, or with a digest that does not match, is incomplete.
- An action is matched when it names a finding id present in the recording, and nothing more. Actions naming a finding the recording lacks, and actions naming none, are counted apart.
- Repeated finding rows are kept and counted. A durable action row repeated exactly is a retransmission and does not count as another outcome; rows that share an action id and version but differ are reported as conflicting, never resolved by taking the latest.
- A result is what the writer recorded. An applied block or a firewall entry is an observation, not a verified effect or a reviewed correct action.
--input-manifest takes the collector’s inventory of what the host has:
{"v": 1, "streams": [
{"kind": "findings", "availability": "present", "sha256": "<digest of the copied file>", "records": 1200},
{"kind": "actions", "availability": "not_recorded"},
{"kind": "firewall_audit", "availability": "absent"}
]}
Every supplied file must appear as a present entry with its digest and record
count, and every present entry must be supplied. absent means the collector
looked and found none; not_recorded means the host does not keep that
stream. Ledger and review streams can only be stated absent or not recorded.
Without an inventory, a stream that was not supplied is reported as not
supplied.
Comparing the admission preview
While the admission ledger only previews or forecasts, the legacy path
decides every automatic block and the ledger sees the same responses.
Compare a dry-run window: once admission executes, the legacy path leaves
the blocks admission owns to it. compare reads the outputs of one joined
run and sets each legacy automatic block in the preview window against what
the preview did with its finding. Older unstamped finding rows still supply
their identifiers and checks for joins:
/tmp/finding-stream compare --findings host-a/findings.jsonl.gz --actions host-a/actions.jsonl.gz --status host-a/status.json
--status takes the host’s csm status --json output, taken at the end of
the run being judged; compare reads only the ledger’s ingress generation,
which names that run, and the admission owner’s block_entries from it.
Each hourly summary names the daemon run that counted it (epoch), and
compare reports each run apart, since a restart loses the hour in progress.
Summaries recorded before they named their run are reported as one unmarked
run. A run’s window spans its first to its last hourly admission summary;
the hour the first one counts may have begun before the run, so it is not
counted. A block
is matched by one observed attempt for the same finding, response kind and
typed target within an hour; a scan block retried from the pending queue may
land later, until its retry ages out two hours after it was queued. Spray
blocks match through their own entries: the incident spray block of one
address through incident_spray, the mail spray subnet block through
mail_subnet. Matching can reassign compatible decisions when their windows
overlap; each step explains at most one legacy row across all runs. An
unused step from before a restart may still explain a pending retry. The
same legacy row in a shared boundary hour needs only one step, even when
both run reports include it. Observed steps and coalesced decisions are
matched before designed refusals, which explain discrepancies but do not
satisfy coverage. Coalesced decisions and designed refusals each spend one
summary count from their own run for the same entry, kind and check in the
same or an adjacent hour. A block the old process made in the hour of a
restart also falls in the next run’s window; when the earlier run’s own
summaries explain it, the next run reports it as explained by another run,
and it adds no coverage there. When both runs can explain several boundary
blocks, matching assigns distinct rows before spending spare counts on rows
already explained by the other run. Each run still spends only its own
counts and earns coverage only from its own matches. Hourly aggregation
cannot prove the exact event delay. A backward wall step clamps counts to
the admission hour floor; larger clock disagreement stays unexplained and
must be checked through clock status.
Attribution explains only scan-pass provenance gaps for checks served through
the matching entry; policy explains only netblock, permanent-block escalation,
challenge timeout, incident or incident spray paths without a retained root. A rootless
derived path may spend its fixed unknown-check policy count without
a finding link. Other policy and attribution refusals stay unexplained.
Summaries must be stamped at an hour boundary, and only refused decisions
may carry a refusal reason. Counts that overflow refuse the comparison,
and output errors never repeat file paths or host data.
Summary matches are aggregate lower bounds, not proof about one finding. Anything else is listed
as unexplained. The report also totals Invalid and queue overflow refusals,
and names what only csm status and the metrics show: Critical deferrals,
queue evictions and the handoff latency.
Each run ends with its gate; only the run the status names is judged. It
passes once its window spans at least 48 hours, nothing is unexplained, both
totals are zero, and three blocks are matched for every entry the status
lists and for every scan check that blocked, even if a later configuration
change removes scan blocking from the status. An entry short of three, or
listed but never blocking, holds the gate until the run spans seven days;
the gate then passes and names it unproven. An unmarked run, or a comparison
without --status, never passes. The report ends with the verdict on the
status’s run: an earlier run that passed does not pass it.
The operator must also verify the named status and metric criteria. Shape
validation cannot authenticate a prior run: compare accepts outputs only
from the joined run that already passed its identity verifier. The report
describes one host’s responses; keep it private. Summary retention discards
whole oldest hours after failed writes exceed its bound. Missing summaries
may mean idle hours, failed writes or a crash; they cannot certify coverage.
Rootless designed-refusal matches consume aggregate counts and provide no
finding join. The operator must prove collector coverage and an undamaged
ledger through status and doctor before accepting the comparison as passed.
What a recording does and does not contain
The audit log is the dispatch record: it holds findings that were alerted, after deduplication. It is not the persisted latest-state set, which is larger because every scan re-emits the findings it still sees and the merge refreshes their timestamps. On one production host a single sweep dispatched 49 critical findings while the active set carried 157 refreshed critical rows.
A replay therefore understates how full the persisted correlation window gets. Compare windows against each other, and treat an absolute rate from a replay as a property of the replay, not of the live host.
Replaying a stream through correlation
scripts/correlation-calibrate replays a recording through the production
cross-account correlation so its thresholds can be re-derived from what hosts
produced rather than from an assumption:
go run ./scripts/correlation-calibrate \
~/.local/share/csm/finding-streams/host-a.jsonl.gz
It reports how often account-and-check pairs repeat, which checks dominate
the input, and what the aggregates did under both derivations:
per dispatch batch, and over the persisted latest-state set. For each it prints
a threshold sweep using the same eligible, windowed accounts as the aggregate.
--window selects the persisted correlation bound: it defaults to
one hour, and --window 0 reproduces the original unbounded correlation.
Longer windows are applied directly, without the production default limiting
them. The source set keeps the store’s retention and size limit. Batch
correlation counts all qualifying rows grouped into that dispatch batch.
Repeated pairs can include distinct findings on the same account; this metric
does not establish how many rows are re-reports of the same finding.
Batch boundaries are inferred from timestamp gaps. Persisted correlation is recomputed at every recorded arrival, including ignored checks that only advance time. Raised duration is measured between those observations, not against an invented scan schedule. Recordings do not contain empty scans, purges or dismissals, so this replay models latest-state accumulation rather than reconstructing every store transition. Rows with no timestamp are counted and skipped because they have no replay position; runtime correlation still counts unstamped stored rows.
The tool needs no host access and reads nothing but the recording. It uses cPanel account roots supplied directly to correlation, without platform discovery on the replay machine.
Replaying scan admission
scripts/response-replay replays a recording through a model of how
automatic scan blocks are admitted today: the hourly block limit, the retry
queue and its age limit, and eviction at the temporary deny limit. It reports
what the model would have done with the recorded findings, not what the host
did, and not whether any block was right.
go build -o /tmp/response-replay ./scripts/response-replay
/tmp/response-replay --findings host-a/findings.jsonl.gz --out replay/host-a.json \
--max-blocks-per-hour 200 --deny-temp-ip-limit 500 --seed 1 --hour-zone UTC \
--manifest host-a/manifest.json
- The live path tries queued candidates in Go map order. The model tries them
in a reproducible random order chosen by
--seed, so compare several seeds rather than reading one run as the order the host used. --hour-zoneis the zone the host’s clock runs in: the hourly limit resets when that zone’s hour changes.Localis refused because it would depend on the machine running the replay.--php-relay-ip-recipient-modemirrorsemail_protection.php_relay.ip_recipient_modeand defaults toobserve. Useenforceto model form-relay client responses. Observed findings never drive a response in either mode. The report records the selected mode.- A recorded block from a challenge timeout, central intel, a credential spray or an incident is applied as recorded: it takes a firewall slot but not the hourly budget. Only a single IP address is accepted as the block target; malformed targets are counted as unclassified rows. A permanent block is counted, not modelled.
- The audit log does not record a finding’s structured source address, so
a reputation finding, which names its address only there and in its
message, has no address in a replay by default and is counted as missing
one.
--reconstruct-reputation-sourcerecovers the address from that message’s fixed form; the report then counts the recovered rows and lists the reconstruction among its assumptions. - A finding keeps the severity it was recorded with. A later change in how a check grades severity is not applied, so an older recording can replay as blockable a finding the current build would only challenge. Every report lists this among its assumptions.
- Batches are inferred from equal timestamps. Recordings hold no empty scans, so queued work drains only when another finding arrives. Rows without a timestamp are counted and left out.
- The report lists the effective policy with every default it applied, the counts, queue delay and eviction residence distributions, blocks per elapsed hour, and the assumptions and mechanisms the model leaves out: infrastructure and allowlist protection, verdict callbacks, subnet blocks, permanent escalation, retries and failures, and manual unblocks. It carries no address, id, name or text from the recording.
- Candidates the queue dropped (aged out or overflowed) are reported apart from work still queued when the recording ended, which the end of the recording cut off rather than the queue.
- The demand is the audit log’s record of dispatched findings, not every finding the admission path was called with.
- The report counts the recorded subnet, ASN-crawl, netblock, permanent and dry-run block rows the model leaves out, and the findings that feed the subnet paths. ASN-crawl subnets spend the same hourly budget the model gives single addresses, so free scan slots in a replay are an upper bound.
- Hourly distributions include empty elapsed hours across the full recorded time span, without allocating a sample for each empty hour. The first stamped row establishes the replay clock, including dates before year one.
- With
--manifest, the report checks that the manifest describes this exact recording and carries its coverage and recorded outcomes beside the replay. Without one, the report says the recording has no statement of what was collected with it. Negative outcome and join counts are refused. - A report is written only to a new path: it never replaces an existing file, and it cannot alias its recording or manifest through symbolic links, parent-directory traversal or hard links. It is staged beside the resolved destination and published as a private file, refusing the run if another writer creates the destination in the meantime. A temporary file cleanup failure is reported with whether the report was published.
- Like a manifest, a report needs a build of a known commit without local changes.