Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

CSM Documentation

CSM is a local security monitoring and response daemon for Linux web servers, with or without a control panel. Platform detection picks the paths and checks for cPanel/WHM, Plesk, DirectAdmin, or no panel, on Apache, Nginx, LiteSpeed, or no web server. cPanel hosts get the deepest account and CMS integration; see supported platforms.

Start here

How CSM works

Real-time watchers process filesystem, authentication, access-log, mail, PAM, BPF, and ModSecurity events. Scheduled scans cover host integrity, accounts, CMS files and databases, packages, mail configuration, and performance. Findings are stored locally and can feed alerts, incidents, the Web UI, API clients, Prometheus, syslog, webhooks, and SIEM exports.

Platform-specific checks skip when their required panel or service is absent. The documentation calls out cPanel-only behavior instead of treating a skipped integration as a failure.

Safety model

Auto-response is off by default. Automatic IP blocking has a dry-run default, while file, process, mail, and BPF actions have their own gates. Infrastructure and operator-allowed addresses are protected from automatic blocks, quarantine records restoration metadata, and firewall configuration supports confirmation-based rollback.

Use Configuration as the source for current defaults and Upgrading for state, config, and package migration rules.

Installation

Supported Platforms

PlatformWeb serverPackageNotes
AlmaLinux / Rocky / RHEL / CloudLinux / CentOS Stream 8+Apache (httpd), Nginx, LiteSpeed, or none.rpmHost, web, and mail checks for the detected stack, with rpm -V package integrity.
Ubuntu 20.04+ / Debian 11+Apache (apache2), Nginx, LiteSpeed, or none.debSame, with debsums/dpkg --verify in place of rpm -V.
cPanel/WHM on CloudLinux / AlmaLinux / Rocky / UbuntuApache (EA4) or LiteSpeed, with or without ea-nginx.rpm or .debAdds cPanel account, WordPress, Exim, and WHM plugin coverage.

Postfix hosts with Dovecot also get mail login detection and optional rspamd spam filtering. On hosts without cPanel, set account_roots for real-time document-root rules, scheduled YARA sweeps and web-root performance checks. Other account and CMS scans still depend on account layouts; see deep check platform support.

The daemon auto-detects the OS, control panel (cPanel/Plesk/DirectAdmin/none), and web server (Apache/Nginx/LiteSpeed) at startup. The detected platform is logged at startup as:

[2026-04-10 08:13:37] platform: os=ubuntu/24.04 panel=none webserver=nginx

Check it with journalctl -u csm.service | grep platform: after starting the daemon.

The package repository at mirrors.pidginhost.com/csm/ is the preferred install method for Debian and Ubuntu. Updates are available through apt upgrade, and APT verifies the repository’s signed metadata before installing packages.

# 1. Install the signing key
curl -fsSL https://mirrors.pidginhost.com/csm/csm-signing.gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/csm.gpg

# 2. Add the repository
echo "deb [signed-by=/etc/apt/keyrings/csm.gpg] https://mirrors.pidginhost.com/csm/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/csm.list

# 3. Install
sudo apt update
sudo apt install csm

Works on Ubuntu 20.04+, Debian 11+, and compatible derivatives. The single stable suite serves every supported release. Production binaries dynamically link glibc with a build floor of 2.28, which is available on all supported distributions; YARA-X itself is linked into the executable.

To upgrade later: sudo apt update && sudo apt upgrade csm.

# 1. Import the signing key into the RPM keyring
sudo rpm --import https://mirrors.pidginhost.com/csm/csm-signing.gpg

# 2. Add the repository
sudo tee /etc/yum.repos.d/csm.repo >/dev/null <<'EOF'
[csm]
name=CSM - Continuous Security Monitor
baseurl=https://mirrors.pidginhost.com/csm/rpm/el$releasever/$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.pidginhost.com/csm/csm-signing.gpg
EOF

# 3. Install (check the fingerprint below before accepting the key)
sudo dnf install csm

rpm --import populates the RPM keyring, which covers gpgcheck – the signature on the package. It does not populate the separate keyring dnf keeps for repo_gpgcheck, the signature on the repository metadata. So the first transaction against a new repository still prompts, even after a successful import:

Importing GPG key 0x81CD59B1:
 Userid     : "CSM Package Signing (CSM Repository Metadata Signing Key) <security@pidginhost.com>"
 Fingerprint: 3A70 4D78 3CF2 6055 B2AA 8F49 4E0F 27F5 81CD 59B1
Is this ok [y/N]:

Without -y, an unanswered key prompt defaults to no. A rejected key can produce an error such as:

Error: Failed to download metadata for repo 'csm':
repomd.xml GPG signature verification error: Bad GPG signature

The error alone does not distinguish an untrusted key from damaged or incorrectly signed metadata. Check the fingerprint against the value above, then verify the metadata signature:

base=https://mirrors.pidginhost.com/csm/rpm/el9/x86_64/repodata
curl -fsSLO $base/repomd.xml
curl -fsSLO $base/repomd.xml.asc
curl -fsSL https://mirrors.pidginhost.com/csm/csm-signing.gpg | gpg --import
gpg --verify repomd.xml.asc repomd.xml
# gpg: Good signature from "CSM Package Signing ... <security@pidginhost.com>"

For unattended installs, approve the repository key in provisioning first, then use sudo dnf -y install csm. DNF’s assumeyes option accepts key-import prompts as well as package prompts. Keep both signature checks enabled; -y is not a substitute for verifying the expected key.

The $releasever variable auto-selects the matching EL major (8, 9, or 10). Both x86_64 and aarch64 are published. Works on AlmaLinux 8+, Rocky 8+, RHEL 8+, CloudLinux 8+, and cPanel-managed hosts.

To upgrade later: sudo dnf upgrade csm.

Online standalone installer

Use the standalone installer when the host has Internet access but cannot use the APT or DNF repository. It downloads the binary and supporting assets from the latest GitHub release, verifies their checksums, requires successful Ed25519 signature verification, and installs outside the package manager.

curl -fsSLo /tmp/csm-install.sh https://raw.githubusercontent.com/pidginhost/csm/main/scripts/install.sh
less /tmp/csm-install.sh
sudo bash /tmp/csm-install.sh

Standalone verification uses OpenSSL 3.0 or newer, an already installed CSM build providing csm verify-release, or python3-cryptography – in that order. EL8 and CloudLinux 8 have the last of these, so the standalone path works there even though their OpenSSL 1.1.1 cannot verify Ed25519. A missing key, missing current-release signature, absent verifier, or failed verification stops the install before executing the binary. See Release signing for the narrowly scoped historical-release exception.

It auto-detects the hostname and alert email, generates a Web UI token, and prompts before applying. Non-interactive mode:

sudo bash /tmp/csm-install.sh --email admin@example.com --non-interactive

This is not an offline or air-gapped installation path. Mirror the signed packages and repository metadata for disconnected environments.

Manual .rpm / .deb download

If you need a specific version or want to install without adding the repository, verify its detached signature before invoking the package manager. Save the trusted public key from Release signing as csm-signing.pub. These commands require OpenSSL 3.0 or newer; older hosts should use the signed repository above. Installing a local package does not by itself establish the repository signature chain:

# RHEL family
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm-VERSION-1.x86_64.rpm
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm-VERSION-1.x86_64.rpm.sig
openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
  -sigfile csm-VERSION-1.x86_64.rpm.sig -in csm-VERSION-1.x86_64.rpm && \
  sudo dnf install -y ./csm-VERSION-1.x86_64.rpm

# Debian/Ubuntu
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm_VERSION_amd64.deb
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm_VERSION_amd64.deb.sig
openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
  -sigfile csm_VERSION_amd64.deb.sig -in csm_VERSION_amd64.deb && \
  sudo apt install -y ./csm_VERSION_amd64.deb

Replace VERSION with a published version. Both files are also available from the package mirror if you need to pin a release without adding the repository.

Filesystem layout

The package uses FHS paths for config, state, drop-ins, and shipped profiles. Upgrades keep /opt/csm/csm.yaml as a compatibility link for older scripts:

ConcernCurrent path
Main config/etc/csm/csm.yaml
Legacy config link/opt/csm/csm.yaml
Drop-in fragments/etc/csm/conf.d/*.yaml
State directory/var/lib/csm/state/
Shipped profiles/usr/lib/csm/profiles/
Shipped config template/usr/lib/csm/csm.yaml.default
Audit log/var/log/csm/audit.jsonl
Binary/opt/csm/csm
Quarantine/opt/csm/quarantine/
YARA / signature rules/opt/csm/rules/

On Debian and Ubuntu, removing the package keeps the main config. Purging the package removes the main config and its package-created compatibility link. If the main config is a symlink, purge leaves its target in place.

The systemd unit declares StateDirectory=csm and ConfigurationDirectory=csm so systemd manages permissions for the FHS directories. /var/lib/csm is mode 0711, searchable but not listable, so Exim can reach the email forward guard files below it; /var/lib/csm/state/ stays 0700. On upgrade, the package copies a real legacy main config into /etc/csm/csm.yaml when needed and points /opt/csm/csm.yaml at it. On first start the daemon copies a non-empty legacy /opt/csm/state/ into /var/lib/csm/state/ (only when the new directory is empty), then continues using the FHS state path. See Upgrading - FHS migration for the manual-binary-swap case.

The binary installer and package scripts also keep the shared state root searchable, including on systemd 239, which does not reset an existing state directory’s mode. Applying the forward guard repairs legacy parent permissions without making private directories listable and rejects symlinks in those parents. Install and csm rehash likewise refuse a symlinked /var/lib/csm; relocate state with state_path instead. Lookup refreshes maintain search access to the lookup directory inside the daemon sandbox; full guard application runs outside it.

When refreshing the service unit, install and csm rehash create any missing directory the unit requires before replacing it, preserving existing modes except for the shared state root’s required search bits. Optional paths and systemd-managed directories are left alone unless needed as parents of a required path. If a required directory cannot be created, the refresh fails and the previous unit stays in place.

Post-install (all methods)

RPM/DEB packages are the maintained installation path and include package-manager ownership, integration profiles, UI assets, rules, and the prebuilt PAM module. The standalone installer supplies the runtime assets but does not register them with a package manager. In either case, set infrastructure IPs and confirm the alert address before starting the daemon.

sudo vi /etc/csm/csm.yaml              # Set hostname, alert email, infra IPs
sudo csm validate                      # Check config syntax (--deep adds connectivity probes; validates merged conf.d too)
sudo systemctl enable --now csm.service
sudo csm baseline                      # Record current state as known-good via the daemon (must be running)

Then open the Web UI at https://<server>:9443/login. The baseline is detailed under Baseline Scan.

Rollback to an older version

Both the APT and DNF repositories retain the last 5 tagged releases at any time. To downgrade:

# Debian/Ubuntu
sudo apt-cache policy csm              # Show available versions
sudo apt install csm=<version>-1

# RHEL family
sudo dnf --showduplicates list csm     # Show available versions
sudo dnf downgrade csm

Verifying platform auto-detection

After systemctl start csm.service, the first line after “CSM daemon starting” reports what CSM detected:

[2026-04-10 08:13:37] CSM daemon starting
[2026-04-10 08:13:37] platform: os=almalinux/10.0 panel=none webserver=apache
[2026-04-10 08:13:37] Watching: /var/log/secure
[2026-04-10 08:13:37] Watching: /var/log/httpd/error_log
[2026-04-10 08:13:37] Watching: /var/log/httpd/access_log

If any field shows none or unknown when you expect something, the auto-detect missed it. File a bug with the output of cat /etc/os-release, systemctl is-active nginx apache2 httpd, and which nginx apache2 httpd.

Optional system dependencies

The daemon runs as one executable plus packaged UI, rule, profile, and PAM assets. Release binaries require glibc 2.28 or newer and the package depends on systemd. A few host packages enable additional checks:

PackagePlatformsEnables
auditdAllShadow file / SSH key tamper detection via auditd
debsumsDebian/UbuntuCleaner system binary integrity output vs. dpkg --verify fallback
logrotateAllRotation of /var/log/csm/monitor.log, /var/log/csm/audit.jsonl, and the PHP Shield event log
wp-cliOptionalWordPress core integrity check; its absence is reported when WordPress sites are found or discovery is incomplete
ModSecurityAllWAF enforcement checks (see platform-specific install below)

Installing ModSecurity

CSM detects ModSecurity but doesn’t install it for you. Platform-specific commands:

# Ubuntu/Debian + Nginx
sudo apt install libnginx-mod-http-modsecurity modsecurity-crs

# Ubuntu/Debian + Apache
sudo apt install libapache2-mod-security2 modsecurity-crs && sudo a2enmod security2

# AlmaLinux/Rocky/RHEL + Apache (requires EPEL)
sudo dnf install -y epel-release
sudo dnf install -y mod_security
sudo systemctl restart httpd

# AlmaLinux/Rocky/RHEL + Nginx (requires EPEL)
sudo dnf install -y epel-release
sudo dnf install -y nginx-mod-http-modsecurity
sudo systemctl restart nginx

After installing ModSecurity, run csm check and the waf_status finding should disappear.

Manual (deploy.sh)

/opt/csm/deploy.sh install
vi /etc/csm/csm.yaml   # set hostname, alert email, infra IPs
csm validate
systemctl enable --now csm.service
csm baseline

Baseline Scan

The csm baseline command scans the entire server and records the current state for change tracking. This is required on first install so CSM knows what’s “normal” for your server. Findings that should never be silently trusted, such as non-standard MySQL superusers or WHM root API tokens, can still be reported on this first scan.

What it does:

  • Scans all cPanel accounts for malware, permissions, and configuration issues
  • Records file hashes, email forwarder hashes, and plugin versions
  • Stores everything in the bbolt database (/var/lib/csm/state/csm.db)

How long it takes: Depends on server size. A server with 100+ cPanel accounts and thousands of WordPress sites can take 5-10 minutes. The daemon must be running because the baseline is coordinated through the control socket.

When to re-run:

  • After a fresh install
  • After restoring from backup
  • After an intentional state reset approved by the operator
  • You do NOT need to re-run for normal deploys/upgrades – the daemon handles incremental state

Important: Start csm.service before running csm baseline. If existing history would be cleared, rerun with csm baseline --confirm only after verifying that reset is intended.

Configuration

CSM is configured via /etc/csm/csm.yaml, with --config <path> to override. Legacy installs that only have /opt/csm/csm.yaml keep working; packaged upgrades migrate that file into /etc/csm/csm.yaml and leave the old path as a compatibility link. Optional drop-in fragments under /etc/csm/conf.d/*.yaml are merged on top of the main file at startup; see conf.d drop-ins below.

Environment-backed tokens and signing secrets use the environment inherited at daemon startup. After changing an environment file, restart the daemon; configuration reload alone does not import those changes. See credential rotation for the systemd procedure.

Installation defaults

Packages and csm install write the same commented configuration, so a new host starts with the same settings whichever way it was installed. A key left out of the file takes the built-in default, which can differ from the value the shipped file sets. Left out or left blank, backdoor_ports and firewall.smtp_allow_users take the shipped lists; write [] for none. Users missing from the host are skipped.

The CLI generates a fresh Web UI admin token and issue time, validates the result, and writes the new configuration with mode 0600. Invalid templates stop installation without replacing the destination configuration.

Operating mode

mode declares what CSM may do to the host. enforce (default) leaves every subsystem under its own switch; observe runs detection and alerting without automatic host remediation or integration updates. Full reference, including required switch settings: Observe mode.

mode: enforce

Platform & Web Server

CSM auto-detects the host OS (Ubuntu, Debian, AlmaLinux, Rocky, RHEL, CloudLinux), control panel (cPanel, Plesk, DirectAdmin, or none), and web server (Apache, Nginx, LiteSpeed, or none) at daemon startup. The detected platform is logged as:

[2026-04-10 08:13:37] platform: os=ubuntu/24.04 panel=none webserver=nginx

The daemon then chooses the correct log paths, config candidates, and check set without any configuration from you. Verify with:

journalctl -u csm.service | grep platform:

Web server overrides

For hosts with a custom layout (reverse proxy, non-standard package locations, chroot), add a web_server: section to csm.yaml. Every field is optional – anything left blank falls back to auto-detection.

web_server:
  type: "nginx"                          # apache | nginx | litespeed -- overrides auto-detect
  config_dir: "/etc/nginx"               # Apache or nginx configuration root
  access_logs:                           # tried in order until one exists
    - "/var/log/nginx/access.log"
    - "/srv/logs/nginx/access.log"
  error_logs:                            # used by ModSecurity deny watcher
    - "/var/log/nginx/error.log"
  modsec_audit_logs:
    - "/var/log/nginx/modsec_audit.log"

modsec_error_log (legacy single-path override) is still honored and takes precedence over web_server.error_logs for the ModSecurity watcher only:

modsec_error_log: "/opt/myapp/logs/modsec_audit.log"

Account roots (plain Linux web-scan coverage)

By default, web-root performance checks iterate /home/*/public_html, which is the cPanel layout. The WP-Cron check and remediation also use cPanel’s domain map for validated addon and subdomain document roots. On a plain Linux host, point CSM at the actual web roots:

account_roots:
  - "/var/www/*/public"            # e.g. Laravel/Symfony sites
  - "/srv/http/*"                  # Arch / generic layouts
  - "/home/*/public_html"          # add if you also have cPanel-style accounts

Each entry is an absolute, normalized glob pattern expanded at scan time. Configured directories also define remediation and restore scope. Custom locations need service write access; see Custom account roots. Non-existent matches are silently dropped. If account_roots is empty and CSM is not on a cPanel host, the account-scan checks return no findings (they run but find nothing, which is the correct behavior for a plain-Linux host with no configured web roots).

The setting covers perf_error_logs, perf_wp_config, perf_wp_transients, perf_wp_cron, real-time php.ini monitoring, and WP-Cron remediation roots. CMS integrity, phishing, .htaccess, and file-index scans use the platform’s account layout: every directory under /home on cPanel, DirectAdmin and plain hosts, under /var/www/vhosts on Plesk.

Minimal Config

hostname: "csm.example.com"

alerts:
  email:
    enabled: true
    to: ["admin@example.com"]
    disabled_checks: []                   # optional: suppress these checks from email only
    smtp: "localhost:25"

webui:
  enabled: true
  listen: "0.0.0.0:9443"
  auth_token: "<fresh random token>"
  auth_token_issued_at: "<UTC RFC3339 time when the token was generated>"

infra_ips: ["10.0.0.0/8"]

Full Reference

hostname: "csm.example.com"

# --- Alerts ---
# Email and webhook may both be off; findings then show only in the web UI and API.
alerts:
  email:
    enabled: true
    to: ["admin@example.com"]
    from: "csm@csm.example.com"
    smtp: "localhost:25"
    disabled_checks: []                 # check names to keep in web/history but exclude from email
  webhook:
    enabled: false
    url: ""
    type: "slack"                       # slack, discord, generic, phpanel
    hmac_secret: ""                     # phpanel webhook signing secret
    hmac_secret_env: ""                 # env var containing phpanel signing secret
  heartbeat:
    enabled: false
    url: ""                             # healthchecks.io, cronitor, dead man's switch
  max_per_hour: 10                      # alert emails/hour; CRITICAL always bypasses and is not counted. Code default 30; the shipped csm.yaml template sets 10
  block_digest:
    enabled: false                      # add per-country counts of auto-blocked IPs to the daily summary
    countries: []                       # empty = trusted countries, then all countries
    interval: "1h"                      # how often counts fold into the summary; live alert dedup window; below 1m runs as 1m and warns
    live: false                         # also send one alert per qualifying block
    send_on: "any"                      # live alerts: any | customer
    channel: ""                         # live alerts only: email | webhook; empty = enabled alert channels
    min_block: 1                        # watched blocks needed to show the summary section; 0 always shows it
  audit_log:                            # SIEM-friendly per-finding stream
    file:
      enabled: false
      path: /var/log/csm/audit.jsonl    # default; logrotate fragment ships with the package
    syslog:
      enabled: false
      network: udp                      # udp | tcp | unix | unixgram | tls
      address: 127.0.0.1:514            # host:port, or filesystem path for unix variants
      facility: local0                  # default: local0
      tls_ca: ""                        # optional CA cert for tls transport

# --- Integrity ---
integrity:
  binary_hash: ""                       # populated by baseline/rehash
  config_hash: ""                       # populated by baseline/rehash/reload
  confd_hash: ""                        # populated by baseline/rehash/reload
  immutable: true                       # apply chattr +i to the installed binary during install/rehash

# --- conf.d drop-in policy ---
confd:
  integrity_exempt: []                  # fragments an integration rewrites; left out of confd_hash (bare filenames)

# --- Thresholds ---
thresholds:
  mail_queue_warn: 500                  # default: 500
  mail_queue_crit: 2000                 # default: 2000
  deep_scan_interval_min: 60            # minutes between deep scans (default: 60)
  exposed_file_scan_depth: 2            # directory levels below each web docroot (default: 2, max: 10)
  multi_ip_login_threshold: 3           # IPs per account before alert (default: 3)
  multi_ip_login_window_min: 60         # time window for multi-IP check (default: 60)
  cred_stuffing_distinct_accounts: 5    # failed accounts from one IP before credential_stuffing (default: 5)
  pam_bruteforce_threshold: 5           # PAM auth failures from one IP before pam_bruteforce + auto-block (default: 5)
  pam_bruteforce_window_min: 10         # window for counting PAM failures per IP (default: 10)
  plugin_check_interval_min: 1440       # WordPress plugin check interval (default: 1440)
  brute_force_window: 5000              # failed auth attempts window (default: 5000)
  domlog_max_files: 500                 # per-domain access logs per WP brute-force scan (default: 500)
  domlog_tail_lines: 500                # trailing lines tailed from each domlog per scan (default: 500)
  domlog_max_age_min: 30                # skip per-domain access logs untouched in this many minutes (default: 30)
  mail_log_tail_lines: 500              # trailing lines of /var/log/exim_mainlog read by the mail-per-account scanner (default: 500)
  syslog_messages_tail_lines: 200       # legacy direct-check FTP tail only; daemon FTP detection follows the log forward and ignores this setting (default: 200)
  ftp_fail_window_min: 30               # sliding window (minutes) for per-IP pure-ftpd auth-failure accumulation before ftp_bruteforce; reported when a scan reads a new failure; diagnostic scans do not consume failures (default: 30)
  account_scan_max_files: 10000         # account and mail-domain paths per scanner cycle (default: 10000)
  # If this cap clips /home/<account>/ paths, account_scan_truncated names the affected account.
  # Rolling content coverage spends this cap once per cycle across the whole host, not once per account.
  crontab_base64_blob_max_bytes: 16384  # encoded bytes per crontab base64 candidate before decoded-content matching; must be a multiple of 4 (default: 16384)

  # Full-scan subsystem (`csm scan --full`) and rolling content coverage.
  full_scan_max_file_mb: 16            # cap on a single file scanned by a full scan, in MiB (default: 16)
  scan_job_retention: 20               # completed full-scan job records kept in the store (default: 20)
  rolling_coverage: true               # tri-state; default on. Normal periodic cycles prioritize new and changed PHP source under discovered document roots, then scan a slice of dormant files. Both share the account_scan_max_files budget; unfinished work resumes across cycles and accounts take turns. Set false to disable
  # Cursor write failures keep progress in memory until storage recovers; a restart loses that unpersisted progress.
  # Changed scans retry incomplete reads and directory listings, preserving earlier findings until coverage succeeds. Clock corrections rewind their progress when needed.
  # Rolling windows also preserve earlier findings when a content read fails. Reboots retain changed-scan progress instead of queuing all old files again.
  # Existing files on first discovery remain the rolling window's job. Coverage follows the same PHP extensions and depth limit as rolling scans.
  # Periodic forced rescans skip rolling coverage and expire old clean-file records without resetting changed-scan progress.
  dropper_detection: true              # tri-state; default on. Real-time flag for a PHP/executable created under a web docroot and unlinked before the TTL probe. Set false to disable
  dropper_unlink_ttl_sec: 300          # seconds a fresh docroot PHP/executable is tracked before the self-delete probe (default: 300, range 30-3600)

  # HTTP request flood, User-Agent spoof, and distributed HTTP detection.
  # These detectors scan the same per-vhost access-log stream as the WP
  # brute-force scanner; no extra log tailer is needed.
  #
  # http_flood_threshold: minimum per-IP request count inside the window
  # that emits http_request_flood. 0 disables the detector. The detector
  # ships disabled so operators can sample local baseline traffic first.
  # Adjust up for CDNs or CGNAT-heavy visitor pools before enabling.
  http_flood_threshold: 0              # 0 = disabled; set after sampling baseline traffic
  http_flood_window_min: 5             # rate window in minutes (default: 5)

  # http_ua_spoof_threshold: per-IP per-window count for non-browser UA
  # kinds (including claimed search-engine bots such as Googlebot/Bingbot/
  # Applebot that fail reverse-DNS confirmation) before http_ua_spoof fires.
  # A single bot-like request from a residential or mobile client no longer
  # hard-blocks it; sustained activity is required. Raise this for visitor
  # pools that legitimately send unusual User-Agents.
  http_ua_spoof_threshold: 30          # default: 30

  # xmlrpc_threshold: per-IP POST /xmlrpc.php count before xmlrpc_abuse fires
  # in access-log based detectors (a hard auto-block). WordPress sites using
  # Jetpack, the mobile app, or WooCommerce make many legitimate xmlrpc.php
  # calls, so this defaults higher than the legacy value. Set 0 to disable the
  # check entirely; an absent key uses the built-in default.
  xmlrpc_threshold: 100                # default: 100; 0 = disabled

  # http_distributed_min_ips: distinct already-abusive source IPs that hit
  # the same vhost in one scan window before a per-vhost distributed flood
  # finding fires. 0 disables the rollup for existing configs that do not
  # opt in.
  http_distributed_min_ips: 10         # sample setting; omit or set 0 to disable

  # URL scanner profile: fires http_scanner_profile for source IPs whose
  # in-window traffic is almost entirely probe-error responses spread
  # across many distinct request paths -- the shape of random-URL
  # enumeration hunting for downloadable files, exposed backups, and
  # dormant shells. Three gates must all pass: a minimum request volume,
  # a minimum error percentage, and a minimum count of distinct error
  # paths (query strings stripped, so cache-buster URLs on one missing
  # endpoint count once). A visitor following dead links, a broken image
  # hammered by one page, and a site migration all stay out of scope.
  #
  # 301 is deliberately not in the default status set: http->https and
  # www redirects make every legitimate visitor 301-heavy, and a site
  # migration redirects entire domains. Add 301 to
  # http_scanner_status_codes only on hosts where that traffic shape is
  # impossible.
  http_scanner_min_requests: 0         # volume gate; 0 = disabled (default); 30 is a safe start
  http_scanner_error_pct: 90           # min % of requests with probe-error status (default: 90)
  http_scanner_min_distinct_paths: 10  # min distinct error paths, max 512 (default: 10)
  http_scanner_status_codes: [404, 403] # statuses counted as probe errors (default: 404, 403)

  # Distributed expensive-request crawl from one ASN. The detector fires
  # only when request breadth, uncacheable volume, account share, and PHP
  # worker saturation agree. Set min_ips to 0 to disable.
  http_asn_crawl_window_min: 60
  http_asn_crawl_min_ips: 25
  http_asn_crawl_min_expensive: 250
  http_asn_crawl_min_share_pct: 50
  http_asn_crawl_high_amp_pct: 50
  http_asn_crawl_high_volume_mult: 4
  http_asn_crawl_saturation: 0          # 0 = performance.php_process_warn_per_user
  http_asn_crawl_max_prefix: 8
  http_asn_crawl_16_pref_pct: 60
  http_asn_crawl_max_tracked_ips: 20000
  http_asn_crawl_allowlist_asns: []
  http_asn_crawl_reverse_proxy_asns: [13335, 54113, 20940]

  # These three opt-in flags extend UA spoof detection to additional UA
  # classes. Leave disabled on busy shared hosts; scripting-language agents
  # and headless browsers appear on many legitimate monitoring stacks.
  http_ua_scripting_enabled: false     # flag curl/wget/python-requests/Go-http style UAs
  http_ua_headless_enabled: false      # flag Puppeteer/Playwright/PhantomJS UAs
  http_ua_empty_enabled: false         # flag requests with no UA at all

  # SMTP brute-force tracker (Exim mainlog, dovecot SASL on submission ports)
  smtp_bruteforce_threshold: 5            # per-IP failed auths before block (default: 5)
  smtp_bruteforce_window_min: 10          # sliding window in minutes (default: 10)
  smtp_bruteforce_suppress_min: 60        # cooldown between repeat findings (default: 60)
  smtp_bruteforce_subnet_threshold: 8     # unique IPs per /24 before subnet block (default: 8)
  smtp_account_spray_threshold: 12        # unique IPs targeting one mailbox before visibility finding (default: 12)
  smtp_bruteforce_max_tracked: 20000      # soft cap on tracked entries; oldest evicted (default: 20000)
  # Slow-brute pair: failed auths from one IP across 3+ distinct mailboxes
  # within the long window. Catches attackers pacing below the fast window.
  # Also fires on breadth alone: 10+ distinct walked mailboxes trigger the
  # block regardless of the failure count. A successful SMTP auth from the
  # source inside the window disqualifies it (live office NAT, not a walker).
  smtp_bruteforce_slow_threshold: 40      # failures before block (default: 40; 3-1024; explicit 0 disables)
  smtp_bruteforce_slow_window_min: 360    # slow sliding window in minutes (default: 360; 1-10080)

  # SMTP probe-abuse tracker (raw connect-rate per IP; catches scanners that
  # never reach AUTH). Threshold sized well above any legitimate MUA usage.
  smtp_probe_threshold: 100               # per-IP connects before block (default: 100; explicit 0 disables)
  smtp_probe_window_min: 5                # sliding window in minutes (default: 5)
  smtp_probe_suppress_min: 60             # cooldown between repeat findings (default: 60)
  smtp_probe_max_tracked: 20000           # soft cap on tracked entries; oldest evicted (default: 20000)

  # Mail brute-force tracker (IMAP/POP3/ManageSieve via mail_logs source)
  mail_bruteforce_threshold: 5            # per-IP failed auths before block (default: 5)
  mail_bruteforce_window_min: 10          # sliding window in minutes (default: 10)
  mail_bruteforce_suppress_min: 60        # cooldown between repeat findings (default: 60)
  mail_bruteforce_subnet_threshold: 8     # unique IPs per /24 before subnet block (default: 8)
  mail_account_spray_threshold: 12        # unique IPs targeting one mailbox before visibility finding (default: 12)
  mail_bruteforce_max_tracked: 20000      # soft cap on tracked entries; oldest evicted (default: 20000)
  # Slow-brute pair, as above. A recent successful login from the source or
  # failures confined to mailboxes it holds established good standing for
  # keep the source on the advisory path instead of blocking. Also fires on
  # breadth alone: 10+ distinct walked mailboxes trigger regardless of count.
  mail_bruteforce_slow_threshold: 40      # failures before block (default: 40; 3-1024; explicit 0 disables)
  mail_bruteforce_slow_window_min: 360    # slow sliding window in minutes (default: 360; 1-10080)
  mail_brute_account_key: "builtin:dovecot-user" # builtin:dovecot-user | builtin:postfix-sasl | regex:<capture>
  # Mail login spray rules (SMTP and IMAP/POP3): long memory of failed and
  # clean logins per IP. A clean login earns the IP trust on that mailbox.
  # Enforce mode blocks last far longer than other blocks, so a busy host
  # holds many at once: size firewall.deny_temp_ip_limit before enabling it.
  mail_spray_mode: observe                # off | observe (findings only) | enforce (block) (default: observe)
  mail_spray_window_min: 10080            # memory of failed and clean logins in minutes (default: 10080; 60-10080)
  mail_spray_breadth_mailboxes: 2         # finding once one IP (an IPv6 /64) has failed on this many untrusted mailboxes, the current one included (default: 2; 2-10; explicit 0 disables)
  mail_spray_missing_networks: 3          # finding once this many networks have tried one non-existent mailbox, the current one included (default: 3; 1-50; explicit 0 disables)
  mail_spray_block_expiry: "720h"         # how long an enforce mode spray block lasts; other blocks keep block_expiry (default: 720h; 24h-2160h)
  modsec_escalation_hits: 3          # denies from one IP before ModSecurity escalation (default: 3)
  modsec_escalation_window_min: 10   # ModSecurity escalation window in minutes (default: 10)
  modsec_low_confidence_escalation_hits: 30 # low-confidence-only backstop (default: 30)

# --- Web server overrides ---
# Leave these empty to use auto-detected paths for the running platform.
web_server:
  type: ""                              # apache | nginx | litespeed; empty = detect
  config_dir: ""                        # optional Apache/Nginx config root
  access_logs: []                       # candidate access logs, replacing detected paths
  error_logs: []                        # candidate error logs, replacing detected paths
  modsec_audit_logs: []                 # candidate ModSecurity audit logs
  # Override the per-vhost access-log glob patterns. Empty uses the
  # auto-detected default for the panel (cPanel, Plesk, DirectAdmin,
  # bare Apache, or bare Nginx).
  domlog_globs: []
  # IPs or CIDRs trusted by periodic access-log scans and the challenge server
  # to append or replace X-Forwarded-For with their observed peer. Other
  # HTTP readers protect these addresses without reading that header.
  # Automatic single-address responses never challenge or block a listed
  # address; queued blocks are discarded even while blocking is deferred.
  # This protection does not cover subnet blocks or manual operator blocks.
  # challenge.trusted_proxies entries count too. Addresses of this host
  # are never trusted. CSM skips an empty or unreadable entry and never
  # trusts one covering every address; csm validate warns about both,
  # and the daemon raises a High finding until a config without them runs.
  # Restart after correcting the lists; a reload keeps the running lists.
  # Legacy null or non-string scalar entries do not prevent startup,
  # reload or plain csm doctor; usable entries beside them still work.
  # Leave both lists empty to use the direct peer.
  trusted_proxies: []

# --- Infrastructure ---
infra_ips: []                           # management IPs/CIDRs/hostnames - never blocked

# --- Mail Logs ---
# Packaged releases include journald support. Custom builds need
# `make JOURNAL=1 build-yara` before `source: journal` can be selected.
mail_logs:
  source: auto                          # auto | file | journal
  file: ""                              # optional path override for file source
  units: ["postfix", "postfix@-", "dovecot"]  # journal units for source=journal or auto fallback; postfix@- is the Debian/Ubuntu instance

# --- Stalwart ---
# Read only on hosts where the stalwart service is the mail server.
stalwart:
  log_dir: /var/log/stalwart            # where Stalwart writes stalwart.log.YYYY-MM-DD
  log_prefix: stalwart.log              # file name before the date suffix
  ports: []                             # non-standard listeners as PORT:PROTOCOL (smtp, imap, pop3, sieve, http), e.g. ["10465:smtp"]
  api_url: http://127.0.0.1:8080        # Stalwart admin API for csm stalwart setup and csm doctor; https unless on this host; file-only. Supply the admin login when running setup
  api_key_file: /etc/csm/stalwart-api.key  # API key CSM uses to act on Stalwart; a regular file owned by root with no group or other access (0600); file-only

# --- State ---
state_path: "/var/lib/csm/state"        # bbolt DB and state files

# --- Suppressions ---
suppressions:
  upcp_window_start: "00:30"            # cPanel nightly update window start
  upcp_window_end: "02:00"              # cPanel nightly update window end
  known_api_tokens: []                  # API tokens to ignore in auth logs (e.g. ["phclient"])
  ignore_paths: []                      # glob patterns to skip in file scans; keep empty, an exempt directory is where an attacker hides
  suppress_webmail_alerts: true         # don't alert on webmail logins
  suppress_cpanel_login_alerts: false   # don't alert on cPanel direct logins
  trusted_countries: ["RO"]             # requires a loaded GeoLite2-City database; download credentials alone do not supply country data

# --- Auto-Response ---
auto_response:
  enabled: false
  kill_processes: false                 # kill malicious processes
  quarantine_files: false               # move malware to quarantine
  max_file_actions_per_hour: 50         # shared quarantine and file-cleaning attempt budget
  max_file_actions_per_account_per_hour: 10  # per-account share of the same rolling hour
  max_file_action_failures_per_hour: 3   # pause automatic file response after repeated failures
  block_ips: false                      # block attacker IPs via firewall
  block_expiry: "24h"                   # positive temporary block duration; omit for the 24h default; admission caps new temporary lifetimes; retries keep their first expiry
  http_asn_crawl_tempban: "24h"         # Critical ASN-crawl subnet ban duration
  # Shared scan-driven block budget; 0/omitted selects 50.
  # Also sets the admission ledger ceiling: 0/omitted selects 2000, capped at 20000.
  # A value sets both, so leave it out unless the ceiling should be lower too.
  # Outside dry run, with the firewall on, the ledger caps the IP blocks it applies.
  # See Automatic response admission in the API reference.
  # max_blocks_per_hour: 0
  enforce_permissions: false            # auto-chmod 644 world/group-writable PHP files
  fix_wp_cron: false                    # on perf_wp_cron findings, auto-disable WP-Cron and install a per-user system cron
  http_scanner_action: "challenge"      # response for http_scanner_profile: "challenge" (default) routes to the PoW page, "block" bans the IP
  block_cpanel_logins: false            # block IPs on cPanel/webmail/FTP/API thresholded brute findings (multi-IP login, webmail/API brute, FTP brute). Single direct cPanel form logins stay audit-only regardless of this flag.
  netblock: false                       # auto-block IPv4 /24 or IPv6 /64 subnets
  netblock_threshold: 3                 # IPs from same IPv4 /24 or IPv6 /64 before subnet block; minimum 2, omit for the default
  netblock_window: "168h"              # blocked IPs count this far back, expired blocks included; omit for the 168h default
  permblock: false                      # promote temp blocks to permanent
  permblock_count: 4                    # temp blocks before promotion; minimum 2, omit for the default
  permblock_interval: "24h"             # positive counting window; omit for the 24h default
  clean_database: false                 # auto-drop confirmed malicious DB objects after backup
  clean_htaccess: false                 # auto-clean .htaccess directives flagged by hardened detectors (backups under /opt/csm/quarantine/pre_clean/)
  virtual_patch_exposed_files: "off"    # off, manual CLI apply, or dry-run-gated auto apply except sample SQL
  disable_enforce_af_alg: false         # suspend periodic AF_ALG hardening re-assertion
  copy_fail_kill_process: false         # kill processes caught opening AF_ALG sockets via the live listener
  mail_auth_recovery:
    restart_enabled: false              # opt-in restart after sustained cPanel mail auth backend outage
    down_grace: "10m"                   # continuously-down duration before restart
    max_restarts_per_hour: 3            # hourly restart-attempt cap
    restart_command: "/usr/local/cpanel/scripts/restartsrv_dovecot"
  dry_run: true                         # safe default; previews IP blocks and web-exposed-file virtual patches
  verdict_callback:
    enabled: false                      # call panel before each auto-block
    url: ""                             # POST target for verdict requests
    hmac_secret: ""                     # signing secret, or use hmac_secret_env
    hmac_secret_env: ""                 # env var read at call time
    allow_unsigned: false               # true only for staged unsigned rollouts
    require_response_signature: true    # reject unsigned callback replies
    timeout_sec: 2                      # callback request timeout

  # PHP-relay auto-freeze. Off by default; only kicks in on cPanel hosts
  # where email_protection.php_relay.enabled is true. dry_run defaults to
  # true even when freeze is true, so an operator who enables freeze
  # without thinking gets a dry-run rather than a live exim -Mf storm.
  # Override at runtime with `csm phprelay dry-run on|off|reset`.
  php_relay:
    freeze: false                       # opt in to wire the exim -Mf hook into the alert pipeline
    dry_run: true                       # safe default; flip with `csm phprelay dry-run off [--persist]`
    max_actions_per_minute: 60          # rolling 60s cap on exim -Mf invocations

# --- Detection ---
detection:
  # Known-vulnerable WordPress plugin matching is default-on and alert-only.
  # Omit for the default, or set false to disable it. Allow entries suppress
  # only the exact reviewed slug and installed version.
  # vulnerable_plugin_scanning: true
  vulnerable_plugin_allow: []           # entries: <slug>@<version>, case-insensitive
  # db_object_scanning is tri-state: omit for the default (on),
  # `false` to explicitly disable. When off, the MySQL persistence
  # scanner emits no findings; the manual `csm db-clean --drop-object`
  # CLI keeps working for operator-driven cleanup.
  # db_object_scanning: true
  db_object_allowlist: []               # entries: <account>:<schema>:<type>:<name> -- suppresses db_unexpected_* warnings only
  admin_overlap_min_accounts: 2         # raise only if routine shared-admin accounts are expected on this host
  admin_overlap_trusted_emails: []       # exact reviewed admin emails that may manage multiple cPanel accounts
  admin_overlap_trusted_domains: []      # exact reviewed email domains for developer or reseller admin accounts
  # rescan_on_signature_update: true    # tri-state; omit for default-on, false pauses queued rescans
  af_alg_backend: "auto"                # auto | bpf | auditd | none
  connection_tracker_backend: "auto"    # auto | bpf | legacy | none
  connection_poll_interval: 30s         # legacy connection tracker interval
  exec_monitor_backend: "auto"          # auto | bpf | legacy | none
  exec_monitor_poll_interval: 30m       # legacy process monitor interval
  sensitive_files_backend: "auto"       # auto | bpf | legacy | none
  sensitive_files_poll_interval: 5m     # sensitive-file poll/watchset refresh interval
  placement_backend: "auto"             # auto | bpf | legacy | none; files renamed or hard-linked into a site; restart required
  direct_smtp_egress:
    enabled: false                      # detect non-MTA local processes opening outbound SMTP
    backend: "auto"                     # auto | bpf | legacy | none
    dry_run: true                       # safe default for detector-scoped action
    ports: [25, 465, 587]               # destination ports to inspect
  bad_asn_outbound:
    enabled: false                      # off by default; third leg of the host_takeover chain. Needs the GeoLite2-ASN database and operator-supplied ASN lists
    blocked_asns: []                    # ASNs always treated as bad (e.g. known bulletproof hosters)
    allowed_asns: []                    # non-empty switches to allowlist mode: any destination ASN outside this set is treated as bad

# --- BPF Enforcement ---
bpf_enforcement:
  enabled: false                        # master switch for in-kernel denial
  dry_run: true                         # log intended denials, allow the connect
  direct_smtp_egress: false             # gate enforcement on direct SMTP egress matches
  verdict_callback: false               # userspace advisory callback after the BPF decision

# --- Challenge Pages ---
challenge:
  enabled: false                        # enable PoW challenge pages instead of hard block
  listen_addr: 127.0.0.1                # bind address; 0.0.0.0 for direct redirects, unix:/run/csm/challenge.sock for a proxy on this host
  listen_port: 8439                     # TCP port; ignored when listen_addr uses unix:
  listen_group: ""                      # group allowed to open the unix: socket; the group your proxy runs as
  tls_cert: ""                          # optional HTTPS cert for direct/public challenge listener
  tls_key: ""                           # optional HTTPS key for direct/public challenge listener
  public_url: ""                        # required by webserver-integration, e.g. https://host:8439/challenge
  secret: ""                            # HMAC secret for tokens (auto-generated if empty)
  difficulty: 2                         # SHA-256 proof-of-work difficulty 0-5 (default: 2)
  trusted_proxies: []                   # proxies in front of this listener; HTTP checks trust them too
  port_gate:
    enabled: false                      # nftables gate for non-loopback challenge listener
  captcha_fallback:                     # widget for JS-disabled visitors (default off)
    provider: ""                        # "turnstile" | "hcaptcha" | "" (off)
    site_key: ""                        # public key embedded in the widget
    secret_key: ""                      # verified server-side
    timeout: 10s
  verified_session:                     # signed-cookie bypass for authenticated operators
    enabled: false
    cookie_name: csm_admin_session
    ttl: 4h
    admin_secret: ""                    # POST'd to /challenge/admin-token to mint cookie
  verified_crawlers:                    # reverse-DNS forward-confirm for search crawlers
    enabled: false
    providers: []                       # names: googlebot | bingbot
    cache_ttl: 15m

# --- PHP Shield ---
php_shield:
  # On CloudLinux, PHP runs inside a CageFS cage that cannot see the Shield
  # event socket. Installing or upgrading the Shield registers its root-owned,
  # non-writable directory in /etc/cagefs/cagefs.mp. Existing cages must be
  # remounted during a maintenance window: cagefsctl --remount-all recreates
  # every LVE on the host and disrupts running cages. A conflicting mount entry
  # is left unchanged.
  # csm doctor checks the event mount in running tenant cages. CSM's own
  # worker services are not cages and are left out of that check.
  # Rewritten command-parameter probes are quieted only when the Shield's
  # source fingerprint matches CSM's verified CMS content cache. Install the
  # updated Shield with csm install --php-shield-only; older events, unverified or
  # modified scripts, and incomplete request paths still alert. All recognized
  # observations remain in the root-only event archive, including quiet probes.
  # This does not verify code included by the entry script; content scanning
  # and runtime protection remain necessary.
  # A failed eval() is High, except when the reported eval() call sits in a
  # root-owned file that no account can change, such as the wp-cli bundled with
  # WP Toolkit running "wp eval". That file and every directory above it must be
  # root-owned, not symlinks, and not group or world writable when the event
  # arrives; such a failure is Warning. Tenants can already disable the
  # Shield in their own PHP processes. Every event field can be forged, so
  # this verifies only the reported path's ownership and permissions, not
  # the sender or evaluated code. A web request reaching such an eval() is
  # still reported at Warning. An ownership check that fails or times out
  # keeps the event at High, and a hung check does not delay later events.
  enabled: false                        # receive PHP Shield events and emit alerts

# --- Reputation ---
reputation:
  abuseipdb_key: ""                     # AbuseIPDB API key for IP reputation lookups
  whitelist: []                         # IPs to never flag as malicious
  # Async PTR + forward-A verification for IPs that claim search-engine
  # bot UAs (Googlebot, Bingbot, Applebot). When an IP claims a bot UA
  # but reverse DNS does not confirm it, the request counts toward
  # http_ua_spoof. Transient DNS lookup failures fail open and are
  # retried later. Set false only if your resolver is unreliable. See
  # docs/src/auto-response.md for the always-block behavior.
  bot_verify_enabled: true              # default: true
  verified_bots: []                     # optional custom crawler identities
  # verified_bots:
  #   - name: "seranking"               # rDNS-verified crawler
  #     ua_substrings: ["serankingbacklinksbot"]
  #     rdns_suffixes: ["seranking.com"]
  #   - name: "perplexitybot"           # AI agent verified by published IP ranges
  #     ua_substrings: ["perplexitybot"]
  #     ip_ranges: ["18.97.9.96/29", "18.97.1.228/30"]
  bot_ranges:                           # built-in AI-crawler range refresh
    auto_update: true                   # default: true; restart required to change
    update_interval: "24h"              # default: 24h, minimum: 1h
  rspamd:
    enabled: false                      # include rspamd rolling history in IP reputation
    url: "http://127.0.0.1:11334"       # rspamd controller URL
    token: ""                           # controller password, or use token_env; not needed for a loopback url
    token_env: ""                       # env var read at query time
  upstream:
    enabled: false                      # include panel-side threat-intel cache scores
    url: ""                             # HTTPS base URL; HTTP only allowed for loopback
    token: ""                           # bearer token, or use token_env
    token_env: ""                       # env var read at query time
    cache_ttl_min: 15                   # local cache TTL for upstream scores
    timeout_sec: 5                      # upstream request timeout
  report:
    enabled: false                      # opt-in abuse report delivery; restart required
    classes: []                         # bruteforce | php_relay | credential_stuffing | bad_asn_egress | spam
    spool_path: ""                      # default: <state_path>/abuse_reports.db
    spool_max: 10000                    # max queued reports, shared by all targets
    targets:
      - name: ""                        # stable target name
        url: ""                         # HTTPS collector URL; HTTP only allowed for loopback
        transport: "hmac"               # hmac | ed25519 (the central abuse database accepts ed25519 only)
        node_id: ""                     # sender node ID
        key_id: ""                      # receiver key ID
        key_env: ""                     # HMAC secret or Ed25519 private key env var
        token_env: ""                   # optional bearer token env var for HMAC targets
  central:
    enabled: false                      # opt-in central scored-set consume; restart required
    set_url: ""                         # HTTPS scored-set endpoint; HTTP only for loopback
    pubkey_env: ""                      # env var with Ed25519 public key hex
    auth_target: ""                     # report target whose node key signs downloads
    refresh_interval: 6h                # pull interval; default 6h
    action: "challenge"                 # off | challenge | block_if_local_corroborated
    block_threshold: 60                 # score needed before local corroboration can block

# --- Signatures ---
signatures:
  rules_dir: "/opt/csm/rules"           # YAML signature rules directory
  update_url: ""                        # remote URL to fetch rule updates
  auto_update: false                    # auto-download rules on schedule
  update_interval: ""                   # how often to check (e.g. "24h")
  signing_key: ""                       # required for any remote rule update path; 64-char hex Ed25519 public key
  allow_rule_count_decrease: false      # permit a deliberate drop below half the installed YAML rule count
  yara_forge:
    enabled: false                      # auto-fetch YARA Forge community rules
    tier: "core"                        # "core", "extended", "full" (default: "core")
    update_interval: "168h"             # how often to check for updates (default: weekly)
    download_url: ""                    # signed ZIP URL/template; supports {tier} and {version}
  disabled_rules: []                    # rule names to switch off, in Forge and in the shipped rules
  # yara_worker_enabled: true           # omit for the default: YARA-X runs in locked-down worker units; `false` runs it inside the root daemon

# signatures.signing_key is mandatory whenever either signatures.update_url
# is set or signatures.yara_forge.enabled is true. It must be the hex
# Ed25519 public key used to verify detached .sig files for rule bundles.
# Remote update URLs must use HTTPS and must not point at localhost,
# loopback, link-local, unspecified, or RFC1918 / ULA private addresses.
#
# YARA Forge upstream GitHub releases do not publish CSM detached signatures.
# To enable automatic Forge updates, mirror the ZIPs, sign each ZIP, publish
# the signature at the ZIP URL plus .sig, and set yara_forge.download_url to
# that signed mirror. Otherwise leave update_url empty and yara_forge.enabled
# false.

# --- Web UI ---
webui:
  enabled: true
  listen: "0.0.0.0:9443"               # address:port for HTTPS server
  auth_token: ""                        # API/login credential (auto-generated on install)
  auth_token_issued_at: ""              # optional UTC RFC3339 issue time; recorded on install
  auth_token_expires_at: ""             # optional UTC RFC3339 expiry; empty never expires
  session_lifetime: "24h"               # browser absolute expiry; maximum 720h; restart required
  session_idle_timeout: "30m"           # browser idle expiry; restart required
  tokens: []                            # scoped tokens: name/token/scope, optional issued_at, expires_at and allowed_ips (scope: admin, read or unban)
  metrics_token: ""                     # optional Bearer token for /metrics only
  metrics_token_issued_at: ""           # optional UTC RFC3339 issue time; rotate with metrics_token, then SIGHUP
  metrics_token_expires_at: ""          # optional UTC RFC3339 expiry; empty never expires
  tls_cert: ""                          # path to TLS certificate PEM file
  tls_key: ""                           # path to TLS private key PEM file
  ui_dir: ""                            # path to UI files on disk (default: /opt/csm/ui)
  allowed_origins: []                   # extra browser origins ("https://host[:port]") allowed to call the API; loopback is always allowed

# --- MCP (read-only endpoint for support agents) ---
mcp:
  enabled: false                        # serves /mcp on the Web UI listener; needs webui.enabled: true; restart required
  tokens: []                            # name/token, optional issued_at, expires_at and allowed_ips; all read-only, never valid on the Web UI

# --- Email AV ---
email_av:
  enabled: false
  clamd_socket: "/var/run/clamd.scan/clamd.sock"  # path to ClamAV daemon socket
  scan_timeout: "30s"                   # per-attachment scan timeout
  max_attachment_size: 26214400         # max single attachment size in bytes (25MB)
  max_archive_depth: 1                  # archive levels unpacked (1: attached archives only); a deeper archive counts as not fully scanned
  max_archive_files: 50                 # max files extracted per archive; also caps encrypted member names reported per message
  max_extraction_size: 104857600        # max total extraction size in bytes (100MB)
  quarantine_infected: false            # true moves infected messages to quarantine; false delivers them with a Critical email_malware finding
  scan_concurrency: 4                   # parallel scan workers
  fail_mode: "open"                     # behavior when a scan cannot complete: "open" (default) delivers; "tempfail" defers so Exim retries
  # Permission events have a five-second scan hold budget. When it expires,
  # the open is allowed ("open") or denied for retry ("tempfail"); an admitted
  # scan continues. A full scan queue applies the same policy immediately.
  # When messages repeatedly cannot start scanning within the hold budget, or
  # find the scan queue full, CSM raises email_av_hold_bypass and stops
  # queueing new scans for five minutes. A hold that runs out while the
  # message is already being scanned does not count. During this cooldown new
  # opens are allowed unscanned ("open") or deferred ("tempfail"). Scanning
  # resumes automatically afterwards; bypassed messages are not scanned later.
  # One sender's messages normally use at most half the scan_concurrency
  # slots (at least one), leaving slots for others when concurrency exceeds
  # one. The sender is the authenticated user, else the sending address (an
  # IPv6 /64), else the local account. Incomplete sender fields are ignored
  # when choosing the sender. Mail relayed through a filtering gateway,
  # unauthenticated mail submitted over SMTP from localhost, and bounces Exim
  # creates each count as one sender. Further messages wait without taking a slot;
  # when their hold budget runs out they follow fail_mode and are still
  # scanned afterwards. When too many messages from one sender are waiting,
  # further ones are scanned at once without waiting their turn. This overflow
  # can use every scan slot; email_av_source_backlog names the sender, at most
  # once a minute.
  # An archive that lists more than ten times max_archive_files entries, or
  # whose decompressed data passes max_extraction_size, counts as not fully
  # scanned and follows fail_mode. Nested archives share that budget. So do
  # corrupt gzip trailers, data after a tar end marker, legacy sparse tar
  # members and zip directory entries that carry data, at any nesting level.
  # Empty members and directories remain valid.
  # Late malware results can quarantine only the original, unlocked message.
  # Messages in active delivery, replaced spool files, and messages with a
  # pending delivery journal are left untouched with email_av_quarantine_error.
  # email_av_late_verdict means an allowed open later received a scan decision
  # to stop delivery; it does not mean a previously deferred message escaped.
  # Password-protected archive attachments are outside fail_mode. Their members
  # cannot be read without the password, so no retry ever makes them scannable
  # and "tempfail" would defer the message until it bounced. CSM delivers them
  # and reports email_av_encrypted_archive naming the archive and the member,
  # at most once an hour. A full alert queue does not consume this allowance.
  # Encrypted member names are capped at max_archive_files per message; the
  # warning counts any additional encrypted members without deferring delivery.
  # Both encryption schemes are covered:
  # legacy ZipCrypto and WinZip AES. Blocking such mail outright is an Exim
  # policy decision and is not something CSM does for you.
  # Attachments are unpacked in locked-down worker processes, one per
  # scan_concurrency slot, started on first use. When a worker cannot finish,
  # the message counts as not fully scanned:
  #   worker deadline reached     "parse worker timed out"
  #   worker memory cap reached   "parse worker ran out of memory"
  #   worker crashed or exited    "parse worker crashed"
  #   reply failed validation     "parse worker reply rejected"
  #   worker unit cannot start    "parse isolation unavailable"
  #   unit keeps failing to start "parse worker failing repeatedly"
  #   message failed twice before "parse worker failed on this message before"
  # The first four and the last are caused by the message. Outside observe
  # mode they request deferral whatever fail_mode says, so Exim retries it.
  # Deferral requires a permission event that is still holding delivery;
  # notification-only mode cannot block it. The two unit failures, and a unit
  # still starting when the scan deadline passes, follow fail_mode.
  # The first four raise email_av_parse_worker_failed, naming the message and
  # its sender, at most once a minute; a rejected reply also raises a Critical
  # email_av_parse_worker_rejected. The next two raise a Critical
  # email_av_isolation_unavailable. Each Critical finding is raised at most
  # once an hour. A message whose worker failed twice within a day is not
  # unpacked again on later delivery attempts, including after a spool watcher
  # restart. Failed starts, cancellation and local descriptor exhaustion do
  # not count against messages or senders. When messages from one sender
  # make workers fail three times, each within ten minutes of the one before,
  # a High email_av_parse_worker_source names the sender, at most once every
  # ten minutes. This advisory reports scan failures, not delivery outcomes.
  # It never blocks automatically, and blocking an address does not hide it.
  worker_memory_limit: 536870912        # memory and swap cap of each attachment worker in bytes (512 MiB); minimum 134217728 (128 MiB)
  # Up to scan_concurrency attachment workers run at once, and each may use up
  # to worker_memory_limit. A worker that reaches it is killed and the message
  # is deferred as "parse worker ran out of memory"; raise the limit if
  # ordinary mail reaches it. The phishing-kit zip worker keeps the default
  # cap. Changing it takes effect after a daemon restart.
  keep_crash_samples: false             # keep copies of messages and tenant zips that crash an attachment worker
  # Crash samples are for diagnosing worker crashes. They hold customer mail
  # and tenant files: they are readable by root only, stored in parse-crashes
  # under the state directory, capped at 20 samples and 500 MiB, and deleted
  # after 7 days, also after this setting is turned off. A link in place of
  # the parse-crashes directory is refused. Failed or incomplete copies are
  # discarded without changing the scan verdict or findings.

# --- Email Protection ---
email_protection:
  password_check_interval_min: 1440     # how often to audit email passwords (default: 1440)
  high_volume_senders: []               # accounts expected to send high volume (skip rate alerts)
  rate_warn_threshold: 50               # emails per window before warning (default: 50)
  rate_crit_threshold: 100              # emails per window before critical (default: 100)
  rate_window_min: 10                   # rate check window in minutes (default: 10)
  known_forwarders: []                  # expected forwarders and Sieve :copy rules

  # PHP-relay detector (cPanel only; gated by platform.IsCPanel at startup).
  # Off by default. When enabled, the daemon spawns the inotify spool
  # watcher, runs a startup spool walk, and starts the Path 2b retro scan
  # on /var/log/exim_mainlog. See docs/src/detection-realtime.md#php-relay
  # for what each path actually triggers on.
  php_relay:
    enabled: false                      # opt in to start the watcher
    rate_window_min: 5                  # Path 1 rolling window
    header_score_volume_min: 5          # Path 1: don't score until script has emitted N msgs
    absolute_volume_per_hour: 30        # Path 2 threshold per script
    account_volume_per_hour: 0          # Path 2b operator override; 0 = auto-derive from cpanel.config maxemailsperhour
    fanout_distinct_scripts: 3          # Path 4 threshold
    fanout_distinct_recipients: 5       # Path 2 + Path 4 recipient-diversity gate; 0 disables only this gate
    fanout_window_min: 5                # Path 4 window
    ip_recipient_spread: 5              # Path 6 threshold: distinct outside recipients per client IP (1-100); whole number; 0 disables Path 6
    ip_recipient_window_hours: 24       # Path 6 window in hours (1-168); whole number; 0 uses the default (24)
    ip_recipient_mode: observe          # Path 6 response: observe (findings only, default) or enforce (challenge, then block)
    policies_dir: "/opt/csm/policies/php_relay"  # optional mailer_classes.yaml + http_proxy_ranges.yaml (replaces the built-in proxy list); SIGHUP-reloadable
  cloud_relay:
    allow_users: []                     # full mailbox opt-outs for cloud-relay detection
    allow_domains: []                   # domain-wide opt-outs for cloud-relay detection

  # Email forward guard (cPanel only). Opt-in MTA-native enforcement for
  # external forward copies. Enforce mode can hold null-sender backscatter,
  # bad-sender-IP and spam-flagged copies before they relay to an external
  # provider, while the local mailbox copy still delivers. Spam-flagged means
  # the message carries X-Spam-Status: Yes, as antispam adds when it tags.
  # Any X-Spam-Status field with a Yes verdict counts, ignoring letter case.
  # Other text does not count as a spam verdict.
  # Malware and auth-fail signals are accounted in dry-run until Exim content
  # scanning is wired. CSM is not in the live mail path; an installed Exim rule
  # can keep holding matching copies even if the daemon is down. Held copies can
  # be released or deleted from the Email page.
  forward_guard:
    enabled: false                      # master switch (default off)
    dry_run: true                       # account/log only, do not actually hold (default true)
    quarantine_retention_days: 14       # held copies older than this are deleted; 0 keeps them
    hold_signals:                       # signal toggles, each default true
      bounce_backscatter: true          # null-sender bounce backscatter (enforceable)
      spam_flagged: true                # message carries X-Spam-Status: Yes (enforceable)
      malware: true                     # message carries malware (dry-run/accounting only)
      bad_sender_ip: true               # originating IP has bad reputation (enforceable)
      auth_fail: true                   # sender failed SPF/DKIM/DMARC auth (dry-run/accounting only)

  # Inbound spam filtering through rspamd on cPanel and Postfix hosts. Opt-in.
  # The operator installs rspamd and adds the scan socket. On cPanel CSM writes
  # the Exim hooks and its own rspamd settings, and each account's Spam Filters
  # switch decides whether its mail is filtered. On Postfix CSM writes only
  # rspamd files, every local domain is filtered and `exempt` lists the
  # exceptions. See docs/src/antispam.md.
  antispam:
    enabled: false                      # master switch (default off)
    dry_run: true                       # scan and log only; no reject, tag or split (default true)
    rspamd_socket: /run/rspamd/csm-normal.sock  # rspamd scan socket, not the reputation.rspamd controller
    reject_score: 15                    # reject at or above this rspamd score (default 15)
    tag_score: 6                        # add X-Spam-Status: Yes at or above this score (default 6)
    feeds:
      enabled: []                       # optional feeds, e.g. [mailspike, psbl]; see docs/src/antispam.md
      spamhaus_dqs_key_file: ""         # root-owned file holding the Spamhaus DQS key; needed only for spamhaus_dqs
    learning:
      enabled: false                    # learn from mail users move to Spam (cPanel) or Junk (Postfix) and keep one message per cluster; needs antispam; see docs/src/antispam.md
      redis_socket: ""                  # Unix socket of a Redis used only by learning, e.g. /run/redis-csm/redis.sock; Postfix needs one, cPanel uses rspamd's Redis when empty
      reporter: domain                  # Postfix only: one learning vote per mail domain (domain) or per mailbox (mailbox)
    exempt: []                          # Postfix only: recipient domains and mailboxes never filtered or learned from, e.g. [example.com, postmaster@example.net]

# --- Firewall ---
firewall:
  enabled: false

  # Open ports (IPv4). SSH (22) is intentionally absent; uncomment in
  # the YAML lists if sshd listens on 22. TCP 853 is DNS-over-TLS;
  # UDP 853 is DNS-over-QUIC.
  # 6277/24441 are DCC/Pyzor network checks used by SpamAssassin.
  tcp_in: [20,21,25,26,53,80,110,143,443,465,587,853,993,995,2077,2078,2079,2080,2082,2083,2091,2095,2096]
  tcp_out: [20,21,25,26,37,43,53,80,110,113,443,465,587,853,873,993,995,2082,2083,2086,2087,2089,2195,2325,2703]
  udp_in: [53,443,853]
  udp_out: [53,113,123,443,853,873,6277,24441]

  # IPv6
  ipv6: false
  tcp6_in: []                           # if empty, uses tcp_in
  tcp6_out: []                          # if empty, uses tcp_out
  udp6_in: []                           # if empty, uses udp_in
  udp6_out: []                          # if empty, uses udp_out

  # Restricted ports (infra IPs only)
  restricted_tcp: [2086,2087,2325,9443] # WHM and CSM Web UI ports

  # One TCP port opened to listed sources only. See firewall.tcp_in_allow
  # below.
  tcp_in_allow: []
  # tcp_in_allow:
  #   - port: 3306
  #     sources: [203.0.113.10, 198.51.100.0/28]

  # Outbound ports a service on this host needs. Checked, never added to
  # the policy: validation warns when an effective family policy omits one.
  # Integrations declare theirs in their own conf.d fragment.
  required_tcp_out: []

  # Passive FTP data ports. Left unset, 49152-65534 is opened on cPanel hosts
  # and hosts with pure-ftpd, proftpd or vsftpd installed, and stays closed
  # elsewhere. Explicit values take precedence; set both to 0 to close it.
  # passive_ftp_start: 49152
  # passive_ftp_end: 65534

  # Destination-scoped outbound TCP. tcp_out holds single ports only, so a
  # port range can only be expressed here -- needed when this host is an FTP
  # *client* and must open passive data connections to a known source.
  # dst takes an IP or CIDR. 0.0.0.0/0 means any IPv4 destination; ::/0 means
  # any IPv6 destination. Each rule applies to its own address family, and
  # validation warns: a wide range to anywhere is the outbound path the
  # output chain exists to close. When smtp_block is on, these rules follow
  # its guard and ranges that overlap smtp_ports are rejected.
  # An IPv6 destination emits no rule unless firewall.ipv6 is true.
  # IPv4-mapped prefixes under ::ffff:0:0/96 are treated as IPv4 prefixes.
  # Lockout warnings only credit valid rules in the connection's family;
  # scoped exceptions still warn because hostname resolution is not checked.
  tcp_out_allow: []
  # tcp_out_allow:
  #   - dst: 203.0.113.10/32
  #     port_start: 49152
  #     port_end: 65534

  # Peer-scoped protocols without ports, such as VRRP between the members of
  # a high-availability pair. See firewall.protocol_allow below.
  protocol_allow: []
  # protocol_allow:
  #   - protocol: vrrp
  #     peers: [192.0.2.11, 2001:db8::11]

  # Infra IPs/CIDRs/hostnames for firewall rules
  infra_ips: []

  # Rate limiting. SYN/conn-rate/UDP are dual-stack (IPv6 keyed per /64);
  # conn_limit is IPv4-only.
  conn_rate_limit: 200                  # new connections/min per source (CGNAT-tolerant; IPv6 per /64; 0 = disabled; null = default)
  syn_flood_protection: true            # per-source SYN flood meter (IPv6 per /64)
  conn_limit: 400                       # max concurrent connections per IPv4 source, IPv4 only (0 = disabled)
  drop_ipv4_options: true               # drop inbound IPv4 TCP carrying IP options, ahead of every accept (CVE-2026-98432 trigger)

  # Per-port flood protection: rate-limit new connections per source IP and IP family.
  # Defaults are sized for a busy mail host: 600/300s = 120 new conns/min/IP,
  # which tolerates a Thunderbird/iPhone client opening 5-15 parallel sessions
  # while still capping single-IP flood storms.
  port_flood:
    - port: 25
      proto: tcp
      hits: 600
      seconds: 300
    - port: 465
      proto: tcp
      hits: 600
      seconds: 300
    - port: 587
      proto: tcp
      hits: 600
      seconds: 300

  # DoS-exempt ranges: bypass configured DoS meters and subnet auto-blocks.
  # See firewall.dos_exempt_ranges below for full details.
  dos_exempt_ranges: []                 # CIDRs or single IPs exempt from connection/mail-port meters and subnet auto-blocks; /0 and bare hostnames rejected at load
  dos_exempt_known_mail_providers: true # also exempt Google and Microsoft mail-provider egress ranges; dynamic, cached, updated every 12h (default: true)

  # UDP flood protection (per-source meter; IPv6 per /64)
  udp_flood: true
  udp_flood_rate: 100                   # packets per second
  udp_flood_burst: 500                  # burst allowance

  # Country blocking
  country_block: []                     # ISO country codes to block
  country_db_path: ""                   # country CIDR directory (default: <state_path>/geoip, filled by `csm firewall update-geoip`)

  # Silent drop (no logging)
  drop_nolog: [23,67,68,111,113,135,136,137,138,139,445,500,513,520]

  # IP limits
  deny_ip_limit: 3000                   # max permanent blocked IPs
  deny_temp_ip_limit: 500               # max temporary blocked IPs; 0 = unlimited for existing blocking; admission forecasts then plan for 10000; a negative value reads as 0 with a warning

  # Outbound SMTP restriction
  smtp_block: false                     # block outgoing mail except allowed users
  smtp_allow_users: [cpanel, mailman]   # extra outbound SMTP users, also exempt from direct SMTP egress detection and BPF enforcement; root and mailnull are always allowed
  smtp_ports: [25,465,587]

  # Local port locks: "PORT:user,user"; only these users and root may open
  # new connections to the port on this host
  local_port_locks: []

  # Dynamic DNS
  dyndns_hosts: []                      # hostnames to resolve and whitelist periodically

  # Logging
  log_dropped: true                     # log dropped packets
  log_rate: 5                           # log entries per minute

# --- GeoIP ---
geoip:
  account_id: ""                        # MaxMind account ID
  license_key: ""                       # MaxMind license key
  editions:                             # MaxMind database editions
    - GeoLite2-City
    - GeoLite2-ASN
  auto_update: true                     # auto-update GeoIP databases (default: true when credentials set)
  update_interval: "24h"                # update check interval

# --- ModSecurity ---
modsec_error_log: ""                    # path to Apache/LiteSpeed error log for ModSec parsing
modsec:
  rules_file: ""                        # path to modsec2.user.conf
  overrides_file: ""                    # path to csm-overrides.conf
  reload_command: ""                    # command to reload web server (e.g. "/usr/sbin/apachectl graceful"); also activates CSM rule updates

# --- Performance ---
performance:
  enabled: true
  load_high_multiplier: 1.0             # load average / CPU cores multiplier for warning (default: 1.0)
  load_critical_multiplier: 2.0         # load average / CPU cores multiplier for critical (default: 2.0)
  php_process_warn_per_user: 20         # per-user PHP process count warning (default: 20)
  php_process_critical_total_multiplier: 5  # total PHP processes / CPU cores for critical (default: 5)
  error_log_warn_size_mb: 50            # error log size warning threshold (default: 50)
  error_log_critical_size_mb: 1024      # error log size that raises the finding to High (default: 1024)
  mysql_join_buffer_max_mb: 64          # MySQL join_buffer_size warning threshold (default: 64)
  mysql_wait_timeout_max: 3600          # MySQL wait_timeout warning threshold (default: 3600)
  mysql_max_connections_per_user: 10    # per-user MySQL connections warning (default: 10)
  redis_bgsave_min_interval: 900        # minimum seconds between Redis BGSAVE (default: 900)
  redis_large_dataset_gb: 4             # Redis dataset size warning threshold in GB (default: 4)
  wp_memory_limit_max_mb: 512           # WordPress memory_limit warning threshold (default: 512)
  wp_transient_warn_mb: 1               # WordPress transient data warning in MB (default: 1)
  wp_transient_critical_mb: 10          # WordPress transient data critical in MB (default: 10)
  wp_cron_fix:                          # tuning for the WP-Cron remediation (manual fix from the Web UI or auto_response.fix_wp_cron)
    interval_minutes: 15                # system cron frequency; default 15, clamped to [1, 60]
    php_bin: ""                         # override interpreter; empty = per-vhost PHP (inherit: .htaccess handler), then auto-detect

# --- Cloudflare ---
cloudflare:
  enabled: false                        # auto-whitelist Cloudflare IP ranges
  refresh_hours: 6                      # how often to refresh Cloudflare IPs (default: 6)

# --- Threat Intel ---
c2_blocklist: []                        # known C2 server IPs; connections with them are blocked
backdoor_ports: [4444,5555,55553,55555,31337]  # ports indicating backdoor activity

# --- Update check ---
updates:
  check_enabled: true                   # notify only; CSM never downloads or applies updates
  interval: "24h"                       # release check interval
  github_api_url: ""                    # optional release API mirror or test endpoint
  package_name: "csm"                   # apt/dnf package name for package-manager fallback

# --- Incidents ---
incidents:
  auto_close:
    enabled: true                       # auto-close idle open/contained incidents
    dry_run: false                      # log decisions without writing status changes
    by_kind:
      mailbox_takeover: 24h
      mailbox_bruteforce: 24h
      credential_spray: 24h
      web_attack: 24h
      web_account_compromise: 168h
  spray_suppression:
    enabled: false                      # collapse one-source credential spray into one incident
    dry_run: true
    distinct_mailboxes: 10
    severity_escalate_at: 50
    per_check: [email_auth_failure_realtime, pam_bruteforce, credential_stuffing]
    max_tracked_ips: 10000
    block_at_severity: ""              # "" | high | critical
  auto_block:
    enabled: false                      # block source IPs from incident correlations
    block_at_severity: ""              # "" | high | critical
    kinds: []                           # empty means all non-spray kinds with remote_ip

# --- Disabled checks (skip whole categories per host) ---
# Listed finding names disable the scheduled check runner(s) that emit them,
# including sibling findings from the same runner. Realtime findings are not
# affected. Use for whole categories that don't apply to a host (e.g. WAF/web
# checks on DNS-only cPanel servers, where httpd is installed but no virtual
# hosts serve traffic).
# For email-only suppression, use `alerts.email.disabled_checks` instead.
disabled_checks: []                     # e.g. [waf_status, waf_rules, waf_detection_only]

# --- Retention (bbolt growth control) ---
retention:
  enabled: false                        # opt-in; when true, a daily sweep prunes old entries
  findings_days: 90                     # keep active findings this long (0 disables the findings sweep)
  history_days: 30                      # keep findings-history entries and proven firewall
                                        # outcomes this long; it also sets how far back a
                                        # durable firewall action can be undone
  reputation_days: 180                  # keep IP reputation/attack entries this long
  sweep_interval: "24h"                 # how often the retention goroutine runs
  compact_min_size_mb: 128              # startup compaction floor; 0 disables auto-compaction
  compact_fill_ratio: 0.5               # compact when used_bytes / file_size drops below this

# --- Debug / diagnostics ---
debug:
  pprof_listen: ""                      # e.g. "127.0.0.1:6060"; MUST be loopback. Empty disables.
                                        # Reach it over SSH: go tool pprof http://127.0.0.1:6060/debug/pprof/heap
                                        # While listening, samples mutex and block contention; adds CPU overhead even without a profile request.

# --- Sentry (error reporting) ---
sentry:
  enabled: false                        # ship panics and selected errors to a Sentry server
  dsn: ""                               # Sentry project DSN
  environment: "production"             # e.g. "production", "staging"
  sample_rate: 1.0                      # 0.0 -> 1.0 (capture all errors)
  debug: false                          # SDK debug logs to stderr

Alert classes

Every finding CSM reports is immediate or summary class. Summary class covers what CSM already handled on its own:

  • an attack from an address CSM blocked, before or in the same batch, including by a subnet block or netblock;
  • an attack a challenge answers, while the address is on the challenge list or was routed there in the same batch. A challenge does not answer FTP or mail authentication failures, critical reputation sightings, or checks configured to require a hard block;
  • applied block and challenge notices, including permanent, subnet and netblock blocks, dry-run block verdicts, and the periodic count of applied automatic responses;
  • mail login advisories that never block, such as spray observe mode;
  • PHP relay freeze dry runs and rate-limit deferrals that need no follow-up.

Summary-class findings are left out of alert email and the generic alert webhook and are counted for the daily summary instead. They never spend the hourly alert budget. The findings page, history, incidents, the audit log, the event stream and the phpanel webhook still receive them. Only findings that would otherwise notify are counted: a suppression rule, or a check that never notifies, keeps its findings out of the summary too. Such a finding can still show that CSM blocked or challenged an address. If the summary fails to take its findings, the alert run reports an error and immediate alerts still go out.

Everything else is immediate. That includes attacks whose source CSM did not block: auto-response off or in dry run, a block that failed, was refused or is still queued, and checks that never block. Compromise evidence, successful logins, unauthenticated WHM script requests, suspicious mail content and findings that summarize several sources stay immediate even when the address is blocked, because blocking one address does not resolve them. A block of an address inside a Cloudflare allow range leaves ports 80 and 443 open, so its notice and the web attacks from that address stay immediate. Failed, paused or skipped automatic actions and auto-block queue and accounting notices are immediate too.

The generic webhook follows the same classes as email. alerts.email.disabled_checks mutes email only: a muted immediate finding still goes to the webhook, and a muted summary finding is still counted in the summary.

There is no setting for this policy, and the same classes decide what csm scan --alert sends. The former suppressions.suppress_blocked_alerts key is ignored; a config that still sets it loads, and csm validate warns so the key can be deleted.

Daily summary

CSM sends one summary of the findings it handled each day at 08:00 server local time. It goes by alert email when email alerts are on, and to the generic alert webhook when that is on, as event daily_summary with the counts under csm.summary (see the API reference). A phpanel webhook does not receive it. alerts.block_digest.channel selects live block alerts only; the daily summary uses every enabled alert channel. It goes out even with nothing to report.

It covers the window since the previous summary:

  • summary-class findings by check and severity, and how many of them come from checks muted by alerts.email.disabled_checks;
  • blocks applied, by kind, including netblocks and promotions to a permanent block;
  • challenges, and mail login advisories that never block, such as spray observe mode;
  • the 10 addresses with the most summary findings, with their country when GeoIP knows it;
  • automatic blocks that hit a known customer, as their immediate alert named them, so a failed alert still reaches you: up to 50 a day, then a count of further blocks;
  • dry-run block verdicts and spray observe-mode verdicts that would have blocked a known customer, listed alongside their finding counts: up to 50 blocked addresses or networks a day, each listed once, then a count of further verdicts;
  • with alerts.block_digest.enabled, watched-country blocks by country and category, once they reach min_block.

CSM tracks up to 1,000 source addresses a day. Past that, a new address takes the place of the one with the fewest findings, so heavy sources stay listed and a count can be overstated, never understated.

Counts are saved at each block_digest.interval (one hour by default), after the final shutdown drain and before delivery. A crash can lose counts since the last save. If the host was down at 08:00, the missed summary goes out at the next start. A failed destination is retried after one minute, then with the wait doubling up to one hour; a restart retries it at once. CSM logs its first failure and again when the wait reaches an hour, and destinations already saved as successful are not retried. While a delivery is pending, new findings accumulate in the next window. When the next summary is due and a destination still has not taken the previous one, CSM drops the previous one for that destination, logs it once and counts it as one lost notification, then sends the new summary to every destination. A summary whose state cannot be saved is still sent; the save warning backs off the same way as a failing destination.

A crash after a destination accepts a summary but before CSM saves that success can repeat it. Clock corrections are checked every minute. Moving back across one or more morning sends does not resend dates already covered, including after a restart. Sending resumes on the first unsent local date. A send recorded more than two days ahead of the clock comes from a clock fault: CSM logs it once, ignores it and sends the next morning.

Only the daemon counts findings. A one-off csm scan --alert prints every finding; its summary-class findings are neither emailed nor counted in the daily summary. The hourly block digest email no longer exists.

Customer login history records the source addresses of authenticated cPanel, webmail, SMTP, IMAP, POP and FTP logins for 30 days. Buffered logs keep their original login time, so reading old logs does not renew expired history. FTP logins retain their authenticated account even when the displayed log record is shortened. FTP command text and failed login names do not count as successful logins. An applied automatic block of a known customer is alerted at once; see the known-customer block alert.

An IPv6 customer is recorded by its /64, so a host that changes privacy addresses stays known. Each account holds a bounded number of addresses per log source; logins without an account name share one budget, and an FTP login counts against the hosting account that owns the FTP account.

History has two fixed halves: logins read from syslog (FTP, IMAP, POP and Postfix SMTP) and logins read from cPanel’s session log and the Exim mainlog. Any local user can write to syslog, so each half trims only itself, and a syslog login never replaces a session log or Exim login for the same address. When a half is full, the accounts holding the most addresses lose their oldest logins first, so a flood of logins evicts only its own accounts. History is saved every minute and after log readers stop during a clean shutdown.

Web UI Exposure

An admin token can change firewall rules, files under account homes and most settings, so treat the Web UI as privileged access. The shipped listen: "0.0.0.0:9443" answers on every interface, which the WHM addon needs. csm validate and csm doctor warn when the UI listens on a non-loopback address, an admin token has no allowed_ips or allows an entire address family, and the CSM firewall policy does not restrict that family. These warnings do not make either command fail.

Enabling the firewall alone does not clear the warning. Keep the UI port out of public tcp_in and tcp6_in entries, or add it to restricted_tcp (which includes 9443 by default). Keep it outside the passive FTP range. Wildcard listeners can accept both address families, so also enable firewall.ipv6. Except for localhost, hostnames are checked against both families without resolving DNS. Alternatively, limit admin tokens with allowed_ips, or listen on 127.0.0.1:9443 and reach the UI through an SSH tunnel. An external firewall may already restrict access, but configuration validation cannot verify it.

TLS Certificates

The Web UI serves over HTTPS. Configure TLS certificates under webui:

webui:
  tls_cert: "/var/cpanel/ssl/cpanel/mycpanel.pem"   # certificate PEM file
  tls_key: "/var/cpanel/ssl/cpanel/mycpanel.pem"     # private key PEM file

On cPanel servers, you can reuse the cPanel self-signed certificate (both cert and key are in the same PEM file). For production, use a proper certificate from Let’s Encrypt or your CA.

The private key may appear before or after the certificates. The first certificate must be the leaf served by the Web UI, followed by any chain certificates. Renewal only replaces an expiring CSM-generated leaf; it preserves the private key and all other contents of a combined file. Certificates issued outside CSM are left unchanged.

If both paths are empty, CSM generates an ECDSA self-signed certificate and key under state_path. Set both paths to use an operator-managed certificate. Configuring only one of the pair is invalid.

Validation

csm validate           # syntax check
csm validate --deep    # syntax + connectivity probes (SMTP, webhooks)
csm config show        # display config with secrets redacted

Editing csm.yaml by hand

CSM stores a sha256 of the main config in integrity.config_hash and a separate digest of loaded drop-ins in integrity.confd_hash. It refuses to start if the on-disk files disagree with those values. This is a tamper-detection feature. It catches edits that were never recorded, but it does not stop anyone who can write /etc/csm, because they can record new hashes too. There are two supported edit workflows depending on which fields you touch.

Fast path: SIGHUP reload (safe fields only)

For fields tagged as hot-reload-safe (alerts, thresholds, detection, suppressions, auto_response, bpf_enforcement, reputation, email_protection, disabled_checks, confd), the daemon can accept the change without a restart:

sudo cp /etc/csm/csm.yaml /etc/csm/csm.yaml.bak-$(date +%s)

# edit /etc/csm/csm.yaml with your favourite editor

sudo systemctl reload csm
sudo journalctl -u csm -n 20 --no-pager

Every save CSM makes itself (the Settings page, csm rehash, the installer) first copies the current file to csm.yaml.bak next to it, so the previous version is one cp away. For a symlinked config, the backup sits beside the resolved target. A first write creates no backup; an unchanged save keeps the previous backup. Backups have mode 0600.

Retired settings are ignored with validation warnings, including settings in YAML merges and conf.d fragments. Removing them does not require a restart.

A reload after a comment edit or retired-setting removal updates the live integrity hashes even when no setting changes. This includes edits to conf.d fragments. If csm rehash already signed the edit, reload adopts the verified hashes without rewriting the main config. A reload with no edits neither rewrites the config nor republishes it, so it leaves dry-run history alone. Signing failures emit a Critical config_reload_error finding and count as reload errors.

systemctl reload sends SIGHUP (wired via ExecReload= in the unit file). The daemon re-reads the file, validates it, diffs it against the running config, and if every change is on a field tagged hotreload:"safe" it swaps the new values into the live config and re-records the integrity hashes on disk. The next check tick sees the new thresholds; fanotify marks are not dropped.

Reloads and Web UI saves share one config-write lock. If a reload signs the config but cannot open the phpanel delivery queue, it reports a reload error and keeps the previous settings and mail account parser active. The live integrity hashes still follow the signed config, so scheduled checks continue.

The tagged-safe top-level fields are alerts, thresholds, detection, suppressions, auto_response, bpf_enforcement, reputation, email_protection, disabled_checks, and confd. The Settings API derives its restart hints from the same manifest that drives config.Diff, so UI hints and SIGHUP behavior cannot drift silently. A save from the Settings page runs the same apply steps as a SIGHUP, so a changed threshold, reputation whitelist, verified-bot list, forward-guard setting or response ceiling reaches the live daemon at once. Changes to their sub-keys are picked up on the next tick by the periodic scanners, the auto-response helpers (block/kill/quarantine/challenge/permission-fix), alert dispatch, and the heartbeat.

reputation.verified_bots is reconciled on reload and the bot-verifier cache is restamped when the list changes.

reputation.bot_ranges controls a long-lived updater goroutine and is tagged restart-required. A reload that changes it emits config_reload_restart_required; restart the daemon to change the auto-update switch or interval.

Two sub-keys are runtime-state exceptions. They live under a safe-tagged parent and SIGHUP accepts edits to them, but they seed long-lived in-memory state at daemon startup. Restart the daemon if you need the running process to use the new value:

  • reputation.whitelist – seeded into the threat database at startup and replaced on every successful reload, so editing the list in csm.yaml and sending SIGHUP takes effect at once. The threat database also exposes its own runtime API for adding and removing whitelist entries (via the Threat Intelligence page in the Web UI or the /api/v1/threat/* endpoints); those entries are persisted to disk and are kept separate from the configured list, so a reload never drops them.
  • email_protection.known_forwarders – captured by the forwarder watcher at startup and read by scheduled forwarder and mail-filter checks. No runtime API yet; send a restart if you edit this list.

auto_response.mail_auth_recovery is a restart-required sub-key under the otherwise safe auto_response section. It is captured by the cPanel mail auth backend probe at startup, so a reload that changes it emits config_reload_restart_required and leaves the live config unchanged.

alerts.block_digest is restart-required under the otherwise safe alerts section. The collector and ticker are built at startup, so a reload that changes these settings emits config_reload_restart_required and leaves the live config unchanged. When countries is empty, the fallback to suppressions.trusted_countries still follows safe reloads. Delivery uses the current email and webhook settings. The daily summary section gives by-country and by-category counts, and live alerts add by-reason counts; WAF categories group ModSecurity escalations and high-volume WAF attacker blocks together.

If you turn off alerts.block_digest.enabled in Settings, restart before turning off the last alert channel. Until the restart the watched-country section and live alerts stay on, and Settings rejects a channel change that would leave them with no channel. Once the running configuration has the watch off, email and webhook may both be off, and then no daily summary is sent.

The rest of the sub-keys in every safe-tagged section are read per-call (inside check functions, auto-response helpers, alert dispatchers) and hot-reload cleanly on the next tick.

Look for one of three log shapes in the journal:

  • SIGHUP: config reloaded; safe fields updated: [thresholds] – success. The new values are live.
  • config_reload_restart_required: SIGHUP reload: restart-required fields changed: [hostname ...]; live config unchanged – the edit touched a field that cannot be hot-swapped. A Warning config_reload_restart_required finding is also emitted. Fall back to the restart path below.
  • config_reload_error: SIGHUP reload: parse failed ... or ... validation error ... – the file on disk is not loadable or fails csm validate. A Critical config_reload_error finding is emitted. The live config is unchanged; fix the file and repeat.

Restart path: unsafe fields

Fields not tagged hotreload:"safe" (the majority, including hostname, state_path, webui.listen, firewall.*, email_av.* and anything that survives only one re-init per daemon lifetime) require a full restart. The integrity hashes must be re-recorded first:

sudo cp /etc/csm/csm.yaml /etc/csm/csm.yaml.bak-$(date +%s)

# edit /etc/csm/csm.yaml with your favourite editor

sudo /opt/csm/csm rehash     # re-records integrity hashes
sudo /opt/csm/csm validate   # syntax + value sanity
sudo systemctl restart csm
sudo systemctl status csm    # confirm active, no crash-loop

If the restart fails (most commonly because rehash was skipped), roll back with sudo cp <backup> /etc/csm/csm.yaml && sudo systemctl restart csm. The backup carries its own matching hash so no second rehash is needed.

Config-management tools

Config-management workflows (Ansible, Puppet, Chef) should:

  • For safe changes, notify systemctl reload csm instead of restart. The daemon re-records the hash itself; no separate csm rehash step is required.
  • For any change that may touch a restart-required field, run csm rehash before the restart notify fires. Or always send reload first, read the journal, and promote to restart only when the reload logs restart-required.

conf.d drop-ins

Files matching /etc/csm/conf.d/*.yaml are loaded after the main config and deep-merged on top of it. Override with --config-dir <path> or CSM_CONFIG_DIR; the flag wins when both are set.

Explicit operator overrides require an existing directory.

  • Order: lexicographic by filename. Scalar keys in 20-overrides.yaml override the same keys in 10-base.yaml. Use a numeric prefix.
  • Merge semantics: maps merge recursively; scalars replace the value from the main file; lists append in fragment order. All-scalar lists drop duplicate entries while keeping the first occurrence; structured lists such as webui.tokens keep every entry.
  • Trust: override directories must be absolute, must exist, and must be owned by root or the running process. The directory and every loaded fragment must not be group- or world-writable. Safe symlinked fragments are allowed, so packaged profiles can still be linked into /etc/csm/conf.d/.
  • Integrity ownership: drop-ins cannot set the integrity block. Integrity metadata is stored only in the main config.
  • Hash: integrity.config_hash covers the main file and integrity.confd_hash covers loaded drop-ins. After editing a drop-in by hand, run csm rehash before restarting, or use systemctl reload csm so the daemon can re-hash after validating the merged config. Rehash re-hashes the resolved main file only; when a real legacy copy still exists at /opt/csm/csm.yaml with the same operator content, rehash replaces it with the compatibility symlink so the two never drift. Web settings saves refuse to bless a drop-in change that has not already been re-hashed. A mismatch is refused at the next daemon start with an error that names csm rehash. Until then the running daemon keeps its old hashes, but its periodic integrity check raises a tamper finding and skips the scheduled checks on every cycle; csm doctor reports the same mismatch, so it can be fixed before a restart turns it into an outage.
  • Integration-owned fragments: a fragment its owning package rewrites on its own schedule (phpanel-server-agent rewrites 10-phpanel-runtime.yaml on every bootstrap) cannot be pinned by a static hash without making every one of those rewrites a failed restart. List such fragments by bare filename under confd.integrity_exempt in the main config, then run csm rehash once. Their content is left out of confd_hash; every other fragment stays covered. The list lives in the main config, outside the integrity block, so it is itself covered by config_hash, and a fragment cannot set confd to exempt itself.
  • Scoped re-hash: an integration that rewrites a covered fragment of its own runs csm rehash --fragment <file> (repeat the flag for several files) instead of a full csm rehash. It re-records hashes only when the named files are the only change since the last recorded state. If the binary, the main config or any other fragment changed, it records nothing and names what changed. A named file may be new, edited or deleted; a deleted one must have been part of the recorded state. Fragments under confd.integrity_exempt stay outside the recorded state either way. CSM keeps a per-fragment record in csm.yaml.confd-fragments.json beside the main config. Every hash update writes it, and each daemon start that verifies conf.d writes one when it is missing or out of date, so upgraded hosts need no extra step. Exit codes:
    • 0: re-hashed, or nothing had changed.
    • 4: refused because something besides the named files changed; stderr lists it.
    • 5: no per-fragment record for the current recorded state yet. A full csm rehash re-hashes everything and writes one.
    • Any other non-zero code: an error, including a corrupt record, an incomplete option, an invalid fragment name or a file that never existed. Do not fall back to a full csm rehash on these errors.
  • Record privacy: the per-fragment record retains covered fragment contents to validate them against the recorded digest. It is written with owner-only access beside the resolved main config and must be protected like the config itself. A corrupt record never authorizes a fallback to a full rehash; a successful startup verification can rebuild it from verified fragments.
  • Use cases: packaged integration profiles (e.g. /usr/lib/csm/profiles/phpanel-agent.yaml symlinked into conf.d/), per-host automation that should not touch the operator’s csm.yaml, secret material rendered from a vault.
ls /etc/csm/conf.d/
# 10-phpanel-agent.yaml   20-tenant-overrides.yaml

csm validate                # validates the merged config
csm config show             # prints the merged, redacted config
csm config schema           # JSON Schema for editor / CI validation

csm validate and csm config show always operate on the merged config so you can audit the effective state without grepping fragments.

detection.direct_smtp_egress

The direct SMTP detector’s backend accepts auto, bpf, legacy, or none; ports must contain TCP ports in the 1-65535 range. See Direct SMTP egress.

bpf_enforcement

BPF enforcement requires a BPF-capable connection tracker at runtime; auto falls back to legacy detection on older servers. See BPF enforcement.

firewall.tcp_in_allow

tcp_in_allow opens one TCP port to listed sources only. Each entry names a port and its sources:

firewall:
  tcp_in_allow:
    - port: 3306
      sources:
        - 203.0.113.10
        - 198.51.100.0/28
        - 2001:db8::11

Use it for a database, replication or monitoring port that a few known hosts need and nobody else should reach. The port does not need to be in tcp_in, and it may be in restricted_tcp. infra_ips is the wrong tool for this: it opens every port to the address.

port is a single TCP port. sources is required and takes IPv4 and IPv6 addresses or CIDRs; a bare address means that one host, the same as /32 or /128. A port out of range, an empty list, a malformed address and a world-wide prefix (0.0.0.0/0, ::/0) are validation errors; open a port to everyone in tcp_in instead. IPv6 sources need firewall.ipv6: true; with it off, IPv6 is not filtered at all and validation warns.

The Web UI and sshd lockout warnings treat a port with at least one usable source here as deliberate and stay quiet for it, in both IP families. An entry whose sources render no rule, such as IPv6 sources while firewall.ipv6 is off, does not count. A usable source also keeps a port in restricted_tcp reachable without infra_ips.

The accepts come right after infra_ips and before the blocked address, blocked subnet and country rules, and before the rate limits. A listed source therefore keeps reaching its port while it is blocked, and only that port: every other port still follows the blocklists. csm firewall deny on a listed source does not close the port for it. Remove the source from the entry and run csm firewall restart instead.

Entries from conf.d fragments are added to the ones in csm.yaml. A change takes effect on csm firewall restart. csm firewall status and csm firewall ports list the applied entries, including changes made by a firewall restart or rollback, and the Web UI firewall page shows and counts the entries with its port exceptions. That page reads the config the daemon loaded, so after a firewall restart or rollback it can differ from the applied policy until the daemon reloads; the CLI shows what is applied. Hosts that support the key report the firewall.tcp_in_allow.v1 capability.

For a single address on one host without a config change, csm firewall allow-port still works. That allow is a soft one: it sits after the blocklists, so an operator block overrides it.

firewall.protocol_allow

protocol_allow opens an IP protocol that has no ports to listed peers only. Each entry names a protocol and its peers:

firewall:
  protocol_allow:
    - protocol: vrrp
      peers:
        - 192.0.2.11
        - 2001:db8::11

The only protocol today is vrrp (IP protocol 112), for keepalived and other VRRP speakers on database or load-balancer pairs. tcp, udp, icmp and icmpv6 have their own keys and are refused here. peers is required and takes IPv4 and IPv6 addresses or CIDRs. An empty list, an unknown protocol, a malformed address and a world-wide prefix (0.0.0.0/0, ::/0) are validation errors.

For each peer the input chain accepts the protocol from that address and the output chain accepts it to that address. For vrrp the output chain also accepts advertisements to the VRRP multicast group of each family that has a peer (224.0.0.18, ff02::12). Advertisements a peer sends to the group match its input rule. IPv6 peers need firewall.ipv6: true; with it off, IPv6 is not filtered at all.

The input accepts come right after infra_ips and before the blocked address, blocked subnet and country rules. A blocked peer therefore still exchanges VRRP with this host, and only VRRP. Without this, a block on a peer makes both members claim the shared address. Blocks of other addresses do not affect these rules.

csm firewall status and csm firewall ports list the applied entries, including changes made by a firewall restart or rollback. Hosts that support the key report the firewall.protocol_allow.v1 capability.

firewall.dos_exempt_ranges

dos_exempt_ranges is a list of CIDRs or single IPs (/32 for IPv4, /128 for IPv6) declared by the operator. Entries are validated at startup; /0 default routes, bare hostnames, and malformed CIDRs are rejected before the daemon starts. Default: empty (no declared ranges).

dos_exempt_known_mail_providers (bool, default true) adds Google and Microsoft outbound mail IP ranges to the effective exempt set automatically. The ranges are sourced dynamically and cached on disk; a built-in snapshot covers startup before the first live refresh. The cache is refreshed every 12 hours.

Sources in the effective exempt set bypass the new-connection rate-limit for their IP family, the IPv4 concurrent connection-limit, and the TCP port 25/465/587 flood meters for their IP family. Subnet auto-block (spray, ASN-crawl, and netblock escalation) skips CIDRs that intersect an exempt range, and exempt IPs do not count toward the netblock threshold.

Exempt sources do not bypass: manual blocks (csm firewall deny or csm firewall deny-subnet), SYN flood protection, UDP flood protection, country blocking, and port policy. A manual block placed inside an exempt range still takes effect because blocked sets are evaluated before the DoS meters. See Firewall - DoS-exempt ranges.

Credential rotation

Environment-backed credentials come from the environment inherited when CSM starts. Reading that environment before each request does not import edits to an environment file or exports from another shell. Restart CSM after changing its launch environment. csm rehash, SIGHUP, and systemctl daemon-reload do not replace a running process’s environment.

The same rule applies to these integrations:

IntegrationEnvironment variable name configured inStatic fallback
Upstream threat intelligencereputation.upstream.token_envreputation.upstream.token
Rspamdreputation.rspamd.token_envreputation.rspamd.token
Verdict callbackauto_response.verdict_callback.hmac_secret_envauto_response.verdict_callback.hmac_secret
Phpanel finding webhookalerts.webhook.hmac_secret_envalerts.webhook.hmac_secret

A non-empty environment value takes precedence over the static fallback. An unset or empty variable uses that fallback. Each verdict exchange keeps one secret for both request signing and response verification.

Configure a systemd environment file

Create /etc/csm/credentials.env, owned by root with mode 0600. Put the actual values in this file through your editor or configuration manager:

CSM_UPSTREAM_TOKEN=replace-with-upstream-token
CSM_RSPAMD_TOKEN=replace-with-controller-password
CSM_VERDICT_SECRET=replace-with-verdict-secret
CSM_WEBHOOK_SECRET=replace-with-webhook-secret

Set the corresponding CSM configuration fields to these variable names. Then create /etc/systemd/system/csm.service.d/credentials.conf:

[Service]
EnvironmentFile=/etc/csm/credentials.env

Apply the new unit configuration and start CSM with that environment:

systemctl daemon-reload
systemctl restart csm.service
systemctl is-active csm.service

For later rotations, update the existing credentials file and run systemctl restart csm.service. daemon-reload is needed when the unit or its drop-ins change, not for a value change in an already referenced file. For a foreground daemon, stop it and launch a new process with the updated environment supplied by its launcher.

Coordinate the change with each receiver. Where the receiver supports overlapping credentials, retain the old credential through restarts and in-flight requests. Verify successful authentication with the new credential at the receiver without logging its value, then retire the old one.

Regression coverage

TestEnvironmentCredentialRotation launches a subprocess from an environment file and captures real HTTP requests from all four clients. Requests before and after an external file edit use the old credential while the process stays running. Restarting the subprocess from the edited file switches all four clients to the new credential. The test does not change the running process’s environment with os.Setenv.

Custom account roots

account_roots accepts absolute, normalized directory paths or glob patterns. The same configured content trees are eligible for manual remediation and quarantine restore. Detected panel account homes and the existing scratch quarantine locations keep their normal scope. A configured content directory itself is not a restore target; restore operates beneath it.

Symlinks in a configured root or its ancestors are rejected. Unsafe trees are excluded from remediation without disabling other tenants’ valid roots. Destination directories are pinned during restore, including when a tenant renames a directory while the operation is running.

Service write access

The packaged service uses ProtectSystem=strict. A custom content tree needs a write grant before the daemon can quarantine, clean, or restore files there. Keep the account container directory owned by root and not writable by tenants:

account_roots:
  - /srv/csm-accounts/*/public

For this layout, /srv/csm-accounts is root-owned; the account directories below it can belong to their tenants. Provision the directories before generating the drop-in:

set -e
fragment="$(mktemp)"
trap 'rm -f "$fragment"' EXIT
csm systemd-roots > "$fragment"
cat "$fragment"
install -d -m 0755 /etc/systemd/system/csm.service.d
install -m 0644 "$fragment" /etc/systemd/system/csm.service.d/50-account-roots.conf
systemctl daemon-reload
systemctl restart csm.service
csm doctor

csm systemd-roots only prints the drop-in. It grants existing content directories or their nearest root-controlled ancestor, and adds mount ordering for those paths. It refuses symlink roots and broad grants such as /, /srv, or /etc. It escapes path quoting and systemd specifiers. The generated fragment appends grants to the packaged service; it does not reset that list.

A tenant-owned directory cannot itself be a grant: its owner could replace it with a symlink before systemd starts. If the command cannot find a narrow, root-controlled ancestor, move the accounts beneath a dedicated root-owned container directory and update account_roots.

Regenerate the fragment after changing roots or provisioning a tree outside existing grants. Changing account_roots requires a daemon restart. If integrity verification is enabled, update its baseline for an intentional configuration change before restarting, as described in Configuration.

Verification

csm doctor checks configured custom roots and detected panel roots outside the packaged home grant. It reports missing roots, unsafe aliases, missing service grants, and read-only mounts in the running daemon’s filesystem view. A stopped service or an unavailable systemd connection is reported as unverified access. Reloading unit files alone does not change the running service’s mount namespace; restart the service and run doctor again.

After setup, verify detection, quarantine, listing, and restore on a test account under the custom root. Confirm that the restored file still belongs to its tenant. A sibling tree outside the configured roots and existing scratch locations must remain ineligible for restore.

The repository includes a disposable systemd service test. Build an image from build/Dockerfile.systemd-test, then use the Linux wrapper:

GO_LINUX_IMAGE=csm-systemd-test scripts/go-linux.sh bash scripts/systemd-account-roots-test.sh
cat .cache/systemd-account-roots/result

The test runs production detection, quarantine, listing, restore, and doctor inside the packaged sandbox, with a generated grant for a separate test volume. It verifies owner, permissions, timestamps, and rejected sibling and symlink destinations. This tests service confinement; it does not start the daemon’s watchers or replace panel integration coverage. Logs and generated grants are saved beside the result file. The harness refuses to run outside its disposable Linux container.

Service configuration writes

The packaged and CLI-installed units use ProtectSystem=strict. The daemon can write these configuration directories when they exist:

DirectoryRuntime writer
/etc/csmCSM configuration and fragments
/etc/auditCSM audit rules and augenrules generated configuration
/etc/modprobe.dRepair of an existing, opted-in AF_ALG mitigation marker
/etc/apache2/conf.dcPanel Apache/LiteSpeed challenge snippet, virtual patches, and default overrides
/etc/apache2/conf-enabledDebian Apache challenge snippet
/etc/httpd/conf.dRHEL Apache challenge snippet
/etc/nginx/conf.dNginx challenge snippet
/usr/local/apache/confLegacy Apache virtual patches and overrides
/usr/local/lsws/conf/templatesStandalone LiteSpeed challenge template

The other state, quarantine, account, spool and cPanel plugin grants remain in place. Optional directory grants tolerate absent software. Install the relevant web server before starting CSM; restart CSM after creating a previously absent granted directory. Custom ModSecurity override locations require a drop-in that grants only their parent directory, followed by systemctl daemon-reload and systemctl restart csm.service. Atomic replacement requires a directory grant. See custom account roots for account grants.

The installer generates the unit for the systemd it finds. systemd 239 (EL8, CloudLinux 8) does not know ProtectHostname, ProtectKernelLogs or ProtectClock; it would log “Unknown lvalue” for each at every start and ignore them, so the installer leaves them out there and prints which ones it omitted. Newer systemd receives the full unit, identical to the packaged one. A binary-swap upgrade keeps the existing unit; run csm install to regenerate it.

The daemon cannot write arbitrary files directly under /etc, SSH configuration, account databases, or systemd unit files. The AF_ALG seccomp drop-in is installed by the operator’s csm harden CLI, outside the daemon. Periodic AF_ALG module removal uses a bounded transient service with fixed module names; the daemon retains ProtectKernelModules=yes and the module syscall restriction.

Exim transactions

The forward guard sends one bounded JSON request over stdin to its own executable in a transient service. The helper accepts only apply/remove operations and policy data. It accepts no paths, command names or shell programs. Applying or removing the managed Exim block, the cPanel rebuild, and rollback all run outside the daemon mount namespace. Operator configuration is preserved. Lookup refreshes remain under /var/lib/csm and do not rebuild Exim.

The helper requires root. An exclusive process lock rejects overlapping config transactions without modifying Exim; the daemon reports the failure, and the operator can reload after the earlier transaction finishes. Its lifetime is bounded, and a failed command is never retried as an unsandboxed duplicate. Existing fixed Exim queue/query operations also use transient services. If the systemd bus is unavailable, the existing wrapper attempts direct execution once; inside the service, restricted writes then fail and are reported. A scope unit is unsuitable because it inherits the calling process’s sandbox.

Parse worker units

The daemon reads tenant zip listings for the phishing-kit check, and unpacks email attachments when email AV is on, in separate workers. Each starts on demand as a transient unit with systemd-run: one unit for zip listings, and one per email_av.scan_concurrency slot for mail. Each unit gets these properties:

  • DynamicUser=yes, CapabilityBoundingSet= (empty), NoNewPrivileges=yes;
  • PrivateNetwork=yes, RestrictAddressFamilies=AF_UNIX;
  • ProtectSystem=strict, ProtectHome=yes, PrivateTmp=yes, PrivateDevices=yes, ProtectKernelTunables=yes, ProtectKernelModules=yes, ProtectControlGroups=yes, RestrictNamespaces=yes, RestrictRealtime=yes, MemoryDenyWriteExecute=yes, UMask=0077;
  • SystemCallFilter=@system-service and SystemCallFilter=~personality, with SystemCallErrorNumber=EPERM. The personality deny stands in for LockPersonality=yes, which systemd-run on systemd 239 rejects;
  • MemoryAccounting=yes, MemoryLimit= and MemoryMax= set to the worker’s memory cap, MemorySwapMax=0;
  • TasksMax=64, LimitNOFILE=256, LimitCORE=0, CPUWeight=50, and a one-hour RuntimeMaxSec. LimitFSIZE= is 64 MiB for zip listings; for mail it is 1 MiB above email_av.max_attachment_size, and never below 64 MiB;
  • OOMScoreAdjust=1000, making parse workers preferred victims under host memory pressure, and Slice= set to a slice of its own per unit name prefix directly under system.slice, for example system-csm\x2dparse\x2dworker\x2dmail0.slice. The slices stay after their units end; they contain no processes but retain their accounting metadata.
  • InaccessiblePaths=- for each account home volume that ProtectHome=yes does not cover: the account roots outside /home (for example /var/www/vhosts on Plesk) and the sibling volumes registered cPanel accounts live on, such as /home2. The list is read once when the daemon sets up its workers, or when csm scan starts one.

Before launching a worker, CSM sets its slice’s runtime CPUWeight=50 through systemd. CPU weights compare siblings: the service’s weight alone would not limit its share against the daemon outside that slice. Slice setup shares the worker’s startup timeout, and a failure prevents the launch. Empty slices and their runtime settings remain until reboot.

On cgroup v2, a replacement waits for earlier units in its slice to disappear before taking the slice’s OOM snapshot. If they outlive the startup timeout, the replacement gets no request and reports a startup failure.

The zip worker receives the zip as an open descriptor and returns entry names in a sealed memfd. A mail worker receives the message’s spool header and body as open descriptors and returns each attachment in its own sealed memfd; the daemon scans those and closes them. It cannot create IPv4 or IPv6 sockets. Unix sockets remain available. /home, /root, /run/user and the account home volumes listed above are hidden; an account volume created after the daemon started is not hidden until it restarts. Most host files are read-only. The worker can still read world-readable host files outside those directories and write in its private temporary directories. The daemon does not forward its environment to the unit; systemd supplies the service environment.

A worker’s standard error reaches the daemon’s log through a pipe, never directly. Each line is cut to 512 bytes and quoted, so control characters and newlines cannot forge log lines, and at most 256 lines or 64 KiB of log entries are logged per worker unit; after that one “further output dropped” line is logged and the rest is discarded. Error text in a worker’s reply is quoted the same way. When the launcher exits, final diagnostics drain for at most one second. The log reader then closes even if another process still holds its writer.

MemoryLimit caps RAM only on cgroup v1, so the daemon also writes memory.memsw.limit_in_bytes in the unit’s own memory cgroup, or memory.swappiness=0 where there is no swap accounting. On cgroup v2 it sets memory.swap.max to 0, and refuses to use a worker on a host with swap on but no swap accounting. The daemon can write those files because csm.service has ProtectControlGroups=no. A worker whose memory cannot be capped or measured gets no request: the phishing check reports itself incomplete, and mail is handled as an incomplete scan under email_av.fail_mode. Without cgroup v2 swap accounting, the watchdog also checks that swap stays off throughout the worker’s lifetime.

The unit runs the daemon’s own binary as its dynamic user, so the binary is installed as mode 0711: others can execute it but not read it, and only root can replace it. Package, CLI and script installs set this mode under any umask and make the installation directory searchable by others; its subdirectories keep their private modes. A rolled-back upgrade restores the previous mode. When the binary or a directory above it is not executable by others, the daemon logs one line at startup, raises a Critical email_av_isolation_unavailable finding, and starts no worker. The phishing check reports itself incomplete, and mail is handled as an incomplete scan under email_av.fail_mode. Failed unit starts raise the same finding, at most once an hour. This includes starts refused because the daemon has exhausted its descriptors; those failures do not open the worker breaker. Mail and zip workers share this allowance across spool watcher replacements. A finding rejected by a full alert queue leaves the allowance available for a retry.

Workers are killed by unit name with systemctl kill, never by a PID. If the daemon dies, a worker waiting for a request reads end of file and exits. A worker stuck parsing may remain until the lifetime cap stops it. At startup, the daemon attempts to kill leftover csm-parse-worker-* units; cleanup failures are logged and do not prevent startup.

A worker the daemon stops on purpose, such as at shutdown, is logged as stopped and never as a memory failure.

disabled_checks: [phishing] disables scheduled phishing checks. Requested account scans can still use the worker, and startup cleanup still runs. email_av.enabled: false starts no mail workers.

The PHP taint worker runs in the same kind of unit, named csm-phptaint-worker-<generation>, with a 512 MiB memory cap and no file writes. At startup the daemon attempts to kill leftover csm-phptaint-worker-* units. The binary’s permissions are checked before each worker launch. If the dynamic user cannot execute it, the daemon logs why and candidate files are reported as not analyzed, just as when a unit fails to start or dies. These failures appear as worker_failure coverage gaps. disabled_checks: [php_taint_deep] stops new analyses; startup cleanup still runs.

YARA-X runs in two units of the same kind. Each rule compile runs in csm-yara-compile-worker-<generation> with a 1 GiB memory cap, from rule text the daemon passes in memory; the compiler cannot open include files. Scans run in one csm-yara-worker-<generation> with a memory cap of 512 MiB plus the larger of thresholds.full_scan_max_file_mb and, with mail scanning on, email_av.max_attachment_size, no file writes and the compiled rules loaded from memory. A reload that changes the file cap applies to the next scan worker. Both allow executable memory, which YARA-X needs for its compiled rules; the other lockdown properties are unchanged. At startup the daemon attempts to kill leftover units of both kinds. While no worker can scan, files and mail attachments are reported as not scanned by YARA, never as clean. signatures.yara_worker_enabled: false runs YARA-X inside the daemon as root instead; csm validate warns about it.

The in-process csm scan <user> command starts its own worker on demand. Its unit prefix includes the CLI process ID, so concurrent scans use distinct slices and cannot attribute another scan’s OOM kill to their own worker. Its report warns when worker setup, startup or archive listing fails, even if later archives are checked successfully. A scan with unchecked archives does not report the account as clean.

Limits and validation

These restrictions reduce accidental and misdirected writes. CSM remains a privileged root daemon: fanotify, firewall administration, BPF, process inspection and remediation require elevated privileges. The unit retains its current capabilities, and root access to the system bus permits transient services. This is not a complete boundary against arbitrary code execution as root. Reducing capabilities or separating privileged operations behind a separately permissioned broker requires its own feature and kernel compatibility review.

scripts/systemd-account-roots-test.sh runs the actual packaged sandbox in a disposable systemd environment. It checks unrelated /etc writes return EROFS, managed directory atomic replacement, ModSecurity overrides, and Exim helper apply/remove/rollback using a fixed command fixture. It also exercises account quarantine, restore and process signaling. The Exim fixture validates the helper transport and filesystem behavior; it does not validate a real cPanel rebuild. The cPanel release tests exercise real daemon restarts and require a licensed image and a live run to establish that platform’s acceptance.

Upgrading

Use the signed APT/DNF repository for maintained package upgrades. Standalone deploy scripts require successful detached signature verification for current releases. They verify with OpenSSL 3.0+ where available, otherwise with the installed CSM binary’s csm verify-release, otherwise with python3-cryptography, so upgrades keep working on EL8/CloudLinux 8 (OpenSSL 1.1.1) including the first upgrade to a build that provides csm verify-release. When no verifier is present the upgrade stops; disabling CSM_REQUIRE_SIGNATURES does not enable unsigned current upgrades. Set CSM_VERIFIER_BINARY to select a specific CSM binary. See Release signing for the historical-release exception.

Use the same signed repository that installed CSM:

sudo apt update && sudo apt install --only-upgrade csm   # Debian/Ubuntu
sudo dnf upgrade csm                                     # RHEL family

The package preserves the operator config and state, updates runtime assets, re-records integrity hashes, and restarts the daemon when it was already active.

Debian and Ubuntu upgrades keep the local main config without a dpkg prompt. Compare it with the shipped template at /usr/lib/csm/csm.yaml.default. RPM keeps a locally modified config and writes a changed template to /etc/csm/csm.yaml.rpmnew. RPM may update an unmodified config in place.

If the main config is missing on upgrade and there is no regular legacy config to migrate, package setup stops before restarting CSM. Restore your config to /etc/csm/csm.yaml and retry.

Standalone installations

sudo /opt/csm/deploy.sh upgrade

The upgrade refuses to install a release older than the one running. To roll back deliberately, run it with CSM_ALLOW_DOWNGRADE=1.

The helper:

  1. Downloads and verifies the binary and supporting assets
  2. Stages the UI, rules, PAM files, and deploy helper before downtime
  3. Stops the daemon and keeps the previous binary and assets as rollback material
  4. Activates the staged release and rehashes the config once
  5. Restarts the daemon, checks sustained liveness, and runs csm doctor

The health gate waits 20 seconds by default. Set CSM_UPGRADE_HEALTH_SETTLE to an integer from 1 to 3600 seconds for a longer or shorter observation window. Invalid values fail the gate. Doctor warnings do not trigger rollback; failed checks do.

If activation, rehash, startup, or the health gate fails, the helper stops the new daemon, restores the previous binary and assets, re-records the restored binary hash, and starts the previous version. It exits nonzero and reports where it retained recovery material. If recovery itself fails, it reports an incomplete rollback that needs operator attention.

Optional nightly upgrades

Packages ship a disabled sample at /opt/csm/configs/cron/csm-auto-upgrade. It runs only after an operator installs it into /etc/cron.d/:

csm version && /opt/csm/deploy.sh check
sudo install -m 0644 /opt/csm/configs/cron/csm-auto-upgrade /etc/cron.d/csm-auto-upgrade

Confirm the host runs a published release first. The checksum check reports any different build as an available update. The downgrade guard rejects a lower version, but a development build with the same version can still be replaced by the published release.

The job invokes the standalone deploy helper every night between 03:30 and 04:30, with a random delay and a nonblocking lock. This updates runtime files directly; it does not update APT/DNF’s installed package version. Hosts that need package-manager ownership and version tracking should use their package upgrade automation instead.

Output goes to /var/log/csm/auto-upgrade.log. Failures also produce cron mail to root; configure and test root’s mail alias before enabling the job. This notification does not depend on CSM’s alert channels or on successful daemon recovery. Successful runs and an already-held lock produce no mail. Staggering spreads upgrades over an hour but does not stop a bad release from reaching the fleet.

Troubleshooting

“store: opening bbolt: timeout” – Most operator commands that need live state now route through the control socket at /var/run/csm/control.sock. This error should only appear from commands that intentionally open the bbolt file directly, such as csm store compact, csm store import, csm store reset-bot-verify, csm db-clean --drop-object, or a second daemon start while one daemon already owns the database.

Fix: stop the daemon before direct-store maintenance commands, then retry:

systemctl stop csm
csm store compact
systemctl start csm

If systemctl says CSM is stopped but bbolt still times out, find the process holding /var/lib/csm/state/csm.db and stop that process after review. Do not delete csm.lock; it is only the daemon instance guard and does not release bbolt’s file lock.

“csm: daemon not running” – CLI commands that talk to the daemon exit 2 with this message when the control socket is missing. This includes csm run*, csm check*, csm baseline, csm status, csm firewall ..., csm store export, csm export --since, and csm phprelay .... Start the daemon with systemctl start csm. Bootstrap commands that run before the daemon exists (csm install, csm validate, csm config schema, csm verify, csm rehash) do not require it.

Never delete csm.db – it contains all historical findings, firewall state, email forwarder baselines, and per-account data. If you delete it, the web UI will show empty data until the next full scan cycle (up to 60 minutes for deep scan findings). Restore from backup when possible; for an intentional reset, run csm baseline --confirm rather than removing the database by hand.

Config changes require rehash – After editing a restart-required field in csm.yaml or any conf.d drop-in, run csm rehash once, validate, then restart. Hot-reload-safe changes can use systemctl reload csm; the daemon validates the accepted config and re-records its hash itself. A restart that fails with conf.d hash mismatch means a drop-in changed since the hashes were last recorded: rehash if the change was intentional, and list fragments an integration rewrites on its own under confd.integrity_exempt so they stop needing one. csm doctor reports the mismatch while the daemon is still up.

FHS migration (state, config, drop-ins, and profiles)

Current packages use FHS paths for state, config, drop-ins, and shipped profiles. Legacy main configs continue to work during the transition.

ConcernLegacy pathCurrent path
Drop-in fragmentsn/a/etc/csm/conf.d/*.yaml
State directory/opt/csm/state/var/lib/csm/state
Shipped profilesn/a/usr/lib/csm/profiles
Binary/opt/csm/csm/opt/csm/csm (unchanged)
Main config/opt/csm/csm.yaml/etc/csm/csm.yaml
Legacy config pathn/a/opt/csm/csm.yaml symlink

The package postinstall creates the FHS directories with the right ownership. If /opt/csm/csm.yaml is a real file and /etc/csm/csm.yaml is absent or still the shipped placeholder, the package copies the legacy config into /etc/csm/csm.yaml and then replaces the old path with a symlink. If both paths are real files with different operator content, CSM refuses the implicit default path until you move one aside or pass --config <path>. Copies that differ only in the integrity hashes CSM writes itself (binary_hash, config_hash, confd_hash) are treated as the same configuration: the daemon starts from /etc/csm/csm.yaml, and the next csm rehash replaces the legacy copy with the symlink.

The daemon copies a non-empty legacy /opt/csm/state/ into the new state directory on first start, but only when the new directory is empty (so a partial migration cannot corrupt it). The legacy directory is left in place; remove it after you have verified the new install.

Operators upgrading by manual binary swap (without re-running the package postinstall) keep the legacy state path if state_path: /opt/csm/state is pinned in the existing csm.yaml. To move state to the FHS layout, either reinstall the package or create the directories by hand and remove the state_path: override.

systemd Type=notify drop-in

The packaged unit file is Type=notify with WatchdogSec=300. The daemon signals READY=1 after watchers attach and pings WATCHDOG=1 on schedule, so systemctl is-active reflects truth and the watchdog kills a hung daemon.

Older units shipped Type=simple. The watchdog still functions because the daemon pings regardless of unit type, but systemctl status only sees the process, not “watchers attached.” If you need the new behavior on an older unit, drop in:

# /etc/systemd/system/csm.service.d/notify.conf
[Service]
Type=notify
NotifyAccess=main

Then systemctl daemon-reload && systemctl restart csm. Verify with systemctl show csm -p Type -p StatusText.

Auto-response dry-run safety default

The first disabled queued response raises admission state to schema 10, which older binaries that read only schema 9 cannot open; they keep blocking with the legacy budget. While admission only previews and forecasts, it creates no genuine charge, attempt or unknown hold beyond the state legacy blocking keeps. Preview accounting is not enforcement state. This is not a downgrade guard for a ledger that has executed real actions. Never delete the state database or replace it with an older copy to get past the format check.

The first positive elapsed episode end this release records raises admission state to schema 13, including on hosts that only preview or forecast. Older binaries cannot open it and keep blocking with the legacy budget, as above. Roll back only to a build that supports both schema 13 and elapsed episode records.

A live admission ledger records each episode’s latest verified block and retains a verified episode’s boundary even after an unattempted return is forgotten. The first verified block raises the ledger to schema 15. Roll back only to a build that reads schema 15, understands the recorded verified generation, and retains and cleans up these rows; older builds refuse this ledger. This writer runs only on hosts where admission executes. Keep the database when changing binaries.

After live re-admissions, roll back only to a build that also keeps the original verified boundary and caps queued returns at it. A build that only reads the format can give an old queued return a full lifetime. Stop the daemon before replacing the binary and keep the database.

auto_response.dry_run defaults to true when the key is absent. Automatic IP blocks are not applied. With the firewall on outside observe mode, the dry-run count holds retained forecast examples, including older entries until pruning; it is not active occupancy or a count of all attempted blocks. If your auto_response: block sets enabled: true and block_ips: true but does not set dry_run, add dry_run: false explicitly before relying on auto-block. Verify with:

csm status --json | jq '.auto_response_dry_run, .dry_run_blocks'
csm firewall status            # check that "Recently Blocked" picks up new entries after the restart

Manual csm firewall ... operations bypass dry-run and always apply.

Admission applies automatic IP blocks

From this release, a host with auto_response.dry_run: false, the firewall on and outside observe mode applies automatic IP blocks through admission, within its hourly ceiling and the free slots of the temporary set. A host that already runs with dry_run: false does so from the first start of the new build. To check the admission comparison first, set dry_run: true before the upgrade, compare a dry-run window as described in finding streams, and set dry_run: false once it passes.

Legacy blocks applied before the upgrade stay until they expire. When admission wants a longer block for an address such a block already holds, the work waits for that block to end. For Critical work the wait raises a capacity notice with reason pending_recovery; the legacy block keeps the address blocked meanwhile.

The automatic block counter now also counts the challenge, central and incident blocks admission applies, so block rates on dashboards can rise after the upgrade.

To stop admission applying blocks without changing the binary, set auto_response.dry_run: true and run systemctl reload csm. Dry run stops every automatic IP block, legacy ones included; blocks already applied stay until they expire. To return automatic blocking to the legacy path, roll back the binary instead.

The supported rollback target is the pre-flip 4.14.0 release. Older builds cannot safely read all admission records; some also reject newer config keys. Removing config keys does not make those versions safe rollback targets. Keep the database and firewall state, including action identifiers and absolute expiries. Do not restore an older state snapshot to bypass a format check.

Before upgrading, retain a verified 4.14.0 package or release artifact and its matching assets; also retain the new artifact for recovery. The upgrade helper deletes its previous-binary copy after a successful upgrade. Resolve the active config and state locations, including drop-ins, and keep a restricted backup.

For a rollback:

  1. Set auto_response.dry_run: true in the effective configuration, reload, and confirm csm status --json reports auto_response_dry_run: true. A drop-in may override the main config. Confirm admission has stopped executing (admission.owner.executing is false) before proceeding; a failed reload is not a successful stop.
  2. Note the installed version, the csm doctor result and the current timed blocks, then stop csm.service and confirm it is inactive. Keep a consistent backup of the stopped database and firewall state.
  3. Install the retained 4.14.0 package with the package manager, allowing the downgrade. The standalone helper installs only the latest release: on a standalone host, put the retained 4.14.0 binary and its assets in place and run csm rehash before starting. Do not substitute a locally compiled binary.
  4. Start the service if the installer did not, verify the running version, sustained service health and csm doctor, and compare remaining timed blocks and expiries with the saved inventory. Keep dry run enabled until these checks pass. Version 4.14.0 only previews admission; with dry run off, the legacy path applies automatic blocks.
  5. To resume legacy automatic blocking, set auto_response.dry_run: false, reload and verify the effective status. Leaving it true continues to suppress automatic blocking. Existing blocks retain their expiry; this procedure does not flush them or promise to restore already expired blocks.

If installation or health checks fail, keep dry run enabled and restore the retained verified new release and its matching assets through the same workflow. Rehash if required, start and repeat the health and block checks. Keep current state in place; recover a damaged state backup only through the documented backup/restore procedure. Do not re-enable automatic blocks on a failed health check.

Admission starts live once per ledger. Upgrading again after a rollback does not repeat that start: blocks the legacy path applied in the meantime stay until they expire, and the legacy path’s hour is not carried over into admission’s hourly ceiling. To compare a dry-run window again before admission blocks, keep auto_response.dry_run: true through the upgrade.

CLI Commands

Packages and the standalone installer expose /usr/sbin/csm, which points to /opt/csm/csm. Commands that talk to the control socket require the daemon to be running; direct-store maintenance commands say explicitly when it must be stopped.

Global flags

csm rehash and csm capabilities reject missing option values before recording hashes or printing output. csm capabilities accepts but does not read config paths.

FlagDescription
--config <path>Override the main config path. Default: /etc/csm/csm.yaml, with fallback to /opt/csm/csm.yaml on legacy installs.
--config-dir <path>Override the conf.d directory. Default: /etc/csm/conf.d. Wins over CSM_CONFIG_DIR when both are set. Override paths must be absolute, trusted, and not group- or world-writable; loaded fragments must meet the same write-safety check.

Daemon

CommandDescription
csm daemonRun as persistent daemon (fanotify + inotify + PAM + periodic checks). Signals systemd READY=1 after watchers attach and pings WATCHDOG=1 on the configured interval. The systemd notification variables are read once at startup and removed from the environment, so no command the daemon runs inherits them.

Notification writes time out if systemd stops reading its socket, so they cannot block the daemon indefinitely. ModSecurity rule actions are checked every five minutes using file contents, including linked rule files. Unchanged rule trees skip parsing; empty or incomplete builds keep retrying, and platform detection resumes when the selected directories no longer yield rules. A temporarily empty tree keeps the last loaded actions until a replacement is available.

Checks

CommandDescription
csm runRun all checks now via the daemon, send alerts
csm run-criticalCritical checks now via the daemon (the daemon also schedules critical checks internally every 10 min)
csm run-deepDeep checks now via the daemon (the daemon also schedules deep checks internally every 60 min)
csm checkRun all checks via the daemon, print findings to stdout, no alerts / auto-response
csm check-criticalTest critical checks only (dry-run via daemon)
csm check-deepTest deep checks only (dry-run via daemon)
csm scan <user> [--alert]Scan a single cPanel account (capped quick scan). --alert sends alerts for the findings. An account whose home is on an additional home partition is found through the cPanel account list and its passwd home.
csm scan <user> --full [--wait] [--json] [--respect-ignores] [--quarantine]Uncapped deep scan of one account, routed through the daemon (bypasses the per-account file cap). It also runs the YARA rules over the account’s document roots, one file at a time through the YARA worker; files over full_scan_max_file_mb are reported as not scanned, and image files are scanned after everything else so a scan that runs out of time has read the code first. It is report-only by default; --quarantine acts on critical file findings the way the scheduled auto-response does (WordPress core, plugin and theme files are cleaned in place unless cleaning would leave them empty or their release does not ship them; directories, symlinks and lower-severity findings are left for review); --wait polls to completion and prints the report, and exits with status 3 when the scan stopped early, was canceled or failed; --respect-ignores honors ignore_paths.
csm scan --all --full [--wait] [--json] [--respect-ignores]Server-wide uncapped full scan across every account. Quarantine per account after review (--quarantine is rejected with --all).
csm scan --status [id] [--json]List full-scan jobs, or show one job by id.
csm scan --report <id> [--json]Print the stored report for a completed full-scan job. Exits with status 3 when that scan did not cover everything.
csm scan --cancel <id> [--json]Cancel a running full-scan job.

Exit status 3 means the scan did not cover everything: a check timed out or failed, a file was skipped as oversize or unreadable, the account was not found, or not every finding was stored.

Passwd home fallback accepts an account-named directory directly under a sibling home volume, such as /home2, and rejects symlinked volumes and homes. Arbitrary passwd locations outside this boundary are not used for account scans.

The full-account YARA pass does not follow tenant symlinks. It reports unreadable or changed files and directories as coverage gaps, and keeps confirmed matches if a later read or worker call times out. Large worker requests use a private temporary snapshot in the daemon state directory, bounded by full_scan_max_file_mb, which is removed after the request. Failure to create or scan that snapshot is reported as incomplete coverage. The default account scan still has no YARA pass.

Management

CommandDescription
csm installDeploy config, systemd, auditd rules, logrotate, WHM plugin
csm uninstall [--purge]Remove the executable and CSM-owned service, audit, logrotate, webserver, and ModSecurity integrations. Config, drop-ins, state, logs, signature rules, and quarantine data are preserved by default. --purge removes all CSM-owned data. Operator ModSecurity rules are never removed.
csm baselineFull server scan via the daemon, records current state for change tracking. Dangerous privileged accounts or WHM root tokens can still be reported on first scan. Takes 5-10 min on large servers. Required on first install. Add --confirm when existing history would be cleared. The daemon must be running.
csm rehashRe-record the binary, csm.yaml and conf.d hashes without scanning. Use once after editing restart-required config, changing a drop-in by hand, or replacing the binary; a stale hash makes the next restart refuse to start. It also applies integrity.immutable to the installed binary. csm rehash --fragment <file> re-records hashes only when the named conf.d files are the only change; see conf.d drop-ins for its exit codes. Unknown arguments are refused rather than ignored.
csm statusShow current state, last run, active findings, and automation rollout state. Add --json for the full health snapshot (watchers, severity counts, store health, blocklist size, capabilities, version, hashes, automation).
csm verify-release <key.pem> <file.sig> <file>Verify a release artifact’s Ed25519 signature without OpenSSL; the supported verification path on EL8 and CloudLinux 8.
csm systemd-rootsPrint a validated systemd drop-in granting write access to configured account trees; see custom account roots.
csm privileges [--json] [--markdown]Print every operation that needs privilege beyond reading CSM’s own files, with what it writes and the config key that stops it. Reads nothing from the host, so it answers “what would this do to my server” before installing. See capability matrix.
csm action show <id>Print, as JSON, what the admission ledger recorded for one automatic response: the candidate with its state, reason and outcome, each attempt, and for a response refused because a block already enforced it, that block. <id> is a candidate ID or the action_id of an admission record. Asks the running daemon. See action log.
csm actions [--since <when>] [--op <id>] [--limit N] [--json]Print what CSM did to this host: quarantines, cleans, process kills and firewall changes, with the digest before and after a file change. Reads the log directly, so it answers after a crash. See action log.
csm selftest [--json]Scan a bundle of samples with known verdicts and report what the installed rules catch, including the gaps the bundle records. Reads no account data. See self-test.
csm doctorConfig + integrity + daemon + watchers (including the YARA-X scanning worker, which fails while it is down or keeps crashing) + store sanity check, including service write access for custom account roots. Config checks include the firewall lockout warnings for inbound and outbound policy; the integrity check reports a binary, csm.yaml or conf.d hash mismatch before the next restart refuses to start, with the remedy. On CloudLinux with PHP Shield enabled it also reports whether the Shield event mount has been applied to the cages, names up to five missing cages, and gives per-account remount commands. Unresolved accounts appear as uid:N; resolve their account names before remounting. For more than five missing cages, it also offers a remount of all cages during a maintenance window. On Postfix hosts with Dovecot, the smtp auth logging line warns when Dovecot does not log rejected logins, which SMTP brute-force detection needs, logs their passwords, or its settings cannot be read. The correlation attribution line names the checks whose findings sit in the active set without a hosting owner (WARN) or confirms every eligible finding carries one (OK), with the cumulative count since start either way. The web client addresses line says whether the web server configuration sets no client address rewriting, replaces the logged client address (OK, naming the setting and its file; web findings then get no automatic response from admission), or cannot be read completely (WARN). It applies the loaded web_server: settings before reading the files. The configuration root selects Apache or nginx, including cPanel LiteSpeed’s Apache tree; LiteSpeed XML stays at its standard location. On Stalwart hosts the stalwart API key line says whether Stalwart accepts CSM’s API key from stalwart.api_key_file, and the stalwart setup lines check Stalwart’s settings, read-only, through that key. Without a working key doctor uses an admin login from STALWART_USER and STALWART_PASSWORD when both are set, and otherwise does not check the settings. The key is sent only to an API address on this host whose port belongs to root or Stalwart’s service user; for a remote address doctor uses the admin login only. csm doctor challenge checks challenge public URL, TLS, port gate, webserver snippets, configtest, and the live /challenge/gate endpoint. Add --json for machine-readable output.
csm validateValidate config (--deep for connectivity probes)
csm config show [--no-redact] [--json]Display the redacted config even when callback secrets are set only in the service environment. With --no-redact, a missing required callback secret stops the command before it prints the config. Use --json for JSON output.
csm config schemaPrint a JSON Schema reflected from the Config struct. Use for CI validation of conf.d drop-ins or panel-side editor schemas.
csm config apply-immutabilityReapply integrity.immutable to /opt/csm/csm. Unsupported filesystems or missing tooling produce a warning and a successful exit; permission and other unexpected errors fail. Package transaction hooks use this after upgrades.
csm verifyVerify binary and config integrity
csm versionVersion and build info
csm capabilitiesPrint this build’s feature tokens, one per line; --json prints a JSON array. Needs no running daemon and no config, so an orchestrator can check a token before it writes a key that depends on it. Tokens for live BPF monitors appear only in the daemon’s GET /api/v1/capabilities.
csm incidents ...List, show, and update correlated security incidents (list, show <id>, status <id> <state>, bulk-status). See Incidents.
csm forensic-snapshot <account> --out <archive.tar.gz>Evidence archive for incident handoff (triggers, admins, sessions, file mtimes).
csm webserver-integration <install|upgrade|status|validate|remove>Install, upgrade, or remove the challenge reverse-proxy snippets for the detected web server.
csm pam <install|uninstall|status>Install or remove the pam_csm.so PAM hook (csm pam --help).
csm report enrollGenerate an abuse-reporting node key pair.

csm doctor also lists protection queue health: waiting and running work, losses and lag. A sustained backlog or drop rate fails the named queue check and includes recovery guidance. Queues whose work is best effort warn instead of failing, so they never change the exit status. While the firewall is disabled, queued automatic blocks pass the check, which says the blocks are not applied and whether dry run is on. The same evidence appears in csm status --json and the HTTP status response.

Each started live monitor also gets a backend: <feature> check naming the backend it runs on, such as backend: placement. A monitor on none passes when its detection.<feature>_backend is none and warns otherwise, with a pointer to the startup log.

Backup & restore

CommandDescription
csm backup <path>Bundle csm.yaml, /etc/csm/conf.d/, the state directory and the antispam learning directory into a tar.gz at <path>. Runtime lock files, in-progress export staging, and unconfirmed firewall rollback artifacts are omitted. The daemon must be stopped; the command verifies both the control socket and state lock before reading data.
csm restore <archive> [--force]Validate and stage the complete archive before atomically replacing the live csm.yaml, conf.d, state directory and antispam learning directory; the replaced learning directory is kept aside and restored learning waits for confirmation. Restore rolls back failed replacements, removes stale state entries, and refuses a running or starting daemon. Stop the daemon first.

csm store export / csm store import (below) is the lower-level alternative: tar+zstd, sha256-verified, finer-grained --only= flags. csm backup/restore is the convenience wrapper most operators want.

Restore and csm store import refuse an archive whose admission ledger is behind this host’s once the ledger has started live: the archive’s ledger never started live, or its clock is behind. Restoring it would spend an hour’s block budget twice and replay the live start from older blocks. --only=baseline is refused too when it carries the firewall state, because it puts that file beside the newer ledger; --only=firewall is not. A host whose ledger never started live, or that has none, is not subject to this comparison. A host ledger this build cannot read, including a truncated database, does not refuse, so a restore can repair it. Pass --force to either command to bypass this comparison.

csm store import rejects archives that place csm.db among state files; the database belongs only in bbolt.snapshot. A baseline import carrying firewall/state.json also verifies the snapshot’s checksum when the manifest provides one. --force does not bypass either format or checksum validation. Baseline imports without firewall state still ignore the unused snapshot.

Backup, export, import and restore check each database snapshot before reading or changing it and refuse a damaged one, including one with leaked pages. They make consistent backups, not salvage copies of damaged databases.

csm store import --only=firewall also checks the live database. When an archive contains antispam learning and omits state, restore checks the staged copy of the live database before replacing any files. Both operations refuse a damaged database or a broken database symlink and accept leaked pages in the live database.

Backup and restore use the state lock to exclude daemon access. Restore stages all data beside its destination before replacement, so validation failures leave the live installation unchanged. Uninstall is intentionally non-destructive unless --purge is supplied.

Restore accepts the format produced by csm backup: one tar archive in one gzip member, with no data or padding after the tar end markers and no trailing compressed bytes or additional members. It validates the gzip checksum and length before replacing any destination, and rejects corrupt or truncated archives.

Both commands stream file contents and limit the total uncompressed tar archive to 16 GiB by default, including headers, manifest, and padding. There is no separate limit on a state file. Use --max-bytes <positive byte count> on both commands to raise or lower the budget; restore also accepts older large backups when they fit the selected budget. An over-limit backup fails without replacing an existing archive. Restore keeps its manifest read limited to 64 KiB.

Backup checks space before copying the state database snapshot. Restore checks before each staged file and each replacement copy, including copies onto a different destination filesystem. These checks leave 64 MiB free beyond the file being copied. Allow space for the old installation, the full extracted archive, and a second copy of the restored files during replacement. Space checks cannot reserve storage against other writers; write failures still abort staging before live replacement. If a limit or space check fails, retain the original archive, provision enough storage, and retry with the same or a higher explicit budget.

Hardening

Operator-driven mitigations applied to the host. Run csm harden with no arguments to print the available subcommands on the current host (the audit detects kernel build, panel, and existing mitigations and only offers what’s relevant). Background, full list, and live-detection details: CVE Mitigations.

CommandDescription
csm hardenPrint the hardening menu for this host.
csm harden --copy-failApply the CVE-2026-31431 (Copy Fail) modprobe mitigation: blacklist algif_aead + af_alg, unload them. Refuses on built-in-AF_ALG kernels.
csm harden --copy-fail-seccompApply the CVE-2026-31431 seccomp mitigation: write systemd RestrictAddressFamilies=~AF_ALG drop-ins for LiteSpeed, Apache/Nginx, every PHP-FPM pool, cron, and mail units. The right path on built-in-AF_ALG kernels (typical cPanel/CloudLinux 8).

The web UI’s Server Hardening page can also apply fixes one section at a time. A failing check that has a safe, mechanical fix shows an Auto-fix checkbox. Apply selected fixes changes only the checked items, then re-runs the audit. The page waits for the worker to finish. Only one hardening worker can run at a time, including across daemon restarts. It uses the daemon’s main configuration and drop-in directory. Applying fixes requires root and systemd-run.

  • SSH MaxAuthTries 4, X11Forwarding no and UseDNS no go into a CSM block at the top of sshd_config. The file is checked with sshd -t before sshd reloads, and restored if the check fails. The previous file is kept as sshd_config.csm-bak.
  • Kernel settings are set with sysctl -w and saved to /etc/sysctl.d/99-zz-csm-hardening.conf. A missing drop-in directory stops the fix before kernel settings change.
  • /tmp and /var/tmp become 1777 owned by root. /etc/shadow becomes 0600, or 0640 when the system shadow group owns it.
  • WHM tweak settings are changed through whmapi1 set_tweaksetting, with 200 for max emails per hour. Compilers are restricted to root, and cPanel updates are set to daily.
  • PHP expose_php, enable_dl, upload progress and fatal error backtraces are turned off in zz-csm-hardening.ini in each build’s scan directory, and in every SAPI’s conf.d on Debian and Ubuntu. If a Debian or Ubuntu web SAPI cannot be updated, the CLI setting stays pending for retry. Linked or unreadable SAPI directories require manual changes. PHP-FPM restarts, Apache reloads when mod_php was changed, and LiteSpeed receives a fresh restart request on each apply.

SSH port, password and root login settings, disable_functions, allow_url_fopen, and the mail, web server and firewall checks stay manual.

Remediation

CommandDescription
csm clean <path>Clean infected PHP file (backs up original)
csm db-clean --option <account> <option_name> [--preview]Sanitize malicious WordPress option values (e.g. injected siteurl / home)
csm db-clean --revoke-user <account> <user_id> [--demote] [--preview]Revoke or demote a compromised WordPress admin and invalidate their sessions
csm db-clean --delete-spam <account> [--preview]Delete published posts whose title or content matches a high-confidence spam keyword. Matches are confirmed on a Unicode-aware word boundary, so a post is never removed for containing a keyword inside a longer word. Keywords that also occur in ordinary writing are reported by scans but never deleted.
csm db-clean --drop-object <account> <schema> <type> <name> [--preview]Drop a MySQL trigger / event / stored procedure / stored function, capturing its CREATE SQL into the db_object_backups bbolt bucket first. <type> must be trigger, event, procedure, or function. <schema> must match a database discovered for <account>. Daemon must be stopped.
csm virtual-patch [--apply]Re-scan web roots and preview reversible access-file deny rules. Requires root and manual or auto mode; a timed-out partial scan applies only findings already confirmed reachable and exits nonzero.
csm enable --php-shieldEnable PHP runtime protection
csm disable --php-shieldDisable PHP runtime protection

Installing or enabling PHP Shield creates /opt/csm/shield.conf.php only when it is missing, so your edits survive. Update its allowed IPs by hand after changing infra_ips. Disabling keeps the file; uninstalling removes it. A PHP pool with open_basedir must allow /opt/csm/, not just the shield script, or the shield ignores /opt/csm/shield.conf.php and uses its defaults. A .csm-shield-disable file only works where open_basedir allows the account’s home folder.

State database

CommandDescription
csm store compactReclaim unused space in the bbolt state file (atomic rename over the live DB). Requires the daemon to be stopped (systemctl stop csm) because bbolt holds an exclusive file lock while running.
csm store compact --previewSnapshot into a temp file next to the live DB and print src/dst sizes without replacing anything. Use to estimate reclaim before scheduling a maintenance window.
csm store export <path>Write a tar+zstd backup containing the bbolt store, the state directory, and the signature-rules cache. Private in-progress staging files and unconfirmed firewall rollback artifacts are omitted. A sibling <path>.sha256 companion file holds the archive hash for verification. Daemon must be running.
csm store import <path>Restore from a backup archive. Daemon must be stopped. Default restores everything; --only=baseline restores only state JSON files (file hashes); --only=firewall merges the firewall buckets into the existing bbolt in one transaction, where a durable firewall journal and its snapshot in the archive replace the target’s, and refuses while either side has a durable firewall action unresolved or its audit undelivered; --force-platform-mismatch allows restoring an archive captured on a different OS / panel / web server; --force allows an admission ledger behind this host’s live one (see Backup & restore).
csm store reset-bot-verifyDrop cached bot PTR verification results and missing-PTR records so the next scan re-runs reverse DNS checks. Requires the daemon to be stopped because bbolt holds an exclusive file lock while running.
csm export --since <when>Dump audit-log events for SIEM backfill. <when> is RFC 3339 (2026-04-01T00:00:00Z) or a duration relative to now (24h, 7d). One JSON event per line on stdout, in the same v=1 schema the live audit_log sinks emit. Pipe to a file or directly into a log shipper. Daemon must be running.

Updates

CommandDescription
csm update-rulesDownload latest signature rules
csm update-geoipUpdate MaxMind GeoLite2 databases
csm update-bot-rangesRefresh built-in AI-crawler IP ranges from vendor feeds

PHP-relay (mail abuse, cPanel only)

Operator controls for the email PHP-relay detector. Talks to the daemon’s control socket; the daemon must be running. See Real-time detection for what the detector fires on, and Auto-response for the freeze action.

CommandDescription
csm phprelay statusPrint the detector’s current state as JSON: enabled, platform, effective dry-run + source (runtime/bbolt/csm.yaml), Path 2b effective account limit, scripts/IPs/accounts tracked, msgID-index size, active ignores. Use to confirm the watcher is wired on a fresh install.
csm phprelay ignore-script <scriptKey> [--for-hours N] [--persist] [--reason ...]Suppress every path for a host:/path scriptKey. Default TTL 168h (7d). --persist writes to the bbolt phprelay:ignore bucket so the suppression survives daemon restarts; without it the entry is in-memory only. <scriptKey> is the value the daemon prints in email_php_relay_abuse and email_php_form_relay_script findings (e.g. shop.example.com:/wp-admin/admin-ajax.php).
csm phprelay unignore <scriptKey> [--persist]Remove an active ignore. --persist also deletes the bbolt row.
csm phprelay ignore-listList all active ignores as JSON: scriptKey, expiresAt, addedBy, reason.
csm phprelay dry-run on|off|reset [--persist]Override the auto-freeze dry-run state at runtime. on = freeze findings emitted but no exim -Mf runs; off = live freezes; reset clears the runtime override and falls back to bbolt or csm.yaml. Precedence: runtime > bbolt > yaml. --persist writes the on/off choice to the bbolt phprelay:settings bucket so it survives restarts; on reset --persist the bbolt row is also deleted.
csm phprelay thaw <msgID>Manually thaw a frozen Exim message. Wraps exim -Mt with msgID validation (rejects anything that isn’t [A-Za-z0-9-]{16,32}) and writes a thaw entry to the auto-freeze JSONL audit at /var/log/csm/php_relay_audit.jsonl.

Antispam learning (cPanel only)

Teaches or corrects learning from Spam folders. Runs as root against the running daemon while learning is enabled; observe mode refuses it.

CommandDescription
csm antispam learn --spam <file>...Learn each saved message as spam; every cluster it matches gains one operator mark.
csm antispam learn --forget <file>...Remove the learned clusters each message matches, including every account’s contribution to them.
csm antispam learn --status [<request-id>...]Show what the daemon recorded for request IDs, including clusters still publishing; without an ID, list unfinished requests.
csm antispam learn --fresh-namespaceWith learning disabled, move the learning keys and stored messages aside and empty the learning state; the next enable starts a new namespace.

Each file gets a request ID, printed before it is sent; csm actions --op integrate.antispam_learn --json shows its recorded outcome and full digests. A lost or truncated reply is retried once with the same ID and snapshot. The exit status is 0 only when every file was learned, already learned, forgotten or matched nothing.

Stalwart

Prepares a Stalwart mail server for CSM through its admin API. See Preparing Stalwart.

CommandDescription
csm stalwart setup [--dry-run] [--password-file PATH | --password-stdin] [--url URL] [--account-domain NAME]Make Stalwart log failed logins and the mail its spam filter rejects or discards, end its bans with CSM’s blocks, and take webmail client addresses from the web server behind CSM’s port lock. Create the role CSM response, the service account csm-response and its API key, which works only from this host and is written to stalwart.api_key_file; the account has no password and can receive mail. A role that lacks CSM’s permissions is reset to CSM’s list, dropping permissions added by hand. Setup refuses an existing csm-response account that has a password or another sign-in credential, has sign-ins stopped, or holds keys without CSM’s role. --account-domain names the account’s domain on a server with several. On a server with tenant administrators, use a domain only the server administrator controls, since whoever can edit the account could give it a password. A key Stalwart accepts is kept while it is the account’s only key; to rotate it, delete the key file and run setup. The key steps run only when the API address is on this host. Also prints whether the spam filter is on and its scores. Adds the spam rule CSM_IP, which scores senders on CSM’s address list so more of their mail goes to Junk, or is rejected or discarded at the scores set in Stalwart; it is the one step that changes how mail is handled, and switching the rule off in Stalwart stops it. An unreadable spam filter status warns without stopping setup. Prints each step as name: state: detail. The admin login comes from STALWART_USER and STALWART_PASSWORD, or the first line of --password-file. --password-stdin reads the password from the first line of stdin instead, with the user from STALWART_USER or admin. On a terminal, setup prompts for missing values and hides password input even with --password-stdin. When you enter both values, the username is hidden too. Press Enter at the user prompt for admin. Ctrl-D at an empty prompt cancels setup. Without a terminal, setup reads stdin only with --password-stdin. --url overrides stalwart.api_url. Before sending the login to an address on this host it checks that root or Stalwart’s service user holds that port and owns each connected peer; unreadable ownership records stop the login. Exits 1 when a setting is unsafe, other Stalwart settings cannot be read, the key file cannot be used or checked, or a change failed. Exits 2 on a usage or config error, an API address that is refused, a missing or refused login, a port held by another user, or a host that is not a Stalwart host.

Firewall

See Firewall for the full reference.

csm firewall status
csm firewall deny <ip> [reason]
csm firewall allow <ip> [reason]
csm firewall tempban <ip> <dur> [reason]
csm firewall deny-subnet <cidr> [reason]
csm firewall grep <pattern>
csm firewall flush
csm firewall apply-confirmed <minutes>
csm firewall confirm
csm firewall restart
csm firewall rollback status|confirm|revert
# ...

csm firewall apply-confirmed and csm firewall restart refuse a pending CLI confirmation window and say whether waiting for its rollback still helps. If startup cannot read its saved configuration, rollback restores the kernel ruleset and state and the window stays pending; once that restore is recorded, later restarts keep the blocks added since and apply the configured settings. csm status and csm doctor show the window. Inspect the current firewall before csm firewall confirm, which accepts the current ruleset and engine configuration; it does not recover unreadable settings. Or repair the saved configuration and restart the daemon, which restores it and rebuilds the ruleset with the current blocks; removing it does not end a recorded window, even one whose marker was empty or corrupt. See CLI confirmation window. The separate Web UI config timer uses csm firewall rollback.

Real-Time Detection

CSM detects threats in under 2 seconds using kernel and log watchers running inside the daemon: a fanotify file monitor, inotify tailers on auth/access/mail/FTP logs and the Exim spool, and a PAM brute-force listener.

fanotify File Monitor (< 1 second)

Monitors the filesystems containing /home, /tmp, /dev/shm, /var/tmp, configured account_roots, and detected cPanel document roots.

A filesystem mark covers the whole filesystem, including all its bind mounts, which makes writes inside CloudLinux CageFS visible. Roots on that filesystem share one mark. A root being a mount point does not restrict a filesystem mark to that path: symlinks and bind mounts can name only a subtree. On kernels that fall back to mount scope, the mark covers the whole containing mount and misses writes through other bind mounts. The startup log describes these scopes and which roots share a filesystem mark, and says when a watch root sits on the same filesystem as /, which is when the mark raises an event for every write on the machine; failures to inspect or mark a root are reported even when other roots succeed. Missing roots are skipped.

The daemon filters delivered file events before queueing them for analysis. In the shared temporary trees (/tmp, /var/tmp, /dev/shm) that decision uses the event descriptor. Non-directory objects with executable permission bits are queued, as are watched user crontabs, PHP source, configuration files such as .htaccess, .user.ini or php.ini, known webshell names, files under .config directories, and images inside hosted trees. Writes needed for self-deleting-file tracking are also queued. Other writes are dropped without analysis. Files retained for tracking keep their normal analyzer route, including suppression and WordPress checksum checks. Staging names written by an atomic save are resolved first, so a configuration file saved through one is still analysed. A descriptor that cannot be inspected is analysed rather than skipped. csm_fanotify_events_total, csm_fanotify_events_admitted_total, and csm_fanotify_events_dropped_total distinguish completed admission decisions, queued events, and events lost to a full analyzer queue. Subtract both admitted and dropped events from the received total to count events rejected before queue admission, including those whose file path could not be resolved.

Atomic-save files enter the same bounded worker queue as ordinary writes. The watcher receives close-write events, not create events. Scans read the original event descriptor even if the file has been renamed, replaced, or deleted before analysis. No rename event is needed to inspect those bytes. Repeated findings use the normal alert cooldown, and queue overflow schedules a directory rescan of files that remain on disk. That recovery runs one pass at a time, under a time budget and with a minimum gap between passes. Unfinished directories stay queued and retry even after new drops stop. Recovery resumes within a directory and retains its original scan window so deferred files do not age out while waiting. Kernel notification loss still relies on the next deep scan. Shutdown drains the queue under a fixed time budget: events still waiting when it expires are released without being scanned; files still on disk are covered by the next deep scan. Both budgets are checked between scans, so a scan or directory read already in progress must still finish before shutdown completes. A file renamed or hard-linked into a watched tree, or a directory moved into one, raises no close-write event at its new path. The placement monitor covers those when a placement backend runs; otherwise they are left to the deep scan.

Realtime YARA scans hand the worker a sealed memory-backed copy of the captured bytes, whatever their size, without reopening the event path for scanning. Alert payload details describe that snapshot. When it is the whole file, the finding keeps the snapshot’s fingerprint. For partial reads, CSM reopens the path to fingerprint the whole file when preparing the finding; unreadable files and files above the fingerprint limit receive no fingerprint. Re-check compares whole-file fingerprints, so an unchanged file can clear when YARA no longer matches it. Content outside the read window remains subject to the scheduled scan limits. A failed scan raises yara_realtime_scan_error.

For WordPress atomic saves, the intended basename can select a core or plugin checksum entry. Only a match against the complete event-file content verifies the file; missing checksums, partial content, and modifications proceed through normal detection. The finding retains the actual event path.

A WordPress update is unpacked under wp-content/upgrade/ before it is moved into place, and each staged PHP file is judged by hash rather than by location. A file that matches the official wordpress.org checksum for its package version is stock and skips detection like an installed stock file. A file whose checksums are still being fetched is content-scanned now and compared once they land, even if the package header was written late or WordPress has already moved the tree into place. An installed plugin header can identify the release only when its directory is the original staging directory moved into place, or when the installed path is still the very file the staged event wrote, so a stock file first examined after the move skips detection like any other. The installed plugin directory must remain the same directory during that check and cannot itself be a symlink. Its identity also connects earlier queued files from the same tree, including when the later file is already verified. Copies and unrelated trees do not establish that connection; every retained digest still needs to match the official release. A core update copies files, so the installed version.php identifies the release only if it was replaced or rewritten after the staged file was seen; a borrowed release still requires the file’s bytes to be the official ones for that path. A tree removed before its release could be identified gets a package warning, not file-by-file comparisons against the old release. This covers refused updates (for example when the new release needs a newer PHP) and plugin copies whose staging identity was never captured; the warning does not establish whether installation succeeded. Reused staging paths follow the normal alert cooldown rather than suppressing later unidentified uploads. A file the official package does not ship gets its own warning. An event identified at its installed location keeps that location while it exists, even if another update reuses the staging name. Other staged events name the installed path when it exists. Themes, packages not published on wordpress.org, a full verification queue, and a package whose checksums do not arrive within 60 seconds raise one warning per staging directory. WordPress picks a new staging directory for every upload. Once the package type and version are known, these warnings are identified by site, package type, name, version and reason. Uploading the same declared release to the same site again is a repeat. It is emailed again only after 24 hours without a report, and history follows the normal 24-hour reminder. This groups checksum-availability warnings; it does not establish that the package contents are unchanged or safe. Incomplete headers keep an upload-specific warning, and a later change of type, version or reason can raise a new warning in the same directory. Per-file checksum warnings repeat across uploads only when their package identity, relative path and complete digest match; files that could not be hashed keep their upload-specific warning. Content findings inside staging keep their own per-file identity and normal alert cooldowns; package-warning dismissal does not dismiss them.

Location warnings for PHP in sensitive WordPress directories are suppressed only for recognized inert content, including comment-only stubs and literal translation or version data. PHP attributes are executable syntax, not comments, and do not qualify a file for that suppression. Content scans still run.

The head-window eval/decoder check ignores PHP comments and inline HTML. String boundaries and legacy line endings preserve the code that follows them. Tail windows retain raw content because they can begin inside a literal whose opening delimiter is outside the window.

A Critical obfuscated_php_realtime, webshell_content_realtime or php_dropper_realtime alert whose file has not already had a file response queues the scheduled content analysis of that file, on one background worker so slow storage cannot delay other alerts. A Critical result goes through normal dispatch and the scheduled response, under quarantine_files and the file response budget. Links in any path component, special files and a file that changed after analysis are refused. A full queue is logged; the original alert stays.

Detects:

  • Executables in shared temporary directories are reported even when named .htaccess, .user.ini, or php.ini, including atomic-save names. Root-owned files with panel maintenance ancestry, package-manager ancestry during an active package window, or verified live root ancestry of an initramfs build, receive Warning. Initramfs provenance requires both real and effective root credentials from the writer through the builder; cached names or unknown credentials do not qualify. The writer’s ancestry is read when the write is seen. Build tools often exit before then; such a file receives the same Warning only when a writer with proven ancestry recently wrote an executable into the same work directory, or a root process with proven ancestry is working inside it, and the executable and every directory below the temporary root are owned by root and cannot be written by other users. An occupant’s actual working directory and every directory back to the work directory must also be root-owned and not writable by other users, without crossing a mount. Occupant scans are spaced per work directory; when scan capacity is full, findings without other proof stay Critical. The vouch is tied to the open file and its directory placement; symlinks, mounts below the temporary root, renamed or replaced files and directories, and a file that was not owned by root when it was written cannot carry that proof. A file the build itself has already removed keeps it. Its lifetime starts when the proven write was seen, and exited writers cannot extend it. Read errors and a known non-root process in the ancestry do not qualify as an exited root writer. The finding of an exited writer with no such proof says so and stays Critical, as does any writer that is still running and proves nothing. Those warnings are grouped: one finding stands for a work directory while maintenance writes into it. Files directly in a shared temporary directory retain separate warnings. Other temporary executable findings remain Critical and are never grouped; an earlier maintenance warning cannot suppress them. Configuration content checks still run, and operator path suppressions still apply.
  • Webshell creation (PHP files in web directories)
  • Self-deleting droppers: a PHP or executable created under a document root and unlinked within thresholds.dropper_unlink_ttl_sec (default 300s), the loader technique that creates a rogue admin then erases itself before any scan. A file whose exact bytes WordPress moved or copied into place before removing the original is not reported: plugin, theme and core packages, language packs, and the version file a core update reads first. Language-pack and version-probe copies also require a complete, stable data-only snapshot with no suspicious content or earlier unsafe write. A core update that stops after reading its version file leaves the old release installed, so no installed copy matches; that version file is not reported when its bytes equal the wordpress.org checksum for the release and locale it declares. Core checksum downloads start in the background when capacity permits. Plugin downloads do not consume core checksum capacity or share its admission limit. Concurrent requests for a release reuse its download and cached result; a file still unverified at the deletion probe is reported. Package headers are read only from regular files, never blocking on an account-writable named pipe or device. Other executable files do not qualify for this copy exception. The script Really Simple Security copies into uploads to test whether PHP runs there, then deletes, is not reported when its content is byte for byte the shipped script; any other content under that name is. Elementor Safe Mode copies its loader into mu-plugins while enabled and deletes it when switched off; that removal is not reported when the loader’s bytes equal the wordpress.org checksum for the same file in the Elementor release the same site had installed when the loader was written. When those checksums cannot be consulted, a loader byte for byte identical to a complete, stable snapshot of the installed plugin’s own file, reached without symlinks, is reported as Warning and the finding names that file. The same package proof applies when a different file replaced the loader’s path. Other capability-probe scripts cannot exempt executable content in this loader. A loader the checksums reject, or whose earlier payload was rewritten, stays Critical even when its containing directory was removed. A core update unpacks the whole release but installs only what changed and never overwrites the bundled themes and plugins; a staged file it deletes that way is not reported when its bytes equal the wordpress.org checksum for that path in the release now installed. The checksum must describe the same complete, stable snapshot as the retained content. Earlier nonempty content must also be known and unchanged, including the first close-write snapshot; later official bytes or an identical installed copy cannot erase that evidence. The same history rule applies to every staged core, plugin and theme file and to the Safe Mode loader: a file whose content changed after it was first written is reported even when WordPress then moves it into place, an identical copy is installed, or its last bytes are official. Repeated analysis of the same snapshot does not count as another write. A single snapshot whose complete bytes could not be read can still match a move by file identity; a second nonempty snapshot leaves the unknown earlier bytes unaccounted for. Files too large for a complete snapshot retain their leading bytes for content checks. Other files removed from staging without an identical installed copy are still reported. Upgrade staging, atomic-save temp files, template compile caches, BackWPup job-state files (the running job’s JSON and its folder list, written behind a PHP comment, including a captured head cut inside a CRLF line ending), a path taken over by a newer file, and a file whose original directory was removed are recognized and reported at a lower severity; files removed together with the directory holding them, such as a language pack WordPress unpacked and discarded, are reported once for that directory; a create/delete burst collapses into one lower-severity notice. Grouping is recalculated after ignored paths are removed: remaining files can form directory groups, while a lone remaining file is reported on its own. Off with thresholds.dropper_detection: false. Candidate tracking and findings held for aggregation each accept up to 16,384 entries; additional work is refused without evicting older evidence. Status and doctor report refusals, exhausted probes, overdue work and stalled processing through protection queue health.
  • Dropper admission and pending findings follow the live suppressions.ignore_paths list after reload. Intentionally suppressed candidates consume no tracker capacity; losses of eligible candidates still raise the capacity warning.
  • PHP in uploads, languages, upgrade directories
  • PHP in .ssh, .cpanel, .gnupg or .cagefs directories, or in the mail directory of an account home, including cPanel accounts on further home partitions such as /home2 (critical escalation). Account mail locations ignore redundant separators but keep Linux directory-name case sensitivity. Directories named mail inside applications or document roots do not count; their PHP content is still scanned.
  • Executable drops in .config
  • .htaccess injection and tampering (auto_prepend, eval/base64 handlers, CGI execution remaps, and ModSecurity disablement). Handler abuse keeps its own finding name. These findings never quarantine the file; automatic cleaning waits for a deep scan and requires auto_response.clean_htaccess.
  • .user.ini tampering and php.ini tampering under configured or detected web roots
  • Obfuscated PHP (encoded, packed, concatenated)
  • Fragmented base64 evasion ($a="base"; $b="64_decode" – function name split across variables)
  • Concatenation payloads (hundreds of $z .= "xxxx" lines with eval at end)
  • Tail scanning: payloads appended to the end of large legitimate PHP files (beyond the 32KB head window)
  • CGI backdoors: Perl, Python, Bash, Ruby scripts in web directories (e.g., LEVIATHAN toolkit)
  • SEO spam: gambling/togel dofollow link injection in PHP/HTML files
  • Phishing pages and credential harvest logs
  • Phishing kit ZIP archives
  • PHP carried inside a file served as an image. Image writes under an account or configured document root are inspected, including hosted .config directories and roots located in temporary directories. Images up to 128 KiB are inspected in one piece; larger files get a 64 KiB head and a 64 KiB tail scan. Payloads outside those windows require a deep scan, subject to thresholds.full_scan_max_file_mb. A PHP opening tag alone is not enough – a screenshot quoting one in its description chunk stays quiet – and a file that only wears an image name while holding PHP source is reported the same way. Finding: php_in_image_realtime.
  • A PHP file that includes or requires an image, archive or other non-executable file while also reading request input. That pairing is the loader half of the technique above: the payload lives in the picture and the one-line loader elsewhere. Literal targets ending an include statement, concatenated paths, paths held in a local, encoded paths and suppressed @include statements match in either statement order. Ordinary templating that pulls in .html, .tpl, .txt or .svg partials is outside the extension-based rule. Encoded targets still count because the source hides their extension.
  • YAML signature matches (PHP, HTML, .htaccess, .user.ini, php.ini)
  • YARA-X rule matches (if built with -tags yara)

Self-deleting file tracking requires a close-write observation and a recent filesystem birth time. Filesystems that do not provide birth time cannot support this detector; ordinary realtime content scanning still runs. Completed upload execution probes remain tracked until the deletion check. Earlier unsafe or uncertain content remains reportable even when a later close-write contains a harmless probe. Image writes use the existing notification-only fanotify instance, its 16,384 event queue and 4-16 analyzer workers. They create no permission-event holds and do not enter the mail scanner. Thumbnail and WebP bursts fill the same queue as other writes: excess events are closed immediately and their parent directories enter the existing capped recovery tracker. Recovery rescans recent surviving files; kernel queue loss and exhausted recovery coverage still rely on the next deep scan. Queue health and content-read truncation remain visible through the existing metrics. Mail permission holds retain their separate hold budget and watchdog.

Both the real-time and the scheduled WordPress admin-creation signature require an administrator role token plus literal or request-derived credentials, and both accept the same ASCII whitespace, including the vertical tab. An importer that creates users with a generated password does not trigger either one just for reading a login from an import form.

Two divergences between the engines are known and left in place. Go folds the Unicode long s into ASCII s and YARA’s nocase does not, so a token spelled with it can match in real time where a scan declines. Bounded expressions also count differently: the real-time engine counts characters and the scheduled one counts bytes. Neither can occur in real PHP, and both belong in the scanner’s regex compilation rather than in hand-written escapes inside every rule. Strictness parity is not enforced across the rulesets: the parity check compares rule names, not how strict each side is.

Complete blank files are excluded from dropper alerts after a close-write observation. Any content or metadata change during the read, including an unlink, leaves the snapshot uncertain. WordPress data and its checksum are read within the same stat interval. A later complete close-write snapshot cannot rule out code that ran during an earlier unstable read. That uncertainty and code seen in any read remain attached to the tracked file. Concurrent firewall-log writes can therefore still produce a warning when the log is replaced, or a critical finding when it is deleted.

Core-release checksum downloads are non-blocking and bounded, including their retry timers. Repeated misses and streams of distinct release names share an admission budget. Release identifiers longer than 64 bytes are rejected before retention or lookup. When that budget is exhausted, cached checksums remain usable; an uncached file stays unverified and receives the normal dropper assessment. A later observation can request checksums again after the budget recovers.

PHP files are also excluded when their first statement stops the interpreter (exit, die, or __halt_compiler, with at most a plain literal argument and no preceding comment). Plugins keep state and firewall data in files of that shape and rewrite them constantly, and the bytes behind the terminator are never compiled. The exemption is refused when those trailing bytes are made only of transport-encoding alphabet: an operator who turns on PHP source conversion can have a file decoded before it is tokenized, which would make an encoded tail the program and this header padding. Comment-bearing PHP remains eligible for the same reason – a comment’s tokens depend on the interpreter’s source encoding. Executable-mode files are not judged by PHP compilation rules, because a shell may read them instead. Content or signature findings and previously observed code always override inert-content filtering.

Features:

  • Per-path alert deduplication (30s cooldown)
  • Process info enrichment (PID, command, UID)
  • Auto-quarantine on high-confidence realtime signature matches (category, size, entropy, and hex/execution validation)
  • Auto-quarantine on confirmed PHP request-to-execution flows and executable packed payloads, using the stable file version that was inspected

Files Moved or Linked Into a Site

A file written elsewhere and then renamed or hard-linked into a site raises no close-write event at its new path. The placement monitor reports these files. Each one goes through the same admission, analysis and response as a write at that path, including quarantine of confirmed shells, whatever the file’s age. The monitor rejects a symlink at the final path component and verifies the file’s identity before analysis. With bpf, that identity comes from the kernel event. With legacy, the event identifies the parent directory and entry name; CSM records the file’s identity when it resolves that entry. A replacement made before that lookup can be analyzed, but a replacement made after it fails the identity check. The event keeps the placing process’s PID for restore detection; legacy cannot prove that process placed a replacement found during lookup.

detection.placement_backend selects how the monitor sees placements. It takes effect after a restart.

  • bpf uses BPF LSM rename and link hooks. It sees renames, including both sides of a rename exchange, hard links, including a temporary file published with linkat, and directories moved in. It needs a BPF-tagged build and a kernel that can attach BPF LSM programs. Chosen explicitly, it has no fallback.
  • legacy uses a second fanotify group that reports directory entries, on kernel 5.9 or newer. It sees files and directories moved in, and new entries, which include hard links and temporary files published with linkat. A new file that its own close-write already analyzed is not analyzed twice.
  • auto (default) tries bpf, then legacy, then runs without a backend.
  • none turns the monitor off. Placed files are left to the deep scan.

Without a backend the daemon logs a startup warning that files moved or linked into watched trees are checked only by the deep scan. The status response names the selection under backends.placement (bpf, legacy or none), and csm doctor shows it as the backend: placement check. That check warns on none unless the config chose it. GET /api/v1/capabilities lists bpf-file-placement while the BPF backend runs. A failed start raises bpf_unavailable, as for the other BPF monitors. It is High when an explicit bpf choice cannot start, or when an explicit legacy choice or auto fails for an unexpected reason, and a Warning when auto falls back to legacy. A kernel that supports neither backend under auto, or lacks the legacy backend when it is chosen explicitly, shows none without a finding.

Placements that do not fit the monitor’s queue go to the same recovery as a full analyzer queue: CSM rescans the directory each file landed in, whatever the files’ age, and queues a moved-in directory for its walk. BPF recovery allows the rename or link to finish before completing a rescan or starting a walk, and covers placements in the watched trees only: a file or directory inside a watch root, or a directory that holds one. Recovery queues remain bounded; evictions and unfinished work count as drops.

When a watched filesystem has several whole-filesystem mounts, BPF checks each watched location separately. Admission uses that location’s name, and a child mount hiding one view does not prevent checks through another view.

A directory moved into a watched tree is walked, because its files raised no events of their own. Each file gets the same identity check and analysis as a placed file. The walk never follows symlinks, runs on its own worker, waits for analyzer room so live writes keep priority, and works through a large tree in bounded passes. When realtime cannot finish a directory, because it holds too many entries, too many moved-in directories are waiting, the analyzers stay busy for a whole pass, or entries cannot be read, it raises one High placed_dir_incomplete_realtime naming the directory. The next deep scan covers the files it did not check.

placement_unresolved_realtime reports what the monitor cannot see, with either backend. It is High for a moved or linked file whose path realtime could not resolve, such as a path deeper or longer than realtime protection follows or a directory the legacy backend could not open, and for a watch root the running backend cannot see. Unresolved files repeat the finding at most once per alert cooldown for each device. A file or directory removed before realtime reached it raises nothing. One Warning per start names the roots on a filesystem this kernel cannot watch for placements. The deep scan covers those files. A kernel queue overflow in the legacy backend raises fanotify_kernel_overflow.

A PHP file saved under an atomic-save temporary name and then renamed to its final name is not scanned twice. A recent clean scan is reused only when the file’s content, size, modification time, permissions and owner match that scan, its final name gets the same analysis, and no setting or rule changed since. Executable files and files too large for a complete read are always analyzed. csm_fanotify_placement_skipped_total counts those reuses, and csm_fanotify_placement_unverified_total counts placements that failed the identity check. Queue health for the monitor appears under bpf.placement and placement.dir_walk in protection queue health.

Sensitive System File Watcher

Tracks a fixed set of system-configuration paths: the account and credential databases, sudoers and its drop-in directory, the SSH daemon config and its drop-in directory, the system cron drop-in directories, and per-user crontabs. The set is not operator-configurable – a path an attacker knows is excluded is a free landing pad.

The backend is chosen by detection.sensitive_files_backend. The bpf backend attaches an LSM hook that catches writes as they happen, and re-expands the watchset every detection.sensitive_files_poll_interval (default 5m). The legacy backend content-hashes the watchset on the same interval.

Both backends key on the path, not the inode. Most tools replace a config file by writing a temporary file and renaming it over the target, which gives the path a new inode and hides the write from the LSM hook. The BPF refresh therefore compares content and security metadata per path: a rename-over is reported as a change on the existing file, only a path no previous refresh had seen is reported as newly appeared, and a regular-file rewrite that leaves bytes, permissions, and ownership unchanged is not reported. It also tracks symlink targets, so retargeting a watched path remains visible even when both targets contain the same bytes. Non-regular entries are tracked by path identity without opening them for content reads. The legacy poller compares content digests by path.

Findings are sensitive_file_modified. A write finding successfully delivered by the LSM hook wins only while the path still has the exact state captured for that finding. The refresh reports a later rename-over separately, retries a refresh finding if the alert queue was full, and always evaluates the bytes used for its digest rather than a later read. Writes inside a package-manager window, or by a process whose ancestor is a package manager or the control panel’s own maintenance tooling, are demoted to Warning unless a cron payload carries obvious persistence tokens. This veto applies to live writes as well as added and modified cron files found by polling. It includes the shared crontab persistence patterns and their bounded base64 inspection, in addition to suspicious shell fragments. A modified file in /etc/cron.d found by the refresh, the legacy poller or the scheduled cron check is also demoted when the only difference from the version CSM saw last is the minute and hour of jobs that already ran once a day at a fixed time, which is how cPanel reschedules its own jobs every night. A changed command, argument, user, environment line, job list or line order, a job that now runs more often or on other days, and a file CSM has not seen before all keep the finding High, and all three paths also refuse schedule-based demotion when the file’s ownership, mode or symlink target changed in the same interval or could not be checked. Files with randomized calendar fields do not qualify: reloading them can choose new execution days, including for jobs whose bytes stayed unchanged. Persisted schedule evidence must belong to the recorded content baseline; an interrupted update cannot justify demotion. Control-panel ancestry is read from the executable a parent process actually runs, not from the name it reports, and the executable must be a root-owned file inside the panel’s own directory that no other user can write. Cached executable evidence must come from a resolved procfs link; an exec-event filename or script argument does not qualify. Ancestry is available on every Linux build before monitoring starts. A cached control-panel hit avoids a live ancestry walk; otherwise CSM also reads procfs and keeps cached package-manager evidence. CSM’s own managed writes are suppressed by content, not by path.

inotify Log Watchers (~2 seconds)

Tails auth, access, and mail logs in real-time. The exact file paths are chosen per platform at daemon startup – see the platform: ... line in the daemon log.

LogPlatformsWhat it detects
cPanel session log (/usr/local/cpanel/logs/session_log)cPanel onlyLogins from non-infra IPs, password changes, File Manager uploads, sessions ended for security token failures
cPanel access log (/usr/local/cpanel/logs/access_log)cPanel onlycPanel-API auth patterns
Auth logAllSSH logins and failures. /var/log/auth.log on Debian/Ubuntu, /var/log/secure on RHEL family and cPanel
Exim mainlog (/var/log/exim_mainlog)cPanel; non-cPanel when the file existsMail anomalies, queue issues, SMTP brute force, probe abuse, and cloud relay abuse
Apache/LiteSpeed/Nginx access logAllWordPress brute force (wp-login.php, xmlrpc.php), real-time. Paths: /var/log/apache2/access.log (Debian), /var/log/httpd/access_log (RHEL), /var/log/nginx/access.log (Nginx), /usr/local/apache/logs/access_log, then /var/log/apache2/access_log and /etc/apache2/logs/access_log (cPanel, also written by LiteSpeed). Automatic candidates exclude the cPanel access log
Mail log (platform file or journal)All hosts with Postfix/Dovecot logsIMAP/POP3/ManageSieve account compromise and mail brute-force; SMTP brute force on Postfix hosts with Dovecot
FTP log (/var/log/messages)cPanel onlyFTP logins and failures
ModSecurity error logAll (if ModSec installed)WAF blocks and attacks. Auto-discovered from the detected web server
Nginx error log (/var/log/nginx/error.log)Nginx hostsGeneral web errors, ModSecurity denies

web_server.access_logs replaces the automatic web log candidates. The realtime watcher selects the first existing log at startup. If none exists, it retries every candidate in order until one can be opened, then watches that path until restart. The periodic WordPress check uses the first candidate with lines and excludes its symlink aliases from the per-vhost scan; other candidates can still be scanned when a per-vhost glob matches them. Reputation collection scans all listed web logs and deduplicates addresses, while the cPanel log remains a separate source for panel checks.

cPanel session log readers match fields by position, so login names and address lists cannot supply the service, address, event or account. Scheduled and realtime login alerts skip loopback and non-address origins. API-created sessions never count as direct logins for multi-address or hijack correlation, and a login that names its method twice counts as a direct login. A password change entry with an unreadable header or timestamp cannot suppress a shadow-file alert.

A cPanel API authentication failure is a 401 or 403 answer to a UAPI, JSON API or XML API call, a WHM DNS cluster call, or another endpoint that accepts API tokens (WebSocket, account transfer, WP Toolkit, Nova), percent-encoded or not. Page requests on a session URL are not API calls and are not counted. A failure on a session URL (/cpsess<token>/...) waits a few seconds for the session log. A browser tab left open after the account logged in again elsewhere still sends its old URL token, and cPanel ends that session after a few token failures. When the session log records that for the same address, the failures that name the session’s account, and the tab’s later requests that reuse its URL token, are not reported: nothing guessed a credential. Any other failure, including one from another client behind the same address, is reported as before. API calls with a token or password carry no session URL and are reported at once. The periodic api_auth_failures check counts failures the same way.

Password-change suppression applies only to confirmed 401 responses. Ambiguous authentication failures remain visible.

The wait covers both matching-window edges and the log watchers’ polling delay. It ends at a fixed deadline. Evidence read after that deadline cannot erase a failure. Correlation evidence expires independently of alert delivery, and queue saturation is reported through the existing queue health counters and daemon warnings.

A frozen Exim message is reported once. Exim logs a still-frozen message again on every queue run, and those lines are not reported again, also after a daemon restart: the daemon keeps the messages it reported in its state database until a day passes without a queue run logging them. A message that froze while the daemon was stopped has not been reported, so the first queue run after the start reports it.

Startup discards expired records and invalid message IDs, and adjusts timestamps ahead of the current clock. These corrections are saved even if no new mail event arrives, so another restart does not extend the corrected retention period.

FTP findings use only the peer field directly after the pure-ftpd program tag, in both realtime monitoring and scheduled scans. Missing or malformed peers and hostname peers produce no finding. To record numeric peers, use pure-ftpd’s existing DontResolve yes setting. Addresses in usernames or later message fields never identify the client.

Successful FTP logins over loopback do not raise an unfamiliar-address warning. Failed authentication remains reportable over loopback, including through local relays.

The FTP and SSH log watchers read the same files as the periodic ftp_logins and ssh_logins checks, so both see every login. Each login is reported once: the watcher and the periodic check build the same finding, and the second one is recognised as a repeat. When the watcher is not running, on a non-cPanel host or before the log file appears, the periodic check still reports the login on its own.

An SSH login is reported only from sshd’s own success record: the sshd program tag, then Accepted <method> for <user> from <address> port <port>. sshd also logs the login names that clients offer, so the same words anywhere else in a line are ignored and cannot name an address to alert on or block. The reputation check uses the same record parser when collecting SSH addresses. Traditional syslog, RFC3339 timestamps, and records without timestamps are supported, with either sshd or sshd-session tags and optional process IDs. IPv4 and IPv6 addresses are accepted; trailing key information does not change the account or source address.

The identity uses the complete log record, including its timestamp and session fields, even when the displayed details are shortened. A later session from the same address is a new finding, even within the 24-hour history reminder window.

A successful FTP login and a cPanel File Manager write are not emailed. On shared hosting every customer connects from an address that is not infrastructure, so both fire on ordinary use of a core feature. They stay on the findings page, in history, in incident correlation and in the attack database, where their value is correlation with other evidence on the same account. Failed FTP authentication, FTP brute force, a login from a brute-force source, and SSH logins are all still emailed.

The phpanel webhook and SSE event stream still receive these successful operations; the email and operator-webhook filter runs after data delivery.

Login check names on upgrade

New findings use ftp_login in place of ftp_login_realtime, and ssh_login_unknown_ip in place of ssh_login_realtime. Update external phpanel, export and SIEM rules to accept the merged names. Historical records and already queued deliveries can still contain the old names; the JSON schema is unchanged.

Existing alerts.email.disabled_checks entries and saved suppression rules for either retired name also match its replacement. The Settings page displays and saves the current names. A mute for one former producer now covers the shared finding from both producers; it does not mute FTP brute-force escalations.

Pending SSH blocks and restored SSH incident evidence retain their blocking policy. Historical successful FTP activity remains excluded from incident blocking, including events carrying its retired name.

cPanel-only log watchers are not registered on non-cPanel hosts, so you will not see “not found, retrying every 60s” warnings for them on plain Ubuntu or AlmaLinux.

The Postfix/Dovecot file reader polls every two seconds. It reads replacement files from the start and rewinds when the current file shrinks below its read position. Truncation also clears buffered bytes from the previous file contents. With copytruncate, a file that regrows past that position between polls can hide the truncation and lose events. Use rename/create rotation with the log writer reopening its file, or journal input, when that loss is unacceptable.

Mail records are emitted only after their newline arrives. A partial record survives temporary EOF up to the 64 KiB limit, including its newline. Longer records are discarded through the next newline even when written across several polls. Rotation and detected truncation clear pending record state.

Mail source attachment retries after failures, starting at one second and doubling to a maximum delay of 30 seconds. The watcher remains unhealthy and reports an unavailable-source finding until a reader attaches successfully. Repeated identical errors are not re-emitted. Retries use the current mail source configuration and start at the current tail, so delayed attachment does not count historical authentication failures as new activity.

With mail_logs.source: auto, each retry chooses the configured or platform file if present, otherwise the configured journal units. A file missing for 90 seconds triggers a new selection. The old reader stops before a replacement starts. Journal input follows new records from the selected services, including services with no prior entries; it does not replay older records on attachment.

Explicit file and journal modes retry their selected source without switching, and a working reader stays attached until it stops or loses its file. Journal input requires a build with journal support. The default units are postfix, postfix@- (the Postfix instance on Debian and Ubuntu) and dovecot. Unit names match exactly, so list any further Postfix instance, such as postfix@postfix-out, in mail_logs.units. Each journal record reaches the mail detectors with its program name, process id and time, as a mail log file records them, so every mail detection works the same with either source. Journal selection uses the trusted systemd unit, not the caller-supplied syslog identifier. Keep mail_logs.units limited to mail services. File input has no sender credentials: a local user who can submit mail-facility syslog messages can forge failures or successes there. On hosts with untrusted local users, use mail_logs.source: journal with the mail service units, or restrict the syslog input that writes the mail log.

SMTP / Dovecot Brute-Force Tracker

Detects credential stuffing, password spray, and raw SMTP probe storms. Runs as part of the Exim mainlog watcher on cPanel hosts and on non-cPanel Exim hosts where /var/log/exim_mainlog exists.

On hosts without that log, such as Postfix servers that check SMTP logins against Dovecot, the tracker counts a failed SMTP login when two lines from the mail log agree:

  • Postfix’s SASL ... authentication failed warning, which is the final result of the request, and
  • Dovecot’s rejection for the same client address shortly before it: Password mismatch, unknown user, or PAM’s authentication failure.

Neither line counts alone. Postfix logs a Dovecot outage the same way as a wrong password, and a Dovecot rejection is one password database’s answer: a later database can still accept the login or fail. A rejection followed by a successful login leaves no Postfix warning, a Dovecot error or warning for the same client within the pairing window withholds the failure, and one Postfix warning counts once however many databases rejected the request. Postfix SASL backend warnings also withhold failures from that SMTP process, including stale replies after a failed reconnect. Replies that blame the authentication server or reject the authentication protocol without checking credentials never count.

Pairing is bounded by both receipt time and, when present, the timestamps in traditional syslog or RFC3339 records. Reading a backlog does not make an old timestamped rejection fresh. The mailbox for smtp_account_spray comes from Dovecot; when pending rejections for an address name different mailboxes, its failures count toward address and subnet detection with no mailbox, and long-memory spray detection counts each rejected name once. Each credential or backend failure consumes one pending rejection, including failures withheld for backend errors. Authentication protocol refusals from Postfix or Dovecot neither count nor consume a pending rejection. Tracking capacity is bounded: new verdicts are dropped when full, and error-mark overflow withholds pairing until the affected window expires. Infrastructure protection is checked again before a paired failure counts.

The pairing bounds apply to every name in an ambiguous batch. Only names eligible when a reply is paired enter that reply’s evidence. A withheld backend reply discards the batch’s pending names, so they cannot return as evidence after recovery. Names already counted or discarded remain consumed for their pairing window; a later rejection can start a new window.

Dovecot mailbox existence is unknown when the logged name is empty, sanitized, abbreviated, or disagrees with another pending rejection. An unknown user answer for a name that is not in the mailbox’s own spelling, such as mixed case or a final dot, is unknown too, because it is about that spelling and not the mailbox. These attempts still count toward address and subnet detection. For one unambiguous mailbox, a password mismatch from any database takes precedence over an unknown-user answer. Forged verdicts inside names or logged passwords do not supply mailbox evidence.

Postfix login names are read only from the username field immediately after the authentication method. Sender and forwarding fields cannot supply a missing login name. Mailbox keys ignore surrounding whitespace and ASCII case, and remain stable when a long name must be shortened. Non-ASCII letters remain distinct and cannot borrow another mailbox’s login trust. Names with invalid UTF-8 count as unnamed failures and cannot earn login trust.

Dovecot writes rejections only with verbose auth logging on:

# /etc/dovecot/conf.d/10-logging.conf
auth_verbose = yes

Leave auth_verbose_passwords at no; lines that carry a logged password are ignored. csm doctor checks both settings on Postfix hosts with Dovecot. Rejections for IMAP, POP3 and ManageSieve logins carry a session id and are counted by the mail tracker below from their login summaries. Those failures carry Dovecot’s unknown user and Password mismatch answers too: CSM pairs each answer with the login summary of the same session, address and name, and the spray rules use it to tell whether the mailbox exists. Answers from several databases combine as they do for SMTP. An answer on its own records nothing, and an authentication error that names the session withholds the session’s answers, so a failing database cannot make a real mailbox look missing.

For IMAP and POP, a cut or unrecognized authentication message that names a session also withholds that session’s answers, and a local mailbox lookup does not replace them. Each mailbox answer expires by its own log timestamp; another name cannot renew it, and a later timestamp cannot answer an earlier login summary. Successful logins consume pending session answers too. Session tracking runs only while the spray rules are on, is held in memory and starts empty after a daemon restart.

A successful SMTP login is read from Postfix’s record of an authenticated client, logged when its first recipient is accepted, or from an authenticated Exim delivery. It counts for the mailbox it names. A login from an address with no recent failures on other mailboxes earns trust for that mailbox only, and the trust counts once it is older than the fast detection window. Failures on just one other mailbox, the shape of an office device with a stale saved password, do not count here unless that mailbox does not exist or the login’s own mailbox also failed. The slow brute-force block skips an address only when it holds such trust for every mailbox it failed on. Any other login that follows failures on other mailboxes earns no trust, unless the address already held trust on that mailbox from before those failures or, for IPv6, another address of the same /64 holds aged trust on it. Trust lasts as long as the spray memory window and never exempts an address from fast brute-force or subnet detection.

Trust earned beside a failing mailbox vouches only for the mailbox that logged in. It cannot excuse missing-mailbox probes or downgrade a compromise while the earlier untrusted failures remain in memory.

Failures without a mailbox prevent the slow brute-force trust exemption until they leave the slow detection window. Exim spray evidence comes from authentication failure records; greetings, subjects and SMTP commands cannot supply a failure record or replace its attempted mailbox.

Long-memory spray detection remembers each address’s failed SMTP logins for days, so a botnet that tries each address only now and then is still caught. It flags an address that fails on several mailboxes it holds no trust for, or that fails on a missing mailbox after enough networks have tried the same name. A mailbox is missing when Dovecot reports it as unknown or when the cPanel lookup below finds no such name. For an address that holds trust on some mailbox, a missing name only counts toward the several-mailboxes rule, so one address typo does not block an office. A login after failures on other mailboxes raises mail_account_compromised, except when they were on just one other mailbox that may exist and the login’s own mailbox did not fail. In observe mode, the default, the spray rules report mail_bruteforce_suspected and block nothing; an address is reported once per memory window, and again sooner only when it fails on more mailboxes. In enforce mode they emit smtp_bruteforce. IPv6 failures are remembered per /64, since one host can rotate through many addresses in it: failures from any address of the /64 count together, and aged login trust any of its addresses earned on a mailbox excuses failures on that mailbox for all of them. The missing-name first-strike exemption and the established-source compromise downgrade also count trust from any address of the /64 that predates its current untrusted failures, and a correct login to a mailbox the /64 already trusts starts trust for a new address. Trust is still earned per address, the slow brute-force exemption still needs the address’s own trust, and each failing address is blocked on its own. In observe mode a /64 is reported as one address would be. An operator unblock, allow or flush clears the address’s spray failure history, for IPv6 that of its /64, and keeps its login trust. Failures during an authentication backend outage, Exim’s temporary authentication failures (a 4xx reply, logged when Dovecot cannot answer), and failures withheld by pairing are never spray evidence. An outage is detected a little late, so a failure during one also discards that address’s spray failures from the fast detection window before it. Successful SMTP logins during an outage leave spray memory unchanged. The memory is saved every few minutes and at shutdown, so a daemon restart keeps it. The thresholds.mail_spray_* keys control the mode, the memory window and both rules.

On cPanel hosts, in observe and enforce mode, CSM checks whether a failed SMTP, IMAP or POP login name is a real mailbox, using cPanel’s own account, domain and mailbox lists. A domain the server does not host, an unknown mailbox on a hosted domain, and a bare name that is not a cPanel account count as missing. A real mailbox or cPanel account counts as existing, and any name CSM cannot tell about stays unknown, so it counts only toward the several-mailboxes rule. A name within two edits of a real mailbox, counting the local part and the domain together, counts as that mailbox: spray memory records the failure against the real mailbox, so aged login trust on it excuses the typo and the typo is never a first strike. Login trust and spray failures treat a valid mailbox domain with or without its final dot as the same name, including saved history. The fast, slow and per-mailbox failure counts use the attempted name with surrounding whitespace removed and ASCII case ignored; they do not use typo matches. CSM holds these lists for a short time, so a mailbox created or deleted just now can get its old answer. When the lists cannot be read every name stays unknown; the daemon log gets one warning, and one line when they can be read again. Only the default thresholds.mail_brute_account_key reads the whole login name, so with any other key IMAP and POP failures skip this check and stay unknown. Each login keeps that setting through the check, even if configuration reloads while it is being handled.

Restored spray memory stays within the configured capacity, and expired entries are discarded. Records dated in the future cannot establish login trust or corroborate a missing mailbox. Saved revocations override conflicting trust records.

Dovecot’s informational and debug authentication messages can contain client text; they never mark the authentication backend as unavailable. Error and warning diagnostics still reach the backend-outage detector. smtp_probe_abuse reads Exim lines only.

Signals:

SignalWhat triggers itAuto-response
smtp_bruteforceA single attacker IP exceeds the per-IP failed-auth threshold within the configured window, or, with spray detection in enforce mode, fails on several untrusted mailboxes or on a missing mailbox many networks triedIP blocked via nftables
smtp_probe_abuseA single attacker IP exceeds the raw SMTP connect-rate threshold before AUTHIP blocked via nftables
smtp_subnet_sprayMultiple distinct attacker IPs from the same /24 subnet exceed the subnet thresholdEntire /24 subnet blocked via nftables
smtp_account_sprayMany distinct attacker IPs targeting the same mailbox exceed the account thresholdVisibility finding only. No auto-block, because attackers span many subnets and no single-IP action helps
mail_bruteforce_suspectedSpray detection in observe mode: the spray rules matched an SMTP source. The message names SMTP and the ruleVisibility finding only
mail_account_compromisedA successful SMTP login after failures on two or more mailboxes including another one, or on a missing one, or after repeated failures on the same mailbox, the first at least an hour before the loginCritical only in enforce mode when a failed name was a missing mailbox many networks tried, and the IP is then blocked and the mailbox suspended under auto-response. A source trusted on two or more other mailboxes is downgraded. Every other case is a High advisory, visibility only

Tunable via the thresholds.smtp_bruteforce_*, thresholds.smtp_probe_* and thresholds.mail_spray_* keys in csm.yaml. Infrastructure IPs (from infra_ips) are never counted or blocked.

Cloud-Relay Credential Abuse

Detects authenticated outbound Exim deliveries where the same mailbox is sending through public-cloud relay sources. The realtime Exim mainlog watcher evaluates new accepted deliveries, and a bounded startup replay covers recent lines already on disk.

Startup replay reads the time zone and process ID that Exim’s log_timezone and pid log selector add, and a restart does not repeat an alert for sends already reported.

The finding is email_cloud_relay_abuse. Under the global auto-response and dry-run settings, the abused mailbox’s logins and outgoing mail are suspended, with the account-wide mail hold as the fallback. Operators with legitimate cloud mailers can opt out specific mailboxes or domains under email_protection.cloud_relay, or use email_protection.high_volume_senders for known high-volume senders.

Authenticated Sender Profile

Catches a stolen mailbox password that is used quietly: under the rate limits, from residential addresses with no cloud reverse DNS, using individually addressed messages. Every authenticated Exim submission updates a rolling hourly window and the mailbox’s own history of source addresses, source countries and envelope recipients, grouped by UTC day. Repeated sends keep each source current, and a busy mailbox continues to accept new source and country evidence.

The finding is email_compromised_account. Four signals raise it:

SignalSeverity
Sends from several countries within one hour, the pattern of a credential checker validating a stolen passwordCritical
Far more distinct source addresses within one hour than the mailbox’s own recent maximumHigh, Critical when the sends also come from a country the mailbox never sent from
Far more distinct source addresses within one day than the mailbox’s own recent maximumHigh, Critical as above
Far more distinct recipients within one day than the mailbox’s own recent maximum, each in its own messageHigh, Critical as above

Each bar adapts to the mailbox’s own prior maximum with a fixed minimum, so a shared office mailbox or a travelling owner is judged against their own habit rather than a global number. A country listed in suppressions.trusted_countries is never treated as new. A mailbox needs established prior activity before a new country counts. A newly seen country remains relevant if unusual sending develops later in the same UTC day. Repeated findings are suppressed during a cooldown, while escalation from High to Critical remains possible.

CSM uses the time in the Exim record to place a loopback send. It uses the country of the mailbox’s most recent webmail login only if that login preceded the send and was recent enough to locate it. An unlocated login clears the previous location.

Recipient fan-out counts distinct envelope recipients of individually addressed messages; bulk announcements do not contribute. Quoted addresses remain single recipients. Records with ambiguous recipient metadata do not contribute recipient evidence, but verified source evidence still counts.

Daily history covers the recent baseline period and expires even for inactive mailboxes. It is cached and saved periodically, before idle eviction and during normal shutdown. An abrupt termination can lose submissions since the last save; a restart always starts a fresh hourly window. Save failures retain pending history for retry. An unreadable baseline pauses profiling for that mailbox until the store can be read again.

Source addresses are never blocked for this finding: they rotate per message and the owner’s own address can be among them. A Critical finding suspends the mailbox’s logins and outgoing mail under the auto-response and dry-run settings, with the account-wide mail hold as the fallback. Mailboxes listed in email_protection.high_volume_senders are not profiled.

Mailbox Compromise Evidence

A stolen mailbox rarely trips a single loud alarm. It shows a few weak signs, each of which an honest owner also produces now and then. CSM keeps a ledger of these signs per mailbox. Each sign is reported once in the audit log, history and the daily summary, and is not mailed on its own.

The signs cover logins from new countries, forged sender domains, bulk recipients, external redirects, and outbound reputation deferrals. Two new countries seen close together carry more weight. A redirect created soon after a foreign login also carries more weight.

A login from a new country is an email_suspicious_geo finding, at most one per mailbox a day. A mailbox needs a few logins of history first, and a country in suppressions.trusted_countries is never new. Webmail, IMAP and POP3 logins share the same history and use the login time in the log. Sessions opened by the panel or a reseller do not locate the mailbox owner. An authenticated send from a country the mailbox never logged in from is the same sign, as is a webmail send after a webmail login from a new country. A send reports email_suspicious_geo the first time the ledger records its country.

A mail filter or Sieve script that sends a mailbox’s mail to an outside address keeps its own email_filter_exfil finding and is also a sign, dated by the script’s file time. The incident finding names the script and its target; CSM leaves the script in place as evidence, so remove it after review.

When large providers refuse or defer this server’s mail because of an address’s reputation, the address is an email_outbound_reputation finding at most once a day, naming the authenticated mailbox that sent most of the refused mail. A tie or an incomplete count leaves the finding at address level. If another mailbox leads a later burst, its ledger records that evidence even after the daily notice.

A mailing is an email_bulk_recipients finding the first time the ledger records it. An authenticated send whose envelope sender is a domain the login’s hosting account does not own is an email_forged_sender finding the first time the ledger records that domain. A subdomain follows its closest mapped parent, including aliases and subdomains delegated to another hosting account.

Each kind of sign counts once, however often it repeats. Mailboxes in email_protection.high_volume_senders send mailings by design, so forged sender domains and bulk recipients do not count for them. Signs expire a week after their last sighting. The ledger groups older details when many distinct values appear.

When the evidence adds up, CSM suspends the mailbox and raises one Critical email_mailbox_compromise finding. It lists the evidence and the restrictions applied. Release restrictions applied by CSM on the Email page, Suspended tab. With automatic response disabled or dry-run enabled, the finding says CSM would suspend the mailbox. Further signs update the same incident. Release discards its evidence, including delayed observations from before release.

Mail Auth Brute-Force Tracker

Detects credential stuffing and password spray against IMAP, POP3, and ManageSieve. Runs through the mail_logs reader: file source uses /var/log/mail.log on Debian-family hosts and /var/log/maillog on RHEL-family and cPanel hosts, while journal source reads configured Postfix/Dovecot units. The wrapper composes with the existing geo-based login monitor, so email_suspicious_geo keeps reporting successful logins from novel countries as compromise evidence.

Signals:

SignalWhat triggers itAuto-response
mail_bruteforceA single attacker IP exceeds the per-IP failed-auth threshold within the configured window without matching successful mailbox activity, or, with spray detection in enforce mode, fails on several mailboxes it holds no trust for or, on cPanel hosts, on a missing mailbox many networks triedIP blocked via nftables
mail_bruteforce_suspectedAn established good source hits the per-IP failed-auth threshold with a confined stale-password pattern, or spray detection in observe mode matched an IMAP/POP source. The spray message names IMAP/POP and the ruleVisibility finding only. No auto-block
mail_subnet_sprayMultiple distinct attacker IPs from the same /24 subnet exceed the subnet thresholdEntire /24 subnet blocked via nftables
mail_account_sprayMany distinct attacker IPs targeting the same mailbox exceed the account thresholdVisibility finding only. No auto-block, because attackers span many subnets and no single-IP action helps
mail_account_compromisedA successful login comes from an IP that repeatedly failed auth against the same mailbox, or follows failures on two or more mailboxes including another one, or on a missing one, or repeated failures on the same mailbox, the first at least an hour before the loginCritical findings block immediately and, under auto-response on cPanel or on Stalwart with its API key, suspend the mailbox (a cPanel account’s own login holds the account’s outgoing mail instead). A source with established history on at least two other mailboxes from before its current untrusted failures emits a High advisory and is not auto-blocked. A login after failures on other mailboxes is Critical only in enforce mode when one of them was a missing mailbox many networks tried; every other spray case is a High advisory

Tunable via the thresholds.mail_bruteforce_* keys in csm.yaml. Independent from the SMTP tracker so the Dovecot noise floor can be tuned separately. Infrastructure IPs are never counted or blocked. Established good sources with a confined stale-password pattern emit mail_bruteforce_suspected instead of an IP block, while wider spraying and compromise from a source without established multi-mailbox history still block. A compromise from an IP with established history on at least two other mailboxes from before its current untrusted failures remains visible as a High advisory but does not feed direct, incident, or spray auto-blocking. The mail_bruteforce and mail_bruteforce_suspected alerts name the mailboxes the source was hitting, with per-mailbox failure counts and a count of attempts that named no mailbox, so a real attack on a live mailbox is easy to tell apart from dictionary noise.

IMAP and POP logins share the long-memory spray detection described for SMTP above. One memory holds the failures and login trust of both, so failed SMTP and IMAP logins from one address on two different mailboxes count together toward the several-mailboxes rule. A login that follows failures on other mailboxes earns neither login trust nor good standing, unless the failures were all on one other mailbox that may exist and none on the login’s mailbox, the address already held trust on the login’s mailbox from before those failures or, for IPv6, another address of the same /64 holds aged trust on it. The slow brute-force block downgrades to mail_bruteforce_suspected only when the address holds good standing or aged login trust on every mailbox it failed on; a login to some other mailbox no longer counts. One address gets at most one mail_bruteforce per suppression period, and one login at most one mail_account_compromised. The High advisory for a login after failures on other mailboxes is raised once per address and mailbox per memory window, so a client that polls every few minutes does not repeat it. In observe mode the same limit applies to a login after a missing name that several networks tried.

A login identified as a short-window password crack revokes trust for that address and mailbox for the spray memory window, even when the compromise alert is suppressed; for an IPv6 address it revokes that mailbox’s trust for every address of its /64. Account cache pressure does not hide a crack while its per-address failure history remains. Further SMTP, IMAP or POP logins cannot restore that trust during the window. Suspect logins do not count toward successful activity that exempts later failures. A crack also removes any good standing from a recent login that had not yet aged into established standing.

When the mail authentication backend itself fails (for example dovecot cannot reach cpdoveauthd), every login fails regardless of password. CSM detects a burst of these backend errors, pauses mail_bruteforce and mail_subnet_spray auto-blocking, keeps the failures out of spray evidence, and raises a mail_auth_backend_degraded warning so the outage is visible instead of mass-blocking legitimate users. IMAP and POP successes during this degraded period do not update spray memory, raise spray alerts, or earn good standing. The short-window compromise detector still uses recorded failures. Detection resumes automatically once the backend recovers.

Mail login attribution uses the connection fields outside the raw login name. Names containing field-like text or angle brackets remain one account name. Records without a usable remote address do not count toward IP blocking or login trust. Reputation exemptions require a successful Dovecot login summary; success wording in a failed login name does not qualify.

Stalwart Mail Server

On hosts where the stalwart service is the mail server and cPanel is absent, CSM reads Stalwart’s daily log instead of the Postfix and Dovecot mail log. It follows the newest stalwart.log.YYYY-MM-DD file in stalwart.log_dir (default /var/log/stalwart), starting at the end of the current file. Files dated later than tomorrow in UTC are ignored. When Stalwart starts the next day’s file, CSM reads the old file to its end and then the new one from its start. A missing directory or file raises mail_log_source_unavailable.

If the next day’s file cannot be found or opened for a while, CSM raises mail_log_source_unavailable and keeps retrying. If the current file is removed, CSM does not go back to an older day’s file.

Each event is assigned a protocol from its listener port: 25, 465 and 587 are SMTP; 143 and 993 IMAP; 110 and 995 POP3; 4190 ManageSieve; 80, 443 and 8080 HTTP (JMAP and webmail). List other listeners in stalwart.ports as PORT:PROTOCOL. Logins and bans on an unmapped port are ignored; sends on one, such as a local submission listener, count toward volume and recipients only.

What CSM reads:

  • Failed and successful logins on SMTP listeners feed the SMTP brute-force tracker, and those on IMAP, POP3 and ManageSieve listeners feed the mail brute-force tracker. The same thresholds, findings, blocks and spray rules apply as on Postfix and Dovecot. Loopback, private and infrastructure addresses never count.
  • Failed logins on the HTTP listener raise stalwart_webmail_bruteforce at the mail brute-force threshold. Stalwart can log an address from a forwarded header there, so this finding never blocks. Stalwart’s own throttling and bans still apply to it.
  • Authentication, abuse and scan bans Stalwart logs for an SMTP, IMAP, POP3 or ManageSieve listener raise stalwart_ban, which blocks the address for the auto-response block length. Stalwart’s ban stays in place. A failed-login ban of an address a customer logged in from within the last 30 days, such as an office network with one stale password, is a High finding and is not blocked. Loiter bans, refused connections and bans on the HTTP listener or an unmapped port are ignored.
  • Authenticated sends on SMTP listeners and unmapped ports are matched to the login on the same connection and feed the rate, sender profile and cloud-relay checks. Sends from a loopback or private address, such as sites on the same server, and sends on an unmapped port count toward volume and recipients only.
  • Unauthenticated messages Stalwart queues are matched by queue id to the lines that record their delivery, so the Email page counts what the spam filter did with each one and the sending address gets spam evidence. Rejected and discarded mail is read from the filter’s own line. Messages on the HTTP listener are ignored. See Stalwart hosts.

Stalwart logs failed logins at debug level by default, so password guessing over SMTP, IMAP and POP3 is invisible until the auth.failed event is raised to info. csm stalwart setup does that (see below). csm doctor reports whether the newest log holds failed logins, whether it is current, and whether its recent lines are in the layout CSM reads.

Preparing Stalwart

csm stalwart setup changes four settings and adds a spam rule through Stalwart’s admin API at stalwart.api_url (default http://127.0.0.1:8080), then reloads Stalwart’s settings once. It also creates the role, service account and API key CSM uses to suspend mailboxes and lift bans. The API address, including a --url override, must not contain a login, query or fragment. Remote addresses require HTTPS and cannot use this host’s port locks as protection. Run it as root with a Stalwart admin login:

STALWART_USER=admin csm stalwart setup --dry-run --password-file /root/stalwart-admin.pw
STALWART_USER=admin csm stalwart setup --password-file /root/stalwart-admin.pw

STALWART_PASSWORD works in place of --password-file, and --password-stdin reads the password from stdin, for example printf '%s\n' "$PW" | csm stalwart setup --password-stdin; the user is then STALWART_USER or admin. On a terminal, setup prompts for missing values. When you enter both values, the username is hidden too. Press Enter at the user prompt for admin. Type the password at the password prompt, where echo is off even with --password-stdin. Ctrl-D at an empty prompt cancels setup. Before sending it to an address on this host, setup and doctor check that the API port and each connected peer belong to root or Stalwart’s service user (stalwart or stalwart-mail). An unreadable ownership record stops the login. This also protects reconnects when Stalwart restarts. --dry-run shows each step and changes nothing; a second run after a successful one changes nothing and does not reload. If a settings step fails, part of it may already be stored: resolve the error and reload settings in Stalwart before running setup again. If one of steps 6 to 8 fails, resolve the error and run setup again. If only the reload failed, restart Stalwart: a second run reads the stored settings as done.

  1. Failed-login logging. Sets the auth.failed event to info unless it is already logged at info, warn or error.

  2. Webmail client addresses. Makes Stalwart’s HTTP listener take the client address from X-Forwarded-For (1 hop), so webmail and JMAP failures name the real client instead of the web server. Anyone who can connect to the listener could forge that header, so setup only does this when the API address names this host on loopback, every HTTP listener has known loopback-only binds, and the running CSM daemon locks each of their ports to the web server and Stalwart. Otherwise it skips the step and prints what is missing, including the line to add under firewall in csm.yaml, for example local_port_locks: ["8080:www-data,stalwart"]; restart CSM after adding it. If Stalwart already takes forwarded addresses without that protection, setup reports the step as unsafe and exits 1 without changing it. See Local port locks.

    With one hop Stalwart reads the last X-Forwarded-For entry, so the web server in front of it must set that header itself. nginx passes a client’s header through unchanged unless told otherwise; add proxy_set_header X-Forwarded-For $remote_addr; (or $proxy_add_x_forwarded_for) to the Stalwart location. Without it a remote client chooses the address Stalwart logs and bans. Setup cannot check this. The users in the lock line must not run site code: run PHP-FPM pools as their own users, not as the web server’s user.

  3. Ban expiry. Sets Stalwart’s failed-login, abuse and scan ban periods to auto_response.block_expiry, so its bans end when CSM’s blocks do. Loiter bans are left alone.

  4. Spam verdict logging. Sets the spam.classify event to info unless it is already logged at info, warn or error. By default Stalwart logs the mail its spam filter rejects or discards only at debug level. This step changes nothing about how mail is handled.

  5. Address list. Creates the spam rule CSM_IP and one scoring tag per kind of listing, so Stalwart adds up to 4 to the spam score of a sender on CSM’s address list and more of their mail goes to Junk, or is rejected or discarded at the scores set in Stalwart. This is the one setup step that changes what happens to mail. Setup creates the rule switched on; to stop it, switch the rule off in Stalwart’s admin interface. Setup never switches it back on and reports a rule that is off as done. It creates missing tags but never changes an existing tag, so a tag’s score is yours to set in Stalwart. When the rule’s condition differs from CSM’s, setup resets the condition and leaves the rule’s other fields alone. A rule named CSM_IP that CSM did not make is left alone with a warning. If Stalwart’s spam rules cannot be read, setup warns and continues with the other steps.

  6. Response role. Creates the role CSM response: the permissions CSM needs to suspend accounts, lift bans, keep its address list in Stalwart and read the settings setup checks, plus Stalwart’s own User role. Including User lets the key change user accounts. When the role lacks any of these or denies permissions, setup rewrites it to CSM’s list and clears denials. If Stalwart has no role named User, setup warns and skips the next two steps.

  7. Response account. Creates the account csm-response holding only that role and no password. It can still receive mail at its address. On a server with several domains, name the domain with --account-domain NAME; otherwise setup skips the step and the next. On a server with tenant administrators, put the account in a domain only the server administrator controls: whoever can edit the account could give it a password. If more than one account is named csm-response, setup warns and creates nothing. Setup also refuses an existing csm-response account that has a password or another sign-in credential, has sign-ins stopped, or holds keys without CSM’s role: it warns, changes nothing and skips the key step.

  8. Response key. Creates an API key of csm-response that works only from 127.0.0.1 and ::1, writes it to stalwart.api_key_file (default /etc/csm/stalwart-api.key, owned by root, mode 0600), then deletes the account’s other keys. The key is never printed. A key already in the file that Stalwart accepts for csm-response is kept while it is the account’s only key; if the account has other keys, setup replaces them all, so a rerun finishes a clean-up that failed. If the file cannot be written, setup deletes the new key and the old one keeps working. A key file that is not a root-owned regular file closed to group and others stops setup with exit 1. To rotate the key, delete the file and run setup again.

Steps 6 to 8 need no settings reload, and they run only when the API address is on this host; otherwise setup skips them and the key is never sent. When one of them warns, setup exits 1: CSM then can neither suspend mailboxes nor lift Stalwart’s bans.

Setup also reports listeners that accept the PROXY protocol, which it does not change, and whether Stalwart’s spam filter is on and at which scores it files spam to Junk, rejects or discards. Doctor prints the same. A filter that is off is a warning; setup never switches it on or off. If its status cannot be read, setup warns and continues with the other steps. A refused admin login still stops setup.

csm doctor checks the same steps read-only through CSM’s API key, and says whether Stalwart accepts the key. Only root can read the key, and a missing key is reported even while Stalwart is down. Without a working key, or for an API address on another host, it uses an admin login from STALWART_USER and STALWART_PASSWORD when both are set. A step setup would change is a warning, as is a response step setup refuses until you clear what it names. Its spam lines say whether Stalwart logs the mail its filter rejects or discards, whether the filter is on and at which scores it acts, and whether its DNS blocklists answer. The blocklist check runs only while the filter is on: it sends Stalwart one test message, which Stalwart never delivers or learns from, and warns when the blocklists refuse this server’s resolver. Blocklist warnings show a bounded, terminal-safe list of refused tags; they do not report the test message’s verdict or score. Stalwart keeps refused answers until it restarts, so restart it after changing the resolver. The same test message comes from the address the daemon keeps on CSM’s address list, so while the address list rule is on, doctor also says whether Stalwart scores senders on that list. The daemon writes the list within 10 minutes of setup, so this check can warn until then; run doctor again after that. If it still warns, check the daemon’s stalwart_address_list finding, and run csm stalwart setup only when that finding says to. Observe mode skips this check, since the daemon then does not write the list to Stalwart. From the newest log alone it says when Stalwart’s spam classifier is still learning and how many ham and spam samples it has. It omits that line if the latest classifier status is malformed or a later training run has started. It also warns when Stalwart’s HTTP listener logs a login from another host on a port CSM does not lock, and when Stalwart bans a CDN or proxy address, which locks out every client behind it.

stalwart_log_degraded warns when an hour of log lines holds none in Stalwart’s layout, as after an upgrade that changes it; when many authenticated SMTP sends in an hour cannot be matched to a login; and when the log stops growing for hours while Stalwart runs.

Known limits:

  • Suspending a Stalwart mailbox stops new sign-ins only. Sessions that are already open stay open until they end.
  • Webmail logins appear in Stalwart’s log from the proxy address, so a webmail login after failures is not detected as a compromise.
  • Sends through webmail or JMAP are not tied to an account, because one proxy connection can carry several users, so the send checks do not see them.
  • A listener behind PROXY protocol logs the client address, not the connecting peer. CSM cannot tell such listeners apart from the log alone.

Suspending Stalwart mailboxes

CSM suspends a hacked Stalwart mailbox through the API key that csm stalwart setup writes. The auto-response and dry-run settings apply as on cPanel. After a Critical mail_account_compromised, email_compromised_account or email_cloud_relay_abuse finding, CSM stops the account’s sign-ins through Stalwart’s API. That also stops it sending. A sign-in through a mailbox alias or a domain alias suspends the account behind it. All aliases share that account’s suspension and release history. Buffered sends logged before a release do not suspend it again. Reassigning an address does not transfer a suspension to its new account. The mailbox is listed on the Email page, Suspended tab, where Release allows sign-ins again on the same account. Release does not change the password. Sign-ins someone else had already stopped are left alone and not listed.

CSM does not suspend admin accounts, its own API key account, accounts its key may not change, or account names without a domain. Bare names containing + or % are not converted to mailbox addresses; that login convention applies only to cPanel. The finding then asks staff to change the password in Stalwart. The API key is sent only to loopback addresses, after verifying the connected peer. Without the key file nothing is suspended. The email_compromised_account and email_cloud_relay_abuse findings then say the mailbox stays active; a mail_account_compromised finding gets no mailbox response. Address blocks apply either way.

Admin-Panel Brute-Force Tracker

Counts repeated POST requests to high-value non-WordPress admin login endpoints. Runs as part of the web access-log watcher.

Covered endpoints (tight set to avoid false positives on shared hosting):

  • phpMyAdmin: /phpmyadmin/index.php, /pma/index.php, /phpMyAdmin/index.php
  • Joomla: /administrator/index.php

When an IP crosses the POST-rate threshold, admin_panel_bruteforce fires and the attacker IP is auto-blocked.

Drupal /user/login and Tomcat Manager /manager/html are intentionally out of scope here. Drupal’s path is too generic on shared hosting, and Tomcat Manager uses HTTP Basic auth (repeated GET requests with 401 responses), not POST form submissions. Both need different detectors and are tracked as follow-up work.

PHP-Relay (Mail Abuse, cPanel Only)

Real-time inotify watcher on /var/spool/exim/input catches WordPress contact-form spam relays where an attacker uses PHPMailer (or similar) with a spoofed From, an external Reply-To, and a script URL that doesn’t belong to the cPanel account. A contact-form relay incident drove the design: a legitimate site running a vulnerable contact-form plugin became a per-message spam relay through the operator’s own mail account.

The detector runs five paths. Paths 1, 2, 2b and 4 fire email_php_relay_abuse (Critical) when one of them crosses threshold; Path 6 has its own findings, described below. Paths 1 and 2 are scoped per-script, using the host:/path from the X-PHP-Script Exim header. Path 2b is per cPanel user. Path 4 is per HTTP source IP across distinct scripts. Path 6 is per client IP across every script and account. Paths 2 and 4 use a recipient-diversity gate that suppresses only known low-recipient notification mail.

The source address is the final, server-observed address in X-PHP-Script. Client-supplied forwarding text cannot replace it. Behind a proxy this is the proxy’s address; CSM does not recover a visitor address from untrusted forwarding text. A missing or invalid final address leaves IP attribution empty.

The script label ends at the first for separator, keeping forwarding text out of per-script counts and queue actions. PHP includes PATH_INFO in this header without escaping separators. Paths containing a literal for therefore share the label before that separator; the source address is parsed separately from the header’s end.

PathWhat triggers itWhy it exists
Path 1: header scorePer-script: From domain not in the account’s authorised domains AND additional signal (PHPMailer / suspicious Reply-To / suspicious User-Agent), evaluated over a rolling 5-min window once the script has emitted at least header_score_volume_min messagesThe shape that matched the original incident: spoofed sender, contact-form-style. FromMismatch is a HARD precondition – the score never accumulates without it
Path 2: absolute volume per scriptA single script emits at least absolute_volume_per_hour messages in the last hour. If Exim envelope recipients are known, fewer than fanout_distinct_recipients distinct recipients suppresses this path; unknown recipients or fanout_distinct_recipients: 0 fail open.Catches a compromised script even if the headers themselves are legit-shaped, while leaving plugin notification mail to a fixed admin set alone
Path 2b: account log-tail volumePer cPanel user: at least effective_account_limit messages in the last hour that the account’s own processes submitted locally (PHP mail(), sendmail), each with at least one recipient outside this host’s mail domains. The effective limit is auto-derived from /var/cpanel/cpanel.config’s maxemailsperhour (60% of it, clamped to 20-60), capped at 95% of the cPanel limit when an operator override is setBackstop for when Path 2 misses the window. Reads /var/log/exim_mainlog directly and names the account only from Exim’s own submission fields, so a subject or sender address cannot. Mail to the host’s own mailboxes, bounces, authenticated SMTP and mail arriving from the network never count. If a present mail-domain file cannot be read, or both /etc/localdomains and /etc/virtualdomains are absent, nothing counts
Path 4: HTTP-IP fanoutPer HTTP source IP: one source IP appears in at least fanout_distinct_scripts distinct script keys in fanout_window_min minutes, after excluding loaded HTTP-proxy ranges, loopback, and the host’s own interface addresses. If Exim envelope recipients are known, fewer than fanout_distinct_recipients distinct recipients outside each sending account’s explicitly listed domains suppresses this path; subdomains are exempt only when listed for that account. Missing account-domain data, unknown or partially parsed recipients, or fanout_distinct_recipients: 0 fail open.Catches one client walking many scripts while avoiding CDN/proxy traffic, local cron or panel callbacks, and fixed-admin notification fanout, including one client commenting on several hosted sites
Path 6: client recipient spreadPer client IP: hosted sites mail at least ip_recipient_spread distinct recipients outside each sending account’s own domains within ip_recipient_window_hours, counted across every script and account. An IPv6 client counts per /64. A message whose account domains cannot be read is not counted; a message whose recipients cannot be parsed counts as one new recipient. ip_recipient_spread: 0 turns the path off.Catches bots that submit a site’s forms with strangers’ addresses, so the site sends each stranger an automatic reply. One bot sends too few of these per script or per hour for the other paths to notice

Path 2b reads recipients only from Exim’s envelope recipient list. Message headers and sender text do not supply recipients. A missing recipient list or a truncated address cannot establish an outbound send. The host’s mail-domain inventory is refreshed once per minute; a missing optional file is allowed when the other file can be read.

Path 5 (behavioural baseline) is deferred to Stage 2.

Per-IP paths skip selected shared addresses: CDN and website firewall edges (Cloudflare, QUIC.cloud, Sucuri) and payment gateway webhooks (Stripe, PayPal, Braintree, PayU, Revolut, Mollie). CSM ships this list built in, taken from each provider’s published list. Membership is not proof that the target site chose the service. Cloudflare Workers can make requests to third-party sites, and Sucuri can proxy requests before the site’s DNS changes. Traffic relayed through these retained providers remains exempt, including abusive requests.

Uptime monitors and cron services are not on the list. Anyone with an account there can schedule POSTs with any form body to any URL, so exempting their addresses would let a bot relay form spam through them, and the per-IP paths would miss it. Self-service CDNs such as Amazon CloudFront, Fastly and BunnyCDN are left out for the same reason: anyone can point a pull zone at any site and post through its edges. If you trust a specific cron service, or your sites sit behind one of those CDNs, you can add its ranges yourself, knowing that anyone with an account at that service can use them.

An http_proxy_ranges.yaml in policies_dir (version: 1 plus a cidrs: list; a bare IP counts as /32 or /128) replaces the built-in list whole and reloads on SIGHUP. Without the file the built-in list applies. A file that cannot be read or parsed, has no cidrs: list, or has no valid entry keeps the last good list, which is the built-in one at startup. Invalid entries, including blank or null list items, are reported and skipped; valid siblings still apply. Only an explicitly empty list (cidrs: []) clears the proxy ranges. Load errors at startup or on SIGHUP raise email_php_relay_policies_reload (Warning). To extend the list, start from the built-in file in the CSM source, internal/emailspool/embed/http_proxy_ranges.yaml. Your file replaces the whole list, so later CSM updates to the built-in entries do not reach you until you copy them in again.

CSM 4.10.0 exempted Amazon CloudFront, Fastly and BunnyCDN; later releases do not. An operator file copied from the 4.10.0 list keeps them until you remove those entries and reload. Legitimate visitors of a site behind these CDNs share their edge’s per-IP counts unless you exempt its ranges. For Path 6, review findings in ip_recipient_mode: observe before enforcing; this setting does not disable Path 4. Cloudflare remains separately exempt from Path 6 through the daemon’s refreshed infrastructure list, even when omitted from the operator proxy file.

Netopia, EuPlatesc, Twispay, JCC and Adyen publish no source address list. If your shops use them, add their ranges yourself. Viva publishes a webhook address list; it is not built in yet, add it yourself.

Path 6 does not count a client on the proxy and payment list, a Cloudflare edge the firewall knows, a loopback or host address, or an infra_ips entry. It also skips mail sent from wp-cron.php, which mails whatever its trigger request queued, and staff: a client that sent mail through a WordPress admin page of the same account in the last 7 days. Every .php page directly in a wp-admin/ directory counts as an admin page except admin-ajax.php, admin-post.php, profile.php and index.php, which logged-out visitors or subscribers can reach. The staff exemption covers only that account. Becoming staff, or an ignore-script for a script, also drops the recipients counted earlier in the window through that account or script.

Staff often mail customers without passing such a page, for example by completing orders with the quick actions of the WooCommerce order list. So when a client is about to cross the threshold, CSM reads the last 4 MiB of each involved site’s current access logs (/usr/local/apache/domlogs/<account>/<site> and <site>-ssl_log). For a site starting with www., it also checks the name with that prefix removed. Each distinct log is read once per crossing.

Access-log evidence requires a GET answered 200 on a recognized WordPress staff page, such as the post editor, the WooCommerce order list, uploads, comments, users, settings, plugins or themes. Both order lists qualify: the legacy one on edit.php and the HPOS one on admin.php?page=wc-orders (or page=wc-orders--<order type>). For HPOS, page must be the first query parameter so PHP’s input limit cannot discard it. Public resources, setup pages, subscriber pages and every other admin.php page do not qualify. Requests with another parameter PHP would interpret as page do not qualify either. Encoded or ambiguous paths and extra path after the PHP filename do not qualify. The request must fall within the last 7 days, and the exemption expires 7 days after that request; rereading a log does not renew it. The client’s address, or its IPv6 /64, becomes staff only on that account, whose mail stops counting; it is challenged only if the other accounts’ mail still crosses the threshold. A missing or unreadable log is no evidence.

Path 6 has known limits:

  • A plugin that handles POST requests in its init or wp_loaded hook without checking for a login, or for DOING_CRON, lets a visitor who is not logged in make it send mail by posting to an admin page or to wp-cron.php. That mail is not counted, and a post to an admin page also earns the client the staff exemption on that account.
  • Behind a proxy or CDN that does not pass the client address to PHP, CSM sees only the proxy’s address. A proxy on the list is skipped, so its visitors are never counted; that includes a bot that sends its requests through a CDN account of its own. A proxy missing from the list is counted as one client for all its visitors, so add its ranges to http_proxy_ranges.yaml.
  • Path 6 names a form by its host in lowercase and its path up to the PHP file, so extra path after the file or a change of case does not split it, and one ignore-script with that name covers every such variant. Other names for the same site, such as with and without www., still count as separate forms.
  • Only WordPress admin pages count as staff evidence. Staff of other back offices, such as PrestaShop, OpenCart or Joomla, who send many order or account mails can be challenged and then blocked. Put a fixed office address in infra_ips.
  • Counts live in memory and are lost when the daemon restarts. The startup walk sees only mail still in the queue, and form replies usually leave it within seconds.
  • A bot that rotates addresses and keeps each one below the threshold is never challenged, and its form is not named.
  • An IPv6 client is counted per /64 but challenged or blocked only at the one address that crossed the threshold.
  • A script finding names the client that triggered it. Later clients through the same form in the window get their own client findings but are not added to the script finding.
  • One stranger mailed many times counts once.

When a client crosses the threshold, CSM raises a High finding for it, once per mode per window. ip_recipient_mode sets what follows. In observe mode, the default, the finding is email_php_form_relay_observed: it goes to the daily summary and never challenges or blocks, so you can review it before switching on enforcement. In enforce mode the finding is email_php_form_relay and the client is sent to the challenge page; a client that does not solve it within 30 minutes is blocked while auto_response.enabled and auto_response.block_ips are on. With the challenge off, the client is blocked directly under the usual auto_response settings. auto_response.php_relay.dry_run applies only to the mail freeze and does not hold this response.

After a reload from observe to enforce, the next eligible message can trigger enforcement using recipients still in the window. The earlier observe report does not delay enforcement. Switching back to observe prevents new responses from retained findings, queued blocks, challenge timeouts and central corroboration. Existing challenges and blocks keep their expiry.

CSM also raises email_php_form_relay_script (High) for each script that sent at least two of that client’s outside recipients, once per script per window. It names the account and the form to fix: add a captcha, or turn off the automatic reply to the submitter. For a legitimate bulk sender, use csm phprelay ignore-script. This finding has no source address field and never challenges or blocks. If the client table fills up, CSM drops the clients with the fewest recipients first and reports email_php_form_relay_table_full at most once a day.

The detector starts a one-shot retrospective scan of exim_mainlog at daemon startup so Path 2b can fire on history already on disk. IN_Q_OVERFLOW triggers a bounded recovery walk of the spool (capped at 1000 files; if more were skipped, a email_php_relay_overflow_scan_truncated Critical fires too – Path 2b backstops the missed messages).

Operator suppressions (csm phprelay ignore-script <host:/path>) short-circuit the pipeline before any path scoring runs, so a known-noisy contact form can be opted out individually without disabling the detector. An ignore on a lowercase host:/path ending at the PHP file also covers that file reached with extra path after it or a different host case. See PHP-relay CLI for the full operator surface.

PAM Brute-Force Listener

Real-time authentication monitoring through the PAM module.

  • SSH login tracking with geolocation
  • SSH failures (service=sshd) drive pam_bruteforce and credential stuffing / password spray breadth: one source IP failing against many distinct accounts inside thresholds.multi_ip_login_window_min. The finding is credential_stuffing; tune the account floor with thresholds.cred_stuffing_distinct_accounts (default 5).
  • Only successful sshd logins clear SSH failure counts.
  • Failures from other PAM services (FTP, mail, webmin and similar) raise pam_auth_failures once per window above thresholds.pam_bruteforce_threshold. It never blocks: CSM reads pure-ftpd, Dovecot and Exim logs separately, and counting one login twice would corroborate it with itself. Services without such a log detector, such as vsftpd, proftpd or webmin, have no blocking signal from PAM.
  • Blocks IPs within seconds of threshold breach
  • Integrates with the nftables firewall for instant blocking

Process Context

Exec and outbound-connection findings carry an optional process object with PID, PPID, UID, user, cPanel account (when known), comm, exe, sanitized cmdline, and a parent chain up to depth 5. The chain is materialized from an in-memory LRU+TTL cache (cap 16384 entries, 30-minute TTL) populated from BPF exec events. Cache misses trigger a bounded async /proc read, so process-context enrichment does not add blocking work to the connection event loop. When neither cache nor enricher has data (e.g., a process that exited before userspace reads its event), the process field is omitted entirely and the finding still emits.

Counters exposed at /metrics:

  • csm_process_context_cache_entries
  • csm_process_context_cache_evictions_total (LRU)
  • csm_process_context_cache_ttl_purges_total
  • csm_process_context_cache_misses_total (includes TTL purges)
  • csm_process_context_enrich_queue_drops_total
  • csm_process_context_enrich_reads_total
  • csm_process_context_enrich_errors_total
  • csm_process_context_enrich_stale_total
  • csm_process_context_enrich_latency_seconds

Caveats:

  • started_at is emitted only when the event source supplies a trustworthy start timestamp. CSM does not infer it from procfs directory metadata.
  • After daemon restart, the csm_process_context_enrich_* counters may show a small enqueued - reads delta. Pending requests in the enricher queue are dropped on shutdown by design.
  • Hosts without BPF support fall back to /proc/net/tcp[6] polling. That path has no PID, so emitted findings do not carry a process field.

HTTP Flood, Scanner Profile, UA Spoof, and Distributed Flood

http_request_flood, http_scanner_profile, http_claimed_bot_unverified, http_ua_spoof, and http_distributed_flood are periodic, not real-time. They run inside the same wp_bruteforce scheduled check that scans per-vhost access logs every 10 minutes. A real-time inotify tailer would need to hold per-IP state across log rotations and is out of scope for the initial release (see the plan non-goals). For attack types where sub-minute response matters, the access-log inotify watcher already covers wp_login_bruteforce and xmlrpc_abuse; the periodic scan adds volume-based rate enforcement, pending claimed-bot handling, scanner-profile detection, and per-vhost distributed attack rollups on top.

Requests that a cPanel proxy subdomain (cpanel., webmail., whm. and similar) passes through the web server to the panel are skipped by the web detectors, realtime and periodic, when cPanel’s own central access log records them as proxied. The panel’s own access log carries those requests, so a busy cPanel or webmail session through a proxy subdomain does not count as web traffic, and one request does not count as two kinds of evidence. Requests in a per-site log, or in a log chosen through web_server.access_logs that is not one of cPanel’s own access logs, always count as website traffic.

Web counters use the request and timestamp written by the web server, including requests with Basic authentication usernames and quoted header values. For periodic client attribution, web_server.trusted_proxies and challenge.trusted_proxies together enable X-Forwarded-For from a configured remote proxy; the log format must record it immediately after User-Agent, optionally preceded by a vhost in host:port form. Trust only proxies that append their observed peer or replace untrusted input with that address; trusted ranges must not include visitors. The client is the first address from the right that is not a trusted proxy. The parser keeps complete tokens at the right end, so discarded client text to the left cannot name a client. If no client survives, including when the retained end contains only trusted proxies, the request is not counted against anyone. A listed address is never counted as a client, even where its header is not trusted.

Realtime checks read the connecting address, not X-Forwarded-For, and act only on a bare address in its canonical form. When that address is a configured trusted proxy, in web_server.trusted_proxies or challenge.trusted_proxies, the WordPress, XML-RPC and admin panel checks skip the request, ModSecurity denies and warnings stay visible without escalating, and cPanel API auth failures are not reported. The scheduled ModSecurity audit scan does not report it either. Other cPanel audit and login findings stay visible without blocking or scoring the proxy, and supply no automatic response forecast. A listed address is skipped even where its forwarded header is not trusted, such as an address of this host. The web server must log the connecting peer, not a forwarded client, in that field.

A verified crawler is dropped from the scan before any counter increments: a source IP whose claimed bot User-Agent passes IP-range or reverse-DNS verification cannot contribute to a flood, scanner, or spoof finding, so verified Googlebot and AI crawler traffic does not produce false positives. Verification is covered in Threat intel.

FindingFires whenKey gates (defaults)
http_request_floodOne source IP makes too many requests inside the rate windowhttp_flood_threshold requests within http_flood_window_min (5 min). Ships disabled (0) so operators sample baseline traffic first
http_scanner_profileOne source IP’s in-window traffic is almost all probe-error responses spread across many distinct paths, the shape of URL enumeration hunting for backups, downloadable files, and dormant shellsThree gates must all pass: http_scanner_min_requests volume (ships 0, disabled), at least http_scanner_error_pct (90) of requests on a probe-error status, and http_scanner_min_distinct_paths (10) distinct error paths. Probe-error statuses default to 404 and 403; query strings are stripped so cache-buster URLs on one missing endpoint count once
http_ua_spoofOne source IP sends non-browser User-AgentsKnown scanner agents (nikto, sqlmap, nmap, wpscan, nuclei, and similar) fire on the first hit. Claimed crawler UAs with a cache-confirmed reverse-DNS negative, scripting (curl/python/wget), headless (Puppeteer/Playwright), and empty agents fire at http_ua_spoof_threshold (30); scripting/headless/empty still require their opt-in flags (http_ua_scripting_enabled, http_ua_headless_enabled, http_ua_empty_enabled)
http_distributed_floodMany distinct already-abusive source IPs hit one vhost in a single scan windowOpt-in: fires once http_distributed_min_ips distinct IPs (sample 10), each having already crossed a per-IP abuse threshold above, hit the same vhost. Built only from IPs that tripped another finding, so a popular site’s normal visitor spread does not trip it

Per-IP findings roll up per vhost, so a confirmed scanner that sprays a few probes across many vhosts still feeds the distributed rollup. Full threshold reference and tuning notes are in Configuration.

Direct SMTP Egress

Outbound connections to SMTP ports from non-MTA local processes emit a direct_smtp_egress finding. See Direct SMTP egress for the full rule set, config schema, and metric.

Critical Checks

Critical checks run every 10 minutes. Typical wall-clock cost on a busy shared host is a few seconds; the runner enforces the 10-minute cadence even when a tick takes longer.

The tables below name the finding identifiers CSM emits, grouped by area. Most are standalone scheduled checks; a few (for example the http_* traffic findings and bad_asn_outbound) are produced by a parent scheduled check – the access-log brute-force scan and the periodic connection scan respectively – rather than being independently scheduled.

Process & System

CheckDescription
fake_kernel_threadsNon-root processes masquerading as kernel threads (rootkit indicator)
suspicious_processesReverse shells, interactive shells, GSocket, suspicious executables
detached_web_processA non-root process running for at least five minutes that holds a PHP handler listener (LiteSpeed LSAPI or PHP-FPM) with no PHP or web server runtime in its ancestry: a web request started it and it outlived the request. A shell is Critical, the loop shape a webshell leaves behind; with kill_processes on it is stopped with its shell parents of the same account under the rules for loops that write back a removed file, unless a terminal or an SSH, cron or cPanel session is in its ancestry. BusyBox is treated as a shell only when running a shell applet. Anything else, such as a site’s PHP worker or an image job, is a Warning for review and is never stopped on this evidence. Processes with incomplete ancestry, and listeners in other network namespaces, are not classified
docroot_binary_execA non-root program running from a document root. The live exec monitor judges each start; this check judges programs still running. Critical (or the rule’s severity) when the program matches a YARA rule, whoever started it; Warning when a PHP interpreter started it and no rule matched. CGI programs run by the web server, build tools such as esbuild and other launchers stay quiet unless a rule matches. Program verdicts are cached by file identity and change time, scan-size limit, and detection version. Rule reloads and worker restarts invalidate cached verdicts. Oversized images and scripts receive only a header read. Scripts are not programs here: their interpreter lives outside the document root. Exited processes are inspected only through canonical absolute paths without symlinks; missing processes and files are quiet. Starts that arrive faster than the live monitor can inspect them are counted in the bpf.execution.docroot queue health row
php_processesPHP process execution, working dirs, environment variables
shadow_changes/etc/shadow modification outside maintenance windows
uid0_accountsUnauthorized root (UID 0) accounts
kernel_modulesKernel module loading (post-baseline). High. af_alg and algif_hash, which upgrades load on demand, are a Warning when a package was installed or updated since the previous scan; every other module, algif_aead (Copy Fail) included, stays High
af_alg_socket_useAF_ALG socket use that may indicate Copy Fail exploit activity
af_alg_enforcementAF_ALG hardening policy drift and correction status

SSH & Access

CheckDescription
ssh_keysUnauthorized entries in /root/.ssh/authorized_keys
sshd_configSSH hardening (PermitRootLogin, PasswordAuthentication, etc.)
ssh_loginsSSH access anomalies with geolocation
api_tokenscPanel/WHM API token usage
whm_accessWHM/root login patterns, multi-IP access
cpanel_loginscPanel login anomalies, multi-IP correlation
cpanel_filemanagerFile Manager usage for unauthorized access

Network

CheckDescription
outbound_connectionsRoot-level outbound to non-infra IPs (C2, backdoor ports)
user_outboundPer-user outbound connections (non-standard ports). System accounts for Knot DNS (TCP 853) and chrony or ntpsec (NTS key exchange, TCP 4460) are exempt on that port only; hosting accounts with those names are still reported
bad_asn_outboundOutbound connection whose destination resolves (via GeoLite2-ASN) to a bad or unexpected autonomous system. Config detection.bad_asn_outbound: blocked_asns (always bad) and/or allowed_asns (allowlist mode – anything outside is bad). Classified for every process including root (the periodic connection scan); non-root connections are also flagged in real time by the live BPF tracker. Off by default; the third leg of the host_takeover incident chain
dns_connectionsDNS exfiltration and suspicious queries
firewallFirewall status and rule integrity

Brute Force & Auth

CheckDescription
wp_bruteforceWordPress login brute force (wp-login.php, xmlrpc.php)
http_scanner_profileSource IPs whose traffic is mostly probe-error responses across many distinct paths in one scheduled scan window
http_claimed_bot_unverifiedHigh-volume claimed crawler traffic while reverse-DNS verification is still pending; challenge-routed when the challenge subsystem is enabled
http_ua_spoofIP hitting the per-IP spoof threshold with a search-engine bot UA that fails reverse-DNS verification, or with scripting/headless/empty UAs when those opt-in flags are enabled
http_distributed_floodMany already-abusive HTTP source IPs hitting the same vhost in one scheduled scan window
http_asn_crawlA single ASN distributing expensive, uncacheable requests across many addresses while one account’s PHP worker pool is saturated. Request classification is a heuristic, not proof of backend cost. Reverse-proxy ASNs and operator allowlists are excluded; Critical findings can carry the specific source CIDRs eligible for temporary blocking.
webshell_callerA client that requested a PHP file CSM flagged as a webshell, one finding per client and account, with request and POST counts, the files and a sample request. Requests keep counting after CSM removes the file. Known site entry points, files the installed WordPress or plugin release ships, and a document root’s own index are excluded from caller attribution. A request alone does not prove the client operated the shell; a compromised page can direct visitors there. cPanel hosts only. It reads the recent tail of each site log, so a single request on a busy site can be missed, and a file detected only in real time counts once a scheduled scan reports it. Alert only: no automatic response
webshell_path_publicA flagged PHP file requested by many distinct clients across all domains serving it. Infrastructure addresses and verified bots count toward this summary but never receive individual caller findings. Its clients are not reported one by one as webshell_caller
ftp_loginsFTP access patterns and failed auth
webmail_loginsRoundcube/Horde access anomalies
api_auth_failuresAPI authentication failure patterns

Webshell caller attribution uses dated requests from the last 24 hours and preserves the full request target before resolving dot segments and PHP path info. Above-root traversal is rejected. URL paths are decoded once, including encoded slashes, before resolving dot segments. For a directory request, a trailing decoded slash permits an index.php lookup. Double-encoded slashes are not decoded again. Attribution counts probes, including 404 responses, and does not prove that PHP ran. This assumes decoded path separators; server settings that preserve them, such as Apache’s NoDecode policy, URL rewrites and custom directory indexes are not inferred. When a client accesses the same file through several domains, its account finding does not name a single domain. These exclusions affect caller attribution only; file scanning is unchanged.

Email

CheckDescription
mail_queueMail queue buildup (spam outbreak indicator). Exim aborts when it cannot write its own log, and the daemon’s sandbox makes /var/log read-only, so the queue query runs as a transient unit forked by PID 1. Hosts without systemd-run query exim directly.
mail_queue_unavailableThe queue depth could not be read, so buildup detection is inactive. Reported instead of assuming the queue is empty.
mail_per_accountPer-account email volume spikes

All Exim queue reads and actions started by the daemon run as transient services outside its read-only filesystem sandbox. This includes queue composition, safe backscatter flushing, and PHP relay freeze or thaw actions. If systemd-run is unavailable, CSM runs them directly; a host without Exim skips the Exim-only queue check. The target command is attempted at most once, and cancellation during the wrapper probe stops it before execution.

Data & Integrity

CheckDescription
crontabsSuspicious cron jobs and scheduled commands
mysql_usersMySQL user accounts and privileges
database_dumpsDatabase exfiltration attempts
exfiltration_pasteConnections to pastebin/code-sharing sites

The MySQL audit exempts mysql@localhost only when MariaDB confirms its stock socket authentication setup with password authentication disabled. Modified authentication, other host literals, and accounts whose setup cannot be verified remain reportable.

Threat Intelligence

CheckDescription
ip_reputationIPs against external threat databases and optional rspamd history. Passive HTTP/cPanel sightings are High; SSH and mail-auth activity is Critical
modsec_auditModSecurity audit log parsing

The attack database counts a finding only against the source address the finding carries as a structured field. Addresses that appear only in message text, such as a connection’s destination, a process name or a file path, never build a score. Scores that earlier releases built from such text are corrected automatically when the daemon starts; blocks they already caused are not lifted. The correction has no setting; auto_response.block_ips controls new automatic blocks.

Raw mail and SMTP login failures are not counted. The mail and SMTP brute-force detectors decide those blocks, and they account for successful logins and mail authentication backend outages, so a mistyped password or a backend outage never raises an address’s local threat score.

At the upgrade restart, stored attack records are re-scored and retired scoring state is removed on the next successful save, including records whose score is unchanged. Existing counts and event history are retained, and existing blocks stay until expiry or operator removal. Rolling back before that save can restore the retired scoring state; an older daemon also resumes scoring new raw mail failures.

The local threat score ranks addresses on the threat page and feeds the unified verdict. No scheduled check turns it into findings or blocks; the checks that provide attack evidence apply their own response policies. No check produces local_threat_score findings any more: the database malware response reports the addresses of live WordPress sessions in its notice instead of blocking them, since the site owner’s sessions are among them. Findings older versions stored under that name clear after the next completed critical scan. A legacy disabled_checks: [local_threat_score] value is still accepted but does not disable ip_reputation or its health checks. Queued score findings and block retries under that name are discarded; existing blocks stay until expiry or operator removal. auto_response.block_ips controls new automatic blocks; there is no setting for the startup correction.

The local threat score retains evidence attributed to the server itself. Such records can represent forwarded attacks or a compromised local process; firewall protection against blocking the server does not establish that traffic is safe.

Performance

CheckDescription
perf_loadCPU load average thresholds
perf_php_processesPHP process count and memory
perf_memorySwap usage and OOM killer activity

Health

CheckDescription
healthDaemon health, binary integrity, required services

Platform Support

Runs on every supported platform unless noted below. The daemon auto-detects OS and panel at startup and silently skips cPanel-specific checks on plain Linux hosts (no “not found” spam).

cPanel-only (skipped on plain Ubuntu/AlmaLinux):

  • api_tokens, whm_access, cpanel_logins, cpanel_filemanager – read WHM API and cPanel session logs
  • wp_bruteforce – iterates /home/*/public_html/*/wp-login.php and per-domain access logs. The domlog pass ranks recent logs first and honors thresholds.domlog_max_files, thresholds.domlog_tail_lines, and thresholds.domlog_max_age_min.
  • webmail_logins – parses cPanel Roundcube/Horde logs
  • mail_per_account – reads /var/log/exim_mainlog

Plain Linux equivalents that still provide coverage:

  • mail_queue runs on any host where Exim is installed and is skipped when there is no Exim queue.
  • Access log brute-force detection (wp_login_bruteforce, xmlrpc_abuse) runs against the detected web server’s access log (/var/log/nginx/access.log or /var/log/httpd/access_log), so WordPress brute-force alerts still fire on non-cPanel hosts – they just rely on the live log watcher rather than per-domain domlog scanning.
  • modsec_audit runs on any host with ModSecurity installed.
  • ssh_logins, SSH brute force, PAM listener, firewall, kernel modules, RPM/DEB integrity, and threat intelligence all run on every supported platform.

Deep Checks

Deep checks run every 60 minutes and cover thorough filesystem, CMS, email, and database scans.

Scheduled and account scan checks share one budget within the daemon, sized from the machine’s core count with a minimum of two and a maximum of five concurrent checks. Web UI scans and CLI jobs submitted with csm scan --full share this budget. Standalone CLI scans run in a separate process. A cancelled or timed-out check retains its slot until it actually exits, while its caller can return promptly. Waiting for slots occupied by another scan does not count as stalled dispatch in queue health.

The real-time scanner has a separate pool of two to sixteen workers, also sized from the core count. Its two-worker minimum lets other events progress while one content scan is slow. Both the packaged and installer-generated service units give CSM a lower CPU weight than the default. Under contention, this favors sibling services with default weights while allowing scans to use spare CPU on an idle host; it is not a CPU quota.

Alert delivery

Alert email and the generic alert webhook share repeat state. A successful send through either channel counts as delivered. Findings held by the hourly cap, failed sends, or disabled notifications remain eligible on later reports, without waiting for the daily history reminder. This includes findings derived from current observations. Baselines, dismissals and suppression rules still apply, including to the sources of cross-account summaries. The signed phpanel stream keeps its separate delivery policy.

SMTP probe, self-deleting dropper, sensitive file write, email AV degraded and auto-block notice findings keep one alert identity per condition and severity: the source address, the removed file, directory or document root, the written file with its writer and, when the write can be tied to the file’s content, that content, the scanner condition, or the kind of notice. Counts, process or message IDs, ages, sizes, error text and the files sampled in a burst can change without a new alert. Dismissing one of these findings covers its later reports until it goes unseen for 24 hours. A return after that alerts again. Cross-account summaries alert again when the affected accounts change, including after a dismissal. More reports for the same accounts do not repeat the summary. Summaries are derived per dispatch batch, so different scan tiers or realtime batches can report different subsets of the affected accounts, and each subset is emailed once.

An undismissed finding can alert again after 24 hours without an observation, even when no other batch arrived to clean up old state. A finding that stays on the findings page counts as observed, also while its check skips runs between reports, so a check that reports once a day does not email daily. After at least 24 hours of downtime, an open finding alerts once on its next report. A completed scan that covered the finding and no longer reports it also permits a later return to alert. Scan coverage warnings follow the policy below.

Keeping acknowledged findings listed does not rewrite alert state, except for an acknowledgment that lapses with its condition as described above. Other retained findings use the existing periodic save, keeping their delivery state across restarts without changing the displayed last-report time.

Scan coverage alerts

PHP, JavaScript, YARA, host-wide WordPress database coverage, and unfinished email password verification warnings keep one alert identity while their counts change. Alert email and the alert webhook send them once; history records them again once a day unless acknowledged. When the owning scan returns without that condition, its acknowledgment clears, so a later recurrence can alert again. An unacknowledged condition that returns is emailed again only if it went unreported for 24 hours, so a gap that is missing from one complete scan and back in the next sends no new email. An unrelated scan, throttled check, timeout, or cancellation does not prove recovery. Disabling a scanner clears its current coverage condition. Existing per-install database limit acknowledgments are kept when a partial scan cannot inspect those installations.

JavaScript scans bound the paths retained for coverage gaps. If that list is truncated, the coverage warning says so and still counts every analysis failure. The scan keeps all earlier JavaScript detections and can report new ones. A later complete scan can clear findings for content it has examined or that is gone.

PHP analyzer crashes and stalls are tracked by the failing file paths, contents, and failure statuses, including files outside the displayed examples. Different failing inputs can therefore alert even after an earlier failure was dismissed. Check crash alerts ignore stack-trace churn, keep account scans separate by account, and re-arm after the same check returns successfully.

Filesystem

CheckDescription
filesystemBackdoors, hidden executables, suspicious SUID binaries
webshellsKnown webshell patterns (c99, r57, b374k, etc.)
htaccess.htaccess injection (auto_prepend_file, eval, base64 handlers) plus ten hardened per-pattern detectors – htaccess_php_in_uploads, htaccess_auto_prepend, htaccess_user_agent_cloak, htaccess_spam_redirect, htaccess_filesmatch_shield, htaccess_rewrite_off_shield, htaccess_header_injection, htaccess_errordocument_hijack, htaccess_cgi_handler_abuse, htaccess_security_disabled. Per-pattern findings use the per-pattern audit for cleaning and re-checks; generic findings use the generic scanner plus the per-pattern detectors for their manual fix and re-check. Realtime detection uses that combined audit. Auto-cleaning is gated by auto_response.clean_htaccess, acts on the per-pattern findings and on htaccess_handler_abuse, whose extensions no site uses, and removes only their directives; other generic htaccess_injection findings keep the manual fix. No .htaccess finding is handled by file quarantine. Every prepend or append target carried in a line is judged, including SetEnv values and RewriteRule environment flags with variable names expanded by Apache. Apache quoting and flag separators are not part of the target. Harmless prepend settings and defensive rewrite patterns are not reported, and handler-abuse findings require a complete extension argument in an active handler directive, with one finding per directive. Disabling CGI suppresses CGI handler mappings only when the active options keep CGI disabled; PHP mappings remain reportable. Automatic cleaning requires removable directives, so retained preludes stay reported without entering the automatic response budget. Manual removal of a malicious rewrite also removes its condition chain so the conditions cannot affect the next rule. Oversized files stay unchanged and require manual inspection; re-checks cannot declare them clean. htaccess_security_disabled covers the mod_security 1.x spellings too, because LiteSpeed obeys them by default; Magento’s stock block that turns ModSecurity off is reported and cleaned like any other. htaccess_rewrite_off_shield reports rewriting switched off beside an access grant whose FilesMatch pattern can match no file name, the unfilled template a dropper kit writes; asset folders that grant named file types stay quiet. Only active access grants count, including quoted and continued directives. Cleaning preserves unrelated bytes and removes an enclosing rewrite module block only when the offending switch is its entire body. A prepend of a security plugin’s own file is not reported. For Really Simple Security the target must be the absolute path of wp-content/advanced-headers.php beside the .htaccess, with no dot segments; a document root reached through a symbolic link counts as the same place. Any other target naming that file is reported wherever it points, and when it is a prepend it is never cleaned automatically.
file_indexIndexed file listing to detect new/unauthorized files
php_contentSuspicious PHP functions (exec, eval, system, passthru)
group_writable_phpWorld/group-writable PHP files (privilege escalation)
js_keylogger_dataflowJavaScript keystroke exfiltration found by AST data-flow analysis: a key-event value that reaches fetch, sendBeacon, XHR, WebSocket, or a resource .src sink, including flows laundered through variables that regex rules cannot express. Runs inside the scheduled deep scan on the same file snapshots as the YARA pass but independent of the YARA backend, with its own rolling cursor. Complete sources up to 2 MiB are analyzed; a file the analyzer could not complete (oversize, parse failure, resource limit) is reported in the Warning-level js_taint_scan_incomplete aggregate and its prior finding is preserved, never silently cleared. Recognized PHP and HTML documents (including HTML-first PHP templates), complete JSON/source maps, CSS stylesheets, and gettext catalogs are skipped rather than counted as JavaScript parse failures. Classification uses content, not file extensions, and valid candidate JavaScript is always analyzed even when comments or strings contain document markers. Embedded JavaScript in templates or data files is not extracted or analyzed; PHP analysis and content rules do not provide equivalent JavaScript data-flow coverage. Ambiguous sources and malformed JavaScript retain their coverage warning, and oversize sources keep the size-limit warning. Disable with js_keylogger_dataflow (or the owner ID js_taint_deep) in disabled_checks; disabling the YARA scan does not disable this analyzer.
symlink_attacksSymlink-based privilege escalation attempts
exposed_filesWeb-downloadable sensitive files under document roots: version-control directories (.git/, .svn/, reported as web_exposed_repo_metadata and virtual-patched by denying the whole directory), database dumps, full-site backup archives, config/credential backups, PHP source-code backups, and phpinfo.php diagnostics. Each candidate is reported only after a headers-only reachability probe pinned to the vhost’s configured serving IP confirms the server serves it (HTTP 200/206, non-executed body) – files the server already blocks (403) and shipped samples such as wp-config-sample.php are never flagged. The domain is preserved for HTTP Host and TLS SNI, while DNS and front-end proxies are bypassed. The probe reads status and content type only, never the file body. The one exception is phpinfo.php: a 200 response alone cannot prove a dump, so CSM reads a bounded portion of that one response and reports web_exposed_phpinfo only when it contains real phpinfo output (the PHP Version banner in a dump-sized body); stub responses are not findings. Findings: web_exposed_db_dump, web_exposed_backup_archive, web_exposed_config_leak, web_exposed_source_backup, web_exposed_phpinfo, web_exposed_sample_sql. A plain SQL file with a sample/schema-specific name under framework/vendor scaffolding (examples/, docs/, vendor/, unpacked *-main/ or *-master/ directories) is reported as the lower-severity web_exposed_sample_sql warning. Archived, renamed, customer-named, and other ambiguous dumps stay Critical. Descent depth is bounded by thresholds.exposed_file_scan_depth (default 2, maximum 10). Optionally, auto_response.virtual_patch_exposed_files (off/manual/auto) writes reversible .htaccess Require all denied rules – manual applies confirmed findings via csm virtual-patch --apply, while auto applies every confirmed class except warning-only sample SQL and honors dry_run. Recognized backup storage directories are denied as a unit so regenerated archives stay blocked. A zip whose name carries no backup wording is classified from its central directory instead, so an archive named after the site it holds is reported when it contains a site-root or structured CMS configuration path, a regular Joomla configuration file at any depth whose same case-sensitive folder also holds a regular Joomla core file, a document-root directory, or a database dump at the archive root; plugin and theme bundles offered as ordinary downloads stay quiet. Readable backup evidence is retained even if later directory records are damaged. The complete entry list is inspected within a fixed metadata budget, without extracting or decompressing payloads; an archive that exceeds the budget leaves the scan incomplete instead of silently clearing an earlier finding. Each change records a rollback entry; restore refuses to overwrite a later customer edit. Findings in this family can be re-checked, and the version-triggered startup sweep re-checks existing findings after verifier changes. A finding clears only after a complete vhost map provides an unambiguous current serving address, both origin protocols answer there, and the shared detection rule no longer confirms an exposure; phpinfo re-checks repeat the bounded body confirmation too. A missing local file alone is not enough. The re-check never falls back to DNS, so a domain that has migrated to another host leaves the finding open rather than clearing on a stranger’s answer, and incomplete routing data or an unreachable or half-answered probe leaves it open too.

The webshell, .htaccess, phishing and filesystem scans run on every deep cycle alongside the file index, including while the realtime monitor is active: the monitor does not report a setuid bit being set, and it sees files renamed or linked into place only where a placement backend runs. The exposed-file scan runs in that tier too, on its own interval, because it confirms each candidate with a live request to the site rather than by reading the file. A cycle that skips it on that interval leaves its existing findings in place. The interval starts only after a complete exposure scan; an incomplete attempt can retry on the next cycle, whether or not the realtime monitor is active. A deleted document root does not make the exposure scan incomplete, even if the panel still lists its domain. A missing root reached through a symbolic link, and a subdirectory that disappears during the walk, keep earlier findings until a later scan completes.

Account discovery failures, unreadable directories or files, and truncated backdoor candidate lists leave the affected scan incomplete. Its earlier findings remain active until a complete scan can replace them; detections from the incomplete attempt are still added, whether or not the realtime monitor is active.

The file index runs on every deep cycle, including while the realtime monitor is active. It checks new paths and rechecks indexed files with active findings; being present in the baseline does not clear an alert. Failed content reads preserve prior findings for that file. After a large deletion, scans rewalk the directories until the smaller baseline is adopted, so cached entries cannot restore removed paths or interrupt the consecutive-scan shrink guard.

The PHP content scan reuses a clean result only when the file’s identity and timestamps still match a stable read. Recently changed files, files that change during inspection, and files without usable identity metadata are read again on the next visit. Older cached results are refreshed as files are visited after upgrading; interrupted scans retain progress. Periodic host scans bypass the cache, and explicit full-content scans always read the files they visit.

PHP execution heuristics distinguish attribute metadata and multiline string contents from executable calls. Literal examples do not establish callable bindings or invoke them, and scanning continues through code after attributes. String boundaries and legacy line endings preserve the executable code that follows them. Attribute declarations do not qualify as comment-only stubs for demotion.

PHP and phishing checks read ${'_POST'} and $GLOBALS['_POST'] as $_POST. The same text inside a quoted string or a comment is ignored.

Phishing title analysis ignores a brand that appears only inside an email address, as on a blog author archive named after its author, even when the page also carries a login form. Elsewhere in the title, password capture or a login title lets a brand glued to other words still count. Malformed address-like text does not hide brand evidence.

Re-verifying Findings

Repeated YARA worker buffer-scan failures are logged at most once per minute per tracked error, with suppressed counts reported when the error recurs after that window. If many distinct failures exhaust the tracking limit, additional errors share a bounded summary instead of resetting suppression. Scan errors still reach callers and preserve incomplete-scan reporting.

A finding records what was true when it was raised. The condition behind it is often resolved by someone else – an operator cleans a file, a virtual patch denies an exposure – and none of that moves CSM’s own rules, so re-verification runs once per deep-scan cycle as well as when the re-check logic changes.

A scan retires a finding it did not raise again only for files it actually examined. Coverage is tracked per file and scanner: a gap that names a file – one past the scan size limit, or one that could not be opened – keeps that scanner’s findings for the file and nothing else, while a gap with no path, such as a directory the walk could not enter or a stat that may hide a subtree, keeps every finding the scanner owns, because the unscanned range is unknown. This matters more than it sounds: a single oversized log that will never fit under the limit is a permanent gap, and treating it as a whole-scanner gap froze every finding on the host indefinitely. On a large server the rolling deep scan covers the files over several runs, so each run retires findings for the files it examined, up to where its time budget ran out. Overlapping and symlinked scan roots are walked in one path order, so an interrupted run never retires findings in a root it has not reached. A finding raised by an older version is not tied to its file this way, so it waits for a complete scan or re-verification.

Each family is re-checked by re-running the test that raised it, giving four outcomes:

  • Cleared. The condition is provably gone: the file was removed, or the server no longer serves it as an exposure. If the same detection returns later, it alerts as a new finding.
  • Demoted to Warning. The flagged content is gone but the file changed since detection, so the cleanup cannot be proven byte-for-byte. The finding is kept – an attacker must not be able to retire one by editing the file – but it stops ranking beside live threats. Only a replacement proven inert qualifies: an empty file, or a comment-only PHP stub with a valid opening tag and no closing tag. A malformed opening tag can leave the file serving page content, so it does not qualify for demotion. This is an allow-list on purpose. A deny-list of dangerous shapes would make every omitted include, callback or inline script a way to buy a lower severity.
  • Restored. A demoted finding whose replacement stops being inert returns to the severity it came from, so a second edit into a detection gap cannot leave live content sitting at Warning.
  • Left alone. Anything uncertain: an unreachable or half-answered probe, a domain no longer served by this host, a replacement that changed while it was being read, a file that still matches.

A finding keeps its identity throughout; only its severity changes, because a finding’s key is derived from its details. Every store mutation is conditional on the snapshot verification actually examined, so a scan or realtime alert that refreshes the same key while a re-check is in flight is never overwritten by the older verdict. An unconfirmed demotion also survives a scan that does not raise the finding again, since that scan is weaker evidence than the verifier that reads the file.

PHP Remote-Source Taint Analysis

This analyzer runs as part of the scheduled deep-content scan, alongside the YARA and JavaScript consumers, and reports php_remote_taint. Disable it with php_taint_deep (or php_remote_taint) in disabled_checks.

It runs in a separate process, and that is not an implementation detail. A single file can put the underlying PHP parser into a state it never returns from, which no in-process deadline can interrupt. Analysis therefore happens in a supervised child process that is killed outright when a file exceeds its deadline; that file is reported as reduced coverage and the scan continues. Repeated failures are rate-limited so no single account can consume the scan budget, and the analyzer recovers without operator action. Whenever a file cannot be examined – including when no worker is available – it is reported as unexamined, never as clean. The byte check that decides whether a file could hold such a flow at all needs no parser, so it runs in the daemon itself: a file without a PHP open tag, a code-execution construct and a remote-fetching call is ruled out there and never reaches the worker. Cancellation during this byte check is still reported as unexamined.

CSM includes a PHP analyzer that looks for code fetching content from a remote server and then executing it, even when the fetch and the execution happen in different functions. This is a flow that regular expressions cannot express: matching it requires binding the value a fetch returns to the value handed to execution, which is beyond what YAML pattern rules or YARA-X can do.

The analyzer parses PHP source and tracks whether a value returned by a remote-fetching call reaches a code-execution construct (eval, include, include_once, require, require_once, create_function, or assert given a string argument), following the value through variable assignment, string concatenation, decoding calls, across function and method boundaries, and through a file_put_contents write into an include of the same path expression. curl_exec, curl_multi_getcontent, wp_remote_get, wp_remote_retrieve_body, and fsockopen are always treated as a remote source, whatever their argument. file_get_contents and fopen are dual-use: only these two calls have their argument inspected, and only when that argument carries an HTTP, HTTPS, FTP, FTPS, php://input, or data:// scheme are they classified as a remote source rather than a local read. Not every php:// stream counts as remote: php://input is request-controlled and treated as a source, while php://memory, php://temp, and a local php://filter resource are not – though a filter wrapping a remote resource still carries its nested remote scheme and is classified accordingly. A URL used directly as a sink’s own argument, with no acquiring call in between – include 'http://evil/x.php' – is out of scope: the pre-filter requires a source keyword before parsing is even attempted, and a bare remote include has none, so it is reported not_candidate, not analyzed.

Only two outcomes mean a file was actually examined: analyzed (parsing and the data-flow pass both completed) and not candidate (a fast pre-check proved the file cannot contain a reportable flow, without needing to parse it). Every other outcome – oversize, a parse failure, a parser recovery that produced only a partial tree, an internal resource limit, cancellation, or an internal error – is a coverage gap, not a clean result, and must never be read as “nothing to see here.”

An oversize file is judged once more before it counts. Only a file whose leading bytes could be source of that language is reported. Both analyzers are handed every readable file the walk produces, and their own pre-check rejects the rest instantly – but that pre-check never runs on a file too large to send, so without this step every large image, archive and compiled catalog on the host arrived as source the scan had failed to examine. PHP is judged by an opening tag anywhere in the inspected prefix, including after binary content. Binary content carrying PHP tokens can therefore remain reportable; the prefix alone cannot establish that it is inert. JavaScript has no such marker. A prefix without NUL is admitted; one containing NUL gets a conservative lexical check, since binary characters are legal in JavaScript literals and comments. Incomplete tokens and ambiguous syntax remain reportable. Neither gate looks for taint source or sink keywords in the prefix. A read that fails answers yes either way, because a file the scan could not examine is exactly what the report exists to name.

A finding’s severity follows how firmly the source was shown to be remote: a decoder-confirmed remote fetch reaching execution is Critical, a fetch carrying a remote URL is High, and a dual-use call whose argument could not be resolved either way is a Warning for review. When one file contains several flows, its strongest flow sets the severity. The last group is where legitimate template compilers and cache layers land, so it is kept visible without paging anyone.

Each flow in a finding’s details reads source -> sink (confidence, basis), for example curl_exec -> eval (high, always-remote) or file_get_contents -> include (low, unresolved). The basis says how the source was identified: always-remote (the call can only read over the network), literal (the argument text carries a remote scheme), decoded (the scheme appears only after escape or builtin decoding), request (a requester can supply the start of the path), call-argument (a call site in the same file passes the remote argument), or unresolved (the analyzer could not decide whether the argument is local or remote). The basis describes the strongest proof that reaches the sink, which can come from a different source call than the one the flow names. Some basis values appear only in later versions.

A finding’s identity is its file, its severity and the source and sink pairs of its reported flows. A new version that changes only the details wording or the basis does not show a dismissed finding again. Any change in those flows or severity, a new file, or a change to the file’s content does. When evidence is capped, basis ranking does not change which equal-confidence flows are retained.

The WordPress database scan reports what it could not fully inspect as db_content_scan_incomplete, counting affected installs against the number discovered and naming one bounded example config path per reason: unreadable_config, missing_credentials, unresolved_table_prefix, query_failed or incomplete_content. Example paths use ASCII escapes for control characters, non-ASCII bytes, quotes and backslashes, with the existing length limit applied after escaping. Content gaps include truncated or unusable query results. A server regular-expression timeout is a statement-local failure: independent checks continue, while the failed check keeps coverage incomplete and preserves earlier findings. Hidden-link selection examines only the leading and trailing parts of each value that the parser reads, so large values cost no more than the parsed sample. When the parser joins those parts into a complete value, selection examines them together too. Ordinary CSS declarations are matched with literal searches; guarded expressions handle commented and encoded styles without admitting unrelated page text. Comment bodies are preserved during matching so normalization cannot change their meaning. If the server stops a regular expression, the selection is repeated without commented styles, so plain and encoded styles stay covered while the coverage gap is still reported. Each install counts once; installs sharing a failed database count as affected without retrying its queries. Incomplete discovery is reported separately in the details because additional installs may be missing from the total. With no attributed reasons, including when discovery stops before reaching any install, the generic three-cause sentence remains the fallback. Multisite safety limits keep their own account-specific warning and do not count again in the summary or hide unrelated failures. Installation findings carry an opaque database scope; the host-wide coverage summary remains unscoped. A complete installation scan retires its resolved findings even when another installation fails. Failed or undiscovered installations retain their findings in the same atomic store transaction. Older findings without a database scope stay until reobserved or until the whole scanner completes; CSM does not guess their database from message text. A partial multisite scan keeps findings for its entire network. Incomplete or interrupted scans protect earlier findings from eviction when new results fill the active list, including when every database or discovery attempt fails, the scanner times out, or an internal panic stops execution. That protection applies to findings already in the active list. Newly detected conditions compete for the remaining space under the normal priority order, so repeatedly incomplete scans cannot grow the list beyond its cap. Retained findings can still refresh their details without losing their first observation. Credential aliases sharing a database scope must all complete before that scope can retire findings, regardless of scan order. This includes an alias whose database and table prefix are known but whose login credentials are missing.

Query diagnostics include the detector stage, failure class and numeric MySQL error code. Repeated errors are counted together, with bounded detail when many causes occur. Raw SQL, server error messages and credentials are never included. Known statement errors, such as a missing table or column, leave the scan incomplete but allow independent detectors to continue. Connection, authentication and unknown failures stop further queries for that database.

Coverage the scan could not reach is reported as php_taint_scan_incomplete, which names how many files were affected and why – a per-file status such as a timeout or a worker failure, or a location the walk could not read at all, where the affected files cannot even be listed. Panics and timeouts are reported in a separate aggregate so hard analyzer failures remain visible beside routine coverage limits. A file that had a finding and later becomes unexaminable keeps its previous finding rather than having it cleared.

Separately from the outcome, an analyzed file can still report reduced precision. Constructs that defeat static variable identity – extract(), compact(), variable variables, a call dispatched through a value, an assignment target the analyzer cannot name, or a value a closure or arrow function captures from its enclosing scope – are recorded alongside the result. A recorded loss means tracking stopped at that point and the file may hold a flow that was not followed; it is never left implicit. The capture case is recorded only when the captured value was itself tainted, so it marks a real loss rather than the mere presence of a closure.

The parser supports PHP syntax up to version 8.1. A file written against a newer PHP version may use constructs the parser does not recognize; when that happens, parsing recovers what it can but the result is incomplete, and the file is reported as reduced coverage (a partial parse) rather than analyzed, so an incomplete view is never presented as a complete one.

WordPress

CheckDescription
wp_coreCore file integrity via official WordPress.org checksums. A command timeout does not mark files as verified, and a timed-out re-check stays unresolved. Repeated verification failures produce wp_core_unverified. A core file proven to differ from the release only in line endings is not reported, and an install with no other difference counts as verified. Standalone PHP deprecation notices do not block this comparison; PHP warnings and ambiguous output still do. For any flagged file not proven so, the daemon log says why at Info level, without copying command output. Incomplete checks stay unresolved.
wp_plugin_inventoryReports persistent plugin inventory failures as wp_plugin_inventory_unverified. Shares the existing refresh and interval with the outdated and vulnerable plugin checks.
nulled_pluginsCracked/nulled plugin detection
outdated_pluginsPlugins behind the latest release, graded by version gap
vulnerable_pluginsInstalled plugins whose version matches a curated known-vulnerable feed (CISA-KEV + confirmed in-the-wild CVEs). Fires only from a fresh shared inventory when a parseable version is inside the affected range; patched, stale, and unparseable versions stay silent. Matched versions are High or Critical regardless of version gap, including inactive plugins because their files remain reachable. Alert-only (never disables a plugin). Toggle detection.vulnerable_plugin_scanning; accept one reviewed build via detection.vulnerable_plugin_allow (slug@version, case-insensitive). For an active install, a host engine that is Off or DetectionOnly, or a disabled account or vhost scope, marks the finding unprotected and raises it to Critical: no request filtering applies and no modsec audit record is written. When CSM ships a virtual patch for that CVE (virtual_patch in the feed) the alert says that patch cannot run; otherwise it claims only the filtering gap. An inactive install is never annotated – its files are why it is reported, but WordPress does not load the vulnerable code path, so a missing filter says nothing about reachability. An addon domain is linked only to its exact cPanel-associated subdomain; unrelated parked, main, and addon sites that happen to share a document root do not inherit one another’s disabled flags. Off cPanel, where CSM deploys no virtual patches and no per-vhost ModSecurity state exists, findings are left as the detector graded them.
vulnerable_timthumbBundled TimThumb (timthumb.php / thumb.php) image-resizer scripts, the abandoned library whose remote-code-execution bug (CVE-2011-4106) is a recurring WordPress entry point. Files are confirmed by TimThumb’s own constants (not just filename) so generic thumbnail helpers are never flagged. A version below the last patch (2.8.14), an unparseable version, or an enabled WebShot / external-fetch feature is reported High; a patched-but-deprecated copy is a Warning to remove. The final release is suppressed only when ALLOW_EXTERNAL, ALLOW_ALL_EXTERNAL_SITES, and WEBSHOT_ENABLED are defined as false in executable PHP and none is also defined as true. Alert-only – TimThumb is never auto-quarantined, since deleting it would break the theme.
db_contentDatabase injection (an external script loader on an ordinary HTTPS host that carries no attacker marker is reported once, as a Warning db_options_new_external_script, the first time that host appears in an option after the site’s first scan; plugin status options that WordPress renders as dashboard notices – LiteSpeed Cache’s CDN setup summary and its stored message list – are read by name instead and reported Critical as db_options_plugin_notice_injection when they carry markup that executes, because those rows hold plugin state and never site content, so neither host reputation nor the first-seen baseline applies and a payload stored before the site’s first scan is still reported. The named notice rows are read without a script-only SQL filter; complete values up to 64 KiB are inspected, while oversized or malformed query rows mark the scan incomplete and retain prior findings. Notice findings are alert-only; stored JSON and serialized data are not rewritten by this check. If another finding triggers option cleanup, cleanup refuses to write while any executable notice markup remains, even when its host has no reputation marker. A value WordPress stored as JSON has every slash escaped, so script URLs are normalised before a host is extracted, in options, in posts, and in page-builder content), siteurl hijacking, rogue admins, spam, a sudden publishing flood measured against the site’s own history (db_post_volume_burst; only on sites older than 400 days, with at least 50 recent posts and more than 5x everything published before – a genuine content migration looks the same, so it is alert-only), spam categories, tags and other taxonomy terms left behind when spam posts are removed (db_spam_taxonomy; a term named after a URL is High, spam vocabulary alone is Warning), PHP-capable snippets stored in the database by WPCode (db_stored_code_execution, Critical when the snippet is published and therefore running, lower when it is a draft or trashed; stored code is invisible to filesystem scanning), stored snippets that both defeat request caching (true DONOTCACHEPAGE, false WP_CACHE, WordPress no-cache headers, or LiteSpeed’s no-cache response header) and inspect the user agent for a search or SEO crawler (db_stored_cloak_logic, High when published and Warning otherwise; direct, concatenated, and ROT13 crawler names are recognised, while a snippet that already matched a malware signature carries the cloak as a bounded note rather than raising a second finding), outbound links wrapped in a container the page hides from readers (db_hidden_link_injection; an off-canvas container – a large negative left/top/text-indent – is High on its own, while display:none, visibility:hidden and fully transparent opacity need corroboration from a second linked domain in the same container or spam vocabulary, because themes legitimately hide panels that link out. Links back to the site’s own registrable domain are ignored, and the scan is skipped when siteurl/home cannot be read, since every absolute link would then look external. Markup is tokenized rather than parsed into a tree, so nesting the injection past a tree parser’s open-element limit does not hide it), autoloaded options named by a 32-character hex digest whose value base64-decodes to one complete PHP-serialized array (db_hostname_keyed_option; kits key the digest to the site’s own hostname so one payload serves many sites, and both halves are required because plugins do write hashed cache keys and base64 alone is ordinary), rewrite rules pairing an exact sitemap<N>.xml route with feed=xmlsitemap<N> in the same rule (db_doorway_sitemap_routes; sitemap plugins add numbered rewrite rules too, so the matching number is what distinguishes a doorway cluster from paginated sitemaps). Bounded option reads that omit any stored bytes mark the scan incomplete, and that row does not produce a finding. The scan also reports published posts attributed to a user that does not exist (db_phantom_post_author, Critical at 100 posts or more and Warning below). Small orphan groups can result from direct SQL user deletion; large groups can identify doorway content whose invented author IDs are hidden from dashboard queries. Direct WordPress installs are covered at every document root the panel serves, at every addon-domain directory in an account home, and one directory below a document root, so a root the panel has stopped serving is still examined, and every finding states which of the two it came from – served roots are reachable now, unserved ones hold a live database that publishes again as soon as a domain is pointed at them, and the statement is omitted entirely when the panel’s domain map could not be read rather than guessing; account backup, cache, staging and metadata directories are not, and installs deeper than one directory below a document root are not discovered recursively. Discovery is shared with the object, admin-overlap, credential-reuse, core-integrity and plugin checks and with their fixers and re-checks, so a finding raised here can always be re-located by the code that resolves it. A site address is reported when its shape cannot be one – a missing host, an invalid port, a backslash, a query string, a fragment, a script for a path, or a non-web scheme – because WordPress builds every asset URL from it. Invalid-address findings are alert-only; only explicit code injection enters database auto-response. Serving a site under a domain hosted elsewhere is ordinary and is not reported on its own – but a root the panel is serving right now, whose address names a domain absent from that account’s panel domains, is reported once per site as db_siteurl_foreign_host (High). Ownership follows the panel’s most-specific exact or wildcard domain mapping, so a delegated subdomain belongs to its own account rather than the account holding its parent. A migrated site is no longer served here, so the served state is what separates the two; an unserved root, an incomplete domain map, or a value the shape check already reports stays silent rather than saying it twice. Multisite-aware: when wp-config.php declares define('MULTISITE', true), up to 100 active secondary blogs (wp_<N>_options / wp_<N>_posts for blog IDs from wp_blogs) are scanned alongside the unprefixed main-site tables; each blog’s PHP-capable WPCode snippets and taxonomy are scanned, and its posts are checked against the network-wide users table. Larger networks emit db_content_scan_incomplete and retain prior findings.
db_content_joomlaJoomla database content scanning. Discovers installs via configuration.php containing class JConfig, parses credentials from public $...; assignments. Scans <prefix>extensions params, <prefix>content article bodies, and joins <prefix>users with <prefix>user_usergroup_map for Super User detection (group_id=8). Findings: joomla_extensions_injection, joomla_content_injection, joomla_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too.
db_content_drupalDrupal 8+ database content scanning. Discovers installs via sites/default/settings.php plus the core/lib/Drupal.php marker. Credentials parsed from the $databases array. Scans config, node_revision__body, and users_field_data joined with user__roles (administrator role). Findings: drupal_settings_injection, drupal_content_injection, drupal_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too. Drupal 7 not yet covered.
db_content_magentoMagento 1.x and 2.x database content scanning. Discovers installs via app/etc/env.php (M2, preferred) or app/etc/local.xml (M1). Credentials parsed via encoding/xml for M1 (CDATA-aware) or field-level regex for M2. Scans core_config_data, catalog_product_entity_text, cms_block, cms_page, and admin_user (with the configured db.prefix). Findings: magento_settings_injection, magento_content_injection, magento_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too.
db_content_opencartOpenCart database content scanning. Discovers installs via the config.php + admin/config.php pair both containing define('DB_DRIVER'. Credentials parsed from DB_HOSTNAME / DB_USERNAME / DB_PASSWORD / DB_DATABASE / DB_PREFIX defines. Scans <prefix>setting (config_url / config_ssl are canonical hijack targets), <prefix>product_description, <prefix>information_description, and <prefix>user (admin/staff). Findings: opencart_settings_injection, opencart_content_injection, opencart_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too.
db_content_prestashopPrestaShop database content scanning. Discovers installs via config/defines.inc.php containing _PS_ROOT_DIR_. Credentials come from the six database_* entries of app/config/parameters.php (1.7.4 and later, 8 and 9), or from the 1.6 config/settings.inc.php defines when neither parameters.php nor parameters.yml exists; no other configuration value is read. A shop on 1.7.0 to 1.7.3, which keeps its live settings in parameters.yml, leaves the scan incomplete. Staging, backup and cache copies and suspended accounts are skipped. Scans <prefix>configuration and <prefix>configuration_lang values, <prefix>cms_lang content, <prefix>product_lang descriptions and short descriptions, and <prefix>category_lang descriptions in every language and shop, with one finding per setting, page, product or category. Setting values are never shown, because the mail password is stored among them. Findings: prestashop_settings_injection, prestashop_content_injection, prestashop_admin_injection. Back-office employees are baselined on the first pass; an employee that appears later is reported once as High, and a new employee with the SuperAdmin profile, or an existing employee moved to it, is reported once as Critical. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too.
db_objectsMySQL persistence mechanisms: triggers, events, stored procedures, stored functions. Critical when the body matches known-malware patterns (sys_+exec, INTO OUTFILE, LOAD_FILE, etc.); Warning when an object exists at all (vanilla CMSes ship none). Toggle with detection.db_object_scanning; suppress Warnings via detection.db_object_allowlist. Manual drop via csm db-clean --drop-object.
admin_overlapWordPress administrator email overlap across cPanel accounts. Reports when the same admin email appears on the configured number of accounts, with reviewed emails and domains suppressible in detection.
credential_reuseWordPress administrator password-hash reuse across cPanel accounts. Groups identical hashes with an in-memory fingerprint and reports only the affected accounts and count.
supply_chainComposer and npm lockfile advisory matching against the local advisory database. Silent when no advisory file is present.

Core verification and plugin inventory report persistent failures separately from malware or integrity findings. A failed attempt first appears in status; repeated failed attempts in distinct scan cycles raise a Warning naming the installation, last attempt and a bounded reason category. A missing wp-cli or an unreachable checksum service stops every installation on the host at once, so when one reason covers more installations than the per-cause limit a host scan collapses the warnings into a single finding naming that reason, the total and a sample of the paths. The limit applies across accounts for each kind of verification and cause. Account scans keep installation-specific warnings. A host summary belongs to an account only when every affected installation shares it; sampled paths do not determine account attribution. Raw wp-cli output is never stored in this history. These warnings do not enter incidents or automatic remediation. The history survives restarts, and successful checks clear the warnings through the normal completed-scan merge. Overlapping scans preserve attempt order and failure streaks even when they finish out of order. New discovery alone does not discard an attempt still finishing in another scan.

Cache reads do not count as attempts. Plugin coverage uses thresholds.plugin_check_interval_min; partial refreshes keep the existing refresh policy. disabled_checks controls these scheduled checks, and plugin coverage performs no refresh when both inventory consumers are disabled. Cancellation and incomplete discovery retain earlier evidence. Complete discovery removes installations that are no longer present, scoped to the account scanned. A domain the panel still lists after its directory was deleted holds no installation and does not make discovery incomplete when the original mapped config path is also confirmed absent. Ambiguous symlink resolution, account boundary violations, and inspection errors keep discovery incomplete and retain earlier evidence. A wp-cli refusal explicitly identifying a non-WordPress directory is recorded as not applicable rather than an outage. A completed core check that finds modified or missing files retains its existing integrity findings and is counted separately. Only the completed checksum summary establishes a negative verification result; partial warnings without that summary remain unverified.

csm status, csm status --json and /api/v1/status report the last observed coverage for core verification and plugin inventory. These counts describe previous attempts, not a fresh scan of every site. See verification status.

WordPress content findings use the account, database server, database name, and table prefix to stay separate. Changes to the panel’s document-root map do not create duplicate findings. Publishing a previously inactive suspicious snippet, or an orphaned-post group growing into a Critical content farm, produces a fresh alert even when the earlier condition was baselined or dismissed. Ordinary count changes within the same orphaned-post severity tier keep the existing identity.

Hidden-link findings are tracked per account, database host, database, and table prefix. More affected rows or a different row order do not raise another alert for the same destinations and concealment strength. Off-screen concealment raises a new High even after an earlier Warning was baselined or dismissed. Changes to destinations outside the displayed sample also produce a new finding. The corrected identity can show existing hidden-link findings once more after an upgrade; previous dismissals do not transfer to the new identity.

Hidden-link corroboration counts registrable domains within one hidden container, not hostnames across a whole row. Multiple subdomains of one linked domain count as one target, and both the WordPress home and site addresses count as local.

Joomla, Drupal, Magento, OpenCart, and PrestaShop configuration reads accept regular files up to 1 MiB. Configuration symlinks and special files are rejected, including during Joomla, OpenCart, and PrestaShop marker probes. Drupal’s version marker must also be a regular file. Reads use the opened file throughout, reject changes observed during the read, and stop when the scan is canceled. Read failures and missing required credentials mark that CMS scan incomplete; manual re-checks keep the finding unresolved when its configuration cannot be inspected.

Joomla, Drupal, Magento, OpenCart, and PrestaShop scans skip suspended accounts before reading their configuration or applying the account scan file limit: a suspended account’s database user is locked, so scanning it would only leave the check incomplete. Both Magento configuration layouts use this filter. Only installs with a resolved owner and a cPanel suspension marker are excluded; unowned paths remain eligible. Active accounts still scan when others are suspended. PrestaShop also skips copies named staging, backup, backups, cache and .trash, matched without regard to case.

CMS re-checks look at every install of the account in the layouts the scanner discovers, the account’s public_html and each folder directly under its home, using each install’s own database login and endpoint, including custom ports and sockets. Administrator findings and every PrestaShop finding record their database host, database name and table prefix, and are re-checked only in that install. Joomla, Drupal, Magento, and OpenCart settings and content findings do not record it, so they resolve only when the row is clean or absent in every install of that CMS on the account. An unreadable install, a recorded database that no install uses, configuration failures and query errors leave the finding unchecked. Home directory listing and install discovery errors also leave it unchecked. Links in the home folder that point nowhere or to regular files are ignored; symlink loops and permission or I/O errors leave the finding unchecked. Re-checks inspect every relevant database even after finding an injection, so a later query failure still leaves the finding unchecked. Linked folders naming the same database are checked once. PrestaShop settings and content resolve only when every language, shop and scanned column is clean or absent. New employees resolve only when deleted; SuperAdmin findings also resolve after demotion.

Administrator baselines for Joomla, Drupal, Magento, OpenCart, and PrestaShop are scoped to the hosting account, CMS, database host, database name, and table prefix. Two sites under one account keep separate baselines when they use different databases or prefixes. Paths that share the same database and prefix share the same administrator set. Upgrading from account-wide baselines starts a fresh baseline for each installation on its first complete administrator query; later additions produce one High finding per new administrator. PrestaShop also records which employees hold the SuperAdmin profile: a new SuperAdmin, or an employee moved to that profile, produces one Critical finding instead, and an employee demoted and later promoted again is reported again. Finding details identify the affected database and prefix.

Database errors, discovery errors, and configuration or query limits keep the affected CMS check incomplete. Earlier findings remain until that CMS completes a scan; another CMS can still complete and clear its own resolved findings. Queries inspect at most 200 rows and request one extra row to detect overflow. Known statement errors and overflow allow independent queries to continue; connection failures and cancellation stop further queries for that installation. Administrator baselines and recorded IDs change only after a complete result; a successful empty administrator result also establishes a baseline. New IDs in a partial result can still be reported against an existing baseline.

PrestaShop scans employees first, so settings or content limits do not prevent employee baselines or SuperAdmin alerts. An employee overflow keeps coverage incomplete and preserves employee state, while settings and content queries still run within their own row limits and the scan deadline.

CMS Scanner Support Policy

New CMS scanner work targets upstream-supported major versions. EOL versions are best-effort when the existing scanner covers them through the same low-risk layout or schema. Adding a new EOL-only scanner needs operator fleet data and an explicit security reason.

Current scanner scope:

  • WordPress single-site and multisite.
  • Joomla installs using the common configuration.php / JConfig layout and standard content/user tables used by supported Joomla releases.
  • Drupal 8 and newer. Drupal 7 is not a planned support target.
  • Magento 1 and 2.
  • OpenCart installs using the standard storefront and admin config pair.
  • PrestaShop 8 and 9; 1.7 and 1.6 best-effort through the same schema.

The supported kinds are declared once in internal/cms. The database scanners and the PHP taint analyzer’s knowledge of each CMS’s bootstrap path constants are tested against that table. The tests also require every typed CMS kind constant, including local declarations and aliases, to have exactly one descriptor. Membership drift fails the tests. The clean-corpus manifest is validated against the same table: every supported CMS is either pinned as a source or listed as pending with a reason. Support does not by itself mean a CMS has clean-corpus false-positive evidence (see the clean corpus). Database object scanning (db_objects) discovers WordPress installs only.

Phishing & Malware

CheckDescription
phishing8-layer phishing detection (kit directories, credential harvesting). Kit archives are judged from their file list, read through an isolated worker without decompression. Unread or malformed directory records leave the check incomplete.
email_contentOutbound email body scanning for credentials and suspicious URLs. Only mail submitted on this server, by an authenticated SMTP client or a local process, is scanned; unauthenticated mail arriving from other servers is not. Messages are found in both flat and split Exim spool layouts. Oversized headers are checked from a bounded prefix; long padding lines do not discard readable evidence or stop body checks.

Outbound scanning covers any authenticated SMTP client, including logins Exim records without a mailbox, and local processes. Quarantine and re-check act only on the message ID the finding names; the envelope sender is shown in the finding details and mailbox field.

If the ZIP worker is unavailable, the check stays incomplete. ZIP parsing never falls back into the daemon. The existing phishing entry in disabled_checks controls the check.

An archive that causes two worker failures within 24 hours is skipped on later scans with a phishing_kit_archive_unscannable warning naming its path and SHA-256. That skip alone does not leave the check incomplete. The failure record expires 24 hours after its last failure; it is held in memory and is cleared by a daemon restart. Changed content is eligible for scanning, and a file rewritten while the worker runs cannot acquire a failure for its replacement contents. Malformed listings, canceled scans, and an open worker breaker leave the check incomplete without adding failures against the archive. Once all failure records expire, scans no longer hash each candidate archive before requesting its list.

The phishing and group-writable PHP scans include dependency and source-control folders within their depth budgets. They do not recurse through directory symlinks. Phishing content reads also refuse file symlinks and special files without blocking, including when a regular file is replaced during a scan. The PHP permissions check uses regular-file permissions, so a symlink’s write bits do not produce a finding.

Phishing directory analysis honors suppressions.ignore_paths for the pages it reads, just as page analysis does. Explicit audits can bypass those exclusions. Directory brand evidence uses the same title rules as page analysis and checks visible body content, so a brand appearing only inside an email address in the title or hidden markup does not establish impersonation. Suspected kit directories are reported for review; auto_response.quarantine_files applies only to eligible page findings.

Files Planted Alongside Malware

A webshell kit is usually dropped in one session: a loader no rule matches, an encrypted payload, a binary dropper, an access file. Removing only the member a scanner recognised leaves the rest in place. When CSM proves one file malicious, it looks at the other files in that file’s directory and reports the ones planted with it.

Files are matched by change time, not modification time. Kits backdate modification times, but the kernel sets the change time on every write and an account cannot forge it.

  • Trigger. A Critical finding from a content scanner, a YARA or signature rule, or a known webshell or backdoor name, from the realtime monitor or a scheduled scan, including the startup scan. Successful quarantine or cleaning through the daemon, including manual dashboard actions and access-file cleanup, also triggers one. Its metadata comes from before the move or rewrite. Refused actions do not trigger sweeps. An account scan sweeps for the detections in its own results and includes the outcome in its report.
  • Scope. The detected file’s own directory, not its subdirectories. Symlinks and special files are never followed or read. Content reads remain bound to the opened directory and selected file version. Only files owned by the detected file’s owner count: an account can only create files it owns.
  • Window. Files changed within ten minutes either side of the detected file. A file caught as it is written may be the first of its kit to land, so its sweep waits until the window has closed.
  • Scan. A first pass reads metadata only. Files in the window, and the directory’s .htaccess whatever its age, then get a full content scan with YARA, the PHP content heuristics, PHP taint analysis and the .htaccess audit, regardless of scan caches. A match is an ordinary finding and follows the normal response policy. Account scans retain their requested file-size limit. Failed PHP analysis is reported as incomplete, not as a clean result.
  • Report. Files in the window that nothing matched are listed in one High malware_companion_files finding when they are PHP, executables, archives, files without an extension or .htaccess, with size, change time and modification time. A modification time more than a day older than the change time is marked as possible backdating; unpacking an archive also keeps old modification times, so treat it as a hint. CSM never quarantines these files on its own: review each one first.

Each plant window of a directory and owner is swept once a day at most, so another member of the same kit, a repeated alert or the quarantine of the detected file does not sweep it again. Sweeps run one at a time and share the scan concurrency limit and per-check execution deadline with periodic checks. Matches are delivered in batches, releasing the scan slot while waiting for delivery. Overlapping windows do not repeat content findings for the same file version. Repeated detections retain one file version, and expired records are removed even when no new work arrives. Pending windows stay claimed until they run. A huge directory costs one metadata read per entry; only files written in the window and .htaccess are opened. A timeout produces a check_timeout warning; directory or content scan failures produce malware_companion_scan_incomplete. The access-file audit checks sibling context once, in batches, and stops on cancellation or when it has enough evidence for its existing sibling gate.

A bulk ownership or permission change, or an account restore, resets the change time of every file it touches. A detection in such a directory lists every file of a reported kind from the same operation. Sweeps still waiting for their window are held in memory, and a restart drops them. Standalone CLI cleanup commands do not run this daemon worker; use an account scan to inspect companions when the action runs outside the daemon.

System Integrity

CheckDescription
rpm_integritySystem binary verification via rpm -V
open_basediropen_basedir restriction validation
php_config_changesSecurity-weakening .user.ini and php.ini files under account web roots

DNS & SSL

CheckDescription
dns_zonesSecurity-sensitive DNS zone changes (delegation, mail, apex, and wildcard records)
ssl_certsSSL certificate issuance (subdomain takeover)
waf_statusWAF mode, staleness, bypass detection. On cPanel, staleness follows vendor configuration files that WHM reports as active and ignores retired trees that remain on disk; the warning means at least one loaded vendor has not refreshed in over a month. Other platforms keep the conservative oldest-artifact check so an unused fresh tree cannot hide stale loaded rules.

Email Security

CheckDescription
email_weak_passwordEmail accounts with weak passwords; in-process verification with supported hash formats and cost limits
email_password_audit_incompletePassword verification was interrupted or encountered a hash outside the supported audit formats or limits
email_forwarder_auditForwarders redirecting to external addresses, piping mail to a command, or discarding it. Pipes that run cPanel’s autoresponder, BoxTrapper or Mailman list software are not reported. Scheduled and real-time scans interpret command quoting and escaped destination quotes using Exim’s rules.
email_mail_filtersExim mail filters and dovecot/Roundcube Sieve scripts that copy mail to an external address while keeping a local copy, forward externally, pipe to a command, or blackhole all mail. Sieve is what webmail-managed rules actually execute, so both are scanned. A forward that leaves the mailbox its own copy is what a webmail forward rule produces, so on its own it reports as a Warning for review; it is Critical when an independent forwarding layer on the same mailbox, mail the mailbox never receives, or the same destination across accounts corroborates it. The full audit runs on the email audit interval. The next deep scan picks up edits to a Sieve script, including the target of an active script link. Each such forward is also compromise evidence for its mailbox.

Performance

CheckDescription
perf_php_handlerPHP handler configuration (DSO vs CGI vs FPM)
perf_mysql_configMySQL my.cnf optimization
perf_redis_configRedis configuration
perf_error_logsError log file growth (bloat)
perf_wp_configWordPress wp-config.php settings
perf_wp_transientsWordPress database transient bloat
perf_wp_cronWordPress cron scheduling (missed crons)
perf_wp_loopbackWordPress self-requests sustained above once a minute
perf_wp_myisamWordPress tables still on the MyISAM storage engine

Platform Support

The deep checks are the most cPanel-biased part of CSM because they iterate account home directories and per-user public_html trees. On plain Ubuntu/AlmaLinux the account-scan based checks do not run today:

cPanel-only (skipped on plain Linux):

  • htaccess, file_index, php_content, group_writable_php, symlink_attacks – iterate /home/*/public_html/**
  • wp_core, outdated_plugins, vulnerable_plugins, db_content, db_objects, admin_overlap, credential_reuse – find WordPress installs through the shared discovery: the panel document-root map, /home/*/public_html, one directory below it, and addon-domain directories in an account home. Unresolved document-root aliases retain prior findings and cached plugin inventory instead of treating a partial walk as a clean result.
  • supply_chain – scans composer.lock and package-lock.json under /home/* and /home/*/public_html
  • phishing, email_content – scan user home directories and Exim spool
  • dns_zones, ssl_certs – read cPanel’s DNS zone store and SSL installation records
  • email_weak_password, email_forwarder_audit – read /etc/valiases, Dovecot/Courier auth databases
  • email_mail_filters – read per-mailbox Exim filters under /home/*/etc/<domain>/<localpart>/filter and domain filters under /etc/vfilters
  • open_basedir – reads EA-PHP php.ini under /opt/cpanel/ea-php*/
  • php_config_changes – recursively scans .user.ini and php.ini below account web roots; incomplete walks emit a coverage finding and preserve prior findings
  • perf_wp_config, perf_wp_transients, perf_wp_cron, perf_php_handler – WordPress and PHP handler introspection via cPanel’s EA-PHP layout; the WP-Cron check also uses cPanel’s domain map for addon and subdomain roots

Runs on every platform:

  • filesystem, webshells – fanotify and file-tree scans over /home, /tmp, /dev/shm
  • rpm_integrity – dispatches to rpm -V on RHEL family or debsums / dpkg --verify on Debian family
  • waf_status – detects ModSecurity on Apache, Nginx, and LiteSpeed across all supported distros
  • perf_mysql_config, perf_redis_config, perf_error_logs – rely on standard service locations
  • perf_wp_myisam – uses shared WordPress discovery in the deep tier, including account home layouts when no panel map is available

Operators on plain Linux can point perf_error_logs, perf_wp_config, perf_wp_transients, and perf_wp_cron at generic web roots with the account_roots glob list (see configuration.md). The remaining account and CMS scans still assume the cPanel /home/*/public_html layout.

Package integrity rechecks retain a modification that the package verifier still reports for the flagged file. Changing its executable mode or removing the file does not by itself resolve that finding.

Hidden temporary files

The filesystem check reports hidden regular files in the shared temporary directories when they have executable permissions, ELF magic, or a leading shebang, long PHP opening tag, or short echo tag. Aliases of the same inode produce one finding; distinct files remain separate even when their names match. Failed content reads leave the filesystem scan incomplete and preserve earlier alerts.

An inert staged blob without these signals is outside this heuristic. There is no guaranteed alternate detection before it is made executable or interpreted; signature and runtime detection depend on recognizable content and activity. Changing its permissions to executable makes it eligible for the next filesystem scan. This check does not establish that every unreported temporary file is safe.

Auto-Response

When enabled, CSM automatically responds to detected threats. All actions are logged in the audit trail.

Actions

ActionDescription
Kill processesFake kernel threads, reverse shells, GSocket, loops that write back a file CSM removed, and shell loops a web request left running in the background. Never kills root or system processes.
Quarantine filesMoves webshells, backdoors, phishing to /opt/csm/quarantine/ with full metadata (owner, permissions, mtime). Restoreable from the web UI.
Block IPsAdds attacker IPs to the nftables firewall with configurable expiry. Admission uses auto_response.max_blocks_per_hour (default 2000/hour); legacy scan blocking defaults to 50/hour.
Clean supported malwareApplies bounded PHP and .htaccess cleaners with pre-clean backups. Database cleanup has a separate opt-in.
Drop malicious DB objectsWhen clean_database is on, confirmed-malicious stored triggers/events/procedures/functions are dropped after a SHOW CREATE backup is recorded, so the drop is reversible. Detection runs regardless; the drop is gated on the operator opt-in.
PHP shieldBlocks PHP execution from uploads/tmp directories and inspects directly executed wp-content scripts for request-fed command sinks and packed eval loaders.
PAM blockingInstant IP block when one address breaches thresholds.pam_bruteforce_threshold SSH failures inside pam_bruteforce_window_min minutes, or fails SSH logins against cred_stuffing_distinct_accounts distinct accounts. Only successful SSH logins clear those failures; failures from other PAM services never block.
Subnet blockingAuto-blocks IPv4 /24 or IPv6 /64 when 3+ IPs from the same range were blocked within netblock_window (7 days by default). Currently blocked addresses count regardless of age, including operator and permanent blocks. Ended blocks count while their latest observed block start is inside the window, unless an earlier subnet block already answered them. A returning operator block starts fresh history after its absence is observed. Whitelist and clear actions, including bulk whitelist, forget the address.
Permblock escalationPromotes temporary blocks to permanent after N repeated offenses.
Suspend mailboxAfter confirmed credential abuse, attempts to suspend that mailbox’s logins and outgoing mail through cPanel. Either success stops escalation; the account-wide outgoing mail hold is the fallback when both fail, no account owns the domain, or the target is a bare domain. A Critical mail_account_compromised (a login after brute force) also suspends the mailbox. When that login is the cPanel account’s own username, the account’s outgoing mail is held instead, because that password unlocks every mailbox. A stolen login never falls back to the account hold: a failed suspension raises a Critical email_mailbox_response finding, as does each suspension or hold it applies. Signs of compromise that add up for one mailbox suspend it with a single Critical email_mailbox_compromise finding; see Mailbox Compromise Evidence. All of these follow auto_response.enabled and dry-run, and each attempted suspension or hold is recorded in the action log with its quoted release command.
Auto-freeze (PHP relay)On cPanel, freezes active Exim messages attributed to a high-confidence PHP-relay finding. It has its own dry-run control and action-rate limit. See PHP-relay CLI.

Of the processes a web request left running, only shells are stopped; anything else, including a non-shell parent of a shell, stays a Warning for review.

Mailbox suspension runs /usr/local/cpanel/bin/uapi as a transient systemd service outside the daemon’s filesystem sandbox; hosts without a usable systemd-run run it directly. Each call is bounded in time, the JSON result decides success, and mailbox names stay literal arguments on both paths.

Partial suspensions name the restriction that succeeded; they do not claim both logins and sending stopped. Mailbox restrictions and account holds already applied outside CSM are not offered for release. If cPanel cannot report the mailbox state, CSM suspends anyway and records the suspension as its own; sending evidence can still fall back to an account hold.

After resolving the credential abuse, release the mailbox or account on the Email page, Suspended tab. The tab lists every suspension and hold CSM made with its live state in cPanel; one lifted directly in cPanel shows as released outside CSM. Incomplete or unreadable status stays unknown, and a delayed status response cannot release a newer suspension.

Release undoes the recorded restrictions and never changes a password. Evidence captured before the release cannot suspend the mailbox again. Cleanup failures are reported so the release can be retried.

Releasing an account hold delivers the mail cPanel held, so delete spam from the Queue tab first; the confirm says so. The action log also keeps the release commands, quoted for the shell, for example /usr/local/cpanel/bin/uapi '--user=account' Email unsuspend_login 'email=alice%40example.com' and the matching unsuspend_outgoing command. The address is URI-encoded because uapi decodes its values.

WHM holds and releases check the JSON API result before reporting success. Dry-run writes one planned-action record without running a command; disabled auto-response writes none.

Subnet history is pruned hourly and saved only when it changes. If the file cannot be read it is left untouched and escalation counts only the addresses blocked right now until it is repaired or removed; status and doctor report the failure. Clear, whitelist and flush actions report history cleanup failures so operators can retry them.

Mail login spray blocks of IPv4 addresses do not contribute to /24 escalation. IPv6 spray blocks count toward their /64 like any other block, since one /64 is treated as one source. The classification comes from committed firewall state, so a missing or stale scan tracker does not turn IPv4 spray blocks into range evidence. Saved spray blocks absent from the kernel do not count as active blocks. If block metadata cannot be read, subnet escalation waits while individual block attempts continue.

Blocks of an address on c2_blocklist never count toward subnet escalation or permanent promotion, whatever finding asked for them: a listed command-and-control server is where local processes connect, not a source of attacks. Other blocked addresses in its subnet still count.

Process termination

An eligible finding receives one automatic termination evaluation across scan and alert delivery. A refused or failed attempt is not retried when the same finding reaches alert delivery. Realtime findings still receive an evaluation; new detections and findings replayed from storage receive a fresh evaluation.

CSM opens a kernel process handle before verifying ownership, executable, start time, or the file referenced by a process. It checks that the captured process is still alive after verification and sends the signal through that handle. An exited process cannot redirect the signal to a replacement with the same PID. Automatic and manual malware termination reject root credentials, including effective and saved root IDs. The separate opt-in AF_ALG reaction requires the current credentials and executable to match its recorded event.

Safe signaling needs pidfd_send_signal (Linux 5.1). Where pidfd_open (Linux 5.3) is absent – EL8 and CloudLinux 8 ship 4.18 kernels without it – CSM pins the target through its /proc/<pid> directory descriptor, which pidfd_send_signal accepts and which refers to the same kernel process. Both paths reject a recycled PID; CSM never falls back to numeric PID signaling. On a kernel without pidfd_send_signal, or when service restrictions deny the call, termination stays disabled: csm doctor reports process termination supported as failed and the health status becomes degraded whenever auto_response.kill_processes is enabled, so an inoperative protection is visible before an incident needs it. Each health snapshot probes this capability again, so transient resource failures clear once signaling becomes available. Automatic termination failures are logged while the original detection remains. Manual kill-and-quarantine reports a termination failure even if the file was successfully quarantined; inspect both the process and recovery entry before retrying. Manual request cancellation is checked before sending a signal.

Automatic file response limits

Realtime quarantine, scheduled quarantine, PHP cleaning and automatic .htaccess cleaning share one rolling-hour budget. auto_response.max_file_actions_per_hour caps host-wide attempts (default 50), max_file_actions_per_account_per_hour caps attempts for one account (default 10), and max_file_action_failures_per_hour pauses these responses after repeated failures (default 3). Zero or an omitted key uses the default; negative values and values above 10000 are rejected.

Each attempt is reserved before touching the file. Successful, failed and interrupted attempts all consume capacity. Reservations live in <state_path>/file-response.json and survive configuration reloads and daemon restarts. Entries expire one hour after admission. A backwards clock adjustment keeps future-dated reservations charged until their window has passed.

Account budgets come from the target’s account-home path, not finding text. Paths outside recognized account homes share an unknown-account budget. One account reaching its limit does not stop other accounts unless the host or failure limit is also reached. Automatic directory and special-file quarantine is refused because one directory move can affect an unbounded number of files. Manual remediation remains available after reviewing the original detection.

A brief shared lock protects each safety reservation and outcome update. The file action runs after reservation without holding that lock, so a slow copy does not block another available slot. Pending actions hold failure capacity until their outcome is saved, so concurrent failures cannot pass the configured ceiling. If the state lock remains busy, the next attempt is refused. Unreadable, incomplete or unwritable safety state also refuses mutations. The original detections remain visible, and a deduplicated auto_response_paused warning reports the cause. Account-limit notices are grouped at host scope so a fault across many accounts cannot flood the alert budget. A pause does not create a retry job; new eligible detections can act after capacity returns. Review outstanding findings and recovery evidence before manual remediation. Do not delete safety state to clear a pause; repair storage faults and let reservations expire.

Duplicate detections of one stable file version share a single realtime response attempt. A different file planted at the same path remains eligible. Within one daemon run, alert delivery does not repeat a file response already evaluated by a scan or admitted to the realtime safety gate, including budget refusals. A realtime detection rejected using sampled content remains eligible for full-file validation during delivery. The original findings still reach alerts and history; a new detection can be evaluated again. Findings still queued at shutdown retain the existing restart replay behavior: the next daemon run evaluates them again under the same persisted limits. Parked findings also keep the offending subnets and targeted accounts needed for subnet response and credential spray correlation. Older parked findings replay without that data. The subnets also appear in the public cidrs field.

Critical realtime PHP findings that prove request data reaches an execution primitive, or that executable PHP unwraps and runs encoded content, enter this file-response gate immediately. Quarantine holds an exclusive Linux lease on the open file that produced the finding until that file is removed. It refuses while another process has the file open or is running it, and when the content or path identity changes. Whole-file malware identified by the cleaner uses this lease too. Other quarantines do not take this lease, so they can still move a file that is open or running. Comment-only matches, URL-only droppers, filename-only checks, tail-window matches and YARA-only findings remain available for review without an automatic move.

File quarantine copies only the file size it verified and checks the file again before removing it. A file that changed is restored and its copy discarded. If its name was recreated meanwhile, the error names the private directory that holds the changed file.

Parked reputation findings keep their intelligence source and expiry. If an expiry cannot be encoded, only that intelligence metadata is dropped; the finding and the rest of the queued batch are still saved and replayed. This handling is automatic and has no setting.

A failed PHP cleaner leaves the file and any pre-clean backup for manual review. It no longer escalates to whole-file quarantine. A cleaner that recognizes no injection or declines an unsupported target refuses the file instead of failing. Sources that change or disappear before mutation are also refusals, including socket replacements and parent paths replaced after quarantine copying. These attempts still use capacity but do not count toward the failure pause. Read, write, backup and durability errors still count as failures. Quarantine and cleaners retain their descriptor-based identity checks, and automatic actions revalidate the file after saving the reservation. Opening a replacement special file cannot block response processing.

Manual full scans leave cleaner refusals for review instead of reporting a failed remediation. csm clean also distinguishes a refusal from an action failure; both return a nonzero exit status when the file was not cleaned.

WordPress core, plugin and theme files are cleaned in place, with three exceptions. When cleaning would leave nothing of a file that sends no page output, the file is quarantined whole: a stub would lose the same content, while quarantine keeps it for a restore. When cleaning would erase a file’s page output, the file stays untouched for review. When the release of WordPress core or of a wordpress.org plugin that a site runs does not ship the file, it was planted and is quarantined. CSM trusts the release a site names only when its version.php or <slug>.php plugin main file is that release’s own copy, and only from the checksums the realtime file monitor has cached. A plugin whose main file has another name stays with the cleaner. Page output is any text outside PHP code, including short and echo tags. Whole-file quarantine rechecks the original content under the same protection used for other verified file responses; a changed file stays in place. csm clean refuses a file it would leave empty.

Automatic permission fixes dismiss the original finding after a successful fix or after confirming its unsafe permission is already gone. A confirmation alone does not report a permission change.

These limits use the existing enabled, quarantine_files and clean_htaccess opt-ins. Observe mode still forbids automatic changes. dry_run continues to control IP blocking and web-exposed-file virtual patches; it does not preview file quarantine or cleaning. Other response families have their own controls.

Files written back after removal

When CSM quarantines or cleans a file, it clears the alert state it kept for that file. If a detection finds the file there again, the finding is reported as new instead of being held back as a repeat. Dismissals remain in effect.

CSM also remembers, for seven days, the content each quarantine or clean took away. A process that writes the same bytes back to the same path raises removed_file_restored_realtime (Critical). With quarantine_files on, CSM then does what it did before: it quarantines the file again, or cleans it again. The daemon keeps this memory across restarts. A file cleaned with csm clean from the command line is not remembered. A restore from the web UI quarantine page, CSM’s own writes and official WordPress release files are never treated as the file coming back.

The saved history is read only from a private file the daemon owns. A damaged or unsafe one is logged and ignored, and detection carries on without it.

Before repeating the response, CSM checks that the file still holds the removed bytes, so a later replacement or a restore from the web UI is never removed. Removed files are watched whatever their extension. A restored .htaccess is cleaned only when clean_htaccess is on.

With kill_processes on, CSM also stops the process that wrote the file back, together with its parent processes of the same account, parents first, so a loop cannot start another writer. It does so only when those processes trace back to the web server or run detached under init, the shape of a loop a webshell leaves behind. A PHP worker that serves the site is not stopped unless it wrote the file itself. A process with a terminal, or one started from an SSH, cPanel or cron session, is left running and the file is still handled; root processes are never stopped. An SSH, cron or cPanel session anywhere above a PHP worker also prevents the kill, and so does any part of the ancestry CSM cannot read. The processes are named when the write is seen, and each is checked again through a kernel process handle just before the signal, as described above.

Restoring quarantined files

Regular-file quarantine and pre-clean backups write and sync the private content copy, metadata, and directory entries before removing or changing the original. Directory quarantine syncs the tree and metadata before moving it on the same filesystem. A directory move across filesystems is refused and leaves the source in place. Storage failures do not count as successful remediation.

New quarantine and pre-clean sidecars record the original modification time, owner, group, permissions, and size. Restore reapplies the saved attributes to the opened destination before syncing it. Linux preserves modification times at the precision supported by the destination filesystem. An ownership or timestamp failure keeps the recovery evidence and reports an error.

On Linux, restore can verify the destination after saved permissions remove all read access. The restored file keeps those permissions, including a mode with no access granted to anyone.

Older sidecars may use quarantine_at; listing and restore also read that historical spelling. Entries without a saved quarantine date sort last. Older entries have no recorded original modification time, so restore leaves the new file’s modification time in place instead of treating the archive’s timestamp as the original. Some older access-file cleanup backups also lack trustworthy ownership and permission data; check those attributes when restoring them.

Configured account roots participate in manual remediation and restore. Set up service write access for roots outside the packaged grants before enabling these operations.

Web UI restore refuses symbolic links in destination directories and does not replace an existing file or directory. If a destination changes during restore, CSM reports a conflict and retains the quarantine entry. Check the original location before retrying; a failed or interrupted file restore can leave a partial file in the directory that was opened for restoration. CSM keeps this file because removing it could discard a concurrent replacement.

Restore syncs the replacement and its containing directory before deleting quarantine evidence. A failure after a move or unlink reports partial completion and retains recovery metadata where possible. Inspect both the original and quarantine locations named in the error before retrying; a copy may already be restored while quarantine cleanup remains incomplete. These guarantees depend on the filesystem and storage device honoring sync requests.

Virtual-patch rollback uses the same directory confinement. It replaces or removes the access file only when the saved content, owner, and permissions still match, preserving later customer edits.

Virtual-patch backups are reused only when their content and saved attributes, including modification time, match. A later edit with identical bytes but a new modification time gets a separate recovery point.

Rollback isolates displaced entries in a private directory. If another writer changes the destination, CSM preserves the live replacement and reports any retained recovery directory in the error. Check that directory beneath the original destination or quarantine location before retrying. If another writer moves a restored directory, CSM cannot recover it from its old name and will not substitute a different entry into quarantine.

Configuration

auto_response:
  enabled: true
  kill_processes: true
  quarantine_files: true
  max_file_actions_per_hour: 50
  max_file_actions_per_account_per_hour: 10
  max_file_action_failures_per_hour: 3
  block_ips: true
  block_expiry: "24h"         # positive temp block duration; omitted defaults to 24h
  # max_blocks_per_hour omitted: 50 scan blocks an hour, admission ceiling 2000
  netblock: true              # enable subnet blocking
  netblock_threshold: 3       # IPs from same IPv4 /24 or IPv6 /64 before subnet block; minimum 2
  netblock_window: "168h"     # blocked IPs count this far back, expired blocks included; omitted defaults to 168h
  permblock: true             # promote temp blocks to permanent
  permblock_count: 4          # temp blocks before promotion; minimum 2
  permblock_interval: "24h"   # positive counting window; omitted defaults to 24h

  # Response to http_scanner_profile findings: "challenge" (default)
  # routes the IP to the PoW challenge when challenge.enabled is true,
  # falling through to a firewall block when it is not; "block" always
  # hard-blocks without offering a challenge.
  http_scanner_action: "challenge"

  # Deny HTTP access to confirmed exposed files with reversible .htaccess
  # rules. off: alert only; manual: `csm virtual-patch --apply`; auto:
  # apply confirmed findings except warning-only sample SQL when dry_run is false.
  virtual_patch_exposed_files: "off"

  # SAFETY DEFAULT: dry_run defaults to TRUE when this key is absent.
  # Automatic IP blocks do not touch nftables. Admission keeps bounded
  # examples of successful capacity predictions, not every attempted block.
  # Manual operator commands (`csm firewall ...`) bypass dry-run and apply.
  # Flip to false only after verifying the policy in dry-run.
  dry_run: true

  # Advisory verdict callback. CSM POSTs each impending auto-block
  # to the panel before applying. The panel can downgrade to "allow"
  # (audit-only), attach `tenant_id` for downstream correlation, or
  # add a reason. CSM fails open on hook errors. Wire contract:
  # verdict-callback-contract.md in this book.
  verdict_callback:
    enabled: false
    url: ""                            # POST target
    hmac_secret: ""                    # signing secret, or use hmac_secret_env
    hmac_secret_env: ""
    allow_unsigned: false              # true only for staged unsigned rollouts
    require_response_signature: true   # reject unsigned callback replies
    timeout_sec: 2

  # PHP-relay auto-freeze (cPanel only). Off by default; opt in
  # explicitly. dry_run defaults to true even when freeze=true so an
  # operator who enables freeze without thinking gets a dry-run.
  php_relay:
    freeze: true                       # enable the exim -Mf hook
    dry_run: true                      # safe default; flip with `csm phprelay dry-run off`
    max_actions_per_minute: 60         # rolling 60s window cap on exim -Mf invocations

Exposed-file virtual patches

Each applied deny keeps a rollback copy under /opt/csm/quarantine/pre_clean/. Re-applying the same rollback state reuses that copy; content, ownership, permissions, or remove-versus-replace changes keep separate restore points.

All-in-One WP Migration rewrites the access file inside wp-content/ai1wm-backups, so CSM also denies only .wpress filenames from the parent wp-content access file. It does not add parent-wide .zip or .gz rules because those extensions can be legitimate downloads elsewhere under wp-content.

Dry-run safety default

auto_response.dry_run defaults to true when the key is absent. This is deliberate: an operator who turns on block_ips: true without reviewing policy keeps automatic IP blocks unapplied while evaluating policy. The retained-example count surfaces in csm status --json and /api/v1/status; forecast status reports capacity decisions separately. On entry into live mode CSM requests a purge; owner-managed purges run in bounded passes and report stale examples until complete. While dry-run remains enabled, records older than a week age out.

With the firewall on and outside observe mode, admission also forecasts each automatic IP block in dry run: whether it would fit the hourly ceiling and the temporary block set, behind the blocks and forecasts before it. Status, csm doctor and findings history show the forecast, with at most one summary a minute. Admission owns new examples, pruning and purge. On a live start, existing examples also wait for its bounded purge, including while admission startup retries. New examples describe committed capacity predictions; the count also includes retained older examples until pruning or purge. It is not unique IPs or active predicted occupancy. Example loss and incomplete comparisons appear in status. A forecast is not a block; restart or re-entry into dry run starts a new session without restoring hypothetical leases from examples. The forecast fills the temporary block set as automatic admission will: of the slots operator, unknown and subnet blocks leave, a fifth is kept for direct or corroborated attacks, and an unlimited set is modelled as 10000 entries. Forecasts and previews cap each new automatic lifetime, including challenges, at auto_response.block_expiry: longer requests are cut and shorter ones stay short. Retries keep their first absolute expiry, even after a lower cap is configured; existing blocking keeps its own lifetimes. In a full set a forecast block may take the place of a lower-priority one, as live admission would: an earlier forecast block of a lower class or severity, or a legacy automatic block that recorded its origin. A new attack may also rotate out an earlier forecast block of the same class and severity once that block has held its slot for 30 minutes, within the share of the set its scope may use; an address whose forecast block was replaced or rotated out cannot rotate again before that block’s expiry. A forecast block for an address that returns after the forecast replaced its earlier block lasts only until that earlier block would have ended. A return with too little time left for a firewall block is refused without a forecast charge or retained example.

Admission checks how each finding’s address was read against its producer’s declared address sources and the check’s current policy, when it mints evidence and whenever retained evidence is used. A producer without a trusted address source cannot mint evidence. This excludes PHP relay findings, whose client address comes from a mail header a local account can forge. With auto_response.dry_run: true, these findings cannot produce a successful admission forecast. Stored evidence keeps its existing format and identifiers.

Scheduled cPanel, ModSecurity audit, per-site and central access log scans can supply automatic response forecasts from complete records with a verified address source and server time. The scheduled command-and-control connection scan can too, for established IPv4 connections this server opened. It names each one by its socket during the current boot, keeps its first-seen time across restarts when the state save completed, and reports both endpoints and the socket owner. Listener changes and overlapping scans do not refresh a matched socket’s first-seen time. Socket inodes can be reused between scans; those reports retain the earlier time, and lost first-seen state can make a forecast conflict with retained evidence. A connection on a local port that also listens still alerts but supplies no forecast, because the socket table does not show which side connected. Connections still waiting for a reply and those in the IPv6 socket table are not reported. Without a socket identity the alert still shows but supplies no forecast. Blocking the address drops its replies but does not stop data this server sends to it. Rereading a record does not refresh its time or add independent evidence. Unfinished or ambiguous records cannot supply a new observation. HTTP findings whose counts or bot-majority decisions use forwarded addresses, including mixed direct and forwarded traffic, name a client behind a trusted proxy and are refused as unreachable. The client is the first untrusted address reached from the right of the forwarded header. Periodic scans retain a bounded suffix of complete address tokens; padding to the left of a retained client does not suppress its count. A missing or unreadable right end, or a retained suffix containing only trusted proxies, names no client and is not counted. The resolver refuses a header over 4 KiB or a walk that must read more than 32 hops, its client included; hops left of the client never count. An address in web_server.trusted_proxies or challenge.trusted_proxies is never read as the client, including when its header is not trusted because it belongs to this host or the host’s addresses cannot be read; an address of this host is never trusted as a proxy. An HTTP, cPanel or IP reputation finding whose address is a known CDN or website-firewall edge, from the Cloudflare, QUIC.cloud and Sucuri ranges CSM ships (a replacement http_proxy_ranges.yaml does not change them) or, when the ASN database names it, a network in thresholds.http_asn_crawl_reverse_proxy_asns, is refused as proxy_peer, whether or not cloudflare.enabled is set. This edge refusal does not apply to SSH, FTP or mail session peers. When Apache (RemoteIPHeader, RemoteIPProxyProtocol On, a loaded or configured mod_cloudflare including DenyAllButCloudFlare, or mod_rpaf with RPAF_Enable On or RPAFenable On), nginx (set_real_ip_from) or LiteSpeed (Use Client IP in Header, in its own configuration or, on cPanel, the Apache configuration it loads) replaces the logged client address, or CSM cannot read those files completely, new HTTP findings are refused as rewritten_log and access-log lines are not IP reputation sightings; cPanel access lines still are, even when the same address also appears in a rewritten web log. An Apache, nginx or LiteSpeed access log format that writes request data into the first client token counts the same way. Leading whitespace and extra spaces after cPanel’s splitlogs virtual-host field cannot hide that token. A leading quote, brace or other non-address character guards normally escaped output. Brackets cannot guard a request value because scheduled scans accept bracketed IPv6 peers, including peers with ports. Apache format nicknames ignore case, including the TransferLog default. A definition in another virtual host or an unrelated conditional section cannot hide a log’s inline or common-format fallback. With HostnameLookups On in any scope, %h counts as a request value, because it then logs the client’s unverified reverse DNS name. Inside a mod_macro <Macro>, a parameter in a log format, format name, piped log target, module name or directive name leaves the state unknown, including a parameter that spells ServerName. Quoted macro names and parameters retain their boundaries. A parameter later in a log file path or in a server name is harmless. A parameter or ${...} variable that starts a log target leaves the state unknown, except Debian’s ${APACHE_LOG_DIR}/. That exception does not apply when Define or UnDefine can change the variable, including inside a macro. The built-in CLF nickname is also checked after piped logging removes its leading field. An ErrorLogFormat request value before an unconditional message in any output record also counts. A message that shares its field with another placeholder cannot guard later request values because that field may disappear. Unresolved Apache configuration variables in log formats, format names and server names are unknown, and so are those that could make or hide a piped log target. Unsupported format handlers, blank Apache server names and custom time formats that can write a line break are unknown, and so are unproven time formats or process, port and duration arguments in or before the field CSM reads. Empty logging conditions, including whitespace-only expressions, are unknown; a nonempty expr= condition does not change how its format is read. LiteSpeed formats with unsupported escapes, modifiers, handlers or ambiguous brace arguments are unknown, in both native XML and cPanel Apache configuration. A valid format no CustomLog, GlobalLog, TransferLog or nginx access_log uses does not count. nginx checks HTTP and stream format namespaces separately. A log selects the first previously declared format with the same name, ignoring ASCII case. Later case variants do not count as used formats. Unresolved names and malformed log declarations leave the state unknown. Entries inside maps are values, including entries read from included files. With escape=none, any request variable can introduce another log record, even after the normal client field or a quote. Binary address variables have unknown output framing. Empty server-name, hostname and port fields cannot hide a following request value. User-defined variables do not inherit trust from a built-in-looking name. nginx original-peer variable names and stream server names remain unproven because their availability depends on the installed modules and version; captures or scripts can supply request values under those names. This conservative check also applies when those variables come from nginx itself. Stream byte counts, session duration and transport protocol remain server values in the stream namespace. This build does not read a separate original-peer log field. CSM reads the files when admission starts, at the start of every reload attempt and web UI settings save, and every 10 minutes. Settings saves refresh this state before applying changes, including saves that fail or require a restart. web_server.config_dir selects the Apache or nginx root, including the Apache tree used by cPanel LiteSpeed; LiteSpeed XML remains at its standard location. The main file is apache2.conf on Debian Apache, conf/httpd.conf on RHEL or cPanel, and nginx.conf on nginx; a cPanel root ending in conf already names that directory. An absent selected Apache main file is unknown, even if a default tree exists elsewhere. A disabled or merely loaded mod_rpaf does not count as a rewrite. Malformed legacy-module arguments leave the state unknown. Cloudflare trusted-proxy hostnames or variables also leave it unknown when CSM cannot establish another active rewrite, because only Apache can resolve those values. Apache section suffixes are ignored only where Apache accepts them. Malformed closing tags and comments on closing tags for read-time conditions such as IfModule leave the state unknown; a rewrite directive found beside a parsing gap still counts as active. Files on disk can differ from what the web server loaded until it reloads, a setting in any virtual host applies to the whole host, and responses already queued are not checked again. SMTP password sprays read from Postfix mail logs can now supply forecast evidence. ModSecurity observations expire with the denial’s recent-log window. The startup Exim history scan names the newest counted send of the account it reports, using Exim’s full timestamp and zone, as the realtime mail log reader names that line. An invalid zone supplies no observation. At its event cap, the realtime cloud relay window keeps the current send’s address. An IP reputation finding supplies a forecast only with a local sighting: a record its log’s parser read the address from, dated by the server within the last 24 hours. An HTTP or cPanel sighting never uses an address listed in web_server.trusted_proxies or challenge.trusted_proxies. Admission still applies its normal evidence freshness check. The existing auto_response.dry_run setting decides whether admission forecasts these blocks or applies them.

A reader that starts inside an unfinished line still delivers its remainder to existing checks, but that fragment cannot supply admission evidence. Complete records that follow it keep their shared references.

Admission also binds each response target to its evidence. Only the mail subnet and ASN crawl entries may widen an attributed address to a containing prefix. Direct queue requests, retained candidates and reservations repeat this check, so previously queued responses cannot bypass it.

This setting is not a universal simulation mode. It gates automatic firewall and related network enforcement paths plus web-exposed-file virtual patches. File quarantine, cleanup, permission changes, and process termination are controlled by their individual auto_response flags. Leave those flags off while evaluating block policy.

IP auto-blocking still requires firewall.enabled: true. The firewall engine owns live nftables mutations and the dry-run gate; with the firewall disabled there is no engine to call, so csm validate warns on auto_response.enabled: true plus block_ips: true.

Verify dry-run state explicitly:

csm status --json | jq '.severities, .blocklist_size'
csm firewall status   # "Recently Blocked" entries with timestamps after the restart confirm live mode

To go live: set dry_run: false, then run systemctl reload csm. The field is hot-reload-safe, and a successful reload validates the config and re-records its hash. For a planned restart instead, run csm rehash once before restarting.

Verdict callback (advisory)

When verdict_callback.enabled: true, every auto-block call POSTs a signed JSON request to the panel before mutating nftables. CSM refuses to start without hmac_secret or a non-empty hmac_secret_env value unless allow_unsigned: true is set for a staged unsigned rollout. Without that opt-in, an unsigned allow response is rejected and the default block continues. When a secret is configured, CSM also requires the panel to sign the response body unless require_response_signature: false is set for a staged rollout. With that opt-out, CSM still checks any echoed nonce or timestamp when a secret is configured; a legacy response that omits both keeps working. The panel can return {"verdict": "block"} (apply), {"verdict": "allow"} (audit-only; CSM logs the decision and skips nftables), or attach metadata (tenant_id, note). The callback runs after local validation and infra-IP safety checks, and before the dry-run gate, so panels can observe dry-run decisions too. In dry run with the firewall on outside observe mode, admission also asks for each locally eligible block attempt, with CSM auto-block: <check> as the reason, up to four requests at a time, and uses the configured allow_unsigned and require_response_signature settings. A full set still allows a request when a lower-priority automatic block could be replaced or an equal-priority forecast block rotated out. Pending requests take no slots or replacement claims; admission checks capacity again after each answer, so competing requests cannot reuse one victim. An allow ends that forecast without a block. Until admission decides blocks, a panel can receive an additional request for the same automatic block. Local refusals, existing effects, capacity waits, coalescing and retries mean request counts need not be two per block. The v1 request has no dry-run field. Forecast examples and summaries are bounded, unsupported promotions are not simulated, and reload discards old policy answers.

CSM fails open on hook errors (timeout, non-2xx, malformed body): local admission continues, still subject to dry-run and capacity checks. An error proves neither a block nor a forecast example. The failure is written to the daemon log. Full request/response schema: Verdict callback contract.

Infrastructure IP DNS guard

Hostnames listed in top-level infra_ips or firewall.infra_ips are resolved every 5 minutes and their current addresses feed the infra auto-block guard. If a hostname stops resolving, the daemon emits an infra_ips_unresolvable Warning finding and keeps the last known addresses protected during the grace period (default 10 min). The finding auto-clears when resolution recovers.

Findings that always trigger IP block

When auto_response.block_ips: true and the firewall is enabled, qualifying findings in this list block the source IP. Per-row severity and challenge exceptions apply. The dry-run gate still applies if dry_run: true. Suppression rules do not stop these blocks; allowlist an address to exempt it.

Suppression rules also leave incident auto-blocking, credential-spray containment and central threat responses active. With database response enabled, suppression stops database cleanup and session revocation; the notice listing the session addresses is still sent. Suppress the action’s own check type to mute its notification; this does not disable enforcement.

Incidents and central threat responses receive new findings, including suppressed findings and checks that do not notify operators. Duplicate observations within a batch count once. Cross-account correlation uses only unsuppressed sources, and its derived alerts can be muted with their own suppression rules without removing them from enforcement.

FindingDescription
wp_login_bruteforceWordPress login flood via wp-login.php
xmlrpc_abuseXML-RPC endpoint flood
email_php_form_relayClient that made hosted sites mail many outside addresses through their forms (responses require email_protection.php_relay.ip_recipient_mode: enforce, including retries; routed to the PoW challenge first when challenge is enabled)
http_request_floodPer-IP HTTP request volume exceeds threshold (disabled by default; enable by setting thresholds.http_flood_threshold > 0)
http_scanner_profileRandom-URL probe pattern from one source IP (disabled by default; enable by setting thresholds.http_scanner_min_requests > 0; routed to the PoW challenge first unless auto_response.http_scanner_action: "block")
http_claimed_bot_unverifiedHigh-volume claimed crawler traffic while reverse-DNS verification is pending (routed to the PoW challenge first when challenge is enabled)
http_ua_spoofIP exceeding the UA anomaly threshold, including confirmed search-engine bot UA spoofing (periodic; see configuration.md for opt-in flags)
ftp_bruteforceFTP authentication flood
smtp_bruteforceSMTP authentication flood
smtp_probe_abuseRaw SMTP connect-rate flood before AUTH
mail_bruteforceIMAP/POP3/ManageSieve authentication flood without matching successful mailbox activity
mail_account_compromisedSuccessful login from an IP that repeatedly failed auth on the same mailbox, or a mail login after failures on two or more mailboxes including another one, or on a missing one. Only Critical findings block; High advisories are visibility only
admin_panel_bruteforcephpMyAdmin or Joomla admin POST flood
ssh_login_unknown_ipSSH login from an IP with no prior history, whether the realtime watcher or the periodic scan saw it first
c2_connectionOutbound connection to a known C2 server
ip_reputationIP flagged by AbuseIPDB / rspamd / upstream threat-intel
modsec_block_escalationModSecurity deny escalation
modsec_csm_block_escalationCSM-internal ModSecurity deny escalation
waf_attack_blockedWAF high-volume attacker
email_compromised_accountEmail account compromise indicator
email_cloud_relay_abuseCloud relay abuse

Successful cPanel, FTP, webmail and PAM login audit events and authenticated File Manager writes do not trigger direct blocks or challenges. Incidents containing only these audit events cannot request a block, even when they retain a higher severity from an older version. Independent attack evidence can still justify a response.

block_cpanel_logins gates cPanel multi-IP logins, API authentication failures, webmail brute force and realtime FTP authentication failures. This includes webmail challenge routing, since an unanswered challenge can become a block.

Queued block candidates are checked against the current check policy and login-blocking setting before retrying. Legacy queue entries without a check identity are discarded on upgrade; new eligible findings can queue them again. Existing firewall blocks and permanent evidence are not removed by this change.

Distributed HTTP flood rollups do not trigger a direct IP block because they describe one targeted vhost, not one source IP. The per-IP findings that feed the rollup still drive normal block decisions.

http_asn_crawl is also handled separately because one finding can carry several offending CIDRs rather than one source IP. Only Critical findings with confirmed PHP worker saturation can tempban those CIDRs, using auto_response.http_asn_crawl_tempban. Dry-run and subnet safety guards still apply.

mail_bruteforce_suspected and the High forms of mail_account_compromised are visibility only. They do not feed direct, incident, or spray auto-blocking. A separate blockable finding can still block the same source. mail_bruteforce_suspected also reports SMTP, IMAP and POP password sprays while spray detection runs in observe mode.

In enforce mode a spray block lasts thresholds.mail_spray_block_expiry (30 days by default) instead of block_expiry; an address already blocked keeps its current expiry. A block that mixes a spray finding with another finding for the address, such as a WAF or per-IP brute force finding, is a normal block: it lasts block_expiry and uses the scan budget. When the temporary block limit is full, the spray block that ends soonest is evicted first, and an operator’s temporary block is never evicted for an automatic one. A blocked address is remembered until one spray memory window after its block ends. One failed login from it in that time, on any mailbox, blocks it again for the spray length. That includes a failure logged just before an authentication outage is detected. On a shared IPv6 /64, an address blocked on the /64’s shared evidence keeps its own re-block until it is released too. An operator unblock, or an undo that removes a block, clears the address’s spray history: its re-block and the failure history of the address, or for IPv6 of its /64. An allow or temporary allow clears it even when the address holds no block, and so does the allow CSM adds for an address that a dyndns_hosts or infra_ips hostname resolves to. Its SMTP, IMAP and POP failure counts restart across all address spellings. Login trust and cracked-password records stay. Pending SMTP rejections are discarded so delayed replies cannot restore old spray evidence. A self-service unban through the panel API with an unban-scope token lifts the block and its re-block but keeps the failure history and counts, so renewed spraying is blocked again on the usual rules. An admin token or browser session on that route performs a full operator release. Failed firewall releases return an error without clearing threat state. A flush clears the spray history, per-address failure counts and pending rejections of every address. Authentication outage guards remain active after a release or flush.

Every auto-response block source - scan findings, challenge-timeout escalations, central-intel corroborated blocks, and incident spray containment - records the same evidence: a temporary threat-DB row, a blocked-IPs tracker entry, an auto_block finding visible to the block digest and alerting, and a step toward permanent-block escalation. A block the admission ledger applied is never promoted to a permanent block while it stands; its offense still counts toward a later block of the address. All of them block only while auto_response.enabled and block_ips are on; with either off, an expired challenge simply leaves the challenge list. Reloading these switches takes effect before the next block attempt, including later entries in a challenge-timeout batch. Incident block refusals caused by these switches keep the incident visible without a failed-block warning. Admission applies auto_response.max_blocks_per_hour to scan, challenge, central, and incident IP blocks. On the legacy path that budget applies to scan-driven blocks only.

Mail login spray blocks use a separate hourly budget of the same size. They reserve credit before the firewall attempt. An uncertain result keeps that credit; a confirmed non-block releases it. Retries wait when the spray budget is full, while other scan blocks keep their own budget. Waiting spray retries hold at most half of the retry queue and yield space to other retries when the queue fills. Deferred retries follow the same limits. One warning an hour names the budgets that are full when it is sent, and says when retries were dropped because the spray share of the queue was full. A failed spray reservation defers spray blocks and reports a warning, without stopping other scan blocks.

The same setting also sets the hourly ceiling of the admission ledger, which covers every IP block it executes, including challenge, central, and incident blocks. For the ledger an omitted or zero value selects 2000 and a value above 20000 is clamped to 20000; the per-IP limit above keeps reading omitted or zero as 50. An explicit value sets both. Packaged configs used to set 50, which keeps the ceiling at 50 until the key is removed; csm doctor warns about a configured ceiling below 2000. At its first start the ledger imports the current hour’s block count, so a restart cannot grant a fresh hour; a block count that cannot be read starts it without saved credit. Status and csm doctor show the ceiling and its source. A changed ceiling applies on reload, and a web UI settings save applies at the owner’s next clock tick; a failed change is retried on later ticks before admission resumes, with elapsed time credited at the saved ceiling before the new one applies. Every automatic response the existing blocking paths select is also handed to the ledger. With auto_response.dry_run false, the firewall on and outside observe mode, the ledger applies automatic IP blocks itself within this ceiling and the free slots of the temporary block set, and the existing paths leave those blocks to it. In dry run it forecasts them. With the firewall disabled or in observe mode it previews them; neither applies them. While a configured firewall engine is starting or unavailable, queued live blocks wait and new responses remain on the legacy path. Work the set has no room for waits as set_full until a slot frees: the slots operator, unknown and subnet blocks leave are divided as in the forecast, legacy automatic blocks occupy general slots, and an unlimited set counts as 10000 entries. A Critical local attack, or one with direct compromise or independently corroborated evidence, may instead take the general slot of a legacy automatic block that recorded its origin. With the firewall on, queued responses disabled by the current policy end as disabled without a withheld alert, including when their evidence expires before a restart or configuration reload. Turning a response back on requires a new eligible observation; ended work is not replayed. Status, csm doctor and findings history show the results. Forecast endings have no new per-attempt row in the action log.

The resulting auto_block findings are output evidence, not new local corroboration for central intelligence or incident correlation. They are deduplicated before the digest, attack database, history, and alert sinks receive them. Only a block that applied marks its address as auto-blocked in the attack database; a dry-run “would be blocked” warning does not.

Central intelligence and incident auto-blocking act on an address only when a finding names it as an attacker. The remote end of an outbound connection, a customer login, and advisories such as ModSecurity warnings or visitors of a mail-sending script do not count, even when another finding raised the incident’s severity. That includes a listed command-and-control server: only the C2 connection check blocks it, when auto_response.block_ips is on.

Known-customer block alert

When an automatic block applies to an address that logged in successfully as a customer in the last 30 days (cPanel, webmail, SMTP, IMAP, POP or FTP), CSM sends an alert at once: one email, and the generic alert webhook as event customer_block with the blocks under csm.customer_blocks (see the API reference). It names the address, the block reason and the check that caused it, and the customer’s login kind, account or mailbox and last login time. An IPv6 customer is named by its /64. A subnet block or netblock that covers known customers sends one message with their count and the five most recent. Every applied block counts: scan blocks, permanent promotions, challenge timeouts, central intelligence and incident blocks. A customer whose own login caused the block is left out, also when the block comes later than its cause: a mail account compromise or an FTP login after brute force names that address and account in its own Critical alert. This also applies when earlier address blocks escalate to a range block: its count and examples include only the remaining customers. Each block names the command that lifts it: csm firewall remove <ip> for an address, csm firewall remove-subnet <cidr> for a range. An allow rule (csm firewall allow <ip>) trusts the address on every port. It does not pass a blocked range, but CSM never blocks a range over an allowed address.

The alert is the false-positive check of automatic blocking, never an exemption. The block stands with its configured length, and its notice still goes to the daily summary, which also lists the block with its customers, so a failed alert still reaches you. A login with stolen credentials also makes an address known, so the alert can show the block of an attacker holding a customer’s password, which is worth seeing too. Each applied block sends at most one message; a later new block of the same address sends another. A promotion to a permanent block is named in its block’s message when both reach CSM together; a promotion that reaches CSM apart from its block sends a message of its own. The alert replaces the live watched-country alert for the same block, and the block still counts in the watched-country section of the daily summary. It does not spend the hourly alert budget, goes out with alerts.block_digest off and ignores alerts.block_digest.channel. A phpanel webhook does not receive it. Dry-run and observe-mode verdicts that would have blocked a known customer are listed in the daily summary instead.

Safety Guards

  • Never kills root processes, system daemons, or cPanel services
  • Infrastructure IPs (infra_ips and firewall.infra_ips) are never blocked or sent to the challenge page
  • A block of an address that recently logged in as a customer is alerted at once and never exempted
  • Subnet blocks refuse the default route and any range that covers infrastructure, local host, allowed, or port-specific allowed IPs
  • Quarantined files preserve full metadata for restoration
  • Every regular file is copied from its verified open descriptor into a private quarantine inode before the detected name is removed. Other hard links are reported after removal; a file swapped into the detected path is reported as a refused remediation, with the captured copy kept as evidence and the replacement left untouched
  • Realtime signature auto-quarantine requires high confidence: category webshell or dropper, file size at least 512 bytes, and either Shannon entropy >= 5.5 or hex density > 20% with an obfuscated-execution signal. This prevents legitimate WordPress plugins from being quarantined.
  • IP blocks are limited by auto_response.max_blocks_per_hour: admission defaults to 2000/hour and legacy scan blocking to 50/hour.
  • CRITICAL alerts and threat-intel reputation sightings always bypass the operator email/webhook rate limit (default 30/hour); lower-severity findings batched with them still count against it
  • Attacks CSM blocked or challenged, and its applied block and challenge notices, go to the daily summary instead of alert email and the generic webhook; an attack whose block failed, is still queued or ran in dry run is alerted immediately, as are a block of a Cloudflare-covered address and the web attacks from it
  • Trusted countries (trusted_countries) suppress login alerts from expected geolocations

What CSM Detects in Real-Time

Beyond standard malware patterns, CSM detects advanced evasion techniques:

  • Fragmented function names: attackers split base64_decode across variables ($a="base"; $b="64_decode") to evade simple string matching
  • Appended payloads: malicious code added to the end of large legitimate files, beyond typical scan windows. Realtime PHP checks scan the first and last 32KB, and periodic PHP content analysis scans a larger head window plus the tail.
  • Non-PHP backdoors: Perl, Python, Bash CGI scripts in web directories (detects toolkits like LEVIATHAN)
  • SEO spam injection: gambling/togel dofollow link injection into theme files
  • WordPress brute force: real-time access log monitoring for wp-login.php and xmlrpc.php floods (blocks within seconds, not the 10-minute periodic scan)
  • Admin-panel brute force: same access-log path, tracks POSTs to /phpmyadmin/index.php, /pma/index.php, /phpMyAdmin/index.php, and Joomla /administrator/index.php. Emits admin_panel_bruteforce and auto-blocks the IP. Path matcher is intentionally tight to avoid false positives on shared hosting; Drupal and Tomcat Manager use different attack shapes and need separate detectors.
  • SMTP brute force and probes: tails /var/log/exim_mainlog on cPanel and non-cPanel Exim hosts where the file exists. Emits smtp_probe_abuse and smtp_bruteforce (per-IP, auto-blocks), smtp_subnet_spray (per-/24, auto-blocks the whole subnet), and smtp_account_spray (per-mailbox, visibility only). Spray detection adds mail_bruteforce_suspected in observe mode and mail_account_compromised for a login after failures on other mailboxes.
  • Mail brute force: tails /var/log/maillog for direct IMAP, POP3, and ManageSieve auth failures. Composes with the existing geo-login monitor so email_suspicious_geo keeps working. Emits mail_bruteforce, mail_bruteforce_suspected, mail_subnet_spray, mail_account_spray, mail_account_compromised, and mail_auth_backend_degraded. Established good sources with a confined stale-password pattern emit the suspected advisory without auto-blocking. A compromise finding from an IP established on at least two other mailboxes stays visible as High without auto-blocking. Wider spraying and compromise without that standing still block. When the auth backend is degraded, mail_bruteforce and mail_subnet_spray auto-blocking pause until backend errors age out.
  • Mail auth backend probe (cPanel): independently of the log signals above, CSM opens the cpdoveauthd socket on a short interval. dovecot keeps answering its IMAP/POP3 ports during a cpdoveauthd outage, so cPanel’s own service checks do not notice, yet every login fails regardless of password. When the probe finds the socket unreachable CSM raises mail_auth_backend_degraded and pauses both mail and SMTP brute-force auto-block. With auto_response.mail_auth_recovery.restart_enabled, CSM restarts the mail service once the backend has been continuously down past down_grace (default 10m, rate-limited), so a brief blip during maintenance never triggers a needless restart. Changes to mail auth recovery settings require a daemon restart.

Network dry-run precedence

Three settings combine for direct-SMTP BPF enforcement. Any applicable dry-run value that is true wins; live network denial requires all three to be false.

LayerKnobDefaultEffect when true
Auto-responseauto_response.dry_runtrueRecord automatic block decisions without applying firewall denial
Detectordetection.direct_smtp_egress.dry_runtrueSuppress detector-scoped action
Kernelbpf_enforcement.dry_runtrueBPF program emits decision but allows traffic

The kernel knob is consulted by the BPF program itself; the others gate userspace action paths. All three default to true on a first install so a configuration mistake cannot start blocking traffic.

Observe mode

mode declares what CSM is allowed to do to the host it runs on.

ValueMeaning
enforce (default)Every subsystem acts under its own switch. This is how CSM has always behaved.
observeDetection, correlation, alerting and the audit sinks run. Automatic host remediation and integration updates are disabled.
mode: observe

Use observe mode to evaluate detection quality on a real host before granting CSM the ability to act, or to run CSM permanently as a reporting sensor next to another response tool.

What observe mode stops

Observe mode skips host integration work that otherwise runs automatically:

  • The auditd rules file is written and augenrules is run, so CSM’s audit layers stay current across package upgrades.
  • The host integration files are refreshed: the WHM plugin CGI and its AppConfig registration, and the CSM section of the ModSecurity user config.
  • Legacy challenge snippets and managed webserver integration snippets are refreshed, with webserver validation and reloads when needed.
  • The WAF check refreshes stale vendor rules and deploys custom ModSecurity rules during periodic scans.
  • The Exim forward guard is reconciled, including removing an installed guard when its switch is disabled.
  • The antispam integration is reconciled, including removing it when its switch is disabled. An installed integration is reported as observe_mode and left in place.
  • The AF_ALG kernel mitigation is re-applied on each critical tick when an opted-in hardening marker is present.

Existing host protections are left in place. Remove or change them explicitly before switching modes if that is the intended posture.

Resolve pending firewall apply/rollback windows in enforce mode before switching to observe. Observe startup refuses a pending settings or ruleset rollback: restoring it would change the host, while ignoring it would abandon the rollback deadline.

Contradictory settings are refused, not rewritten

A config that sets mode: observe and still enables a subsystem that changes host state is rejected at load, naming every conflicting key at once:

mode: observe forbids changing host state, but these keys still enable it:
auto_response.enabled (set false), firewall.enabled (set false)

The keys checked are auto_response.enabled, firewall.enabled, php_shield.enabled, bpf_enforcement.enabled, email_protection.forward_guard.enabled, email_protection.antispam.enabled, auto_response.php_relay.freeze, auto_response.mail_auth_recovery.restart_enabled, and auto_response.virtual_patch_exposed_files set to auto. It also refuses auto_response.copy_fail_kill_process: true and, when the antivirus scanner is enabled, email_av.quarantine_infected: true and email_av.fail_mode: tempfail, all of which act on a host independently of auto_response.enabled. Mail that makes the attachment parse worker fail is allowed with a finding instead. In enforce, CSM requests deferral whatever fail_mode says when a permission event still holds delivery.

CSM refuses rather than silently turning those switches off in memory, because the config re-hash path marshals the in-memory config back over csm.yaml: an in-memory override would eventually be written into the operator’s file.

What still runs

Detection is unchanged. Real-time watchers, scheduled checks, correlation, incidents, alerts, webhooks, the SSE stream and the audit-log sinks all behave as they do under enforce, except that checks report host drift without fixing it. CSM still writes its own configured state, log, cache and quarantine trees, and creates its runtime sockets under /run/csm (/var/run/csm). Explicit config edits and reloads can update the config integrity hashes.

Manual operator commands still work. csm firewall deny, csm clean, csm virtual-patch --apply, csm db-clean and csm harden are explicit actions an operator takes, not daemon behaviour, so observe mode does not block them. auto_response.virtual_patch_exposed_files: manual is accepted for the same reason.

Signature and GeoIP updates still run: they write only inside CSM’s own directories.

Two host writes remain, and the capability matrix marks both as not configurable:

  • On a BPF build, capability discovery loads and briefly attaches probe programs to find out what the kernel supports.
  • The Copy Fail check runs kcarectl --patch-info to see whether a livepatch covers the host. kcarectl refreshes its own cache under /var/cache/kcare on every run, including a read-only query.

Neither changes the host’s security configuration, and skipping them would cost the detection they exist for. They are listed here so “observe” is not read as a promise of zero writes.

Confirming the posture

csm doctor                 # "operating mode: observe (no automatic remediation or integration changes)"
csm status                 # mode: observe
csm status --json          # .mode
curl .../api/v1/status     # .mode

The capability string mode.observe.v1 reports that a build understands the setting.

Changing mode requires a restart (systemctl restart csm), not a reload. Doctor compares the running mode with the configured mode and warns if they differ. When the daemon is unreachable, it reports only the configured mode.

Capability matrix

Every operation CSM performs that needs privilege beyond reading its own files, or that writes outside its own directories, with the privilege it needs and the setting that stops it.

Print it on any host, before or after install:

csm privileges              # table
csm privileges --json       # machine-readable

The command prints a static inventory. It does not load CSM configuration or probe host privileges, so it can be used before installing the service. Output failures produce a nonzero exit status in text, JSON and Markdown formats.

How to read it

Needs is what the operation requires from the kernel. root means uid 0 rather than one capability: the operation reads or writes files owned by many different accounts, or drives a panel tool that assumes root. Today the daemon runs as a single root process; splitting it into a small privileged helper plus a reduced-privilege main process is planned, and this inventory is its first stage. Capability names describe the privileged kernel interfaces as well as the root filesystem access. For BPF, tracing and LSM programs need CAP_BPF plus CAP_PERFMON; cgroup socket programs need CAP_BPF plus CAP_NET_ADMIN. CAP_SYS_ADMIN is the broader fallback on older kernels. These requirements follow the kernel program-load checks.

Trigger is automatic when the daemon may start the operation on its own, and operator when it runs only in response to a command or a button. Not running the command is how an operator operation is turned off.

Writes lists what the operation writes while it runs, including writes made by a tool it invokes. Paths starting with / are filesystem paths; everything else is a resource named <kind>:<name>. “nothing (read-only)” means the operation only reads. Paths describe default locations; configured account roots, state, rule, log, spool and policy paths replace those defaults and may require service drop-ins. Shared CSM state and audit-log writes are grouped in the state rows. This is not a list of every possible path written by an external panel tool.

Turn it off is the config key and YAML value that stops the named operation. Dotted keys describe nested YAML mappings; they are not literal top-level keys. Apply changes before restarting the daemon: several controls are read only at startup. A row saying “not configurable” has no single config switch that stops all its callers. disabled_checks takes a list and only affects scheduled checks. Stopping new work does not undo installed rules, hooks, quarantines or blocks; use the subsystem removal commands for that. Disabling the forward guard or antispam removes its existing Exim and rspamd configuration, which is itself a host write, so both rows name mode: observe.

Action record says whether the operation writes to the action log. “no” does not mean the operation is silent; it means it is not on that stream yet and the daemon log is where it appears.

Without the privilege is what an operator loses by withholding it, so the row reads as a decision rather than a demand.

Relationship to the systemd sandbox

The daemon runs under ProtectSystem=strict with an explicit writable-path allow-list; see service confinement. The two lists are compared by tests: every declared in-daemon filesystem write must fit a packaged grant, and every grant must have an in-daemon writer. Empty inventories and grant lists fail. Paths are compared on directory boundaries, and an operation outside the sandbox cannot justify a daemon grant. The unit parser rejects invalid paths and unsupported syntax, including continuations, specifiers and directory flags, instead of guessing which locations are writable. Tests cannot discover omitted operations or prove the privilege and disable claims; those require tracing the runtime code.

Operations marked “outside the systemd sandbox” run as standalone operator commands or transient services forked by PID 1. The latter use direct execution only where their helper permits it; the archive parse worker has no direct fallback when systemd is unavailable. Mixed operations have separate rows for their in-daemon writes, such as the AF_ALG marker repair and forward-guard lookup refresh. The service sandbox does not bound external tools. Archive worker memory limits are also written inside the daemon sandbox. Their cgroup: resources are kernel controls, writable because the packaged unit keeps ProtectControlGroups=no, rather than ordinary directory grants.

Turning most of it off at once

mode: observe stops automatic remediation and integration deployment, and refuses configurations that enable those subsystems. It still maintains CSM state, loads temporary BPF capability probes in BPF builds, and invokes the KernelCare probe, which can update its own cache. Archive worker startup cleanup and on-demand unit launches also remain active. It is not a guarantee of zero host writes. See observe mode.

The matrix

The risk tier says what can go wrong if an operation acts on the wrong target: 0 changes nothing outside CSM’s own directories, 1 records a recommendation or dry-run decision, 2 is a low-risk change such as a probe, challenge gate or mail hold, 3 quarantines, blocks or denies a target, and 4 signals processes, reloads services, or rewrites content or configuration. Rows describe maximum live effect, so preview-only tier 1 has no separate row today. A tier does not promise automatic rollback; see recovery gaps below and the current contracts in the JSON inventory. The TIER column of csm privileges and the JSON inventory use the same 0 to 4 numbers.

OperationNeedsTriggerRisk tierWritesTurn it offAction recordWithout the privilege
state.control_socket
create the private control socket used by operator commands
rootautomatic0/var/run/csmnot configurablenoCLI commands cannot reach the daemon
state.mail_relay_policies
load operator-supplied mailer classes and proxy ranges for the PHP-relay detector
noneautomatic0nothing (read-only)email_protection.php_relay.enabled: falsenoonly policy files readable by the daemon uid can be loaded
state.php_shield_events
receive PHP Shield events over a socket and append their local archive
rootautomatic0/var/log/csm-php-shieldphp_shield.enabled: falsenoPHP runtime events are not collected
state.sign_config
rewrite CSM’s own integrity hashes into csm.yaml after an approved change
rootoperator0/etc/csmdo not run the commandnothe integrity hashes cannot be re-recorded, so the next restart refuses to start after any config edit
state.update_forge
download and verify YARA Forge rules independently of YAML updates
rootautomatic0/opt/csm/rulessignatures.yara_forge.enabled: falsenoYARA Forge rules are not refreshed
state.update_signatures
download and signature-verify YAML malware rule updates
rootautomatic0/opt/csm/rulessignatures.update_url: ""nodetection freezes at the ruleset shipped with the installed package
state.write_logs
write the daemon log and the audit-log sinks that feed a SIEM
rootautomatic0/var/log/csmnot configurablenono local record of findings or actions
state.write_store
write the bbolt state database, baselines, incidents and scan reports
rootautomatic0/var/lib/csm, /opt/csm/statenot configurablenoCSM cannot run: without state there is no baseline, no dedup and no incident history
detect.account_databases
read account MySQL credentials and scan databases for injected content and stored objects
rootautomatic0nothing (read-only)not configurable: db_object_scanning only stops stored-object scans; content and requested scans remainnono database detection: injected admin users, poisoned options and malicious triggers stay invisible
detect.af_alg_sockets
deny AF_ALG sockets through BPF LSM, or observe socket use through the audit-log fallback
CAP_BPF, CAP_PERFMON, rootautomatic3kernel:AF_ALG socket denialdetection.af_alg_backend: nonenothe live monitor is gone; the periodic critical check still reports an exposed kernel
detect.audit_rules
query loaded audit rules with auditctl to check detection coverage
CAP_AUDIT_CONTROL, rootautomatic0nothing (read-only)not configurablenoaudit-rule coverage cannot be verified
detect.bpf_probe
load and briefly attach BPF LSM, tracepoint and cgroup programs to discover kernel support
CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN, rootautomatic2kernel:temporary BPF programs and mapsnot configurable: capability discovery runs independently of monitor settings in BPF buildsnoBPF capabilities report unavailable; configured automatic backends use their fallbacks
detect.file_placement_bpf
watch files renamed or hard-linked into watched trees through BPF LSM rename and link hooks
CAP_BPF, CAP_PERFMON, rootautomatic2kernel:BPF programs and mapsdetection.placement_backend: nonenoauto mode falls back to the fanotify placement group where the kernel has it; an explicitly selected BPF backend stays unavailable
detect.file_placement_fanotify
watch files renamed or hard-linked into watched trees through a fanotify group that reports directory handles, opened with open_by_handle_at
CAP_SYS_ADMIN, CAP_DAC_READ_SEARCH, rootautomatic0nothing (read-only)detection.placement_backend: nonenofiles renamed or hard-linked into a site are checked by the next deep scan instead of at once
detect.filesystem_events
fanotify stream over account roots and world-writable temp directories
CAP_SYS_ADMINautomatic0nothing (read-only)not configurablenono real-time file detection; scheduled scans still run, so a webshell lives until the next cycle
detect.kernel_livepatch_probe
run kcarectl –patch-info to see whether a KernelCare livepatch covers Copy Fail; kcarectl rewrites its own cache on every run
rootautomatic2/var/cache/kcarenot configurable: the startup kernel probe and hardening audits also invoke kcarectl, including in observe modenoa patched kernel reads as vulnerable, so the AF_ALG finding cannot be cleared
detect.kernel_oom
read the restricted kernel message buffer with dmesg to find recent OOM kills
CAP_SYSLOG, rootautomatic0nothing (read-only)not configurablenoswap statistics remain available, but OOM kills are not reported
detect.mail_queue_probe
query the Exim queue through a transient unit, because Exim opens its logs even to answer a read
rootautomatic2/var/log/exim_mainlog, /var/log/exim_paniclog, /var/log/exim4 (outside the systemd sandbox)disabled_checks: [mail_queue]noqueue depth is unknown, so a queue-size spam outbreak may go undetected
detect.outbound_connections
watch outbound connections through BPF cgroup hooks, or by polling /proc/net/tcp
CAP_BPF, CAP_NET_ADMIN, rootautomatic2kernel:BPF programs and mapsdetection.connection_tracker_backend: nonenoauto mode uses polling; an explicitly selected BPF backend stays unavailable
detect.pam_events
receive authentication attempts from the pam_csm.so hook over a private socket
rootautomatic0/var/run/csmnot configurablenono immediate brute-force or credential-stuffing block; log parsing lags by up to a minute
detect.parse_worker_memory
cap the memory and swap of the archive and mail parse workers through kernel cgroup controls inside the daemon sandbox
rootautomatic2cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dparse\x2dworker*.slice/csm-parse-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dparse\x2dworker.slice/csm-parse-worker-*.servicenot configurable: disabled_checks only suppresses scheduled phishing checks and email_av.enabled: false stops the mail workers; requested account scans remainnoa worker gets no request if its memory and swap cannot be capped: the phishing check reports itself incomplete and email attachments are handled as incomplete scans under email_av.fail_mode
detect.parse_worker_units
start the archive and mail parse workers as locked-down transient units and attempt to kill leftover parse worker units at startup
rootautomatic2systemd:transient units, systemd:runtime slice CPU weights (outside the systemd sandbox)not configurable: disabled_checks only suppresses scheduled phishing checks and email_av.enabled: false stops the mail workers; requested account scans and startup cleanup remainnotenant zip archives are not checked for phishing kits, the phishing check reports itself incomplete, and email attachments are handled as incomplete scans under email_av.fail_mode
detect.phptaint_worker_memory
cap the memory and swap of the PHP taint worker through kernel cgroup controls inside the daemon sandbox
rootautomatic2cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dphptaint\x2dworker.slice/csm-phptaint-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dphptaint\x2dworker.slice/csm-phptaint-worker-.servicenot configurable: disabled_checks with php_taint_deep stops new analyses; startup cleanup of leftover units remainsnoPHP files are not analyzed for remote-input taint
detect.phptaint_worker_units
start the PHP taint worker as a locked-down transient unit and attempt to kill leftover PHP taint worker units at startup
rootautomatic2systemd:transient units, systemd:runtime slice CPU weights (outside the systemd sandbox)not configurable: disabled_checks with php_taint_deep stops new analyses; startup cleanup of leftover units remainsnoPHP files are not analyzed for remote-input taint
detect.process_exec
watch process execution through a BPF tracepoint, or by walking /proc
CAP_BPF, CAP_PERFMON, rootautomatic2kernel:BPF programs and mapsdetection.exec_monitor_backend: nonenoauto mode uses periodic /proc sampling; an explicitly selected BPF backend stays unavailable
detect.read_service_logs
read mail, authentication, web server and ModSecurity logs
rootautomatic0nothing (read-only)not configurablenono mail abuse, brute-force or WAF detection: these logs are root-readable only
detect.scan_account_files
read every account’s files for scheduled, real-time and on-demand scans
CAP_DAC_READ_SEARCH, rootautomatic0nothing (read-only)not configurable: disabled_checks only suppresses scheduled checks; realtime and requested scans remainnoonly files readable by CSM’s own uid are scanned, which on a shared host is close to nothing
detect.sensitive_file_writes
watch writes to /etc/shadow and comparable files through a BPF LSM hook
CAP_BPF, CAP_PERFMON, rootautomatic2kernel:BPF programs and mapsdetection.sensitive_files_backend: nonenosensitive-file changes are found by periodic hashing instead of at the moment of the write
detect.yara_worker_memory
cap the memory and swap of the YARA scan and compile workers through kernel cgroup controls inside the daemon sandbox
rootautomatic2cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dyara\x2dworker.slice/csm-yara-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dyara\x2dworker.slice/csm-yara-worker-.service, cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dyara\x2dcompile\x2dworker.slice/csm-yara-compile-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dyara\x2dcompile\x2dworker.slice/csm-yara-compile-worker-.servicesignatures.yara_worker_enabled: falsenofiles and mail attachments are reported as not scanned by YARA
detect.yara_worker_units
start the YARA scan and compile workers as locked-down transient units and attempt to kill leftover YARA worker units at startup
rootautomatic2systemd:transient units, systemd:runtime slice CPU weights (outside the systemd sandbox)signatures.yara_worker_enabled: falsenofiles and mail attachments are reported as not scanned by YARA
integrate.antispam
install, update or remove CSM’s rspamd settings, optional feeds, learning files, the webmail Junk-button include and Exim spam-filter hooks, rebuild the Exim configuration and reload Exim and rspamd; on Postfix hosts only CSM’s rspamd files and an rspamd reload
rootautomatic4/etc/rspamd/local.d, /etc/rspamd/override.d, /etc/rspamd/lua.local.d, /usr/local/cpanel/base/3rdparty/roundcube/config/inc.d, /usr/local/cpanel/etc/exim/acls, /etc/exim.conf.local, /etc/exim.conf, /var/lib/csm, exim:rebuild artifacts and service reload, service:rspamd reload (outside the systemd sandbox)mode: observenoCSM’s spam-filter settings and hooks cannot be installed, updated or removed; existing mail filtering keeps its current configuration
integrate.antispam_ip_map
write the CSM IP map rspamd reads, readable only by root and rspamd’s group, and delete it once no rule reads it
rootautomatic2/var/lib/csm-rspamdmode: observenorspamd gets no scores from CSM’s own evidence; other filtering is unchanged
integrate.antispam_learn
learn an operator’s file as spam in rspamd’s local fuzzy storage, or forget the clusters it matches, keeping one anchor message per cluster
rootoperator2/var/lib/csm/antispam/learning, rspamd:fuzzy storageemail_protection.antispam.learning.enabled: falseyesoperators cannot teach or correct antispam learning; rspamd’s other scoring is unchanged
integrate.antispam_record
invalidate pending antispam workers before reconciliation and record each new request under the shared Exim lock inside the daemon sandbox
rootautomatic0/var/lib/csm/forward_guardnot configurable: pending workers must be invalidated even when antispam is disabled or mode is observenopending antispam workers cannot be invalidated and new antispam changes are not handed to the root worker
integrate.auditd_rules
write CSM’s auditd rules and reload them, so audit-backed detection survives package upgrades
CAP_AUDIT_CONTROL, rootautomatic4/etc/audit, kernel:audit rulesmode: observenoaudit-backed detection layers stay inactive after an upgrade
integrate.challenge_port_gate
install the separate nftables port gate for the public challenge listener
CAP_NET_ADMIN, rootautomatic2nftables:challenge port gatechallenge.port_gate.enabled: falsenothe listener remains reachable without the challenge port filter
integrate.challenge_snippet
refresh the web server snippet and rewrite maps that route challenged visitors to CSM’s proof-of-work listener, and reload the web server when the snippet changed
rootautomatic4/etc/apache2/conf.d, /etc/apache2/conf-enabled, /etc/httpd/conf.d, /etc/nginx/conf.d, /usr/local/lsws/conf/templates, /var/cache/csm, service:web server reloadmode: observenosuspicious visitors are blocked outright instead of being offered a challenge
integrate.firewall_ruleset
build and load CSM’s nftables table, including the operator’s port policy and rate limits
CAP_NET_ADMIN, rootautomatic4nftables:csm tablefirewall.enabled: falseyesno firewall management; another tool owns the host’s packet policy
integrate.modsec_section
refresh the managed ModSecurity section at startup and during periodic WAF checks, preserving operator configuration
rootautomatic4/etc/apache2/conf.d, /usr/local/apache/confmode: observenoCSM’s virtual-patch WAF rules are not installed
integrate.panel_plugin
deploy the WHM plugin CGI and its AppConfig entry, and register it with the panel
rootautomatic4/usr/local/cpanel/whostmgr/docroot/cgi, /var/cpanelmode: observenono panel plugin; the web UI is still reachable on its own port
integrate.php_shield
install the PHP runtime hook and register its shared event directory for an operator-scheduled CageFS remount
rootoperator4/opt/csm, /var/log/csm-php-shield, /etc/cagefs, /etc/csm, /opt/cpanel, /opt/alt, /usr/local/lsws, php:runtime configuration (outside the systemd sandbox)do not run the commandnono PHP runtime blocking of uploads and temp-directory execution
integrate.waf_vendor_rules
ask the panel to update stale ModSecurity vendor rulesets when a periodic check finds them out of date
rootautomatic4/etc/apache2/conf.d, /usr/local/apache/conf, /var/cpanel, service:web server reloadmode: observenostale WAF rules are reported but not refreshed
operate.apply_hardening
apply the hardening audit’s mechanical fixes for one section on an admin’s web UI request, in a root worker started outside the sandbox, then reload the services that read them
rootoperator4/etc/ssh/sshd_config, /etc/ssh/sshd_config.csm-bak, /etc/sysctl.d, kernel:sysctl values, /etc/shadow, /tmp, /var/tmp, /etc/cpupdate.conf, /usr/bin, cpanel:tweak settings, /opt/cpanel, /etc/php, /etc/php.d, /usr/local/lsws/admin/tmp, service:sshd, PHP-FPM and Apache reload (outside the systemd sandbox)do not run the commandnoeach fix is applied by hand from the audit’s suggested fix
operate.clear_stalwart_ban
lift Stalwart’s own automatic bans of an address unblocked, allowed or unbanned in CSM or allowed through a DynDNS name, through the Stalwart API; other Stalwart bans stay
rootoperator2stalwart:blocked address list (outside the systemd sandbox)do not run the commandyesStalwart keeps refusing the address until its ban ends or is lifted in Stalwart
operate.delete_forwarder
delete one destination of a mail forwarder through uapi, as the account that owns it, on an admin’s web UI request
rootoperator4cpanel:mail forwarders (outside the systemd sandbox)do not run the commandnoforwarders are deleted in cPanel
operate.export_archives
read protected configuration, state or account evidence and export backup or forensic archives, temporary snapshots and checksum sidecars
rootoperator4filesystem:operator-selected archive destinations (outside the systemd sandbox)do not run the commandnoprotected configuration, state and account evidence cannot be exported completely
operate.harden_host
apply a supported CVE mitigation: a modprobe blacklist, or seccomp drop-ins for the services that need one
CAP_SYS_MODULE, rootoperator4/etc/modprobe.d, /etc/systemd/system, service:restart, kernel:modules (outside the systemd sandbox)do not run the commandnothe mitigation is applied by hand from the command the audit prints
operate.install_service
install or remove CSM itself: the systemd unit, the PAM hook, logrotate and the panel integrations
CAP_AUDIT_CONTROL, CAP_LINUX_IMMUTABLE, rootoperator4/opt/csm, /etc/csm, /var/lib/csm, /var/log/csm, /etc/systemd/system, /etc/pam.d, /etc/logrotate.d, /etc/audit, /usr/sbin/csm, /lib64/security, /usr/lib64/security, /lib/security, /lib/x86_64-linux-gnu/security, /usr/lib/x86_64-linux-gnu/security, /lib/aarch64-linux-gnu/security, /usr/lib/aarch64-linux-gnu/security, /usr/local/cpanel/whostmgr/docroot/cgi, /var/cpanel, service:csm and panel integrations, /etc/cron.d, /var/cache/csm, /opt/cpanel, /var/run/csm, /etc/apache2/conf.d, /etc/apache2/conf-enabled, /etc/httpd/conf.d, /etc/nginx/conf.d, /usr/local/apache/conf, /usr/local/lsws/conf/templates (outside the systemd sandbox)do not run the commandnoCSM cannot be installed as a service
operate.manual_firewall
block, allow, tempban or flush addresses on request, and roll a firewall apply back
CAP_NET_ADMIN, rootoperator3nftables:csm setsdo not run the commandyesfirewall changes are made with nft or the panel’s own tooling
operate.manual_remediation
clean or quarantine files and spool messages, truncate malicious crontabs, kill verified malware processes, or change database objects on request
CAP_KILL, rootoperator4/home, /tmp, /var/tmp, /dev/shm, /var/spool/cron, /var/spool/exim/input, /var/spool/exim4/input, /opt/csm/quarantine, mysql:account databases, process:signaldo not run the commandnoremediation is done by hand over SSH
operate.rehash
re-hash configuration, converge legacy config copies, set binary immutability and refresh the launcher, service and log rotation
CAP_LINUX_IMMUTABLE, rootoperator4/opt/csm, /etc/csm, /etc/systemd/system, /etc/logrotate.d, /usr/sbin/csm, service:daemon reload (outside the systemd sandbox)do not run the commandnohash recording and upgrade integration refresh cannot complete
operate.restore_backup
stage and restore protected configuration, drop-ins and state from a backup while the daemon is stopped
rootoperator4/etc/csm, /var/lib/csm, /opt/csm, /tmp (outside the systemd sandbox)do not run the commandnoconfiguration and state cannot be restored
operate.stalwart_setup
prepare Stalwart through its admin API: log failed logins, log rejected spam, trust the webmail client address behind CSM’s port lock, match ban length to CSM’s blocks, and create the role, service account and API key CSM acts through
rootoperator2stalwart:failed-login and spam-verdict log levels, HTTP client address source, ban periods, settings reload, stalwart:response role, service account and API key, /etc/csm/stalwart-api.key (outside the systemd sandbox)do not run the commandnoStalwart is prepared by hand in its admin interface
operate.truncate_error_log
empty an account error log that has grown large enough to threaten the filesystem
rootoperator4/homedo not run the commandnoa bloated log is reported and truncated by hand
operate.release_mailbox
release a mailbox suspension or account outgoing-mail hold CSM made, on an operator’s request
rootoperator2cpanel:mailbox login and outgoing suspension, cpanel:account outgoing mail hold, stalwart:account sign-in permission (outside the systemd sandbox)do not run the commandyesthe suspension stays until it is released in cPanel or Stalwart
respond.af_alg_enforce
unload the AF_ALG kernel modules again when an opted-in mitigation marker is present
CAP_SYS_MODULE, rootautomatic4kernel:modules (outside the systemd sandbox)auto_response.disable_enforce_af_alg: truenoa module reload silently reopens the Copy Fail exposure
respond.af_alg_kill
kill a verified AF_ALG socket caller through the separate Copy Fail response setting
CAP_KILL, rootautomatic4process:signalauto_response.copy_fail_kill_process: falsenothe process is reported but remains running; this path is independent of kill_processes
respond.af_alg_marker
restore a changed AF_ALG mitigation marker after the operator has opted in
rootautomatic4/etc/modprobe.dauto_response.disable_enforce_af_alg: truenoa changed module blacklist is reported but cannot be repaired
respond.block_ip
add an attacker address or subnet to the firewall’s deny sets
CAP_NET_ADMIN, rootautomatic3nftables:csm setsauto_response.block_ips: falseyesattacks are reported but not stopped; dry-run records what would have been blocked
respond.bpf_deny_egress
deny matched outbound connections in the kernel through a BPF cgroup hook
CAP_BPF, CAP_NET_ADMIN, rootautomatic3kernel:bpf cgroup programbpf_enforcement.enabled: falsenodirect-to-MX spam egress is detected but not stopped at the source
respond.clean_file
strip injected code from a PHP or access file, keeping a pre-clean backup
rootautomatic4/home, /tmp, /var/tmp, /dev/shm, /opt/csm/quarantineauto_response.enabled: falseyesinjected files are reported and left in place
respond.database_cleanup
revoke a rogue CMS admin, sanitize poisoned options, and drop confirmed malicious stored objects after recording their definition
rootautomatic4mysql:account databasesauto_response.clean_database: falsenodatabase persistence survives file cleanup and re-infects the account
respond.enforce_permissions
chmod a world-writable or group-writable PHP file back to 644
rootautomatic3/homeauto_response.enforce_permissions: falsenowritable code files are reported only
respond.fix_wp_cron
disable WP-Cron for an account and install a per-user system cron entry instead
rootautomatic4/home, /var/spool/cron, /tmpauto_response.fix_wp_cron: falsenorunaway WP-Cron load is reported only
respond.forward_guard
install or remove the managed Exim block that holds forwarded spam and backscatter, and rebuild the Exim configuration
rootautomatic4/etc/exim.conf.local, /etc/exim.conf, /var/lib/csm, exim:rebuild artifacts and service reload (outside the systemd sandbox)mode: observenoforwarded spam keeps damaging the host’s sending reputation
respond.forward_guard_check
record the latest forward-guard request and check whether a rule is installed under the shared Exim lock; cancel pending requests in observe mode without waiting for the lock, inside the daemon sandbox
rootautomatic0/var/lib/csm/forward_guardnot configurable: pending forward-guard workers must be canceled even in observe modenoforward-guard changes are not handed to the root worker, pending workers cannot be canceled, and the installed guard stays as it is
respond.forward_guard_lookup
refresh the forward-guard bad-sender lookup inside the daemon sandbox
rootautomatic0/var/lib/csm/forward_guardemail_protection.forward_guard.enabled: falsenoExim continues using the last successfully written sender lookup
respond.freeze_mail
freeze queued Exim messages attributed to a confirmed PHP-relay finding
rootautomatic2/var/spool/exim/input, /var/spool/exim4/input, /var/log/exim_mainlog, /var/log/exim_paniclog, /var/log/exim4 (outside the systemd sandbox)auto_response.php_relay.freeze: falsenoa compromised script keeps sending until an operator freezes the queue
respond.hold_outgoing_mail
request a cPanel account outgoing-mail hold through whmapi1 after sustained mail abuse
rootautomatic2cpanel:account outgoing mail holdauto_response.enabled: falseyesthe mail abuse finding is reported but the account can continue sending
respond.kill_process
signal a malicious process through a kernel process handle, never a recycled PID, never root
CAP_KILL, rootautomatic4process:signalauto_response.kill_processes: falseyesreverse shells and miners keep running until an operator kills them
respond.mail_delivery_gate
defer Exim delivery with fanotify permission responses when tempfail policy requires it or a message makes its parse worker fail
CAP_SYS_ADMIN, rootautomatic2fanotify:mail delivery decisionsemail_av.enabled: falsenomail AV falls back to notifications and cannot defer delivery on scan failures
respond.quarantine_file
move a confirmed malicious file out of an account tree into CSM’s quarantine, preserving owner, permissions and mtime
CAP_LEASE, rootautomatic3/home, /tmp, /var/tmp, /dev/shm, /opt/csm/quarantineauto_response.quarantine_files: falseyesmalware is reported and left in place
respond.quarantine_mail
move an infected message out of the Exim spool after an antivirus match
rootautomatic3/var/spool/exim/input, /var/spool/exim4/input, /opt/csm/quarantineemail_av.quarantine_infected: falsenoinfected mail is reported and delivered
respond.restart_mail_auth
restart the panel’s mail authentication service after a sustained outage
rootautomatic4service:mail authenticationauto_response.mail_auth_recovery.restart_enabled: falsenoan authentication backend outage is alerted but not repaired
respond.suspend_mailbox
suspend one mailbox’s logins and outgoing mail through uapi, or its Stalwart sign-ins through the Stalwart API, after its credentials are abused
rootautomatic2cpanel:mailbox login and outgoing suspension, stalwart:account sign-in permission (outside the systemd sandbox)auto_response.enabled: falseyesthe credential abuse finding is reported but the mailbox keeps logging in and sending
respond.virtual_patch
write a reversible deny rule into an account access file to close an exposed file
rootautomatic3/home, /opt/csm/quarantineauto_response.virtual_patch_exposed_files: offnoexposed backups, dumps and configs stay reachable until the account owner fixes them

Regenerate the table with go run ./cmd/csm privileges --markdown. A gate in internal/ci fails when the page and the inventory disagree.

Risk tiers and recovery coverage

csm privileges --json reports each operation’s risk tier as a number from 0 to 4. A row with no tier would report -1; the test suite refuses such an inventory, so a release never ships one. The tier is the operation’s maximum live effect on a wrong target. It is not the severity of the finding that triggered the operation.

Eight operations carry a safety contract in the JSON inventory: respond.block_ip, operate.manual_firewall, integrate.firewall_ruleset, integrate.challenge_port_gate, integrate.challenge_snippet, integrate.antispam_ip_map, respond.quarantine_file and respond.clean_file. Each states the authority it needs, how the target is revalidated before the change, how the change is reversed, and the limit that bounds it. The contracts describe current behaviour, including what it does not cover. Every other operation that changes the host states the recovery this inventory does not yet cover:

OperationsRecovery gap
detect.bpf_probe, detect.outbound_connections, detect.process_exec, detect.sensitive_file_writes, detect.file_placement_bpf, detect.af_alg_sockets, respond.bpf_deny_egressThis inventory does not yet specify verified detach, map restoration and crash recovery for these kernel hooks.
detect.kernel_livepatch_probe, detect.mail_queue_probeThis inventory does not specify recovery of incidental external cache or log writes by probe commands.
detect.parse_worker_memory, detect.phptaint_worker_memory, detect.yara_worker_memoryMemory and swap limits end with the worker cgroup; this inventory specifies no separate cleanup contract.
detect.parse_worker_units, detect.phptaint_worker_units, detect.yara_worker_unitsWorker units and their cgroups end with the unit; empty slices and their runtime CPU weights remain until reboot. Startup attempts to kill leftover workers and logs failures.
respond.mail_delivery_gate, respond.hold_outgoing_mail, respond.suspend_mailbox, respond.freeze_mail, respond.quarantine_mail, operate.release_mailboxThis inventory does not yet specify identity checks for releasing or restoring mail, or recovery after a restart.
respond.af_alg_enforce, respond.af_alg_marker, respond.forward_guard, respond.fix_wp_cron, integrate.auditd_rules, integrate.modsec_section, integrate.panel_plugin, integrate.php_shield, integrate.waf_vendor_rules, operate.harden_host, operate.install_service, operate.rehash, operate.restore_backupThis inventory does not specify an action-wide snapshot and verified rollback of configuration, service and external-tool side effects.
respond.af_alg_kill, respond.kill_process, respond.restart_mail_authProcess termination and restart cannot restore lost process state; this inventory does not yet specify a full recovery contract.
respond.database_cleanup, respond.virtual_patch, respond.enforce_permissions, operate.manual_remediation, operate.truncate_error_logCurrent remediation may retain local recovery evidence, but per-operation identity-checked undo and partial-failure recovery are not specified by this inventory.
integrate.antispamRollback cannot change existing SMTP sessions; failed rollback or an interrupted worker needs another apply or remove transaction.
integrate.antispam_learnForget removes CSM’s contribution but cannot restore the near-match coverage other learned messages lost to it, and a lost reply is settled by reconciliation, not undone.
operate.export_archivesExport can replace an existing operator-selected archive; removing the new archive does not restore overwritten bytes.
operate.delete_forwarderThe web UI audit log names the deleted destination and its account; nothing re-creates the forwarder.
operate.apply_hardeningsshd_config keeps a .csm-bak copy and is restored when sshd -t rejects the change; sysctl, permission, WHM tweak, cPanel update and PHP drop-in changes are undone by hand.
operate.stalwart_setupSetup prints the previous values; undoing a change is done by hand in Stalwart’s admin interface.
operate.clear_stalwart_banStalwart bans the address again on new abuse; a cleared ban is not restored.

Self-test

csm selftest scans a bundle of samples whose verdicts are known and reports what the installed rules catch. It reads no account data and writes nothing, so it is safe to run on a production host, and it answers the question an evaluator actually has before pointing a scanner at real sites.

csm selftest
csm selftest --json

Exit status is 0 when every sample matched what the bundle records for each available engine, and 1 when anything needs attention. Incomplete or empty rule loads, scan errors, invalid samples and an empty bundle fail the run. A build without YARA-X can pass its realtime checks, but explicitly reports that YARA coverage was not tested.

The command uses the configured rules directory and disabled-rule list for both engines. Disabling a detection can turn a sample into MISSED. A missing or unreadable configuration fails the command rather than measuring a full packaged ruleset that the host may not run. Pass --config and --config-dir when the daemon uses custom paths.

What the bundle contains

Adversarial samples and benign controls, in one list. The controls are the half that decides whether a rule set is usable: a scanner that flags an ordinary WordPress plugin is worse than one that misses a shell.

Samples are stored base64-encoded and decoded only in memory. Endpoint antivirus deletes files that look like web shells, and a decoded copy on disk would make the bundle disappear from the machine it is meant to test.

The split-assert sample represents legacy PHP string assertions, not PHP 8 behaviour. The chr-built sample constructs a callable command-execution function; the test suite checks that it is callable without running its payload.

Reading the output

VerdictMeaning
detectedAn adversarial sample the rules caught.
cleanA benign control the rules left alone.
known gapAn adversarial sample the shipped signature rules do not catch, recorded in the bundle.
MISSEDAn adversarial sample that should have been caught and was not. A regression.
FALSE POSITIVEA benign control the rules fired on.
GAP CLOSEDA recorded gap that now fires. Good news; the bundle needs updating.
ERRORThe sample could not be decoded or scanned. The error is printed below it.

The summary counts missed samples, false positives, closed gaps and errors separately. A skipped engine has no sample results; JSON reports the reason in its skipped field.

If configuration disables every rule in an engine, the command still measures its samples, reports zero loaded rules and the resulting misses, and exits with failure. Missing or malformed rule files remain load errors.

Known gaps are recorded, not hidden

A gap is a measurement, and the test suite fails when one closes as well as when one opens. A rule change that starts catching a recorded gap breaks the build until the bundle is updated, so no gap can quietly become permanent and no improvement goes unnoticed.

A gap is a gap in the signature engines. CSM’s other layers – PHP taint analysis, the behavioural checks, PHP Shield, and the correlation that turns findings into incidents – are not measured by this bundle. A sample listed as a known gap may still be caught in production by one of them.

Engines

CSM ships two rule sets, and the command reports each separately:

  • realtime, the YAML rules the real-time watchers use.
  • yara, the YARA-X rules used by scheduled and email scanning. Present only in builds compiled with YARA-X; a build without it reports SKIPPED with a reason. A compiled engine whose rules cannot load fails the command.

The regular CI test job runs the realtime gate. The required test:production job runs scripts/production-tests.sh portable, which includes all packages with the yara,journal,bpf tags and the pinned YARA-X library, so it runs the YARA bundle gate too. Tagged builds and lint alone do not execute that test.

To measure the bundle locally, install PHP CLI for the fixture validity check, then run:

go test -count=1 ./internal/selftest
go test -count=1 -tags yara ./internal/selftest

The second command requires the pinned YARA-X library. The untagged command does not measure YARA detection.

Incidents

CSM groups related findings into Incident objects so operators see one escalating story per account, mailbox, or process instead of a stream of unrelated findings. Original findings are not mutated or suppressed – the Incident is layered on top.

Classification

Each incident is assigned a kind at create time. Inbound attacks are keyed on the attacker source IP, not on the victim they name:

  • web_attack – a WAF block, scanner probe, or login brute-force. When the finding names a victim domain or account, that is the attack target, not evidence the account is compromised, so the incident keys on the attacker IP and one attacker’s hits across many vhosts collapse into one incident.
  • mailbox_bruteforce – failed mailbox logins, mail brute-force bursts, and pre-auth SMTP probes from one source. A failed login is an attack attempt, not a takeover, so it keys on the attacker IP rather than the targeted mailbox. A threat-intelligence match on an address seen failing IMAP or POP3 logins opens this kind too, including alerts queued during shutdown and delivered after restart.

Compromise kinds are reserved for evidence that an attack succeeded: web_account_compromise (on-disk or behavioural signals such as a webshell or suspicious PHP) and mailbox_takeover (post-authentication abuse such as outbound spam, cloud relay, or a compromised-account signal). Inbound-attack kinds carry short attacker-grade retention; compromise kinds get the longer review window.

Two further classes of finding never open an incident at all. Findings that record an action CSM took, or how well it can see, are excluded so CSM’s own output cannot re-enter its decision path. Findings that report a standing weakness in installed software – a known-vulnerable or outdated plugin – are excluded because nothing has happened yet and there is nothing to contain. Known-vulnerable plugin findings remain alertable. Routine outdated-plugin notices remain informational, and both classes appear in the findings list.

Lifecycle

StatusMeaning
openActive. New findings for the same correlation key keep merging in.
containedOperator marked under control. Findings still merge in window.
resolvedClosed. Future findings start a new incident.
dismissedFalse positive. Future findings start a new incident.

Resolved and dismissed incidents are pruned 30 days after their last update when an operator closed them, and 7 days after their last update when the daemon closed them (closed_by starting with auto:). Older records without close attribution keep the 30-day period. Confirming an automatic closure, changing a closed status, or recording a block on a closed incident gives that decision 30 days of retention. Recorded operator decisions on older rows with stale automatic attribution also keep this longer period. Reopening clears close attribution; the next closure determines retention again.

Open and contained incidents are never auto-pruned by the retention loop, but they may be auto-resolved by the per-kind idle threshold described under “Auto-close” below. Retention sweeps use bounded transactions so a large backlog does not hold the store writer for the entire cleanup. Pruning frees space for reuse; it does not shrink the state file. Finding history has its own retention, so a retained incident can refer to findings already evicted from history.

Auto-close

To stop the open-incident backlog from growing without bound on busy hosts, the daemon scans Open / Contained incidents shortly after startup and then once an hour, auto-resolving any whose updated_at exceeds the per-kind idle threshold. A live sweep closes at most 1000 stale incidents at a time; if more stale incidents remain, follow-up sweeps run every 30 seconds until the backlog drains. Dry-run sweeps still scan the full set so the counters show every would-close decision. Auto-resolved incidents carry closed_by: "auto:stale" and an incident_auto_closed action in their timeline so reporting can distinguish them from operator closes.

Defaults (configurable in csm.yaml):

incidents:
  auto_close:
    enabled: true            # set false to disable
    dry_run: false           # set true to log decisions without writing back
    by_kind:
      mailbox_takeover: 24h
      mailbox_bruteforce: 24h
      credential_spray: 24h
      web_attack: 24h
      web_account_compromise: 168h

Kinds absent from by_kind are never auto-closed. The default map omits host_integrity_risk, host_takeover, and post_exploit_process because those host-level incidents should stay open until an operator reviews them. host_takeover is the compound escalation raised when any two of three host-takeover legs (a new uid-0 account, a planted suid binary, an outbound connection to a bad ASN) are correlated for the same host inside the merge window.

If a fresh finding for the same correlation key arrives after the auto-close, the merge-window stale-binding logic creates a new open incident – nothing about auto-close blocks re-detection. History is preserved on the closed record.

Tuning on high-volume hosts. Each by_kind threshold is the idle time before a kind auto-resolves; they are independent and operator-set. A host under sustained brute-force keeps a large open set mostly from the longer-lived kinds (web_account_compromise defaults to 168h). If the open-incident count is higher than you want to triage, shorten the relevant by_kind entry (e.g. web_account_compromise: 72h) rather than disabling auto-close. Untouched auto-resolved records are retained 7 days, measured from when the incident resolves, so shortening the threshold also moves the eventual prune point earlier relative to the last finding. Auto-close still keeps a resolved record for follow-up instead of deleting history at close time.

Metrics: csm_incidents_auto_closed_total and csm_incidents_auto_close_dry_run_total.

Credential-spray suppression

Failed mailbox logins already collapse onto the attacker IP as a single mailbox_bruteforce incident (see “Classification”). Spray suppression extends that across protocols: it tracks the distinct-mailbox/account set per source IP for the configured per_check detectors (mail and PAM) across the merge window and, once an IP exceeds distinct_mailboxes, opens a single credential_spray super-incident keyed on the IP with breadth-based severity escalation. Subsequent findings from that IP attach to the spray incident’s timeline.

If per_check includes mail_subnet_spray or smtp_subnet_spray, grouping uses their single cidrs subnet. Older findings with the subnet in source_ip keep the same grouping, including when replayed or joining restored incidents.

Defaults (configurable in csm.yaml):

incidents:
  spray_suppression:
    enabled: false           # default OFF; opt-in
    dry_run: true            # default ON; counters move, routing unchanged
    distinct_mailboxes: 10   # threshold to trip
    severity_escalate_at: 50 # bump severity to CRITICAL at this many
    per_check:
      - email_auth_failure_realtime
      - pam_bruteforce
      - credential_stuffing
    max_tracked_ips: 10000
    block_at_severity: ""    # "" detection-only, "high" block on open,
                             # "critical" block on escalation

Setting block_at_severity hands the source IP to the firewall as soon as the spray detector trips at the chosen tier, once spray_suppression.dry_run is false. The detector also requires auto_response.enabled and auto_response.block_ips; the firewall still honors auto_response.dry_run, so a dry-run host logs the would-be block without applying nftables rules. Live accepted requests are recorded on the incident timeline as a credential_spray_block_requested action. Non-live outcomes (dry-run, verdict-allow, already blocked) and failed attempts do not latch the incident, so a later finding can retry after blocking is live again. Concurrent findings for the same incident share one in-flight firewall call, and resolved or dismissed spray incidents do not make new block decisions.

A spray incident blocks only an address that a finding with address evidence attests, the same gate every incident block uses. Raw mailbox failures such as email_auth_failure_realtime count mailboxes and open the incident, but cannot block the source alone; pam_bruteforce or credential_stuffing from the same address can. Visibility-only findings, including mail_bruteforce_suspected and the High mail_account_compromised advisory for an established multi-mailbox source, never make a spray incident blockable either.

Incident blocks name the finding that supplied address evidence. The attesting check and severity survive trimming and restart and are checked again at startup. Retained active proof in older incidents is migrated; missing or retired proof cannot authorize another block until a fresh qualifying finding arrives. Existing installed blocks keep their expiry, and incidents.auto_block and incidents.spray_suppression still control responses. At startup, older incidents learn the link from an attesting timeline event even when their evidence flag was already set. If an older timeline has no finding identity or already lost that event, the original link cannot be recovered; it is never made from message text. The next finding that attests the key’s address records its own identity when that link is missing, and the new link survives trimming and restart. Existing blocks are unchanged.

Automatic incident blocks use the address-evidence gate without a separate exclusion list. Advisory events alone never attest an address, but can raise an incident’s severity after an earlier attestation, even when that event left a trimmed timeline. An ip_reputation match never attests either: an incident it opens blocks only after a local attack finding from the same address joins it. Retained evidence in existing incidents follows this rule on the next finding after upgrade. The existing incident and spray blocking settings still govern the action.

Whitelisted IPs (entries in reputation.whitelist and the live bbolt whitelist updated via the Web UI) are skipped from spray detection so internal mail relays, NAT egresses, and known-good infrastructure never produce a spray incident. Changes to reputation.whitelist apply on config reload.

Choosing block_at_severity:

  • "" (default) – detection-only. Spray incidents open, no firewall hand-off. Use during dry-run validation and on hosts where blocking is owned by a separate system.
  • high – block at the distinct_mailboxes trip. Recommended once the dry-run counter looks clean. Trips on the first sustained burst before the source IP goes idle for longer than the merge window.
  • critical – block only after severity escalates, i.e. one IP hits severity_escalate_at distinct mailboxes before the source IP is idle for more than the merge window. A low-and-slow attacker that stays below that count before each idle reset never escalates and never blocks. Pick this only when you have strong shared-NAT exposure and accept that slow sprayers evade the gate.

Rollout:

  1. Ship the daemon with enabled: false, dry_run: true. The detector tracks per-IP mailbox sets and increments csm_credential_spray_dry_run_total whenever the threshold would have tripped, but routing stays on the legacy per-mailbox path.
  2. Validate the counter on your own infrastructure for 24h. If a trusted IP shows up in the dry-run trips, add it to reputation.whitelist.
  3. Flip enabled: true, dry_run: false. New attacker IPs route through the spray path; existing per-mailbox backlog drains via the auto-close path.
  4. After another 24h, set block_at_severity: high. The firewall hand-off runs on every spray decision (open + merge), so an incident opened before the flag was armed still blocks on the next finding from the same IP.

Metrics: csm_credential_spray_opened_total, csm_credential_spray_suppressed_mailbox_takeover_total, csm_credential_spray_dry_run_total, csm_credential_spray_tracked_ips.

Incident auto-block

spray_suppression only handles the credential_spray super-incident kind. Low-and-slow scanners that never trip a per-detector window (modsec escalation, mail brute-force, smtp probe) still produce web_attack, mailbox_bruteforce, mailbox_takeover, or web_account_compromise incidents but never get firewalled. The incidents.auto_block block adds a generic incident-driven firewall hand-off:

incidents:
  auto_block:
    enabled: false           # default OFF; opt-in
    block_at_severity: ""    # "" / "high" / "critical"
    kinds: []                # empty = any non-spray kind with one source IP

When the gate trips, the correlator hands the source IP to the firewall through the same dry-run / block_ips gate as the spray path. A live accepted request records incident_block_requested; non-live outcomes (dry-run, verdict-allow, already blocked) do not latch the incident, so an operator who arms auto_block AFTER an incident has already crossed the gate still gets a block on the next finding while the incident is open or contained. Incidents with multiple source IPs are left for manual review. If a long-running incident’s timeline was truncated and the source IP is not part of the incident key, auto-block also stays off because the remaining visible timeline may not contain every source IP. The incident blocks an address only when a finding that names it as an attacker joined the incident. An outbound connection’s remote end, a customer login or an advisory never qualifies, whatever raised the incident’s severity; see Auto-Response.

Qualifying address evidence is retained when findings join an incident and across restarts, including for incidents saved by earlier versions. Reopening an incident keeps its evidence; a new incident starts with its own evidence.

The same visibility-only findings are excluded from generic incident auto-blocking. In particular, setting block_at_severity: high does not turn an established multi-mailbox mail_account_compromised advisory into firewall evidence. A Critical compromise finding remains blockable.

credential_spray is explicitly excluded from this path; the dedicated spray hand-off owns it. Set kinds to narrow the surface (e.g. only web_account_compromise) if you do not want every CRITICAL mailbox_takeover incident to block its source IP.

This pairs naturally with the ModSecurity escalation thresholds (thresholds.modsec_escalation_hits, thresholds.modsec_escalation_window_min) – raising the window from the shipped default of 10 minutes to e.g. 4 hours lets the modsec detector promote paced scanners to a Critical escalation finding, which then trips the generic auto_block gate.

ModSecurity escalation is confidence-gated. Each deny is classified as high-confidence (a specific attack/probe rule – SQLi, RCE, traversal, URL-encoding abuse, CSM custom, or an OWASP CRS rule from an APPLICATION-ATTACK rule file, which is how LiteSpeed logs, since it omits the rule message and tags), low-confidence (policy/anomaly scoring rules such as COMODO content-type 210710 or anomaly-points 214930, and OWASP CRS anomaly-score rules), or unknown. Message and tag evidence comes from the rule configuration CSM reads, with macro references removed, so text a request puts into a logged message cannot change confidence. Without that configuration, known rule IDs and the rule file decide; other rules stay unknown. A burst escalates to a firewall ban at the normal hit count only when it contains a high-confidence deny or an unknown deny; a low-confidence-only burst emits one non-actioned modsec_low_confidence_burst finding for visibility instead of banning, then stays quiet until that active low-confidence window drains. This stops false bans of legitimate traffic (e.g. an unusual checkout that only trips content-type/anomaly rules) without blinding CSM to real attacks, which trip the high-confidence rules. A determined source that floods only low-confidence rules is still banned once it reaches the thresholds.modsec_low_confidence_escalation_hits backstop (default 30). Unknown blocking rules are escalation-eligible (fail-secure) and raise a modsec_classifier_gap finding once per rule for the whole host (repeated daily while the rule stays unclassified) so a new vendor rule pack is noticed rather than silently given a no-ban path. A deny whose request text imitates the rule fields ModSecurity writes counts as an unknown rule, so a request cannot borrow another rule’s confidence or escalation setting. A few CSM rules that match one specific exploit’s request shape (for example the PHP2Shell upload progress field or the wp2shell tool User-Agent) escalate on their first deny, whatever the hit threshold; turning firewall escalation off for that rule on the ModSecurity rules page opts out, including its retained evidence in later decisions. Upload progress requests carrying browser headers are denied by a separate rule whose escalation is off by default. Requests lacking those headers retain first-deny escalation. See ModSecurity Integration for the browser-header limitation and behavior when saved exclusions cannot be read. Existing bans keep their configured expiry. Warnings remain non-blocking even when request fields contain text resembling a deny message. While qualifying evidence remains in the window, duplicate escalations stay suppressed until auto_response.block_expiry allows fresh activity to renew the ban.

Kinds

  • web_account_compromise – findings attributable to a hosted account or script (PHP relay, webshell, account-scoped login bruteforce, etc.).
  • web_attack – an inbound web attack or remote-IP reputation/threat-score signal not tied to failed mailbox logins. When the finding includes a victim domain or account, that field is recorded as target context, not as the correlation key. Keyed on the source IP and given attacker-grade retention (default 24h) so defended probes and attacker-IP reputation hits do not inflate the account-compromise count or its longer review window.
  • mailbox_bruteforce – failed mailbox authentication, mail brute-force bursts, account-spray signals, pre-auth SMTP probes, and remote-IP reputation matches on addresses seen failing IMAP or POP3 logins. Keyed on the attacker source IP with attacker-grade retention (default 24h).
  • mailbox_takeover – post-authentication mailbox abuse such as suspicious geo, credential leaks, cloud relay abuse, spam outbreaks, and confirmed compromised-account signals.
  • post_exploit_process – process exec from /tmp, /var/tmp, /dev/shm.
  • host_integrity_risk – daemon/kernel-level signals (sensitive file changes, fake kernel threads, binary/config tampering). Periodic binary/config tamper findings join the local host incident even without account or IP attribution. Startup verification still alerts and refuses to start before normal incident processing is available.
  • host_takeover – any two of a new uid-0 account, a planted suid binary, and an outbound connection to a bad ASN, seen for the same host inside the merge window.
  • credential_spray – one source IP brute-forcing many distinct mailboxes/accounts inside the merge window. Keyed on the source IP rather than per-mailbox, so a scanner spraying thousands of usernames produces one super-incident instead of thousands of mailbox_bruteforce rows. Findings from the same IP after the trip attach to this incident’s timeline. See “Credential-spray suppression” below.

The host-integrity set, all five compound sets, kind selectors and identity exclusions are checked against the detector registry. The independent test fixture internal/incident/testdata/check-policy.json records an explicit classification role and selecting-set membership for every registered check, including checks with no named override. Tests reject unknown names, missing eligible members, new tables without a contract and new checks without a policy decision. They also exercise each check with account, mailbox, process and source-IP attribution to test classification precedence.

These tests live in the external incident_test package, which can import the check registry without adding a production dependency from incident to checks. That dependency would cycle through checks -> control -> incident. Changes to the fixture require a review of the detector’s emitted evidence; do not regenerate it from the selecting tables. Classification coverage does not calibrate correlation weights or prove that broader compound membership is safe.

Severity policy

Severity escalates only. Each incident keeps the highest severity any joined finding has carried. Findings themselves are never re-emitted at a higher severity. The audit trail records an incident_severity_changed action when an incident’s severity bumps.

Correlation window

15 minutes by default. Findings outside the window for the same key start a new incident. The window is a named constant in code; not yet exposed via config.

Open threshold

Non-Critical findings normally need at least two correlated sightings inside the merge window before an incident opens. The first sighting is held in a pending bucket and counted toward the threshold; the second promotes both into a new incident with a two-event timeline. A mail-filter exfil finding remains a first-hit incident even when an uncorroborated copy-forward is graded Warning, so the operator review path is not lost. Stale pending entries are pruned by the daily retention sweep.

Critical-severity findings (account compromise, cloud-relay abuse, modsec rule escalations) bypass the threshold and open immediately so escalations still page on first hit.

The threshold suppresses one-shot scanner noise (a single modsec deny from a wandering scanner, an isolated mistyped password) without hiding sustained activity. The current pending-bucket size is exposed as the csm_incidents_pending gauge.

The stored incident includes the full correlation key, including process PID/UID and remote IP when those are the only available dimensions, so active incidents keep merging after daemon restart.

Cross-account correlation of findings

Separately from incidents, the scan runner, the latest-state merge and the realtime dispatcher derive two findings from the findings they see:

  • coordinated_attack (Critical) when at least three distinct hosting accounts each carry at least one Critical finding from a check classified as a security event or malware artifact. The checks may differ between accounts. Repeated findings or several installs inside one account never raise the count.
  • cross_account_malware (Critical) when the same malware-artifact check (webshell, new_webshell_file, backdoor_binary, new_executable_in_config) is present on two or more accounts at any severity. Different malware checks on different accounts do not combine.

Over the persisted latest-state set, both aggregates only combine findings whose condition was first observed in the last hour. A scan re-emits every finding it still sees and the merge refreshes its report time, so judging by that would let a months-old condition re-enter the window on every cycle; the merge therefore carries each finding’s first observation across re-reports and correlation reads that instead, including when a completed scan replaces its owned findings. A condition removed by a completed scan starts a new observation if it is found again later. The window is relative to the merge time, so the next completed scan clears expired aggregates even if it produces no findings. Source rows are retained when their aggregates expire, a finding carrying no first observation falls back to its report time, and a finding carrying no timestamp at all still counts. Derived timestamps never affect the window. JSON output omits first_seen when no first observation was recorded.

The first observation is not retroactive. On upgrade, a stored finding adopts its existing report time, because nothing recorded when that condition actually started. A long-lived finding with a recent report can therefore count once after upgrading; it ages out one hour after that saved report time, and the next completed scan clears expired aggregates even if it reports the same conditions again.

The realtime and scan batch paths keep their existing grouping and count all qualifying rows in the batch. A scan can carry forward an older finding for a file it could not examine; its original timestamp does not exclude it from batch correlation or get refreshed by correlation.

Every registered check is classified as a security event, a malware artifact, ignored with a stated reason, or derived. Derived findings are never inputs. A nonempty TenantID wins verbatim, including case, over the file path and text. It must identify the host-local hosting account. Existing stored values and verdict callbacks use the same precedence; callbacks must supply the same owner keys as producers. External identifiers can split one owner or combine distinct owners, and correlation does not validate them.

If that field is empty, correlation uses the account home containing the cleaned absolute FilePath. Relative paths, root/home-only paths and paths that escape the home after cleaning do not identify that home. This is a lexical mapping through the platform’s account roots, including Plesk roots; it does not resolve symlinks or prove that directory aliases are distinct owners. Nonstandard content roots need producer-supplied identity.

The final fallback scans Message before Details for an account-root path. It recognizes <account-root>/<user>/, not (account: user) labels. Within each field it searches roots in configured order, can match an embedded root substring, and can select an incidental path. These compatibility limits are why account-aware producers should supply identity.

Qualifying rows with no identity from any source do not count toward either aggregate. They are counted once per call and logged once per check name per process, using only that call’s row count and no finding text. Ignored, derived, unknown and below-threshold security-event rows produce no diagnostic count. A check with a declared attribution gap is still eligible: when its producer does supply an authoritative owner, that Critical counts.

The health snapshot (csm status --json, /api/v1/status) carries a correlation_attribution block with two views: current is the per-check count of unattributed qualifying rows retained in the active set and inside the correlation window at its latest merge, including any eviction caused by the size limit. Unstamped legacy rows also count. It clears when a later merge attributes those rows or they age out of the window; cumulative sums every unattributed row since the daemon started, across active-set merges and per-batch derivations, so a producer that recovered stays visible as having failed. Counters are published together in merge order. The block is absent until the first merge. csm doctor reports the same state as correlation attribution: OK when current is empty, WARN naming the checks and their counts otherwise, with the history in both cases.

The two per-batch derivations (scan runner, realtime dispatcher) see only their batch and produce alerts. The latest-state merge derives from the merged, deduplicated, capped persisted active set under the same lock as the purge, so evidence from separate scans combines: three accounts compromised in three different scans still produce a persisted coordinated_attack, and it clears only when fewer than three accounts still carry a qualifying Critical there. Each completed runner snapshot replaces only its own checks’ rows; rows kept for an unscanned owner or a file coverage gap stay as inputs; the previous derived findings are dropped and recomputed from the merged set. Demotion, dismissal or re-verification of a contributing row takes effect at the next nonempty scan merge, and only if fewer qualifying accounts remain; there is no immediate recompute and no promise that every file is re-verified. Demoting a malware artifact below Critical cannot clear the all-severity malware aggregate by itself. There is no time window, no shared-signature or causal requirement, and no precision multiplier: the widened Critical inputs accumulate across scans and can include unrelated events and false positives, which is why the threshold calibration stays open on the roadmap.

Callers initialize platform detection before correlation; the latest-state caller does so before taking the store lock. Correlation reads cached roots, and attribution warnings are reported after the merge releases that lock.

Class membership does not mean an emitter currently reaches Critical: the non-WordPress administrator checks emit High with a stored baseline and Warning without one, and the outbound backdoor-port and bad-ASN variants emit High, so all of them are eligible but contribute nothing until a Critical variant exists.

Eligible producers supply verified identity when available. The test inventory names the producer test for each check, including unresolved branches:

  • Database and CMS scanners stamp the owner of the install’s configuration path, resolved through the account roots. An install outside every root keeps its display label but no owner.
  • Mail producers (rate windows, mail holds, credential leaks, bulk-service logins, forwarders, filters, mail brute force, cloud relay, geo logins, PHP relay volume) resolve a mailbox or domain to its owning account through the panel’s domain ownership table. Without that table (any panel other than cPanel) the owner stays empty and the row is reported, not counted. A bare account name must resolve to a passwd home directly under an account root. Credential and bulk-service findings use the authenticated identity, never the envelope sender. A sender-domain volume aggregate carries an owner only when every counted arrival proves the same local account through authentication or local submission. Arrivals with a remote ident username after the connecting address stay unattributed: that unquoted text can imitate later authentication metadata. Records whose greeting hides the connecting address also remain unattributed and supply no address. Mixed or unverified aggregates stay unattributed without reducing the volume count. Owner lookups run after tracker locks are released. Mail hold and governor findings require a local mail-server permission decision. Submission identities are read only from reception metadata before the message size; message IDs, subjects, addresses and login names cannot supply, replace or remove an owner or connecting address. Address literals in message IDs and recipients remain message data, and local arrivals do not acquire a peer from later metadata. An optional MAIL AUTH value follows the authenticated identity and does not change that identity or the peer. Subject credential alerts read the arrival’s subject, with Exim’s quote and backslash escapes decoded; escapes for non-printing characters stay as written. Literal tabs remain as logged. Sender addresses, greetings, message IDs and recipients cannot supply that subject. Records with remote ident or optional MAIL AUTH text are skipped for subject alerts because those unquoted values can imitate later fields. Subject text remains sender-controlled and is not proof of authentication. TCP Fast Open connections retain the same peer and ownership checks. The submission boundary follows Exim’s reception log fields and assumes Exim’s default greeting syntax check.
  • Process, login and crontab producers accept a system user as owner only when its home directory sits directly under an account root, so root, service users and unknown uids never become an account. Direct SMTP findings apply this validation to both socket users and enriched process accounts. SSH logins from unknown addresses use this validated account as their tenant only when the sshd record confirms successful authentication. Root and service-user logins group by source address after upgrade, and their findings carry no tenant_id and no attack database account. The attack database uses only this validated tenant for SSH attribution; certificate text in the log cannot supply an account. Existing incidents and attack records keep their attribution. No setting changes this behavior.
  • File families (content, phishing, htaccess, file index, core integrity, realtime file events, PHP shield events, self-deleting droppers) carry the judged file’s path, which resolves as described above. The collapsed core-integrity finding has no single path and carries the install owner.
  • Socket checks resolve the kernel UID through the same passwd and direct account-home validation. Bad-ASN events also retain that owner when realtime process enrichment misses. Root, service and unknown UIDs stay unattributed. An attributed socket or mail finding includes its owner in the message so different accounts retain separate dispatch and audit identities.
  • A sender-domain mail aggregate with mixed or unverified submitters remains a declared attribution gap. The owning account for a mailbox still requires the cPanel domain table. Any unattributed qualifying finding reaches the diagnostic count rather than contributing an invented account.

Correlation policy table

The table below is generated from the check registry: every registered check with its class, its ignore reason when it is excluded, and its declared attribution gap. A test compares it with the registry byte for byte and fails when a row is stale, missing or duplicated; regenerate it with go test ./internal/checks -run '^TestCorrelationDocumentation$' -args -update-correlation-docs rather than editing rows by hand. A class says how correlation treats a check when it fires; it does not say the check currently reaches Critical, that its owner is available on every panel, or how precise it is.

Regeneration preserves the surrounding prose and marker line endings, even when they use CRLF. The generated block itself uses LF line endings.

Ignore reasons:

  • account-aggregate: already summarizes several accounts without a single victim identity
  • attacker-side: attacker activity or attempted access, not evidence of compromise of the named victim
  • host-scope: host-wide condition with no account to attribute; a cross-account count cannot use it even when it is a real compromise
  • informational: audit trail or inventory event with no compromise claim
  • performance: resource usage
  • posture: static configuration, hardening or hygiene state; a Critical means a misconfiguration, not an attack on the account
  • response: record of an automatic action already taken; feeding it back would double count
  • self-health: CSM’s own health, capacity or coverage state

Attribution gaps:

  • envelope-sender: sender-domain volume aggregate is unattributed when contributing submissions are unverified or belong to different accounts
CheckClassIgnore reasonAttribution gap
account_scanignoredself-health
account_scan_errorignoredself-health
account_scan_truncatedignoredself-health
admin_cross_account_overlapignoredaccount-aggregate
admin_panel_bruteforceignoredattacker-side
af_alg_enforcement_correctedignoredself-health
af_alg_socket_usesecurity event
antispam_apply_failedignoredself-health
antispam_controller_openignoredself-health
antispam_feedignoredself-health
antispam_file_replacedignoredself-health
antispam_ip_mapignoredself-health
antispam_learningignoredself-health
antispam_preconditionignoredself-health
antispam_rspamd_downignoredself-health
antispam_rspamd_recoveredignoredself-health
antispam_scan_errorignoredself-health
api_auth_failureignoredattacker-side
api_auth_failure_realtimeignoredattacker-side
api_tokensignoredinformational
auto_blockignoredresponse
auto_responseignoredresponse
auto_response_pausedignoredresponse
auto_response_withheldignoredself-health
backdoor_binarymalware artifact
backdoor_portsecurity event
backdoor_port_outboundsecurity event
bad_asn_outboundsecurity event
bpf_ringbuf_errorignoredself-health
bpf_unavailableignoredself-health
bulk_password_changeignoredaccount-aggregate
c2_connectionsecurity event
cgi_backdoor_realtimesecurity event
cgi_suspicious_location_realtimesecurity event
challenge_routeignoredresponse
check_panicignoredself-health
check_timeoutignoredself-health
config_reload_errorignoredself-health
config_reload_restart_requiredignoredself-health
coordinated_attackderived
cpanel_file_uploadsecurity event
cpanel_file_upload_realtimesecurity event
cpanel_loginignoredinformational
cpanel_login_realtimeignoredinformational
cpanel_multi_ip_loginsecurity event
cpanel_password_purgeignoredinformational
cpanel_password_purge_realtimeignoredinformational
credential_log_realtimesecurity event
credential_reuseignoredposture
credential_stuffingignoredattacker-side
crond_changeignoredhost-scope
crontab_changeignoredinformational
cross_account_malwarederived
csm_healthignoredself-health
database_dumpignoredinformational
db_content_scan_incompleteignoredself-health
db_doorway_sitemap_routessecurity event
db_hidden_link_injectionsecurity event
db_hostname_keyed_optionsecurity event
db_magic_token_usersecurity event
db_malicious_eventsecurity event
db_malicious_functionsecurity event
db_malicious_proceduresecurity event
db_malicious_triggersecurity event
db_options_injectionsecurity event
db_options_new_external_scriptsecurity event
db_options_plugin_notice_injectionsecurity event
db_phantom_post_authorsecurity event
db_post_injectionsecurity event
db_post_volume_burstsecurity event
db_rogue_adminsecurity event
db_siteurl_foreign_hostsecurity event
db_siteurl_hijacksecurity event
db_siteurl_invalidignoredposture
db_spam_injectionsecurity event
db_spam_taxonomysecurity event
db_stored_cloak_logicsecurity event
db_stored_code_executionsecurity event
db_suspicious_admin_emailsecurity event
db_unexpected_eventignoredinformational
db_unexpected_functionignoredinformational
db_unexpected_procedureignoredinformational
db_unexpected_triggerignoredinformational
detached_web_processsecurity event
direct_smtp_egresssecurity event
dns_connectionignoredhost-scope
dns_zone_changeignoredinformational
docroot_binary_execsecurity event
dpkg_integrityignoredhost-scope
drupal_admin_injectionsecurity event
drupal_content_injectionsecurity event
drupal_settings_injectionsecurity event
email_auth_failure_realtimeignoredattacker-side
email_av_degradedignoredself-health
email_av_encrypted_archiveignoredself-health
email_av_hold_bypassignoredself-health
email_av_isolation_unavailableignoredself-health
email_av_late_verdictignoredself-health
email_av_parse_errorignoredself-health
email_av_parse_worker_failedignoredself-health
email_av_parse_worker_rejectedignoredself-health
email_av_parse_worker_sourceignoredattacker-side
email_av_quarantine_errorignoredself-health
email_av_queue_overflowignoredself-health
email_av_scan_errorignoredself-health
email_av_scanner_panicignoredself-health
email_av_source_backlogignoredself-health
email_av_timeoutignoredself-health
email_bulk_recipientssecurity event
email_cloud_relay_abusesecurity event
email_compromised_accountsecurity event
email_credential_leaksecurity event
email_defer_fail_governorignoredinformational
email_dkim_failureignoredposture
email_filter_blackholesecurity event
email_filter_exfilsecurity event
email_filter_forwardersecurity event
email_filter_pipesecurity event
email_forged_sendersecurity event
email_mail_filtersignoredself-health
email_mailbox_compromisesecurity event
email_mailbox_responseignoredresponse
email_malwareignoredattacker-side
email_outbound_reputationsecurity event
email_password_audit_incompleteignoredself-health
email_phishing_contentignoredattacker-side
email_php_form_relayignoredattacker-side
email_php_form_relay_observedignoredattacker-side
email_php_form_relay_scriptsecurity event
email_php_form_relay_table_fullignoredself-health
email_php_relay_abusesecurity event
email_php_relay_account_volume_cappedignoredself-health
email_php_relay_action_dry_runignoredresponse
email_php_relay_action_failedignoredresponse
email_php_relay_action_skippedignoredresponse
email_php_relay_cpanel_limit_unreadableignoredself-health
email_php_relay_disabledignoredself-health
email_php_relay_inotify_overflowignoredself-health
email_php_relay_inotify_overflow_recoveredignoredself-health
email_php_relay_msgindex_persist_failedignoredself-health
email_php_relay_no_eximignoredself-health
email_php_relay_overflow_scan_truncatedignoredself-health
email_php_relay_path2b_disabledignoredself-health
email_php_relay_policies_reloadignoredself-health
email_php_relay_rate_limit_hitignoredresponse
email_php_relay_sweep_failedignoredself-health
email_php_relay_watcher_failedignoredself-health
email_pipe_forwardersecurity event
email_rate_criticalsecurity event
email_rate_warningsecurity event
email_spam_outbreaksecurity event
email_spf_rejectionignoredposture
email_suspicious_forwardersecurity event
email_suspicious_geosecurity event
email_weak_passwordignoredposture
executable_in_config_realtimesecurity event
executable_in_tmp_realtimesecurity event
exfiltration_paste_sitesecurity event
exim_frozen_realtimeignoredhost-scope
fake_kernel_threadsecurity event
fanotify_kernel_overflowignoredself-health
fanotify_overflowignoredself-health
firewallignoredhost-scope
firewall_ipv6_unmanagedignoredhost-scope
firewall_portsignoredhost-scope
firewall_state_unreadableignoredself-health
ftp_auth_failure_realtimeignoredattacker-side
ftp_bruteforceignoredattacker-side
ftp_loginignoredinformational
ftp_login_after_bruteforcesecurity event
full_scan_file_too_largeignoredself-health
group_writable_phpignoredposture
htaccess_auto_prependsecurity event
htaccess_cgi_handler_abusesecurity event
htaccess_errordocument_hijacksecurity event
htaccess_filesmatch_shieldsecurity event
htaccess_handler_abusesecurity event
htaccess_header_injectionsecurity event
htaccess_injectionsecurity event
htaccess_injection_realtimesecurity event
htaccess_php_in_uploadssecurity event
htaccess_rewrite_off_shieldsecurity event
htaccess_security_disabledsecurity event
htaccess_spam_redirectsecurity event
htaccess_user_agent_cloaksecurity event
http_asn_crawlignoredattacker-side
http_claimed_bot_unverifiedignoredattacker-side
http_distributed_floodignoredattacker-side
http_request_floodignoredattacker-side
http_scanner_profileignoredattacker-side
http_ua_spoofignoredattacker-side
infra_ips_unresolvableignoredself-health
integrityignoredhost-scope
ip_reputationignoredattacker-side
joomla_admin_injectionsecurity event
joomla_content_injectionsecurity event
joomla_extensions_injectionsecurity event
js_keylogger_dataflowsecurity event
js_taint_scan_incompleteignoredself-health
kernel_moduleignoredhost-scope
local_threat_scoreignoredattacker-side
magento_admin_injectionsecurity event
magento_content_injectionsecurity event
magento_settings_injectionsecurity event
mail_account_compromisedsecurity event
mail_account_sprayignoredattacker-side
mail_auth_backend_degradedignoredself-health
mail_bruteforceignoredattacker-side
mail_bruteforce_suspectedignoredattacker-side
mail_log_source_unavailableignoredself-health
mail_per_accountsecurity eventenvelope-sender
mail_queueignoredhost-scope
mail_queue_unavailableignoredself-health
mail_spam_sourceignoredattacker-side
mail_subnet_sprayignoredattacker-side
malware_companion_filessecurity event
malware_companion_scan_incompleteignoredself-health
modsec_block_escalationignoredattacker-side
modsec_block_realtimeignoredattacker-side
modsec_classifier_gapignoredself-health
modsec_csm_block_escalationignoredattacker-side
modsec_disabled_vhostignoredposture
modsec_low_confidence_burstignoredattacker-side
modsec_warning_realtimeignoredattacker-side
mysql_superuserignoredhost-scope
new_executable_in_configmalware artifact
new_php_in_languagessecurity event
new_php_in_sensitive_dirsecurity event
new_php_in_sensitive_dir_cleanignoredinformational
new_php_in_upgradesecurity event
new_php_in_uploadssecurity event
new_php_in_uploads_cleanignoredinformational
new_suspicious_phpsecurity event
new_webshell_filemalware artifact
nulled_pluginignoredposture
obfuscated_phpsecurity event
obfuscated_php_realtimesecurity event
open_basedirignoredposture
opencart_admin_injectionsecurity event
opencart_content_injectionsecurity event
opencart_settings_injectionsecurity event
outdated_pluginsignoredposture
pam_auth_failuresignoredinformational
pam_bruteforceignoredattacker-side
pam_loginignoredinformational
password_hijack_confirmedsecurity event
perf_error_logsignoredperformance
perf_loadignoredperformance
perf_memoryignoredperformance
perf_mysql_configignoredperformance
perf_php_handlerignoredperformance
perf_php_processesignoredperformance
perf_redis_configignoredperformance
perf_wp_configignoredperformance
perf_wp_cronignoredperformance
perf_wp_loopbackignoredperformance
perf_wp_myisamignoredperformance
perf_wp_transientsignoredperformance
phishing_credential_logsecurity event
phishing_directorysecurity event
phishing_iframesecurity event
phishing_kit_archivesecurity event
phishing_kit_archive_unscannablesecurity event
phishing_kit_realtimesecurity event
phishing_pagesecurity event
phishing_phpsecurity event
phishing_realtimesecurity event
phishing_redirectorsecurity event
php_config_changeignoredposture
php_config_realtimeignoredposture
php_config_scan_incompleteignoredself-health
php_dropper_realtimesecurity event
php_in_image_realtimesecurity event
php_in_sensitive_dir_realtimesecurity event
php_in_uploads_realtimesecurity event
php_remote_taintsecurity event
php_shield_blocksecurity event
php_shield_evalsecurity event
php_shield_webshellsecurity event
php_suspicious_executionsecurity event
php_taint_scan_incompleteignoredself-health
placed_dir_incomplete_realtimeignoredself-health
placement_unresolved_realtimeignoredself-health
prestashop_admin_injectionsecurity event
prestashop_content_injectionsecurity event
prestashop_settings_injectionsecurity event
protection_queue_degradedignoredself-health
protection_queue_recoveredignoredself-health
realtime_scanner_panicignoredself-health
removed_file_restored_realtimesecurity event
reputation_quota_exhaustedignoredself-health
response_capacity_exhaustedignoredself-health
root_password_changeignoredhost-scope
rpm_integrityignoredhost-scope
self_deleting_dropper_overflowignoredself-health
self_deleting_dropper_realtimesecurity event
sensitive_file_modifiedignoredhost-scope
shadow_changeignoredhost-scope
shared_senders_staleignoredself-health
signature_match_realtimesecurity event
signature_update_rescan_queuedignoredself-health
signature_update_rollbackignoredself-health
smtp_account_sprayignoredattacker-side
smtp_bruteforceignoredattacker-side
smtp_probe_abuseignoredattacker-side
smtp_subnet_sprayignoredattacker-side
spam_evidence_evictedignoredself-health
ssh_keysignoredhost-scope
ssh_login_unknown_ipignoredinformational
sshd_config_changeignoredhost-scope
ssl_cert_issuedignoredinformational
stalwart_address_listignoredself-health
stalwart_banignoredattacker-side
stalwart_log_degradedignoredself-health
stalwart_spam_filter_offignoredself-health
stalwart_webmail_bruteforceignoredattacker-side
suid_binarysecurity event
supply_chain_vulnignoredposture
suspicious_crontabsecurity event
suspicious_fileignoredhost-scope
suspicious_php_contentsecurity event
suspicious_processsecurity event
symlink_attacksecurity event
test_alertignoredinformational
threat_feed_staleignoredself-health
trusted_proxy_unusableignoredself-health
uid0_accountignoredhost-scope
user_outbound_connectionignoredinformational
vulnerable_pluginsignoredposture
vulnerable_timthumbignoredposture
waf_attack_blockedignoredattacker-side
waf_bypassignoredposture
waf_detection_onlyignoredposture
waf_rulesignoredposture
waf_rules_staleignoredposture
waf_statusignoredposture
web_exposed_backup_archiveignoredposture
web_exposed_config_leakignoredposture
web_exposed_db_dumpignoredposture
web_exposed_phpinfoignoredposture
web_exposed_repo_metadataignoredposture
web_exposed_sample_sqlignoredposture
web_exposed_source_backupignoredposture
webmail_bruteforceignoredattacker-side
webmail_login_realtimeignoredinformational
webshellmalware artifact
webshell_callerignoredattacker-side
webshell_content_realtimesecurity event
webshell_path_publicignoredinformational
webshell_realtimesecurity event
whm_account_actionignoredinformational
whm_login_realtimeignoredinformational
whm_password_changeignoredinformational
whm_password_change_noninfrasecurity event
whm_unauth_scripts_realtimeignoredattacker-side
world_writable_phpignoredposture
wp_core_integritysecurity event
wp_core_unverifiedignoredself-health
wp_login_bruteforceignoredattacker-side
wp_plugin_inventory_unverifiedignoredself-health
wp_user_enumerationignoredattacker-side
xmlrpc_abuseignoredattacker-side
yara_backend_unavailableignoredself-health
yara_forge_rollbackignoredself-health
yara_match_realtimesecurity event
yara_match_scheduledsecurity event
yara_realtime_scan_errorignoredself-health
yara_scan_incompleteignoredself-health
yara_worker_crashedignoredself-health

Findings from retired checks

A check name that no version of CSM emits any more stays registered while older installations can still hold findings under it, because a finding is only ever cleared when its name appears in the owning runner’s purge list. Two file-index names, new_php_in_languages and new_php_in_upgrade, are in that state: findings written by releases before the content-first file index are cleared by the next completed file_index scan, are kept while that scan is incomplete, and are kept per file while the scan reports a coverage gap for that file. Nothing emits them again. php_dropper was never emitted by any release, is not registered, and no response table lists it: the manual, automatic and full-scan quarantine sets and the attack database mapping are each declared once and tested against the registry, so a renamed or never-emitted name cannot sit inert in a response table. The same guard removed modsec_block and waf_block from the attack database mapping; neither was ever emitted, so WAF blocks have never fed local reputation scoring through that database. Whether the emitted ModSecurity block names should is an open scoring decision, not something the guard decides.

Directory enumeration and PHP handler-configuration read errors make the file-index scan incomplete, including when only one account root is unreadable. The scanner keeps its previous index and directory cache, preserves its active findings, and still reports new findings from readable directories. A later completed scan clears the retired names; absent optional directories do not prevent completion. The first scan after startup and every retry after an incomplete or interrupted walk enumerate directories again, even if their cached modification times still match.

API

  • GET /api/v1/incidents – list, newest first. Without query parameters the response is a bare JSON array (compat with the existing wire shape phpanel/SIEM consumers decode against). When ?limit=, ?offset=, or ?status= is present the response switches to an envelope: {"items":[...], "total":N, "offset":N, "limit":N, "status":"..."}. Status accepts the four spec values plus active (open + contained, the default web UI filter). Limit is capped server-side at a safe maximum.
  • GET /api/v1/incidents/<id> – one incident.
  • POST /api/v1/incidents/<id>/status – transition status.

See api.md for endpoint detail.

Web UI

Open Monitor -> Incidents. The page has three tabs:

  • Correlated – the default flat list of incidents with status filter, page size, and detail panel. The detail panel shows the current firewall block state for the incident’s source IP (permanent, temporary, cphulk, or not blocked) when an IP is known.
  • Grouped – rolls up incidents by (kind, source) so a credential spray that produced thousands of mailbox_bruteforce incidents shows as one row per attacker IP. Pageable with the same page-size selector as Correlated. Click a group to see member incidents in the detail panel, which also surfaces the source IP’s firewall block state; clicking a sample id jumps back to the Correlated tab focused on that incident.
  • Timeline Search – the older IP/account history search across the audit log.

Admin tokens can transition incident status (open / contained / resolved / dismissed); read-scope tokens can browse all three tabs.

Control socket

csm incidents list [--status all|active|open|contained|resolved|dismissed] [--limit N] [--offset N] [--all]
csm incidents show <id>
csm incidents status <id> <open|contained|resolved|dismissed> [details]
csm incidents bulk-status --older-than 24h [--last-seen-before RFC3339] [--status active|open|contained] [--kind K] [--domain D] [--account A] [--mailbox M] [--limit N] [--to resolved|dismissed] [--apply --confirm]

csm incidents list returns the first 100 incidents by default. Use --offset for the next page, --status active for open + contained incidents, or --all for an explicit full dump.

csm incidents bulk-status defaults to dry-run. It prints the total match count and a bounded preview of the incidents that would change. At least one age guard is required: --older-than, --last-seen-before, or both. To mutate incidents, pass both --apply and --confirm.

Metrics

  • csm_incidents_open – gauge of currently open + contained incidents.
  • csm_incidents_created_total
  • csm_incidents_severity_changed_total
  • csm_incidents_status_changed_total
  • csm_incidents_findings_merged_total
  • csm_incidents_compacted_total
  • csm_incidents_pending – gauge of findings held in the threshold gate, awaiting a second correlated sighting.

Incident Response Runbook

Use this flow when CSM flags account compromise, mailbox takeover, malicious database triggers, or outbound spam on a production cPanel host.

Safety rules

  • Do not delete customer files during first response.
  • Do not thaw, release, or purge queued mail until affected credentials are rotated or an operator approves the specific queue action.
  • Do not close incidents until the account was reviewed, credentials were rotated or explicitly deferred, and a fresh scan is clean.
  • Take a CSM backup before upgrading CSM or changing incident state.

1. Verify the deployed binary

Deploy only after the required GitLab pipeline passed and the package was published.

/root/deploy-csm.sh check
/root/deploy-csm.sh upgrade
/opt/csm/csm version
/opt/csm/csm doctor --json

2. Take a backup

mkdir -p /root/csm-backups
systemctl stop csm.service
/opt/csm/csm backup /root/csm-backups/csm-pre-response-$(date +%Y%m%d%H%M%S).tar.gz
systemctl start csm.service
/opt/csm/csm doctor --json
sha256sum /root/csm-backups/csm-pre-response-*.tar.gz

Backup refuses a live or starting daemon and verifies the state lock before reading data. Restart the service immediately after the archive is created. If backup fails, start the service before investigating the failure.

Confirm the archive is readable:

gzip -t /root/csm-backups/csm-pre-response-*.tar.gz
tar -tzf /root/csm-backups/csm-pre-response-*.tar.gz | sed -n '1,80p'

3. Preserve evidence

mkdir -p /root/csm-forensics
/opt/csm/csm forensic-snapshot <account> --out /root/csm-forensics/<account>-$(date +%Y%m%d%H%M%S).tar.gz
sha256sum -c /root/csm-forensics/<account>-*.sha256
tar -xOzf /root/csm-forensics/<account>-*.tar.gz manifest.txt

Check the manifest for private-path exclusions, schema count, capture errors, and recent_mtimes_status=ok.

4. Map affected accounts

Map incident domains and queued local senders to cPanel users before rotating credentials or changing mail queue state.

/opt/csm/csm incidents list --status open --all
exim -bpc
exim -bp | exiqsumm
grep -E '^example.com:' /etc/trueuserdomains /etc/userdomains
whmapi1 listaccts searchtype=user search=<account> --output=json

Use native cPanel APIs for inventory:

uapi --user=<account> Email list_pops --output=json
uapi --user=<account> Ftp list_ftp --output=json
uapi --user=<account> Mysql list_users --output=json

5. Rotate credentials

Rotate the cPanel account password, FTP accounts, affected mailboxes, WordPress administrator users, database users, and application secrets for the affected account. Prefer WHM and UAPI calls or the control panel over direct file edits.

Do this before releasing mail or marking incidents resolved unless the operator explicitly defers rotation for a documented reason.

6. Review queued mail

Start with read-only summaries:

exim -bpc
exim -bp | exiqsumm
exim -bp

Review headers before any queue action:

exim -Mvh <message-id>

Group messages into:

  • safe to remove: frozen bounces, obvious backscatter, duplicate failed delivery notices with no customer value
  • do not touch: current customer conversations, invoices, form leads, or any message where the business value is unclear
  • needs review: suspicious local sender messages, mixed external bulk mail, or messages tied to an account whose credentials are not rotated

Only remove or thaw message IDs that were reviewed:

exim -Mrm <message-id>
exim -Mt <message-id>

7. Review stale incidents

Preview first:

/opt/csm/csm incidents bulk-status --older-than 72h --status active --kind web_account_compromise --limit 20
/opt/csm/csm incidents bulk-status --older-than 24h --status active --kind mailbox_takeover --limit 20

Apply in bounded batches only after review:

/opt/csm/csm incidents bulk-status --older-than 72h --status active --kind web_account_compromise --limit 100 --apply --confirm --details "operator cleanup after review"

For mailbox incidents, confirm mailbox rotation or explicit operator deferral before applying status changes.

8. Confirm recovery

/opt/csm/csm status --json
/opt/csm/csm doctor --json
exim -bpc

Keep the forensic archives, CSM backup, command notes, and queue decisions with the incident record.

Direct SMTP egress

CSM watches the local mail stack via spool + log scanning. Non-MTA processes that open outbound SMTP connections directly bypass that path. The direct SMTP egress detector catches that at connect time and feeds the incident correlator.

What fires

A finding with check: "direct_smtp_egress" is emitted when:

  • A non-root process opens an outbound TCP connection.
  • Destination port is one of the configured SMTP ports (default 25, 465, 587).
  • Destination IP is not loopback, infra, or in the operator’s infra_ips list.
  • The process user is NOT a known MTA user (mail, mailnull, postfix, dovecot, dovenull, mailman, plus exim on cPanel). On non-cPanel hosts, Stalwart’s stalwart and stalwart-mail users are also recognized when their UID is below 1000. Hosted UIDs do not receive this automatic exemption, even when they use a Stalwart username.
  • The process user is NOT listed under firewall.smtp_allow_users. Those users may send through firewall.smtp_block, so their connections are expected. List any other mail server’s service user there.

Process names are never a standalone allow condition. A hosted account renaming malware to smtp or smtpd still emits a finding.

The detector always emits findings when enabled. The dry_run knob does not suppress findings; it participates in the BPF enforcement gate, where any dry_run=true layer keeps kernel denial in observe-only mode.

Configuration

detection:
  direct_smtp_egress:
    enabled: true
    backend: auto       # auto / bpf / legacy / none
    dry_run: true       # safety default for detector-scoped action
    ports:             # each value must be 1-65535
      - 25
      - 465
      - 587

Backends

  • auto – allow both BPF and legacy scan paths. Live backend choice still follows detection.connection_tracker_backend.
  • bpf – emit only from the cgroup/connect4,6 consumer.
  • legacy – emit only from the /proc/net/tcp[6] polling path (live poller or scheduled critical scan). This path lacks PID/comm; MTA matching uses the username and UID.
  • none – detector disabled even when enabled: true is set elsewhere; useful for staged rollout.

The generic outbound connection tracker is still governed by detection.connection_tracker_backend; this setting only gates direct_smtp_egress findings.

Metric

csm_direct_smtp_egress_findings_total – monotonic counter, incremented per finding emitted by the BPF connection consumer. The legacy poller does not bump this counter today; operators who run backend=legacy should track findings via the audit log.

rDNS enrichment

When the BPF backend is active, finding details include a Domain field populated from a TTL-cached reverse lookup (30 min TTL, 1 second per-lookup deadline). The lookup runs only after the cheap direct-SMTP filters match. On resolver miss or timeout the field is omitted; the finding still fires.

Caveats

  • 2525 is intentionally NOT in the default port list. Many operators run unrelated services on it. Add it to ports if your infra uses it for submission.
  • The detector emits regardless of the dry_run knob. Kernel denial requires auto_response.dry_run, this dry_run key, and bpf_enforcement.dry_run to all be explicitly false.

BPF cgroup-deny enforcement

Optional in-kernel denial of outbound connections that match an enabled userspace detector. Direct SMTP egress is currently the only enforcement gate. Every layer starts in dry-run so operators can review telemetry before allowing live denial.

What it does

When bpf_enforcement.enabled=true, direct_smtp_egress=true, the connection tracker is running on BPF, and all dry-run layers are false:

  • The cgroup/connect4 + cgroup/connect6 BPF program inspects each outbound TCP connect.
  • If destination port is in the protected set AND the source UID is not in the safe-UID map AND the gated detector matches, the program returns 0 (kernel denies the connect).
  • Userspace observes the decision via the decision field on the ringbuf event and emits an audit-log entry.

When any dry-run layer is true (the default), the program emits the decision but always returns 1 (allow). Operators can run dry-run for as long as they need to gather telemetry before flipping to live denial.

The safe-UID map includes system UIDs below 1000, traditional MTA accounts, and accounts listed in firewall.smtp_allow_users. Stalwart usernames are exempt only as system UIDs; a hosted account cannot gain an exemption by choosing a Stalwart name. CSM rereads /etc/passwd every 5 minutes and drops the exemption of hosted accounts that were deleted or renamed, so a new account reusing such a UID is exempt for at most one refresh.

What it does NOT do

  • It does NOT wait on remote verdict callbacks in-kernel. That would add HTTP latency to every connect. The verdict callback (if enabled) runs in userspace after the BPF decision and enriches the emitted finding; it cannot undo a kernel denial.
  • It does NOT enforce on UDP, ICMP, or non-cgroup paths.
  • It does NOT replace detection. Findings still emit regardless; enforcement is a separate, layered control.

Configuration

bpf_enforcement:
  enabled: false              # master switch; default off
  dry_run: true               # safety default; flip after telemetry review
  direct_smtp_egress: false   # gate enforcement on the direct SMTP detector
  verdict_callback: false     # userspace post-decision callback

bpf_enforcement.enabled=true requires at least one feature gate. Today the only gate is direct_smtp_egress, which itself requires detection.direct_smtp_egress.enabled=true. The connection tracker backend must be auto or bpf, and the direct SMTP backend must be auto or bpf.

Kernel requirements

  • Linux >= 4.10 with CONFIG_CGROUP_BPF=y.
  • cgroup/connect4 and cgroup/connect6 BPF program types.
  • The capability surface bpf_enforcement.available.v1 is the wire signal that the binary supports the feature; combined with bpf_enforcement_active on the health snapshot, operators can detect both feature presence and runtime state.

On older kernels or default builds without the BPF tag, detection.connection_tracker_backend: auto falls back to the legacy /proc/net/tcp[6] poller. In that state direct SMTP findings still work when detection.direct_smtp_egress.backend is auto or legacy, but BPF enforcement is inactive.

When CSM attempts BPF and cannot start it, it emits a bpf_unavailable finding. The message reports whether the daemon is running on a fallback backend or has no live fallback active.

Metrics

  • csm_bpf_enforcement_decisions_total{decision="allow|dry_run|deny"}
  • csm_bpf_enforcement_uid_map_refresh_total – successful periodic refreshes of the safe-UID BPF map.
  • csm_bpf_enforcement_uid_map_refresh_failures_total – failed refreshes (e.g. /etc/passwd unreadable).

Dry-run precedence

Three independent dry_run knobs interact:

  1. auto_response.dry_run: keeps automatic network response in observe-only mode.
  2. detection.direct_smtp_egress.dry_run: detector-scoped action knob.
  3. bpf_enforcement.dry_run: kernel-side denial knob.

Rule: any dry_run=true wins. Live denial requires all three to be false at the layer they apply, plus a BPF runtime backend. Defaults are dry_run=true everywhere on first install.

Rollout recipe

  1. Enable the direct SMTP detector without BPF enforcement. Watch csm_direct_smtp_egress_findings_total for a week.
  2. Enable BPF enforcement with dry_run: true. Watch csm_bpf_enforcement_decisions_total{decision="dry_run"} and confirm dry-run denials track expected hosted-account egress.
  3. Set BPF dry_run: false on a single canary host. Audit incidents for false positives.
  4. Roll out to fleet.

CVE Mitigations

CSM treats CVEs as a three-layer problem:

  1. Operator-driven hardening via csm harden ... - applies the right preventive control for the host (modprobe blacklist, seccomp drop-ins, sysctl tweaks).
  2. Continuous enforcement by the daemon - re-asserts the control on every startup and as a periodic check, so a package upgrade or manual modprobe does not silently undo the mitigation.
  3. Live detection - auditd/BPF listeners flag exploit signatures the moment they fire, even on hosts where the kernel itself cannot be patched.

The hardening audit detects what the host actually has (kernel build, KernelCare livepatches, seccomp coverage) and refuses to claim protection it cannot deliver. Run csm harden with no arguments for the full list of available mitigations on the current host.

Active mitigations

CVE-2026-31431 - “Copy Fail” (Linux kernel AF_ALG)

Two operator paths depending on whether AF_ALG is loadable on the kernel:

  • csm harden --copy-fail - writes /etc/modprobe.d/csm-copy-fail-mitigation.conf blacklisting algif_aead and af_alg, then unloads them. Refuses on kernels where AF_ALG is built in (typical cPanel / CloudLinux 8), since the blacklist would have no effect there.
  • csm harden --copy-fail-seccomp - writes systemd RestrictAddressFamilies=~AF_ALG drop-ins for the units that spawn untrusted code: LiteSpeed, Apache/Nginx, every PHP-FPM pool, cron, mail. The right path on built-in-AF_ALG kernels.

The audit recognises KernelCare CVE-2026-31431 livepatches via kcarectl --patch-info and reports pass only when the host is genuinely mitigated (module blacklisted, seccomp drop-ins applied, or KernelCare livepatch active).

Live detection and BPF blocking

BPF-tagged builds (make BPF=1 or go build -tags bpf) load an LSM socket_create program on kernels with BPF LSM and ringbuf support. That program refuses socket(AF_ALG, ...) from non-root UIDs before the AF_ALG socket is allocated, returns EPERM, emits a ringbuf event, and feeds the existing Critical af_alg_socket_use finding path. UID 0 keeps AF_ALG access for system crypto users. There are no UID-range or alert-only BPF tunables today; use detection.af_alg_backend to select auditd or none if a host needs to avoid kernel-side refusal.

Default builds, BPF-tagged builds on unsupported kernels, and hosts forced to detection.af_alg_backend: auditd keep the audit-log listener. The audit path catches non-system AF_ALG socket attempts at Critical within about 500 ms but cannot stop the syscall before it reaches the kernel. Hosts that are not exploitable skip the live listener.

If CSM attempts the AF_ALG BPF path and cannot start it, it emits a bpf_unavailable finding. The finding says whether the audit fallback is active or no live fallback is available.

Auto-response

  • auto_response.copy_fail_kill_process: true - SIGKILL the offending process when the live listener fires. Default off (alert-only).
  • auto_response.disable_enforce_af_alg: true - suspend the periodic re-assertion of the module blacklist without removing the hardening marker. For triage only.

The daemon self-heals its auditd rule file on startup if it has drifted from the embedded copy, closing the upgrade gap where a new binary shipped without re-running postinstall would leave detection silently inactive.

Configuration knob

  • detection.af_alg_backend - auto (default) | bpf | auditd | none. auditd is the kill switch for a misbehaving BPF-tagged release. bpf is strict mode (no fallback). none disables the live listener entirely.

The csm_af_alg_backend{kind="bpf-lsm"|"auditd-tail"|"none"} Prometheus gauge surfaces which backend the coordinator selected at startup, so dashboards can see the active path without parsing logs.

BPF validation

On a BPF LSM host with a BPF-tagged CSM build:

  1. Set detection.af_alg_backend: bpf for strict validation, or leave it as auto and confirm BPF was selected.

  2. Start the daemon and check metrics:

    curl -k -H "Authorization: Bearer $CSM_TOKEN" https://127.0.0.1:9443/metrics \
      | grep -E 'csm_af_alg_backend|csm_bpf_backend'
    

    Expected selected series:

    csm_af_alg_backend{kind="bpf-lsm"} 1
    csm_bpf_backend{feature="af_alg",kind="bpf"} 1
    
  3. As a non-root user, attempt an AF_ALG socket:

    sudo -u nobody python3 - <<'PY'
    import errno
    import socket
    import sys
    
    try:
        socket.socket(socket.AF_ALG, socket.SOCK_SEQPACKET, 0)
    except OSError as exc:
        print(exc.errno)
        sys.exit(0 if exc.errno == errno.EPERM else 1)
    
    raise SystemExit("AF_ALG socket unexpectedly succeeded")
    PY
    
  4. Confirm the command prints 1 (EPERM) and CSM emits a Critical af_alg_socket_use finding. The finding details should say the call was refused by the BPF LSM program.

CVE-2026-41940 - cPanel/WHM auth-bypass

Detection in the access-log path:

  • Non-infra WHM login attempts surface at Warning (suppressible alongside other cPanel-login alerts) so an operator can see brute force traffic against the bypass surface.
  • The tokenless WHM-script request the published exploit uses for cache promotion surfaces at Critical, always-on, and feeds auto-block.

No operator hardening command is required. The host fix is to apply the cPanel patched build. CSM provides the detection layer for windows where patching has not yet rolled out.

CVE-2026-109618 and CVE-2026-109619 - “PHP2Shell” (PHP core)

Two critical flaws in PHP itself, reachable over the network without a login (advisories GHSA-w8hw-w9fx-j79w and GHSA-6wmf-4rv5-p3rp). They are fixed in PHP 8.2.35, 8.3.36, 8.4.27 and 8.5.12, and updating is the only complete fix.

  • CVE-2026-109618 - when memory runs out at the wrong moment, PHP leaves request memory in an invalid state that request shutdown then uses. Crafted multipart bodies can force that during request startup, before any script runs. Apache mod_php is exposed without any application involvement; PHP-FPM and LiteSpeed LSAPI need an application-specific path.
  • CVE-2026-109619 - a fatal error or memory exhaustion can leave worker state pointing at freed request memory, and some operations in the next request use it before normal initialization. Session upload progress is one of them. It runs on any multipart POST carrying the upload-progress field, whether or not the site uses sessions.

CSM provides a request filter and configuration audits until patched PHP is installed:

  • ModSecurity filter - rule 900133 denies requests other than GET and HEAD whose form field names contain the default PHP_SESSION_UPLOAD_PROGRESS field name, including the single-quoted spelling PHP accepts. Requests carrying browser Origin or Sec-Fetch-Site headers are denied by 900134 instead, whose firewall escalation is off by default. Keep that exclusion to prevent browser-steered bans. Requests carrying neither header retain first-deny escalation, including a browser whose extension or proxy strips Origin when posting to a plain-HTTP site. A custom session.upload_progress.name is outside this filter’s coverage. It needs SecRequestBodyAccess On, and it sees only what ModSecurity’s request body limits let it inspect, so the PHP setting below is the stronger stopgap.
  • Hardening audit - php_upload_progress_<version> warns while session.upload_progress.enabled is on or cannot be confirmed off. For cPanel EA4 PHP the audit reads php.ini and then php.d in PHP’s own order, so a drop-in that turns the setting off is recognised. Conditional site settings and FPM pools are checked separately; disabling one does not hide another that is on. Unreadable overrides and unresolved values retain the warning. On bare servers, the audit follows the additional files reported by the system PHP CLI in their load order. A pass covers only inspected settings, not running workers, per-user CageFS copies, .user.ini files, or a separate web SAPI configuration. Confirm the effective value through each site’s web handler after restarting. The php_version_<version> item marks a branch end of life the day after its upstream security support ends.
  • PHP 8.5 backtraces - php_fatal_error_backtraces_<version> warns on PHP 8.5 and later while fatal_error_backtraces is on or cannot be confirmed off, using the same scope and pool rules. Scripts can still turn it on at runtime, so a pass covers the configured default only.

Both configuration audits recognise quoted FPM setting names. Within each pool, admin values override ordinary values; the first duplicate at the same level wins, matching PHP-FPM’s application order. Remove conflicting pool entries instead of relying on a later value to override an earlier one. The backtrace audit does not determine the installed patch level and still checks PHP 8.5 and later after an update.

To turn the handler off on cPanel EA4, add session.upload_progress.enabled = Off to a file in each /opt/cpanel/ea-phpXX/root/etc/php.d/ directory. For CloudLinux alt-php, add the same line to /etc/cl.selector/global_php.ini and run selectorctl --apply-global-php-ini. With CageFS, PHP inside each cage reads its own copy of the EA4 configuration directory, so run cagefsctl --force-update after adding the file. Restart PHP workers afterwards and confirm the value from inside a cage (cagefs_enter_user <user> <php binary> -r '...') or from a web request, not only from the host CLI.

On PHP 8.5, also set fatal_error_backtraces = Off the same way; the advisory names it as the mitigation for its strongest confirmed case. The advisory’s setting for CVE-2026-109618, memory_limit = -1 behind a separate memory cap, is partial. Do not cap PHP with an address-space limit such as RLIMIT_AS or a CloudLinux VMEM limit: failed allocations reach the vulnerable path.

None of these replaces the PHP update. PHP published source backports for 7.4, 8.0 and 8.1 but no releases, and nothing for older branches. Vendors such as CloudLinux may ship fixes in their builds, so check the package changelog for the CVE IDs.

NGINX2Root - nginx worker to root through kernel SCTP

NGINX2Root is five nginx flaws (CVE-2026-123007, CVE-2026-123633, CVE-2026-123634, CVE-2026-121810, CVE-2026-113425) that run code as the nginx worker. Upstream fixed them in 1.30.7 and 1.31.9; see the nginx security advisories. Update and restart nginx, since a reload keeps the old binary. CSM does not check nginx versions: distributions backport the fix without changing them.

The published chain reaches root through a separate Linux kernel SCTP bug. Any local foothold can use it, a PHP webshell included, once the sctp module loads, and an unprivileged user can get it loaded on demand. Web hosts do not use SCTP, so the hardening check os_sctp_blocked asks kmod for the module’s effective load plan and passes only when:

  • the kernel has no SCTP support, or
  • the module is unloaded, modprobe reports it missing, and CSM confirms its absence in the running kernel’s module tree (EL8 ships it in kernel-modules-extra), or
  • the module is not loaded and kmod would run a no-op install command, or
  • the module is not loaded and the kernel has module loading disabled.

blacklist sctp is not enough: another module can still load SCTP as a dependency. Block it with an install directive instead:

echo 'install sctp /bin/false' > /etc/modprobe.d/csm-disable-sctp.conf
modprobe --show-depends sctp   # must print "install /bin/false"

If SCTP is already loaded, stop what uses it and run modprobe -r sctp, or reboot. A kernel with SCTP built in fails until it is replaced. The check warns when it cannot read the kernel config, /proc/modules, the modprobe configuration, or the load plan. A missing-module report also warns if CSM cannot inspect the module tree and its index, or finds an SCTP module there.

Future CVEs

New mitigations land here as they ship. The bar is:

  • The host can be measurably hardened (modprobe / seccomp / sysctl / config), and/or
  • An exploit-signature detector can fire reliably without false positives.

CVEs that are purely “patch the package”, with no preventive control we can apply and no signature we can detect, do not get a CSM mitigation; the right answer is the vendor patch. The daemon’s package-integrity check (rpm -V / debsums) only shows that installed files still match their package; it cannot tell whether that package contains a fix.

Firewall (nftables)

CSM includes a native nftables firewall engine that replaces LFD and fail2ban. It writes rules through the kernel netlink API directly via google/nftables. Integrity monitoring reads the installed structure with the nft command.

Features

  • Atomic ruleset - single netlink transaction, no partial application
  • Named IP sets with per-element timeouts (blocked, allowed, infra, country)
  • IPv4 options drop - inbound IPv4 TCP packets with IP options are dropped before conntrack, NAT to containers and every accept, closing the CVE-2026-98432 trigger on unpatched kernels; set drop_ipv4_options: false only for workloads that need IP options, such as CIPSO labels
  • Rate limiting - SYN flood, UDP flood, and per-IP connection rate are dual-stack (IPv6 metered per /64); per-port flood meters are dual-stack per source address; the concurrent connection limit is IPv4-only
  • Country blocking via MaxMind GeoIP CIDR ranges
  • Outbound SMTP restriction by UID (prevent spam from compromised accounts)
  • Local port locks - only listed users and root may open new connections to a port on this host
  • Subnet/CIDR blocking with auto-escalation from individual IPs and safety guards for infra, local, and allowed addresses
  • Permanent block escalation after repeated temp blocks
  • Dynamic DNS hostname resolution (updated every 5 min) with grace-period guard against transient resolver failures
  • IPv6 dual-stack with separate sets
  • Commit-confirmed safety - Juniper-style auto-rollback timer
  • Infra IP protection - refuses to block infrastructure IPs
  • Auto-response dry-run - safety default that forecasts eligible IP blocks and retains bounded examples without applying them
  • Verdict callback - optional advisory hook to the panel before each auto-block (allow / block / attach metadata)
  • cphulk integration - unblock flushes cphulk too
  • Mail spray history - unblock, allow, undoing a block and flush also clear the released addresses’ mail login spray history and failure counts
  • Audit trail - JSONL log with 10MB rotation
  • State persistence with atomic writes

Storage contract preparation

The durable firewall action service is implemented and tested through engine injection. Production activation, reader cutover, migration, restore, downgrade and operator recovery interfaces remain open.

Firewall actions record the actor, source, linked finding or incident, and complete before and after state before changing the kernel. Pending intent is separate from committed state. Recovery verifies target identity and expiry before recording an outcome; it does not blindly replay a mutation. Repeated request IDs reuse the original action and admission accounting. Audit delivery retries use the same action ID. Typed undo checks that the affected targets still match the recorded result.

Mail login spray blocks keep their eviction priority through restart, durable batches, and undo. Undo restores the original expiry. An operator replacement does not inherit the spray classification. Undoing an unblock restores the block, not the spray history the unblock cleared.

Proven outcomes are retained for undo and review, bounded two ways. The retention sweep drops delivered outcomes older than the findings-history setting, and a hard cap on retained outcomes and on their total size applies even when sweeps stay off. The size cap includes retained audit evidence. Existing journals are indexed on their first retention operation. Pending actions and outcomes whose audit has not been delivered are never dropped. The newest outcome also survives the hard cap even if it alone exceeds it. Hourly scan counters keep only the newest windows. Scan admission refuses pruned windows after a backward clock correction, preserving budget safety. Deleting an outcome ends the undo window for that action.

When an action cannot be proven, for example because the kernel did not answer in time, its outcome stays uncertain and the engine refuses further firewall changes rather than guess. The daemon retries recovery at startup and on its maintenance tick, which settles the outcome as soon as the kernel can answer. csm firewall actions shows what is waiting and why. If the kernel can never prove it, an operator inspects the host and records the answer with csm firewall actions resolve, which takes applied or rejected and an optional note. Kernel evidence wins over the operator: if recovery can prove the outcome at that moment, the proven one is recorded and the operator’s answer is not used. The decision and its note reach the action log.

Startup recovery runs before applying the firewall. If startup fails, pending actions remain available for automatic recovery and operator resolution. After resolving them, restart the daemon to retry firewall setup. A storage error while saving kernel evidence stops resolution; it never permits an operator assertion to replace that evidence. Failure to refresh committed state is also reported, even when the outcome was saved.

Applying an action writes only the entries that change, so the kernel write for one block does not grow with the size of the blocked set. Large removals use bounded messages within the same atomic update. Ranged sets are written whole because their start and end markers move together. If the kernel already expired an entry the action meant to remove, the set is rewritten instead, and both paths end at the same state.

Firewall state storage provides complete snapshot reads and revision-checked replacement using the existing database. Reads preserve expired entries, original timestamps, explicit provenance, duplicate allow entries and collection order. An uninitialized store, corrupt data or a failed read returns an error without usable state. Rejected or rolled-back writes preserve the previous snapshot and revision. An error after the transaction body succeeds reports an uncertain commit; callers must reconcile stored state before retrying and must not assume rollback or confirmed durability. Legacy bucket edits invalidate a committed snapshot instead of silently changing its revision. The contract does not activate runtime cutover.

Firewall storage metrics separate write wait from transaction duration and expose read time, pending writes, failures and snapshot batch size using fixed labels. Repeatable benchmarks exercise competing writers and local snapshot copying. Backup restore and downgrade compatibility require the later migration stage.

Mutation failures

Single-address blocks refuse loopback, unspecified, link-local, multicast and the limited broadcast address. The kernel drops multicast and limited-broadcast sources before the input chain. Subnet blocks refuse ranges overlapping loopback, link-local or multicast ranges, infrastructure, host interface addresses, or full-IP and port-specific allows, plus unspecified or broadcast individual addresses and default routes. Other ranges containing those addresses, such as 0.0.0.0/8 or 240.0.0.0/4, remain blockable. All these safety refusals are recorded as refused in the action log, like single-address refusals. Invalid targets and storage or kernel errors remain failures. Refusing a permanent promotion leaves the prior temporary block and its expiry unchanged.

Protected targets are refused before timed-block lifetime checks. Subnet dry-runs validate the target before reporting a proposed block. Manual requests also record safety refusals when durable action tracking is enabled. Legacy entries for protected addresses survive reloads and can still be unblocked.

WAF attacker reports for link-local addresses stay visible but advise investigating the traffic source instead of a block CSM would refuse.

Integrity checks compare live rule structure with the snapshot captured after CSM last applied the firewall. Editing or rehashing configuration alone never approves a changed ruleset. A failed snapshot capture reports a monitoring gap; after correcting nft availability, re-apply the firewall to restore its baseline. Dynamic set membership does not affect this comparison.

Firewall changes persist their intent before changing kernel rules. A failed atomic kernel transaction restores the previous state. Failed writes and rollbacks return errors without success audit records; expiry cleanup logs the failure and retries on a later pass. DNS refreshes also retry failed removals. Removing one allow source preserves any other active source for that address.

An unconfirmed-durability error means the replacement is visible on disk but storage did not confirm its survival across power loss. For kernel changes, CSM attempts to restore the previous state before returning the error. A failed rollback is reported as well. Correct the storage problem, inspect the saved state and live rules, and repeat the intended operation or run csm firewall restart to apply the saved state. Do not treat an error as a successful rule change.

Port-specific allow additions and removals update saved state only. Run csm firewall restart to apply them to the kernel; the command acknowledgement states that a reload is required.

Clearing a stale local threat score

Run csm firewall forget <ip> as root against the running daemon to clear one address’s accumulated local threat score. It accepts exactly one IPv4 or IPv6 address, including equivalent IPv6 spellings; CIDRs and extra arguments are rejected. There is no dry-run option.

The command clears equivalent stored spellings together and reports their highest score and total event count. Forgetting an IPv6 address also forgets the spam evidence of its /64 network, and the reply names that network. Blocks, allow lists, whitelists and raw event history remain intact. New findings immediately start a fresh scoring record, even if they arrive while the command runs. This does not suppress findings about the host’s own address.

Persistence is attempted immediately but remains best effort: a success reply confirms removal from memory, not durability across a restart. Check the daemon logs for attack database write failures if an old score returns after restart.

Attack statistics and event queries use the daemon’s state database or its configured attack database directory. If neither is available, they do not read event files from the working directory.

Stalwart bans

On a host whose mail server is Stalwart, Stalwart keeps its own list of banned addresses. When one address is unblocked, allowed or undone in CSM, unbanned from a hosting panel, or allowed through a hostname in dyndns_hosts or infra_ips, CSM also lifts Stalwart’s automatic bans of it through the Stalwart API key from csm stalwart setup. An allow through a hostname counts the same as an operator allow. This runs in the background, so the unblock never waits on Stalwart. Only the bans Stalwart makes on its own, for failed logins, failed recipients, loitering and port scans, are lifted. Blocks made by hand in Stalwart and bans with any other reason stay, and the daemon log says so. A flush and a subnet removal leave Stalwart’s list alone. Each lifted ban is written to the action log as operate.clear_stalwart_ban.

A lift that fails is written to the daemon log, which names the address Stalwart may still refuse, and to the action log when it got as far as the delete. Every unblock ends with Stalwart reloading its ban list, also when no ban was left to lift. So unblocking the address again retries a ban that Stalwart still lists, and finishes a lift whose reload did not happen. Until then Stalwart can keep enforcing a ban it no longer lists.

Startup failures

Overlapping, nested, duplicate, and adjacent ranges are merged for the kernel, including IPv4 and IPv6 ranges ending at the last address. This applies to infrastructure, country, Cloudflare, DoS exemption, and blocked subnet sets. Stored subnet entries keep their own source and expiry; removing or expiring one entry rebuilds the remaining coverage in an atomic transaction. Default routes remain forbidden as subnet blocks to prevent operator lockout.

CSM tries to initialize and apply an enabled firewall up to three times, with one-second and two-second delays between attempts. Each attempt builds a fresh atomic transaction. A failed apply keeps the previous kernel rules in place. The retry delays stop when the daemon shuts down.

If all attempts fail, CSM continues monitoring but reports degraded health. /api/v1/status and csm status --json expose automation.firewall_enabled: true, firewall_managed: false, and firewall_startup_error. The error remains available for the lifetime of that process. csm doctor reports a failed firewall check and a recovery step.

Inspect journalctl -u csm.service, correct the reported configuration or nftables permissions problem, and restart csm.service. A successful startup clears the error and enables the firewall-dependent services. Disabling the firewall deliberately does not degrade health.

If the firewall state file cannot be read at startup, CSM renames it to state.json.unreadable-<UTC time> (with a numbered suffix if needed) in the same directory before anything can replace it, then starts without the blocked and allowed addresses it held. It raises a Critical firewall_state_unreadable finding, and csm doctor fails its firewall state check, naming the kept file, until the next restart. automation.firewall_state_set_aside carries the same path. Review the kept file and add back the entries that must stay. This startup alert bypasses the finding queue and finding suppression rules. If the file cannot be renamed or synced, startup fails without retrying, even if installing the base rules also fails. A running CSM ruleset stays in place with its blocks; after a reboot, CSM installs its base rules without saved blocks, allows or port exceptions. The failed engine does not adopt replacement state or resume management when storage becomes readable. Correct the storage problem before restarting CSM. A running engine keeps its known state across an unreadable rewrite, including failed saves, and writes that state back when storage accepts a later save.

Before restoring blocks at startup, CSM asks the admission ledger about every block an automatic response attempt wrote. Blocks whose attempt is still running, ended unknown or failed are marked unverified and held back from the kernel. Firewall views omit them once the mark is saved; if the mark cannot be saved, the block still stays out of the kernel, but views can list it until a later save succeeds. Blocks whose attempt failed, or ended unknown and was later found expired and absent, are then removed from the saved state; if that removal cannot be saved, they stay held back and the next start removes them. If the ledger cannot be read, eligible blocks are restored as before and existing unverified marks are kept; none of those automatic blocks is evicted to make room for a new one, and csm doctor warns until a restart reads the ledger. automation.firewall_ledger_error carries the reason. A ledger missing its admission queue state counts as unreadable. The same check runs after a confirmation window restores its saved blocks. With a readable ledger such blocks are never evicted either: an operator block that finds the temporary limit reached only because of them is added above the limit. New timed automatic blocks record their origin explicitly; operator blocks and operator undo do not acquire that origin from their reason text. Older unstamped rows keep unknown origin. Automatic admission may replace a block that recorded its origin with a higher-priority one; a block of unknown origin is never replaced. An automatic block replaces only blocks that recorded an automatic origin: at the temporary limit it never replaces an operator’s temporary block or a block of unknown origin, and is refused when only those remain. An automatic alias of an operator block keeps that operator protection and the longer lifetime. An operator block may still replace them.

Rows of an address family without a blocked set neither fill the temporary limit nor become replacement victims. Turning off IPv6 filtering therefore does not make saved IPv6 blocks refuse a new temporary block. Checking the limit does not remove blocks. Dry runs and allowed verdicts keep existing blocks, even when the limit would require a replacement.

At the temporary limit, replacement uses the same live set image as the limit count. A saved row without an element cannot free a slot. When no replaceable row has an element, the new block is refused; if a complete image cannot be read, the saved rows remain the fallback. The element must belong to the address’s own family: an IPv4-mapped key in the IPv6 set does not prove that an IPv4 block is present.

Temporary IP blocks use a firewall clock that counts elapsed time and never moves backward. Setting the system clock back does not add that step to their lifetime; a forward step can shorten restored blocks. While the firewall clock leads system time, new blocks show expiry times that much later. Elapsed time alone does not close the lead, but a later clock correction can. The journal reports the first lead greater than one second once per firewall start.

lease.json, beside the state file, stores the firewall clock reading, boot identity and time since boot. CSM attempts to save it after each successful ruleset install and saves it before a new temporary block changes state or kernel rules. A restart on the same boot resumes no earlier than the saved time plus elapsed time since that checkpoint. A reboot supplies no proof of elapsed time from the previous boot, so the saved time alone provides that bound. The current wall clock and the admission ledger’s saved clock can move the resumed time forward.

An unreadable or damaged checkpoint logs a warning; restoration uses the remaining clock bounds. A save failure after ruleset installation warns without removing those rules. A save failure before a new temporary block refuses that block with an error. Older CSM versions ignore the checkpoint; if system time still trails the firewall clock at downgrade, saved temporary blocks can be restored for longer by that remaining lead.

Firewall and threat views omit unverified addresses, including duplicate spellings. When firewall state exists, threat views use it alone for block status; legacy tracking entries cannot make held-back blocks appear enforced.

CLI Commands

# Status
csm firewall status                              # Show status and statistics
csm firewall ports                               # Show configured port rules

# Block / Allow
csm firewall deny <ip> [reason]                  # Block IP permanently
csm firewall allow <ip> [reason]                 # Allow IP (all ports)
csm firewall allow-port <ip> <port> [reason]     # Allow IP on specific port
csm firewall remove <ip>                         # Remove from blocked and allowed
csm firewall remove-port <ip> <port>             # Remove port-specific allow

# Temporary
csm firewall tempban <ip> <dur> [reason]         # Temporary block
csm firewall tempallow <ip> <dur> [reason]       # Temporary allow

# Subnets
csm firewall deny-subnet <cidr> [reason]         # Block subnet
csm firewall remove-subnet <cidr>               # Remove subnet block

# Search
csm firewall grep <pattern>                      # Search blocked/allowed IPs
csm firewall lookup <ip>                         # GeoIP + block status lookup

# Bulk operations
csm firewall deny-file <path>                    # Bulk block from file
csm firewall allow-file <path>                   # Bulk allow from file
csm firewall flush                               # Clear all blocked IPs (subnet blocks kept)

# Safety
csm firewall apply-confirmed <minutes>           # Apply the firewall block from csm.yaml with auto-rollback timer
csm firewall confirm                             # Confirm applied changes
csm firewall rollback status|confirm|revert      # Manage pending config rollback
csm firewall restart                             # Reapply full ruleset

# Profiles
csm firewall profile save|list|restore <name>    # Profile management

# Audit
csm firewall audit [limit]                       # View audit log

# GeoIP
csm firewall update-geoip                        # Download country IP blocks

# Cloudflare
csm firewall cf-status                           # Show Cloudflare IP whitelist status

CLI confirmation window

csm firewall apply-confirmed <minutes> applies the firewall settings from disk with a rollback timer. csm firewall confirm accepts the current ruleset and engine configuration and cancels that timer. While confirmation remains pending, csm firewall restart and another apply-confirmed are refused. A rollback restores the ruleset and the saved blocks of the moment the window opened. When the window ends, CSM retimes temporary blocks from readable saved state, including IPv6 blocks when a restart during the window had disabled IPv6 filtering. Normal rollback restores the saved engine configuration first; confirmation after an unreadable saved configuration accepts the current configuration. A retiming failure logs a warning and releases the hold, so restored timers may still outlast the displayed expiry. Older windows with no saved block snapshot keep their restored timers.

If startup cannot read the saved engine configuration, rollback still restores the kernel ruleset and state, but the window stays pending. Once that restore is recorded, later restarts keep the blocks added since instead of restoring the window’s start again; a missing, truncated or mismatched record makes the next restart restore again, and an empty or corrupt marker is recorded like any other. After such a ruleset-only rollback a restart applies the configured settings, so the old kernel ruleset does not survive a reboot. csm status and csm doctor show a pending window, when it rolls back and why it cannot settle by itself. Inspect the current ruleset and configuration before confirming: confirmation accepts the current state; it does not recover the unreadable settings. To recover them instead, repair the saved configuration and restart the daemon: the settings are restored and the ruleset is rebuilt with the current blocks, and the window ends only once that rebuild applies. Removing the saved configuration does not end a recorded window. While a window is pending, a second apply-confirmed or a restart is refused with what settles it; waiting helps only while its rollback is scheduled and can restore the saved configuration. Confirmation or rollback can report a leftover snapshot cleanup error after ending the window. Fix that cleanup error before starting another window.

This CLI window is separate from the Web UI config rollback below, which uses csm firewall rollback status|confirm|revert.

Configuration

Firewall defaults can be edited in two places:

  • Web UI: Settings -> Firewall section. Port lists, rate limits, flood protection, deny caps, country block, and outbound SMTP restriction are all editable. Changes are restart-class. The save endpoint warns if the WebUI listen port is missing from tcp_in. The port_flood per-port rule list is YAML-only for now.
  • YAML: edit /etc/csm/csm.yaml directly. Run csm rehash then systemctl restart csm.

Tentative apply (rollback timer)

The Firewall section in the Web UI offers two save buttons. Save writes the new config and prompts you to restart. Apply with rollback timer writes the new config, restarts the daemon, and starts a timer (default 5 minutes, range 1-30). If you do not click Confirm before the timer expires, the daemon restores the previous config and restarts again. This protects against locking yourself out by, for example, removing the WebUI port from tcp_in.

When the Web UI is unreachable (firewall mistuned, daemon broken), use the CLI escape hatch:

csm firewall rollback status
csm firewall rollback confirm
csm firewall rollback revert

Rollback state survives daemon restarts (the snapshot and its firewall configuration are persisted in the state directory). On startup the daemon checks for a pending rollback: if the deadline has already passed it restores the previous config and restarts; otherwise it restores the running firewall configuration and rearms the timer for the remaining window. Backup and store export omit this transient state, so restoring an archive cannot re-arm an old confirmation window.

firewall:
  enabled: true
  ipv6: false                  # false = ALL IPv6 traffic bypasses the firewall; CSM raises a finding on dual-stack hosts
  conn_rate_limit: 200         # new connections per minute per source (IPv6 per /64; 0 = disabled; null = default)
  syn_flood_protection: true   # per-source SYN flood meter (IPv6 per /64)
  conn_limit: 400              # max concurrent connections per IPv4 source (0 = disabled)
  drop_ipv4_options: true      # drop inbound IPv4 TCP carrying IP options (default: true)
  smtp_block: false            # restrict outbound SMTP
  log_dropped: true
  dyndns_hosts:                # resolved every 5 min and whitelisted
    - "monitoring.example.com"

Passive FTP range

When passive_ftp_start and passive_ftp_end are both unset, CSM opens TCP 49152-65534 for passive FTP data connections only on cPanel hosts and on hosts with pure-ftpd, proftpd or vsftpd installed. This includes Plesk’s ProFTPD and Debian’s MySQL, PostgreSQL and LDAP Pure-FTPd packages. Other hosts keep the range closed. Explicit values take precedence; set both endpoints to 0 to keep the range closed.

Full firewall reference: Configuration - Firewall.

Auto-response interaction

Auto-block calls require firewall.enabled: true because they go through the firewall engine. The engine consults two policy hooks first:

  1. auto_response.verdict_callback - when enabled, the engine POSTs a signed JSON request to the panel after local validation and infra-IP safety checks. When a secret is configured, CSM rejects unsigned callback replies by default. The panel can downgrade to allow (audit-only), attach tenant_id for downstream correlation, or add a note. CSM fails open on hook errors. In dry run, admission also asks the panel about each block it forecasts. Wire contract: verdict callback contract.

  2. auto_response.dry_run - when true (or absent; safety default), BlockIP() skips the automatic IP mutation. With the forecast owner active, only committed predictions that pass capacity checks create new bounded examples; a legacy dry-run outcome alone does not guarantee an example. Manual csm firewall ... operator commands bypass via BlockIPForce and always apply. Verify with csm firewall status after policy changes; “Recently Blocked” timestamps newer than the last restart confirm live mode. See Auto-response - Dry-run safety default.

Subnet blocks refuse the default route and any range that contains an infrastructure IP, a resolved infra hostname, a local host address, a full-IP allow, or a port-specific allow. Remove the allow or narrow the CIDR before applying the block.

Allowlist precedence

Inbound IPv4 TCP packets with IP options are dropped in prerouting before any of the rules below, infra included, while drop_ipv4_options is on. Hosts that support the key report the firewall.drop_ipv4_options.v1 capability. The nftables input chain accepts infra_ips first, then drops blocked_ips, then accepts allowed_ips. Because the drop is evaluated before the allowed_ips accept, an allowlisted IP that lands in blocked_ips would still be dropped. The same applies to port-specific allows, because those rules are evaluated after blocked_ips too. To keep operator allows effective, the auto-block path refuses to add an IP to blocked_ips when it is on allowed_ips (set by csm firewall allow), has a port-specific allow (csm firewall allow-port), or is in a verified-bot range (built-in or reputation.verified_bots). Precedence:

  • infra_ips - hard protect. Never blocked by anything, auto or manual; subnet blocks containing one are refused.
  • tcp_in_allow sources - firm, one port. The accept for a listed source on its declared port sits right after infra_ips, ahead of the blocked address, blocked subnet and country rules. A block of that source, automatic or manual, takes effect on every other port and never on the declared one. To cut a source off, remove it from the entry. See firewall.tcp_in_allow.
  • allowed_ips, port-specific allows, and verified-bot ranges - soft allow. The auto-block path skips them, but an explicit operator deny (csm firewall deny, Web UI manual block) still applies, because operator commands go through BlockIPForce and bypass the soft-allow gate.

An operator block carries the lifetime the operator chose. csm firewall deny and the Web UI permanent block never expire; the Web UI 24 hour block expires in the firewall and takes its threat-database evidence with it, so the address does not keep scoring as malicious after the block is gone.

The Web UI 24 hour action refuses to shorten a permanent or longer block; unblock explicitly before changing its lifetime. Undo of a bulk block or unblock restores each original deadline, including permanent blocks, and never extends an expired block. A later operator decision supersedes the older undo action.

Lockout warnings

Config validation warns when an enabled firewall would cut off the management plane. csm doctor, daemon startup, and the Web UI save path all run the same checks, and the Web UI returns each warning once:

  • The enabled Web UI port is missing from tcp_in, or from an explicit tcp6_in override when IPv6 filtering is enabled. An empty tcp6_in inherits tcp_in. A port opened to at least one usable source in tcp_in_allow does not warn.
  • A restricted_tcp entry also appears in an effective public TCP allow list, but no infra_ips are configured. The restricted list only filters public accepts; it does not open ports itself. Matching ports are therefore reachable only through the port-agnostic infrastructure-IP accept rule, and with no infrastructure addresses they are reachable from nowhere. A port with a usable tcp_in_allow source stays reachable from that source and is left out of the warning.

These stay warnings and never block a save or a start: fronting the Web UI with a reverse proxy or reaching it over a VPN are legitimate reasons to leave the port out of tcp_in.

One more warning compares the policy against the host instead of against the config, so it runs in csm doctor, csm validate --deep, and the Web UI firewall save rather than on every load:

  • sshd listens on a port that tcp_in (or an explicit tcp6_in) does not allow. The shipped tcp_in leaves 22 out, because many hosts move sshd, so a host that never moved it loses SSH on the first apply. Every Port directive counts, including ones in Included drop-ins under /etc/ssh/sshd_config.d/, and a port named in restricted_tcp with infra_ips set is treated as a deliberate infra-only listener. A port opened to listed sources in tcp_in_allow is treated as deliberate too, unless none of its sources renders a rule. AddressFamily and ListenAddress limit the check to the IP families sshd exposes, and loopback-only listeners do not count because the inbound firewall always accepts loopback traffic. Hosts with no sshd config get no warning.

Egress

tcp_out is default-drop as well and ends in a TCP reset, so a host whose policy omits a port it dials does not lock an operator out; it goes silent. Every heartbeat, finding delivery or intel lookup fails at once with “connection refused”, which reads like the far end being down, while the host looks healthy locally. The same validation pass therefore warns when an enabled firewall’s outbound policy would refuse a connection the daemon itself needs:

  • The port of every enabled outbound endpoint in the config is checked against tcp_out: alerts.email.smtp, alerts.webhook.url, alerts.heartbeat.url, alerts.audit_log.syslog.address (tcp and tls transports), auto_response.verdict_callback.url, reputation.rspamd.url, reputation.upstream.url, reputation.report.targets[].url, reputation.central.set_url, signatures.update_url, signatures.yara_forge.download_url, sentry.dsn and updates.github_api_url. HTTP endpoints use their explicit numeric port, or the scheme default when the port is omitted. SMTP and TCP/TLS syslog addresses also accept TCP service names, matching their dialers. Disabled features are skipped, and so are loopback destinations, which the output chain accepts ahead of any port rule.
  • Port 443 is checked once for the built-in HTTPS endpoints (threat feeds, AbuseIPDB, MaxMind, YARA Forge, AI-crawler range feeds, release check), because dropping it silences all of them at once.
  • Every port under firewall.required_tcp_out is checked. That list is a declaration, never added to the policy: a conf.d fragment owned by an integration can state the ports its service needs, and csm doctor reports when the effective policy drops one instead of the operator discovering it from a silent node. The check runs against the merged tcp_out, so any config layer that permits the port satisfies the declaration.

On a restricted output chain, smtp_block installs per-user accepts for the mail ports ahead of the port rules, and those ports never get a port rule of their own. The daemon runs as root, so its alert mail is not warned about when tcp_out omits a port that smtp_block still lets root reach. A port declared for another service still warns under smtp_block, because a port declaration cannot prove that service’s user is allowed. When IPv6 is managed, an explicit tcp6_out is checked separately; an empty one inherits tcp_out, and the single warning covers both families. When only the IPv6 lists are set, IPv4 egress is accepted wholesale and the warning names tcp6_out. A literal IPv4 or IPv6 destination is checked only against its own family.

This catches the daemon’s own egress and whatever has been declared. It does not see what an arbitrary third-party process on the host dials; an agent that ships its own conf.d fragment should declare its ports under required_tcp_out there.

Value validation

Unlike the warnings above, these are errors, because the value cannot do what the operator meant:

  • Ports outside 1-65535 in any port list, including drop_nolog and the passive FTP range, and a passive FTP range that ends before it starts.
  • country_block entries that are not two-letter ISO codes.
  • port_flood entries with an out-of-range port, a protocol other than tcp or udp, or a non-positive hit count or window.

Two enums elsewhere get the same treatment because their consumers fall back to a default branch rather than failing: reputation.central.action (an unrecognised value became a challenge policy) and incidents.*.block_at_severity, which accepts only high or critical and silently disabled incident blocking on anything else.

Incident auto-block escalation

An incident-driven block used to be requested once per incident and never again. The block it applied expired after auto_response.block_expiry, but the marker saying “already blocked” did not, so an attacker who kept going past the expiry was never blocked a second time while the incident stayed open and kept collecting evidence.

A block is now re-requested whenever the previous one has lapsed and the incident is still active and still receiving qualifying findings, and each request lasts longer than the last:

blocklifetime
firstauto_response.block_expiry (24h by default)
second7 days
third and laterpermanent

Admission caps new temporary lifetimes at auto_response.block_expiry, live and in dry-run forecasts; shorter requests stay short and retries keep an earlier attempt’s absolute expiry, even after the cap is lowered. Admission does not turn a non-positive lifetime into a permanent block. The lifetimes in the table are what existing blocking applies. A block admission applied is never promoted to a permanent block while it stands.

A permanent block is never re-requested. Closing an incident, by an operator or by the stale-incident sweep, resets the ladder, so a later recurrence starts at the bottom rather than inheriting a months-old episode. Concurrent findings still collapse into one firewall call, and a declined or dry-run request is not recorded, so it can retry.

The ladder survives restarts and quiet intervals while the incident remains active. Closing the incident, manually or automatically, resets it; a pending block callback cannot restore the old ladder after that close.

The incident view carries a Block button whenever the incident has one unambiguous source address. Mixed-source or truncated timelines without an address in the correlation key do not offer a block target. The button asks for confirmation, blocks permanently, notes the block on the incident timeline as operator_block, and settles the ladder so the automatic hand-off does not re-request a block for an address the operator just blocked.

The block API accepts an optional incident_id. An invalid, unknown, or address-mismatched incident ID does not prevent the firewall block, but does not change the incident. Refreshing an existing temporary block does not advance the escalation rung. Blocks recorded on closed incidents remain audit actions without restarting the ladder.

Local port locks

firewall.local_port_locks limits which local users may open new TCP connections to a port on this host. Each entry is PORT:user,user; root is always allowed. Other users get a connection reset. Connections to other hosts on the same port and connections already open are not affected.

firewall:
  local_port_locks:
    - "8080:www-data,stalwart"

Use it for a service that trusts what local clients tell it, such as a mail server that takes the webmail client’s address from a forwarded header. List only users that run no site code: a PHP pool running as a listed user can connect too. A user name that does not exist is reported at startup and skipped; the port stays locked to the others. The status page and csm doctor read the locked ports from the running daemon after successful installation. Unapplied changes, failed replacements, and live rules that no longer match the installation do not count as protection.

Infrastructure IP DNS guard

Hostnames listed in top-level infra_ips or firewall.infra_ips are resolved every 5 minutes and their current addresses feed the infra auto-block guard. If a hostname stops resolving, the daemon emits an infra_ips_unresolvable Warning finding and keeps the last known addresses protected during the grace period (default 10 min). This prevents a transient DNS outage from deprotecting the management plane. The finding auto-clears when resolution recovers.

DoS-exempt ranges

Operators can declare IP ranges that bypass the new-connection rate meter for their IP family, the IPv4 concurrent connection-limit, and mail-port flood meters, preventing false-positive throttling and subnet auto-blocks for carrier CGNAT pools or mail-provider egress. Configure under firewall.dos_exempt_ranges (your own CIDRs) and firewall.dos_exempt_known_mail_providers (adds Google and Microsoft mail ranges, on by default). See Configuration - firewall.dos_exempt_ranges.

What exempt sources bypass

Sources in the exempt set skip three categories of metering:

  • Connection rate-limit - the new-connection rate meter (configured via conn_rate_limit) does not apply for the source’s IP family.
  • Concurrent connection-limit - the IPv4 concurrent connection cap (conn_limit) does not apply.
  • Mail-port flood meters - the port_flood rules on TCP 25, 465, and 587 do not apply for the source’s IP family.

Subnet auto-block (spray, ASN-crawl, and netblock escalation) also skips any subnet block whose CIDR intersects an exempt range, and exempt IPs are excluded from the per-subnet threshold count so they cannot push a subnet over the netblock limit. Auto-response subnet blocks whose range falls inside an exempt range are removed automatically at daemon startup and at the start of each auto-block cycle, but only while IP auto-response is live; dry run leaves them in place. Manually created IP and subnet blocks are never pruned, even if they fall inside an exempt range.

What exempt sources do not bypass

The following protections remain in force regardless of exempt status:

  • Manual blocks - csm firewall deny <ip> and csm firewall deny-subnet <cidr> go through BlockIPForce, which bypasses the exempt check. An IP or range that is both exempt and manually blocked is still dropped.
  • SYN flood protection - the SYN flood meter is not affected by the exempt set.
  • UDP flood protection - the UDP flood meter is independent of the exempt set.
  • Country blocking - country CIDR blocks apply unconditionally.
  • Port policy - tcp_in, tcp_out, and restricted_tcp port rules are not modified.

The rule ordering that makes this work: the nftables input chain evaluates blocked_ips (and subnet blocks) before the DoS-meter rules. So a manual block inside an exempt range still drops the traffic – the block is hit before the meter that exempt sources bypass.

Dynamic mail-provider ranges

When dos_exempt_known_mail_providers is true (the default), the daemon resolves Google and Microsoft outbound mail ranges at startup and pushes them into the firewall exempt sets before the first rule application. The ranges are discovered from the providers’ published SPF records (the Google and Microsoft mail SPF roots), so they track provider changes without a CSM update. They are cached on disk so the previous set is available immediately on subsequent starts. A built-in snapshot is used if the cache is missing or the first live refresh has not completed. The cache is refreshed every 12 hours; if a refresh fails or the nftables reapply fails, the previous overlay is preserved unchanged.

ModSecurity Integration

CSM detects ModSecurity (WAF) on Apache, Nginx, and LiteSpeed across cPanel and plain Linux hosts. Custom rule deployment, override writes, and reload management are currently cPanel-only; other platforms still receive status, staleness, and event detection.

Supported Web Servers

Web serverConfig candidatesStatus checkCustom rule deployment
Apache on cPanel EA4/usr/local/apache/conf/*, /etc/apache2/conf.d/modsec*, whmapi1 modsec_is_installedYesYes (via cPanel modsec user conf)
Apache on Debian/Ubuntu/etc/apache2/mods-enabled/security2.conf, /etc/apache2/conf-enabled/*, /etc/apache2/conf.d/modsec2.confYesNo
Apache on RHEL/Alma/Rocky/etc/httpd/conf.d/mod_security.conf, /etc/httpd/conf.modules.d/*YesNo
Nginx on any distro/etc/nginx/nginx.conf, /etc/nginx/modules-enabled/50-mod-http-modsecurity.conf, /etc/nginx/modsec/main.confYesNo
LiteSpeed/usr/local/lsws/conf/httpd_config.xml, /usr/local/lsws/conf/modsec2.confYescPanel only

When ModSecurity is not installed, the waf_status check emits a platform-specific install hint:

# On Ubuntu + Nginx:
Install: apt install libnginx-mod-http-modsecurity modsecurity-crs

# On Ubuntu + Apache:
Install: apt install libapache2-mod-security2 modsecurity-crs && a2enmod security2

# On AlmaLinux + Apache:
Install (requires EPEL): dnf install -y epel-release && dnf install -y mod_security

# On AlmaLinux + Nginx:
Install (requires EPEL): dnf install -y epel-release && dnf install -y nginx-mod-http-modsecurity

# On cPanel:
Install: WHM > Security Center > ModSecurity

Rule-staleness alerts scan both the flat CRS layout (/usr/share/modsecurity-crs/rules/*.conf) used by distro packages and cPanel vendor trees, including nested layouts such as modsec_vendor_configs/VENDOR/rules/*.conf. On cPanel, CSM maps WHM’s active configuration files to their vendor trees and checks the newest artifact in each loaded tree. This honors individual configuration overrides, keeps retired trees out of the result, and prevents a fresh unloaded vendor from hiding a stale loaded one. LiteSpeed also keeps the on-disk check as a backstop while cPanel rebuilds the active configuration list and rule tree. If WHM cannot provide the mapping, and on other platforms, the check keeps the conservative oldest-artifact behavior. Update instructions are platform-specific (apt update && apt upgrade modsecurity-crs, dnf upgrade modsecurity-crs, or WHM on cPanel).

Features

  • Custom CSM rules - IDs 900000-900999 in configs/csm_modsec_custom.conf (cPanel only today)
  • Rule override management - SecRuleRemoveById directives for false positive suppression
  • Escalation control - change rule severity or action per-rule
  • Live deny escalation - repeated ModSecurity deny events from one IP emit an escalation finding that feeds auto-response blocking. CSM-owned rules keep their existing per-rule escalation controls. CSM rules that match one specific exploit escalate on the first deny.
  • Disabled-scope detection - reports domains and accounts with the engine switched off, covering both the userdata flag and the per-account and per-domain config includes used by Apache and LiteSpeed in the std and ssl trees. A malware finding from a scheduled or account scan on such an account, or on a file served by such a domain, gets a context line naming the scope and the file that switches the engine off. It shows in alerts, the web UI, scan reports, webhooks and the audit log. Only the accounts and sites with malware findings are read for this. The line is not part of the finding’s identity: the finding keeps its severity, and turning the engine off or on neither re-alerts it nor undoes a dismissal
  • WAF event log parsing - correlates events by IP, URI, and rule ID
  • Hot-reload - apply changes without Apache restart (cPanel only)
  • Rule activation - ModSecurity reads rules only when the web server starts or reloads. When CSM’s installed rule sections change, for example after an upgrade or csm install, the daemon runs modsec.reload_command at startup or during its WAF check. Each rule change reloads once. Standalone csm check runs never reload. A failed reload raises a waf_status warning and is retried. Without a command, CSM only warns at startup and the rules wait for the next web server restart.

First-deny escalation requires a blocking action for an exact CSM rule ID. Request text cannot turn a warning into a deny. Disabling a rule’s firewall escalation excludes both new events and retained evidence from later blocking decisions. Existing firewall bans keep their configured expiry.

ModSecurity error-log detections take the client address only from the structured prefix before ModSecurity: or [MODSEC]. Fields later in the same record cannot supply a client address. Live monitoring, scheduled WAF checks, and the Web UI use the same parser. The module marker must follow the server’s structured header directly. Markers inside ordinary file errors, PHP errors, or other message text do not create WAF detections. Standard Apache, Nginx and LiteSpeed headers are accepted; custom headers outside those formats are ignored. Apache’s module-qualified error prefix, including [:error], identifies its unbracketed IPv6 connection port. Legacy Apache and libmodsecurity client fields keep the complete IPv6 address. Bracketed addresses with a port are also accepted. Custom Apache logs must retain the module field to distinguish unbracketed IPv6 addresses with ports from bare addresses. LiteSpeed IPv6 connection formats are unsupported: events keep an unknown client and cannot trigger an IP escalation. The Web UI uses the finding’s recorded source address, with a prefix-only raw-log fallback for older findings; it leaves the client unknown when neither is available. On Apache 2.4 a line must carry ModSecurity’s own module tag (security2 or security3). Lines that another module tags, such as PHP or CGI output, and LiteSpeed PHP stderr lines are never ModSecurity events, even when their text holds a ModSecurity message. A LiteSpeed line must have its standard header, with the connection field directly before [MODSEC].

Apache 2.2-style headers without a module tag remain accepted for compatibility. They cannot distinguish ModSecurity output from a hosted site’s message that starts with the same marker, so false alerts and visitor escalation remain possible with that format. Use module-tagged Apache logs to separate writers.

Header validation does not authenticate file-based records. Nginx can write literal line breaks from decoded filenames into its error log, allowing request text to create an entire apparent record on a new line. File-based real-time escalation and scheduled error-log scans remain exposed to that case. Native ModSecurity audit logs provide transaction framing for scheduled scans; fixing real-time ingestion requires a source that preserves authenticated record boundaries rather than splitting arbitrary log bytes into lines.

Scheduled audit scans read native serial logs and concurrent indexes from ModSecurity 2 and libmodsecurity 3, plus libmodsecurity 3 JSON logs. Counts use the recorded client address and denial action or response status. Request and response content cannot select another client or cancel another client’s counts. The existing web_server.modsec_audit_logs setting selects the candidate logs.

To scan error-log records instead, include the source in that candidate list and identify it with modsec_error_log or web_server.error_logs. Detected platform error logs are also recognized. These sources count only messages starting with Access denied with code after the module marker. Request text cannot switch their interpretation to an audit index or suppress a denial. Other audit candidates accept native audit records only. Custom error-log candidates must be identified this way when upgrading. Changing a source’s type restarts its scan so previous counts cannot carry across formats.

Only complete transactions within the recent line window contribute to these counts. The first scan and scans after log rotation read from the file’s start to establish transaction boundaries. Later scans resume from the saved position. Large existing logs can take longer on the first scan. Memory use stays bounded, and incomplete writes wait for the next scan.

CSM adds the default escalation exclusion for browser-marked upload progress requests once, including on upgrade, and keeps later operator changes. If the saved exclusions cannot be read, CSM keeps running and blocking requests but pauses ModSecurity firewall escalation, and the rules page reports the error instead of overwriting them. The next start retries the default update.

LiteSpeed logs every rule match as triggered!, whether or not the rule denied the request, so CSM looks up each rule’s action in the installed rule files. Apache logs say whether a request was denied and do not need the lookup. A rule whose action is block does what the SecDefaultAction for its phase says, and pass when there is none, as in ModSecurity. OWASP CRS in anomaly scoring mode sets pass, so its detection rules count as warnings, not denies. CSM does not follow Include order: a SecDefaultAction applies to the rules in its own directory and below, so a vendor pack’s setup file governs that pack and no other.

CSM reads rule IDs, phases and actions from the rule’s action argument. Text inside an operator, message or log data cannot override them. Explicit disruptive actions take precedence over inherited block actions. When setup files in the applicable directory cannot be read or parsed, inherited block actions remain blocking until CSM can resolve them. Explicit actions are still usable, and rules without a disruptive action remain unknown.

The directory walk follows symlinked packs and rule directories. Defaults follow the configured directory layout, including nested overrides. Directory cycles are reported as incomplete reads and prevent caching the result.

CSM checks the rule-action registry every five minutes and rebuilds it when rule or setup file contents change. Read failures leave the build uncached so the next check retries. Files that were read in full but exceeded the parser’s line limit are reported without forcing unchanged files to be parsed again. Rules appended after parsing reaches the end of a file are picked up on the next check.

The reload command runs inside CSM’s systemd sandbox. Use a service-manager command such as systemctl reload lsws for LiteSpeed, so the web server’s service performs the reload. Direct reload scripts inherit CSM’s filesystem restrictions and may fail. A LiteSpeed reload restarts workers and can briefly raise load.

The LiteSpeed Cache role-simulation filter covers privileged routes and writes, including WordPress REST method overrides, when requests carry simulation cookies with a weak hash. Ordinary public GET/HEAD crawling remains allowed. This is a request-scoped mitigation: public reads still run under the simulated identity, so upgrading the vulnerable plugin remains necessary.

The usual hash discriminator is 1-16 alphanumeric characters versus the fixed plugin’s 32-character hashes. Numeric equivalents are also filtered because the vulnerable plugin compares hashes loosely; padding or exponent notation must not turn a weak hash into an exempt one. These checks also keep public crawler reads allowed.

The WordPress user enumeration filter blocks anonymous requests for the REST users route, whether the route follows wp-json/ in the path or starts at wp/v2/users in the rest_route query parameter, in any letter case. Other page paths and REST namespaces are left alone. Query values are matched as already decoded by the query parser. Requests that carry an Authorization header or a WordPress logged-in cookie pass, so admin screens, the editor and Application Password clients keep working. Both are presence checks: the filter turns away anonymous scanners, and WordPress still decides who is signed in.

The filter runs in phase 1 and does not inspect request bodies. A route supplied only in a POST body, including with a REST method override, is outside its scope. Sites that need to restrict the public users endpoint must enforce that policy in WordPress. Disabling rule 900112 disables this filter for both route forms; its helper rules only set transaction-local flags and do not block requests.

The PHP session upload-progress filter denies requests whose form field names contain PHP’s default PHP_SESSION_UPLOAD_PROGRESS field name, matched case-insensitively so quoted spellings are covered. A custom session.upload_progress.name is outside this filter’s coverage. GET and HEAD pass. Other methods use the first-deny escalation rule when both Origin and Sec-Fetch-Site are absent; browser-marked requests use the separate rule whose escalation is off by default. Keep that exclusion to avoid browser-steered bans. A browser whose extension or proxy strips Origin on a plain-HTTP request can still reach the first-deny rule. PHP handles that field before any script runs, which leaves the web server as the only layer that can drop the request. See CVE Mitigations for the PHP-side setting that removes the handler.

For Apache ModSecurity v2 regression validation, run python3 scripts/test-litespeed-modsec.py in a disposable Debian Linux environment with apache2, libapache2-mod-security2, libapache2-mod-php, and python3 installed. The test loads the complete shipped configuration and exercises HTTP requests through the actual engine and PHP parser. It does not establish LiteSpeed runtime compatibility; verify changed rules on the supported LiteSpeed engine before deployment.

Web UI Pages

ModSecurity (/modsec) - WAF status overview, event log, active block list, filterable by time range, minimum severity, and source country

ModSec Rules (/modsec/rules) - per-rule management:

  • View the CSM rules with descriptions and hits in the last 24 hours
  • Enable or disable individual rules; changes are staged and applied with one reload
  • Turn firewall escalation off for a rule: the rule still denies the request, but CSM does not block the IP in the firewall
  • Escalation exclusions are listed and edited here even when rule management (modsec.rules_file, modsec.overrides_file, modsec.reload_command) is not configured, since the daemon applies them either way

API Endpoints

GET  /api/v1/modsec/stats            WAF statistics
GET  /api/v1/modsec/blocks           Blocked request log
GET  /api/v1/modsec/events           WAF event details
GET  /api/v1/modsec/rules            Loaded rules list
POST /api/v1/modsec/rules/apply      Apply the set of disabled rules and reload
GET  /api/v1/modsec/rules/escalation Rule IDs excluded from firewall escalation
POST /api/v1/modsec/rules/escalation Exclude one rule from escalation or turn it back on

Signature Rules

CSM uses YAML rules for real-time scanning and finding re-checks. Optional YARA-X rules also run during deep scans and email attachment scanning. Rules are stored in /opt/csm/rules/. An engine that loads zero rules (a mistyped signatures.rules_dir, an empty rule sync) raises a realtime_rules_missing finding at startup and after every reload, because both engines otherwise treat an empty directory as a successful load and scan every write against nothing.

Deep scans are rolling: each scheduled run resumes from a persisted cursor and scans as much as fits in its time budget, so the whole content set is covered across runs even when a single run cannot finish it. A warning finding is raised if no full pass has completed within 30 days.

The same rolling walk also feeds the JavaScript keystroke taint analyzer (js_keylogger_dataflow) and the PHP remote-source taint analyzer (php_remote_taint); see the deep checks reference for both. Each consumer keeps its own cursor and completion record, so none of them stalls the others: a missing or failed YARA backend does not hold up either taint analyzer, and the PHP analyzer being unavailable does not affect YARA or JavaScript coverage.

A rule declaring file_types: [".php"] is applied to every extension a stock PHP handler executes (.php2 through .php8, .phtml, .pht) and to .phps, because that source-view extension still contains PHP source. .phps stays outside the set of extensions CSM treats as executable, so this does not change which files the real-time dropper tracker considers runnable.

Both engines skip ZIP, gzip, bzip2, xz, 7z, and RAR containers, but only when the file name carries an archive extension as well as the archive signature. Matching compressed bytes or stored filenames produces false positives without inspecting the archived file, so real containers are scanned when they are extracted onto monitored storage instead. A file whose name a web server would execute is always scanned whatever its first bytes look like, because PHP echoes any leading bytes and runs the rest. Uncompressed tar files and executable PHP archives (PHAR) remain scannable, and filename-based phishing-kit archive detection is unchanged.

YAML Rules

Encoded spam-link detection applies to real-time scanning and finding re-checks only. It looks for external-link attributes in PHP that directly prints a base64 literal. Embedded tracking pixels, local image references and inline logos without external links do not supply that evidence. An encoded external link is a heuristic, not proof of spam or proof that the destination is outside the site’s own domain. Scheduled YARA scans do not use this rule.

The WordPress REST API exploit signature is a YAML-only heuristic. It requires a users-endpoint URL literal and a password query parameter or a nearby PHP, JSON or form-encoded password field, including payloads prepared before the URL. Endpoint prose alone does not qualify. This is bounded textual evidence, not PHP data-flow analysis or proof that a request is unauthorized.

rules:
  - name: webshell_c99
    severity: critical
    category: webshell
    file_types: [".php"]
    patterns: ["c99shell", "c99_buff_prepare"]
    min_match: 1

  - name: phishing_login
    severity: high
    category: phishing
    file_types: [".html", ".php"]
    patterns: ["password.*submit", "credit.*card.*number"]
    exclude_patterns: ["legitimate_form_handler"]
    min_match: 2

Fields:

  • name - unique rule identifier
  • severity - critical, high, or warning
  • category - webshell, backdoor, phishing, dropper, exploit
  • file_types - file extensions to match (or ["*"] for all)
  • patterns - case-insensitive literal strings
  • regexes - regex patterns, compiled case-insensitively
  • regex_pairs - entries with first, second and max_distance; each pair counts as one regex match when its match starts are within that many bytes, in either order
  • exclude_patterns - literal patterns that suppress a match (false positive reduction)
  • exclude_regexes - regex patterns that suppress a match
  • min_match - minimum total number of matching literal and regex entries; each entry counts once
  • require_regex - require at least one regex among the matches counted toward min_match
  • max_file_bytes - skip this rule when the complete scanned file is larger than the byte limit; omitted or 0 is unbounded
  • max_file_bytes_exempt_regexes - high-confidence regexes that let the rule continue normal evaluation above max_file_bytes

A regex only runs on a file that contains the fixed text every match of it must contain, compared without regard to case: for eval\s*\(\s*base64_decode that is both eval and base64_decode. Write regexes around distinctive words such as function names: a regex with no fixed text of two or more characters runs over every file of its types, and realtime pays that cost on each write. Required text can include escaped bytes such as \x00; these remain part of the literal when checking whether a file could match.

Regex pairs compare the ordered, non-overlapping matches of two expressions. Both expressions must be non-empty and compile; max_distance must be non-negative. A distance of zero requires both matches to start at the same byte. Use pairs for nearby evidence instead of joining expressions with a wide wildcard gap, which can slow finding re-checks on large files.

When a regex includes a literal listed in patterns, the same content can satisfy both entries. Use independent entries when a rule needs multiple pieces of evidence. The bundled HTTP tunnel rule requires both socket creation and a CONNECT request. The legacy PHP callback rule uses the same narrow signature in YAML and YARA-X: a direct function call with a quoted parameter list, a variable, null, a simple array lookup or a short helper call as its first argument. The second argument is a decoder or request lookup, optionally preceded by one concatenated literal. Quoted lists can contain commas, semicolons and escaped quotes. Array indices accept a single quoted key or an unquoted scalar. Helper arguments accept at most one literal among unquoted scalar operands, including implode(',', $args). These bounded forms consume quoted operands whole and exclude comments, interpolation and nested expressions, so delimiters inside data cannot supply the body-source evidence. Double-quoted array keys, helper literals and body prefixes must escape dollar signs; unescaped dollars require interpolation analysis. Shared positive and benign fixtures check both engines. Generated socket and funchand wrappers and ordinary legacy callbacks stay silent under these rules.

The cron downloader signature shares its expression between both engines. It recognizes numeric five-field schedules and the standard @ shortcuts, including lists, range steps and randomized ranges. The month and weekday fields also accept a single three-letter ASCII name in either case. Unicode lookalikes are not schedule names. Name lists and ranges are not matched, because they made crafted lines too costly to scan. Schedule tokens must be complete and separated by spaces or tabs. Fields are limited to 64 characters. Root entries that disable cron logging and scripts that install a quoted cron entry remain covered. It requires a download followed by a complete shell, eval or exec command on the same line. Quoted shells, command groups, common execution wrappers and shell lookup commands are recognized. Bounded spans keep crafted lines cheap to scan; unrelated lines and shell words inside longer command names are not evidence.

The download-and-pipe, miner download, miner cron and shell startup file signatures also look for their parts within bounded spans of one line, in both engines. A download command is found wherever it sits on a long line, and a # shortly before it still marks a commented-out usage line. The YARA-X rules treat a command as documentation only inside a fenced code block, whose opening fence starts a line and whose closing fence stands on its own line, or as the text of a Markdown link. The shell startup file signature judges a file with more Markdown blocks and links than any published document carries without that suppression, so its scan time stays linear. The download-and-pipe and shell startup file signatures need a whole shell or interpreter name after the pipe, so a download piped into a checksum tool or another program whose name starts the same way is not an execution. Versioned Perl and Python names, Python build variants and standard Windows executable names retain detection. Both engines use the same shell name boundaries, including commands embedded in strings and binary data. The miner cron signature ignores letter case in both engines, accepts only spaces and tabs between schedule fields, and counts its bounded command span in characters so non-ASCII paths have the same coverage. The other bounded spans are measured in bytes by the YARA-X rules and in characters by the YAML rules, so the two can disagree on a line with non-ASCII text in the span. Measuring characters in YARA-X slowed scans of crafted lines too much. A finding is re-checked by the engine that raised it.

Both engines report download-and-pipe commands even when a file begins with HTML or has an HTML extension. This includes install commands printed on web pages. Leading markup does not prove that later lines cannot run as shell code.

The unauthenticated uploader signature skips a framework helper class: a file holding only a direct-access guard that exits and one class declaration, which writes the upload through the object and creates no object itself. Code after the class, or syntax the check cannot read, keeps the alert.

The PHP goto-obfuscation rule requires three independent signals in both engines: a PHP opening tag, at least nine jumps to digit-bearing generated labels or eleven to alphabetic labels, and a decode call, execution call, dynamic include, or request input. Fixed-path bootstrap includes do not supply this evidence. Variable and array callback calls count even when the function name is constructed and the argument is a literal. Comments between a callable and its opening parenthesis do not hide the call. Line breaks and keyword case do not change the label counts. Long encoded strings and data URIs alone are not execution evidence. These are source-text heuristics, not PHP dataflow analysis; they cannot resolve arbitrary dynamically generated code or distinguish every benign use of these operations.

The PHP content heuristic that runs during scans applies the same evidence rule: generated goto labels and descriptive goto labels both need a decode call, execution call, dynamic include, or request input before they count as an obfuscation indicator. call_user_func is deliberately not evidence in either place, because plugin loaders dispatch their own callables through it. The content heuristic uses the signature’s evidence expression, including comment-separated calls, grouped and array callbacks, and case-sensitive PHP superglobal names. A regression check guards against expression drift.

The archive loader signature reports PHP that runs code read out of an archive, in both engines. A ZipArchive member read, or a zip:// or phar:// path, must sit inside an eval, assert or create_function call, lie within 1000 bytes of an eval that leaves PHP mode first, before or after it (measured between match starts in both engines), or follow a password set on the archive and precede eval of a variable. Extracting archives to disk, parsing data members and template engines evaluating local text are not evidence, nor is a bundled tool such as a .phar whose stub names an archive path far from a template engine’s eval. Including a zip:// path is reported with or without parentheses. These are source-text heuristics, not PHP dataflow analysis. Execution arguments stop at the end of their call while allowing balanced prefix calls. Assertions that parse or compare archive data are not execution evidence. Both strings passed to create_function remain source text, as described in the PHP manual. Regression fixtures run through both engines, including mixed-case calls and ordinary archive and template code.

Three signatures report code an injection kit leaves in WordPress files, in both engines. The first-line stub signature matches only a file that opens with an array holding an empty string followed by an isset test with an empty body. The line does nothing, outlives cleaners that strip only the payload, and no generator writes it. The visitor logger signature needs a site or host hash, cookie handling, an append-mode fopen and a visitor address read, each starting within 1200 bytes of a lookup of the upload temp directory. The hook injector signature needs add_action or add_filter called with a hook name returned by a call on an encoded literal, starting within 4096 bytes of both base64 decoding and a byte-wise XOR loop. Plugins that name hooks in plain text, keep caches in the temp directory or mask licence keys with XOR are not evidence.

Legacy callback parser follow-up

The callback signature does not inspect quoted function bodies. Doing so needs PHP string decoding, tokenization and expression analysis: for example, assert($x > 0) is an ordinary boolean check, and "eval($x)" can be data. The rules scan source text, so they do not promise general PHP comment or string awareness, nor complete coverage of dynamically generated code.

The following cases were covered by the expanded regex and are deliberately outside the narrowed signature. They remain acceptance cases for a parser follow-up, not claims of current detection by this rule:

Deferred caseExamples to restore
Constructed parameter listsConcatenation, nested array indices, helper calls with multiple quoted operands, and chr(100/(1+1)) as the first argument; only the bounded simple forms above are covered
Comments inside constructed parametersComments in array indices or helper arguments, especially those containing closing delimiters or body-source names
Interpolated parameter operandsDouble-quoted array keys or helper operands containing unescaped dollars, including interpolation with nested quoted keys
Comments between argumentsBlock comments with embedded commas, and line comments before the body source
Constructed body expressionsGrouped concatenation, parenthesized decoders and trim(base64_decode($payload)); a single literal concatenated onto request input is covered
Interpolated bodiesA double-quoted body such as "return {$_POST['code']};", or a concatenated double-quoted prefix containing unescaped dollars
Literal executable bodieseval($x) or string-capable assert($x), with statements, strings or comments before them; both outer quote styles and escaped quotes
Literal expression contextsreturn, or, do, case, include, include_once, require, require_once, clone, yield from, comparisons, shifts and inequality before an execution sink
Literal lexical edgesGlobal assert, comment backslashes before * or */, and quoted operands before a comparison
Outer call contextsCalls immediately following a ternary colon, case-label colon or comparison operator

This work belongs in internal/phptaint, which already uses VKCOM/php-parser and records the second argument of create_function as a sink. Extend that analysis to decode and parse statically known callback bodies under its existing budgets, distinguish boolean assertions from string execution, and report unresolved dynamic bodies as analysis gaps. It currently feeds a separate scheduled check; it is not a post-filter for YAML or YARA. Realtime coverage would need explicit integration and latency tests, and standalone YARA would still have the narrower coverage.

Acceptance requires restoring the deferred cases as positive parser fixtures, retaining the shared benign fixtures, checking both quote styles and comment forms, and passing the clean-corpus gate without new baseline entries. Do not expand another regex into a PHP tokenizer to recover these cases.

YARA-X Rules (Optional)

Build CSM with YARA-X support:

CGO_LDFLAGS="$(pkg-config --libs --static yara_x_capi)" go build -tags yara ./cmd/csm/

The rules directory and rule files must be owned by root or the scanner user and must not be group- or world-writable. The standard service runs as root and reads the rule files; the YARA workers get the rule text from it and never open them. The installer uses 0750 for the directory and 0640 for shipped rules. Custom rules must also remain readable by the service. With the default worker enabled, rule files cannot include other files.

Place .yar or .yara files alongside YAML rules in /opt/csm/rules/. CSM compiles them at startup and uses them for:

  • Real-time fanotify file scanning
  • Scheduled deep-scan filesystem sweeps
  • Email attachment scanning

Scans hand every file to YARA-X regardless of its name, so a rule may decide on the leading bytes rather than the extension. The bundled rule for PHP carried inside an image does exactly that: it requires a genuine PNG, JPEG, GIF, WebP, ICO, BMP or TIFF container, a PHP opening tag, and an execution or remote-fetch construct beside it. A picture is a working payload store, and nothing about its name says so.

Its other half is the loader: a PHP file that includes or requires a non-executable file while reading request input. The extension list covers images, archives and opaque data files; source partials such as .html, .tpl, .txt and .svg are deliberately absent, because including those can be ordinary templating. Literal targets must end the include statement; an image named later in a rendered HTML expression is not an include target. Encoded targets remain suspicious regardless of the hidden extension. Findings include up to five distinct non-executable path literals near an include or require. These are candidate payload references: proximity does not prove variable identity, and concatenated paths may be partial. Extraction advances through the content once and stops when the output cap is reached.

Native Linux programs are judged by content too. A program that carries a wget or curl download command with an open output path, together with either a URL to a PHP script or .htaccess file (including at the URL root), or PHP source that writes files, is reported as a Critical dropper wherever it sits. The ELF header decides what counts as a program: non-PIE, PIE and static PIE executables do, shared libraries (native Node.js modules, Python extensions) do not. The executable bit is not consulted. A program in a web root without those traits is not reported: vendor download binaries, bundled build tools such as esbuild, image optimizer binaries and old CGI programs are normal there.

Scheduled YARA sweeps scan regular, non-empty files under configured account_roots, or cPanel /home/*/public_html roots when no override is configured. Files larger than thresholds.full_scan_max_file_mb, symlinks, and special files are skipped. An unreadable or oversized file, or a scanner backend error, emits yara_scan_incomplete and preserves findings from the previous complete sweep. Scheduled findings and real-time fanotify findings have separate ownership, so one path cannot purge the other’s results.

A real-time scan that cannot inspect a changed file emits yara_realtime_scan_error instead, so a scanning outage stays separable from the scheduled coverage report. The dashboard’s Components matrix shows this finding as Fanotify’s last event; scheduled coverage reports do not advance that event time. This error finding is suppressed while the file monitor is stopping: the YARA backend is stopped before the file monitor has finished draining, and a clean restart is not an outage.

With signatures.yara_worker_enabled: true (the default), files are reported as not scanned until the first rule load succeeds, including while a failed startup compile is retried. An empty rules directory counts as loaded for files, but mail parts are reported as not scanned until at least one rule loads. A failed reload keeps the rules already loaded. A file or mail part whose content made the scan worker fail twice within a day is temporarily reported as not scanned without another attempt, including when copies are already queued. A successful rule load clears these failures; they also expire a day after the last failure. The oldest failures may be forgotten earlier if many different inputs fail.

Without the yara build tag, YARA rules are not loaded or evaluated, and the worker backend reports files as not scanned.

Updating Rules

Before merging bundled YARA changes, run the rules against an unpacked corpus of clean WordPress core and plugin files:

YARA_FP_CORPUS=/path/to/corpus go test -tags yara ./internal/yara/ -run TestRepositoryRulesAgainstCleanCorpus -v

The same corpus measures the YAML rules that only realtime and finding re-check run:

YARA_FP_CORPUS=/path/to/corpus go test ./internal/signatures/ -run TestRepositoryYAMLRulesAgainstCleanCorpus -v

The same corpus also checks that no YAML regex is skipped on a file it matches:

YARA_FP_CORPUS=/path/to/corpus go test ./internal/signatures/ -run TestYAMLGatesSoundOnCleanCorpus -v

Both gates require at least 5,000 non-empty files within the default scheduled scan size limit. Rule-load, traversal, and read failures fail the relevant run instead of counting as clean; YARA backend errors do too.

The measured YARA baseline is empty. The YAML baseline records rules that already fire on clean plugin and core code and are named in the realtime-rule porting backlog. Tighten a noisy rule rather than excluding paths or filenames.

csm update-rules          # download latest rules and reload the running daemon

csm update-rules now asks the daemon to reload through the control socket once the download completes. If the daemon is not running, the next start picks the files up automatically. kill -HUP $(pidof csm) still works.

Or from the web UI: Rules page > Reload Rules button.

Remote rule updates are now signature-verified. Any configuration that enables signatures.update_url or signatures.yara_forge.enabled must also set signatures.signing_key to the 64-character hex-encoded Ed25519 public key that verifies the downloaded .sig files. Both URLs must use https; a plain-http URL fails validation.

A valid signature proves who published a rules file, not that it is the current one. The YAML updater therefore also refuses a download whose version is lower than the installed file’s, or that carries fewer than half as many rules, and keeps the installed rules. The daemon emits a Critical finding for either refusal so a stale mirror or a replayed old release is visible instead of silently stripping detection. A missing or unparsable installed file is not compared, so a signed update remains the way to recover from a corrupt rules file.

For an intentional reduction below half the installed YAML rule count, set signatures.allow_rule_count_decrease: true, restart CSM, and run the signed update. Then return the setting to false and restart CSM again; this exception never permits a version downgrade.

Remote update URLs must use https and must not point at localhost, loopback, link-local, unspecified, or RFC1918 / ULA private addresses.

YARA Forge Integration

CSM can automatically fetch curated YARA rules from YARA Forge, which aggregates and quality-tests rules from 40+ public sources including signature-base, Elastic, Malpedia, and ESET.

Configuration

signatures:
  signing_key: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
  yara_forge:
    enabled: true
    tier: "core"              # core (5K rules, low FP), extended (10K), full (12K)
    update_interval: "168h"   # weekly
    download_url: "https://mirrors.pidginhost.com/csm/yara-forge/{version}/yara-forge-rules-{tier}.zip"
  disabled_rules:             # rule names to switch off, in Forge and in the shipped rules
    - SUSP_Example_Rule

The project operates the signed mirror shown above. A ready-to-use drop-in is shipped at /usr/lib/csm/profiles/yara-forge.example.yaml; copy or include it under /etc/csm/conf.d/ to enable Forge without editing the main csm.yaml. The matching signing_key is the project Ed25519 public key (hex), published on the release signing page.

signing_key must be a hex string for the Ed25519 public key that matches the private key used to sign the remote Forge artifact. It is not a PEM block and not a file path.

YARA Forge’s upstream GitHub releases publish ZIP files, but not CSM detached signatures. CSM therefore requires yara_forge.download_url to point at a mirror you operate. The URL may contain {tier} and {version} placeholders. The detached signature must be available at the resolved ZIP URL plus .sig.

When the URL has a standalone {version} path segment, CSM first reads a plain-text latest pointer from that version directory’s parent, for example https://mirrors.pidginhost.com/csm/yara-forge/latest. If the pointer is not published, CSM falls back to the upstream GitHub release tag. Mirror network errors and server errors fail the update instead of falling back, because GitHub may name a release the mirror has not signed yet. CSM accepts only short tag tokens from the pointer; the ZIP signature still gates installation.

If you do not have a signed update source yet, disable remote updates instead:

signatures:
  signing_key: ""
  update_url: ""
  yara_forge:
    enabled: false

Tiers

TierRulesSizeFalse Positive Risk
core~5,0001.6 MBLow (quality >= 70, score >= 65)
extended~10,5003.3 MBMedium
full~11,6003.7 MBHigher (includes score >= 40)

Update Flow

  1. CSM resolves the latest YARA Forge version from the mirror pointer, falling back to GitHub only when no pointer is published
  2. If different from the installed version, downloads the ZIP for the configured tier from yara_forge.download_url and its detached signature
  3. Verifies the download against signatures.signing_key
  4. Filters out any rules listed in disabled_rules
  5. Compile-tests the rules with YARA-X before installing, in the compile unit when worker mode is enabled
  6. Atomically replaces the previous Forge rules file
  7. Reloads the YARA scanner

Custom rules in malware.yar are never overwritten by the Forge fetcher.

Disabling Rules

If a rule produces false positives, add its name to disabled_rules in the config and restart the daemon:

signatures:
  disabled_rules:
    - SUSP_XOR_Encoded_URL
    - php_goto_obfuscation

The list covers every rule CSM loads, not only YARA Forge: Forge rules are stripped from the download, rules shipped in malware.yml are skipped when the real-time engine loads them, and rules shipped in malware.yar are stripped before the scheduled engine compiles them. The self-test measures the ruleset that is left, so disabling a rule shows up as the coverage it costs.

Names are matched in full, ignoring surrounding whitespace and letter case; prefixes and substrings do not match. Removing a YARA rule preserves neighboring rules, including when declarations share a line or literals contain braces. The daemon strips disabled rules before every compile, so rule reloads and worker restarts keep honoring the list.

A valid ruleset whose rules are all disabled loads as an empty set, including on reload; stale rules are not retained. The self-test reports the resulting misses. An empty or invalid replacement still reports a load error.

csm validate warns about a name that matches no rule, because a typo here otherwise reads as “that rule is off” while the rule keeps firing. Disabling a rule is a last resort and a standing gap in coverage; prefer fixing the rule.

Validation recognizes a disabled YAML rule even if its regular expression is invalid, and counts repeated names only once.

Signature settings require a daemon restart. SIGHUP does not apply a changed disabled list; rule-file reloads keep the current list.

How Rules Avoid False Positives

Signature rules require structural nesting, not co-presence of strings. Two dangerous function calls appearing in the same file but in unrelated code paths won’t trigger a rule. The call must directly wrap or chain with the other for a match.

YARA-X rules cannot express nesting, so multi-string rules state how their evidence is related. A bounded distance is used when closeness is part of the malicious shape. When valid malware can carry padding between its signals, PHP diagnostic records are kept as separate contexts so one record cannot borrow evidence from another. Anchoring works the same way: a CGI webshell rule requires its shebang at offset 0, because that is where the web server needs it. None of these controls refers to a file’s name or path.

Realtime signature auto-quarantine adds a safety gate: only webshell and dropper matches are eligible, and the file must be at least 512 bytes and either have Shannon entropy >= 5.5 or hex density > 20% plus an obfuscated-execution signal. Legitimate plugin code (well below 5.5 entropy) passes through; obfuscated malware (5.8+) is caught.

Alert Rate Limiting

Default: 30 operator alert dispatches/hour (configurable via max_per_hour). CRITICAL findings and threat-intel reputation sightings always get through by email or generic webhook regardless of the rate limit, and they do not count against it. Other lower-severity alerts are rate-limited, including when they are batched with a CRITICAL finding: once the budget is spent, only the urgent findings in that batch are sent.

A dispatch uses one slot only when email or a generic webhook successfully delivers routine findings. If only urgent findings reach a channel and routine delivery fails, the slot remains available for a later dispatch. Email-disabled findings do not reserve a slot unless a generic webhook will carry them. The phpanel webhook stream bypasses this budget.

Summary-class findings never reach the cap: attacks CSM already blocked or challenged, its applied block and challenge notices, and the other findings listed under Alert classes in the configuration reference go to the daily summary instead of email or the generic webhook.

Suppressions

Create suppression rules to silence known false positives:

  • From the Findings page: click the suppress button on any finding. The path pattern is a glob; the dialog pre-fills the finding’s own file with [, ], *, ? and \ escaped, so the rule matches that file only
  • From the Findings page with several findings selected: bulk Suppress creates one rule per selected file
  • From the Rules page: manage suppression rules directly
  • Via API: POST /api/v1/suppressions

A suppression rule hides matching findings from the Findings page, stops their email and webhook alerts, and stops file, process and account remediation for them. It does not stop IP blocking, challenge routing or attack scoring: a rule that mutes a whole check would otherwise leave every attacker that check reports unblocked. To exempt an address that was blocked by mistake, whitelist it on the Firewall page under Allow Rules or with csm firewall allow.

Incident auto-blocking, credential-spray containment and central threat intelligence also use suppressed findings. Database response never blocks session IPs; a suppressed database finding cannot trigger cleanup or session revocation. IP action notifications are separate findings; suppressing their check type mutes those notifications without stopping the action. These rules apply to startup, scheduled and real-time scans, control-socket runs with alerts enabled, and replay after restart.

To suppress email alerts for specific checks while keeping them visible in the web UI, use disabled_checks in your config:

alerts:
  email:
    disabled_checks:
      - "email_spam_outbreak"
      - "perf_memory"

Email AV

CSM scans email attachments in real-time using ClamAV and YARA-X on the Exim mail spool.

How It Works

  1. fanotify watches the Exim spool directory for new messages, including every split_spool_directory hash subdirectory (the cPanel default layout); hash directories Exim creates later are picked up within a minute
  2. Attachments are extracted and scanned by ClamAV (socket) and YARA-X (if available) Base64 and quoted-printable decoding tolerates stray characters, missing padding and malformed escapes. Encoded multipart bodies are decoded at every level. Ambiguous base64 is scanned under quartet and alphabet-only interpretations, and a valid padded prefix is scanned separately when data follows it. At most 16 additional interpretations are scanned per message. If decoding or multipart parsing reports an error, recovered content and later attachments are still scanned and the message is reported as incompletely scanned. Extraction size and MIME nesting limits still apply
  3. Zip and tar.gz attachments are unpacked within configured size and file limits
  4. Attachments are unpacked in a separate locked-down process with no network access, one per concurrent scan (scan_concurrency), started on first use. Extracted parts reach the scanners in memory and are never written to disk; each scan holds at most max_extraction_size of them. A process that times out, runs out of memory, crashes or sends an invalid reply makes the message an incomplete scan. Outside observe mode, CSM requests deferral whatever fail_mode says. Deferral requires a fanotify permission event that is still holding delivery; notification-only mode cannot block it
  5. Attachment names written to logs and the UI use sanitized, shortened base names. Archive detection uses the full name, so shortening a displayed name does not prevent unpacking
  6. Infected messages raise a Critical finding; with quarantine_infected: true they are also quarantined with full metadata instead of delivered
  7. Sender, recipient, and message ID are logged

Quarantine and release preserve each queue file’s ownership, permissions, and modification time, including when the move crosses filesystems. Release returns the message to the queue directory it came from, including the hash directory Exim uses when split_spool_directory is on.

Release accepts the flat queue root or the message’s hash directory, named for the sixth character in both 16- and 23-character Exim IDs. Other subdirectories and hash-directory symlinks outside the trusted queue roots are rejected.

Quarantine persists recovery metadata before moving spool files and syncs file contents and directory changes. A failed rollback retains the remaining quarantine files. Release restores the body before publishing the queue header; storage failures retain metadata and report the affected paths. Inspect both the spool and quarantine locations before retrying a partially completed operation.

Web UI

The Email page (/email) shows:

  • AV watcher status (active, engine health)
  • Scan statistics (scanned, infected, quarantined)
  • Quarantined email list with release/delete actions

API Endpoints

GET  /api/v1/email/stats         Scan statistics
GET  /api/v1/email/quarantine    Quarantined email list
GET  /api/v1/email/av/status     AV watcher status
POST /api/v1/email/quarantine/   Release or delete quarantined email
  • email_content - scans outbound email body for credentials and suspicious URLs; every base64 body and part, attachments included, is decoded with the same tolerance as attachment scanning, using live MIME headers and the declared multipart boundaries
  • email_weak_password - detects email accounts with weak passwords
  • email_forwarder_audit - audits forwarders for exfiltration redirects
  • mail_queue - alerts on queue buildup (spam outbreak indicator)
  • mail_per_account - per-account sending volume spikes

Email password audit

The deep scan checks mailbox passwords against account-derived candidates and the bundled weak-password list. Verification runs inside CSM; passwords and stored hashes are never passed to subprocesses or included in findings. Confirmed matches can still use the HIBP range API for breach counts, sending only a SHA1 prefix.

Supported stored formats follow Dovecot password schemes:

SchemeAccepted format and audit limit
CRYPT, SHA512-CRYPT, SHA256-CRYPTSHA crypt with a nonempty salt of up to 16 characters; 1,000 to 1,000,000 rounds, or the standard 5,000 when omitted
CRYPT, MD5-CRYPTMD5 crypt with a nonempty salt of up to 8 characters
CRYPT, BLF-CRYPTbcrypt 2a, 2b, or 2y; cost 4 through 14
ARGON2I, ARGON2IDVersion 19; at most 64 MiB memory, 4 passes, and 4 lanes; salt 8 to 64 bytes, digest 16 to 64 bytes
PLAINPlaintext
PLAIN-MD5, LDAP-MD5, SMD5MD5 digests; salted variants allow 1 to 64 salt bytes
SHA, SHA1, SSHA, SHA256, SSHA256, SHA512, SSHA512SHA digests; salted variants allow 1 to 64 salt bytes

Unprefixed hashes use CRYPT. cPanel shadow files, which add password aging fields after the hash, are read the same way. Scheme names and .hex, .b64, and .base64 encoding suffixes are case-insensitive. Unsalted digests also accept Dovecot’s hex/base64 autodetection. DES crypt, bcrypt 2x, yescrypt, PBKDF2, and mechanism-specific formats such as SCRAM are not audited.

Stored values are limited to 4 KiB and candidates to 256 bytes without embedded NULs. At most three hash calculations run concurrently. Cancellation returns promptly; a calculation already running keeps its worker slot until its bounded work finishes.

An unsupported, malformed, or over-budget hash produces email_password_audit_incomplete. CSM keeps earlier password findings and does not record that mailbox as successfully audited. Other mailboxes continue to be checked. When these are the only failures, the audit waits for password_check_interval_min before trying again; temporary verification failures remain eligible for retry on the next deep scan. Cycles skipped by the interval preserve both earlier password findings and the audit warning. A completed audit replaces them with its current results. Upgrading starts a fresh audit even when an older CSM version recorded the same hash.

Inbound spam filtering

CSM can filter inbound mail through rspamd on cPanel hosts, where each cPanel account decides for itself whether its mail is filtered, and on Postfix hosts, where every local domain is filtered. It is off by default.

The operator installs rspamd. CSM configures it, wires it into Exim on cPanel, keeps the integration current and reports when filtering is not working. CSM is never in the live mail path: Exim or Postfix talks to rspamd directly, and stopping CSM leaves the installed filtering in place.

Most of this page describes cPanel hosts. Postfix hosts lists what differs there.

What each account gets

The switch is cPanel’s own Spam Filters page, “Process New Emails and Mark them as Spam” (the ~/.spamassassinenable file). New accounts start opted in. Who may change the switch is controlled by the spamassassin feature in cPanel feature lists; CSM adds no setting for it.

With dry_run: false, inbound mail is scored by rspamd and Exim applies the account’s choice. For an opted-in account, Exim:

  • rejects it with 550 when the score is at or above reject_score;
  • adds X-Spam-Status: Yes when the score is at or above tag_score and below reject_score. If the user turned on cPanel’s spam box, the mail is filed in the Spam folder; otherwise it is delivered normally with the header. When the forward guard enforces spam_flagged, a copy forwarded to an outside address is held instead of relayed.

Mail for an opted-out account is delivered with no CSM rejection and no spam verdict header. Opting out never affects antivirus, outbound checks, SMTP-time controls such as RBLs, rate limits and anti-spoofing, or CSM’s detections.

A change to the switch applies from the next recipient Exim accepts. CSM does nothing and Exim does not reload.

The Spam Filters page’s own score threshold and allow and deny lists are not used: reject_score and tag_score apply to every opted-in account.

Who decides for a message

Policy follows the account that owns the addressed domain, including aliases, forwarders and catch-alls. It is not re-evaluated for the mailbox a forwarder finally delivers to:

  • a forwarder in an opted-out account delivers unfiltered into an opted-in mailbox it points at;
  • an address in an opted-in account can reject mail its forwarder would have passed to an opted-out mailbox.

When one message has recipients with different choices, Exim accepts the first recipient and answers 452 for each later recipient whose choice differs. The sending server retries those recipients in a separate delivery, which follows their own choice.

If /etc/global_spamassassin_enable exists, cPanel treats every account as opted in and no account can opt out. CSM reports this as a warning.

Mail that is never scanned

Only unauthenticated mail from remote hosts to the host’s local domains is scanned. Authenticated submissions, local mail and these peers are not: 127.0.0.0/8, ::1, trustedmailhosts and cpanel_mail_netblocks, the same set cPanel exempts from its own spam scan.

cPanel also accepts some recipients before the scan hook runs, so they are never filtered: mail to a domain from that domain’s remote MX hosts, hosts in skipsmtpcheck_hosts, always-relay and pop-before-SMTP hosts, Mailman -bounces addresses and suspended recipients.

Before you enable it

This section is for cPanel hosts. Postfix hosts have their own requirements.

Install rspamd from the upstream rspamd.com repository and keep its systemd unit’s User= setting (the upstream package runs as _rspamd); CSM checks the scan socket’s owner against it. CSM does not install rspamd or Redis. A first install waits for the conditions below; a definite failure after installation removes owned integration content when safe.

  1. SpamAssassin is off. Disable spamd in WHM so /etc/spamddisable exists. Otherwise cPanel’s own scan uses CSM’s scanner and adds its own headers and subject tag whatever the account chose.

  2. A scan socket for Exim. Add a Unix socket to rspamd’s normal worker, in /etc/rspamd/local.d/worker-normal.inc:

    bind_socket = "/run/rspamd/csm-normal.sock mode=0666";
    

    This replaces the normal worker’s TCP listener. The socket and every parent directory must be owned by root or the rspamd user. Parent directories must not be writable by group or others, so a hosting account cannot replace the listener. Extra ACL entries on the socket or any parent directory block installation. Exim’s runtime user must be able to search every parent directory and write to the socket; mode=0666 grants socket write access.

  3. rspamd accepts every message Exim accepts. rspamd’s default max_message (50 MiB) equals cPanel’s default Exim limit, and Exim adds headers before scanning. Set max_message to Exim’s message_size_limit plus at least 1 MiB in /etc/rspamd/local.d/options.inc, for example max_message = 60mb; with cPanel’s 50M default.

  4. Validate and reload rspamd (rspamadm configtest, then systemctl reload rspamd).

  5. Exim is running. CSM probes an Exim listener before and after each change.

CSM also refuses to install while any of these hold, and names the cause:

  • Exim settings, ACLs or filters that already act on spam scores, such as deny_spam_score_over_*, ratelimit_spam_score_over_* or automatic spam deletion and forwarded-spam restrictions;
  • another spamd_address in /etc/exim.conf.local;
  • cutthrough delivery, PRDR or CHUNKING enabled;
  • an Exim main log other than the one CSM reads (log_file_path);
  • an Exim or rspamd unit that cannot reload;
  • disabled custom ACL hooks or invalid named lists used by the hooks;
  • unsafe mutation targets or malformed CSM markers in the locations CSM manages;
  • an rspamd override that defeats CSM’s settings below.

CSM does not change WHM options, account toggles, Redis or rspamd worker settings to satisfy these conditions.

Configuration

email_protection:
  antispam:
    enabled: false
    dry_run: true
    rspamd_socket: /run/rspamd/csm-normal.sock
    reject_score: 15
    tag_score: 6
    feeds:
      enabled: []
      spamhaus_dqs_key_file: ""
    exempt: []
  • enabled turns the integration on. Setting it back to false requests removal of CSM’s owned content; conflicts or failures can prevent removal and produce findings. On a host that is neither a cPanel host running Exim nor a Postfix host it stays inactive; the daemon logs a warning at startup and whenever a configuration change is reloaded. Once such a host runs Postfix, for example after Postfix was installed or started after CSM, the next reload (systemctl reload csm) starts antispam without a restart.
  • dry_run scans and logs what would happen without rejecting, tagging or splitting recipients. On a Postfix host CSM’s rspamd files already apply in dry run, and the host’s own action thresholds decide on CSM’s scores; see Dry run and enforce.
  • rspamd_socket is the scan socket above: an absolute path of at most 107 characters using letters, digits, /, ., _ and -.
  • reject_score and tag_score are whole rspamd scores with 0 < tag_score < reject_score <= 1000.
  • feeds selects optional blocklists; see Feeds.
  • exempt lists recipient domains and mailboxes that are never filtered on a Postfix host, such as example.com or postmaster@example.net, and whose Junk folders are never learned from. Entries are lower-cased, and only plain domains and mailboxes are accepted. cPanel hosts ignore it: there each account’s Spam Filters switch decides. See Which mail is filtered.

All of these are hot-reloadable: after antispam starts, systemctl reload csm or a Settings save updates health tracking and requests a reconcile pass before forward-guard reconciliation. Settings saved before antispam starts are picked up when it initializes, after the startup forward-guard reconciliation. enabled: true is refused in observe mode.

rspamd_socket is not the reputation.rspamd controller URL. That optional setting reads rspamd’s scan history for IP reputation and is unrelated to filtering.

The capabilities API lists mail.antispam.v1 when a build supports this integration, and mail.antispam.postfix.v1 when it also supports Postfix hosts; neither means filtering is installed on the host. mail.antispam.learning.v1 means the build can learn from Spam folders, and mail.antispam.learning.postfix.v1 that it can also learn from Junk folders on Postfix hosts; while learning runs, csm status --json carries an antispam_learning block.

Feeds

Feeds and map redirects need rspamd 4 or later. On an older or unreadable rspamd version CSM leaves rspamd’s own lists and maps alone and reports it.

Every feed is off until you name it:

email_protection:
  antispam:
    feeds:
      enabled: [psbl, mailspike]
      spamhaus_dqs_key_file: ""   # only for spamhaus_dqs

enabled is a list of feed names; an unknown or repeated name, or spamhaus_dqs without a clean absolute spamhaus_dqs_key_file, makes the configuration invalid. Check each list’s terms against your use before you enable it:

NameTerms
mailspikeFree for sites under 100,000 messages and 100,000 DNS queries a day; contact Mailspike above that.
psblFree for anyone.
uceprotect1Free for private and commercial use; above 100,000 requests a day UCEPROTECT asks for its rsync feed. Level 1 also lists some forwarders that break SPF.
nordspamFree for commercial and non-commercial use; contact Nordspam above 10,000 queries a day.
semFree; for scoring only, never to reject mail on its own. Tell Spam Eating Monkey before going above 100,000 queries a day.
spamcopFree; SpamCop calls its list aggressive and recommends tagging, not rejecting.
barracudaFree after registering the addresses of the resolvers that query it with Barracuda.
dnswlFree through DNS under 100,000 queries a day; its pages conflict on non-commercial use. A listing lowers the score slightly.
uriblFree for low-volume use only; URIBL blocks public and large shared resolvers.
surblFree only for organisations under 1,000 users or 250,000 messages a day, and never inside a product or service sold for a fee; other use needs SURBL’s paid data service. See the SURBL usage policy.
spamhaus_dqsNeeds a DQS key. The free key covers non-commercial use under 100,000 queries a day; commercial use needs a paid key.
csm_ip_mapLocal data only: CSM’s block history, its threat feeds and the spam evidence it records.
rspamd_mapsKeeps rspamd’s own map downloads from maps.rspamd.com; free use is non-commercial and under 5,000 queries a day, other use needs an agreement with rspamd. See rspamd’s usage policy.

Maps without any feed

Without rspamd_maps, CSM redirects the rspamd.com maps it manages to local files: the copies the rspamd package installs in /etc/rspamd/maps.d, refreshed by package updates. The freemail and disposable-address rules stay quiet until you put your own lists in /etc/rspamd/local.d/maps.d/freemail.map and /etc/rspamd/local.d/maps.d/disposable.map. Phishing exceptions read /etc/rspamd/local.d/maps.d/phishing_whitelist.inc when it exists.

This does not block network access to rspamd.com. Operator-owned files and additional maps can keep downloads active; CSM reports those it finds in enabled modules after the integration is installed and current.

DNS feeds need a local resolver

Some blocklists refuse queries from public resolvers. CSM enables a DNS feed only when every nameserver rspamd uses is loopback, private, or inside the top-level infra_ips. rspamd uses dns.nameserver from its options, or /etc/resolv.conf when that is unset.

When your own resolvers sit on public addresses and refuse other networks, list their addresses or CIDRs in infra_ips; hostnames do not count for this check. infra_ips changes need a CSM restart, not a reload. Otherwise run a recursive resolver on the host and add this line to /etc/rspamd/local.d/options.inc:

dns { nameserver = ["127.0.0.1"]; }

cPanel’s local BIND usually recurses for localhost; cPanel’s PowerDNS does not. When rspamd uses the systemd-resolved stub (127.0.0.53 or 127.0.0.54 on port 53), CSM asks systemd-resolved for every server it can forward to, including its fallback servers when no other server is configured, and judges those. If CSM cannot read that list, DNS feeds stay off with a warning. Other ports on the stub addresses count as separate local resolvers. When PowerDNS holds port 53, run the resolver on another port and name it with that port, for example dns { nameserver = ["127.0.0.1:5353"]; }. A loopback or private address passes CSM’s address check, but a resolver that forwards to a public resolver can still be refused. CSM never edits options.inc. After changing it, run rspamadm configtest, reload rspamd, then reload CSM to request a check.

Resolver lists accept master-slave:, round-robin:, random:, and hash: prefixes without regard to case. Rspamd 4.2.2 does not support sequential:; CSM leaves DNS feeds off for that setting. A warning about an unreadable /etc/resolv.conf applies only when no explicit nameserver setting exists; invalid explicit settings get their own warning.

Spamhaus DQS

Put the key alone in a root-owned file, mode 600, and name the file in spamhaus_dqs_key_file. CSM reads it on every check, never logs it, and writes it only into /etc/rspamd/override.d/rbl.conf, mode 640, owned by root and rspamd’s group. CSM must know that group: set User= (and optionally Group=) in the rspamd unit. A root-started rspamd that changes user itself leaves the group unknown, and the feed stays off.

rspamd writes the name of every failed lookup, which contains the key, to its log. CSM keeps spamhaus_dqs off, with a feed warning, while that log file can be read by other users, which is the rspamd package’s default. Make it private, for example chmod 750 /var/log/rspamd and chmod 640 on the files in it, and set rspamd’s log rotation to create new files with mode 640. The DQS rules also run without rspamd’s own list monitoring. CSM probes ZEN’s test points and reports failures without stopping lookups. Turn the feed off when a critical warning reports listings for a never-listed test point.

The log and its directories must be owned by root or rspamd, with no extra ACLs or write access for other users. A private file or enclosing directory must restrict readers to root and rspamd, including group access. Symlinks and paths with dot components cannot establish privacy. CSM cannot verify syslog or console routing; use logging.type = file with a private log. The worker rechecks log privacy and rspamd version before applying feeds.

CSM IP map

csm_ip_map gives rspamd CSM’s own evidence about sender addresses. CSM rebuilds the map at start and every 10 minutes into /var/lib/csm-rspamd/csm_ip.map, and rspamd scores listed senders through one rule, CSM_IP:

SymbolListed whenScore
CSM_IP_DROPthe address is in a Spamhaus DROP network from CSM’s threat feeds4
CSM_IP_BLOCKEDCSM blocked the address permanently3
CSM_IP_MAIL_AUTHthe address attacked SMTP or mail logins in the last 30 days3
CSM_IP_SPAMits spam evidence score is 50 or more, see Spam evidence3
CSM_IP_BLOCKLISTDEblocklist.de lists it in CSM’s threat feeds2
CSM_IP_CINSCINS Army lists it in CSM’s threat feeds2

Each address or network gets one line with its strongest symbol, so the map adds at most 4 to a message. Whitelisted addresses (operator, temporary and reputation.whitelist), infra_ips, private, loopback and link-local addresses and the server’s own addresses are never listed, and neither is a network that contains one of them.

IPv4-mapped IPv6 addresses and CIDRs in infra_ips protect the same senders as their IPv4 forms. Infrastructure hostnames use the last successful DNS answers from CSM’s firewall resolver. This requires a running CSM firewall; without it, use literal addresses or CIDRs. DNS changes affect the map on its next rebuild and spam evidence on the next scan record. Entries only in firewall.infra_ips grant firewall access without suppressing spam evidence or map listings.

The feed stays off with a warning until the map exists as a regular root-owned file, mode 0640, in rspamd’s service group, without extra ACLs. Every parent must be a root-owned directory with no group or other write access or extra ACLs, and rspamd must be able to traverse it. Symlinks cannot establish safe map access.

CSM must know rspamd’s group, as for DQS above. The worker rechecks map access and rspamd version before applying a queued request. A failed check leaves the installed configuration intact. The next regular check retries with the current host facts.

The feed needs CSM to own /etc/rspamd/local.d/multimap.conf, even with rspamd_maps selected. CSM replaces an operator-owned file there, see Replaced files; one that is not safe to replace leaves the feed off. Like the other feeds, it needs rspamd 4 or later.

CSM checks that rspamd uses the current map by scanning a canary message from 192.0.2.255 every 30 seconds while the rule is in use. The scan stays out of rspamd’s history and statistics. antispam_ip_map reports a map CSM could not write, or two consecutive probes showing rspamd still not using the current map after its 5-minute load grace; a matching answer clears it. Earlier probe failures are forgotten when the map is removed, monitoring is disabled, observe mode is entered, or the scan socket changes. Dismissing the warning acknowledges the current outage; recovery clears the dismissal so a later failure can notify again. A failed write keeps the previous map, which rspamd keeps using. A missing map, a changed file mode or a changed rspamd group makes CSM write the map again even when its content is the same. Once no installed rule reads the map, CSM deletes it; a failed removal raises antispam_apply_failed and is retried on the next pass. If removal fails and a later apply reuses the surviving map, CSM resumes monitoring it and subtracts its score from new spam evidence.

Spam evidence

While antispam is enabled, CSM reads each scan result Exim logs and records a verdict about the sending address: spam when the score reaches the reject score, ham when it stays below the tag score, and nothing in between or when the scan failed. The score the CSM IP map gave the sender is subtracted first, so the map never feeds its own evidence. Dry run and enforce record the same way, and so do recipients with filtering off: the verdict is about the sender. Whitelisted, infra_ips, private and the server’s own addresses are skipped. Postfix hosts record no verdicts; see Scan records. This works with or without csm_ip_map, so you can review the data before turning the map on. After a CSM restart with csm_ip_map enabled, verdicts are not recorded until CSM has written the map again, because the map’s share of a score is unknown until then.

A spam verdict creates or updates the sender’s record on the Threat page. Ham only updates a record that already exists, so busy legitimate senders do not fill the attack database. The spam score is the share of spam among recent verdicts, each verdict losing half its weight every 7 days; two recent spam verdicts and no ham reach 50, which lists the sender as CSM_IP_SPAM. Spam evidence never blocks an address, never changes its threat score and is never reported to AbuseIPDB.

CSM forgets a record that holds only spam evidence once its spam and ham verdicts have both lost nearly all their weight: about 47 days after a single spam message, later for a sender with more verdicts. Ham on the record keeps it, so the next spam from a mostly clean sender does not list it. A record that also has attack findings, a block or a mail login attack in the last 30 days is kept. Every record leaves 90 days after its last spam verdict or finding; ham does not extend that limit. Stored evidence without a decay timestamp keeps its weight, as it does when scoring the sender. CSM checks at start and with every deep scan, hourly by default. The spam evidence budget can drop a record that holds only spam evidence sooner.

An IPv6 sender is recorded and listed per /64 network, for example 2001:db8:1:2::/64, so a sender that rotates addresses builds one score. IPv4 senders, including IPv4-mapped IPv6 addresses, stay per address. Clearing or whitelisting an address inside the network forgets the network’s evidence, and the CSM IP map does not list a network that holds an excluded address. Evidence recorded per IPv6 address by an earlier release stays until it expires or the spam evidence budget drops it.

With spam in reputation.report.classes, a sender that rspamd keeps rejecting, and that has sent no clean mail recently, is reported to the central abuse database. Senders of the shared mail services are never reported. Dry-run nodes count their verdicts too. See Abuse Reporting.

The Threat page’s Spam column shows spam and ham counts and the spam score, with a badge once the score reaches the listing threshold of 50. The badge does not confirm a listing: the feed may be off, and the map still skips excluded addresses. The “Spam evidence only” filter lists senders by spam score. The IP lookup shows mail login attacks and spam verdicts.

Spam evidence budget

The spam evidence budget is 50,000 records, about 22 MB of memory for records held only for spam evidence. It is fixed and has no setting. Each time CSM records a verdict, it keeps those records at or under that budget, counted at the verdict’s observation time. A missing time uses the current time. Invalid addresses and ham from an untracked sender are ignored and do not trigger the budget.

Startup loads unexpired evidence without trimming it to the budget. The count can therefore exceed the budget until a verdict is recorded. It can also grow as login evidence ages: a record with a mail login attack counts toward the budget only when that attack is more than 30 days old at the verdict’s time and the record has no attack findings or block. Records with attack findings, a block or still-protected login evidence never count toward the budget and are never dropped for it. A delayed verdict can leave more than 50,000 such records at the current time because it counts them at its own earlier time.

When a verdict is recorded and the count is over the budget, CSM drops records until nine tenths of the budget remain, rounded down. Each record has a level: the weight its verdicts still have, rounded up to whole messages, and at least 2 once it has two or more verdicts. Records of the lowest level go first. Inside a level, records in the network that holds the most spam-only records go first, counted per IPv6 /48 and per IPv4 /16 at the start of the pass. When those counts tie, the record with the least weight goes first.

A flood of single messages from one network, one /48 or /16, loses its own records first once that network holds more spam-only records than any other. A flood spread over many networks competes with the other records of the same level: the most crowded networks go first, then the records with the least remaining weight when network counts tie, whoever sent them. A flood of single messages cannot remove a sender with two or more verdicts, a listed sender or a sender’s ham history while records with two or more verdicts fill less than nine tenths of the budget; above that, one pass trims the weakest of them once. Higher levels are reached only after every lower-level record has been dropped; they can still be dropped when they alone exceed the trim target. A flood that sends as many messages from each address or /64 as a level competes with every record of that level, listed senders included. Senders that share a crowded /16 or /48 with a flood can lose records of the same level with it.

CSM writes a warning to the daemon log at the first drop, then at most once per day of recorded verdict time while drops continue. It also queues the warning for alerts; a full queue can refuse it. The warning gives the number of records dropped, names the most crowded network and says how many records that network held. The count in a later warning includes everything dropped since the previous one. Drops made after a warning are reported by the next warning. It comes with a later drop or with a deep scan, once a day has passed since the previous warning. A restart forgets drops not yet reported. Evicted records are removed from storage on the next successful flush; they can return after a crash before that flush. The API threat statistics include the total dropped since the daemon started.

Look at the named network and its count. The count is how many records holding only spam evidence the network had when the drop began, not how many of the dropped records were its own. The count alone does not show whether the network sent the flood. Inspect its senders; if the network is hostile, block it in the firewall. Records with attack findings, a block or a mail login attack in the last 30 days are never dropped by this budget. Records with two or more verdicts are dropped only once such records fill nine tenths of the budget.

Scores

No feed symbol weighs more than 5, and the symbols of one feed add up to at most 14, so no feed alone reaches the default reject score of 15. Each symbol counts once per message. SURBL’s four scoring categories weigh 3.5 each; Spamhaus DQS gives 5 for ZEN, 3 and 2 for the DBL categories and 3 and 1 for newly seen domains. URIBL’s grey and red lists weigh 0.5 because they also list domains that legitimate bulk senders use.

Updated feed weights apply on the next successful reconcile while antispam is enabled and observe mode is off.

Files

CSM owns these files while a feed or map redirect needs them:

  • /etc/rspamd/override.d/rbl.conf and /etc/rspamd/override.d/rbl_group.conf, with a DNS feed. They replace rspamd’s own lists and their weights, so only the feeds you named run;
  • forged_recipients.conf, mid.conf, mime_types.conf, whitelist.conf and multimap.conf in /etc/rspamd/local.d, without rspamd_maps. The local IP feed also needs multimap.conf when rspamd_maps is selected.

CSM replaces an operator-owned file at one of these locations while it needs the file, see Replaced files. A file that is not safe to replace makes CSM omit the DNS feeds or the affected map redirect instead. An operator file CSM does not need stays, and its contents can still conflict with required settings and block core filtering. In particular, an override.d/rbl.conf that enables the RBL module defeats CSM’s attempt to turn it off when no DNS feeds are usable. Resolve that conflict before enabling antispam.

Feed findings

Each antispam_feed finding names the feed, rspamd_maps for maps, or rbl for loaded DNS-rule checks:

  • feed off: the reason is in the details, such as a public resolver, a key file with the wrong owner or mode, or a file CSM cannot own. Fix it; the feed returns on the next check.
  • map redirect skipped: CSM could not manage that optional file. Check the operator’s map settings; they may already use local copies. Move those settings out of CSM’s target file if CSM should manage the redirect.
  • rspamd.com paths still loaded: rspamd still fetches those maps from rspamd.com. Redirect the named maps yourself, or enable rspamd_maps only if your use meets rspamd.com’s terms.
  • loaded rules differ during a routine check: rspamd runs a different set of blocklist rules from those CSM wrote, usually through an operator override. This is a warning only; CSM leaves the installed filtering in place. Correct the overrides, validate and reload rspamd, then reload CSM.
  • IP-map rule differs or cannot be verified: with csm_ip_map enabled, CSM checks that multimap is enabled and that CSM_IP uses the managed map, expected score and complete symbol set. Symbol order does not matter. Extra rule options that can change matching or force an action are conflicts. Failed configuration dumps also produce a warning. Routine checks leave core filtering installed; correct the override or command failure before retrying an apply.
  • refuses CSM’s queries: the list answered with an error code, such as a refused resolver, a query limit or a rejected key. Those error-code answers add no score.
  • does not answer: three checks in a row got no answer through rspamd’s resolver. Check that the resolver recurses and can reach the list.
  • unexpected answer: the test point returned an unrecognised answer. Check the provider and resolver; only the configured listing codes can score. A listed test point returning both listing and refusal codes also raises this warning; it does not prove that the list stopped scoring.
  • answers for addresses it never lists (Critical): turn the feed off and reload CSM. CSM’s probes only report findings; they do not disable feeds. A listing code still triggers this warning when the reply also contains a refusal code, since rspamd can score the listing code independently. rspamd stops querying a list only after its own monitoring detects it. DQS monitoring is disabled, so its lookups continue until you turn it off.

An apply that fails at feeds_check is reported as antispam_apply_failed and attempts to roll back. CSM omits DNS feeds and csm_ip_map from subsequent applies until the enabled feed names or DQS key-file path changes, or CSM restarts, and does not rewrite the IP map meanwhile. Fix the override before retrying. Reloading unchanged settings or replacing the key’s contents does not clear this hold.

CSM probes only while antispam is enabled outside observe mode and the installed integration is current without blocking preconditions. It checks two test names per supported zone through rspamd’s first nameserver, on the port rspamd uses, looking up A records. It does not probe SEM’s IPv6 zone or Spamhaus DBL and ZRD. A healthy probe result does not establish the health of those zones or of other configured resolvers.

What CSM writes

  • three Exim ACL hooks under /usr/local/cpanel/etc/exim/acls (custom_begin_spam_scan_check, custom_begin_spam_scan and custom_begin_check_message_post), inside # CSM-ANTISPAM BEGIN and # CSM-ANTISPAM END markers;
  • one spamd_address line in the @CONFIG@ section of /etc/exim.conf.local;
  • actions.conf, greylist.conf, rbl.conf, emails.conf, asn.conf, fuzzy_check.conf, phishing.conf and regexp.conf in /etc/rspamd/local.d, each starting with # csm-managed-version: 1. Greylisting, the emails module, sender-IP lookups to rspamd.com, phishtank URL lookups to rspamd.com and public fuzzy storage are off. DNS block lists stay off unless a DNS feed is enabled. regexp.conf holds the CSM_INVISIBLE_CJK_SUBJECT rule: a Subject with an invisible character, such as a zero-width space, next to a Chinese, Japanese or Korean letter. It checks the decoded Subject, including MIME-encoded ones, and covers kana, Han and Hangul letters with their half-width and supplementary forms. Punctuation does not trigger it. Spam uses invisible characters to split words so filters miss them. On its own the rule lifts a message above the default tag_score but not to the default reject_score. Keep your own regexp rules in /etc/rspamd/override.d/regexp.conf;
  • the map redirect files and, with a DNS feed, the blocklist files listed under Feeds;
  • groups.conf in /etc/rspamd/local.d, which lowers rspamd’s HIDDEN_TEXT weight from 5 to 2. Newsletters hide their preview line, and at the default tag_score that rule plus one minor symbol tagged them. It also counts URL_VERY_LONG once per message instead of once per long link, so newsletters full of tracking links are not tagged for that. It raises DMARC_POLICY_REJECT from 2 to 3, so forged mail from a domain whose DMARC policy is reject reaches the tag score sooner. A symbol you define in any *_group.conf in /etc/rspamd/local.d keeps your weight: CSM leaves it out of its file. If one of those files cannot be read or uses includes, macros or multiline strings, CSM leaves all three weights to rspamd. Your override.d group files already take precedence. CSM replaces an operator-owned groups.conf. One that is not safe to replace is left alone: rspamd keeps its own weights and CSM reports an rspamd_scores warning;
  • with learning on, the learning files listed under Learning from Spam folders;
  • a completion record in /var/lib/csm/antispam.

Content outside CSM’s markers, including forward-guard blocks, is preserved. CSM regenerates its own blocks and files, so manual edits to them are overwritten. A file of the same name without CSM’s header is replaced, see Replaced files; removal leaves it alone.

The loaded phishtank_enabled setting must be the boolean false. Missing, duplicate or non-boolean values are configuration conflicts, just like true. CSM reports them and removes its integration when safe. Failed commands or malformed command output leave the integration in place for a later check.

Each change runs as one transaction in a short-lived root helper started by systemd outside CSM’s sandbox, under the same lock as the forward guard. It rebuilds the Exim configuration, checks it, reloads Exim and rspamd only when needed, and restores the previous state if a step fails. If systemd-run is not available, CSM changes nothing.

CSM checks the integration at startup, after each config reload and every 15 minutes, and repairs drift, such as a cPanel rebuild that dropped the hooks. After fixing a condition that blocked it, run systemctl reload csm to request another check, even when csm.yaml is unchanged. A check already running finishes first; the reload does not wait for it. When another CSM change holds the lock, it tries again 20 seconds later.

Before each check, CSM cancels antispam changes still waiting to run from an earlier check or an earlier daemon, also while antispam is disabled or in observe mode. A change that is already running finishes first.

If cancelling fails, CSM raises a warning and retries: after 20 seconds while antispam is enabled, at the next regular check while it is disabled.

Existing Exim SMTP sessions retain their old hooks; new sessions use the updated hooks. Account toggles are still read at each recipient. Reloading rspamd can change scoring for scans from either generation of Exim hooks.

Replaced files

CSM owns its rspamd files and the Roundcube include outright. When antispam is enabled and one of those paths holds your own file, CSM copies it to /var/lib/csm/antispam/replaced before writing its own. Each copy is readable by root only. Its name is the UTC time followed by the escaped original path, for example 20261101T120000Z_%2Fetc%2Frspamd%2Flocal.d%2Factions.conf.

CSM replaces only a regular file owned by root, with no group or other write access and no extra ACL entries, in directories protected the same way. Any other file at one of these paths blocks the install, or leaves the optional feature that needs it off. CSM repeats these checks before copying the file and before replacing it. The Exim hook files and /etc/exim.conf.local are shared: CSM edits its own block there and never replaces them.

Each retained copy raises one antispam_file_replaced warning naming the file and the copy in the same check that runs the update. Move any settings you still need into a file CSM does not manage, then delete the copy to clear the warning.

When a change fails, CSM restores your file with its original owner, group and permissions. It deletes that copy only after the restore is durable, even if another file could not be restored. Copies made for replacements that never happened are removed too. If a restore cannot be confirmed, its copy remains for recovery.

On a cPanel host, disabling antispam deletes CSM’s files and does not restore the copies. On a Postfix host, disabling antispam puts the newest copy of each file back over CSM’s own file and deletes that copy, and returning to dry run does the same for actions.conf, see Dry run and enforce. Older copies of the same file stay, and CSM never restores over a file it did not write. Restores require root-owned backups with mode 0600 in a root-only directory. A symlink, unsafe backup or unsafe destination permissions stop the transaction before mail configuration changes. Correct the permissions and retry. If a backup changes before completion, CSM rolls back and keeps that edit. Restores and their rollbacks never follow directory symlinks. A restored file gets the owner and mode checked before the restore, which after a takeover are your original ones. /etc/rspamd/override.d/rbl.conf written while a DQS key is set is the exception: CSM’s file is owned by root, in rspamd’s group, with mode 0640, and the restored file keeps that. Each warning says what CSM will do with its copy. The warnings describe past replacements, including after antispam is disabled.

Rolling it out

  1. Meet the conditions above and enable with dry_run: true.
  2. Leave dry run on for at least 7 days. Review each would-reject decision in the Exim main log (CSM-ANTISPAM event=scan ... decision=reject), and watch for scan-error findings. Do not enforce until all predicted rejects are classified, none rejects legitimate mail, and no scan or configuration failure finding remains.
  3. Review which accounts are opted out today, and decide whether users may use the spam box (WHM Tweak Settings, skipspambox). That setting only shows or hides the choice: accounts that already have ~/.spamassassinboxenable get tagged mail in their Spam folder either way, and turning the spam box off in WHM deletes that file from every account.
  4. Set dry_run: false. Check every 550 for a week. Return to dry run on a confirmed false positive.

To roll back, set enabled: false while CSM is in enforce mode. Successful removal reloads Exim, and new sessions no longer use CSM filtering. Check for removal failures before assuming rollback completed. Observe mode leaves the installed integration untouched.

Sessions opened before removal retain their old hooks and may still scan. Those scans use operator or default rspamd settings after CSM’s settings are removed; scores can change and external lists can become active again.

Postfix hosts

On a host without cPanel that runs Postfix, CSM manages rspamd only. Postfix already passes mail to rspamd’s proxy worker as a milter, and rspamd applies the decision. CSM never edits Postfix’s configuration. The sections above about cPanel accounts, Exim and Roundcube do not apply.

Requirements

  1. Postfix passes mail to rspamd’s proxy worker. smtpd_milters must name the address of an rspamd proxy worker that has milter = yes. CSM checks this read-only with postconf and rspamd’s worker settings. When smtpd runs chrooted, a unix: milter path is resolved inside Postfix’s queue directory, as Postfix does. Without this wiring CSM reports a definite postfix_milter precondition and installs nothing. milter_mail_macros must also include {auth_authen}, with the braces, as Postfix’s default does: rspamd knows authenticated mail by it. Without it CSM reports the same precondition, because customers’ outbound mail would be filtered as remote mail. CSM reads both settings from main.cf only; per-service -o smtpd_milters overrides in master.cf are not read.

  2. rspamd refuses the mail it rejects. With discard_on_reject on the proxy worker, rspamd’s milter drops a message it rejects; with quarantine_on_reject it accepts the message into Postfix’s hold queue, where it stays until someone releases or deletes it. Either way the sender is never refused. CSM does not enforce while either is set on a proxy worker Postfix passes mail to, and removes an enforcing install, with a definite milter_rejects precondition. In dry run, where the host’s own rejects go the same way, it is a warning. An option given more than once counts as set unless every value is false, and so does any value that is not a boolean, such as null. Set both to false, once each, for example in /etc/rspamd/local.d/worker-proxy.inc.

  3. rspamd 4.2 is recommended. CSM accepts rspamd 3 or later, but its scope rule is tested with rspamd 4.2.3.

  4. /etc/rspamd/lua.local.d exists, owned by root and not writable by group or others. CSM’s scope rule goes there.

  5. A scan socket for CSM. CSM uses it to check that rspamd answers. Add a Unix socket to rspamd’s normal worker in /etc/rspamd/local.d/worker-normal.inc:

    bind_socket = "localhost:11333";
    bind_socket = "/run/rspamd/csm-normal.sock mode=0600";
    

    Only CSM uses this socket, as root, so mode 0600 is enough. Keep the first line if your proxy worker passes mail on to the normal worker instead of scanning it itself. The socket and every parent directory must be owned by root or the rspamd user and not writable by group or others. Run rspamadm configtest, then systemctl reload rspamd.

  6. rspamd logs to a file at info level. CSM reads its scan records from rspamd’s log file, for example /var/log/rspamd/rspamd.log. When rspamd logs to the journal or syslog, CSM warns and sees no scan records. info is rspamd’s default. At notice, warning or error level rspamd does not write them, and CSM does not warn; set level = "info"; in /etc/rspamd/local.d/logging.inc.

  7. rspamd accepts every message Postfix accepts. CSM warns when rspamd’s max_message is below Postfix’s message_size_limit.

  8. Postfix and rspamd are running.

Which mail is filtered

All unauthenticated mail from remote hosts is filtered, whatever domain it is addressed to. That includes mail Postfix relays for hosts in mynetworks on public addresses; list those in infra_ips. CSM never rejects or tags mail:

  • from a sender who authenticated to Postfix;
  • from rspamd’s local networks (local_addrs), which include loopback and the private address ranges by default;
  • from an address in infra_ips.

Mail whose every recipient is in exempt is scored and logged, but never rejected or tagged. A domain entry matches that domain only, not its subdomains. One recipient outside the list is enough: the whole message is then filtered.

Your own rspamd settings rules, for example in /etc/rspamd/local.d/settings.conf, keep working. For mail CSM does not filter, CSM only raises the action thresholds on top of what your rule set. A rule that limits which symbols run (symbols_enabled or groups_enabled) also skips CSM’s scope rule: mail it matches keeps the normal thresholds, even when it is authenticated, local or exempt, and gets no scan record.

Dry run and enforce

A dry run on Postfix already changes how rspamd scores mail. rspamd is in the mail path, and CSM installs its scoring files from the first dry run, so it can predict its own decisions. From then on:

  • greylisting, the emails module and public fuzzy storage are off;
  • rspamd’s own DNS block lists are off, unless you name DNS feeds in feeds.enabled;
  • sender-IP and phishtank lookups to rspamd.com are off, and without the rspamd_maps feed the rspamd.com maps CSM manages are read from local copies, so the freemail and disposable-address lists stay empty until you fill them, see Maps without any feed;
  • CSM’s symbol weights and its invisible-character Subject rule apply;
  • CSM’s milter_headers.conf replaces yours, so other header routines you enabled there stop;
  • your own files at those paths are replaced, with a copy kept and an antispam_file_replaced warning, see Replaced files.

What CSM leaves to the host in dry run is actions.conf: rspamd keeps rejecting and tagging at the host’s own thresholds, now applied to CSM’s scores, for all mail, including authenticated, local and exempt mail. Mail can therefore score differently than before CSM, in both directions. CSM logs what it would have done with each scored message: decision=reject, tag or none, and unfiltered for exempt mail.

When the host’s own reject threshold is at or below reject_score, as with rspamd’s built-in actions when there is no actions.conf, the dry run still rejects mail. CSM raises a dry_run_rejects precondition warning, for example “dry run on this host still rejects: rspamd rejects at 15, at or below CSM’s reject score of 15”.

With dry_run: false CSM takes over /etc/rspamd/local.d/actions.conf. An existing file is copied first, see Replaced files. rspamd then rejects at reject_score and adds headers at tag_score. CSM’s thresholds never reject or tag authenticated, local, infra_ips or exempt mail; other rspamd modules, such as antivirus, can still reject it.

Going back to dry run puts your actions.conf back from its newest copy, deletes that copy and clears its warning. The file gets the owner and mode CSM’s file had, which after a takeover are your original ones, or root ownership and mode 0644 when CSM’s file was loosened. Your thresholds decide again. Without a copy CSM deletes its own file, and rspamd’s built-in actions apply, which reject too.

Headers and the Junk folder

CSM owns /etc/rspamd/local.d/milter_headers.conf. rspamd removes any X-Spam-Status header a message arrives with and adds its own: X-Spam-Status: Yes, score=... at the add header threshold, No below it. For authenticated and local mail, which rspamd’s milter headers skip, CSM’s scope rule removes the inbound copy itself. In dry run the header follows the host’s own add header threshold.

rspamd also adds its own Authentication-Results header with the SPF, DKIM, DMARC and ARC results, and removes any copy the message arrived with, since a sender could forge one. Authenticated and local mail get no new Authentication-Results header and keep any existing copies, so rspamd does not add login names to outbound mail. Their inbound X-Spam-Status is still removed. Other header routines, such as X-Spam-Level or rspamd’s extended headers, are off.

CSM leaves the X-Spam: Yes header of rspamd’s proxy worker alone. Filing tagged mail into the Junk folder stays your Sieve rule, for example one that files X-Spam: Yes into Junk. CSM does not manage Sieve.

Scan records

CSM’s scope rule writes one CSM-ANTISPAM event=scan line per scored message into rspamd’s log, at info level: rspamd logging at notice or above hides them. Authenticated, local and infra_ips mail get none. These records count toward scan health and the Email Security page only. They do not feed spam evidence, the CSM IP map, the Threat page or abuse reports: any local process can make rspamd log such a line through its local scan and milter listeners.

Failed scans come from Postfix’s mail log: Postfix logs a warning each time its milter call to rspamd fails, and CSM raises antispam_scan_error. What happens to that mail depends on Postfix’s milter_default_action: accept delivers it unfiltered, tempfail defers it, reject rejects it and quarantine holds it in Postfix’s hold queue.

Files and learning

CSM writes the rspamd files and the completion record listed under What CSM writes, actions.conf only while it enforces, plus milter_headers.conf in /etc/rspamd/local.d and csm_scope.lua in /etc/rspamd/lua.local.d. It writes no Exim or Roundcube file. Each change reloads rspamd only. Disabling antispam removes these files and puts back the newest copy of each file CSM replaced, see Replaced files.

Learning installs the same rspamd files as on cPanel, without the webmail include; see Learning from Junk folders.

If CSM cannot tell which mail server runs, for example while Postfix restarts on a host that also has Exim installed, it leaves an installed integration in place and reports a temporary platform precondition.

Learning from Junk folders

With learning.enabled: true CSM learns from mail users move into their Dovecot Junk folder. Scores, limits, retention, stored messages, operator commands and findings work as described under Learning from Spam folders, with these differences:

  • Every mailbox Dovecot lists takes part. There is no per-mailbox switch and no opt-in file. Mailboxes and domains in exempt are not watched: CSM does not read their Junk folders.
  • CSM reads the new and cur directories of each Junk folder. A report must be owned by the uid Dovecot stores that mailbox’s mail as.
  • Mail already in a Junk folder is never learned. The first pass records what is there, and only messages moved in later are reports.
  • No webmail Junk-button include is installed.

Learning requirements

The learning prerequisites apply, and also:

  1. Dovecot 2.3. CSM lists users with doveadm user '*' and reads mail settings with doveconf. A userdb must be able to list its users. A passwd or passwd-file userdb can, but not a passwd-file whose path holds % variables, such as one file per domain. An SQL or LDAP userdb needs its iterate settings. A static, checkpassword or prefetch userdb cannot. When no userdb can list users, and on Dovecot 2.4, whose mail settings changed, CSM reports unsupported and leaves learning off. When a userdb that can list users fails, for example while its SQL database is down, CSM reports backend_unavailable: installed learning stays and new reports wait for a later check. A temporary failure looking up an individual user also pauses discovery, without cancelling reservations or pruning state. Unusable or case-colliding user names pause discovery because their identities and mail owners cannot be established safely.

  2. Junk marked as \Junk. CSM finds Junk folders by special_use = \Junk in Dovecot’s namespace mailbox settings, as Dovecot’s example 15-mailboxes.conf sets for Junk. A single configured mailbox is supported too. A folder named Junk without the flag is not watched, and with no \Junk mailbox at all every mailbox is skipped.

  3. Maildir in its default layout. CSM skips a mailbox stored another way: another driver such as mdbox or mbox, LAYOUT=fs, or the zlib or mail_crypt plugin. It also skips a mailbox whose uid is 0, whose home or mail location it cannot use, such as a location with DIRNAME, MAILBOXDIR or a %{if} condition, or whose lookup confirms the user does not exist. A %{if} in mail_location or a private namespace location skips every mailbox; CSM still reads their mail owners for isolation checks. Known owners remain part of those checks even when their mailboxes are skipped for another reason. Two users whose mail directories are the same, including a symlink alias, or one inside the other, are both skipped: CSM cannot tell their moves apart. Private namespaces without Junk also count. An unresolved mail location holds other mailboxes sharing its uid until discovery can distinguish them. Skipped mailboxes are counted by reason in one mailboxes_skipped warning that names none of them; learning goes on for the others.

  4. A Redis of its own. Learning never uses rspamd’s Redis on Postfix, because other rspamd modules share it. Run a second Redis instance and set learning.redis_socket to its socket; without it CSM reports unsupported. On Debian and Ubuntu the packaged redis-server@.service template runs an instance from /etc/redis/redis-<name>.conf. For an instance named csm, create its data directory:

    install -d -o redis -g redis -m 0750 /var/lib/redis/csm
    

    Then write /etc/redis/redis-csm.conf:

    port 0
    unixsocket /run/redis-csm/redis.sock
    unixsocketperm 770
    pidfile /run/redis-csm/redis-server.pid
    dir /var/lib/redis/csm
    maxmemory 256mb
    maxmemory-policy noeviction
    appendonly yes
    

    and start it with systemctl enable --now redis-server@csm. The template’s sandbox lets Redis write below /var/lib/redis, /var/log/redis, its run directory and, for cluster mode, /etc/redis; systemctl cat redis-server@csm shows the exact list on your system. dir must be in one of them for persistence to work. Use the data directory created above, not /var/lib/redis itself, where the main instance keeps its files. Size maxmemory for the hashes you expect to learn.

    Give rspamd’s user access through the socket’s group, for example with usermod -aG redis _rspamd; no other user may be in that group. Then restart rspamd: a reload keeps the groups it started with, and its fuzzy worker would be refused by the socket. The Redis checks of the prerequisites apply to this socket. rspamd’s global redis settings must not use sentinels, TLS or a named user, because rspamd would apply them to the learning Redis too.

  5. A private controller socket for CSM, as on cPanel; see rspamd admin interface.

  6. A mail user that is not a service user. Users’ Sieve scripts and Dovecot’s mail processes run as the uid Dovecot stores mail as, such as vmail, so CSM treats every name of that uid as a user that can run tenant code: none may be in the Redis socket’s group. Mail stored as rspamd’s or Redis’s uid is refused with unsafe_prerequisite.

CSM checks these on every reconciliation pass while learning is requested. The learner and its Junk folder watcher start only once none of them fails.

Reporters

learning.reporter decides who casts one learning vote:

  • domain, the default: each mail domain is one reporter. Its mailboxes share one vote per cluster and one learning allowance. A domain that reaches its allowance pauses only its own new reports.
  • mailbox: each mailbox is a reporter of its own, as each cPanel account is.

A user without a domain, such as a system user, reports under its own uid in either mode.

Changing the mode, or exempt, restarts the Junk folder watcher. Votes already cast stay until their cluster expires, so a cluster reported both before and after a mode change can count the same domain or mailbox one extra time. Reports the old reporters had not sent yet are dropped, and the Junk folders of domain mailboxes start new baselines.

Enabling learning on Postfix

  1. Set up the Redis instance and the private controller socket above, then run rspamadm configtest and restart rspamd. A reload is not enough after adding rspamd’s user to the socket’s group.
  2. Set learning.enabled: true and learning.redis_socket, then run csm validate and systemctl reload csm.
  3. Check antispam_learning findings and csm doctor. Once learning runs, csm status --json carries an antispam_learning block.
  4. Move a test message into a Junk folder. The next pass, within five minutes, picks it up.

Rolling it out on Postfix

  1. Meet the requirements above. Choose feeds.enabled first: the dry run already turns off rspamd’s own DNS block lists and greylisting. Enable with dry_run: true. Leave exempt empty unless a recipient must never be filtered.
  2. Check that rspamd’s loaded actions are still the host’s (rspamadm configdump actions) and that no dry_run_rejects or milter_rejects warning appears. Review each antispam_file_replaced copy, and check that greylisting and DNS block lists are as you intend. Then check that CSM-ANTISPAM lines appear in rspamd’s log and that mail at the host’s add header threshold still carries X-Spam: Yes.
  3. Leave dry run on for at least 7 days. Review each would-reject decision in rspamd’s log (CSM-ANTISPAM event=scan ... decision=reject) and watch for scan-error findings.
  4. Set dry_run: false. Check every reject for a week. On a confirmed false positive, return to dry run: CSM puts your own actions.conf back.

To roll back, set enabled: false. CSM removes its rspamd files, puts back the files it replaced and reloads rspamd.

Stalwart hosts

On a host whose mail server is Stalwart, CSM does not install or manage rspamd and puts nothing in the mail path. Stalwart’s own spam filter and its own scores decide what happens to each message. csm stalwart setup makes Stalwart log the mail its filter rejects or discards and reports whether the filter is on and at which scores it acts; csm doctor checks the same and whether the filter’s DNS blocklists answer. Neither switches the filter on or off or changes the scores at which it files mail to Junk, rejects or discards it. Setup also adds a spam rule that adds up to 4 to the spam score of senders on CSM’s address list, so more of their mail goes to Junk, or is rejected or discarded at the scores set in Stalwart. It is the one setup step that changes what happens to mail; switch the rule CSM_IP off in Stalwart to stop it. Setup keeps existing tags and reports scores only for scoring tags; it lists other tags as non-scoring. See Preparing Stalwart.

Filter verdicts

CSM reads Stalwart’s log and counts one verdict per message on the Email page: delivered, Junk, rejected or discarded. A message to several recipients that went to Junk for any of them counts as Junk. Delivered and Junk messages are counted two minutes after CSM reads them, so the page lags by that much. A message with no local delivery by then, because it was relayed, deferred or failed, is not counted. Mail from logged-in senders, local mail between mailboxes included, is not counted, and neither is mail Stalwart logs without a client address. When more messages wait than CSM can hold, as under a flood, the extra ones count as “not matched”. If a held queue ID names conflicting messages, CSM counts that ambiguous join once as “not matched” and records no sender evidence. Repeated copies of the same queued line neither add a verdict nor extend the hold.

Delivered counts every delivered message, also while the filter is off. It also counts mail a mailbox’s own Sieve rule filed into a folder, Junk included, because Stalwart logs such mail as not spam. Rejected and discarded show a dash until Stalwart has logged one.

Spam evidence on Stalwart

Each counted message, except one not matched, also records a verdict about the sending address, as on cPanel (see Spam evidence): Junk, rejected and discarded count as spam, delivered as ham. Junk is spam at Stalwart’s Junk score, not at a CSM score. The same addresses are skipped: whitelisted, infra_ips, private and the server’s own. IPv6 senders are recorded and listed per /64 network, and clearing or whitelisting an address inside forgets the network’s evidence. Spam evidence never blocks an address, never changes its threat score and is never reported. It does not need email_protection.antispam. The spam evidence budget applies here too.

A sender on CSM’s address list gets no verdict unless CSM sees that the rule CSM_IP is off. The rule raised the score of its mail, so Stalwart’s verdict on it says more about the list than about the sender. The list that counts is the one CSM builds now, not what Stalwart holds. While Stalwart’s spam filter is off, no verdict is recorded at all. The Email page counts these messages as usual.

CSM’s address list on Stalwart

CSM keeps its address list in Stalwart’s lookup lists, where the rule CSM_IP reads it. The list holds the same addresses as the CSM IP map, from the same sources and with the same exceptions. CSM checks it at start and every 10 minutes, writes only what changed, and then has Stalwart load it. If loading fails twice, CSM keeps retrying on later checks, even when no keys need changing.

The lookup namespaces whose names start with csm-ip belong to CSM. CSM deletes any key there that it did not add, so edits made there in Stalwart are undone within 10 minutes. To stop the scoring, switch the rule CSM_IP off in Stalwart: CSM then empties these namespaces. A missing rule or a rule whose type is not Any also empties them. CSM fills them again once the rule exists, has type Any, and is enabled. In observe mode CSM reads Stalwart but writes nothing, so the namespaces keep what they hold, even with the rule off. An unfinished reload keeps its warning in observe mode.

stalwart_spam_filter_off warns while Stalwart’s spam filter is off, since Stalwart then scores no mail. A failed filter read leaves an existing warning in place until CSM can confirm the filter is on. stalwart_address_list warns when CSM cannot inspect or read its key file, read the rule or list, write the list, or have Stalwart load it. When Stalwart refuses CSM’s key, its message says to run csm stalwart setup. These warnings clear once recovery is confirmed.

CSM’s key needs more access to keep the list, so run csm stalwart setup again after upgrading. Without a key file CSM writes nothing to Stalwart and raises neither finding; setup and csm doctor report the missing key. An unreadable key file instead raises stalwart_address_list.

Not done on Stalwart

Apart from the rule for its address list, CSM does not change Stalwart’s scores or rules, set up its DNS blocklists or check outgoing mail.

rspamd admin interface

rspamd’s controller answers admin and statistics requests. By default it trusts every connection from 127.0.0.1 and ::1 without a password, and every process on the server can connect there, including customer scripts. Such a process could read rspamd’s history, which lists senders and recipients of scanned mail, or change scores and actions at runtime.

While antispam is enabled, CSM reports each controller listener that other users may reach as an antispam_controller_open finding. The finding names the listener and the reason, including cases where CSM cannot prove isolation:

  • a trusted address that local processes can use (loopback, a wildcard, or one of the server’s own addresses);
  • a trust entry CSM cannot resolve, such as a map file or a hostname;
  • no controller password at all, only an enable password, or an empty password;
  • a controller configuration CSM cannot read in full, such as an unreadable worker type, an unsupported enabled or count value, a systemd socket, or a second controller worker;
  • a Unix socket, or one of its parent directories, that other users can write to, or whose access CSM cannot verify.

For symlinked Unix sockets, a private directory protects only its own descendants. CSM also checks the resolved socket path: a private directory on a detour does not hide a socket reachable through its real path.

These warnings also update in observe mode and when an apply is busy. When rspamadm cannot run, CSM keeps the previous warnings until it can. Turning antispam off clears them even if CSM cannot cancel pending changes.

To close TCP access, keep a controller password set and put this in /etc/rspamd/override.d/worker-controller.inc:

secure_ip = [];

rspamd ships with the placeholder password q1, which it rejects as client authentication, so this also works without setting your own. With both password and enable_password unset, an empty secure_ip opens every command to everyone. Setting only enable_password leaves read-only access open.

For local administration add a private Unix socket in the same file. Direct Unix socket connections get admin access without a password. Forwarded headers trigger additional trust and password checks, so local clients must send no forwarded headers. Keep the socket and its parent directories owned by root or rspamd, with no extended ACLs. Use socket mode 0600 and non-symlink directories that other users cannot write to. Socket mode alone does not protect a replaceable parent:

bind_socket = "/run/rspamd/csm-controller.sock mode=0600";
bind_socket = "localhost:11334";
secure_ip = [];

Reload rspamd after the change. CSM never edits the controller settings.

The Email Security page reads rspamd’s counters through a Unix controller socket owned by rspamd, such as the private socket above, or else a loopback listener that secure_ip trusts, and shows “Stats unavailable” without either; CSM never uses the controller password.

Learning from Spam folders

email_protection.antispam.learning.enabled: true makes rspamd learn from mail users move to their Spam folder. When several accounts report the same content, later copies of it score higher. Learning is off by default and requires antispam to be enabled. Enabling it is refused in observe mode. This section describes cPanel hosts; for Postfix hosts see Learning from Junk folders.

Enabling learning is consent to two things: CSM reads new messages in the Spam folders of participating accounts, and it keeps one full message per learned cluster. Clusters normally retire 90 days after their last report, but pauses and cleanup failures can retain messages longer (see Stored messages).

How reports are found

Every five minutes CSM checks the main mailbox and each domain mailbox of every account whose Spam Filters switch is on, or of every account when /etc/global_spamassassin_enable exists. It looks at the new and cur directories of .spam, the folder spam-box delivery and the webmail Junk button use, and .Junk, which many mail clients use. Folders are read without following links; CSM never creates folders or changes permissions. The exception is a .spam or .Junk that links to another folder of the same mailbox, such as .spambox on some older cPanel accounts, by its bare name or its exact path under the account’s home. CSM reads that folder, and pointing the link elsewhere starts a fresh baseline.

The first pass over a folder records what is already there without reading it. Only messages that arrive after that are reports, so existing mail is never learned retroactively. Moving a message within the folder or changing its flags does not report it again. A pass reads at most 20 reports per account, takes turns between accounts, and stops before the next one is due; the rest wait for the next pass.

What counts as a report

A report is a regular file owned by the account, at most 10 MiB, that did not change while CSM read it. A message still linked from another folder, such as one delivery shared by several mailboxes, counts once it and its links have stayed unchanged for a full pass, so a move into or out of Spam can finish first. Messages are skipped when any X-Spam-Status header says Yes, when that header cannot be parsed, and when the header block is malformed or too large, linked or not. Only the text parts of a report are learned, attached text included; images and other attachments never are. A message whose MIME structure CSM cannot read is skipped as no_digest, for operator requests too.

Reports are not authenticated. Tenants control their own mail: they can move any message they received, or deliver one locally, and a forged X-Spam-Status: Yes header keeps a message from being learned. In dry run senders’ headers are not removed, so forged headers have this effect on any message.

Scores and limits

Each reporting account adds a fixed weight to the cluster. One, two, three and four accounts give CSM_FUZZY_SPAM scores of about 2.9, 5.4, 7.2 and 8.4, so with the default tag score of 6 a message needs reports from about three accounts to be tagged on that evidence alone. An operator mark from csm antispam learn --spam weighs as much as nine accounts. Copies in multiple Spam folders or mailboxes of the same account still contribute only once to that cluster.

Messages that share most of their text join the same cluster, so later variants of a campaign score too. A new message whose plain text and HTML versions differ starts its cluster from one of them, normally the larger; attached messages are never used. Later copies score through that version. Newsletters built from one template can also be close enough to join. Lowering or correcting a learned value removes and re-adds it, which can leave a short gap in which the message scores lower.

Learning writes scores with either value of antispam’s dry_run setting. In dry run they affect the measured score; what happens to the message still follows the dry-run policy.

Opt-out, disable and retention

Unless /etc/global_spamassassin_enable exists, turning an account’s Spam Filters switch off stops CSM reading its folders at the next check and drops its unsent reports, including reports the learner accepted but has not yet sent to rspamd. Each cluster of a report is handled on its own: a part whose update rspamd may already have received still completes. If the learner cannot cancel, CSM reports it and retries at the next check while the account stays opted out. Turning the switch back on starts new baselines. The global override prevents per-account opt-out, including learning. Removing a cPanel account drops its learner reports the same way, once its UID belongs to no remaining account. An account whose home directory CSM cannot use keeps them. While the account list is changing or cannot be read, every account keeps them and CSM retries at the next check. What the account already contributed stays until the cluster expires or an operator forgets it; opting out or removal does not delete it sooner.

Turning learning off keeps stored messages, keys and the learning ledger, but clears folder baselines and unsent reports; re-enabling starts new baselines. Observe mode keeps everything and resumes where it stopped.

Folders absent for 90 days are forgotten with their pending reports. Unresolved accounts, unreadable homes, incomplete mailbox lists and unsafe or unreadable folders keep their state until they can be checked again. Links in an account’s etc directory, such as cPanel’s Roundcube database link, are not mail domains: CSM does not follow them and they do not hold back this cleanup.

A backend error stops further submissions for that pass; the next pass starts with the accounts after the one that failed. Reports stay pending while rspamd’s fuzzy storage answers no lookup, however long that lasts. A failure counts against a message only when storage answered a lookup for another report earlier in the same pass, or when rspamd cannot process the message; such a message does not stop other accounts’ reports. When rspamd answers a message’s inspection with an error, CSM inspects a fixed test message: if storage answers it, the failure counts against the message and the pass goes on with the next report. If storage does not answer it, submissions stop and the report stays pending without a counted failure, even if storage answered earlier in the pass. A message with three counted failures, including unclassified learner errors, is skipped as failed. Canceled or expired passes, reconciliation pauses, unsent controller requests, a wrong controller peer and local state failures never count. Unreadable learning state holds reports even when the learner cannot finish startup; the reports resume once the state is available.

Prerequisites

While learning is requested, CSM checks these during reconciliation. Definite failures request removal of the learning configuration. Temporary probe failures keep an existing installation. Learning findings do not disable core filtering; antispam_learning names the reason.

  • rspamd 4.2.x and Redis 5.0.x, 6.2.x or 7.x; other versions are refused;
  • exactly one private controller Unix socket owned by rspamd, mode 0600, in directories only root and rspamd can write to, with no extended ACLs, and no other controller listener granting other users admin or read-only access (see rspamd admin interface);
  • rspamd’s control socket private to root and rspamd, with protected directories and no extended ACLs. Check the effective control_socket path and its permissions as well as the controller socket;
  • one Redis server for fuzzy storage, reached only through a Unix socket: port 0 and tls-port 0, a socket owned by root, rspamd or the redis service user, no access for other users or extended ACLs, and any group access limited to root, redis and rspamd’s service user. Service users must not be hosting accounts. Parent directories must protect the socket from replacement. Redis needs a positive maxmemory and maxmemory-policy noeviction. A Redis build without TLS needs no tls-port setting;
  • with learning.redis_socket set, that Redis instead of rspamd’s: CSM applies the checks above to it, refuses it while any other rspamd module or rspamd’s global Redis settings name the same socket, or while those global settings use sentinels, TLS or a named user, and points the fuzzy worker at it with no password. Learning moves to a new socket only after it passes these checks; until then it keeps its current Redis;
  • rspamd’s uid used by rspamd alone: no cPanel account, and on Postfix no user but the service users, may share it. If CSM cannot read every account’s uid, learning waits: an installation stays, and nothing new is published until a later pass can check;
  • rspamd’s log unreadable to other users, by file permissions or by CloudLinux CageFS. CageFS qualifies only for a file log under /var/log/rspamd, with every account confined and that directory absent from its skeleton;
  • no local fuzzy rule and no enabled fuzzy worker of your own. CSM’s files replace them, so CSM refuses instead. The stock rule for rspamd.com is replaced on purpose.

CSM sends the Redis password only to a socket it has verified, and only on a connection served by that socket’s owner.

A cPanel account record with no system user, such as cPanel’s own system record, does not count as an account sharing the Redis owner when /etc/passwd does not list it either. If /etc/passwd cannot be read in full, or an account lookup fails for another reason, CSM does not trust a non-root Redis owner. On Postfix these account read failures pause learning and retain its installed configuration. The worker rechecks Dovecot mail owners before using a dedicated Redis, including owners of skipped mailboxes.

Redis persistence is recommended. CSM warns only when both append-only persistence (appendonly yes) and snapshots (save) are disabled. While learning is active, CSM rebuilds missing learned values from stored messages.

What learning installs

  • fuzzy_check.conf and worker-fuzzy.inc in /etc/rspamd/override.d, mode 0640 for root and rspamd’s group: local fuzzy scoring and storage;
  • fuzzy_group.conf in /etc/rspamd/local.d: the CSM_FUZZY_SPAM score and its group cap;
  • csm_learning.lua in /etc/rspamd/lua.local.d: a controller plugin for message inspection;
  • /usr/local/cpanel/base/3rdparty/roundcube/config/inc.d/zz-csm-antispam.inc.php, which points Roundcube’s Junk button at the Spam folder. This changes the Junk button for every webmail user on the server, regardless of the account’s Spam Filters switch. CSM replaces an operator-owned file at that path, see Replaced files. The markasjunk plugin must already be enabled; CSM does not enable it.

CSM checks the include by loading Roundcube’s configuration with cPanel’s PHP as the nobody user, never as root. If Roundcube does not load the Junk settings, or the check fails, CSM restores the previous include and reports roundcube_unavailable. The rest of the learning configuration stays installed. Later passes omit CSM’s include until the learning configuration changes or CSM restarts. A later include can override the settings.

After writing these files CSM checks what rspamd loaded, including the effective CSM_FUZZY_SPAM weight the controller reports, which can differ from the files after a runtime weight change. A failed learning check attempts to roll back the change. Later passes omit learning until its configuration changes or CSM restarts. Fix the reported cause before restarting; reloading unchanged settings does not clear the hold.

On an otherwise current, completed installation, changing only learning reloads rspamd without rebuilding Exim. Changing only the Roundcube include reloads neither service. An incomplete installation or Exim configuration drift can still require an Exim rebuild.

Keys live in /var/lib/csm/antispam/learning/keys, root only: directories are mode 0700 and key files mode 0600. CSM creates a complete set when none exists. It refuses a partial or invalid set with state_mismatch, leaving learning off. Do not delete the surviving keys to start over: new keys cannot match hashes made with the old ones. The learning ledger is bound to the hash keys and hash settings. A changed generation stops learning instead of reusing the previous learning state.

The Redis capacity warning starts at 75% of maxmemory and clears below 70%. It leaves installed learning configuration in place and pauses the learner, including pending changes, until usage falls.

Turning learning off requests removal of CSM’s learning files and Roundcube include, while keeping the keys. Conflicts or transaction failures can prevent removal. Check findings before assuming learned scoring is gone. Disabling learning or entering observe mode stops native learning calls before configuration reconciliation. Calls already in flight are canceled and drained within their deadlines. Reconciliation pauses the learner until a completed pass confirms that learning can resume.

Stored messages

Each learned cluster keeps one message, the first one reported, so the learner can restore its value if Redis loses it. CSM keeps its headers and text parts, without images or other attachments, under /var/lib/csm/antispam/learning/anchors, readable only by root. A cluster started from one version of a message keeps only that version. CSM deletes it when the cluster retires, 90 days after its last report, or when an operator forgets it. Backups and file system snapshots can keep a copy longer. Messages saved by older versions keep their attachments until their cluster retires. If deleting a retained message fails, CSM still removes its retired score and reports the cleanup error. New reports wait until cleanup succeeds. Disabling learning, observe mode or unmet prerequisites can delay retirement and message deletion. Turning learning off keeps stored messages and state. rspamd can hash the same message differently from one check to the next, because the words it hashes depend on the language it detects. When a stored message no longer gives its cluster’s value, CSM retires that cluster, removes its score and logs a warning; other clusters and new reports are not affected. A failed inspection leaves the cluster intact. CSM can still publish the message if its text-only form reproduces the learned digest; otherwise publication waits for a later retry. Reports joined to a retired cluster contribute no vote; operator spam requests report skipped:superseded.

Retained messages consume their learning allowance until deletion succeeds. The record that prevents a retired score from returning remains charged, including when a new cluster retires before its first contribution completes.

CSM’s state archive (csm store export) leaves learning state out; csm backup includes it, see Backup, restore and a fresh namespace.

Teaching and correcting learning

Root can learn a saved message as spam, or remove what one taught:

csm antispam learn --spam <file>...
csm antispam learn --forget <file>...
csm antispam learn --status [<request-id>...]
csm antispam learn --fresh-namespace

The daemon must be running. --spam and --forget require learning enabled and refuse observe mode. Each file must be regular with one link, at most 10 MiB, and unchanged during reading. Symlinks anywhere in its path are refused. The snapshot is sent to the daemon as bytes. --spam gives each matching cluster one operator mark, which weighs as much as nine reporting accounts. --forget removes the clusters a file matches, with every account’s contribution to them, and deletes their stored message. Other learned messages may still score similar mail.

Each file gets a request ID, printed before it is sent. If printing fails, that file is not sent. A lost or truncated reply is retried once with the same ID and snapshot. Refusals and completed replies are not retried.

A file reported as pending was reserved; the daemon keeps publishing it in the background. A file reported as deferred changed nothing, because learning was busy or waiting on another change; run the command again later.

csm antispam learn --status <request-id> shows what the daemon recorded for a request, including clusters still publishing and their phase. Status only reads the ledger and works with learning disabled or in observe mode. Without an ID it lists the requests that are not finished. If the command is interrupted, look its ID up there, or in the action log with csm actions --op integrate.antispam_learn --json, which records the request ID, escaped basename, outcome and full digests, never message content or headers. A missing record does not prove that no change occurred. Running the command again makes a new request.

Forgetting works for the whole server: the clusters a file matches lose every account’s contribution. CSM keeps a zero-value record for each forgotten value and keeps auditing it, so a late write cannot bring the score back unnoticed. Forget is not a permanent allowlist; new reports of the same content start a new cluster.

Exit statusMeaning
0Every file was learned, already learned, forgotten or matched nothing; --status and --fresh-namespace succeeded.
1At least one file was refused, failed or is still pending, or the daemon could not complete the request.
2The command line is not valid.

Backup, restore and a fresh namespace

csm backup stores the learning state as one bundle: the ledger in the state database and /var/lib/csm/antispam/learning, which holds the hash keys, transport keys and stored messages. csm restore brings the bundle back and moves any learning directory it replaces to learning.old-<UTC time>-<suffix>. Installation failures trigger rollback, but rollback itself can fail. Cleanup errors can also occur after the new bundle is installed. After any restore error, check the reported paths and resolve it before restarting CSM.

Restored learning state holds new reports until the daemon has run with learning disabled outside observe mode; enable learning again only when the backup is the current bundle. CSM cannot tell an old backup from the current one by comparing scores.

If the backup is older, or storage was lost along with the keys, start a fresh namespace instead. With learning disabled, run:

csm antispam learn --fresh-namespace

It waits for an active reconciliation pass, moves /var/lib/csm/antispam/learning to learning.old-<UTC time> and empties the learning and Spam folder state. If emptying fails, CSM attempts to move the directory back and reports any failure to do so. Resolve that failure before enabling learning. After a successful reset, the next enable creates new keys and a new storage prefix, and every folder starts a new baseline. Old reports are not replayed. The old directory still holds full messages; remove it once you no longer need it. CSM never flushes Redis; old hashes stay under the old prefix until they expire.

Learning findings

Each antispam_learning finding names one reason. Each reconcile pass reports server readiness; the running learner adds the reasons from anchor_drift on. On Postfix the learner also reports a Dovecot listing that failed in its last pass, until a later pass reads it.

ReasonMeaning
unsupportedAn rspamd or Redis version is outside the supported versions above, a Postfix host has no learning.redis_socket, or Dovecot runs 2.4 or has no userdb that can list its users; learning is disabled.
unsafe_prerequisiteRequired isolation or Redis settings are unsafe or cannot be established; learning is disabled.
state_mismatchLearning keys could not be created or loaded safely, restored learning state awaits confirmation, or the learner found its state out of step with its storage; learning is disabled and nothing is adopted or deleted. The CSM log names the disagreeing fuzzy digests when the learner finds it.
config_conflictOperator configuration conflicts with CSM’s learning setup; learning is disabled.
apply_failedA learning check failed during apply; subsequent passes hold learning off until its configuration changes or CSM restarts.
backend_unavailableA probe failed temporarily, including a Dovecot listing that failed or timed out; existing learning configuration is retained unless a definite failure also requires removal. A first installation waits.
worker_absentThe installed fuzzy worker is not listening. Restart rspamd after fixing the cause; CSM leaves the configuration installed.
redis_capacityRedis has reached the capacity guard above; CSM leaves the configuration installed.
redis_persistenceRedis has neither append-only persistence nor snapshots enabled; CSM warns without disabling learning.
roundcube_unavailableThe Junk-button include could not be installed, verified or removed cleanly. The other learning files are unaffected.
removal_incompleterspamd still loads CSM’s learned scoring while learning is disabled; resolve the reported conflict or removal failure.
anchor_driftA learned value moved away from its target more than once. CSM keeps correcting it; repeated drift means something else writes to the learning storage.
mutation_uncertainA write to the learning storage got no answer. CSM reads the stored value before it sends anything else.
anchor_body_invalidA stored message is missing or changed. That cluster’s value is held until it is forgotten.
state_ioLearning state could not be written, for example on a full disk. Nothing unrecorded is sent and installed scores stay.
audit_overdueThe check of every learned value has not finished a round within 24 hours.
request_errorsLearning requests to rspamd failed three times in a row.
backend_rebuildingLearned values are missing from storage, for example after Redis lost its data. CSM restores them from the stored messages and holds new clusters until it is done.
mailboxes_skippedOn Postfix, Dovecot mailboxes CSM does not watch, counted by reason; see Learning requirements. Learning goes on for the others.
account_capacityAn account, or the operator, reached its learning allowance, or an account’s Spam folder state reached its limit of 65536 messages. Its new reports are skipped or wait; existing learning stays. Confirmed absent folders are forgotten after 90 days, which frees their share.

Findings

CheckSeverityMeaning
antispam_preconditionWarningA condition above is not met. The message names the cause, such as rspamd_socket or spamassassin_enabled. A temporary cause keeps an installed integration; a definite one removes it when that is safe.
antispam_apply_failedWarning, or Critical when the previous state could not be confirmedAn install, update or removal failed at the named step, or pending changes could not be canceled.
antispam_file_replacedWarningCSM replaced your own file at one of its paths and kept a copy. Deleting the copy clears it, and so does CSM putting the copy back on a Postfix host. See Replaced files.
antispam_rspamd_downWarningThree probes of the scan socket in a row failed. CSM probes every 20 seconds while antispam is enabled.
antispam_rspamd_recoveredWarningTwo probes in a row succeeded after an outage.
antispam_scan_errorWarningThe mail server logged a failed scan. One finding covers every failure until a scan succeeds.
antispam_controller_openWarningOther users may reach rspamd’s admin or read-only interface, or CSM cannot verify isolation. See rspamd admin interface.
antispam_feedWarning, or Critical when a list answers for everythingA feed or map redirect is off, loaded rules differ, a list misbehaves, or rspamd still fetches maps from rspamd.com. These findings alone do not disable filtering. See Feeds.
antispam_ip_mapWarningThe IP map could not be written, or rspamd has not confirmed the current map 5 minutes after a write. Core filtering is unaffected. See CSM IP map.
antispam_learningWarningOne finding per learning readiness or removal problem. Core filtering is unaffected. See Learning findings.
stalwart_spam_filter_offWarningStalwart’s spam filter is off, so Stalwart scores no mail. See CSM’s address list on Stalwart.
stalwart_address_listWarningCSM could not inspect or read its key file, read its rule or list, update the list, or have Stalwart load it. Run csm stalwart setup when the message says so. See CSM’s address list on Stalwart.
spam_evidence_evictedWarningThe spam evidence budget was reached and CSM dropped the weakest records. The message names the most crowded network and how many records it held. See Spam evidence budget.

While rspamd is down or scans fail on a cPanel host, mail is delivered without spam filtering. Exim adds no rejection for a scanner failure, and other mail checks keep running. On a Postfix host, Postfix’s milter_default_action decides, see Scan records. Disabling antispam clears the down and scan-error findings without a recovery notice. Preconditions or removal failures can still be reported.

Scan records in the Exim main log include the connecting sender’s address as ip=. Scan health ignores records with a malformed non-empty address. Older records without that field, and records with an empty address, still count.

Dismissing a scan-error finding acknowledges the current episode. A successful scan clears that dismissal immediately, even between health probes. Disabling antispam or restarting CSM also resets it. Undismissed scan errors keep the normal daily notification deduplication so intermittent failures do not send one alert per message.

Rspamd outage and recovery notices do not consume the routine hourly alert budget. They honor notification suppressions and email check exclusions. Suppressed notices remain in history, the audit log and event streams.

In observe mode CSM changes no mail configuration; it only cancels changes still waiting to run. An installed integration is reported as observe_mode.

rspamd timeouts

When a scan reaches rspamd’s task_timeout, rspamd answers with the score it had so far. A slow rule that would have lowered the score is lost, so a timed-out scan can tag or reject mail a complete scan would pass. Exim cannot tell the difference; rspamd’s log records the timeout. If it shows timeouts, raise task_timeout or remove the slow rule.

Tested versions

cPanel 11.138 with cPanel’s Exim 4.100.1: on AlmaLinux 8 with rspamd 4.2.1 and 4.2.2, and on AlmaLinux 9 with rspamd 4.2.2. CSM needs Exim 4.97 or later with content scanning and rspamd 3 or later.

Threat Intelligence

CSM tracks, scores, and correlates attacks using a local attack database enriched with external feeds and GeoIP data.

cPanel service-subdomain alerts retain the logged address as unverified context, separate from per-IP reputation and attack history.

Attack Database

  • Per-IP event tracking (brute force, reconnaissance, reputation, authenticated activity)
  • Local scoring from attack volume, types and targeted accounts, used to rank addresses and in the unified verdict; the score never raises a finding or a block on its own
  • Top attackers leaderboard

Successful cPanel, FTP, webmail and PAM login audit events, and authenticated File Manager writes, remain in event history and account counts as auth_success (Authenticated Activity). They add no event-volume or multi-account score. Multi-IP login and authentication-failure signals keep their existing scoring.

file_upload remains readable for historical records; no current check produces it. Existing scores recorded under older classifications are not rewritten because aggregated login counts cannot distinguish ordinary logins from multi-IP alerts. For a confirmed false positive, follow Clearing a stale local threat score. That operation preserves historical events and does not remove firewall or permanent-blocklist entries; review those separately.

Mail evidence

The attack database keeps mail login attacks and spam/ham verdicts in a separate mail section. Equivalent IPv4 and IPv6 spellings share one record; invalid sender addresses are ignored. Spam and ham do not change the local threat score or add attack events. Ham only updates an existing record.

Spam and ham verdicts about an IPv6 sender are recorded per /64 network, not per address. A record that holds only spam evidence is dropped once that evidence has faded, or sooner to keep the 50,000-record spam evidence budget. The budget is checked when a verdict is recorded, using that verdict’s observation time; startup can load more records until then. See Spam evidence for the details.

Verdict weights decay from their observation times, including when older verdicts arrive after newer ones. A missing observation time uses the current time. Spam refreshes the record’s activity time. Remaining ham weight keeps a spam-only record until both spam and ham evidence have faded, but never past the 90-day limit since the last spam verdict or finding. The spam evidence budget can drop it sooner. Retained mail evidence survives a restart with its original counts and observation times.

IP Intelligence

Combines multiple sources into a unified verdict:

SourceData
Local attack DBEvent count, types, score
AbuseIPDBExternal reputation (if API key configured)
RspamdPer-IP rolling history (if controller access configured)
Upstream HTTP cachePanel-side shared score (if reputation.upstream configured)
Permanent blocklistOperator-managed persistent blocks
Free public feedsSpamhaus DROP, blocklist.de, CINS Army
Firewall stateCurrently blocked/allowed status
GeoIPCountry, city, ASN, ISP
RDAPNetwork name, organization (cached 24h)

Verdicts: clean, suspicious, malicious, blocked

Free public feeds

CSM loads cached Spamhaus DROP, blocklist.de, and CINS Army data at startup and refreshes it daily. A failed download keeps that feed’s previous data. These lists supplement local evidence without requiring an AbuseIPDB key.

Spamhaus merged EDROP into DROP, so CSM no longer fetches EDROP. An old <state_path>/threat_db/spamhaus-edrop.txt cache is ignored and can be deleted. Existing firewall blocks, whitelists and findings are unchanged.

Pluggable sources

Threat-intel sources implement a small Source interface (lookup-by-IP returning a score + reason). The aggregator queries every enabled source in parallel, applies per-source weighting, and produces the unified verdict above. Adding a new source means implementing the interface and registering it; no existing source code changes.

Currently shipped:

  • AbuseIPDB (reputation.abuseipdb_key) - external IP reputation feed. CSM caps uncached lookups per cycle and reserves store-backed daily quota before sending requests. While the quota is exhausted (daily budget or an API 429/402 backoff) CSM emits a reputation_quota_exhausted Warning so the degraded coverage is visible; a threat_feed_stale Warning fires when previously downloaded free threat feeds have not refreshed in over 7 days. Persistent conditions remain visible in Findings; alert email and the webhook send each one once, and history records a reminder at most once a day, including across daemon restarts; these coverage warnings never trigger reputation scoring or blocks.
  • Rspamd (reputation.rspamd.*) - per-IP rolling-history signals from the local rspamd controller. Delivered ham dilutes the score, temporary deferrals are neutral, and definitive spam actions count against the sender. Token resolution reads token_env from the process environment at query time. Changing the external environment requires a daemon restart; see credential rotation.
  • Upstream HTTP cache (reputation.upstream.*) - shared panel-side cache of AbuseIPDB or proprietary scores. Useful in fleets: agents pay a bounded local cache hit (cache_ttl_min, default 15 m) instead of hammering the upstream once per agent. CSM temporarily opens a fail-open circuit breaker after repeated upstream failures and lets only one cooldown probe through at a time. Use HTTPS for remote panels; plain HTTP is accepted only for loopback. Wire contract: Upstream threat-intel contract.

Verified crawlers

reputation.bot_verify_enabled verifies claimed crawler User-Agents with static IP ranges first, then strict forward-confirmed reverse DNS. reputation.verified_bots adds operator-defined crawler identities with name, ua_substrings, and one verification method: rdns_suffixes or ip_ranges. With rdns_suffixes the source IP must forward-confirm under a registrable domain (public suffixes and shared-hosting suffixes are rejected; a PTR-only match is not trusted). With ip_ranges the source IP must fall in one of the published CIDRs – this is for crawlers such as GPTBot and PerplexityBot that publish address ranges instead of crawler reverse DNS. Over-broad or non-public ranges are rejected. All checks run at config load and on reload.

Built-in rDNS verification covers Googlebot, Bingbot, Applebot, DuckDuckBot, Amazonbot, the Facebook and Meta crawlers, Brave, and the SERanking backlink bot. Googlebot, Bingbot, and Applebot also match a shipped IP-range snapshot first and fall back to reverse DNS; DuckDuckBot, Amazonbot, Facebook/Meta, Brave, and SERanking are rDNS-only.

Reverse-DNS verification is asynchronous. A newly admitted verification job receives a short, bounded pending window, including time spent waiting in the queue. High-volume traffic in that window can route to the proof-of-work challenge when it is enabled. A full queue, stopped worker, unsupported identity, or expired pending window uses the ordinary flood and scanner controls instead.

DNS failures, missing reverse DNS, and failed cache writes do not prove a spoofed identity. They leave verification unresolved, delay retries, and do not renew pending treatment on each retry. Attempt history is bounded. When it is full, new sources can replace completed entries after their initial cooldown; retries cannot extend that reservation. Live jobs and newly granted pending windows keep their history. If no entry can be replaced, admission is refused until capacity becomes available instead of running untracked lookups. Evicted or expired sources without a persisted missing-PTR record can receive another pending window, but the initial cooldown prevents continuous renewal under churn. Expiry preserves live jobs and their retry delay after completion. A missing PTR suppresses further DNS lookups for one fixed hour in the state database, including across restarts. Scans do not extend that hour. After it lapses, DNS retries resume, but the record remains as attempt history for a day, so a restart or in-memory eviction cannot grant fresh pending treatment. Records older than that are removed as new ones are written. Cleanup is attempted at most once an hour, including after a failed cleanup. A definitive verification result replaces that history. The record is not a verdict: it grants neither the verified-crawler exemption nor pending treatment, and it never counts as a spoof. csm store reset-bot-verify and a verified_bots change clear these records together with the cached results. A confirmed negative remains eligible for spoof detection. A cached positive receives the normal verified-crawler exemption.

GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot and ClaudeBot are recognized out of the box: their published IP ranges ship as an embedded snapshot and are refreshed from the vendor endpoints by an auto-updater (reputation.bot_ranges, default on, outbound HTTPS, configurable interval; restart required for setting changes). Fetched ranges are validated with the same over-broad and non-public guards as operator entries, and the embedded snapshot is the trusted fallback when a fetch fails. Anthropic publishes one combined feed for ClaudeBot, Claude-User and Claude-SearchBot and documents IP-list verification rather than reverse DNS, so CSM verifies ClaudeBot by address from that feed; the legacy anthropic.com reverse-DNS suffix is kept only as a fallback.

csm update-bot-ranges refreshes these ranges on demand (mirroring csm update-geoip): it fetches the vendor feeds, writes the on-disk snapshot, and asks the running daemon to apply them without a restart. The auto-updater and the manual command both export metrics – refresh success/failure, prefix count per crawler, and the last successful refresh time – under the csm_botranges_* names.

Abuse Reporting

reputation.report can send minimized confirmed-abuse reports to a central database or private collector. It is off by default. Remote targets must use HTTPS; plain HTTP is accepted only for loopback collectors. Keys and target wiring are read at daemon startup, so changes to this block require a restart. The central abuse database accepts ed25519 targets only; its report URL ends in /report and its scored-set URL in /decisions. The request format is in the Abuse Report Contract.

Reported findings

Each class covers these checks. Only Critical findings are reported, except bad_asn_egress and spam, which report High findings too.

ClassChecks
bruteforcepam_bruteforce, wp_login_bruteforce, xmlrpc_abuse, ftp_bruteforce, smtp_bruteforce, mail_bruteforce, admin_panel_bruteforce
credential_stuffingcredential_stuffing
php_relayemail_php_relay_abuse
bad_asn_egressbad_asn_outbound
spammail_spam_source

Nodes accept spam entries in downloaded central lists. With spam in reputation.report.classes, a node raises mail_spam_source for a sender that rspamd keeps rejecting and that has sent it no clean mail recently, and reports that sender. Dry-run antispam nodes count their verdicts too. The finding never blocks or challenges the sender on the reporting node.

Shared mail services are never reported: Google, Microsoft, Amazon SES, SendGrid, Mailgun, Brevo, Yahoo and iCloud. CSM reads their sending hosts from each provider’s published SPF records and refreshes the list in the background. Spam reports wait while that list cannot be loaded. When a refresh has failed and the list is not current, CSM raises a shared_senders_stale Warning. It stays in Findings until the list is current again. Alert email and the webhook send it once; history records a reminder at most once a day, including across daemon restarts.

CSM keeps sender history across restarts. If that history cannot be read at startup, spam source detection stays off for that run and the daemon log says why. A restart turns it back on once the state database can be read.

The class needs email_protection.antispam.enabled. Turning either on or off takes a restart.

A report carries only the source address, the class, a count and the time. List the classes to send in reputation.report.classes. An empty list keeps reporting off.

Set up the central abuse database

  1. On the node, generate a key pair:

    csm report enroll
    

    It prints a 64-byte private key as 128 hex characters and a 32-byte public key as 64 hex characters. Put the private key in the CSM environment file described in credential rotation, for example as CSM_REPORT_KEY. The private key is a secret: never paste it into a ticket or chat. Give only the public key to the central operator.

  2. The central operator approves the node and returns its node_id and key_id, the report and scored-set URLs, and the central public key. Add the central public key, encoded as 64 hex characters, to the same environment file, for example as CSM_CENTRAL_PUBKEY.

  3. Configure the target. The central block is optional; it is described below.

    reputation:
      report:
        enabled: true
        classes: [bruteforce, credential_stuffing, php_relay, bad_asn_egress]
        targets:
          - name: central
            url: https://abuse.example.net/report
            transport: ed25519
            node_id: "<node_id from the operator>"
            key_id: "<key_id from the operator>"
            key_env: CSM_REPORT_KEY
      central:
        enabled: true
        set_url: https://abuse.example.net/decisions
        pubkey_env: CSM_CENTRAL_PUBKEY
        auth_target: central
        action: challenge
    
  4. Restart CSM: systemctl restart csm.service.

reputation.central pulls the database’s signed list of scored addresses. With action: challenge, a qualifying local attacker finding for an unprotected listed address adds it to the challenge list. This requires challenge.enabled: true, an active firewall and a working challenge setup.

With action: block_if_local_corroborated, CSM attempts a block instead once the central score reaches reputation.central.block_threshold; below that score it still uses the challenge path. A live block also requires an active firewall, auto_response.enabled: true, auto_response.block_ips: true and auto_response.dry_run: false, and remains subject to the normal block protections. Dry-run records what would have been blocked. Central data alone never blocks. An ip_reputation match is outside intelligence too, so it never counts as the local finding for either action. action: off pulls the list without taking central-driven actions.

auth_target names the report target whose node key signs each download. The central abuse database refuses unsigned downloads. The target must use transport: ed25519, and its URL must share scheme, host and port with set_url, so a signed request never goes to another host. Leave auth_target unset only for a central service that serves the list without signatures.

Signed downloads work with reputation.report.enabled: false; keep the target configured even when the node does not send reports. csm validate refuses an auth_target that names no target or several, a target that is not Ed25519 or lacks node_id, key_id or key_env, or a set_url on another origin, and CSM will not start with that config. The settings page refuses the same changes. If the target’s key is unset or invalid, CSM starts with the central consumer off and logs one line naming reputation.central.auth_target. It never falls back to unsigned downloads while auth_target is set.

If the service refuses a download, CSM keeps its last verified list, tries again at the next refresh, and logs central-intel: pull unauthorized; check node enrollment, node clock and reputation.central.auth_target. Each successful refresh logs central-intel: refresh complete (version=N).

To rotate the node key, run csm report enroll again and have the central operator register the new public key. Then update key_id and the environment file, and restart CSM.

Send to a private collector

Use transport: hmac with a shared secret. You choose node_id and key_id; the collector uses them to find the secret. Set the named secret and optional token in CSM’s environment file and restart CSM. The secret is used as raw string bytes, not hex-decoded.

reputation:
  report:
    enabled: true
    classes: [bruteforce, credential_stuffing]
    targets:
      - name: collector
        url: https://collector.example.net/csm/report
        transport: hmac
        node_id: web-01
        key_id: k1
        key_env: CSM_COLLECTOR_SECRET
        token_env: CSM_COLLECTOR_TOKEN   # optional bearer token

The collector must implement the Abuse Report Contract. CSM queues one copy per target, so one node can report to the central database and a private collector at the same time. Give each target a unique, stable name: duplicate names select only the last usable target, and removing or renaming a target, or a key error that makes it unusable at startup, discards its queued reports on the next delivery pass. The spool_max capacity is shared across all targets, not reserved per collector. Overflow drops the oldest entries across that shared queue.

Check delivery

When reporting starts successfully, the log shows abuse-reporting: enabled for 1 target(s), 4 class(es) with your counts. A target with a missing field, a non-HTTPS remote URL, an unknown transport or an unusable key is skipped with a log line. Reporting stays off when no usable targets or valid classes remain, or when its spool cannot be opened. Failed deliveries are logged as reporting: delivery incomplete. The abuse_reporting.ingress and abuse_reporting.spool entries in protection queue health show waiting and lost reports.

Delivery and retries

Reports wait in a durable queue and are delivered once a minute. When a target is down, throttling, or refuses the node (for example before the node is approved, or when the node’s clock is wrong), that target’s reports stay queued in order and it is retried a minute later; other targets keep receiving theirs. A target that answers HTTP 429 with Retry-After, in seconds or as an HTTP date, is retried after that wait instead, up to an hour. Throttling while the target still accepts reports is not logged. If it keeps throttling without accepting any, the delay doubles on each attempt up to a minute, or the requested wait if longer, with a warning at most once a minute per target. A report the collector refuses as invalid (HTTP 422) would be refused again, so it is dropped and counted as lost.

Reporting and central intelligence actions share the address protection checks. These cover literal infrastructure IPs/CIDRs and crawler IP ranges, not infrastructure hostname resolution or rDNS crawler results. For NAT64 and 6to4 sources, protection of either the original IPv6 address or its embedded IPv4 address applies.

Web UI

The Threat Intel page (/threat) provides:

  • IP lookup with composite scoring
  • Two separate block actions per IP: a 24 hour block and a confirmed permanent block, singly or over a selection of attackers
  • Top attackers with GeoIP enrichment
  • Attack type breakdown chart
  • Hourly trend chart
  • Whitelist management (permanent and temporary)

A 24 hour block records threat evidence that expires with the firewall block, so a mistaken block of a customer address stops counting against it once the block lapses. A permanent block records evidence that stays until an operator clears it. When an address is no longer blocked but still holds permanent evidence, the lookup says so, because that address is flagged again the next time it is seen.

The 24 hour action refuses to shorten an existing permanent or longer firewall block. Unblock explicitly before changing that lifetime. Bulk requests skip those addresses and report warnings. Existing permanent threat evidence remains until cleared; feed updates do not remove a timed operator record before its expiry.

Bulk undo restores each prior firewall lifetime, using the original deadline for timed blocks. Expired blocks stay expired, and a later block, clear, or whitelist decision invalidates the older undo action. Only successful firewall reversals restore the corresponding threat evidence.

When CSM starts it drops the attack records of addresses inside the top-level infra_ips list, so an internal host listed after it was recorded stops ranking as an attacker. Only addresses and CIDRs count; hostnames and firewall.infra_ips do not. Event history is kept.

Unblock & Clear accepts private addresses for an internal host that is not in infra_ips. Clear and whitelist remove the attack record under every equivalent spelling of the address, such as an IPv4-mapped form. Whitelist and allow actions refuse private addresses and point to infra_ips instead.

API Endpoints

GET  /api/v1/threat/stats            Attack stats and type breakdown
GET  /api/v1/threat/top-attackers    Top attacking IPs with GeoIP
GET  /api/v1/threat/ip               IP threat lookup
GET  /api/v1/threat/events           IP event history
GET  /api/v1/threat/whitelist        Whitelisted IPs
GET  /api/v1/threat/db-stats         Attack database statistics
POST /api/v1/threat/block-ip         Block IP for 24 hours
POST /api/v1/threat/block-ip-permanent  Block IP with no expiry
POST /api/v1/threat/whitelist-ip     Permanent whitelist
POST /api/v1/threat/temp-whitelist-ip  Temporary whitelist
POST /api/v1/threat/clear-ip         Clear from attack DB
POST /api/v1/threat/unwhitelist-ip   Remove from whitelist

GeoIP

MaxMind GeoLite2 integration for IP geolocation and ASN enrichment.

Features

  • City database - country, city, latitude/longitude
  • ASN database - ISP, organization, autonomous system number
  • Auto-download on first use
  • Auto-update every 24 hours (configurable)
  • RDAP fallback for detailed ISP/org info (cached 24h)

Where It’s Used

  • Threat intel page (top attackers, IP lookup)
  • Firewall audit log (country flags)
  • Login alerts (geographic context)
  • Country-based login suppression (trusted_countries)
  • Country blocking (firewall CIDR ranges)

Configuration

geoip:
  account_id: "YOUR_MAXMIND_ACCOUNT_ID"
  license_key: "YOUR_MAXMIND_LICENSE_KEY"
  editions:
    - GeoLite2-City
    - GeoLite2-ASN
  auto_update: true
  update_interval: 24h

Free account: maxmind.com/en/geolite2/signup

CLI

csm update-geoip                    # Manual database update
csm firewall update-geoip           # Download country CIDR blocks
csm firewall lookup <ip>            # GeoIP + block status lookup

Address lookups report country, city, ASN, organization, and network when the local databases supply them. Country-block matches retain all matching countries and do not prevent ASN or city enrichment. Missing fields are omitted.

API

GET  /api/v1/geoip              IP geolocation (?ip=&detail=1)
POST /api/v1/geoip/batch        Batch lookup (array of IPs)

Challenge Pages

JavaScript proof-of-work challenge pages - a CAPTCHA alternative for suspicious IPs.

How It Works

  1. Suspicious IP hits a protected resource
  2. CSM serves a challenge page requiring client-side SHA-256 proof-of-work
  3. Browser computes the proof (shows progress bar)
  4. On valid solution, CSM issues an HMAC-verified token
  5. Subsequent requests pass through automatically

Features

  • SHA-256 based difficulty - configurable 0-5 levels
  • Client-side computation - no server load
  • HMAC token verification - prevents replay attacks
  • Nonce-based anti-replay
  • User-friendly - progress bar, instant feedback
  • Bot filtering - clients that do not run JavaScript never pass. Headless browsers and scripts written for the puzzle can solve it, and a higher difficulty costs real visitors more time than such clients

Use Cases

  • Gray-listing alternative to hard IP blocks
  • Protecting WordPress login pages
  • Rate limiting without blocking legitimate users
  • DDoS mitigation layer

Routing Behavior

When challenge.enabled: true, CSM routes eligible IPs to the challenge page instead of hard-blocking them. This works independently of auto_response settings.

Addresses the firewall refuses to block are never routed: entries in infra_ips or firewall.infra_ips and, while the CSM firewall runs, the current addresses of infra hostnames and the server’s own addresses. Entries only in firewall.infra_ips still produce alerts and spam evidence.

Challenge-Eligible Checks

Pre-auth, browser-visible attack signals only: wp_login_bruteforce, xmlrpc_abuse, wp_user_enumeration, webmail_bruteforce, http_scanner_profile, http_claimed_bot_unverified, ip_reputation, email_php_form_relay (routing and timeout blocks require ip_recipient_mode: enforce). Post-auth audit events (cPanel, webmail, file upload, WHM logins), WAF high-volume attacker findings, and non-browser protocols (SSH, FTP, DNS recursion, outbound traffic, API auth) are excluded - their IPs have no useful challenge step or no browser session to render the PoW page.

Local threat scores no longer route addresses to the challenge page, and no check produces local_threat_score findings any more.

http_scanner_profile routing is operator-selectable: auto_response.http_scanner_action: "challenge" (default) routes the IP here; "block" skips the challenge and hard-blocks directly. With the challenge subsystem disabled, both values block.

Always Hard-Blocked

Confirmed malware (webshells, YARA/signature matches), WAF high-volume attackers, C2 connections, backdoor ports, phishing pages, database injections, and spam outbreaks are hard-block candidates immediately, even when challenge is enabled.

Timeout Escalation

If an IP doesn’t solve the PoW challenge within 30 minutes, it is automatically escalated to a hard firewall block while auto_response.enabled and auto_response.block_ips are on; otherwise the entry only expires. The pending claimed-bot path (http_claimed_bot_unverified) is the exception: it expires without timeout escalation because the reverse-DNS verifier decides the next action. A confirmed spoof is hard-blocked by the later http_ua_spoof finding after it reaches the UA threshold, while a real crawler is removed from the challenge list once verification succeeds. A client named only by a trusted proxy’s X-Forwarded-For header is the other exception: its requests reach this host from the proxy, so neither the challenge page nor a firewall block reaches it, and its entry expires without escalation. Counts that include direct requests still allow timeout escalation, and so does a direct finding for the same address in the same scan. An existing non-escalating listing stays until it expires.

A timeout block of an address that recently logged in as a customer sends the known-customer block alert at once. The block still applies.

Passed challenges

Each pass is logged as CHALLENGE-PASS: <ip> passed (<method>) and counted in csm_challenge_passed_total and the passed field of the challenge stats. CSM remembers passes for 7 days, across restarts, and expires stored passes during normal operation:

  • A challenge that times out within 4 hours of the address’s last pass expires without a block. A returning browser is waved through by its verification cookie for the same 4 hours.
  • WordPress login, XML-RPC and user enumeration counts read each site’s last log lines, which on a quiet site can be days old. Lines an address logged before or at its last pass no longer count against it or attribute a distributed attack to that site; newer ones do. While the challenge service is active, these counts also ignore lines older than the pass retention period, so forgotten passes cannot revive answered requests. Lines without a readable timestamp still count because CSM cannot establish their age.

A pass does not exempt an address from new challenges or hard-block checks. A client without a valid verification cookie must solve a new challenge when routed again, including during the timeout grace period.

Bind address

The listener binds to 127.0.0.1 by default, so enabling the challenge server alone does not expose a new public port. The webserver integration uses direct redirects to challenge.public_url; installed direct mode therefore needs a non-loopback listener and a public URL ending in /challenge. For your own proxy on this host, listen on a Unix socket instead (see Proxy on the same host).

challenge:
  enabled: true
  listen_addr: 0.0.0.0
  listen_port: 8439
  public_url: https://cpanel.example.com:8439/challenge
  tls_cert: /var/cpanel/ssl/cpanel/mycpanel.pem
  tls_key:  /var/cpanel/ssl/cpanel/mycpanel.pem

When CSM’s firewall is enabled and challenge.port_gate.enabled is true, the daemon also opens challenge.listen_port in the main firewall rules. The port-gate chain still drops traffic to that port unless the source is loopback, an infra_ips entry, or an IP currently on the challenge list. Port-gate rules follow the configured listener address family. An IPv6-only listener gates only IPv6 clients; IPv4 challenge entries stay in the webserver map but are ignored by the IPv6 nftables set.

Run csm doctor challenge after changing these fields. The command checks the public URL shape, TLS files, port-gate setting, installed webserver snippet version, webserver configtest, and the live /challenge/gate endpoint. Add --json for automation.

TLS

The challenge listener serves HTTPS when challenge-specific TLS material is configured. Loopback listeners stay on plain HTTP by default. Direct/public listeners can reuse the Web UI cert.

Resolution order:

  1. challenge.tls_cert + challenge.tls_key (explicit per-service).
  2. webui.tls_cert + webui.tls_key (shared cert; cPanel mycpanel.pem covers both webui and the challenge port without extra config) only when challenge.listen_addr is not loopback or a Unix socket.
  3. Plain HTTP. This is expected for the default loopback-only path. Public listeners without TLS log a startup warning. HSTS-pinned parent domains (cPanel, phpanel, customer apex) will fail with ERR_SSL_PROTOCOL_ERROR because the browser auto- upgrades the URL to https; ship TLS material in production.
challenge:
  tls_cert: /var/cpanel/ssl/cpanel/mycpanel.pem
  tls_key:  /var/cpanel/ssl/cpanel/mycpanel.pem

Trusted Proxies

By default, the challenge server uses RemoteAddr to identify clients. The shipped webserver integration redirects browsers directly to challenge.public_url, so it does not need trusted_proxies. Configure trusted proxies only for a custom proxy deployment where CSM receives traffic from a proxy on another machine and must trust X-Forwarded-For from that proxy.

challenge:
  enabled: true
  trusted_proxies:
    - "192.0.2.10"
    - "198.51.100.0/24"

Entries are addresses or CIDR ranges. Legacy zero-padded prefix lengths remain accepted. CSM skips other entries and never trusts a catch-all, such as 0.0.0.0/0. A peer covered only by a refused catch-all names no visitor when it forwards a request, even if truncation leaves an empty header. This includes native IPv6 peers covered by a broad IPv4-mapped range. A direct request from such a peer still names its own address. csm validate and plain csm doctor warn about unusable entries; they do not reject the configuration. Saving the challenge section in Settings rejects them. Trust only proxies you control, not ranges that also contain visitors. Each proxy must append its observed peer address to X-Forwarded-For or replace the header with that address. The challenge server also trusts the proxies in web_server.trusted_proxies, the list the HTTP checks use.

CSM reads all X-Forwarded-For field lines as one ordered chain, from the right, past other trusted proxies, and takes the first other address as the visitor. An entry that is not an address ends the read.

If no visitor can be identified because the chain is missing, malformed, over 4 KiB, more than 31 trusted hops before the visitor, or contains only trusted proxies, challenge endpoints return HTTP 400. They do not issue a puzzle or cookie, accept a bypass, or clear a pending challenge for the proxy. This also applies to the gate and admin-token endpoints.

CSM rejects proxy trust for loopback, the unspecified addresses, and every address on the host’s interfaces, link-local ones included. Any local account can connect from those and write X-Forwarded-For. An entry containing one of them does not prevent the daemon or challenge listener from starting or reloading. Requests attributed to a listed local address receive HTTP 400. Remote proxies in the same range remain usable. csm validate and plain csm doctor warn; csm doctor challenge reports the unusable local proxy as a failure, and the webserver integration installer refuses it. This applies to both TCP and Unix socket listeners. For a proxy on this host, use the socket and remove host-local entries from trusted_proxies.

While serving requests, CSM uses a cached interface address list, refreshed on demand after five minutes. If a refresh fails, it keeps the last successful list. An address added while the daemon runs can still be trusted until a successful refresh discovers it. Before assigning this host an address covered by trusted_proxies, remove or narrow the covering entries to remote proxies and restart CSM. Only then assign the address. Run csm doctor challenge to check the resulting configuration.

With trusted_proxies set, point challenge.public_url at the proxy. listen_addr holds a host only; the port goes in listen_port.

csm doctor challenge leaves TLS to the proxy and does not ask for the port gate. It does not check the proxy itself, so confirm HTTPS works at public_url and keep listen_port closed to everything except the proxy. A public_url that names the listener’s own address and port is direct mode, and the usual TLS and port-gate checks apply.

With both challenge.trusted_proxies and web_server.trusted_proxies empty, X-Forwarded-For from a TCP peer is ignored to prevent IP spoofing. Only challenge.trusted_proxies says the listener itself sits behind a proxy: the installer and csm doctor challenge’s TLS and port gate rows read it alone, so a site CDN in web_server.trusted_proxies does not change them. A listed address is never a visitor in either mode.

Proxy on the same host

A proxy on the same host reaches the challenge server through a Unix socket. Only root and members of challenge.listen_group can open it, so local accounts cannot pose as the proxy. Shared hosts that do not need a proxy should use direct mode (see Bind address), which trusts nothing.

challenge:
  enabled: true
  listen_addr: unix:/run/csm/challenge.sock
  listen_group: nginx     # the group your proxy runs as; nobody on cPanel
  public_url: https://challenge.example.com/challenge

CSM creates the socket with mode 0660, owned by root and listen_group, and removes it on shutdown. Its directory must be owned by root, not writable by anyone else, and writable by the CSM service; with the shipped systemd unit that means /run/csm. The path must be absolute, without repeated separators, dot components or a trailing slash. It may use at most 107 bytes, and its directory at most 91 bytes, since CSM first binds under a temporary name that needs up to 16 more bytes. listen_port, the port gate and the Web UI TLS fallback do not apply. The proxy must send X-Forwarded-For; a request without a readable visitor address gets HTTP 400. trusted_proxies stays optional and lists any proxies in front of the local one.

nginx:

location /challenge {
    proxy_pass http://unix:/run/csm/challenge.sock;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
}

With a CDN or another proxy in front of nginx, list those proxies in trusted_proxies and use proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; so the visitor address they add is kept.

Apache 2.4.7 or later, with mod_proxy and mod_proxy_http (it adds X-Forwarded-For itself):

ProxyPass "/challenge" "unix:/run/csm/challenge.sock|http://localhost/challenge"

The socket proves only that a request came through a process in listen_group. Two things outside CSM decide whether that process can be trusted, and CSM cannot check them:

  • Tenant code must not run as the proxy’s user or group. On cPanel the web servers run as nobody, which is safe when PHP and CGI run as each account’s own user (suEXEC, LSAPI with suEXEC, per-account PHP-FPM). It is not safe with mod_php (DSO), CGI without suEXEC, mod_ruid2 or mpm-itk. Use direct mode on those hosts.
  • The proxy must take the visitor’s address from its own connection. It must not accept forwarded-address headers from addresses on this host, such as nginx set_real_ip_from 127.0.0.1 or Apache RemoteIPInternalProxy 127.0.0.1 (set up for ea-nginx in front of Apache). The proxy browsers connect to must be the one that talks to the socket.

LiteSpeed’s proxy emulation does not support this setup, and SELinux in enforcing mode may stop a confined web server from opening the socket. Use direct mode on those hosts.

csm doctor challenge checks the group and the socket directory, then probes the gate through the socket.

Successful Verification

When a client passes the challenge:

  1. The IP is removed from the challenge list so the webserver stops sending that visitor to the challenge flow
  2. A verification cookie is set, so a visitor who is challenged again within its lifetime passes without seeing the puzzle

Passing the challenge changes nothing in the firewall. The visitor is treated like any other client from then on: normal rate limits and blocks still apply, and a later attack signal can list the IP again. The challenge page, the verify endpoint and the CAPTCHA fallback answer only for IPs currently on the challenge list; anyone else gets 404.

Webserver Integration

The webserver integration redirects challenge-listed IPs to challenge.public_url. The installer refuses to run until that URL is an absolute http or https URL ending in /challenge, and the configured challenge listener uses non-loopback TCP or a Unix socket. For a socket, challenge.listen_addr must use unix:/absolute/path and challenge.listen_group must name the proxy’s group. Invalid socket settings are rejected before the installer writes runtime maps or changes the webserver.

csm webserver-integration install     # initial wire-up
csm webserver-integration upgrade     # re-apply after a CSM upgrade
csm webserver-integration status      # show detected stack + version drift
csm webserver-integration validate    # run the webserver's configtest
csm webserver-integration remove      # uninstall the snippet

The installer auto-detects the active webserver via internal/platform. Supported stacks and snippet paths:

StackSnippet path
cPanel + Apache (EasyApache)/etc/apache2/conf.d/csm-challenge.conf
Debian/Ubuntu Apache/etc/apache2/conf-enabled/csm-challenge.conf + per-site include
RHEL family Apache (httpd)/etc/httpd/conf.d/csm-challenge.conf + per-site include
LiteSpeed (LSWS)/usr/local/lsws/conf/templates/csm-challenge.conf
Nginx (plain + Engintron + phpanel)/etc/nginx/conf.d/csm-challenge.conf

Apache without cPanel applies the server-wide snippet inside no <VirtualHost> on its own, HTTP or HTTPS. The installer also writes the rule to /var/cache/csm/apache-challenge-site.conf, next to the maps, and prints the one line each site needs. Add it inside every <VirtualHost>:

<VirtualHost *:443>
    ...
    Include /var/cache/csm/apache-challenge-site.conf
</VirtualHost>

Then run apachectl configtest and reload Apache. CSM never edits site files; csm doctor challenge warns while any <VirtualHost> in the files Apache loads lacks the line. The check follows loaded includes through shared configuration fragments and ignores includes in inactive conditions. If Apache does not report its loaded configuration, doctor warns that coverage is unknown. Uninstall leaves a disabled per-site include so existing site configuration remains valid. Remove the Include lines before deleting that retained file.

On cPanel, Apache and LiteSpeed alike, HTTPS site blocks do not inherit the server-wide snippet, so the installer also writes the rule to the per-site include /etc/apache2/conf.d/userdata/csm-challenge.conf. It skips /.well-known/ paths so certificate validation keeps working. cPanel adds that include to every site block when it rebuilds httpd.conf with a file present. csm webserver-integration install and upgrade run that rebuild once when needed, inside the same configtest-then-reload run. The daemon’s startup refresh writes the file but never rebuilds httpd.conf; HTTPS sites are then covered after cPanel’s next rebuild, and csm doctor challenge warns until they are. Coverage checks inspect each HTTPS site block and require the per-site snippet to be present.

If a rebuild, configtest, or reload fails, the installer restores the previous snippets and the saved httpd.conf, including its permissions. It does not run a second rebuild during recovery. A failed restore or recovery configtest prevents a recovery reload and reports the failure.

csm webserver-integration remove disables both redirects even if one snippet is missing. It leaves a managed, comment-only per-site snippet: cPanel’s required wildcard include must still match a file for the web server to start. Removal never rebuilds httpd.conf. Installing again replaces the placeholder with the redirect rule. Automatic startup refreshes leave both snippets alone if either has a newer template version than the running binary.

The snippets are rendered from the effective CSM config. Apache and LSWS read their RewriteMap from /var/cache/csm/challenge_ips.txt; Nginx reads a native map include from /var/cache/csm/challenge_ips.nginx.map. The directory is the service’s systemd cache directory: it is world-readable so the webserver user can read the maps, and unlike the runtime directory it survives systemctl stop csm, package upgrades and reboots. That matters because Apache and LSWS validate the map at config-parse time and Nginx fails on a missing include, so a map that vanished with the daemon would take every site down at the next webserver reload. CSM rewrites the Nginx include on challenge-list changes and reloads Nginx only when the file content changes.

Older installers also deployed a legacy proxy-mode snippet on cPanel Apache and LiteSpeed (/etc/apache2/conf.d/csm_challenge.conf). Installs no longer do. On those stacks csm webserver-integration install and upgrade remove the legacy snippet in the same configtest-then-reload run that writes the integration snippet, and restore it if that run fails. If either file cannot be restored, the error reports the incomplete rollback and no recovery reload runs with that incomplete configuration. While both are present and the integration template is not newer than the binary, status reports the integration as stale, so the daemon’s startup refresh below removes the legacy snippet on hosts that already run the integration. Newer templates are left alone at startup, including after a binary rollback. Observe mode skips this refresh.

At startup the daemon creates the default maps if they are absent, including when challenge mode is disabled. It also checks the legacy snippet when one is still present. If its RewriteMap points at a map file the daemon does not maintain – which makes webserver config validation fail host-wide once that file goes missing – the daemon re-deploys the shipped template. A CSM-managed integration snippet from an older template version is refreshed the same way, through the configtest-then-reload flow below. Files without the directive, operator-edited snippets, and snippets already pointing at the daemon maps are never touched. While such a snippet still references a map under the old runtime directory, the daemon keeps that file present until the snippet is refreshed with csm webserver-integration upgrade.

csm uninstall and the package post-removal hook remove the snippets before the maps, and keep the maps whenever a snippet cannot be removed (for example an Nginx server {} block that still uses $csm_challenged).

On every run, the installer:

  1. Writes the new snippet to a sibling temp file and renames it into place atomically.
  2. Runs the webserver’s own configtest (apachectl configtest, nginx -t, lswsctrl conftest).
  3. On pass: reloads the webserver gracefully and exits 0.
  4. On fail: restores the previous snippet bytes (or removes the file if it did not exist before) and exits non-zero with the captured configtest output. The webserver is never reloaded with a broken config.

Nginx config validation runs outside the daemon’s read-only sandbox via systemd-run, so it can open its logs. The availability probe and the configtest each get a separate 15-second command timeout. If the probe fails or times out, or systemd-run is absent, CSM tries nginx -t directly. A failed configtest is reported without retrying it.

The snippet header carries a version marker; upgrade is a no-op when the on-disk content matches the rendered template and no legacy snippet remains to retire. Hand-edited files (missing or mismatched marker) trip a “modified” status and the installer refuses to overwrite them - remove or rename first.

Hosts with no detectable webserver exit with status=skipped so package post-install hooks succeed cleanly on, e.g., a plain phpanel worker that doesn’t run nginx locally.

Bypass Paths

Three opt-in bypass mechanisms let legitimate traffic skip the PoW page entirely. All default off; an upgraded csm.yaml with no new blocks behaves exactly as before.

CAPTCHA Fallback (JS-Disabled Visitors)

The PoW solver requires JavaScript. Visitors with JS off (older mobile browsers, accessibility tooling, text browsers, scripted integrations) would otherwise be locked out. When configured, CSM renders a Cloudflare Turnstile or hCaptcha widget inside a <noscript> block; on completion the form posts to /challenge/captcha-verify and CSM validates the token server-side against the provider’s siteverify endpoint. Provider rejections do not spend the page nonce, so a visitor can retry the same challenge page after a mistyped, expired, or failed widget response.

challenge:
  captcha_fallback:
    provider: turnstile         # turnstile | hcaptcha | "" (off)
    site_key: "0xAAAA..."       # public key embedded in the widget
    secret_key: "0xBBBB..."     # verified server-side; never sent to client
    timeout: 10s

Verified Operator Sessions

Operators who repeatedly hit the challenge during normal admin work can mint a signed cookie that bypasses PoW for the cookie’s TTL. The signing key is generated at daemon startup and rotates on every restart – old cookies stop working automatically.

challenge:
  verified_session:
    enabled: true
    cookie_name: csm_admin_session    # default
    ttl: 4h                            # default
    admin_secret: "long-shared-secret" # required

To issue a cookie, POST the secret to the challenge server:

curl -i -X POST -d "secret=long-shared-secret" \
  https://your-host:8439/challenge/admin-token
# 204 No Content
# Set-Cookie: csm_admin_session=...; Path=/; HttpOnly; Secure; SameSite=Lax

The cookie binds to the requester’s IP, so a stolen cookie does not work from a different network.

Requests from this server’s own addresses, loopback included, get HTTP 403 without a check of the secret, because every local account can send from them. Request the cookie from the network you browse from.

Verified Search Crawlers

Googlebot and Bingbot can be allow-passed by reverse-DNS forward-confirm. CSM looks up the visitor’s PTR, checks it ends in a known crawler suffix (e.g. .googlebot.com), then forward-resolves that name to confirm the original IP appears in the result. A spoofed User-Agent: Googlebot from a residential IP fails forward-confirm and falls through to PoW.

challenge:
  verified_crawlers:
    enabled: true
    providers: [googlebot, bingbot]
    cache_ttl: 15m

Positive results cache for cache_ttl; negative results cache for one-fifth that long so a transiently-broken resolver does not lock out a real crawler for the full window.

Operational

Backups

csm store export and csm store import capture the bbolt store, state JSON files (baseline file hashes), and signature-rules cache into a single tar+zstd archive. Use these for re-provisioning, cluster cloning, and disaster recovery rather than re-baselining a 200k-file account tree from scratch. The daemon writes the archive under its state directory (the only tree its systemd sandbox can write) and the CLI moves it to the path you give, so any destination the CLI can write works, including /var/backups. The destination directory and every directory above it must be owned by root or by the account running the command, and must not be writable by other accounts unless they carry the sticky bit as /tmp does; the destination itself must not already exist as a symlink or as another account’s file. Otherwise another account could swap the archive out after the CLI has verified it, and the export is refused before the daemon writes anything. In-progress staging files and unconfirmed firewall rollback artifacts are excluded from backups, so restoring an archive cannot re-arm an old confirmation window.

csm store export /var/backups/csm-$(date +%F).csmbak
sha256sum -c /var/backups/csm-$(date +%F).csmbak.sha256
# transfer the .csmbak + .sha256 to the target host
systemctl stop csm
csm store import /var/backups/csm-2026-04-27.csmbak
systemctl start csm

Partial restore: --only=baseline restores only the file-hash state (useful after a full re-install where firewall and history should stay fresh); --only=firewall merges the firewall buckets into an existing daemon (useful for cloning blocklists across a cluster); a durable firewall journal and its snapshot in the archive replace the target’s. It refuses while either side has a durable firewall action unresolved or its audit undelivered.

Performance Monitor

CSM monitors server performance metrics and generates findings when thresholds are exceeded.

A successful Fix clears only the finding for that file, and the finding alerts again if the problem returns.

Critical Checks (every 10 min)

CheckWhat it monitors
perf_loadCPU load average vs core count (critical/high/warning thresholds)
perf_php_processesPHP process count and total memory usage
perf_memorySwap usage percentage and OOM killer activity

Host-wide OOM events are Critical; memory-cgroup limit events are Warning. Both are reported when present in the last hour, with separate deduplication identities for each scope and victim process.

Deep Checks (default every 60 min, thresholds.deep_scan_interval_min)

CheckWhat it monitors
perf_php_handlerPHP handler type (DSO vs CGI vs FPM) and configuration
perf_mysql_configMySQL my.cnf settings (buffer pool, connections, query cache)
perf_redis_configRedis memory limits, persistence, eviction policy
perf_error_logsBloated error_log files in every document root, with growth rate
perf_wp_configWordPress wp-config.php hardening and debug settings
perf_wp_transientsWordPress database transient bloat
perf_wp_cronWordPress cron scheduling (missed crons, excessive events)
perf_wp_loopbackWordPress sites calling themselves faster than WordPress’s own schedulers, hour after hour
perf_wp_myisamWordPress database tables still on the MyISAM storage engine

Bloated error logs

perf_error_logs scans the configured web roots plus every document root in cPanel’s domain map, so addon and subdomain sites are covered as well as public_html. It walks three levels deep. Heavy trees such as wp-content, wp-admin and vendor are not descended, but an error_log directly inside one is checked: PHP writes the log next to the running script, so admin-ajax.php errors land in wp-admin/error_log.

A log over performance.error_log_warn_size_mb (default 50) is a Warning and only shows on this page. A log over performance.error_log_critical_size_mb (default 1024) is High and alerts. Once a log has been seen for at least an hour, the finding also shows how fast it grows per day. A log that shrinks or is replaced by a new file, as on truncation or rotation, starts a new baseline. A cancelled scan changes nothing; a scan that could not read part of the tree keeps the logs it did not reach and updates the ones it did.

The finding keeps the same identity while the log grows, so alert email and the webhook send it once, history records it at most once a day, and a dismissal sticks until the log crosses into the higher tier. When more than 20 logs are bloated, the largest are reported.

WordPress loopback requests

A WordPress site runs background work by sending a request to itself, a loopback. WordPress’s cron spawn and Action Scheduler’s admin dispatch each hold a 60-second lock, so one job reaches its own site at most once a minute.

perf_wp_loopback counts the last three complete local clock hours in every active vhost log. It counts POST requests with WordPress’s own User-Agent from the loopback interface or one of the server’s addresses. The count is kept per job, meaning the full logged path plus the admin-ajax action; shortened display names do not merge jobs. A job over 60 an hour in each of the three hours is reported as a Warning on this page. The finding shows the hourly counts, how many runs failed with a 5xx error, and the User-Agent.

Request timestamps record when requests started, but log lines are written when they finish, so no part of a log can be skipped by its timestamps. The check follows each active log instead: every run reads only what the log gained since the previous run, and the hourly counts are kept in the scan state. A rotated, truncated or replaced log is read from its start, and the first run reads at most the last 8 MB of each log. Memory for distinct jobs is bounded, and the scan stops at its deadline, including inside oversized lines. A log that cannot be read, or exceeds the job budget, keeps its saved position and its prior findings until a later run succeeds.

There are two usual causes:

  • A plugin schedules its job for “now” and fires it from every page view. The job then runs about as often as the site is visited, crawlers included.
  • A background queue such as Action Scheduler keeps re-dispatching itself because its backlog never drains.

The hour in progress is not counted. A site that reaches itself through a proxy such as Cloudflare is missed when its log records the proxy’s address rather than the server’s. Other tenants can also make requests from the server’s addresses, so this Warning is advisory: it does not establish which site initiated a request, send alerts, or trigger a response action.

WordPress MyISAM tables

MyISAM locks a whole table for every write. While one request writes, every other request that reads or writes that table waits. On a busy site a burst of uncached requests queues behind those locks until it holds every database connection the account may open, and new visitors get database errors. InnoDB locks single rows, so the same traffic keeps flowing.

perf_wp_myisam reads MySQL’s table catalogue once per run, as root and without site credentials, and matches the MyISAM tables against the WordPress installs found by the shared discovery. Each install’s database and table prefix come from explicit string literals in its wp-config.php, including an explicitly empty prefix. PHP expressions are not evaluated. A table belongs to the install whose prefix matches it most closely, so two sites sharing one database are reported separately. Dormant and suspended installs still reserve their prefixes. Each database and prefix gets one Warning on this page, listing the tables largest first with their total size.

Not reported:

  • installs whose DB_HOST is another server; accepted hosts are localhost, loopback addresses and IP addresses bound to this server’s interfaces
  • TCP connections whose port differs from the catalogue server’s port, and explicit local sockets whose path differs from that server’s socket path
  • hostname aliases other than localhost; DNS is not used to infer ownership
  • document roots the panel no longer serves, and suspended accounts

An unreadable or unresolved configuration, or a failed catalogue query, keeps prior findings until a later run succeeds. If a sibling’s database or prefix cannot be established, affected scopes are withheld rather than assigning its tables to a shorter prefix. Configurations using dynamic or conditional database settings need explicit literals for this check to resolve their scope.

To convert a site without losing data:

  1. Put the site in maintenance mode so nothing writes during the change.
  2. Take a full database dump and confirm it completed.
  3. Run ALTER TABLE <table> ENGINE=InnoDB for each listed table, one at a time, and stop at the first error. Each statement copies the table and blocks writes to it while it runs.
  4. Compare row counts before and after, then end maintenance mode.

Keep the server’s strict SQL mode for the conversion. A table holding values InnoDB would reject then fails with an error instead of having those values silently adjusted.

Web UI

The Performance page (/performance) shows real-time metrics:

  • Server load and CPU usage
  • PHP process and memory charts
  • MySQL and Redis health
  • WordPress performance indicators

PHP worker counts include LiteSpeed and PHP-FPM pool workers. The PHP-FPM master is excluded from these request-worker counts; security process checks continue to inspect it. Database memory comes from the MySQL or MariaDB server process, independent of its PID-file location or wrapper-related arguments.

The findings list also exposes admin-only fixes, per-row and as a Bulk fix dropdown that applies one fix to every matching finding at once:

  • perf_error_logs: truncate a bloated error_log in place. The inode is preserved so running PHP processes keep writing to the same file.

  • perf_wp_config: disable display_errors in .user.ini, php.ini, or .htaccess by commenting the matched line and appending an Off override.

  • perf_wp_cron: add define('DISABLE_WP_CRON', true) to wp-config.php and install a per-user system cron that runs wp-cron.php on a fixed interval. Disabling WP-Cron alone would stop scheduled WordPress tasks, so the cron is installed in the account owner’s own crontab (visible and editable by the customer). The cron is installed before the define is written, so a crontab failure leaves WordPress scheduling unchanged. The define is inserted before the “stop editing” marker (or the wp-settings.php require); insertion points inside multiline comments or heredocs are ignored, and the fix refuses a wp-config.php with no safe insertion point rather than corrupt it.

    The installed schedule is staggered per account and docroot (for example 7-59/15) instead of a wall-clock-aligned */15, so many managed sites do not all fire in the same second and spike the host load. Non-divisor intervals use a shifted minute list so the gap stays within the configured interval. The command also runs under flock -n with a per-docroot lock file in the account home, so a slow pass skips the next run instead of overlapping it. When both auto_response.enabled and auto_response.fix_wp_cron are enabled, managed crontab lines installed by older releases are upgraded on daemon start to this format. Their PHP interpreter changes only when an override is configured or the site’s PHP version can be resolved unambiguously; otherwise, they keep their existing interpreter. Only lines under the # CSM WP-Cron marker are touched; customer-authored cron entries are never rewritten.

These actions are limited to configured account roots, reject symlinks anywhere in the path and unsupported file types, refuse a file that changes before the edit lands, and remove the fixed row from the active findings view after a successful edit.

WP-Cron fix settings

Tune the WP-Cron remediation under Settings -> Performance:

  • performance.wp_cron_fix.interval_minutes (default 15, range 1-60): how often the installed system cron runs wp-cron.php. The interval only bounds task latency – WordPress keeps its own event schedule and a cron pass with nothing due is a wasted full bootstrap – so 15 minutes is right for most sites. Lower it per host only when busy stores need tighter Action Scheduler latency.
  • performance.wp_cron_fix.php_bin (default empty): overrides the PHP interpreter for the cron line. Leave it empty and each site runs under the PHP version its own vhost is set to when cPanel’s domain map provides an unambiguous version. A vhost set to inherit runs the version named by the nearest cPanel-generated handler block that maps .php in its .htaccess chain, up to the account home. Empty blocks leave the parent mapping in effect. The lookup refuses symlinked directories and files, special files, and oversized files; an unreadable nearer file stops inheritance. CSM falls back to the detected CLI interpreter when neither gives a usable, installed version; an existing managed job keeps its interpreter while the map is unavailable. Setting a value pins that one interpreter for every managed site. CLI php is used instead of an HTTP request so the job never ties up a web worker.

To let the daemon apply this fix automatically on every WP-Cron finding, set auto_response.fix_wp_cron: true (default false; requires auto_response.enabled: true). It is opt-in because it edits customer wp-config.php files and crontabs.

MySQL telemetry auth

The MySQL panel runs mysql -e "SHOW STATUS LIKE 'Threads_connected'" from the csm process. The client needs to authenticate against the local server, and csm supports two setups out of the box:

  • A ~/.my.cnf for the csm runtime user with credentials for a MySQL account that holds at least the PROCESS privilege. cPanel and CloudLinux ship /root/.my.cnf for the root user; csm running as root picks it up automatically.

  • A unix-socket grant for the csm OS user, e.g. on Debian/Ubuntu MariaDB:

    CREATE USER 'root'@'localhost' IDENTIFIED VIA unix_socket;
    GRANT PROCESS ON *.* TO 'root'@'localhost';
    

If neither is configured, the MYSQL card renders n/a / n/a instead of a misleading 0 conn. csm makes no attempt to connect over TCP or store credentials on its own.

Redis telemetry auth

The Redis panel connects to local Redis at 127.0.0.1:6379. If Redis requires a password, set REDISCLI_AUTH in the csm daemon environment. The dashboard uses that password for its in-process Redis client.

If csm cannot read Redis, the REDIS card renders n/a instead of an empty Redis. A Redis with no memory limit shows no limit with neutral styling.

API

GET /api/v1/performance    Current performance metrics snapshot
POST /api/v1/perf/fix-error-log
POST /api/v1/perf/fix-display-errors
POST /api/v1/perf/fix-wp-cron

See the API reference for which metrics fields can be null.

Web UI

HTTPS dashboard that updates live from the finding event stream (see Live updates). Dark/light theme toggle.

Static assets use content-versioned URLs. Only a URL matching the served file receives immutable caching; older or unversioned URLs must revalidate. Replacing a file changes its version even when its size and modification time are preserved. Text assets use gzip when accepted by the client, except for byte-range requests.

The sidebar groups pages by operator workflow. URLs are stable; the groups only reorder visibility:

  • Overview - Dashboard
  • Triage - Incidents, Findings (Active and History tabs)
  • Response - Firewall, Quarantine, Cleanup History, Email Security, ModSecurity, Threat Intelligence
  • Operations - Performance, Server Hardening, Audit Log
  • Configuration - Rules, ModSecurity Rules, Verified Bots, Settings

A page has the same name in the sidebar, the browser tab and its heading. The old /blocked address redirects to /firewall.

Sidebar group expand/collapse state is saved in the browser. On viewports under 992px the sidebar collapses into a top-bar drawer toggled from the hamburger button. Account detail (/account?name=<account>) is not in the sidebar; the finding detail, account groups on Findings, incident detail, the accounts targeted in a Threat Intel lookup and the dashboard’s accounts-at-risk card link to it, and the command palette opens an account typed by name. The palette also lists Sessions. Browser logins require administrator scope. The header links to session management.

Pages

PageURLPurpose
Dashboard/dashboardTriage queue, daemon status strip, Components matrix, system posture, 24h stats, accounts at risk, auto-response summary, brute-force summary, timeline charts. A queued finding opens its own detail, and each 24h severity count opens the History tab for the last 24 hours at that severity
Findings/findingsActive findings with search, check/account filters, header grouping toggle, detail panel, fix/dismiss/suppress actions, a permanent Block for findings that report an attacker address, sticky bulk operations (fix, dismiss, suppress), modal account scan. The open finding is kept in the URL as ?key=<finding key>, so the link reopens it
Findings > History/findings?tab=historyPaginated archive of all findings, newest first, with date range and severity filters, 25 to 200 rows per page (hperpage), CSV export; window=24h (1 to 720 hours) shows a rolling window instead of calendar days
Quarantine/quarantineEvery file backup: quarantined files and cleaners’ pre-clean backups, with type, live state of the original path, content preview, restore and delete; filters by account, detector, type and date
Cleanup History/cleanup-historyDB-object backups with preview and restore controls; file backups are on the Quarantine page
Firewall/firewallSubview-tabbed page (?view=overview/blocks/allow/config/audit/danger; ?ip=<address> opens the lookup for that address): blocked IPs/subnets with GeoIP, bulk unblock of selected rows (with undo), the whitelist and allow rules (Allow Rules tab), search, audit log; the lookup links to Threat Intel for the same address; destructive actions live under the Danger tab
ModSecurity/modsecWAF workbench: status strip, Active WAF pressure summary list (top attackers by hits), top rules / domains side panel, Blocked IPs / Events tabs, and a Manage Rules link to ModSecurity Rules. Block detail panels show first-seen, top URIs, sample events, and direct links to Threat Intel, Firewall lookup, and rule management
ModSecurity Rules/modsec/rulesEnable or disable CSM rules (applied with one reload) and firewall escalation exclusions; the exclusion list works even when rule management is not configured
Email Security/emailHeadline numbers for the selected range, each opening the tab that explains it, then job tabs with counts: Overview (mail decisions chart, rspamd, groups needing attention, protection), Queue (why each message is stuck, with retry, return to sender, delete and thaw), Held mail (quarantined malware and held forward copies, to release or delete), Suspended (mailboxes and accounts CSM suspended, with live status and Release; releasing never changes the password), Rejects, Sending accounts (mail per hosting account, or per mailbox login on a Postfix host, and abuse events), Forwarders (each forwarder and where it sends, with delete per destination; a domain’s default address is changed in cPanel), Provider delays (providers and sending IPs deferring mail, and why) and Findings. ?tab= names the open tab.
Verified Bots/verified-botsEditor for the verified-crawler allowlist (reputation.verified_bots): UA, reverse-DNS suffix, and IP-range identities, plus auto-update posture, with apply-and-reload. Admin scope
Threat Intelligence/threatIP lookup with scoring/GeoIP/ASN (?ip=<address> runs it on load), 24 hour and permanent block and whitelist actions (single and bulk), top attackers, attack type charts, trends; the lookup links to Firewall for the same address, and the whitelist itself is kept under Firewall > Allow Rules
Server Hardening/hardeningOn-demand hardening audit, stored report, score, and remediation guidance. Web server directive checks follow Apache quoting, section comments, continuations, and shell include patterns. Quotes in ignored section suffixes do not cause warnings. Malformed section endings, unreadable includes, unresolved directive names, and unsupported wildcard classes produce warnings. On Apache or cPanel hosts, a configured Apache root takes precedence over default locations; a missing selected main configuration never falls back to another tree. Other servers skip the Apache audit when that main file is absent.
Incidents/incidentCorrelated incident list and grouped view, both filterable by every status, with detail panel and bulk status changes (contained, resolved, dismissed) for the selected incidents on the page, plus forensic timeline search by IP or account
Rules/rulesYAML/YARA rule management, suppressions, state export/import, test alerts
Account/accountPer-account analysis: findings, quarantine and history. An on-demand account scan runs from the Findings page
Audit Log/auditEvery operator action in the Web UI and API, including logins, logouts and session revocations, with the credential that acted, search, action and date filters, URL state, and export. Failed logins go to the daemon log instead
Performance/performanceServer load, PHP processes, MySQL, Redis, WordPress metrics
Settings/settingsSearchable config editor with grouped large sections, field-level validation errors, restart notices, redacted secret updates, and firewall tentative apply with rollback timer. Commands, file paths, sockets and environment variable names are shown read-only and change only in csm.yaml; changing the rspamd or upstream address requires entering its credential again
Sessions/sessionsActive browser logins, individual revocation and logout of every session (confirmed first)

Audit attribution is captured when the action is authorized and remains available if the browser session expires or is revoked while the action runs.

Account views use the recorded finding owner when available, with account paths as a fallback. They also recognize resolved paths under linked account roots, including files already moved into quarantine.

Threat intelligence spam evidence

The Spam column shows spam and ham verdict counts and the current spam score. Click its heading to sort by score. The Spam evidence only filter requests senders with spam verdicts, including those with no attack events. The filter is kept in the URL for bookmarks and history navigation. When switching filters, previous rows stay unavailable for actions and export until a load succeeds.

Spam evidence about IPv6 senders is listed per /64 network. Such a row is named by the network and carries a Network badge. It has no checkbox and no block or whitelist buttons, and clicking it does not open the IP lookup. Looking up an address inside the network shows whichever spam evidence is stronger: the address’s own or the network’s. Ties use the network, which is named whenever its evidence is shown.

The Listing threshold badge means the spam score qualifies for the CSM IP map. It does not confirm an active listing: the feed must be configured and working, and address exclusions still apply. Spam scores remain separate from the unified threat score and firewall block status.

Dates and time zones

Every page shows dates in the time zone chosen under Preferences: the browser’s, the server’s, or a named zone. Date filters pick whole days in that zone. Changing the zone reloads the page so dates already on screen follow it, provided the browser can save and read back the preference. If browser storage is unavailable, new renders use the preference without forcing a reload. Days with a midnight clock change start at the first valid time of that day; a repeated midnight uses its first occurrence.

Email chart hours also follow partial-hour daylight saving changes. Custom date links include both selected days, even when supplied in reverse order. Long custom ranges group adjacent days into labeled spans without dropping counts. Changing the range discards pending results for the previous range.

Live updates

The Web UI keeps an event stream open (/api/v1/events) and shows a Live indicator while it is connected. Findings are batched for up to a couple of seconds before pages update:

  • Findings shows the “new findings” banner at once instead of on its next check
  • Dashboard sends desktop notifications and refreshes the 24h counts and the triage queue
  • Incidents reloads the current list, unless incidents are selected for a bulk change

While the stream is connected, the checks those pages run on a timer slow down to once a minute as a safety net; when it drops they return to their normal pace. Pausing auto-refresh also pauses live updates.

Desktop notifications handle findings that arrive out of order within a batch or share a timestamp, without repeating them on the next history poll.

Refresh

The header shows when the page’s data was last loaded (“Updated N ago”). It moves when the page loads its data, on each automatic refresh and on the Refresh button; an action or a detail lookup does not change it. The pause button appears only on pages that refresh on a timer, and pauses that refreshing in this browser.

Refresh reloads the page’s data in place, keeping filters, scroll and open panels. On a page with unsaved edits (Settings, Verified Bots, staged ModSec rule changes) it asks before discarding them. Hardening’s Refresh reloads the stored report and does not run a new audit. Pages rendered by the server, such as Sessions, reload.

A panel that fails to load says what failed and why, with a Retry button that loads only that panel again. The panel’s earlier content returns once a later load succeeds.

The dashboard shows only what the host supports. Email Security appears when Exim, Postfix or Stalwart is detected; a mail server started after CSM shows up within five minutes. Its Queue, Held mail, Forwarders and Provider delays tabs, the antivirus lines of the Protection card, cPHulk actions on the Firewall page, the dashboard’s Email AV chip, and the Email AV, cPanel and webmail login, and known API token settings need cPanel and are not shown elsewhere.

On a host with no PHP interpreter, the PHP Shield, WordPress, WP-Cron, XML-RPC and PHP process settings and the Performance page’s PHP cards are hidden; PHP installed after CSM starts shows them within five minutes. PHP-FPM or lsphp in a container counts as PHP. A container running only Apache with mod_php has no separate PHP process and is not found. The PHP Shield settings show for container PHP only when that container has the shield script at its usual path. Hidden settings keep their values in csm.yaml, including when you save another field in the same section.

The Suspended tab shows on cPanel and Stalwart hosts. On a Stalwart host it lists the mailboxes whose sign-ins CSM stopped, and Release allows them again. On a Stalwart host the first three Email Security cards show the mail Stalwart’s spam filter did not file to Junk, the mail it rejected or discarded, and the mail it filed to the Junk folder. The state line and mail decisions chart use Stalwart’s filter verdicts, independently of CSM’s antispam mode. The Spam rejected card opens Overview and has no mode badge. It shows a dash and “none logged yet” until CSM has a retained rejection or discard, including events outside the selected range. After that, it shows the range’s rejection count and the discarded count below it. The Mail delivered card also names unmatched verdicts. Missing verdict counts from an older daemon display as zero; they never fall back to rspamd counts. Large card counts are shortened, with the full count available in the value’s tooltip. The rspamd card and the Antispam line of the Protection card are not shown, as they describe CSM’s own antispam.

Rejects and Sending accounts read the Exim main log, or the Postfix mail log on a host without Exim. On Postfix a send is mail accepted from a client that logged in, counted under that login; mail relayed for a trusted host without a login is not counted. A Postfix reject whose reason cannot be told apart from text the client sent is listed under Other. Both tabs stay empty on a Stalwart host.

On Email Security, failed refreshes mark protection status as unavailable and hide stale chart data. An older response cannot overwrite newer health status or the selected outbound abuse range. Headline counts refresh every 30 seconds. Queue messages and composition, held mail, rejects, sending accounts, provider delays and Findings refresh every minute while their tab is open. Hidden tabs do not poll their lists; the Held mail, Sending accounts and Provider delays counts refresh every minute on any tab. On hosts without Stalwart’s filter, the rspamd state in the header also refreshes every minute on any tab. Unchanged results keep their rows; changed results retain paging, sorting, selection, open details and focus for items still present. Pending message, bulk and firewall actions stay disabled across refreshes. Queue actions wait up to a minute; an unreadable outcome is reported as unknown. Held mail custom ranges include only items held during the selected days; the summary also identifies held items outside that range. The other ranges have no end, so a server clock ahead of the browser cannot hide mail just held. Rejects, Sending accounts and Suspended use the Held mail layout, with a search box, a row count and 25 rows per page. Search rejects by sender, recipient, IP or detail, sending accounts by name, and suspensions by mailbox, owner account or reason. The address keeps the search. A new search starts on the first page. A refresh keeps the current page while the search and filters stay the same. Rejects lists up to the newest 200 rejects in the range and says when the list reaches that limit.

The visible History tab refreshes with its current filters and page size. Editors wait for an ongoing save or load before accepting another refresh; Verified Bots keeps edits made while a reload is pending. ModSecurity exclusion controls wait for the exclusion list to load, and a failed load stays visible instead of appearing as an empty list. Late responses cannot replace a newer account tab, settings section, history page, or finding detail, or reopen a finding detail that was closed.

Notifications

Success and information notices fade after five seconds. An error stays until you close it, and the same error is not shown twice while it is on screen.

Confirmations

A confirmation for an action that deletes data, blocks traffic, turns protection off or ends sessions names the action on a red button, such as Delete or Block, and starts with Cancel focused, so pressing Enter does not carry it out. Logging out every browser session asks first. Cancel keeps focus after the dialog finishes opening. A later text prompt uses its own normal OK button, and bulk actions keep one confirmation open at a time.

Bulk file actions

Select-all and every bulk action reach only the rows the table currently shows. Rows on other pages or hidden by a search or filter are never selected or acted on; set the page size to All to act on every row. Quarantine selection counts and buttons are refreshed whenever the visible rows change.

A failed file restore cleans up its own destination copy while retaining the quarantined evidence. A replacement created by another writer is preserved.

A mail message quarantined from a phishing finding is listed as two files, its header and its body. Restoring either one returns the whole message to the mail queue, and Exim delivers it on its next queue run. The complete body is returned before the header becomes visible to Exim. Restore refuses an incomplete message or a conflicting queue entry and keeps the quarantine copy. If restore or quarantine cleanup fails, retry the same entry. Progress survives a daemon restart, and a cleanup retry does not requeue an already delivered message. A header quarantined on its own can be restored if its body remains in the queue.

Quarantine deletes large file selections in sequential batches. If a request fails, later batches are not sent; the page reports the confirmed deletion count and refreshes the list. File restore and delete controls stay disabled until the operation and refresh finish, and Refresh waits for them.

Threat Intel bulk block and whitelist actions accept up to 100 selected IPs and retain one undo action. Larger selections must be narrowed before sending. Findings bulk fix also asks for a smaller selection when the request would exceed the API body-size limit, which includes finding details.

Findings bulk suppress creates one path rule per selected file, up to 100 files, with the file name matched literally. Selected findings that name no file are skipped; a rule for a whole check is made from one finding. Rules are saved one at a time, and a failure stops the rest and reports how many were saved.

Security

  • Authentication - API bearer tokens in the header; opaque server-side browser sessions in HttpOnly/Secure/SameSite=Strict cookies
  • CSRF - HMAC-derived token bound to the browser session on cookie-authenticated POST, PUT, PATCH, and DELETE requests; a form sends it in the body, never the query string
  • Headers - X-Frame-Options DENY, Content-Security-Policy (scripts, styles and forms from the Web UI only; no plugins, <base> or framing), HSTS, nosniff, and the legacy XSS auditor turned off
  • TLS - Auto-generated self-signed certificate, renewed automatically within 30 days of expiry and picked up without a restart; a certificate you install is never replaced, and replacing its files takes effect on the next connection. Renewal keeps the existing private key, so a failed certificate write leaves the working pair intact; explicitly configured certificate and key files must already exist
  • Rate limiting - 5 credential attempts/min per IPv4 address or IPv6 /64, shared by login submissions and unknown bearer tokens on all authentication routes, including login GETs and unknown API paths. Concurrent requests share the limit; valid UI and metrics tokens used outside their scope do not consume it. Once exhausted, the budget rejects bearer requests before checking credentials. API and /metrics traffic also has a 600 requests/min limit.
  • Token expiry - dashboard, API and metrics credentials expire at their configured expires_at and never expire without one; expired credentials, malformed dates and future issue times are refused
  • Token sources - a token with allowed_ips is accepted only from those addresses and CIDR ranges, matched against the TCP peer
  • Token length - tokens shorter than 32 characters are reported as warnings at startup and by csm validate and csm doctor; they keep working
  • Bearer auth skips CSRF (for API-to-API calls)

Credential rotation

A credential expires only at its expires_at, a UTC RFC3339 time: set it on a webui.tokens entry, or use webui.auth_token_expires_at and webui.metrics_token_expires_at for the legacy and metrics tokens. A credential without expires_at never expires, so configurations written before these settings existed keep working. The optional issued_at (webui.auth_token_issued_at, webui.metrics_token_issued_at) records when the token was generated; expires_at must be later. Integrations should narrow the scope (unban for customer self-unblock), list their source addresses in allowed_ips, and set an expires_at that matches how often they rotate. Reloads and restarts preserve the deadline. Active event streams recheck the credential before sending each event or heartbeat. Browser sessions and their form tokens use the credential that created the session. Expiry of another administrator’s credential does not interrupt them.

To rotate, replace the token with a fresh random value of at least 32 characters, update its dates and distribute it to clients through your secret manager. Restart CSM after changing login or API credentials. Metrics token and date changes can be applied together with SIGHUP. Changing a date alone is not rotation.

Package installation keeps newly generated credentials out of subprocess arguments. Rotate credentials generated by earlier packages because local users could have observed them in process listings during installation.

An expired credential or an invalid date disables that credential and produces a warning at startup and through csm validate and csm doctor. Detection and alerting continue. If all admin credentials expire, update the root-owned configuration locally and restart CSM to recover dashboard access. Use token.expires_at.v1 in the capabilities API to detect this policy.

Browser sessions

Log in with an administrator credential from webui.tokens (or the migrated legacy webui.auth_token). The cookie contains a new random session secret; the reusable API credential never appears in it. Old token-valued cookies are rejected, so an upgrade requires a fresh login. Read-scope API tokens cannot create browser sessions.

Use Sessions in the header (/sessions) to see login names, client address, browser, creation time, last activity and absolute expiry. Revoke one session or log out every browser, including your own. These operations do not rotate API credentials. Logout uses a CSRF-protected POST. Logout and revocation accept the same browser origins as API writes (webui.allowed_origins); the login form does not check the origin.

webui:
  session_lifetime: "24h"
  session_idle_timeout: "30m"

Both durations require a restart. Lifetime must be between one second and 30 days; idle timeout must be at least one second and no longer than lifetime. Zero does not disable expiry. Idle time means time without operator activity: page loads and API requests made within a minute of keyboard, pointer or scroll input. Background polling by an open page does not count, including metrics scrapes, HTML page fetches and event-stream connections. Browser navigation to a page counts as a page load; fetching that page on a timer does not. The activity marker is recalculated when each API request is sent, so a dashboard left open still logs out after the idle timeout. Activity is committed at bounded intervals, so idle expiry can occur slightly early, never late. Passive event-stream heartbeats do not extend the session; streams check revocation and expiry before each event and heartbeat.

A browser session also loses access when its login credential expires. Every daemon restart invalidates all browser sessions. Token removal, rotation, name or scope changes take effect after the required restart; log in again with a current administrator credential. Reauthentication creates a new session and revokes the previous one. Operator preferences remain tied to the login credential, so a new session does not reset them.

The local transactional store keeps session verifiers, never raw cookie secrets. Failed persistence cannot issue a login or claim successful revocation. If the previous session cannot be read during reauthentication, login fails without issuing a replacement cookie or changing that session. Concurrent requests preserve the latest committed activity even when they arrive out of order. If the session store is unavailable, browser authentication fails closed; API bearer authentication remains independent. Session admission is bounded and refuses new logins at capacity instead of evicting active sessions. The login response says so; wait for idle sessions to expire, or revoke sessions from a logged-in browser or with an admin API token (DELETE /api/v1/sessions). Expired sessions are removed during admission, and startup clears the session records. Backup exports exclude live session records, and full restores discard any session records from older archives.

MFA is a separate planned feature. See the session management guidance for the security principles behind opaque identifiers, expiry and revocation.

Keyboard Shortcuts

General

KeyAction
?Show shortcut help
/Focus search input
Ctrl-K / Cmd-KOpen command palette
KeyAction
g dGo to Dashboard
g fGo to Findings
g hGo to Findings > History tab
g tGo to Threat Intel
g rGo to Rules
g bGo to Blocked IPs (Firewall)

Findings page

KeyAction
j / kMove selection down/up (focus moves to the row)
o / EnterOpen selected finding
dDismiss selected finding
fFix selected finding

Finding and incident rows, finding group headers and sortable table headers also work from the keyboard: Tab to them and press Enter or Space. A sorted header reports its order to screen readers. Closing the detail panel or a dialog returns focus to what opened it, and a dialog opened from the detail panel keeps Tab and Escape to itself. If a refresh removes the opener, focus returns to the open detail panel or main content. Shortcuts act on the focused finding and stay inactive while a dialog or detail panel is open.

Each finding row offers up to four actions: Fix (apply the automated remediation, shown only when one exists), Re-check (re-evaluate the finding against the live filesystem and clear it if the condition is gone, useful after fixing something by hand instead of waiting for the next scan), Dismiss (stop alerts for it while it stays unchanged; a later scan that still finds it lists it again, and undo is offered for 30 seconds), and Suppress (create a rule to hide similar findings for good). A finding cleared by Fix or Re-check is new again when a later scan finds it, so a reinfection alerts. A detection or dismissal that arrives while Fix or Re-check runs stays in place.

Dismissing a finding about a file’s content, such as a malware match, covers that content only. When the file changes, the finding alerts again as a new detection, and automatic responses can act on it. A scan that could not read the file does not end its dismissal. The “new findings” banner appears when a listed file’s content changes. If the file changed after the page loaded, the dismissal is refused and the list refreshes with a warning. A bulk dismissal still dismisses the unchanged findings, and undo covers only those.

Dismissal undo preserves later dismissals, successful re-checks and baseline resets. Findings first received in real time stop alerting when dismissed, even before the next scheduled scan records them.

Re-check appears only when CSM can test a current condition again. Supported targets include file permissions and content, phishing and .htaccess files, selected accounts and system integrity checks, WordPress core/plugins, CMS database rows, administrator accounts, and database objects. Re-check uses the stored finding identity and current host state; the browser cannot substitute a different path, row, account, or object.

YARA Re-check compares the complete scanned content with the finding’s recorded fingerprint, including large files admitted by thresholds.full_scan_max_file_mb. If the content is unchanged and YARA no longer flags it, the finding clears.

The operation fails closed. Missing or unreadable evidence, package-manager errors, failed CMS discovery, and failed database queries leave the finding active. A file whose bytes changed since detection is never cleared either, because a partial clean and an evasion edit look alike. When the replacement can be proven inert, an empty file or a comment-only stub, the finding drops to Warning instead of clearing, and the original severity comes back if the file stops being inert. Historical events such as login attempts, WAF blocks, and IP reputation cannot be re-evaluated and therefore have no Re-check action. Broad aggregates and dependency findings require a new account or full scan.

WHM Plugin

CSM installs a WHM plugin (addon_csm.cgi) that redirects operators from WHM to the daemon Web UI. After the redirect, API calls are same-origin requests to the daemon.

API requests that carry a browser Origin header are accepted from https://<hostname>:<port> (the configured hostname and webui.listen port), from an https loopback origin such as https://localhost:9443 over an SSH tunnel on any local port, and from every origin listed in webui.allowed_origins (bare https://host[:port] entries, hot-reloadable). A loopback origin is accepted only when it is the origin the request was sent to (its Host): another local service in the same browser shares the Web UI’s cookies, which ignore the port, and is refused. Any other origin gets 403 Cross-origin request blocked, which shows up as a read-only UI: pages load but every action fails. The Host header is used only for this loopback comparison; it never admits a non-loopback origin. A preflight (OPTIONS) from an accepted origin is answered with the methods and request headers the API takes, so a credentialed JSON write from a listed origin goes through.

API Reference

Machine-readable HTTPS API. All endpoints require token authentication. State-changing POST, PUT, PATCH, and DELETE requests require CSRF protection for browser cookie sessions.

Authentication

# Bearer token (header)
curl -H "Authorization: Bearer YOUR_TOKEN" https://server:9443/api/v1/status

# Cookie-based (after login)
curl -b "csm_auth=SESSION_COOKIE_FROM_LOGIN" https://server:9443/api/v1/status

Cookie-authenticated state-changing requests require the X-CSRF-Token header (obtained from the authenticated page meta tag). The token belongs to the browser session that loaded the page; another session’s token is refused, and a form field is read from the request body only. Admin-scope Bearer requests are CSRF-exempt because the Authorization header is the write credential.

Browser session management

A successful browser login exchanges an admin token for an opaque session cookie. API tokens are never valid cookies. Sessions expire on idle or absolute deadlines and all end on daemon restart. The header’s Sessions page provides the same management operations as these admin-only endpoints:

MethodPathResult
GET/api/v1/sessionsitems list of sessions with id, name, created, last_seen, expires, remote_ip, user_agent, current
DELETE/api/v1/sessions/<id>Revoke that session; an unknown ID answers 404
DELETE/api/v1/sessionsRevoke all browser sessions, including the caller’s

Revocation returns {"ok":true} only after committing to the store. Failure returns 503; invalid session IDs return 404. Lists contain no credential hashes or session verifiers. Cookie-authenticated DELETE requests require CSRF; admin bearer callers are exempt. Read-scope bearers cannot list or revoke sessions. POST /logout requires authentication and CSRF for browser callers; it revokes the current session and clears the cookie. GET /logout cannot log users out.

Token scopes

Configure tokens under webui.tokens: with a scope of admin, read or unban:

webui:
  tokens:
    - name: "operator"
      token: "..."
      scope: admin       # full read+write
      issued_at: "<UTC RFC3339 time when this token was generated>"
    - name: "panel-readonly"
      token: "..."
      issued_at: "<UTC RFC3339 time when this token was generated>"
      scope: read        # status, findings, history, stats, challenge stats, blocked IPs, scan jobs, health, components, capabilities, SSE
    - name: "panel-unban"
      token: "..."
      scope: unban       # GET /api/v1/firewall/check and POST /api/v1/firewall/unban only
      issued_at: "<UTC RFC3339 time when this token was generated>"
      expires_at: "<UTC RFC3339 time when this token stops working>"
      allowed_ips: ["203.0.113.10", "198.51.100.0/24"]

A credential expires at its expires_at and never expires without one. The optional issued_at records when the token was generated. Replace the placeholders above with fresh random tokens and real times. A malformed or future issued_at, an expires_at that is malformed or not after issued_at, and a past expires_at refuse authentication. See credential rotation.

allowed_ips limits a token to the listed addresses and CIDR ranges, for API calls, browser login and the browser sessions it starts. CSM matches the TCP peer address and ignores forwarded headers. A token without allowed_ips is accepted from any address. An entry that is not an address or CIDR fails configuration loading.

IPv4-mapped addresses and CIDRs within the mapped range match IPv4 peers: ::ffff:192.0.2.0/120 matches the same peers as 192.0.2.0/24. CIDR host bits are masked. Entries with interface zones, such as 2001:db8::1%eth0, fail configuration loading; use an address without a zone. A peer’s interface zone does not affect address matching.

Unban-scope tokens are meant for hosting panels that let customers lift their own blocks. They cannot log in to the dashboard, read status or reach any other route, and they are exempt from CSRF only on those two routes.

On POST /api/v1/firewall/unban, an unban-scope token removes the address’s block and its mail spray re-block marker. Spray failure history and SMTP, IMAP and POP failure counts remain, including shared IPv6 /64 evidence and other addresses’ markers. Renewed spraying can still trigger a block. An admin token or browser session on the same route performs a full operator release, clearing the address’s mail failure history and counts as described in automatic blocking. Either credential also removes subnet blocks CSM added automatically that cover the address (subnet_removed in the response). A subnet an operator blocked is kept and named in subnet_kept: the address stays dropped until that subnet is unblocked. Operator descriptions do not grant automatic-unban eligibility. If a subnet changes during the request, CSM leaves the replacement intact and returns HTTP 500 with a retry message. Older subnet entries without recorded automatic ownership are also kept, even if their source label says automatic; they expire normally or can be removed explicitly through the admin-only subnet removal endpoint. The credential accepted at authorization selects the release mode; expiry while reading the request body cannot change it. A valid browser session takes precedence over a bearer token and still needs CSRF with an unban bearer.

A failed individual-IP removal returns HTTP 500 without clearing threat state or reporting a successful unban. A failed subnet removal also returns HTTP 500; the individual-IP release and any earlier subnet removals have already taken effect. A firewall backend without self-service unban support returns HTTP 501 for an unban-scope caller; it never falls back to an operator release. A backend without conditional subnet removal returns HTTP 501 when a covering automatic subnet needs removal. The cPHulk flush remains best effort and runs only after the firewall removals succeed.

The legacy single-token webui.auth_token:, with its optional webui.auth_token_issued_at and webui.auth_token_expires_at, is loaded into a legacy-auth-token admin entry. Read-scope tokens are intended for orchestrators and dashboards that consume status, findings, history, stats, challenge stats, blocked-IP summaries, scan jobs, health, components, capabilities, and SSE events. Admin scope is still required for write routes and for sensitive reads such as quarantine, settings, firewall internals, threat-intel detail, rules, account detail, exports, incident timelines, and audit history. (ModSecurity stats/blocks/events are read scope; only the ModSecurity rules and escalation routes need admin.) metrics_token: is a separate, read-only credential for /metrics only.

Read-only MCP endpoint

Support agents that speak the Model Context Protocol get these reads at /mcp on the same listener, with their own credentials; see MCP endpoint. It cannot reach any route that changes state.

Errors

Every failure answers a non-2xx status with a JSON body that has an error message. That includes CSRF, origin, rate-limit and wrong-method refusals. Some failures add detail next to it; a settings save that fails validation adds errors, a list of fields and messages. An unknown path under /api/ answers 404. No 2xx response reports a failure.

{"error": "invalid IP address"}

Actions

A request that changes state answers "ok": true with the action’s own fields, such as undo_token or warning. Work that continues after the answer, such as a daemon restart, a firewall rollback or a scan job, answers 202. A batch where some items failed answers 200 and lists the failures; a batch where nothing changed answers an error status. A read-only route answers 405 to any method but GET.

An empty fix batch is rejected. Threat actions report firewall failures, and bulk actions list invalid addresses as well as failed changes. A failed action can have applied some steps before the error; inspect the current state before retrying it. A response that cannot be encoded answers 500 with an error body.

/api/v1/firewall/check and /api/v1/firewall/unban also send "success": true for callers written against the older API. It will be removed; use the status code and ok.

Lists

Every GET route that returns a list answers a JSON object, never a bare array. The list is under items. An empty list anywhere in a response is [] and an empty map {}, never null; null means a value is not known.

  • total is the number of matches the server counted. It is larger than the length of items when the route pages or cuts the list.
  • offset and limit come with routes that page or cap the list. Capped routes that do not support paging report offset: 0.
  • truncated is true when matches were left out: past the page, past the limit, or past a scan cap. When a scan cap stopped the count, total counts only what was scanned. Incident groups also send scan_truncated to distinguish a scan cap from a page limit; their total is exact when scan_truncated is false.
  • Other keys next to items describe the whole list, such as check_types on /findings/enriched or summary on /email/forwarders.

/audit, /threat/top-attackers and /threat/events do not count every match. They send limit and truncated without total.

{"items": [{"ip": "203.0.113.9", "reason": "wp_login_bruteforce"}], "total": 1}

Times and durations

Every time is an RFC 3339 instant in UTC with sub-second precision, such as 2026-09-22T10:04:05.123456Z. The same applies to the event stream. A time that is not set is left out, never sent as 0001-01-01T00:00:00Z.

The API does not send times it formatted for reading, such as “5m ago”, “1h2m” or a clock time without a date. Clients format instants in their own time zone and count down to an expires_at themselves.

Durations are numbers of seconds in keys that end in _seconds, such as uptime_seconds, elapsed_seconds, duration_seconds, oldest_age_seconds and update_interval_seconds. Temporary whitelist responses use duration_seconds, rollback status uses remaining_seconds, and incident timelines use window_seconds. Request parameters such as hours and editable configuration values retain their documented units.

Two values keep a text form. started_at_token is an opaque token for restart polling. The temporary reason on /api/v1/firewall/check keeps its “(expires in …)” text for existing callers; expires_at carries the instant.

Severity

A severity is its label: CRITICAL, HIGH or WARNING. That covers severity, severity_max, demoted_from and the attack event sev. Clients derive colours and sort order from the label. A severity filter takes the label in any case, or the older 0, 1 or 2 level. The findings inside an email quarantine entry keep the scanning engine’s own rating.

Status & Data

GET  /api/v1/status              Full health snapshot: version, uptime, watchers, severity counts,
                                 store health, blocklist size, capabilities[], config_hash, binary_hash,
                                 automation rollout state, challenge pending count, rollback state.
                                 `started_at_token` changes after a daemon restart and is suitable
                                 for restart polling.
                                 `security_posture` is `healthy`, `warning`, or `critical` after
                                 combining daemon faults with active high/critical incidents.
                                 `incidents_open_by_severity` breaks open and contained incidents
                                 down by `critical`, `high`, and `warning`.
                                 `correlation_attribution` (present after the first active-set
                                 merge) lists per check the findings correlation could not
                                 attribute to an account: `current` for the active set now,
                                 `cumulative` since daemon start.
                                 `queues` reports named protection queues, including depth,
                                 capacity, running work, recent and cumulative drops, and lag.
                                 A degraded queue changes `status` and `security_posture`.
                                 `backends` names the backend each started live monitor
                                 selected (`af_alg`, `connection_tracker`, `exec_monitor`,
                                 `sensitive_files`, `placement`): `bpf`, `legacy` or `none`.
                                 `latest_scan` is the canonical last-scan timestamp; `last_scan_time`
                                 is a legacy alias kept for older clients and will be removed.
                                 `uptime_seconds` is the time since the daemon started.
GET  /api/v1/challenge/stats     Challenge-routing activity for the UI: `pending`, `escalated`
                                 (timeouts that became hard blocks), `passed` (challenges solved
                                 or bypassed, since restart), `routed_by_check` (per source
                                 check, since restart), and `recent` routes. Read scope.
GET  /api/v1/capabilities        Static feature list (e.g. `confd.dropins.v1`, `events.sse.v1`,
                                 `webhook.phpanel.v1`, `webui.prefs.v1`, `webui.undo.v1`,
                                 `mail.queue.composition.v1`, `mail.forwarders.delete.v1`,
                                 `detect.http_scanner_profile.v1`, `challenge.stats.v1`,
                                 `firewall.rollback.v1`, `firewall.dos_exempt.v1` on Linux builds).
                                 Use for orchestrator feature-detect. `csm capabilities` prints
                                 the same build tokens without a running daemon.
GET  /api/v1/components          Watcher/component matrix with attachment, event, and upstream freshness state.
GET  /api/v1/events              Server-Sent Events stream of findings as they dispatch.
                                 Read-scope token sufficient. One JSON event per `data:` line.
                                 Writes and flushes have a three-second deadline. A failed
                                 write or flush closes the stream and frees its subscriber slot.
GET  /api/v1/health              Daemon health (fanotify, watchers, engines)
GET  /api/v1/findings            Current active findings
GET  /api/v1/findings/enriched   Enriched findings with GeoIP, accounts, fix info, and a list version.
                                 ?limit=N orders by severity, then newest, even when all rows fit;
                                 returns at most N rows with limit and truncated, while total and
                                 the severity counts cover all.
                                 ?fields=version returns only {version, total}, for change polling.
                                 block_ip is set only for checks that report an attacker address,
                                 the same evidence auto-block acts on
GET  /api/v1/finding-detail      Finding detail with action history (?check=&message=)
GET  /api/v1/history             Paginated history (?limit=&offset=&from=&to=&severity=&search=&checks=&scope=).
                                 checks is a comma-separated list of check names to include.
                                 scope=email limits results to the /api/v1/email/checks list when checks is absent.
                                 total counts every match; truncated is true when matches exist past the page
GET  /api/v1/history/csv         CSV export of the newest 5,000 entries matching the /history filters
GET  /api/v1/stats               24h severity counts and auto-response summary, accounts with active high or critical findings
GET  /api/v1/stats/trend         30-day daily severity counts; each `date` is a calendar day in the server's time zone
GET  /api/v1/stats/timeline      Hourly severity counts for the last 24 hours; each bucket names its `start` instant
GET  /api/v1/quarantine          Quarantined files with metadata (incl. htaccess pre_clean backups)
GET  /api/v1/quarantine-preview  Preview quarantined file content (?id=)
GET  /api/v1/db-object-backups   db_object_backups bucket (MySQL trigger/event/procedure/function drops)
GET  /api/v1/db-object-backup-preview Preview captured CREATE SQL (?key=)
GET  /api/v1/blocked-ips         Blocked IPs with reason and expiry
GET  /api/v1/accounts            Accounts a server-wide scan covers
GET  /api/v1/account             Per-account findings, quarantine, history (?name=)
GET  /api/v1/audit               Newest 200 UI audit log entries; truncated marks older ones. Each entry names the credential that acted (actor) and whether it came as an API token or a browser login (via)
GET  /api/v1/export              Export state (suppressions, whitelist)
GET  /api/v1/incident            Incident timeline (?ip=&account=&hours=)
GET  /api/v1/performance         Performance metrics snapshot (admin scope)
POST /api/v1/perf/fix-error-log  Truncate a fixed-row error_log finding (admin scope, CSRF)
POST /api/v1/perf/fix-display-errors
                                  Disable display_errors for a fixed-row config finding (admin scope, CSRF)
POST /api/v1/perf/fix-wp-cron    Disable WP-Cron and install a system cron for a perf_wp_cron finding (admin scope, CSRF)
GET  /api/v1/hardening           Last stored hardening audit report (admin scope)

Performance telemetry

GET /api/v1/performance returns metrics and findings. In metrics, mysql_mem_mb, mysql_conns, redis_mem_mb, redis_maxmem_mb and redis_keys are null when csm could not read them; zero is a measured zero. The two Redis memory fields are read together, redis_keys separately. A redis_maxmem_mb of 0 means Redis has no memory limit. The csm_performance MCP tool returns the same response.

Accounts at risk

accounts_at_risk in GET /api/v1/stats lists accounts with a current high or critical finding, whatever its age. Suppressed findings and internal records are left out. findings counts all of the account’s current findings, warnings included, and severity is the highest. At most 50 accounts, ordered by severity, then finding count, then name. An empty list is [].

Date ranges

/api/v1/history, /api/v1/email/groups, /api/v1/email/overview, /api/v1/email/rejects, /api/v1/email/senders and /api/v1/email/relay-abuse take from and to as a calendar date (YYYY-MM-DD) or an RFC 3339 time. A date is a day in the server’s time zone, and to includes the whole day. An RFC 3339 to is exclusive. The web UI sends RFC 3339 times so a day follows the operator’s time zone preference. A value in neither form is rejected with 400. An empty or whitespace-only bound is treated as absent. Calendar days start at their first valid local time, including when midnight is skipped or repeated; a skipped calendar date covers an empty range. Fractional seconds are preserved when filtering RFC 3339 bounds. On the /api/v1/email/ endpoints a to without from covers the 24 hours before to.

WordPress verification coverage

wordpress_verification contains core and plugins coverage when installation history is available. The same fields appear in csm status --json; human status shows the counts. Each entry reports verified, modified, unverified, unknown, not_wordpress and last_attempt. unknown means discovery found a directory but no completed attempt is recorded. modified means core verification completed with integrity differences, including missing core files. A verified plugin inventory can still contain vulnerable versions. The timestamp is the latest recorded attempt in that group. Counts retain the last observed results across restarts and cached scan cycles. Overlapping scans retain the latest attempt and its consecutive failure history in scan order. Discovery without an attempt does not replace verification evidence. An absent group means there is no retained installation history, not proof of coverage. An error field reports unreadable history instead of clean counts.

The first failed attempt increases unverified; repeated failures also produce an installation-specific Warning. In host scans, when one cause stops more installations than the per-cause limit, the warnings collapse into a single finding that names the cause, the total and a sample, so a host-wide fault cannot spend the whole alerts.max_per_hour budget. The limit applies across accounts for each kind of verification and cause. Account scans keep installation-specific warnings. A summary has no installation path and names an account only when all affected installations share it. These findings are separate from queue health: a command that completed with a refusal did not lose queue work. Existing queue loss accounting remains responsible for interrupted or missing command work.

Firewall confirmation

automation.firewall_confirmation describes a pending CLI apply-confirmed window, also when the firewall is disabled or its engine failed to start; it is absent when none is pending. deadline is the saved deadline in UTC when it can be read, and error says why the window cannot settle without csm firewall confirm or repair of the reported recovery problem. It is separate from the Web UI configuration rollback timer.

Automatic response admission

status.admission.v1 advertises the admission object on status responses. The same view appears under admission in csm status --json and as admission rows in csm doctor. The daemon reads it on a timer, so checked_at says how old it is. Every automatic response the blocking paths select is also handed to the ledger, which refuses disabled responses. With auto_response.dry_run false, the firewall on and outside observe mode, the ledger applies the automatic IP blocks it is handed (owner.executing) and the blocking paths leave those blocks to it. Otherwise it previews or forecasts eligible work and never applies it, and the view is not an enforcement record.

  • ledger holds the stored admission time, queue, lifetime queue counters (including newly checkpointed ingress drops), hourly and daily outcomes, ingress generations, the hourly ceiling with its general and reserved lanes, storage, the outbox, notice records and unresolved outcomes (running and unknown counts and when the next unknown one can retire), and effects: the temporary set slots the ledger’s own automatic blocks hold, pending and verified, in general and reserved, with next when the first of them expires. The effects section reports an error if its records, tally or schema disagree, including damage beyond the first expiry. Each section carries its own error, so one damaged record does not hide the rest. Ingress drops checkpointed before this accounting change are not added retroactively to lifetime counters. On an upgraded host, retained outcome windows can therefore exceed the corresponding lifetime totals. A restart does not fill that historical gap. Once the ledger has started live, a preview ends as a dry run does: the outcome windows and the candidate’s state show it as dry_run, while the hourly comparison counts it as observe.
  • ingress says whether automatic responses are admitted and, while they are not, since when and how many Critical arrivals were refused.
  • owner says whether admission runs automatic blocks (executing), whether it has completed a turn since the daemon started (settled), and names why the ledger owner is not running (error), why its last turn failed while it runs (turn_error), the last fault met while retiring uncertain blocks (retirement_error, kept until none is left faulted), what keeps unknown outcomes from retiring while the ledger holds some (retirement_held_by: engine for an owner without the firewall engine, as with the firewall disabled or in observe mode, policy for a refused execution policy; their addresses stay held). It is omitted otherwise, including while a configured firewall engine has not started. Policy refusals and recovery update this cause with the execution hold, without waiting for the next ledger sample. Once the cause is gone, each block still retires only after it has expired and the firewall shows nothing of it. While the owner has the firewall engine, block_entries names the entries whose automatic IP blocks the live configuration lets admission run on this host (scan, challenge_timeout, incident, incident_spray, central); the comparison gate expects each to be seen. The owner also reports why the last clock reading or its live start failed (tick_error), whether the clock is degraded, where the ceiling came from (ceiling_source: default, configured or clamped), the legacy hourly count imported at the first start (import, which counts at least until an hour after at, or names why the count could not be read at the first start or when execution began; a committed read error survives startup retries), the last inventory read and why responses naming a damaged record were discarded (damage_error). The first completed turn publishes settled: true with any earlier turn_error cleared. Later failed turns and policy reloads do not reset settled; a daemon restart does. preview_replacements, preview_replacements_last_hour, preview_rotations and preview_rotations_last_hour remain in the API for compatibility and read zero in this build. Live blocks execute and dry-run blocks forecast; neither increments these preview counters.
  • An owner that runs on the firewall engine, with the firewall on and outside observe mode, adds execution_held with execution_held_by (policy for a refused execution policy, firewall for the firewall engine), a verdict object for the panel verdict callback, whose counters count this process and restart at zero, a dry_run object for its dry-run forecast and a set object for the temporary block set outside a dry run, both described below, with set_error when the last turn could not read the set. verdict.enabled is true while the owner itself asks the panel: with the callback configured, during a dry-run forecast and while it runs automatic blocks itself (executing). This staging route does not advertise admission.dry_run_verdict.v1; that capability requires the complete admission-gates contract and activation evidence.

owner.set combines the non-admission occupants from the last scheduling turn with the current ledger holds, divided as automatic admission divides it: limit, unlimited, capacity, unclaimed, imported and drain_seconds as in the forecast’s occupancy below, and general and reserved with each partition’s size, the legacy and admission blocks it holds (used) and its free slots. A schedule serves block work, previewed or run, into free slots of the partition its lane occupies, or into the slot of a lower-priority automatic block it may replace, including a legacy automatic block that recorded its origin; other block work waits as set_full and is checked again when a held slot expires or after a minute. The object is absent when the last scheduling turn found no ready block work, in dry run and when the set could not be read. A maintenance-only turn keeps the last non-admission snapshot unless the owner enters dry run; status still refreshes the ledger holds. Entering dry run, including through a reload, clears both set and set_error before the next clock reading or forecast is attempted. Engine admission counts are not added again when the ledger holds their slots. Any excess counts as unclaimed and makes drain_seconds unknown (-1), since the engine’s drain estimate excludes admission rows. This view is not a fresh kernel inventory.

owner.dry_run is a capacity forecast, not a record of blocks. active is true while the owner forecasts instead of blocking. session_id, since and reset_reason (startup, mode_change or model_error) name the forecast session and are null while none has started, including after a failed start; a session starts at startup, on entry into dry run or after its model failed, and leaving dry run keeps the last session’s identity. modelled_kinds and prediction_scope say what the model covers: block_ip against the hourly ceiling and the temporary block capacity. occupancy gives the real temporary occupancy (actual, addresses plus two elements per timed subnet), the forecast addresses the real sets do not hold (shadow_only), the listed legacy automatic blocks the forecast replaced (replaced), their sum minus replaced (predicted), the limit, whether the set is unlimited and whether retained forecast state fills its modelling bound (limited); it is null while no snapshot is available. actual and imported count at least the listed imports, matching the partition counts. A replaced import is subtracted only while listed, until its own expiry, even if its replacing forecast block expires or is replaced first. Unlisted imports count again; a renewal with a new name is a new import. capacity is the finite capacity automatic admission is modelled within: a positive limit, or 10000 for an unlimited set. Until validation rejects negative limits, they use the same fallback as zero and retain their configured value in limit. unclaimed counts the real timed entries it never evicts (operator blocks, blocks of unknown origin, two per timed subnet and admission blocks no ledger record holds), imported the real verified stamped legacy automatic blocks, and drain_seconds estimates how long the last of them has left at the snapshot, or -1 when a kernel expiry is unavailable. Zero means no remaining known whole seconds. Treat -1 as unknown, not as expired; a combined horizon is unknown if any contributing horizon is unknown. New blocks, renewals and subnet wall-clock changes can extend that estimate; it is not an automatic removal deadline. general and reserved divide the slots the unclaimed entries leave, a fifth rounded up reserved for direct or corroborated work, each with its size, the real and forecast automatic blocks it holds (used; after a live period the slots the ledger’s records still hold count in their own partition, read each turn) and its free slots, also bounded by total remaining capacity. The drain is unknown while admission blocks have no ledger record. Real blocks are never forecast replacement victims and their scopes do not share a partition, so a forecast scope may be due more than live admission would give it. shares gives, for general and reserved, the scopes that share the partition (those holding its forecast blocks the real sets do not hold and those the last turn’s picks wait in) and the fewest slots each is due (share, the size divided among them; the remainder goes one each to the scopes after the session’s cursor), both zero without a scope. All share keys remain present, including zeros; without a snapshot, occupancy is null instead of an object with missing share keys. Existing occupants keep their partition when limits change. In a full set, a forecast block may replace a strictly lower-priority automatic block: a forecast block of a lower class or severity, or a listed legacy automatic block that recorded its origin, which the forecast counts as gone while still listed, until the import’s own expiry. A pick of a new attack may instead rotate out a forecast block of exactly its class and severity in the partition of its own lane once that block’s 30-minute hold has passed on both the admission clock and the session’s elapsed time (capacity_rotate). The set’s scopes share each partition: a pick at or over its share rotates only its own scope’s blocks, one below it only the scope that borrows most, and one without a share waits. A pick that waits while such a block holds its partition predicts equal_tier_wait, not set_full. A rotated or replaced forecast block’s address cannot rotate again until that block would have expired. An overfull set admits none, and entries counted in unclaimed are never replaced. Real coverage replaces forecast occupancy: a manual timed block reduces available capacity, an automatic block counts in general until ledger partitions are wired, and permanent coverage takes no temporary slot. ceiling is the forecast’s own copy of the hourly ceiling: limit, general_used, reserved_used, general_budget and reserved_budget, null while no forecast runs. picks counts the session’s predictions by name and last_hour the last sixty one-minute buckets of process monotonic time, an approximation that a wall-clock change cannot age; every prediction name is present, and both are zero while no forecast runs. examples_lost, summaries_lost and record_errors count this process’s losses and failed writes, examples_stale is true while a purge of earlier sessions’ examples is pending, and comparison_incomplete is true after any loss in this process; a restart alone does not set it, since each run’s hourly summaries carry its epoch. forecast_error names why forecasting last failed (engine_unavailable, clock_degraded, ledger_unavailable, snapshot_mismatch, session_id_unavailable, invalid_policy or model_error) until a turn succeeds, and last_error_at keeps its time until the next session starts. Absent values are null, never missing keys. The durable outcome windows in ledger keep counting dry-run endings across sessions and restarts.

The hourly ceiling follows auto_response.max_blocks_per_hour, capped at 20000. An omitted or zero value selects 2000 for the ledger and 50 for the shared scan-driven block budget. An explicit value is used by both, and csm doctor warns when it puts the ceiling below 2000. After editing the configuration, run systemctl reload csm; web UI settings saves apply at the next ledger tick. See automatic response for the scope of the current budget.

While admission previews or forecasts, existing blocking paths still decide every block. Admission rows say existing blocking is unaffected, and admission execution warns while execution is held. While the owner executes, the rows say admission decides automatic blocks, and the outages that stop automatic blocks, held execution included, fail. The dry-run rows of an owner that forecasts (admission dry run, admission dry-run capacity and admission dry-run examples) show the forecast’s session, real against predicted capacity and lost examples. The capacity row also shows the automatic slots of each partition and how long the entries automatic admission never evicts take to drain, the legacy automatic blocks the forecast counts as replaced, the forecast blocks of the last hour that would have replaced a lower-priority automatic block and how many scopes share each partition; a replacement is an admission, not a wait. A dry run alone is not a failure; a held forecast, model limits, waits the model predicts (equal_tier_wait among them), forecast blocks of the last hour that would have rotated out an equal-priority block, a set those entries fill, no free automatic slot, an unknown drain time and lost or stale examples warn. Recent rotation and wait warnings remain visible when the forecast has no current capacity snapshot; current occupancy and scope shares are then unavailable. These warnings clear when they leave the last-hour window or the forecast session ends. Outside a dry run, admission set shows the slots the last turn budgeted block work by. It warns about unavailable capacity or an unreadable set. The retained preview replacement and rotation counters remain zero in this build and do not describe live replacements or rotations. Rows warn when the ledger owner is not running, its clock reading is refused, an owner turn fails (admission turns), retiring an uncertain block fails (admission retirement), the ledger is damaged, admission has stopped or a Critical finding would not have received its response in the last hour; a partial-containment notice, whose block ran, does not count. They also show a degraded clock, a clamped ceiling, a ceiling of 1 (only the reserved lane runs), a legacy hourly count that could not be read, a failed inventory read and automatic blocks with an unproven outcome, which hold their addresses until they expire and the firewall shows nothing of them. Once owner.executing is set, rows say admission decides automatic blocks, and held execution, a stopped owner, a refused clock reading, a failing turn, a damaged ledger, stopped admission, discarded responses, a Critical gap in the last hour and a ledger the firewall could not read at startup fail. Until owner.settled is set, and for at most two minutes after the daemon starts, those rows warn and say so: the check that follows an upgrade runs before the first turn, and its failure would roll the upgrade back. A Critical gap recorded before the daemon started only warns, and says so: a flood of the hourly ceiling could otherwise fail every check after a restart for an hour. The admission execution fix names the configuration for a refused policy and the firewall otherwise.

admission.audit and admission.notices in queues report the delivery of admission records to the action log and of admission notices. admission.owner.audit_error names the last failed audit delivery, including an applied-block consumer’s state read or threat-store write. The batch stays pending and later rows can wait behind it; queues["admission.audit"] reports no-progress stalls. Successful delivery clears the error. Keep the database and pending rows; repair the reported storage failure before retrying.

A ledger that cannot start, or admission that stops, raises one auto_response_withheld alert through the same path as protection queue alerts, a Warning while admission previews or forecasts and Critical while it executes; it follows the notification channel settings but bypasses alerts.max_per_hour. Ledger notices (withheld responses, exhausted capacity, partial containment of an outbound destination and their summaries) raised while the owner previews or forecasts describe what the ledger would have done, so they are recorded in findings history as Warnings starting with “Admission preview:” and sent to no alert channel. Those raised while it executes go to the alert channels with their kind’s severity and no prefix. The ledger paces them per notice key and summary. A preview is acknowledged only after its history write succeeds. Delivery errors leave notices pending for retry: the next two cycles retry at once, then each attempt waits twice as long as the one before, up to an hour. Ledger notices and the stopped-admission alert retry independently, so a failing path does not defer the other; a successful delivery resets only its path’s wait.

Protection queue health

status.queue_health.v1 advertises the queues map on status responses. The same measurements appear under snapshot.queues in csm status --json and as named checks in csm doctor.

findings.ingest, fanotify.analyzer, fanotify.staged_packages, fanotify.dropper, fanotify.dropper_findings and spool.scanner report waiting work, capacity, running work, cumulative losses, losses during the last minute, the oldest waiting item’s age and the oldest running item’s processing time. Waiting work includes producers blocked on admission. Ingest work remains running while the dispatcher holds or processes its batch, including startup. Ages pause while the startup hold is in place and resume from its release, so a long baseline scan is not reported as a stall; work lost during the hold still counts. Releasing a hold preserves any future eligibility time for deliberately deferred work. Loss totals include the undelivered tail of a batch canceled during shutdown; scan warnings intentionally excluded from alerts do not count as lost work. Recovered file and spool scanner panics count as lost scan work. The workers continue processing later events, but repeated failures still degrade health. fanotify.reconcile reports the recovery queue in directory tasks, with 1,024 waiting slots. Repeated drops in a directory retain its original queue age. A detached scan batch remains running while new drops queue separately, including new work for a directory that is already being scanned. Eviction, work older than the recovery scan window, unreadable directories or candidate files, interrupted batches and unfinished shutdown work count as failed recovery tasks. An incomplete task may still have scanned some files; each directory task counts at most once. Directories and files that no longer exist when recovery runs hold nothing left to scan and complete the task, which keeps bulk extraction and package restores out of the loss counters. The recovery window includes its cutoff; work completed exactly at that boundary does not count as expired. Staged package verification reserves capacity for its whole running batch. Its full-queue timer starts when admission fills the queue and continues while the verifier retains those slots, including between retries. Files awaiting another attempt retain their original waiting age; retrying does not reset lag. Shutdown counts files still awaiting verification after the analyzer workers have stopped. Package metadata I/O cannot block health polling for this queue. Dropper candidates and held findings each have 16,384 waiting slots. Their lag starts after the configured unlink TTL or the finding’s 45-second grace period, respectively; intentional waiting does not count as overdue work. Detached probes and emission batches remain visible as running work. Processing time starts when each batch leaves its waiting queue, independently of the timestamp used to decide which work is eligible. Retries and refreshed observations retain earlier eligibility, and exhausted probes, capacity refusals and unfinished shutdown work count as losses. mail.delivery reports the selected mail-log reader’s 64-slot delivery queue. Its running time includes parsing and delivery of resulting findings. Losses include oversized complete file records, unreadable journal entries, canceled admission, consumer failure and buffered records abandoned on shutdown. Counters survive reader replacement and changes between file and journal sources. They count records already read, including complete records abandoned in file read-ahead, not unread source history or partial file records that have not reached a newline.

mail.file_source reports sampled untransferred bytes after the first file attachment, with depth_unit: bytes and capacity_unavailable: true. This includes disk backlog, buffered read-ahead and raw partial-line bytes, even when the reader retains only a bounded prefix of an oversized line. Transfer to mail.delivery removes the source bytes before output admission can block. These byte and record measurements are separate stages and must not be added.

A reader-owned sampler checks the current descriptor every two seconds, so appends remain visible while delivery is blocked. Health reads memory only. lag_basis: consumer_progress measures time without consumption while unread work remains. Sampling or new arrivals do not reset this clock. A partial line at EOF waits for more input and does not report a stalled consumer. Reader and sampler operations also report processing_lag after one minute without progress, including blocked reads, metadata calls and cleanup.

File I/O failures report source_io. A successful read cannot clear a failed cursor check; each operation must recover. Failed metadata sampling makes depth unavailable until a successful sample. Rotation and observed truncation retire the old generation, and late samples cannot overwrite its replacement. The initial attachment still starts at EOF; replacements start at the beginning. Copytruncate detection retains its existing limit: truncation followed by growth past the old descriptor offset between checks may be indistinguishable from an append.

Complete records already held in read-ahead count once in delivery loss when discarded. Unread disk data and unterminated fragments have no measured record count; generation changes, source failures and shutdown preserve dropped_lower_bound in the source row. A working file or journal replacement clears the retired source’s current error while retaining that history. Source path disappearance still follows the watcher’s existing grace period.

mail.journal_source reports the journal reader after its first successful attachment. The cursor exposes no exact unread-record count or queue capacity, so depth_unavailable and capacity_unavailable are always set, and lag_basis: unavailable marks the missing backlog age rather than reporting a waiting time of zero. The row measures the current operation instead. A selected entry remains in flight until delivery acquires it; known unreadable or abandoned selected entries count once in mail.delivery. The two stages must not be added together.

Journal progress follows cursor advancement, entry reads, bounded idle waits, output admission and close. One minute without operation progress reports processing_lag, including a stuck cursor with no known selected entry. Cursor, entry, wait and close failures report source_io; known abnormal exits remain visible through actual cleanup. An unknown cursor outcome or unread shutdown sets dropped_lower_bound without inventing a record count. Successful reads restore current health; historical uncertainty survives reader replacement. Successful attachment of a file source clears the retired journal’s current error. Failed attachment leaves it visible. Health reads memory only and does not call the journal or change its tail positioning, retry or delivery policy.

When their BPF backends are active, bpf.af_alg.output, bpf.connection.output, bpf.execution.output and bpf.sensitive_files.output report the 256-slot userspace delivery queues. Their losses include decoding failures, admission refusals, consumer panics and buffered output left after the reader stops. A received event remains running until evaluation and delivery to the finding queue finish, including events intentionally filtered during evaluation. Kernel ring occupancy and reservation failures are separate from these userspace measurements. bpf.placement.output, bpf.placement.kernel and bpf.placement.unresolved report whichever placement backend runs: placements waiting for a worker, placements the kernel lost, and placements whose path could not be resolved. The legacy backend adds bpf.placement.create_settle for new files waiting for their own close-write. When this queue fills, its oldest entry is sent for early analysis and counted as a drop from the settle queue. Pending entries discarded at shutdown also count as drops. A full output queue counts the placements it refuses as drops and hands them to overflow recovery, which rescans the directory a file landed in and walks a moved-in directory. placement.dir_walk reports directories moved into a watched tree that wait for their walk, counted in directories. Evicted directories and walks cut short count as lost and raise placed_dir_incomplete_realtime. bpf.execution.docroot reports up to 1,024 non-root process starts waiting for a check of a program in a document root. One worker resolves the running image before reading and scanning it. All non-root starts enter this queue because a launch path outside a document root can be an alias into one. Starts refused by a full queue, starts left waiting at shutdown, and findings suppressed during shutdown count as lost. The scheduled deep scan still reads programs in its scope, and the periodic process check judges running programs. bpf.connection.verdict reports the advisory annotation pool’s 256 waiting slots and up to four running callbacks. Repeated findings for a pending destination, reason and severity share one request and retain its original age. Capacity refusal, callback failure and abandoned shutdown work count as lost annotations, separately from lost findings. Callback errors release the pending key and allow a later finding to retry; a successful retry preserves earlier loss totals. Shutdown closes admission, waits for running callbacks and counts the queued requests left behind. Cached answers remain available without new work. Findings continue immediately when an annotation is unavailable. processctx.enrichment reports 1,024 waiting requests and up to two running process-context workers. It appears only after a BPF consumer initializes the shared pool; reading health does not start workers. Overflow replaces the oldest queued request and preserves the waiting age of the remaining requests. Running time includes the process read, latency observer, account resolution and cache write. Read errors and interrupted processing count as lost enrichment; vanished processes and stale or unverified identities are completed filtering. The queue dropped metric counts refused, evicted and abandoned requests; queue health also includes failures after a worker receives the request. The daemon stops the pool after BPF producers have joined, lets running reads finish and counts buffered requests discarded at shutdown. Final loss evidence remains available. This row measures enrichment requests, not each individual deadline-based file read inside a request. processctx.proc_reads reports the 64 shared slots for deadline-limited process file and symlink reads, including process-start captures on the BPF path. Waiting and running reads share this capacity. A slot remains occupied until both the underlying read and its caller have finished; an undelivered result still occupies its slot. A timeout counts the lost result immediately and keeps a blocked syscall visible as running work until it returns. Timeout and a later read failure count as one loss. Refusals at capacity and read failures count as losses; missing process files are expected churn. Synchronous reads without a deadline consume no slots. The initial process-directory check and the cached boot-time read are synchronous and are not measured by this row. smtp_rdns.resolves reports the 64 reverse-DNS lookup slots used by direct SMTP egress detection. A slot remains occupied until its resolver and caller finish. The one-second caller deadline also bounds healthy processing age; timing out counts a lost result once while keeping a resolver still running visible. Capacity refusals and resolver failures count as losses, while NXDOMAIN and successful empty responses are normal negative results. Cache hits perform no queued work. The bounded result cache is retained data, not waiting lookups. Status can initialize the empty cache but never performs DNS or waits for its cache lock. Synchronous lookups without a deadline use no slots or queue row. email_password.mailboxes reports waiting mailboxes and the five concurrent audits per scan, retaining each audit through finding collection and cache writes. Concurrent scans have no fixed combined waiting capacity. A busy pool uses each audit’s five-minute budget or shorter scan deadline; an idle slot with no dispatch progress for one minute reports backlog lag. Admission and post-audit work have their own one-minute budgets. Cancellation removes waiting demand without adding losses, while actual audits remain visible until they return. Expired work, failed verification, cache write failures and abnormal exits count once per mailbox. Unsupported, malformed and over-budget hashes remain normal incomplete-audit outcomes. Health retains loss counts after the batch drains and reads only memory. Deadline losses are recorded where an audit actually stops unfinished. A deadline arriving after a successful cache write does not turn that mailbox into a loss, and context evaluation during drain never holds the health lock. email_password.hashes reports the three shared password-verification slots. Each slot remains occupied until its KDF and caller have both finished, including after scan cancellation. email_password.waiting reports callers awaiting a slot; it sets capacity_unavailable because concurrent scans have no fixed global waiting limit. Both rows use the password audit’s five-minute check budget for lag. The occupied pool also reports sustained full capacity. A deadline while waiting counts one admission loss; a deadline after admission counts one lost verification result. A later KDF error or abnormal exit cannot count it twice. Explicit cancellation withdraws demand without loss; actual verification errors still count. Successful matches and nonmatches are normal results. Rejected input and already canceled callers start no queued work. These rows contain no password, hash, mailbox or account data, and status never runs a verification. The outer mailbox scan’s discovery, network enrichment and persistence are separate work from these hash slots. checks.executions reports dispatched check functions across host and account scans. An execution remains present until both its function and caller finish, including while its result awaits consumption or its function outlives a timeout. Lag is evaluated against each call’s original deadline, including a shorter parent deadline; a delayed function start does not reset it. Timeouts, panics and abandoned results count once per execution. Explicit cancellation withdraws demand without counting a loss, but a later function failure still counts. This aggregate sets capacity_unavailable: separate scans have their own wrapper limits, and timed-out functions can outlive those slots. It does not measure checks awaiting dispatch, scan-job persistence or later automatic actions. Status reads memory without waiting for a check or accessing the state database. checks.plugin_inventory reports sites waiting for the five inventory workers per refresh, retaining each site through command execution, result collection and cache storage or failed-result cleanup. Concurrent refreshes have no fixed combined capacity. Inventory uses a four-minute budget for its two bounded commands, or the shorter check deadline; admission and result handling each have one minute. A full pool stays healthy within those budgets. A free slot with no dispatch progress for one minute reports backlog lag. Command, decoding and storage failures count once per site, including when cleanup also fails. A command that ran and returned output (on stdout or stderr) with an error, such as a tree wp-cli will not inventory, answered the check and counts no loss. Missing command output still counts as lost work, including when the command exited with an error. Cancellation adds no losses; deadline withdrawal counts unfinished sites. Buffered sites stay visible until the original workers stop consuming and the refresh joins them. Actual commands remain in flight until they return. Health reads memory only and retains loss evidence after recovery. Shared-refresh waiters remain owned by checks.executions; they do not create another set of site jobs. Optional domain lookup fallback and plugin metadata enrichment keep their existing behavior. checks.wordpress_core reports installations waiting for the five checksum workers per scan, retaining each through its command, integrity findings and verified-file caching. The unit is installations; concurrent scans have no fixed combined capacity. Command lag uses the two-minute command budget or a shorter parent deadline. Result and cache work use one minute without progress. A full finite batch stays healthy within those budgets; a free worker with no dispatch progress for one minute reports backlog lag. Returned operational failures and abandoned work count once per installation. A command killed by a signal counts as failed work while retaining any partial integrity findings. Recognized integrity results from a completed command, including deliberately filtered output, complete without a queue loss. So does a command that ran and refused the tree, such as a directory that is not a WordPress installation or one whose configuration fails to load. Missing command output still counts as lost work. Cancellation withdraws unfinished demand without loss, while deadline withdrawal counts unfinished installations. Commands ignoring cancellation remain in flight until they return. A deadline during caching cannot undo completed verification, and an ordinary cache read failure does not turn a verified site into lost work. Health reads only metadata, including while result or cache locks are occupied, and retains confirmed losses after recovery. Discovery is separate check work. auto_block.waiting reports scan, direct-block, firewall-flush and startup observation calls waiting for shared state, with no fixed waiting capacity. auto_block.active reports the single state owner through firewall operations, state writes and cleanup. lag_basis: operation_progress times the current operation within a batch; advancing batches do not degrade solely because their total duration exceeds a minute. One minute without progress degrades the active row and any waiting callers. A free state slot with no admission for one minute also reports lag. Returned direct-block or flush errors, failed state reads and writes, and abnormal exits count once per call; protected-address refusals do not. Known write failures are recorded before readback and diagnostic output. Known errors remain visible during later cleanup, and the common loss threshold and recovery policy apply. These rows count state-lock callers, independently of persisted per-IP retry records. Health snapshots use memory only and cannot wait for the state lock or I/O.

auto_block.pending reports durable retry records, including records loaded at startup when automatic blocking is disabled. Its capacity is the retry admission limit of 1,000 records. Records selected for a cycle remain in flight until their state-file outcome is known; newly requeued records also remain in flight until persistence finishes. auto_block.candidates reports distinct IPs admitted to the current cycle, with no fixed capacity. Candidate and record counts describe different stages and must not be added together. A candidate remains visible through its firewall callback, subsequent bookkeeping and durable requeue.

Pending age uses its original queue timestamp, or first observation when that timestamp is absent. An eligible record receives a timestamp on its first requeue; records without check identity are withdrawn under the current block policy. Refreshing the reason, check or severity does not reset the original queue age. Normal quota waits stay healthy within the existing two-hour retry lifetime; older waiting records report backlog_lag. Active record and candidate work use one minute without operation progress, so advancing batches can run longer without a false warning. These measurements add no retry scheduler and do not change the hourly quota, expiry or overflow policy.

An unsuccessful attempt whose retry survives reports retry_failed without a loss. Successful blocks, dry runs and expected refusals remain completed even if later bookkeeping fails. Withdrawal under the current check policy or login-blocking setting is an expected refusal and adds no loss. Persisted record identities include check and severity, so a refused record cannot acknowledge a different eligible retry after a failed write. Confirmed removal of expired, invalid or overflowed records counts as pending loss; a fresh candidate that neither completes nor survives on disk counts as candidate loss. Existing duplicate coalescence adds no loss while a retry survives. The common loss threshold and recovery policy apply.

State read or write failures report state_io. Failed writes are read back because an error after rename can leave the new state in place. If that read also fails, depth_unavailable and dropped_lower_bound expose the uncertainty. A later successful read restores measured depth; lifetime loss totals remain lower bounds when earlier outcomes could not be established. New work with known outcomes still contributes exact losses. Completed candidate history is released after settlement, and health snapshots never read files or wait for firewall callbacks.

auto_block.cleanup reports distinct IPs awaiting bookkeeping cleanup after a successful firewall flush. It observes saved cleanup retries at startup and admits the flushed engine entries before reading the tracker. Cleanup also includes tracked blocks, with duplicate IPs counted once. Work remains in flight through store removal, threat-record cleanup and the final state-file outcome. This row has no fixed capacity and is independent of the block-candidate stages.

Cleanup waiting age starts at first observation with lag_basis: deferred_checkpoint. Age alone does not degrade health: cleanup retries run on the next explicit flush, without a background retry deadline. One minute without active operation progress reports processing_lag. Failed cleanup whose retry survives reports retry_failed; an old tracker block can retain a retry even when saving its cleanup marker failed. Completed cleanup stays acknowledged across failed saves, while a newly blocked IP creates fresh cleanup demand. Recovery first establishes the known block baseline. Cleanup of a later block has an exact outcome even when older cleanup history remains uncertain.

Cleanup state and snapshot failures report state_io. An unreadable tracker sets depth_unavailable and dropped_lower_bound; speculative waiting records are not reported as measured depth. The last bounded batch retains its identity for later read recovery. A subsequent unreadable flush replaces that history with its own batch. A readable state restores measured depth and counts confirmed unfinished work that no longer has a retry source. An incomplete pre-flush engine snapshot marks lifetime losses as a lower bound without inventing a missing count. Health reads memory only. These measurements preserve the existing flush policy, returned errors and cleanup retry behavior.

attackdb.events reports events awaiting persistence, including historical imports. The buffer has no fixed capacity. Waiting age starts at admission, independently of an event’s timestamp. A detached batch remains in flight through writes, close and error logging; newly arriving events stay in the waiting count. Processing age uses lag_basis: operation_progress and resets when a write returns, so a progressing batch does not appear stalled solely because of its total duration. Waiting age or absent write progress of one minute degrades health. Completed JSONL records accepted by the file writer and successful database writes count as persisted. Buffered data alone does not. Confirmed unwritten events count as losses, including interrupted work left buffered or unencoded, and returned encoding failures. These losses are visible before cleanup; close errors or abnormal I/O exits preserve uncertainty with dropped_lower_bound and report persistence_uncertain for one minute, ahead of a backlog or a stalled write, as the record queue already does. The common loss threshold and recovery policy apply to confirmed losses. These measurements preserve the existing write, retention and shutdown policy; they do not add retries or promise storage durability beyond the writer’s result. Memory-only databases have no persistence backlog. Health reads queue memory, independently of database state locks and I/O.

attackdb.records reports distinct IP records awaiting a saved update or deletion. Repeated changes to one IP coalesce before the snapshot; a mutation during an active write remains separate pending work. A successful deletion can also satisfy a repeated deletion. Loaded records needing normalization and expired records needing removal enter the same queue. Retained scoring records are not counted as pending persistence. There is no fixed capacity.

Waiting age starts at the first pending change. The active snapshot stays in flight through writes, error logging and retry bookkeeping. Processing age uses lag_basis: operation_progress and resets at each returned store write or delete, at the commit of their batch, or at the complete flat-file write result. The store commits one flush as one batch, so a failed commit retries every record and deletion in it. A refusal affects only that operation; a successful save cannot clear a failed deletion for the same address, or vice versa. A returned commit refreshes processing age even if every operation was refused. One minute without progress or with waiting work degrades health. Returned failures report retry_failed while the retry remains queued, preserving its original age without counting it as lost. Successful retries clear that condition. A failed shutdown flush retains the in-memory retry; the stopped background saver does not schedule another attempt.

An interrupted operation counts confirmed abandoned demand only when no pending update or deletion survives. Unreturned write outcomes report persistence_uncertain for one minute and retain dropped_lower_bound afterward. This includes a transaction interrupted before its callback starts or after all its operations were refused. Known losses remain counted through recovery. Health reads queue metadata only, without waiting for database state or storage locks. These measurements preserve the existing persistence, retry and shutdown behavior; they do not make retained in-memory retry state durable across process exit.

block_digest.records reports the actual buffer of watched-country block records, bounded to 5,000 with drop-oldest overflow. The buffer folds into the daily summary at each configured interval, before each summary send and at shutdown. Age starts when a record enters the buffer, independently of the block timestamp. Normal batching and a full buffer alone do not report a stall: waiting becomes overdue one minute after the configured interval. A detached fold remains in flight; one minute without progress reports processing_lag. The row uses lag_basis: operation_progress. Records arriving during a fold belong to the next one. Expected per-IP coalescing is successful completion, while an abandoned fold counts its discarded records as losses.

block_digest.email and block_digest.webhook report configured destinations in notifications, with capacity_unavailable. Each eligible destination owns one notification before delivery starts, including live alerts and the daily summary, which is sent even when empty. Default delivery follows current alert settings at admission, attempt and interrupted cleanup. Disabled destinations add no new waiting work or loss; destinations enabled during an earlier send are still handled. Explicit destination selection applies only to live alerts and keeps its error when that alert channel is disabled. Daily summaries use all enabled alert channels except phpanel. One minute waiting or active reports lag. A returned sink error counts one lost notification and stays in flight through error logging. A daily summary sink, or the summary rendering, that panics fails the same way and is logged with the panic, so it cannot stop the daemon. If an operation exits without returning, its attempted delivery reports delivery_uncertain for one minute and retains dropped_lower_bound; a remaining destination that was never attempted counts one confirmed loss. Known losses use the common warning threshold and survive recovery in lifetime totals.

Live delivery remains best effort. Daily summaries save their window before delivery and retry failed destinations; successful destinations are saved separately. A failed destination retries after one minute, doubling up to one hour, and counts one lost notification for that summary however often it is retried. Between attempts that destination keeps one waiting notification, so its row reports backlog_lag while the outage lasts; switching that destination off releases it within a minute. When the next summary is due, a still unfinished one is dropped, logged once and counted once, and the new one goes to every destination. A crash between remote acceptance and saving success can repeat a delivery. Records retained after the collector stops report consumer_stopped. Health reads metadata independently of collector state locks, country lookups and delivery callbacks. The rows are always present: every daemon runs the collector for the daily summary, and without alerts.block_digest.enabled the records buffer stays empty. The summary and customer block payloads are described under Summary and customer block webhooks.

state.pending reports findings parked for the next startup, bounded to the newest 10,000 findings. The store observes existing parked work when opened. Depth is the last confirmed file contents; in-flight findings include incoming appends and cleared batches still in replay. The bound applies to the stored batch, not concurrent callers or an already detached replay. Overflow and new findings that fail to reach storage count as losses. Duplicate occurrences remain separate findings.

Waiting age starts at admission or startup observation, independently of the finding timestamp. Surviving findings keep their age through later appends; evicted findings no longer determine it. Waiting uses lag_basis: deferred_checkpoint: a parked or full batch awaiting restart does not itself report a stall. state.pending_operations measures callers waiting for the state lock and active persistence or replay, in operations with no fixed bound. One minute waiting or without operation progress reports lag. Actual read, write and clear results advance progress; replay stays active through dispatch.

Read, write and clear errors report state_io. Failed mutations are read back: a returned error after replacement does not imply the new findings were lost. Readback compares the recovered JSON payload, including internal subnet and spray-target data and its normal repair of invalid UTF-8, so repaired log text does not invent uncertainty or hide overflow. Unreadable outcomes set depth_unavailable and dropped_lower_bound. A later read restores measured depth, while lifetime losses remain lower bounds. Known encoding failures and overflow are counted even when other outcomes are unknown. Indistinguishable retained payloads keep the earlier age after a failed write. An interrupted write or clear marks disk state unknown before releasing the state lock. Interrupted replay retains measured disk depth but cannot claim an exact loss for a partially dispatched batch; uncertainty reports persistence_uncertain for one minute. Confirmed losses use the common warning threshold and remain in lifetime totals after recovery.

Parked findings add optional storage-only keys to the public finding fields so subnet response and credential spray correlation keep their inputs at restart. Subnets are in the public cidrs field and also keep their storage-only copy, so a daemon rolled back after a restart still replays them. Older parked records still replay, and older daemons ignore the added keys. Process-local delivery markers are cleared on read, so stored findings receive a fresh evaluation.

Append order, returned errors and clear-before-replay policy are unchanged. These observations add no retry or extra replay. Health snapshots read metadata only and do not wait for state locks, files or dispatch callbacks.

incident.persist.waiting reports immutable incident snapshots waiting for the ordered writer, with no fixed waiting capacity. incident.persist.active reports the single occupied writer. A writer or free-slot admission stalled for one minute reports lag. Failed writes and abnormal callback exits count once; abandoned bulk snapshots count as waiting losses and release their ordering slots for later writes. Callbacks remain owned through cleanup and error logging. Store failures retain in-memory transitions and the existing warning log. incident.persist.deferred reports coalesced bookkeeping waiting for a later mutation or explicit flush, including shutdown. Its oldest age uses lag_basis: deferred_checkpoint; age alone does not degrade health because there is no periodic flush deadline. Full snapshots supersede earlier bookkeeping; restoration and retention discard the affected markers. Memory-only correlators have no persistence work. All three rows read memory independently of state locks and database I/O. The common loss threshold and recovery policy apply to writes.

checks.file_index.waiting reports live scans waiting for the shared baseline slot, with no fixed waiting capacity. Each wait keeps its shorter parent deadline or the file-index check budget; a free slot with no admission for one minute also reports backlog lag. checks.file_index.active reports the single occupied slot. Walking and file analysis use the check’s original execution deadline (normally 15 minutes) or the shorter parent deadline. Setup, persistence and cleanup each have a separate one-minute budget. A healthy long scan alone does not report a full queue. Actual filesystem work remains visible after cancellation until it returns. Deadline withdrawal, incomplete walks, unreadable PHP content, failed executable metadata or state reads and writes, and abnormal exits count once per live scan. Executable entries disappearing during enumeration and explicit cancellation alone add no loss. The common three-loss warning threshold applies, and recovery retains total losses. A successful late baseline commit adds no loss. Findings and shrink protection keep their existing behavior. Force-file-index audits bypass this stateful slot and remain owned by the check execution row. Snapshots read memory only. checks.reputation_queries publishes accepted lookups immediately after quota reservation, including while refused lookups receive fallback scoring. It follows the accepted work through HTTP, response cleanup, buffered results, supplemental scoring and cache or quota-backoff storage. Each scan has at most five requests; concurrent scans have no fixed combined capacity. HTTP uses the client’s timeout, while cleanup, local result handling and persistence have separate one-minute budgets. Supplemental scoring uses the combined timeout of its enabled sources or the shorter parent deadline. Buffered results follow their own active consumer’s deadline; another scan cannot hide their delay. Query failures, failed storage and abnormal exits count once per result. Quota responses and refusal before dispatch remain expected outcomes with the existing quota health warning. Parent cancellation does not cancel the existing HTTP requests; they remain visible until response handling and storage finish. Successful late results add no loss. Health reads memory only and retains losses after recovery. Local feed matches, cache hits and serial pre-query discovery do not create query jobs. A failed cache write retains the findings already produced. checks.dispatch reports pending checks and occupied runner wrappers across host and account scan batches. Concurrent batches have no fixed global waiting limit, so capacity is unavailable. Lag uses consumer_progress within each batch: pending work degrades after one minute without progress when that batch has a free worker slot. A busy pool may keep working within each check’s own deadline. Setup and result handling have a separate one-minute budget; they cannot borrow a heavy check’s longer deadline. A wrapper panic or abnormal exit counts a loss, as does a parent deadline before a check can start. Explicit cancellation withdraws queued demand without loss. Execution failures belong to checks.executions; dispatch tracks the surrounding scheduling operation. Reading status never takes a scan, context or database lock. scans.jobs reports eight waiting full-scan jobs and one worker. Work remains visible through enumeration, scanning, remediation, persistence and cleanup. Its consumer_progress lag follows that worker and its own check batches: healthy check deadlines allow long scans to proceed, while an overdue child cannot be hidden by another child’s progress. Enumeration, result handling, individual file actions and database operations each have a one-minute progress budget. Thirty seconds at full waiting capacity also degrades health. scans.admission separately reports callers waiting for admission and writing their initial job record, with a one-minute lag budget and unavailable capacity. Status reads only memory; operator polling does not advance the worker’s clock. Queue refusal, failed job operations and abandoned work count once per job. Configured finding-history truncation, explicit cancellation and successful shutdown draining do not count as loss. A terminated worker refuses new jobs. At startup, both queued and running records from the previous process become errors with reason daemon_restarted; their lost requests count in job health. email_av.scans reports antivirus engine work through execution and both result handoffs. Timed-out engines remain in flight until they return; buffered results remain queued until the message scan consumes them. Execution lag uses the original per-part deadline, while result waiting and processing each have a one-minute budget. Moving a result between buffers preserves its waiting age. Concurrent messages and engines outliving timeouts have no fixed global limit, so capacity is unavailable. Timeouts, errors and abandoned work count once per engine scan; later failures cannot count the same scan twice. Detections and unavailable engines are normal outcomes, and mail verdict behavior is unchanged. Status reads only memory, and watcher restarts reuse the same engine health. php_taint.requests follows callers waiting for the worker lock, active requests and pipe operations that outlive their callers. Waiting, setup, reply handling and cleanup each have a one-minute lag budget. Active worker communication uses its configured timeout or the shorter caller deadline; a buffered reply cannot borrow a long worker timeout. The request stays in flight through reply decoding and cleanup, and until any outstanding pipe operation finishes. Capacity is unavailable because callers have no fixed global waiting limit. Worker failures, timeouts, breaker refusals and abnormal exits count once per request; known failures are recorded before cleanup. Oversize input, caller cancellation and refusal after an intentional stop do not add losses. Status reads memory without the worker lock, and shutdown retains the supervisor’s health evidence. Recovered analyzer panics also count as failed work when delivered in a valid worker reply. The report and worker reuse policy remain unchanged. yara.requests follows scans waiting for the single YARA scan worker, the active scan and the rules load of a freshly started worker. Capacity is unavailable because callers have no fixed global waiting limit. Worker failures and breaker refusals count once per request. central.actions reports 1,024 waiting central-intelligence actions and one running action. Backlog remains visible while the signed feed refreshes; processing time includes the action handler and its evidence delivery. Overflow, action failure and abandoned shutdown work count as losses. A protected-address refusal or absent firewall engine completes the queue task without claiming that a block happened. Challenge tasks measure delivery to the challenge list; they do not measure its later file or firewall writes. Shutdown cancels feed refreshes, waits for an already-running action and discards the remaining queue. Captured dispatch hooks refuse later work and preserve its loss count after shutdown. Logging does not reset loss totals. bot_verification.requests reports 256 waiting bot-identity requests and one running verification. Duplicate requests share their original waiting age. Running time includes DNS lookups and the cache write. Queue overflow, DNS timeouts or transient failures, failed cache or missing-PTR record writes and abandoned shutdown requests count as losses. Missing PTR records, unknown bot identities and definitive positive or negative answers remain expected outcomes. A source with a recorded missing PTR is not queued again until its one-hour suppression lapses; suppressed requests do not count as losses. For a day after the lookup, lapsed records prevent renewed pending grace without blocking DNS retries. Queue health does not turn a resolver failure into a spoof finding. Shutdown cancels DNS, waits for any active cache write and discards waiting work. Late submissions are refused, pending keys are released and loss totals remain available. abuse_reporting.ingress measures reports awaiting durable storage, with a capacity of 10,000 or the configured spool limit when smaller. Running time includes persistence to every configured target. Memory overflow, incomplete persistence and work abandoned by a worker failure count as losses; one source report counts once even when several targets fail. Outbound delivery can delay memory work, and that backlog remains visible. The queue stays open through the daemon’s final finding flush. Reporter shutdown then closes admission and persists every accepted report before closing the spool; a failed write does not discard unrelated waiting reports. Captured hooks refuse later reports. This row measures the memory queue, not reports already retained on disk for retry; normal shutdown does not count those durable reports as lost. abuse_reporting.spool measures durable reports per destination, with the configured spool capacity. Reports already being sent remain in flight through the database acknowledgment. Failed admission, discarded records and reports the collector refuses as invalid count as losses; a delivery retry, failed acknowledgment or normal shutdown retains the report and does not count it as lost. A failing destination pauses only its own reports for the rest of a pass. An evicted or refused record counts as lost only if no send was acknowledged during this process. An active send settles that count when it finishes; a failed retry cannot undo a prior acknowledgment. lag_basis: observed_age means waiting age starts when this process first sees the record. Existing records start at spool open; retries keep that age. Doctor labels this as observed_lag, since time spent waiting before restart is unknown. Waiting or running work degrades after two minutes, allowing for the normal one-minute delivery interval. The common full-queue and recent-loss thresholds also apply. spool_io names failed admission, read or acknowledgment operations; delivery_failed names a failed or panicking sender and stays set while any destination failed in the last pass or is waiting for its retry deadline. Throttling without progress remains visible here even between log warnings. These states clear when the affected operation recovers. Health reads remain independent of database writes and outbound requests. Reopening with a smaller cap preserves existing records until the next admission applies the configured overflow policy. phpanel.spool measures the durable panel webhook queue, capped at 100,000 records per state directory. In-flight work includes writes waiting to commit and deliveries awaiting database acknowledgment. Retried findings keep their original waiting age; a failed attempt is not a lost finding while its record remains durable. An overflowed record counts as lost only when no send was acknowledged during this process; an active send settles that count when it finishes. Malformed findings, and findings the panel refuses with HTTP 400, 413 or 422, count once when removed from delivery, and the next record is sent at once. Their bounded diagnostic archive is retained history, not pending work. A long run of refusals with nothing accepted between them looks like a panel refusing every request, so from then on refused findings stay queued and are retried as a delivery failure until the panel accepts one. This hold survives config reloads and daemon restarts. Only a 2xx response from the configured endpoint acknowledges a finding; redirects are not followed. Every other failure is retried in order with backoff. Waiting age uses the persisted enqueue timestamp after restart. Boundary records absent from the accounting rebuilt at open are adopted with the same timestamp, so retrying them does not reset their age. Missing, damaged or future timestamps are timed from queue open or adoption. A minute of waiting or processing degrades the row; delivery and database errors remain visible until the affected operation succeeds. Health reads use memory only, so a stalled database or collector cannot block status. Disabling delivery preserves the durable backlog and cumulative loss; enabling it resumes the stored work. Stopped queue instances refuse late admissions. actionlog.writes measures the 64 process-wide action-log write slots. A sink write that outlives the caller’s 250ms wait budget stays in flight until the sink and any panic reporting finish; a batch of actions serialising behind one file lock is normal, and only a minute without the write returning degrades the row. A caller deadline after admission does not count as loss, since the sink may still record the action. Refused admission, sink errors and panics count as lost records, once per record. Changing or disabling the sink preserves outstanding work and cumulative loss. Health reads do not wait for the sink. Normal writes still complete before the caller returns; saturated or stalled recording keeps the existing caller budget. events.deliveries aggregates live event-stream subscribers in one row without client identities. Capacity sums their buffers (64 findings per daemon stream); depth counts waiting deliveries and in-flight work includes encoding, writing and flushing. One full subscriber can degrade this row even while others drain. The common one-minute lag, 30-second fullness and recent-loss thresholds apply. Overflow, encoding errors and failed streams count as losses; a failed stream also counts its abandoned buffered events. Normal request cancellation and server shutdown withdraw pending demand without adding losses. Cumulative loss survives subscriber removal, and an outstanding write remains visible until it returns. Closing the bus preserves buffered work for consumers still draining it. HTTP writes retain their three-second deadline; successful delivery here means the write and flush returned, not that the remote application acknowledged it. Each active BPF backend also exposes a .kernel row. depth_unit: bytes labels ring occupancy and capacity. lag_basis: consumer_progress means lag_seconds measures time without observed consumption while data remains, starting at the first sample that sees pending data. It is not an event age. One minute without progress degrades the row; recently observed reservation failures use the same loss threshold as userspace queues. Kernel loss counters are separate from decode failures and userspace admission loss. After a reader unmaps its ring, depth_unavailable: true and lag_basis: unavailable prevent zero fields from claiming an empty live ring. The last counter sample adds submitted records the reader never consumed. Submission accounting precedes publication, so a fast reader cannot consume an event before it has been counted. dropped_lower_bound: true marks this final shutdown total: kernel detachment can leave callbacks finishing after the sample. Doctor prints dropped>=... for that bound. A counter lookup failure or an incoherent occupancy reading marks the measurement unavailable and degrades the row as measurement_unavailable once it lasts half a minute, so one artefact of reading a live ring raises nothing; a failed final sample degrades at once and remains degraded. A stopped reader is reported instead of the measurement artefacts it causes. Invalid occupancy cannot inherit fullness or consumer stall alarms from an earlier sample; independently counted losses remain visible. The required kernel suite fills the shipped connection program’s ring with real non-root connect calls and checks reservation loss and retained output. fanotify.kernel and spool.kernel report pending notification records and use the same consumer-progress lag measurement. Their group capacity is not exposed by the kernel: capacity_unavailable: true marks it as unknown, and doctor prints depth=N/unknown records. The current system queue limit may differ from the limit captured when the watcher was created. Their loss totals are lower bounds: each overflow record proves at least one loss, and shutdown adds the records known to be unread before closing the descriptor. Events can still arrive between that sample and close. An unavailable pending-record measurement degrades the row once it lasts half a minute; the reading taken at close degrades at once and remains degraded. Records the kernel has already dropped are reported ahead of an unreadable depth. Closing a descriptor leaves known zero occupancy. fanotify.reader and spool.reader track batches after a kernel read, until all records have been dispatched or filtered. A stalled batch remains visible even when the kernel queue is empty. Reader losses count batches interrupted by consumer failure, separately from kernel-record losses. Spool replacements retain loss and running-batch evidence, while the new descriptor starts its own occupancy and progress measurements. Health reads, event reads and permission responses cannot use a descriptor after close. The required kernel suite verifies pending records, stalls, drain recovery and unread shutdown loss using real fanotify events.

forwarder.kernel and phprelay.kernel report inotify backlog in bytes, because records include variable-length filenames. Capacity is unknown and lag follows observed consumer progress. Overflow markers each prove at least one lost event. Closing with unread bytes adds one further known loss; the byte count does not reveal the number of discarded events. These totals use dropped_lower_bound: true. forwarder.reader and phprelay.reader track the running read batch, including synchronous callbacks. Failed callbacks count as interrupted batches. PHP relay replacements retain earlier losses with fresh occupancy measurements. Descriptor reads, watch changes, polling and close share one lifetime guard.

phprelay.index.persistence reports the message attribution writer’s 4,096 waiting slots. Writes remain running from channel receipt through the pending batch and its database transaction. Batches contain at most 256 writes, also during an explicit flush. Each flush covers the backlog present when the writer accepts it; concurrent arrivals cannot extend that flush indefinitely. Refused submissions, encoding failures and writes in a failed transaction count as losses. Failed writes are settled before emitting their error finding, so a blocked reporter cannot hide the loss. A failed transaction counts each of its writes once and does not prevent subsequent batches from committing. Shutdown closes admission and drains accepted writes; later submissions count as refused work. The existing persistence dropped metric counts refused submissions, while queue health also includes writes that fail after admission.

Some rows carry advisory. Their work is best effort: a client that stops reading its event stream, an unreachable panel asked for an optional annotation, and expired process context reads all lose detail around findings that are still detected, stored and delivered. An advisory row reports its own degradation with the same evidence, but leaves the host status and security posture unchanged, warns instead of failing csm doctor, and raises no notification.

php_shield.eval_sites is advisory grading work. It reports running ownership walks and completed results still held for callers. A timeout or refused lookup counts as a lost proof, while the finding stays High. A completed walk releases its single admission slot before publishing, so later events do not wait for that caller. Concurrent callers can hold completed results, making the aggregate capacity unavailable. Health reads perform no filesystem lookup.

Each degraded row names its condition: backlog_lag for the oldest waiting item past its budget, processing_lag for the oldest running item, queue_full for capacity held continuously, dropped_work for recent losses, consumer_stalled for a sampled queue whose consumer stopped advancing, reader_stopped for a released kernel descriptor, measurement_unavailable for a reading that stayed unavailable, and persistence_uncertain for a write whose outcome is unknown. The owner-specific conditions retry_failed, source_io, spool_io, state_io, delivery_failed, delivery_uncertain and consumer_stopped are described with their rows above.

Doctor names each age by what it measures: observed_lag for an age taken at observation, consumer_stall for consumer progress, operation_stall for the current operation, deferred_age for work parked until a restart, and lag=unavailable where no age exists.

A queue becomes degraded after three losses in a minute, thirty seconds continuously full, or a minute waiting or processing. These are operational alert budgets, not measured throughput guarantees. Health is computed directly from the counters, independently of finding delivery. The daemon records and dispatches protection_queue_degraded at most once per queue every five minutes while pressure remains, then one protection_queue_recovered event. The five-minute bound spans recoveries, so a queue that clears and degrades again inside the window stays degraded in status without a second event, and no recovery event follows a degradation the bound suppressed. These are CSM health events and do not feed account-compromise correlation or automatic response. Recovery preserves cumulative loss evidence; restarting a spool watcher also preserves it. Restarting the daemon resets the counters.

Inspect worker errors and CPU, memory and I/O pressure when a queue degrades. Reduce competing bulk work and confirm the queue drains and recent losses stop. This surface currently covers finding delivery, file and spool kernel readers and scanners, recovery scans, staged package verification, dropper processing, BPF queues, process context enrichment and deadline reads, mail-log delivery, forwarder and PHP relay notification queues, and PHP relay index persistence; other bounded queues remain in the roadmap.

Summary and customer block webhooks

The generic alert webhook gets two events of CSM’s own: the daily summary and the known-customer block alert. A phpanel webhook gets neither. Each is one JSON object: text holds the email body, for chat tools, and csm holds the data.

{
  "text": "[CSM] host.example.com - daily summary 2026-06-10\n...",
  "csm": {
    "event": "daily_summary",
    "host": "host.example.com",
    "version": "X.Y.Z",
    "summary": {
      "since": "2026-06-10T08:00:00Z",
      "due": "2026-06-11T08:00:00Z",
      "checks": {"wp_login_bruteforce": {"CRITICAL": 12}},
      "blocks": 3,
      "block_kinds": {"wp-bruteforce": 2, "netblock": 1},
      "muted": 4,
      "sources": [{"ip": "203.0.113.5", "count": 9, "country": "NL"}],
      "watched": {"total": 0, "by_country": {}, "by_category": {}}
    }
  }
}

daily_summary fields:

  • csm.event is daily_summary; csm.host and csm.version name the host and the CSM version.
  • summary.since is the start of the window and summary.due its scheduled send time, both RFC 3339.
  • summary.checks counts summary findings per check, then per severity (CRITICAL, HIGH, WARNING).
  • summary.blocks counts applied blocks; summary.block_kinds counts them by kind, such as wp-bruteforce, mail-bruteforce, netblock, permanent or other.
  • summary.muted counts summary findings of checks in alerts.email.disabled_checks.
  • summary.sources lists the 10 addresses with the most summary findings: ip, count, and country when GeoIP knows it.
  • summary.watched counts watched-country blocks: total, by_country and by_category. It stays empty without alerts.block_digest.enabled.
  • summary.customer_blocks lists the applied blocks that hit known customers, as customer block objects (below), at most 50; summary.customer_blocks_more counts further blocks. Both are left out when empty.
  • summary.would_block lists dry-run and observe-mode verdicts that would have blocked known customers, as customer block objects, at most 50; summary.would_block_more counts further verdicts. Both are left out when empty.

Each count map holds at most 512 keys; later keys count under other.

{
  "text": "[CSM] host.example.com - automatic block hit a known customer: 203.0.113.5\n...",
  "csm": {
    "event": "customer_block",
    "host": "host.example.com",
    "version": "X.Y.Z",
    "customer_blocks": [{
      "target": "203.0.113.5",
      "reason": "SMTP brute force from 203.0.113.5",
      "check": "smtp_bruteforce",
      "customers": [{"ip": "203.0.113.5", "kind": "cpanel", "account": "alice", "at": "2026-06-10T08:15:00Z"}]
    }]
  }
}

customer_block fields:

  • csm.event is customer_block; csm.host and csm.version as above.
  • csm.customer_blocks lists the blocks of the alert. Each has target, the blocked address or network; reason, the block reason or, for a notice without one, its message; check, the check that caused the block, left out when unknown; customers; and more, the covered customers not listed, left out when 0.
  • Each customers entry is one known customer, the most recent first: ip (an IPv6 customer as its /64), kind (cpanel, webmail, smtp, imap, pop or ftp), account (left out when the login named none) and at, the last successful login in RFC 3339.

The watched-country live alert, event block_live, keeps its own shape: there csm.blocks lists blocked addresses.

GeoIP

GET  /api/v1/geoip               IP geolocation (?ip=&detail=1)
POST /api/v1/geoip/batch         Batch GeoIP lookup (body: {"ips":["192.0.2.1"]}, maximum 500)

Threat Intelligence

GET  /api/v1/threat/stats        Attack stats, type breakdown, hourly trend
GET  /api/v1/threat/top-attackers Top attacking IPs with GeoIP (?limit=&spam_only=1)
GET  /api/v1/threat/ip           IP threat lookup (?ip=)
GET  /api/v1/threat/events       IP event history (?ip=&limit=)
GET  /api/v1/threat/whitelist    Whitelisted IPs
GET  /api/v1/threat/db-stats     Attack database statistics
POST /api/v1/threat/block-ip     Block IP for 24 hours
POST /api/v1/threat/block-ip-permanent Block IP with no expiry
POST /api/v1/threat/whitelist-ip       Permanent whitelist
POST /api/v1/threat/temp-whitelist-ip  Temporary whitelist (with expiry)
POST /api/v1/threat/clear-ip           Clear IP from attack database
POST /api/v1/threat/unwhitelist-ip     Remove from whitelist
POST /api/v1/threat/bulk-action        Bulk block (24h or permanent) / whitelist across many IPs

/api/v1/threat/stats includes mail_only_ips, the attack database records that hold only spam evidence, mail_only_new_24h, those first seen in the last 24 hours, and mail_evicted, those dropped to keep the spam evidence budget since the daemon started.

In /api/v1/threat/top-attackers, ip is an IPv6 /64 in CIDR form, such as 2001:db8:1:2::/64, for a record that holds the spam evidence of a network. /api/v1/threat/ip takes an address only. Its attack_record.spam_network names the /64 when the spam evidence shown was recorded for the network. Equivalent IPv6 spellings use the same address record and return the canonical address. The stronger spam evidence wins; ties use the network.

The 24 hour endpoint returns 409 Conflict when the address already has a permanent or longer firewall block. Unblock it explicitly before shortening its lifetime. Bulk actions use action: "block", "block_permanent", or "whitelist"; refused blocks are excluded from count and explained in warnings. Repeated canonical IPs count once. Permanence cannot be selected through an extra request field.

Bulk undo restores the original per-IP firewall deadlines. A later change to any target invalidates that undo action; it cannot restore dismissed evidence or replace the later decision.

Firewall

GET  /api/v1/firewall/status         Config, blocked/allowed counts
GET  /api/v1/firewall/allowed        Whitelisted IPs
GET  /api/v1/firewall/subnets        Blocked subnets
GET  /api/v1/firewall/audit          Firewall audit log
GET  /api/v1/firewall/check          Check if IP is blocked (?ip=); admin or unban scope
POST /api/v1/block-ip                Block an IP
POST /api/v1/unblock-ip              Unblock an IP
POST /api/v1/unblock-bulk            Bulk unblock IPs
POST /api/v1/firewall/allow-ip       Allow an IP
POST /api/v1/firewall/remove-allow   Remove IP from allow list
POST /api/v1/firewall/deny-subnet    Block subnet
POST /api/v1/firewall/remove-subnet  Remove subnet block
POST /api/v1/firewall/flush          Clear all blocks
POST /api/v1/firewall/unban          Unblock IP + flush cphulk; admin or unban scope
POST /api/v1/firewall/cphulk-clear   Flush cphulk bans only

The audit log reports each timestamp as an RFC 3339 instant in UTC and the block or allow lifetime as duration_seconds. Blocked addresses, subnets and allow rules carry expires_at, left out for a permanent entry.

ModSecurity

GET  /api/v1/modsec/stats              WAF statistics (read scope). Accepts ?window=1h|6h|24h, ?severity=warning|high|critical.
GET  /api/v1/modsec/blocks             Blocked requests log, aggregated per IP, with resolved source country (read scope). Accepts ?window=1h|6h|24h, ?severity=warning|high|critical.
GET  /api/v1/modsec/events             WAF event details with resolved source country (read scope). Accepts ?window=1h|6h|24h, ?severity=warning|high|critical.
GET  /api/v1/modsec/rules              Loaded rules list
POST /api/v1/modsec/rules/apply        Apply the set of disabled rules and reload
GET  /api/v1/modsec/rules/escalation   Rule IDs excluded from firewall escalation, sorted
POST /api/v1/modsec/rules/escalation   Exclude one rule from escalation or turn it back on

POST /api/v1/modsec/rules/escalation takes {"rule_id": 900112, "escalate": false}. The rule ID must be a CSM rule (900000-900999). escalate: false adds the exclusion and escalate: true removes it; other excluded rules are left as they are.

A failed rules reload reports rolled_back: true only when restoring the previous overrides succeeds. A rollback failure is reported in the response and audit log; inspect the overrides before trying another reload.

Rules & Suppressions

GET  /api/v1/rules/status        YAML/YARA rule counts, version
GET  /api/v1/rules/list          Rule files
GET    /api/v1/suppressions      Suppression rules
POST   /api/v1/rules/reload      Reload signature rules from disk
POST   /api/v1/suppressions      Add a suppression rule
DELETE /api/v1/suppressions      Delete a suppression rule by id

POST /api/v1/suppressions takes {"check", "path_pattern", "reason"}. The path pattern is a glob and must be valid. A rule that covers every path of a check, hiding all its findings and stopping their remediation, needs "all_paths": true and no path_pattern; an empty pattern without it returns 400. check must be a check name, not a pattern: letters, digits, _, ., : and -. A name that is neither a known check nor the check of a current finding is saved, and the response carries a warning saying the rule matches nothing yet.

Email

GET  /api/v1/email/stats         Email scanning statistics
GET  /api/v1/email/forwarders    Mail forwarder inventory with destination providers and local-copy flags (read scope)
POST /api/v1/email/forwarders/delete  Delete one destination of a forwarder on cPanel hosts through uapi, as the account that owns it; takes {"source", "destination"} as the inventory lists them; 404 when the inventory has no such pair, 500 when cPanel or command execution fails, 503 when unavailable (admin scope, CSRF)
GET  /api/v1/email/deferrals     Outbound deferral rollup by provider and sending IP with reason codes and last_seen, parsed from exim_mainlog; each provider has a sample error text; lines is the number of log lines read and window_start/window_end the times of the first and last of them; one read is reused for 60 s (read scope)
GET  /api/v1/email/queue-composition  Mail queue makeup: real vs null-sender bounce backscatter, frozen count, unreadable count, oldest age, top stuck recipients; 500 when exim could not list the queue, rather than an empty queue (read scope)
POST /api/v1/email/queue/flush-backscatter  Request removal of frozen null-sender messages from the exim queue on cPanel hosts; returns the count of targeted messages no longer queued, reports incomplete verification as 500, or returns 503 when unavailable (admin scope, CSRF)
GET  /api/v1/email/queue         One page of the exim queue on cPanel hosts, oldest first, each message with why it is stuck; page (default 1), limit (default 25, max 100), filter=all|frozen|backscatter, q (a substring of the sender or of one of its first 50 undelivered recipients) (admin scope)
GET  /api/v1/email/queue/{id}    One queued message with all undelivered recipients (up to 1,000), why it is stuck and the last 20 lines of its msglog; 404 when it is not queued (admin scope)
POST /api/v1/email/queue/{id}/{action}  retry, bounce (return to sender), delete or thaw one queued message; answers the outcome the queue shows afterwards (admin scope, CSRF)
GET  /api/v1/email/held          Forward copies held by the forward guard, each with its decoded subject and size (admin scope)
POST /api/v1/email/held/{id}/release   Re-inject a held forward copy to its external recipient (admin scope, CSRF)
DELETE /api/v1/email/held/{id}   Discard a held forward copy (admin scope, CSRF)
GET  /api/v1/email/checks        Email checks the Email page shows, each with its group kind ("" = never grouped) (read scope)
GET  /api/v1/email/groups        Server-grouped action rows (kind=compromised_account|spam_outbreak|auth_failure|queue_alert|malware) with from/to/limit (read scope)
GET  /api/v1/email/overview      Mail decision totals and 15-minute chart series for from/to (default the last 24 h), antispam mode, live queue and mail-related block counts, and Stalwart's filter verdicts: totals delivered, junk, filter_rejected, discarded and unmatched, series delivered, junk, filter_rejected and discarded, and filter_rejects_seen (read scope)
GET  /api/v1/email/rejects       Recent rejected mail (time, class, sender, recipient, source IP, detail), newest first, kept 7 days; class=content (spam and malware, default)|all|<class>[,<class>...], from/to (default the last 24 h), limit (default 50, max 200); totals has the permanent rejects of every class in the range (admin scope)
GET  /api/v1/email/senders       Messages each hosting account sent in from/to (default the last 24 h), busiest first, up to 100 rows, with total, unresolved and abuse_events, the outbound abuse events relay-abuse lists for the range, counted up to 5,000 like relay-abuse, with abuse_events_truncated true when that cap cut the count, and abuse_by_account, those events counted per account they name and per kind, before any list limit (admin scope)
GET  /api/v1/email/relay-abuse   Outbound PHP-mail abuse detections (spam outbreaks, high-volume scripts/accounts) with per-site script breakdown; from/to/limit; total counts up to 5,000 and total_truncated is true when that cap cut it, while truncated is also set when only limit cut the list (read scope)
GET  /api/v1/email/rspamd       rspamd status (up|down|off), antispam mode, version and rspamd's counters since it started (scanned, learned, actions, top 8 rules), read from its controller without a password and cached 60 s; available false with a reason when they cannot be read (read scope)
GET  /api/v1/email/quarantine    Quarantined email list, each with the size of its spooled body and its subject, decoded and cleaned as on held forward copies
GET  /api/v1/email/av/status     Email AV watcher status
GET  /api/v1/email/quarantine/{id}          One quarantined message
POST /api/v1/email/quarantine/{id}/release  Release a quarantined message (admin scope, CSRF)
DELETE /api/v1/email/quarantine/{id}        Delete a quarantined message (admin scope, CSRF)

For Stalwart, the overview’s totals.delivered, totals.junk, totals.filter_rejected, totals.discarded and totals.unmatched count its own filter verdicts in the selected range. unmatched counts messages whose verdict CSM could not tie to a sending address. Chart points use delivered, junk, filter_rejected and discarded, omitting these keys when zero. totals.rejected and the chart’s rejected key count rspamd decisions, not Stalwart filter rejects.

The top-level filter_rejects_seen is true when any retained bucket contains a Stalwart filter rejection or discard, even outside the selected range. False means no such event is retained, not that the filter rejected no mail. The Email page shows a dash and “none logged yet” for rejected mail until this flag is true. Older responses may omit the flag or verdict counts; the page treats the flag as false and missing counts as zero.

The mail.forwarders.delete.v1 capability advertises the forwarder delete endpoint. A 500 can follow a delete that partly ran, so read the inventory again before retrying. A domain’s default address (*@domain) is changed in cPanel.

The senders abuse count and relay abuse list apply the exact from and to bounds, including fractional seconds, even when requests share cached history. from is inclusive and to is exclusive. A cached history limit cannot let findings outside those bounds hide matching findings.

Email groups report handled: true only for auth failures whose complete, current history is answered by active blocks on every attacker address or covering subnet. Missing addresses, truncated or stale history, successful logins, and account sprays stay unhandled. Unreadable firewall state sets blocks_unavailable: true and leaves every group unhandled.

The overview’s counting_since begins when mail counting starts and survives restarts. Stops appear in gaps, ending when counting resumes even if loading saved counts needs a retry. Gap bounds round outward to whole seconds so short stops remain visible. uncounted_minutes rounds the total gap duration to the nearest minute, with a minimum of one for any positive duration.

The overview reads queue depth on every request. Queue detail is refreshed at most once per minute, even if the queue empties and refills. An empty queue discards its previous detail; new mail has detail_unavailable: true until the next listing. Failed listings can reuse detail for at most ten minutes.

The queue list and the overview share one queue listing, refreshed at most once per minute and after any queue action. total counts the messages that match filter and q; queued, frozen, backscatter (null-sender messages) and oldest_seconds describe the whole queue. listed_at is when that listing ran, so these counts can trail the overview’s live queue size. When no listing has succeeded in the last ten minutes the list answers unavailable: true with no items and no counts. A failed refresh cannot reuse an empty listing, and a positive queue count invalidates any earlier empty result. The next list request then lists the queue at once rather than waiting out the minute, once for each such refill. A message exim cannot read is listed with unreadable holding exim’s error and reason class unreadable; without its envelope it has no sender, recipients, size or frozen state, and only delete runs on it. An exim spool error without an immediately preceding message line makes the listing unavailable, including a missing-file error. Truncated entries and unrecognized output also fail instead of reporting an empty queue. The composition counts messages without envelopes only in total and unreadable, not in real or bounce. Backscatter flushing excludes every unreadable message and validates the whole listing before and after removal; a targeted message that becomes unreadable still counts as queued. Quoted recipient local parts remain intact, including spaces. Each message’s reason (class, text, raw) comes from its msglog tail; a partial first line is discarded. Up to 100 reasons are cached for a minute. The current page takes priority when older entries must be evicted. A message thawed since it froze shows retry_wait until its next delivery attempt.

Message reads and actions run exim as a systemd transient unit and never inside the daemon’s sandbox: without systemd-run the message route and the actions answer 503 and exim does not run. Each action first checks that the message is still queued; a missing-file reply must name the requested message. When it is not queued, the answer is 409 with outcome: "gone". Nothing ran, unless the message left during the action. retry answers attempt_started as soon as the delivery attempt has started. bounce and delete answer bounced and removed when the message left the queue and exim reported no error. When exim reported an error and the message left anyway, as when exim -Mrm finds it locked by a delivery that then completes, the answer is 409 with outcome: "gone". An unreadable message answers removed despite an error exit only when exim confirms its removal and the message is no longer queued. A failed or interrupted delete without that confirmation answers gone when another caller takes the message. thaw answers thawed when the queue shows the change, even if exim reported an error. Otherwise the answer is unchanged with an error. A thaw on a message that is not frozen answers unchanged without running exim -Mt. Every attempt is written to the UI audit log as email_queue_<action> with the message ID, the outcome and any error exim reported.

Held forward release requires a recorded sender and recipient, each a single mailbox without commas, whitespace or a leading dash. An explicitly empty sender is allowed for bounces. Ambiguous or missing envelope metadata causes a release error and leaves the message held without invoking sendmail. Release runs sendmail as a systemd transient unit with the message on its stdin, never inside the daemon’s sandbox: without systemd-run the release fails and the message stays held. The unit uses KillMode=process, so the delivery exim forks before it exits is not stopped with the unit and the message leaves at once. A release or delete first claims the held copy, so of concurrent requests one acts and the others answer that the copy is not held; a failed release puts the copy back. A copy whose release or delete the daemon never finished, as when it stopped mid-release, is held again on the next 15-minute refresh once the claim is at least five minutes old; such a release may already have been delivered. Recovery never replaces another held copy in either Maildir folder and makes each claim visible only once.

Deferrals are grouped under the provider whose MX host deferred them (Google, Microsoft and Yahoo hosts), so a business domain those providers host counts under them. The recipient’s domain decides when the log line names no MX or an unknown one.

The rspamd status follows live health probes. Dismissing an outage finding does not change it to up. Version display strings are bounded whether they come from the controller or the installed program. A read of rspamd’s controller finishes even when the request that started it goes away, so its result is cached for the other requests; a failed read is logged once per cache period.

Hardening

GET  /api/v1/hardening           Load last hardening audit report (admin scope)
POST /api/v1/hardening/run       Run hardening audit and save report (admin scope, CSRF)
POST /api/v1/hardening/apply     Apply automatic fixes for one section, then re-run the audit (admin scope, CSRF)

Scan Jobs

csm scan --full enqueues full-scan jobs that run inside the daemon and persist to the store. Jobs are report-only unless an account-scope request sets quarantine: true; server-wide jobs reject quarantine.

GET  /api/v1/scan-jobs              List full-scan jobs (read scope)
GET  /api/v1/scan-jobs/{id}         Job status and stored report (read scope)
GET  /api/v1/scan-jobs/{id}/findings
                                      Paginated findings for one job (?offset=&limit=, limit 500 by
                                      default and at most 5000; truncated marks more pages) (read scope)
POST /api/v1/scan-jobs              Enqueue a full-scan job (admin scope, CSRF)
POST /api/v1/scan-jobs/{id}/cancel  Cancel a queued or running job (admin scope, CSRF)

Verified Bots

GET  /api/v1/verified-bots        Configured verified-crawler allowlist plus live verification state (admin scope)
POST /api/v1/verified-bots/apply  Validate, apply, and reload an edited verified-bots list (admin scope, CSRF)

Actions

POST /api/v1/fix                      Apply fix for a finding
POST /api/v1/fix-bulk                 Bulk fix multiple findings
POST /api/v1/dismiss                  Dismiss one finding {key} or up to 500 {keys}, optional {contents}; returns undo_token if any were dismissed
POST /api/v1/scan-account             On-demand account scan
POST /api/v1/verify-finding           Re-check a single finding on demand (admin scope, CSRF)
POST /api/v1/quarantine-restore       Restore quarantined file
POST /api/v1/quarantine/bulk-delete   Bulk-delete quarantined files; returns count and the ids it could not delete
POST /api/v1/db-object-backup-restore Restore a dropped MySQL object from its db_object_backups record
POST /api/v1/test-alert               Send test alert through all channels
POST /api/v1/import                   Import state bundle (suppressions, whitelist)

fix and fix-bulk act on the file the stored finding names. A request may repeat that path in file_path, but a different path is refused, and a target is never a remediation root itself (/home, /tmp, /var/tmp, /dev/shm) or an account’s home directory.

verify-finding returns the verifier verdict in checked, resolved, demote, and detail. When that verdict also changes the stored finding, severity_change is demoted or restored. A demote verdict without severity_change means no stored severity changed; for example, the finding was already demoted or a scan replaced its snapshot while verification was running. Callers must not report a state change from the verdict alone.

Settings

GET  /api/v1/settings             List editable config sections
GET  /api/v1/settings/<section>   Read a config section (secrets redacted)
POST /api/v1/settings/<section>   Update a config section (safe fields reload, restart fields queue)
POST /api/v1/settings/restart     Request a daemon restart. Returns 202 with `started_at_token`
                                  for polling until the restarted daemon reports a new marker.
POST /api/v1/settings/firewall/tentative-apply  Save firewall config, restart, and arm rollback timer
GET  /api/v1/settings/firewall/rollback         Read pending rollback state
POST /api/v1/settings/firewall/confirm          Confirm tentative firewall changes
POST /api/v1/settings/firewall/revert           Revert tentative firewall changes now

Sections map to top-level config keys: alerts, auto_response, challenge, reputation, performance, infra_ips, sentry, etc. Writes persist to csm.yaml, re-record the integrity hash, and hot-reload where possible; restart-required changes are queued for /api/v1/settings/restart. Invalid field values return 422 and do not touch disk.

Fields marked file_only in the schema are shown but refused on write with a 422: anything that names a command, an executable, a file path, a socket or an environment variable the daemon acts on as root can only be changed in csm.yaml. Changing reputation.upstream.url or reputation.rspamd.url also returns 422 unless the same request enters the effective credential for that address again, and is refused outright when the credential comes from an environment variable. A credential entered in the request must match the resulting configuration after conf.d merging; an overridden replacement cannot authorize sending the stored credential to another address. Firewall tentative apply is restart-class by design: it snapshots the previous config, writes the new one, restarts the daemon, and auto-reverts unless the operator confirms before the timer expires.

Operator preferences

Per-operator state (UI density, timestamp display, default auto-refresh, saved filter views) is keyed server-side by SHA-256 of the auth token, so preferences follow the operator across browsers and devices without the daemon ever storing the raw credential. Capability flag: webui.prefs.v1. These endpoints require admin scope because they read or mutate operator-private UI state.

GET    /api/v1/prefs/user        Read this operator's UI preferences
PUT    /api/v1/prefs/user        Replace the prefs blob (CSRF on cookie sessions)
GET    /api/v1/prefs/views       List saved views; `?page=findings` filters by page
PUT    /api/v1/prefs/views       Upsert one view {page, name, params} (CSRF on cookie sessions)
DELETE /api/v1/prefs/views       Delete one view {page, name} (CSRF on cookie sessions)

Response shape for GET /api/v1/prefs/user:

{
  "density":       "comfortable",
  "timezone":      "local",
  "auto_refresh":  "on",
  "table_columns": { "findings-table": ["check","severity","when"] }
}

density is comfortable or compact. timezone is server, local, or an IANA-shaped zone string (e.g. Europe/Bucharest). auto_refresh is on or off. Server-side sanitisation drops any other value. Unset prefs encode as empty strings; the UI applies comfortable, local, and on defaults.

Response shape for GET /api/v1/prefs/views:

{
  "items": [
    {
      "name": "Critical SSH",
      "page": "findings",
      "params": { "severity": "critical", "check": "smtp_bruteforce" },
      "updated": "2026-05-25T21:47:35Z"
    }
  ],
  "total": 1
}

Saved views are operator-scoped and capped at 200 per operator. The saved view collection is stored as one 64 KiB preference blob. page and params keys must be simple identifiers: ASCII letters, digits, underscore, hyphen, or dot, up to 64 bytes. Each view has at most 32 params, and param string values are capped at 256 bytes. name must be 1-80 bytes with no control characters. PUT and DELETE return "ok": true on success.

Bulk-action undo

Finding dismissals, bulk threat block / whitelist and bulk firewall unblock responses return an undo_token when the daemon queues an inverse operation server-side for 30 seconds. The UI surfaces a banner with the same TTL; CLI callers can act on the token through the endpoints below. Each successful undo writes an undo_<original_action> audit entry. Capability flag: webui.undo.v1. These endpoints require admin scope because they read or mutate operator-private action state.

GET  /api/v1/undo/pending    Latest pending undo entry for this operator (empty object if none)
POST /api/v1/undo/run        Consume an entry and dispatch its inverse {id}; empty id uses latest

Non-empty response shape for GET /api/v1/undo/pending:

{
  "id": "188d1f2a6c8b0000",
  "action": "threat_bulk_block",
  "inverse": "threat_bulk_unblock",
  "summary": "Blocked 2 IPs",
  "recorded_at": "2026-05-26T00:07:09Z",
  "expires_at": "2026-05-26T00:07:39Z"
}

POST /api/v1/undo/run returns {ok, action, inverse, count} on success, or 410 Gone when the entry is missing, already consumed, or past its 30-second TTL. Recognised inverse action keys are threat_bulk_unblock, threat_bulk_block, threat_bulk_unwhitelist, threat_bulk_whitelist, firewall_bulk_reblock, and finding_undismiss. Other bulk actions (quarantine delete, generic fix) do not surface an undo token because they have no clean inverse.

Dismissals accept at most 500 keys per request and deduplicate repeated keys. contents maps a key to the content_sha256 its row showed: an empty string for a finding without one, or a string of 64 lowercase hex characters. All other values, including null, answer 400 before any findings are dismissed. A key whose listed finding now carries different content is not dismissed: a single key answers 409, and a bulk request dismisses the rest, returns the refused keys in refused, and counts only the applied dismissals in count. A request with every key refused returns no undo_token. A key missing from contents is dismissed as before and covers the listed content. Undo skips findings changed by a later dismissal, successful re-check or baseline reset, and count reports only the dismissals it actually reversed. Newer scan copies remain in the current findings list.

Finding fields

Every finding in /api/v1/findings, /api/v1/events, and the JSONL audit log carries optional correlation fields when CSM can attribute them:

FieldMeaning
tenant_idTenant attribution from the verdict callback or panel-side webhook reply
domainDomain associated with the event (e.g. PHP-relay scriptKey host, mailbox domain)
mailboxMailbox attribution (e.g. mail brute-force target, PHP-relay envelope-from)
cidrsOffending subnets the finding names (e.g. a distributed crawl’s collapsed subnets, or a mail or SMTP password spray’s /24; sprays leave source_ip empty)
relay_totalPHP-relay trigger count for the path that fired
relay_breakdownPHP-relay script samples that contributed to the alert, with script key, hit count, last seen time, and a bounded sample subject when available

Fields are omitted when the daemon could not attribute them. Orchestrators should treat absence as “unknown,” not “global.”

JSONL and syslog audit events include cidrs for new findings and replayed parked findings after upgrade. Existing audit records are not backfilled.

Cleanup fields

GET /api/v1/quarantine also powers the Cleanup page’s file-backup list. Entries include:

FieldMeaning
kindquarantine or pre_clean
live_stateoriginal_missing, live_differs, original_not_file, archive_missing, archive_not_file, or unknown. Byte-identical restored entries are hidden.

GET /api/v1/db-object-backups returns restored and restored_at when a captured MySQL trigger/event/procedure/function backup has already been replayed.

Incidents

GET /api/v1/incidents/groups is a read-scope rollup of active incidents by kind and source. It accepts status=active|all|open|contained|resolved|dismissed, kind, limit and offset, allowing a credential spray to render as one row per attacker rather than one row per target. total counts the groups and scanned_incidents the incidents they came from; truncated is true when the scan cap left incidents out.

GET /api/v1/incidents

Returns one page of incidents sorted by updated_at descending, with items, total, offset, limit and status. limit defaults to 50 and is at most 500. status filters by open, contained, resolved, dismissed, or active for open and contained; empty means all.

GET /api/v1/incidents/<id>

Returns one incident by id. 404 if not found.

POST /api/v1/incidents/<id>/status

Body:

{"status": "resolved", "details": "operator-marked"}

Status values: open, contained, resolved, dismissed. Closing an incident (resolved/dismissed) means future findings for the same correlation key start a fresh incident. Reopening an incident binds the same key again. Incident JSON includes correlation_key when CSM has a stored account, mailbox, domain, process, or remote-IP key.

MCP endpoint (read-only)

CSM can expose what it knows about a server to an LLM support agent through the Model Context Protocol. The endpoint is off by default, answers at /mcp on the Web UI listener with credentials of its own, and is read-only by construction: every tool is answered by dispatching a GET to one route from a fixed allowlist of API reads, in-process, so no tool can block, unblock, fix, scan, quarantine or change settings. It is meant for agents that verify support tickets about the server (“is my IP blocked”, “why does this account have findings”, “is mail stuck”).

Enable it

webui:
  enabled: true           # the endpoint is served by the Web UI listener
mcp:
  enabled: true
  tokens:
    - name: "support-agent"
      token: "<64 random hex characters>"
      issued_at: "2026-11-02T10:00:00Z"
      allowed_ips: ["198.51.100.0/24"]

Restart the daemon (systemctl restart csm); the endpoint is wired at startup. csm validate checks the block: it refuses mcp.enabled without webui.enabled, a configuration without a usable token, and a token that is also a Web UI or metrics token, and warns about short tokens.

The endpoint shares the Web UI’s TLS certificate, port and firewall exposure. Limit who reaches it with the token’s allowed_ips.

Tokens are bearer credentials with the same dates and source restriction as webui.tokens, without a scope: there is only one, read-only. A Web UI token or browser session never opens /mcp, and an MCP token opens nothing else. A known credential sent to the wrong endpoint, or used outside its dates or addresses, is refused without spending the guess budget below. Unknown tokens draw on the same budget as Web UI login and API token guesses: five failed attempts per minute per client (IPv4 address or IPv6 /64), after which the client is refused for a minute. /mcp also counts toward the API request rate limit. Up to four tool calls run at once and each request, a whole batch included, has 30 seconds. A read still running at the deadline keeps its slot until it finishes.

Transport

The MCP Streamable HTTP transport, JSON-RPC 2.0, stateless:

curl -sk https://server.example.com:9443/mcp \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"csm_lookup_ip","arguments":{"ip":"203.0.113.9"}}}'
  • POST /mcp only. GET answers 405 (no server-initiated stream); there is no session id to keep.
  • Protocol versions 2025-06-18, 2025-03-26 and 2024-11-05; initialize answers the client’s version when it is one of those.
  • Methods: initialize, ping, tools/list, tools/call, notifications/initialized (202, no body). resources/list and prompts/list answer empty lists. Batches of up to 20 requests are answered as a batch.
  • A request body is limited to 1 MiB. A tool answer larger than 4 MiB is refused with a message asking for limit, offset or a filter.
  • A browser Origin header must pass the same check as on the API: the configured Web UI origin, an entry in webui.allowed_origins, or a loopback origin that matches the request host.

A client configuration for an agent runtime that takes a URL and a header:

{
  "mcpServers": {
    "csm-web01": {
      "url": "https://web01.example.com:9443/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}

Tools

Every tool carries readOnlyHint: true. A tool answers the route’s JSON as text content; the route’s own error (400, 404, 503) is returned as a tool error with the API message, so the agent can read it. Lists keep the API shape {items, total, truncated}; a tool takes only the arguments its route reads.

Composite tools:

ToolReads
csm_overviewhostname, platform, version, status, health, capabilities
csm_lookup_ipfirewall/check, the blocked-ips entry, threat/ip, geoip for one address
csm_config_summarynon-secret configuration facts from an explicit field list: mode, platform, Web UI listen address, enabled protections, auto-response limits, brute-force thresholds. Token values, keys, DSNs and URLs are never read

Route tools, one per API read (the description names the route):

AreaTools
Daemoncsm_status, csm_health, csm_capabilities, csm_components, csm_challenge_stats, csm_scan_jobs, csm_scan_job
Findingscsm_findings, csm_findings_enriched, csm_finding_detail, csm_history, csm_stats, csm_stats_trend, csm_stats_timeline
Responsecsm_blocked_ips, csm_quarantine, csm_db_object_backups, csm_audit
Accounts and incidentscsm_accounts, csm_account, csm_incident_timeline, csm_incidents, csm_incident_groups, csm_incident
Mailcsm_email_stats, csm_email_quarantine, csm_email_av_status, csm_email_checks, csm_email_groups, csm_email_overview, csm_email_rejects, csm_email_senders, csm_email_rspamd, csm_email_relay_abuse, csm_email_forwarders, csm_email_deferrals, csm_email_queue_composition, csm_email_queue, csm_email_held
Hostcsm_performance, csm_hardening
Threat and rulescsm_threat_stats, csm_threat_top_attackers, csm_threat_ip, csm_threat_events, csm_threat_db_stats, csm_threat_whitelist, csm_rules_status, csm_rules_list
WAF and firewallcsm_modsec_stats, csm_modsec_blocks, csm_modsec_events, csm_modsec_rules, csm_firewall_status, csm_firewall_allowed, csm_firewall_audit, csm_firewall_subnets, csm_firewall_check, csm_firewall_rollback, csm_geoip

Not reachable, on purpose: the findings event stream, CSV and state exports, settings, sessions, preferences, undo, the verified-bots editor, every POST, PUT, PATCH and DELETE route, and the reads that return content rather than metadata: quarantined file previews, queued message headers and the SQL of backed-up database objects. The allowlist is tested against the public route registrations, so a route added to it must be a documented GET read.

Some reads touch the live host rather than CSM’s store: csm_email_stats and csm_email_queue run the MTA queue tools, csm_firewall_check asks cPHulk on cPanel, csm_accounts walks account directories and csm_performance samples /proc.

Logging

Each tool call writes one daemon log line, mcp tool call, with the token name, tool, peer, HTTP status of the underlying route, outcome and duration, and the arguments cut to one line. Authentication failures log mcp: refused token with the credential name, when known, and peer. Reads are not written to the Web UI audit log, which records actions only.

The capability mcp.readonly.v1 in GET /api/v1/capabilities advertises that the build carries the endpoint; whether it is enabled is a matter of configuration.

Metrics (Prometheus)

CSM exposes a /metrics endpoint on its HTTPS web UI port (default 9443). The endpoint serves the Prometheus text exposition format (Content-Type: text/plain; version=0.0.4) and is safe to scrape every 15 seconds.

“Available metrics” below is the shipped set. New call sites are instrumented in ongoing releases; check CHANGELOG.md under ## [Unreleased] for the latest additions.

Enabling

Metrics are on whenever webui.enabled: true is set in csm.yaml. The endpoint has its own auth knob:

webui:
  enabled: true
  auth_token: "<UI login token>"
  auth_token_issued_at: "<UTC RFC3339 time when the UI token was generated>"
  metrics_token: "<long random string for Prometheus scraper>"
  metrics_token_issued_at: "<UTC RFC3339 time when the scrape token was generated>"
  metrics_token_expires_at: "<optional UTC RFC3339 expiry; leave empty for none>"

metrics_token is optional. When set, a Bearer header containing this exact value grants access to /metrics. The UI auth_token or a valid UI session cookie is also accepted so the dashboard can self-scrape, but keeping the two tokens separate is recommended: rotating auth_token does not then break Prometheus scraping, and giving your monitoring stack the scrape token does not also give it UI access.

Each credential expires at its expires_at setting and never expires without one. Rotate the metrics token and its dates together, update the scraper’s secret, then send SIGHUP to CSM. A malformed or future issue time, or a malformed or past expiry, refuses the metrics credential. See credential rotation for recovery instructions.

Prometheus scrape config

scrape_configs:
  - job_name: csm
    scheme: https
    tls_config:
      # CSM serves a self-signed cert by default; either skip
      # verification here or pin the CA you chose.
      insecure_skip_verify: true
    authorization:
      type: Bearer
      credentials: "<metrics_token from csm.yaml>"
    static_configs:
      - targets:
          - csm-host-1.example.internal:9443
          - csm-host-2.example.internal:9443

A complete, validated version of this snippet (with global: block) ships as docs/src/examples/prometheus-scrape.yml. The CI pipeline runs promtool check config against that file in the promtool-check job; if the example ever stops validating, the pipeline fails.

Quick check

curl -sk -H "Authorization: Bearer $METRICS_TOKEN" \
    https://localhost:9443/metrics | head

Available metrics

Build / process

  • csm_build_info{version} (gauge, always 1): build metadata. Scrape once to discover the running version. Join on it in queries via group_left(version).

Go runtime

Always exposed. Use these to watch daemon memory over time and tell live growth from GC headroom (rising heap_alloc = real growth; large heap_idle = retained-but-freed memory).

  • go_memstats_heap_alloc_bytes (gauge): heap bytes allocated and still in use (live).
  • go_memstats_heap_inuse_bytes / go_memstats_heap_idle_bytes / go_memstats_heap_released_bytes (gauges): in-use spans, idle spans, and bytes returned to the OS.
  • go_memstats_heap_sys_bytes / go_memstats_sys_bytes (gauges): heap and total bytes obtained from the OS.
  • go_memstats_next_gc_bytes (gauge): heap-size target for the next GC.
  • go_memstats_gc_cpu_fraction (gauge): fraction of CPU time spent in GC since start.
  • go_goroutines (gauge): live goroutine count.

For a deeper leak hunt, enable the loopback pprof endpoint (debug.pprof_listen) and run go tool pprof http://127.0.0.1:<port>/debug/pprof/heap over an SSH tunnel.

The same endpoint serves profile (CPU, 30s by default), goroutine, mutex and block. Contention sampling for the last two starts with the listener and stops when the last listener exits, including on a listener failure. Failed or refused binds do not enable sampling.

While enabled, mutex profiling samples one in 100 contention events. Block profiling samples waits of at least 10 microseconds and a proportion of shorter waits. Timing and collecting these samples adds CPU overhead even when nobody fetches a profile, so use this endpoint for diagnostics and leave debug.pprof_listen empty when it is not needed. Profiles accumulate for the process lifetime; disabling sampling does not erase previously recorded data.

For a CPU investigation, collect the CPU profile first and the goroutine dump alongside it:

curl -o cpu.pb.gz 'http://127.0.0.1:<port>/debug/pprof/profile?seconds=60'
curl -o goroutine.txt 'http://127.0.0.1:<port>/debug/pprof/goroutine?debug=2'

YARA-X worker (default-on; off only if signatures.yara_worker_enabled: false)

  • csm_yara_worker_restarts_total (counter): cumulative number of counted YARA scan worker failures (failed start, timeout, crash, out of memory or protocol), including launches that never create a worker. Rule reloads and lifetime retirement do not count. Alert on sustained growth: a steady climb means scans keep failing and are reported as not scanned.

Findings

  • csm_findings_total{severity} (counter): every finding CSM records is counted here. Severities are CRITICAL, HIGH, and WARNING (matching the alert.Severity enum). Use rate(...) for arrival velocity; watch for sudden CRITICAL spikes.

Alert delivery

  • csm_alert_dispatch_failures_total (counter): alert channel sends that failed after CSM detected findings. Counts email, webhook, and phpanel delivery failures. Sustained growth means findings are not reaching operators; check SMTP, webhook reachability, and credentials.

  • csm_audit_sink_degraded{sink} (gauge): one when an enabled JSONL or syslog destination failed to open or write, zero when healthy or disabled. Values appear after the audit pipeline first runs and remain degraded during retry backoff. Syslog reports local write results, not receiver acknowledgement.

  • csm_audit_events_dropped_total{sink} (counter): events whose destination was unavailable or whose write failed. Use csm export --since <when> for backfill; failed live deliveries are not replayed automatically.

State

  • csm_store_size_bytes (gauge): on-disk size of the bbolt state database (/var/lib/csm/state/csm.db by default). Retention sweeps bound logical growth. Startup compaction reclaims freelisted pages automatically when the file is large and mostly slack; csm store compact does the same immediately with the daemon stopped. The retention sweep logs a restart hint only under that same condition. It includes deleted pages still held by active readers, which become reclaimable when the daemon stops. Later writes can change whether compaction is due at startup. History and attack-event writes pack pages densely when their timestamps arrive mostly in order. A write uses balanced page splits when events older than the newest stored one are most of its records or at least half its inserted data. Space savings depend on record sizes and arrival order. Archive imports keep the snapshot’s page layout; compaction repacks an existing file.

Fanotify realtime monitor

  • csm_fanotify_queue_depth (gauge): current number of queued events waiting for the analyzer pool. The queue capacity is 4000; sustained values near that cap mean drops are imminent. Alert target: max_over_time(csm_fanotify_queue_depth[5m]) > 3500.
  • csm_fanotify_events_total (counter): delivered fanotify file events whose admission decision has completed. Filesystem marks receive writes anywhere on their superblocks, including other bind mounts. Kernel queue overflow notifications are counted separately.
  • csm_fanotify_events_admitted_total (counter): events queued for analysis, including dropper tracking. Subtract this counter and csm_fanotify_events_dropped_total from csm_fanotify_events_total to count events rejected before queue admission. This includes path-resolution failures as well as filter rejections. Use rates over the same interval to compare these independently scraped counters.
  • csm_fanotify_events_dropped_total (counter): cumulative events dropped because the analyzer queue was full. The reconcile pass still rescans drop-affected directories 60 s later, so dropped events do not disappear from detection – they arrive delayed. Alert target: rate(csm_fanotify_events_dropped_total[5m]) > 0 paired with a short for-clause.
  • csm_fanotify_placement_unverified_total (counter): files and directories moved, linked or created into watched trees that did not open as the file the placement monitor reported, so realtime did not check them there. A file replaced again right after it was placed is counted here. Sustained growth means placements are being raced; the deep scan covers those files.
  • csm_fanotify_placement_skipped_total (counter): placed files not analyzed again because the same version was analyzed recently under the same conditions, such as an atomic save renamed to its final name.
  • csm_fanotify_kernel_queue_overflow_total (counter): kernel-side fanotify queue overflows (FAN_Q_OVERFLOW). Unlike analyzer-queue drops, the kernel gives no file descriptor for lost events, so only directories with concurrent analyzer drops can be reconciled; the rest are covered by the next scheduled deep scan. Any nonzero rate during a write storm means realtime coverage had a gap.
  • csm_spool_fanotify_queue_overflow_total (counter): kernel queue overflows on the mail spool watcher. In permission mode an overflowed event means the message was delivered without being scanned; a Warning finding is raised when this happens.
  • csm_fanotify_reconcile_latency_seconds (histogram): how long the post-overflow reconcile pass takes to walk drop-affected directories and rescan recent files. Buckets: 0.01 s .. 60 s. Watch p95: reconcile stealing tens of seconds means bulk events are piling up faster than the walker can keep up.
  • csm_checks_domlog_discovery_dropped_total{reason} (counter): per-vhost access-log paths the WP brute-force domlog discovery helper dropped before scanning. Labels: reason is evalsymlinks_error (broken symlink, attacker-removed log file) or stat_error (file vanished between glob and stat, permission regression on the log directory). Steady growth means a real chunk of vhosts is being silently skipped each cycle. Stale-mtime drops are intentional filtering and are NOT counted here.
  • csm_realtime_content_scan_truncated_total{check} (counter): cumulative real-time content checks where the underlying file was larger than the main read window, so the full-rule pass saw only the leading window. The read cap protects RE2 cost on huge files; sustained growth on a label means full-rule coverage is capped on large files. Labels currently emitted: phpcontent_inline (known webshell filename), phpcontent_uploads (PHP in uploads), php_check (generic PHP content scan), crontab (per-user /var/spool/cron write), htaccess (per-vhost .htaccess write), user_ini (per-vhost .user.ini write), html_phishing (HTML phishing heuristic), and cgi_backdoor (CGI backdoor heuristic). Compare against finding history for webshell_content_realtime (or the matching check name for non-PHP labels) to judge whether a raised cap would surface real findings.

Mail scanning and parse workers

  • csm_spool_hold_expired_total{stage} (counter): mail opens answered under email_av.fail_mode because the hold budget ran out. stage records where the message was when its hold expired: queued (no scan slot yet; these count toward the hold bypass), waiting (behind another message from the same sender) or scanning (a slow scan; the scan continues and can still quarantine). Sustained queued growth means scan capacity is short.
  • csm_spool_source_waits_total{outcome} (counter): mail opens whose sender was already using its share of the scan slots. outcome is waited (the message waited its turn without taking a scan slot) or overflow (too many messages from that sender were already waiting, so it was scanned at once without waiting its turn). Overflow scans can use every scan slot. overflow growth raises an email_av_source_backlog finding naming the sender, at most once a minute.
  • csm_parse_worker_jobs_total{pool,outcome} (counter): parse worker jobs. pool is mail (email attachments) or zip (tenant zips checked for phishing kits). outcome is ok, timeout, oom, crash, protocol (reply rejected), spawn (the unit did not start), refused (the worker is failing repeatedly) or skipped (the message killed workers before, so it was not unpacked again). Anything but ok means a message or zip was not fully checked. Canceled requests, stopped workers and local descriptor exhaustion do not increment this counter.
  • csm_parse_worker_starts_total{pool} (counter): worker units started. A worker serves many jobs; a start rate close to the job rate means workers keep dying or being retired.
  • csm_parse_worker_memory_bytes{pool} (histogram): worker memory and swap in use after a successful job. Buckets: 32 MiB .. 1 GiB. Compare the high buckets with email_av.worker_memory_limit. Unavailable memory readings are omitted; earlier readings are not reused.

Periodic check runner

  • csm_checks_crontab_base64_truncated_total (counter): crontab base64 candidates that exceeded the per-blob decode cap before decoded-content pattern matching ran. Sustained growth means encoded cron content is larger than the scanner currently inspects; review affected crontabs and tune the scanner before redeploying.

  • csm_check_duration_seconds{name,tier} (histogram): wall-clock time each check takes to complete. Label name is one of the periodic check-runner names (fake_kernel_threads, webshells, …); label tier is critical, deep, or all. Buckets: 0.01 s .. 900 s. Most checks keep the 300 s timeout ceiling; heavy filesystem checks can run up to 900 s. Useful aggregations:

    # p95 of the slowest check in the critical tier:
    histogram_quantile(0.95,
      sum by (le, name) (
        rate(csm_check_duration_seconds_bucket{tier="critical"}[10m])
      )
    )
    
    # total time each cycle spends in deep-tier checks:
    sum by (tier) (rate(csm_check_duration_seconds_sum{tier="deep"}[1h]))
    

Threat intelligence

Registered when reputation.upstream.enabled: true.

  • csm_threatintel_cache_hits_total (counter): upstream threat-intel lookups served from CSM’s local per-IP cache.
  • csm_threatintel_cache_misses_total (counter): upstream threat-intel lookups not served from the local cache. A miss may still fail open without an HTTP request when the breaker is open.
  • csm_threatintel_backend_failures_total (counter): upstream backend failures from network errors, non-200 responses, malformed JSON, response IP mismatches, or out-of-range scores.
  • csm_threatintel_breaker_open (gauge): 1 while the upstream circuit breaker is refusing calls, 0 when closed or allowing its single cooldown probe.

AI-crawler IP ranges

  • csm_botranges_refresh_total{result} (counter): AI-crawler IP-range refresh attempts, labelled success (at least one vendor feed updated) or failure. Driven by the daemon’s auto-updater and by csm update-bot-ranges.
  • csm_botranges_prefixes{bot} (gauge): current number of published IP prefixes per crawler identity in the active overlay.
  • csm_botranges_last_success_timestamp_seconds (gauge): Unix time of the last successful refresh.

Firewall

  • csm_blocked_ips_total (gauge): number of IPs currently on the firewall block list. Excludes expired temp bans – the store’s LoadFirewallState filters those before the gauge reads.
  • csm_firewall_rules_total (gauge): total firewall rules across all four categories (blocked IPs, allowed IPs, blocked subnets, port-specific allows). Excludes expired temp blocks and allow-list rows. Sudden drops are worth investigating; expected drops happen when temporary block or allow deadlines pass.

Config reloads

  • csm_config_reloads_total{result} (counter): SIGHUP reload attempts, by outcome. Labels: result is one of:
    • success – safe fields swapped in place, integrity hash re-recorded, live config updated.
    • restart_required – one or more fields that need a full restart changed; live settings unchanged.
    • error – YAML parse, validation, re-hash, or subsystem apply failure; live settings unchanged. If signing succeeded before an apply failure, the live integrity hashes follow the signed config.
    • noop – file edit produced no semantic change (identical values, whitespace edit, etc.). Alert target: rate(csm_config_reloads_total{result="error"}[5m]) > 0 paired with a short for-clause.

Auto-response

  • csm_auto_response_actions_total{action} (counter): every auto-response action fired, by class. Labels: action is kill, quarantine, or block. Incremented once per finding the corresponding Auto* helper produces, so a batch blocking four IPs in one cycle adds 4 to action=block. Useful for detecting response storms. Verified admission blocks also increment this counter, from every source admission runs (scan, challenge, central and incident), once per engine result, without waiting for alert delivery: rate(csm_auto_response_actions_total[5m]).
  • csm_admission_handoff_seconds (histogram): how long a response funnel waits to hand an automatic response to the admission ledger, which applies, previews or forecasts it. Preparing evidence for a possible later response adds no sample; selected refusals and responses each add one. Buckets: 0.001 s .. 1 s. The share of handoffs at or under 10 ms is csm_admission_handoff_seconds_bucket{le="0.01"} / csm_admission_handoff_seconds_count; it should stay at or above 0.99.

Challenge

  • csm_challenge_routed_total{check} (counter): IPs routed to the proof-of-work challenge, labelled by the source check that flagged them (e.g. http_scanner_profile, wp_login_bruteforce, ip_reputation). Graph per-detector challenge volume with sum by (check) (rate(csm_challenge_routed_total[5m])).
  • csm_challenge_escalated_total{outcome} (counter): challenge entries that timed out and were escalated to a hard block, by firewall outcome: live (a new block landed), noop (the IP was already blocked), plus dry_run, allowed, and allowlisted. outcome=live is how many challenges actually became blocks, including those applied by admission after the timeout handoff.
  • csm_challenge_passed_total{method} (counter): challenged addresses that got through, by method: pow (solved the puzzle), captcha, verify_cookie (passed earlier and came back), admin_cookie, and crawler (verified search crawler).

Firewall blocks

  • csm_firewall_block_outcome_total{outcome,source} (counter): every IP block attempt, labelled by firewall outcome (live, dry_run, allowed, allowlisted, noop, protected for refused infrastructure IPs, error for engine failures or an unavailable engine) and by source (scan, challenge, incident, central_intel, cli, web_ui, or unknown for an invalid internal source). All combinations are exported at zero from startup, and unexpected outcome or source values collapse into the fixed error or unknown buckets instead of creating new labels. Admission execution reports into these same counters. Failed or unknown batch outcomes count as errors; forecasts and queued deferrals add no attempts. Retrying an outcome write or audit delivery adds no count. This is the aliveness signal for auto-response: alert when sum(rate(csm_firewall_block_outcome_total{outcome="live",source!~"cli|web_ui"}[1h])) == 0 while attack findings keep coming, and watch outcome="error" for a dying engine.

Retention (when retention.enabled: true)

  • csm_retention_sweeps_total (counter): number of retention sweep cycles completed since daemon start. A flatline after a restart means the sweep goroutine is not scheduling; a healthy daemon increments this on every sweep_interval tick.
  • csm_retention_deleted_total (counter): cumulative entries deleted across the history, attacks:events, and reputation buckets. Spikes on the first sweep after enabling retention (initial backlog), then settles to the steady-state churn. Useful for estimating when the file might benefit from a restart or a csm store compact maintenance window.

PHP-relay (email abuse, cPanel only)

All series are prefixed csm_php_relay_. Registered when email_protection.php_relay.enabled: true and the host is cPanel; otherwise zero across the board. See Real-time detection.

  • csm_php_relay_findings_total{path} (counter): findings emitted per detection path. Labels: path is one of header, volume, volume_account, fanout (and later baseline, reputation for Stages 2-3). Use rate(...) to spot detection storms; a sudden rate jump on header typically means a contact-form vulnerability is being exploited, on volume_account typically means an account password was leaked.
  • csm_php_relay_actions_total{action,result} (counter): auto-freeze invocations attempted. Labels: action is currently freeze; result is ok or fail. Pair with csm_php_relay_findings_total to confirm freeze keeps up with detection.
  • csm_php_relay_action_gone_total (counter): messages already absent from the spool by the time exim -Mf ran. Normal queue churn; not a failure. Sustained growth means the spool is moving fast and the freezer is racing the queue runner.
  • csm_php_relay_path_skipped_total{path,reason} (counter): path evaluation that bailed before producing a finding. Labels: path matches the finding labels above; reason enumerates the gate that fired (e.g. ignore-list match, missing scriptKey).
  • csm_php_relay_spool_scan_fallbacks_total{reason} (counter): AutoFreeze fell back to a full spool walk to find msgIDs. Labels: reason is capped (the in-memory activeMsgs per script hit its cap, so a fresh disk walk was needed) or reputation (a late reputation finding arrived for a script with no live activeMsgs). Sustained growth on capped means a single script is firing faster than the in-memory window keeps state for; consider raising header_score_volume_min or adding an ignore.
  • csm_php_relay_active_msgs_capped_total (counter): per-script activeMsgs set hit its cap and dropped the oldest entry. Counts the eviction event itself; the next freeze for that script will land in csm_php_relay_spool_scan_fallbacks_total{reason="capped"}.
  • csm_php_relay_windows_active{kind} (gauge): retained per-script / per-IP / per-account window state. Labels: kind is script, ip, or account. Sized by Flow E sweep cadence (5 min for windows, 24 h retention for accounts); flat values across hours are normal.
  • csm_php_relay_msgid_index_size{layer} (gauge): msgID dedup index size by storage layer. Labels: layer is memory (in-process map) or bbolt (persisted batch writer). Memory ceiling is 200k entries; bbolt grows freely until the 25 h Flow E sweep prunes it.
  • csm_php_relay_msgindex_persist_dropped_total (counter): bbolt persist queue overflow drops (the 4096-deep buffered channel was full when the watcher tried to enqueue). Should be zero in steady state; a non-zero value means the bbolt writer is blocked on disk and the in-memory dedup is the only thing protecting against double-fire on a queue-runner re-write.
  • csm_php_relay_msgindex_persist_errors_total (counter): bbolt commit failures from the async batch writer. Each bump also emits a Critical email_php_relay_msgindex_persist_failed finding. Disk-full or permissions issue on /var/lib/csm/state/csm.db.
  • csm_php_relay_inotify_overflows_total (counter): kernel IN_Q_OVERFLOW events on the spool watcher. Each one triggers a bounded recovery scan (default cap 1000 files); if the cap fires, also emits email_php_relay_overflow_scan_truncated Critical. Sustained growth means the spool is churning faster than inotify can keep up – usually a backup restore or a real attack.
  • csm_php_relay_spool_read_errors_total (counter): emailspool.ParseHeaders errors on -H files the watcher tried to consume. Usually transient (file disappeared between inotify event and open) and self-correcting; sustained growth points at a permissions or filesystem problem.
  • csm_php_relay_userdata_errors_total (counter): cpanelUserDomains resolver errors reading /var/cpanel/userdata/. Used by the Path 1 From mismatch check; errors here mean Path 1 is potentially undercounting until the read recovers.

Anti-spam learning

All series are prefixed csm_antispam_learning_. Pass totals and state gauges update after each completed Spam folder pass while learning runs. Contributions are counted when committed, including background retries. No series carries an account, mailbox or UID label. See Anti-spam.

  • csm_antispam_learning_reports_admitted_total (counter): Spam folder reports the learner accepted.
  • csm_antispam_learning_contributions_committed_total (counter): cluster contributions committed from those reports.
  • csm_antispam_learning_skips_total{reason} (counter): entries and reports skipped. reason is a fixed code such as tagged, malformed, opted_out, oversize or account_capacity. On Postfix hosts it also counts Dovecot mailboxes not watched, such as driver or no_junk, and cancel_unconfirmed when a pass could not confirm that a domain or mailbox was removed and so dropped no reports.
  • csm_antispam_learning_pending_reports (gauge): reports waiting after the last pass. When discovery was incomplete, including unreadable folders or discovery limits, this is a lower bound.
  • csm_antispam_learning_backlog_age_seconds (gauge): age of the oldest waiting report at the last pass.
  • csm_antispam_learning_last_pass_timestamp_seconds (gauge): when the last pass completed.
  • csm_antispam_learning_live_clusters, csm_antispam_learning_zero_target_records, csm_antispam_learning_body_bytes (gauges): learned clusters, zero-target guards and stored anchor message bytes.
  • csm_antispam_learning_uncertain_intents (gauge): anchor writes whose outcome is still being reconciled.
  • csm_antispam_learning_oldest_unchecked_timestamp_seconds, csm_antispam_learning_audit_completed_timestamp_seconds (gauges): anchor audit progress.
  • csm_antispam_learning_reconstructed_clusters (gauge): missing live anchors newly scheduled for reconstruction since the learner started. Retries, ordinary value corrections and removal of retired anchors do not increase it.

If the learning ledger cannot be read after a pass, the engine-state gauges (live_clusters, zero_target_records, body_bytes, uncertain_intents, oldest_unchecked_timestamp_seconds, audit_completed_timestamp_seconds and reconstructed_clusters) report NaN until a successful read.

Signature retroactive rescans

  • csm_signature_rescans_total (counter): rescans queued by a change to a tracked signature file’s content that completed. A rescan completes when the rolling deep YARA scan has scanned every file since the change. Re-installing identical rules, as a package upgrade or a repeated download does, does not count. Multiple updates before a rescan completes coalesce into one.

The watcher compares content hashes when a file’s mtime or size changes; symbolic links use the target file’s metadata. Failed reads and files replaced during hashing are retried without discarding the last verified state. First observations and file removals do not queue a rescan.

The rolling deep YARA scan reads files in path order and resumes each deep run where the previous one stopped, so one pass spans many deep runs. A rules update does not restart the pass: restarting on every update would never reach the files that sort last when updates come faster than a pass. The scan records where it stood when it took the update, and the rescan completes once the scan has wrapped and come back to that point. Every file is then scanned with the new rules within one pass. An update during a rescan starts a new one from where the scan then stands.

If the set of scan roots changes, or a root cannot be listed, the rescan starts a new lap from where the scan stands; the scan itself keeps going and still covers every root it can read. On a host whose accounts change faster than one pass, the rescan can stay pending; scanning is unaffected.

The rolling scan uses the YARA rules. YAML rule updates reach existing files only when those files change, as before.

A queued rescan is stored with the rule state that caused it, and the scan stores its progress with its cursor, so daemon restarts keep both. The saved queue is restored even if the state store becomes available after the first watcher tick. Invalid queue records log a warning and are repaired by queuing a fresh rescan; the scan neither tracks nor clears a queue it cannot read. A deep run cancelled by shutdown or by the check timeout saves no progress. Coverage gaps, reported by yara_scan_incomplete, do not keep the queue pending.

Setting detection.rescan_on_signature_update to false stops the watcher and pauses the queue: the scan keeps running but neither starts nor completes a rescan. Re-enabling it resumes the saved work.

csm check-deep advances the same rolling scan, so it can complete a queued rescan and increment this counter. Its findings are returned and kept in scan history but not alerted or acted on, and the live scan does not revisit those files until its next pass. Use csm run-deep when findings should alert.

State from older builds contains only mtimes. The first successful read adds a hash without queuing a rescan if the recorded mtime still matches. A moved mtime without a recorded hash conservatively queues one rescan.

Counter reset semantics

Prometheus counters in CSM live in process memory. They reset to zero whenever the daemon restarts (config change, binary upgrade, crash recovery). This is the standard behaviour for every Prometheus-instrumented daemon; Prometheus’s scrape pipeline detects counter resets on its own and rate(), increase(), and rate_over_time() all handle them correctly.

Operators should not alert on “counter decreased across a scrape” as a failure condition. Alert on rate() or increase() of a counter over a window long enough to absorb expected restarts.

Persisting counters across restarts would require writing to bbolt on every increment, which would not pay for itself. If a specific metric needs restart-stable behaviour later, a gauge-over-the-bbolt-counter pattern can be added for that one case without affecting the rest.

Caveats

  • Scrape the web UI’s HTTPS port, not a separate listener.
  • curl -k / insecure_skip_verify is appropriate only when the cert is self-signed and the network path is trusted. Pin a CA for anything else.
  • Prometheus label cardinality: per-account and per-IP labels are deliberately not exposed. Shared-hosting deployments with 1000+ cPanel users would otherwise overwhelm a Prometheus server.
  • Metric vectors cap label-value combinations at 1000 children per metric, including the overflow bucket. Once a vector reaches that cap, new combinations are aggregated under _overflow_.

Not instrumented (yet)

  • Per-account labels on any metric. Deliberately off: shared-hosting deployments with 1000+ cPanel users would blow out Prometheus cardinality.
  • Fanotify inline auto-response actions (the quarantine-while- seeing-the-write path in fanotify.go). The periodic csm_auto_response_actions_total does not count those; a follow- up may split the metric or add a source label.
  • bbolt per-bucket size breakdown, csm_store_used_bytes, and csm_store_last_compact_ts. Deferred to the online-compaction follow-up of the retention work (see ROADMAP.md).

Audit Log (SIEM)

Audit Log

CSM ships source observations and notification findings to one or more SIEM-friendly sinks, deduplicated by observation identity within each batch and against each destination’s recent successful deliveries. Audit records include sources suppressed by notification filtering or rate limits, so SIEM correlation can still identify the original observation.

Two sink types ship today, both opt-in via csm.yaml. They can be enabled together or independently.

Schema

Every event, regardless of transport, has the same shape:

{
  "v": 1,
  "ts": "2026-04-28T10:32:14.512938Z",
  "finding_id": "8e3f1c204c1d8b95",
  "severity": "CRITICAL",
  "check": "webshell_realtime",
  "message": "PHP execution primitive in uploads/",
  "details": "...",
  "file_path": "/home/customer/public_html/uploads/x.php",
  "hostname": "host.example.com"
}

The v field is the schema version. CSM bumps it on incompatible changes and will not bump it for additive fields, so SIEM parsers can pin on v: 1 and ignore unknown keys.

An optional context key carries background beside the details, such as ModSecurity being switched off for the account a malware finding concerns. It is not part of finding_id or receipt matching, so a change in it alone never produces a new record. Recognized credentials in context are redacted before alert and audit output, using the same policy as messages and details.

finding_id is a stable 16-hex-char hash of the canonical fields (timestamp, check, severity, message, file path). Two emits of the same finding produce the same ID, so downstream dedup works across re-runs.

Firewall actions caused by a finding carry this same identity in the action log, including failed and refused attempts. The identity is captured before reason text is shortened and survives queued retries, challenge timeouts, central-intelligence decisions, and permanent-block promotion. Subnet and incident escalation link the latest known contributing finding. Older stored evidence without an identity and manual or maintenance operations remain unlinked; CSM does not reconstruct an identity from display text. Database session blocks link the original database finding, not a synthetic IP candidate.

The daemon audits source observations even when they repeat an earlier finding or are filtered from operator notifications. Distinct observations keep their own identities; recent replays of the same observation are suppressed per sink. Receipts are kept in bounded memory and survive temporary sink failures. A destination that missed a record can receive its replay without duplicating a healthy destination’s record. Replays refresh their receipt’s position, keeping recently replayed observations ahead of inactive receipts. Receipt matching includes original details, scanner identity and source attribution; distinct findings can share a legacy finding_id and still need separate records. Process enrichment alone does not create a new observation. Restarting or reconfiguring sinks clears receipts; observations no longer in the receipt cache can be emitted again. The legacy finding_id remains an action correlation key; collectors must not treat it alone as a unique-record key. Notification suppression and downstream finding observers keep their existing behavior. Audit delivery still depends on the configured sink and scan findings reaching the dispatcher.

The ts field records when CSM raised the finding, including process monitoring, automatic actions, scanner health, and mail relay storage errors. It is independent of when a sink delivers the event. A producer that builds a finding without a time gets the moment the daemon received it, so ts is never the zero time.

Alert dispatch fills missing times on its own copy of the findings. Reusing an unstamped input for a later occurrence gets a fresh time and finding id; replaying a finding with an existing time preserves its time and id.

Process context

Exec and outbound-connection findings on BPF-backed hosts carry an optional process object with PID, PPID, UID, user, cPanel account (when known), comm, exe, sanitized cmdline, and a parent chain. The field is omitted when no context is available, so existing parsers that ignore unknown keys see no schema change.

{
  "severity": "HIGH",
  "check": "outbound_connection",
  "message": "Suspicious outbound connection",
  "process": {
    "pid": 4242,
    "ppid": 4200,
    "uid": 1001,
    "user": "alice",
    "account": "alice",
    "comm": "ncat",
    "exe": "/usr/bin/ncat",
    "cmdline": ["ncat", "203.0.113.10", "587"],
    "parent": {
      "pid": 4200,
      "ppid": 4100,
      "uid": 1001,
      "comm": "sh"
    }
  },
  "timestamp": "2026-05-07T12:34:56Z"
}

The parent chain may be truncated at depth 5 and may stop early if an intermediate parent has been evicted from the cache.

File sink (JSONL)

alerts:
  audit_log:
    file:
      enabled: true
      path: /var/log/csm/audit.jsonl    # default

The default path is created with mode 0640 and the parent dir with 0750. The packaged logrotate fragment uses copytruncate mode so the daemon’s open file descriptor stays valid across rotation – no SIGHUP needed. It rotates daily and keeps 14 compressed rotations. The 100 MB threshold permits early rotation when the host runs logrotate more often than daily; it does not cap growth between runs. The sink never drops records to stay under that size, because a detection flood would otherwise blind the audit trail until the next rotation. On busy hosts, run logrotate more often than daily so the size trigger can take effect. Installation and upgrades refresh the fragment, including upgrades through csm rehash.

If you move the audit log off the default path, add your own logrotate stanza for it: the packaged fragment names the default path only.

Tail it for an interactive view:

tail -F /var/log/csm/audit.jsonl | jq -c

Or hand it to a log shipper like Vector, Filebeat, or Fluentbit.

Syslog sink (RFC 5424)

alerts:
  audit_log:
    syslog:
      enabled: true
      network: udp                  # udp | tcp | unix | unixgram | tls
      address: 127.0.0.1:514        # host:port for udp/tcp/tls, path for unix*
      facility: local0              # default
      tls_ca: ""                    # optional PEM file for tls transport

Wire-line is RFC 5424 with the JSON event embedded as the MSG body, so receivers that already understand the JSONL schema parse it the same way regardless of transport. UDP and unix-datagram emit one datagram per message; TCP, TLS, and unix-stream use LF framing.

Severity mapping onto the standard syslog level set:

CSM severitySyslog levelNumeric
CRITICALcrit2
HIGHerr3
WARNINGwarning4

Automated tests cover RFC 5424 output and UDP, TCP, TLS, Unix datagram, and Unix stream framing. Validate the chosen receiver configuration in a staging environment before production rollout.

Delivery failures and recovery

Each enabled destination retries independently after an open or write failure. The delay starts at one second and doubles up to one minute. A later dispatch retries a missing destination once its delay has elapsed; a quiet daemon waits until another finding arrives. Healthy destinations stay open during retries. Reloading audit configuration waits for in-flight writes before replacing sinks.

Failures and recovery are logged to journald. Monitor csm_audit_sink_degraded{sink="jsonl"} and csm_audit_sink_degraded{sink="syslog"} on /metrics: one means a configured destination failed, zero means it is healthy or disabled. These values are initialized when the audit pipeline first runs. Syslog health reflects local connection/write results, not an acknowledgement from the receiving SIEM.

csm_audit_events_dropped_total{sink} counts events whose destination was unavailable or whose write failed. Failed deliveries are not replayed automatically; use the backfill command below to recover stored findings. Backoff resets after a successful delivery, and configuration changes take effect on the next dispatch.

Backfill

When you first turn on the audit log, the SIEM has no history. Use csm export --since <when> to dump prior findings in the same JSONL schema:

csm export --since 24h > recent.jsonl
csm export --since 2026-04-01T00:00:00Z > q2.jsonl

<when> is either an RFC 3339 timestamp or a duration relative to now (24h, 7d). The output is one JSON event per line on stdout, identical in shape to what the live sinks emit, so you can pipe it straight into the same ingest pipeline.

Requires a running daemon.

What gets logged

Source observations and notification findings reaching the audit dispatcher, deduplicated by observation identity within each batch, before:

  • the per-account rate limiter (so audit signal is not lost when email and webhook are throttled);
  • the “blocked IP suppression” filter (so SIEM correlation sees events that operators were spared);
  • the per-sink disabled-checks list (audit log is not subject to email’s disabled_checks).

This means audit-log volume is generally higher than the email or webhook stream. Plan SIEM retention accordingly.

What does not get logged

Before a record is written, its message and details replace recognized password fields, API tokens, command-line secrets and cPanel session identifiers with [REDACTED]. Session redaction covers cPanel, WHM, Webmail, the shared server daemon, DAV and security purge log lines. It keeps the account name beside a session identifier, leaves unrelated lines of a multiline finding alone, and leaves already redacted text unchanged when it runs again. Repeated and quoted values are all covered, including the displayed form of NUL-separated arguments.

The same redaction runs on email digests, on new finding history in both the bbolt and the legacy JSONL backend, and therefore on the history the web UI serves and exports as CSV. Attack event messages are redacted before truncation, so a truncated line cannot hide a credential behind a cut service tag; attack events store no finding details. Account and IP attribution is read from the original finding, and finding IDs are computed from it too, so audit records still correlate with remediation records. Other structured fields are copied unchanged.

Two limits are worth knowing. Records written by earlier versions are not rewritten, so an existing log keeps whatever it already holds. The active-finding snapshot and the pending queues are process-local state that is read back and compared by the daemon itself, so they are not redacted.

The audit log is not a replacement for csm.history (the bbolt history bucket). Only findings that pass through the audit dispatcher are emitted. Internal state changes – daemon startup, reload events, config changes – live in journald via csm.service and are not mirrored here.

Action log

The audit log records what CSM found. The action log records what CSM did: one record per action, one file, one schema, whether the action came from the daemon, an operator command or the web UI.

/var/log/csm/actions.jsonl
csm actions                      # recent actions, one line each
csm actions --since 24h          # RFC 3339 timestamp or a duration
csm actions --op respond.block_ip
csm actions --json               # the raw records
csm action show <action_id>      # one admission response and its attempts

Schema

{
  "v": 1,
  "ts": "2026-09-09T10:32:14.512938Z",
  "hostname": "host.example.com",
  "op": "respond.quarantine_file",
  "actor": "daemon",
  "finding_id": "8e3f1c204c1d8b95",
  "target": "/home/alice/public_html/uploads/x.php",
  "account": "alice",
  "reason": "webshell_realtime",
  "before": {"exists": true, "sha256": "5f2b...", "size": 1841, "mode": "-rw-r--r--", "uid": 1001, "gid": 1001},
  "after": {"exists": false},
  "result": "applied",
  "recovery_path": "/opt/csm/quarantine/2026-09-09T10-32-14-x.php"
}
FieldMeaning
opThe privileged operation, using the IDs from the capability matrix. The matrix says what an operation may do; this says what it did.
actionThe specific change within an operation, such as block, unblock, apply or rollback.
actordaemon, cli or webui. actor_detail carries the operator’s source address or the command name.
finding_idThe finding that caused the action, using the same ID the audit log emits, so the two streams join.
commandThe exact argv when CSM ran a program. Absent when the action used system calls only.
before / afterThe target file’s digest, size, mode and owner around the change. "exists": false after a quarantine is the record that the file is gone.
resultapplied, dry_run, failed or refused. A dry-run record says what CSM would have done.
undoA command that reverses the action, when an exact inverse is available.
recovery_pathRetained quarantine content or a pre-clean backup, with a .meta sidecar. Use the quarantine recovery workflow; csm restore accepts state archives, not these files.

Attempted operations include failures and safety refusals. An already blocked address does not produce another applied record. A successful file replacement or quarantine with a later warning stays visible with its recovery path.

A refused action is recorded too. “The safety rules stopped this” and “CSM never looked” are different operational states, and only one of them is a reason to change the configuration.

v is the schema version. It is bumped only on an incompatible change, so a parser can pin on v: 1 and ignore unknown keys.

Long reasons and error messages are truncated with an explicit marker so one action cannot make the history unreadable. Targets and command arguments are preserved.

A file larger than 64 MiB is recorded without a digest rather than stalling the action while it is hashed. Quarantine digests come from the captured copy, after the move. Cleaning digests describe the exact bytes read and installed through the pinned file handles. Directories and symlinks have no digest.

What is covered

The capability matrix has an “Action record” column. These operations write here today: firewall blocks and unblocks (automatic and operator), whole-ruleset changes, file quarantine, surgical file cleaning, process termination, mailbox suspension, the account-wide outgoing mail hold, and an operator’s release of either (operate.release_mailbox, one record per part undone), and lifting Stalwart’s ban of an address unblocked in CSM (operate.clear_stalwart_ban, one record per ban). A hold record carries the quoted whmapi1 release_outgoing_email command that lifts it. Everything else appears in the daemon log only. The column is the authoritative list. Tests exercise the operation paths, including failures and concurrent blocks.

The firewall keeps its own audit file because the web UI and the API read it. Its entries also appear here; dry-run decisions, failed attempts and whole ruleset apply/rollback records are recorded directly on this stream. Startup rollback records are written before the daemon requests a restart.

Admission records describe each step of an automatic response the admission ledger admitted: op is respond.block_ip, action the kind of response (block_ip, block_subnet, block_service, promote or challenge), target its canonical target, reason the lane and the absolute expiry, and result the step (reserved, executing, then applied, narrowed, failed or unknown). action_id and action_version name the attempt and the step. A record is written durably before the ledger lets it go, so after a crash it can appear twice; drop a copy whose action_id, action_version and ts match another. An admitted IP block takes these steps while the ledger executes, with auto_response.dry_run false, the firewall on and outside observe mode. Otherwise the ledger previews each response it is handed, so an admitted response is reserved and then observe and is never executed; in dry run with the firewall on outside observe mode it forecasts block responses instead. Their terminal outcomes are counted in the ledger without a new per-attempt action-log row; bounded forecast examples and summaries are separate.

csm action show <id> asks the daemon for what the ledger holds on one response, by the action_id of an admission record or by its candidate ID. It prints JSON: the candidate’s kind, target, check, state, reason, outcome and expiry, each attempt with its lane, reservation, finish and outcome, and for a response refused because a block already enforced it, that block’s action and end (link). A response whose history has retired, or that was never admitted, is reported as not recorded. A retained retired attempt still has its unknown outcome and retired: true; retirement is not proof of success. Retention removes the candidate and all its attempts together. If a retained candidate is missing an earlier attempt, the command reports ledger damage; it does not claim the missing part aged out or return a truncated history.

The command uses candidate_id for the candidate and attempts[].action_id for each attempt, matching the action log’s action_id. It never uses the log’s action operation field for an identifier. The other top-level fields are kind, target, optional check and finding_id, state, optional reason, disposition, expires_at and link, and attempts (always an array, including when empty). Each attempt has seq, state, optional disposition and lane, reserved, optional finished, expires_at, and retired only when true. Times use RFC 3339. link.effect identifies the existing block; an omitted link.expires_at means permanent. Omitted candidate expiry means no expiry was assigned; it is not a permanent block.

This command requires a running daemon and never changes admission or firewall state. Each read is audited with op: respond.block_ip, action: show, actor: cli, actor_detail: csm action show, and the requested ID as target. result: read means the lookup succeeded, never that a block was applied; failed lookups use result: failed and error. These read records have no lifecycle action_id or transition version. They use the existing bounded, best-effort action log outside the admission outbox, so a sink failure does not disable the diagnostic read. The finding-stream exporter accepts these records as reads, salts their targets as opaque references and drops error text; they cannot claim a block was applied.

It offers no resolve, retry or undo operation; durable firewall actions remain under csm firewall actions. The admission.action_show.v1 capability advertises this local read command; it does not indicate that automatic execution is on.

Once an hour the ledger owner appends respond.block_ip summary records that count its decisions on the responses handed to it during the hour ending at ts. A summary sets count, how many responses it covers, and names no target: action is the kind of response, reason the check that asked for it, actor_detail the entry it came through (scan, or a derived entry such as challenge_timeout or central), result the decision (queued, coalesced, observe or refused) and error the refusal reason. An attribution refusal also names its cause: no_observation (no reader stamped the finding), kind_not_allowed (no parser vouched for the address in a way the check accepts), target_mismatch (the response names an address its evidence does not attribute), unreachable (the address is a client behind a trusted proxy, which sends this host no packets of its own), proxy_peer (the address is a known CDN or website-firewall edge, which carries many visitors) or rewritten_log (the web server replaced the logged client address, or CSM could not read its configuration, so the connecting peer is unknown). Refusals at submission and queue placement keep the same cause as refusals at evidence minting. A queued response that a later check refuses (its revalidation, an inventory refresh, a configuration opt-out, a lost queue position after reassessment, or the recheck of a pick before an attempt or a preview) counts as refused too, with its reason and cause, in the hour the ledger ended it. A displaced queued response also counts once. These rows retain that hour even after a newer hour was reported. Hours discarded when a logging outage exhausts summary retention stay discarded; later refusals for them never return as partial summaries. An owner that runs automatic blocks and forecasts them in dry-run mode also counts dry_run decisions, each with a prediction of what live admission would have done: capacity_admit, existing_actual, existing_shadow, ceiling, set_full, pending_recovery, forecast_limited, not_modelled, capacity_evict, capacity_rotate or equal_tier_wait. A prediction is not a refusal reason. A clean stop writes the open hour early, so summaries for the same hour add up; a crash loses the open hour, so the counts are lower bounds. Each summary also sets epoch, the daemon run that counted it: a number that rises at every start of the owner, so summaries of two runs never share one.

Refused previews and submissions displaced from the memory queue count as refused with their reason, including every report merged into a displaced submission. While the owner executes, a block it runs shows only in its attempt rows above: the summaries count a live block pick only when the ledger refuses it at its recheck or reservation. A batch the firewall engine refuses while applying it, and a pick the panel verdict callback allows, add no refused row; the first shows in its attempt’s outcome, the second in the log and in the verdict counters of status. Summaries continue while the ledger or its clock is unavailable, and stopping flushes completed decisions even if the final drain fails. Their hour never precedes the latest recorded admission hour, including clock readings from an incomplete startup.

actor uses the caller’s attribution when available and otherwise identifies the process performing the action. Some web UI requests therefore record as daemon; the web UI’s own action log retains the operator’s source address.

Rotation

The file rotates to actions.jsonl.1 at 10 MB. csm actions reads both. Readers and writers coordinate rotation through actions.jsonl.lock. Readers pin both files before releasing the lock, so rotation cannot hide records and a slow reader does not hold up a writer. CSM handles this rotation itself; the installed logrotate configuration leaves the live file alone and archives actions.jsonl.1, which the sink never writes to again, for 90 days. Older history is compressed there and is not read by csm actions; ship the stream to a SIEM if you need it queryable.

Recording is best effort. Sink errors or panics do not change an action’s outcome. A write waits at most 250 ms, with at most 64 writes outstanding; a stalled sink or saturation can lose records. Normal CLI writes finish before the command exits. Keep this log on a local filesystem and monitor write-error warnings. The stream is not a transactional guarantee that every host change survives a crash.

PAM hook for CSM

The CSM daemon listens on /var/run/csm/pam.sock for authentication events emitted by a PAM module (pam_csm.so). The module is shipped with the package at /usr/lib/csm/pam/pam_csm.so; without it installed the daemon’s PAM listener stays attached but never hears anything, and the dashboard reports the watcher as deaf.

Quick install

sudo csm pam install        # preview first with --dry-run
sudo csm pam status         # confirm

csm pam install stages pam_csm.so into the platform’s security directory (/lib64/security on RHEL, /lib/x86_64-linux-gnu/security on Debian) and appends two lines to the standard service files. These report successful logins:

auth     optional   pam_csm.so # managed-by-csm
session  optional   pam_csm.so # managed-by-csm

Targets: /etc/pam.d/sshd, /etc/pam.d/su, /etc/pam.d/sudo, /etc/pam.d/password-auth (RHEL) or /etc/pam.d/common-auth (Debian). Files that don’t exist on the host are skipped. Every edit creates a timestamped .csm-backup-YYYYMMDDTHHMMSSZ next to the original.

Failed logins need one more line. A PAM module cannot see whether the modules before it accepted the password, so the failure hook goes where only a failed attempt arrives: directly before the terminal pam_deny.so of the shared auth stack (password-auth on RHEL, common-auth on Debian), which the standard remote login services include:

auth     optional   pam_csm.so authfail # managed-by-csm

A jump that skips pam_deny.so on success, such as Debian’s [success=1 default=ignore] pam_unix.so, is widened by one so it skips the hook too; uninstall narrows it back, including jumps to or past the end. Each attempt that reaches this denial is reported once, whichever service ran it. Earlier failures that return before it are not reported by this hook. SSH failures drive the PAM brute-force and credential-stuffing findings; only a successful SSH login clears those failures. Blocking follows auto_response.block_ips. Failures from other services raise the non-blocking pam_auth_failures alert. CSM reads pure-ftpd, Dovecot and Exim logs separately; other services, such as vsftpd, proftpd or webmin, get no blocking signal from PAM.

The installer refuses the failure-hook edit when it cannot prove the placement is safe. Any shared stack that refuses it makes the command exit non-zero, even if another stack was hooked. Success hooks may already have been added to service files or the shared stack. Placement is refused when:

  • the stack is a symlink, which is how authselect manages it on RHEL 8 and later. authselect rewrites the file on its next run, so the hook needs a custom authselect profile; add the line above to that profile’s password-auth before pam_deny.so, widening any jumps that skip the denial so they also skip the hook.
  • there is no single required or requisite pam_deny.so auth line.
  • a jump crosses an include, substack or @include line, so its target cannot be counted. An auth include or @include before the denial is also refused because its own jumps can escape into this stack.
  • a required denial is followed by an auth reset action or an expanded auth include that could clear its failure.
  • a jump cannot be adjusted within PAM’s integer range, or the file uses line continuations or syntax the editor cannot parse.

On Debian and Ubuntu, pam-auth-update --force regenerates common-auth and drops every CSM line; run csm pam install again after it. csm pam status shows whether failed logins are reported.

Hosts installed before the failure hook existed report successful logins only. Run csm pam install again after upgrading; it adds the failure hook and leaves the existing lines alone.

Logins from loopback addresses are ignored. The module passes on the remote host sshd reports; with UseDNS yes that is a host name, which CSM cannot track or block. The daemon logs that once and ignores those logins, so keep UseDNS no, the OpenSSH default.

Safety rails

  • The optional control flag is mandatory: a CSM outage must not block authentication. csm pam install writes nothing else.
  • Test from a second SSH session before closing the one that installed. If SSH or sudo breaks, rename the .csm-backup- file back over the broken target and run csm pam uninstall --keep-module on the live session.
  • The PAM module never reads or modifies passwords, never decides whether to permit auth, and never writes to disk. It opens a Unix socket, writes one line, closes.

Rebuild from source

The package ships the C source alongside the compiled module so operators can rebuild on hosts with a different libc:

cd /usr/lib/csm/pam
make
sudo install -m 0755 pam_csm.so /lib64/security/pam_csm.so   # RHEL
# or
sudo install -m 0755 pam_csm.so /lib/x86_64-linux-gnu/security/pam_csm.so  # Debian

Requires gcc and libpam-devel (RHEL) or libpam0g-dev (Debian).

From a source checkout, make -C build/pam check runs the module through real libpam stacks. It writes test services under /etc/pam.d, so run it as root in a throwaway container.

Uninstall

sudo csm pam uninstall       # removes managed hooks and the module
sudo csm pam uninstall --keep-module   # only edits, leave .so in place

Uninstall is idempotent, removes only lines marked # managed-by-csm, restores any jump count install widened, and creates one fresh .csm-backup- per file it edits. Files with no managed hooks are left alone. If an edited file has an uncountable jump, an auth include before the failure hook, or syntax it cannot parse, uninstall stops with an error naming the file and leaves it as it is; restore the backup or remove the marked lines by hand and fix the jumps. Parse errors identify the line without printing its module arguments. The PAM listener’s dashboard verdict returns to deaf the next time the dashboard polls.

What the dashboard tells you

Dashboard labelMeaning
okAttached AND has emitted at least one finding in the last 7 days.
idleAttached, upstream alive, no recent findings (healthy quiet).
deafAttached, but no upstream process is feeding the watcher.
degradedThe watcher failed to attach on startup (config error, missing kernel).

For the PAM listener specifically, deaf is the verdict whenever the socket has not received a single connection in the last 24 hours and the daemon has been up longer than the 15-minute grace window. Hover the badge in the UI for the install hint.

Pre-start hooks for the CSM daemon

CSM does not implement its own hook protocol because systemd already ships one: drop-in units. This document shows how phpanel-server-agent (or any operator-supplied automation) wires a pre-start script that runs before the daemon binds.

Why drop-ins, not a custom hook directory

Inventing a hook_pre_start: config or a magic /etc/csm/hooks/ directory would duplicate systemd.unit(5) drop-ins, which already support: per-service ordering, ExecStartPre= (cancel start on failure), EnvironmentFile= (load secrets), User=/Group= (privilege drop), and Conditional*= (skip when prerequisites absent). Every distro CSM supports already runs systemd. Use it.

Recipe for phpanel-server-agent

  1. Author your hook script. Convention: /opt/phpanel/hooks/csm-prestart.sh.

    #!/bin/bash
    set -e
    # Re-resolve the panel hostname and stash the IP for CSM's infra_ips.
    PANEL_IP=$(getent hosts panel.example.com | awk '{print $1}' | head -1)
    if [ -n "$PANEL_IP" ]; then
        cat > /etc/csm/conf.d/10-phpanel-runtime.yaml <<YAML
    infra_ips:
      - $PANEL_IP
    YAML
    fi
    
  2. Install the drop-in. Copy the example from /usr/lib/csm/profiles/csm-prestart.example.conf (shipped by the CSM package) to /etc/systemd/system/csm.service.d/10-phpanel-prestart.conf, editing the ExecStartPre= line to point at your script.

  3. Reload systemd.

    systemctl daemon-reload
    systemctl restart csm.service
    
  4. Verify with systemctl cat csm.service - your drop-in should appear at the bottom under # /etc/systemd/system/csm.service.d/....

Standard environment available to the hook

  • CSM_HOSTNAME - set by CSM at runtime, not available in ExecStartPre. If you need the hostname pre-start, read /etc/hostname directly.
  • CSM_INFRA_IPS - same caveat. Resolve from your panel agent’s context (it knows the panel address).

Failure semantics

ExecStartPre= blocks startup on non-zero exit. Use ExecStartPre=-/path (dash prefix) when you want CSM to start even if the hook fails. For phpanel-server-agent’s reconcile step, the dash prefix is recommended so a transient panel outage doesn’t keep CSM from running.

Supply-chain Advisory Database

CSM’s supply-chain check (supply_chain_vuln) parses customer dependency lockfiles – composer.lock and package-lock.json – and matches the resolved versions against a local advisory database. The scanner ships in the binary; the advisory data does not. With no advisory file present the check is a silent no-op.

This mirrors the YARA-forge mirror posture (see ROADMAP item 1): CSM carries the matching machinery, and the data is delivered out of band by an operator or a sync job. Shipping a stale CVE snapshot inside the binary would give false confidence and rot between releases.

Location

<state_path>/advisories/supply-chain.json

state_path is the daemon’s configured state directory. The file is read on each scan; updating it requires no daemon restart and no recompile.

Format

OSV-subset JSON. A dependency version is flagged when it falls inside any range: version >= introduced AND (fixed == "" OR version < fixed). introduced: "0" means “from the first release”.

{
  "advisories": [
    {
      "ecosystem": "npm",
      "package": "lodash",
      "ranges": [{ "introduced": "0", "fixed": "4.17.21" }],
      "id": "CVE-2021-23337",
      "severity": "high",
      "summary": "Command injection via template."
    },
    {
      "ecosystem": "composer",
      "package": "vendor/package",
      "ranges": [{ "introduced": "2.0.0", "fixed": "2.4.1" }],
      "id": "CVE-2024-xxxxx",
      "severity": "critical",
      "summary": "..."
    }
  ]
}
  • ecosystem: composer (Packagist vendor/name) or npm (package name as it appears under node_modules/).
  • severity: critical -> Critical finding, high -> High, anything else (medium/low/empty) -> Warning.
  • Version comparison is numeric per dotted segment; a leading v and any pre-release/build suffix (-rc1, +build) are ignored.

Populating it

Generate the file from a public feed and write it to the path above on a schedule. Two common sources:

  • OSV (https://osv.dev) for the npm/Packagist ecosystems. Export the relevant ecosystem ranges into the subset format above.
  • WPScan / wpvulndb style feeds for WordPress plugin advisories (future ecosystem; not parsed by the current lockfile scanner).

A signed mirror job analogous to the YARA-forge mirror is the intended long-term delivery mechanism; until then an operator cron that fetches and transforms a feed is sufficient.

Scope (current)

  • Parsed: composer.lock, package-lock.json (npm lockfile v1, v2, v3).
  • Not yet parsed: requirements.txt / Pipfile.lock (Python), Gemfile.lock (Ruby), WordPress plugin manifests. These are follow-on ecosystems for the same matcher.

Upstream Threat-Intel HTTP Contract (v1)

This document specifies the HTTP API CSM expects when reputation.upstream.enabled: true. A panel host (or any HTTP server) can implement it; phpanel’s reference implementation lives in phpanel-server-agent.

Endpoint

GET <reputation.upstream.url>/lookup?ip=<ip>

reputation.upstream.url must be an https:// URL with a host. Plain http:// is accepted only for loopback hosts such as 127.0.0.1, [::1], or localhost. CSM appends /lookup to that base URL.

Authentication

CSM sends Authorization: Bearer <token> when reputation.upstream.token (or its env var) is set. The server SHOULD reject requests with a missing/invalid token via HTTP 401.

Before each HTTP request, CSM reads reputation.upstream.token_env from its process environment. A non-empty value overrides the static token field. Editing an environment file or exporting a value in another shell requires a daemon restart to take effect; configuration reload does not replace the process environment. See credential rotation.

Request

  • ip query parameter: IPv4 dotted-quad (e.g. 1.2.3.4) or IPv6 in any RFC 5952 form. The server SHOULD normalize before lookup.

Response

200 OK

{
  "ip": "1.2.3.4",
  "score": 75,
  "source": "upstream",
  "ttl_sec": 900
}
  • ip (required): echoes the input.
  • score (required): integer 0..100. Higher means worse. 0 is “no signal” and the aggregator excludes it from averaging.
  • source (optional): free-form identifier the server may set for audit-log surfacing.
  • ttl_sec (optional): number of seconds CSM should cache this answer. Overrides reputation.upstream.cache_ttl_min. The server SHOULD return the smaller of its own cache lifetime and a sane default.

401 / 403

CSM treats as “no signal” - does not retry, does not cache. Logged as a warning so operators see auth misconfigurations.

4xx / 5xx other

Treated as “no signal.” The aggregator continues with whatever scores the other sources returned.

CSM rejects malformed 200 responses, including an ip field that does not match the requested IP or a score outside 0..100.

Caching

CSM caches each (ip, score) pair locally for cache_ttl_min minutes (default 15) or for ttl_sec if the response provides it. A cache hit does not contact the upstream. Bursts of detection events that touch the same IP within the TTL window produce a single upstream call.

The local cache is capped at 10,000 IPs. When the cap is reached, CSM prunes expired entries first and then removes the entry with the nearest expiry time until the cache is back under the cap.

After repeated upstream failures, CSM opens a short circuit breaker and temporarily treats the source as “no signal.” When the cooldown expires, only one probe request is allowed through; concurrent callers keep using the fail-open path until that probe succeeds or fails.

Sample server stub (Go)

http.HandleFunc("/api/csm/ti/lookup", func(w http.ResponseWriter, r *http.Request) {
    if r.Header.Get("Authorization") != "Bearer "+os.Getenv("PHPANEL_CSM_TOKEN") {
        http.Error(w, "unauthorized", 401); return
    }
    ip := r.URL.Query().Get("ip")
    score := lookupCachedScoreFromAbuseIPDBOrRedis(ip) // your impl
    json.NewEncoder(w).Encode(map[string]interface{}{
        "ip": ip, "score": score, "source": "phpanel-cache", "ttl_sec": 900,
    })
})

Verdict Callback HTTP Contract (v1)

When auto_response.verdict_callback.enabled: true, CSM POSTs each auto-block decision to the configured URL and reads an advisory response. The endpoint is not a per-tenant nftables enforcement plane - it lets phpanel observe and label decisions, downgrade them to audit-only, and attach tenant attribution.

CSM runs its local block validation before calling the endpoint. Malformed IPs, disabled IPv6 targets, infra IPs, and configured block limits are rejected locally and are not sent to the callback.

Callers

Two parts of CSM call the endpoint, told apart by the request source:

  • auto_response: each automatic block of a single address, after local validation and before the auto_response.dry_run gate. severity is "auto". An allow verdict skips that block. In dry run with the firewall on outside observe mode, the admission ledger also asks for locally eligible block attempts, so one block can bring an additional request. Its requests carry CSM auto-block: <check> as the reason, run up to four at a time and honour allow_unsigned and require_response_signature; an allow ends the forecast without an example. Existing effects, protected targets and full capacity need no owner request; retries ask anew and reload discards old policy answers. The v1 request contains no dry-run field, and request totals are not an exact multiple of legacy calls.
  • bpf_enforcement: only when bpf_enforcement.verdict_callback: true is also set. CSM asks about the destination of an outbound connection the BPF hook denied, or would have denied in dry-run. reason is bpf_enforcement:<check>:<port> and severity is the finding’s severity (WARNING, HIGH or CRITICAL). The answer only annotates findings: the verdict, tenant_id and note are added to the finding details. It never changes the kernel decision, which is made before the call. Answers are cached for one minute per destination, reason and severity, and the finding that triggers a lookup is sent without one.

Subnet blocks, promotion of an existing block to permanent, and manual blocks from the CLI, web UI or API do not call the endpoint.

Endpoint

POST <auto_response.verdict_callback.url>
Content-Type: application/json
Accept: application/json
X-CSM-Signature: sha256=<hex>          (HMAC-SHA256 over raw body, omitted if no secret configured)

auto_response.verdict_callback.url must be an http:// or https:// URL with a host. CSM POSTs directly to that URL (no path appended).

Authentication

CSM signs the raw request body with HMAC-SHA256 using the configured secret and sends the digest in X-CSM-Signature: sha256=<hex>. The secret is resolved once per request/response exchange: if hmac_secret_env is set and the env var is non-empty, it wins over the static hmac_secret field. Request and response verification use the same key for that exchange. The value comes from the running process environment; editing an environment file requires a daemon restart, even if configuration is reloaded. See credential rotation.

CSM refuses to enable the callback without a non-empty hmac_secret or resolved hmac_secret_env value. Operators can set allow_unsigned: true only for staged rollouts that intentionally send unsigned requests and accept unsigned responses. Without that explicit opt-in, CSM rejects an unsigned allow verdict and proceeds with the default block.

The server MUST verify the signature with constant-time compare and reject unsigned or invalid requests with 401.

Response signing (required by default)

The server MUST sign its response body with the same HMAC-SHA256 scheme (same secret, same X-CSM-Signature: sha256=<hex> header). CSM verifies the response signature with constant-time compare before parsing the verdict; an unsigned or forged response is rejected and CSM falls back to its default block. Without this check an on-path attacker could downgrade every “block” decision to “allow”.

If the server cannot sign responses yet (e.g. mid-rollout), operators can set auto_response.verdict_callback.require_response_signature: false in csm.yaml to temporarily accept unsigned responses. The default is true and there is no other way to disable the signature requirement.

When response signing is disabled but a secret is configured, CSM still checks any nonce or timestamp fields the server echoes. A nonce must match the request nonce, and a timestamp must stay within the five-minute skew window. A legacy response that omits both fields keeps working.

Request body

{
  "ip": "203.0.113.10",
  "reason": "mail_bruteforce",
  "severity": "auto",
  "source": "auto_response",
  "nonce": "9a7e84f0c123ab45e102d4f8b9c1d3e0",
  "timestamp": 1748345600
}
  • ip (required): the address CSM is about to block, or for bpf_enforcement the connection’s destination.
  • reason (required): short reason string, for example CSM auto-block: wp_login_bruteforce.
  • severity (optional): "auto" for auto-response decisions; the finding’s severity for bpf_enforcement.
  • source (optional): "auto_response" or "bpf_enforcement"; see Callers.
  • nonce (always sent): a 128-bit hex random value unique to this call. The panel MUST echo it verbatim when response signing is on. When response signing is disabled, CSM still checks the value if the panel echoes it.
  • timestamp (always sent): unix seconds when CSM produced the request. The panel does not need to validate it but MAY use it to reject stale calls.

Response body (200 OK)

{
  "verdict": "block",
  "tenant_id": "tenant-42",
  "note": "matched dovecot user pattern",
  "nonce": "9a7e84f0c123ab45e102d4f8b9c1d3e0",
  "timestamp": 1748345600
}
  • verdict: "block" (default; also returned if field omitted) or "allow" to override. Any other string is rejected by CSM as a protocol error, treated as “no decision” (CSM proceeds with the default block per fail-open semantics).
  • tenant_id (optional): attribution string; CSM logs it alongside the decision.
  • note (optional): free-form note logged for the verdict.
  • nonce (required when response signing is on): MUST equal request.nonce. CSM compares with constant-time equality and rejects mismatches as replay attempts. When response signing is disabled, CSM still checks this field if present.
  • timestamp (required when response signing is on): unix seconds when the panel produced the reply. CSM rejects replies whose timestamp drifts more than five minutes from local clock. When response signing is disabled, CSM still checks this field if present.

Failure semantics

  • Network error / timeout / non-200 HTTP response: CSM logs a warning and proceeds with the default block. The hook is fail-open at the transport level.
  • Local nonce generation failure: CSM does not send the request and proceeds with the default block.
  • 200 OK with verdict: "allow": CSM does not modify nftables; logs the override to stderr. If no HMAC secret is configured, this is accepted only when allow_unsigned: true is set.
  • 200 OK with verdict: "block" or omitted: standard block path runs (which still honors auto_response.dry_run if set).
  • 200 OK with unknown verdict string: rejected; treated as fail-open (default block).
  • 200 OK with verdict: "allow" and no HMAC secret while allow_unsigned is false: rejected as an unsigned downgrade; treated as fail-open (default block).
  • 200 OK with missing or invalid X-CSM-Signature (and require_response_signature not turned off): rejected as forged; treated as fail-open (default block). Operators see the rejection in stderr; recurring rejections indicate either a panel-side rollout gap or an active on-path attack.
  • 200 OK with mismatched nonce or timestamp drift over five minutes: rejected as replay whenever a secret is configured and the field is present; treated as fail-open (default block).
  • 200 OK with missing nonce or timestamp: rejected as replay when response signing is required. Missing replay fields are accepted only when require_response_signature: false is set for a legacy response shape.

Limits

  • CSM caps response body at 64 KB. Larger responses are rejected.
  • Default request timeout is 2 seconds (configurable via timeout_sec, range 1-30).

Sample server stub (Go)

http.HandleFunc("/api/csm/verdict", func(w http.ResponseWriter, r *http.Request) {
    body, _ := io.ReadAll(r.Body)
    if !verifySignature(r.Header.Get("X-CSM-Signature"), body, secret) {
        http.Error(w, "bad signature", 401); return
    }
    var req struct {
        IP, Reason, Severity, Source string
        Nonce string
    }
    _ = json.Unmarshal(body, &req)

    tenant := lookupTenantOwning(req.IP) // your impl
    verdict := "block"
    if isPanelInfra(req.IP) {            // never block our own infra
        verdict = "allow"
    }
    respBody, _ := json.Marshal(map[string]any{
        "verdict": verdict, "tenant_id": tenant,
        "nonce": req.Nonce, "timestamp": time.Now().Unix(),
    })
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(respBody)
    w.Header().Set("X-CSM-Signature", "sha256="+hex.EncodeToString(mac.Sum(nil)))
    _, _ = w.Write(respBody)
})

When to graduate this to per-tenant enforcement

This v1 hook surfaces tenant attribution but does not synthesize per-tenant nftables rules. If/when phpanel wants real per-tenant enforcement (block IP X from tenant Y’s mailboxes only, allow elsewhere), that’s a separate plan that will need:

  • Multi-set nftables rule generation per tenant
  • A way to map an IP+tenant pair to an nftables verdict
  • Lock-step config sync between CSM and the panel’s tenant table

Do that work then, against a real call site, not on spec.

Fleet Correlation Contract (v1)

CSM is a per-host sensor. It does not correlate findings across hosts itself; cross-host (“fleet”) correlation is owned by phpanel, which already receives every deduplicated finding through the per-finding webhook. This document defines the contract phpanel implements to turn that stream into one fleet incident per attacker instead of N identical per-host incidents.

This is the phpanel-side correlation design: CSM stays a sensor with no new inbound network surface on hosts. A peer-to-peer ingest endpoint inside CSM was considered and rejected for v1 (it would add an inbound auth plane and key distribution to every host).

What CSM already provides

With alerts.webhook.type: phpanel, CSM POSTs one HMAC-signed message per deduplicated finding that reaches alert.Dispatch (see the per-finding webhook in internal/alert/webhook.go). This sink runs before operator alert suppression and the operator alert rate limit, so phpanel receives the correlation stream even when local notifications are muted:

POST <alerts.webhook.url>
Content-Type: application/json
X-CSM-Signature: sha256=<hex>      (HMAC-SHA256 over the raw body)
X-CSM-Hostname: <hostname>

{
  "hostname":  "host-7.example",
  "timestamp": "2026-05-29T10:00:00Z",
  "finding": {
    "check":     "ssh_bruteforce",
    "severity":  2,
    "source_ip": "203.0.113.10",
    "message":   "...",
    "timestamp": "2026-05-29T10:00:00Z",
    "tenant_id": "...",   // when known
    "domain":    "...",   // when known
    ...
  }
}

CSM keeps each finding in a durable queue and delivers them in order. A 2xx answer acknowledges a finding. HTTP 400, 413 or 422 refuses the record itself: CSM moves it to a bounded local archive, counts it as lost and sends the next finding. Use these codes only for a problem with that one record. After a long run of refusals with nothing accepted, CSM stops archiving and retries instead, so a panel that refuses every request does not lose the queue. The hold survives config reloads and daemon restarts and ends after an accepted delivery. Redirects are not followed and do not acknowledge findings. Any other failure is retried with backoff, and later findings wait behind it.

Every field phpanel needs to correlate across hosts is present: hostname identifies the reporting sensor, and finding.source_ip identifies the attacker when the detector has a remote source. For mail_subnet_spray and smtp_subnet_spray, use source_ip when present (older findings); otherwise use the single subnet in finding.cidrs. Other checks’ cidrs, including http_asn_crawl, are response targets and must not become an attacker key. Severity is the CSM numeric enum (0 warning, 1 high, 2 critical). No extra CSM API or peer transport is required; fleet correlation is a matter of phpanel aggregating the stream it already receives.

Correlation rule (phpanel side)

  1. Verify X-CSM-Signature (HMAC-SHA256 over the raw body) with the shared secret before trusting any field. Reject unsigned or mismatched messages.
  2. Group incoming findings by the attacker address described above within a sliding window (recommend matching CSM’s 15-minute incident merge window).
  3. When the same attacker address appears in findings from two or more distinct hostname values inside the window, open one fleet incident keyed on that address rather than per-host incidents.
  4. Attach each contributing (hostname, finding) to the fleet incident’s timeline so an operator sees the spread across the fleet.
  5. Account/mailbox/domain identifiers are host-local; do not use them as the cross-host key (they collide across tenants). Use only the attacker address as the cross-host key.

Severity and de-duplication

  • A fleet incident’s severity is the max of its contributing findings.
  • Findings carrying no attacker address (host-integrity events, config drift) are not fleet-correlated; they stay per-host.
  • The same (hostname, attacker address, check) arriving repeatedly is the normal heartbeat of an ongoing attack; collapse on the attacker address and count contributions rather than opening a new incident per message.

Out of scope (v1)

  • Peer-to-peer ingest between CSM hosts.
  • A CSM-side fleet incident store. CSM remains stateless with respect to other hosts; the fleet view lives in phpanel.
  • Cross-host auto-block orchestration. phpanel may drive per-host blocks through each host’s existing control surface; CSM does not coordinate blocks across the fleet.

Abuse Report Contract (v1)

When reputation.report.enabled: true, CSM queues a minimized report for each eligible confirmed-abuse finding, one copy per usable target. This page defines the request a collector receives and the answers CSM understands. Use it to build a private collector. The central abuse database implements the same contract and accepts ed25519 targets only. Node setup is in Threat Intelligence.

What CSM reports

CSM reports a finding only when all of these hold:

  • Its check maps to a class listed in reputation.report.classes. The mapping is in Threat Intelligence.
  • It is Critical, or High or above for bad_asn_egress and spam.
  • It names a source address that parses as an IP address.
  • Its finding timestamp is not zero.
  • That address passes CSM’s protection checks. CSM excludes private, loopback, link-local, multicast, carrier-grade NAT, documentation, benchmarking and selected special-purpose ranges, literal IPs and CIDRs in infra_ips and firewall.infra_ips, Cloudflare edges, and built-in or operator-configured crawler IP ranges. This check does not resolve infrastructure hostnames or consult rDNS crawler results. Teredo clients are not excluded solely for using Teredo. For NAT64 and 6to4 sources, the same checks also apply to the embedded IPv4 address.
  • For spam, the address does not belong to a shared mail service.

Request

POST <target url>
Content-Type: application/json
X-CSM-Node: <node_id>
X-CSM-Key: <key_id>
X-CSM-Timestamp: <unix seconds, UTC>
X-CSM-Nonce: <32 lowercase hex characters>
X-CSM-Signature: ed25519=<hex> | sha256=<hex>
Authorization: Bearer <token>          (hmac targets whose token_env variable is set)

CSM posts to the configured URL as written and appends no path. Remote URLs must use HTTPS; plain HTTP is accepted only for loopback hosts. CSM does not follow redirects: a 3xx answer is a failed delivery and is retried. Each request times out after 15 seconds.

Body

{"ip":"203.0.113.10","class":"bruteforce","count":1,"first_seen":"2023-11-14T22:13:20Z","last_seen":"2023-11-14T22:13:20Z"}
  • ip: the source address in canonical text form. IPv6 is compressed and lowercase. An IPv4-mapped IPv6 address is sent as dotted IPv4.
  • class: bruteforce, php_relay, credential_stuffing, bad_asn_egress or spam.
  • count: observations in this report. Version 1 always sends 1.
  • first_seen, last_seen: RFC 3339 UTC time of the finding, which may carry fractional seconds. Version 1 sends the same value in both.

The body carries no hostnames, account names, domains, mailboxes, paths or process data. The signature covers the SHA-256 of the body, so verify against the raw bytes received, not a re-encoded copy.

Signature

CSM signs a message built from six fields followed by the timestamp. Each field is written as a 4-byte big-endian length, then its bytes:

OrderFieldValue
1NodeX-CSM-Node value
2KeyX-CSM-Key value
3MethodPOST
4PathDecoded path of the configured URL, or / when it has none. The query string is not included.
5Body hashSHA-256 of the raw body, 32 raw bytes
6NonceX-CSM-Nonce value

The X-CSM-Timestamp value follows as an 8-byte big-endian signed integer.

The path is the one in CSM’s configuration. A collector behind a proxy that rewrites the path must verify against the original path.

The transport selects the algorithm:

  • ed25519: an Ed25519 signature (RFC 8032, not prehashed) with the node’s private key. key_env must hold the 64-byte private key encoded as 128 hex characters, not a 32-byte seed. The signature is sent as X-CSM-Signature: ed25519=<128 hex characters>. The collector verifies it with the public key it registered for that node and key ID.
  • hmac: HMAC-SHA256 keyed with the key_env value exactly as set, not hex-decoded, sent as X-CSM-Signature: sha256=<64 hex characters>.

This differs from the per-finding webhook, whose HMAC covers only the body.

Collector checks

A collector must:

  1. Find the key for X-CSM-Node and X-CSM-Key.
  2. Rebuild the signed message from the raw body and headers, then verify the signature. Compare HMAC values in constant time.
  3. Reject a timestamp outside a window you choose, for example five minutes either side of your clock.
  4. Remember each node and nonce pair for that window and reject a repeat.

Answer these failures with 401 or 403, never 422. CSM keeps retrying queued reports so approval or clock repair can restore delivery. Queue overflow can still discard reports during the outage; see Delivery.

Test vector

This is a signature test using a documentation address. CSM’s reporting gate excludes that address from real reports.

Inputs:

  • X-CSM-Node: node-1
  • X-CSM-Key: key-1
  • Target URL https://abuse.example.net/report, so the path is /report
  • The body shown in Body, without a trailing newline
  • X-CSM-Nonce: 0123456789abcdef0123456789abcdef
  • X-CSM-Timestamp: 1700000000
  • HMAC secret example-secret

Body SHA-256:

56f3e2c77c1456d41f210109cd069f5795c66ea3d825af3ac5f1d0bc1a008d2c

Signed message, hex:

000000066e6f64652d31000000056b65792d3100000004504f5354000000072f7265706f7274
0000002056f3e2c77c1456d41f210109cd069f5795c66ea3d825af3ac5f1d0bc1a008d2c
000000203031323334353637383961626364656630313233343536373839616263646566
000000006553f100

Header:

X-CSM-Signature: sha256=ae5e149cc7789d78c65b30d8d52a372a6d3bb999a84355d88d42f9c1874666e8

An ed25519 target signs the same message bytes.

Responses

CSM ignores the response body.

AnswerWhat CSM does
2xxDelivered. The report leaves the queue.
409Treated as delivered. Use it only for a report already stored.
422Dropped and counted as lost, never resent. Use it only for a body you will never accept.
429Retried after a positive Retry-After, in seconds or as an HTTP date, capped at one hour. A missing, invalid, zero or past value uses one minute. Repeated throttling can increase short waits as described below.
Any other status, timeout or network errorRetried after one minute.

Threat Intelligence describes how waits grow while a collector keeps throttling.

Delivery

  • Reports first enter a bounded memory queue before being written to disk. Memory overflow, failed persistence or a crash before persistence can lose reports. Normal shutdown drains accepted memory work to disk.
  • Persisted reports survive restarts and are sent to each target in queue order. When a target fails, its later reports wait for the next pass while other targets can continue receiving theirs.
  • reputation.report.spool_max is one shared capacity across all targets. Each report uses one entry per target. When full, CSM drops the oldest entries across the whole queue, including entries for other targets.
  • Target names identify queued destinations. Use unique, stable names: the last usable target with a duplicate name wins. Removing or renaming a target, or a key error that makes it unusable at startup, discards its queued entries on the next delivery pass.
  • A retry sends the same body with a new nonce, timestamp and signature, so the same report can arrive more than once, for example when an answer is lost. Deduplicate on node and body hash, and answer 409 for a body already stored.

Signed scored-set downloads

When reputation.central.auth_target names an ed25519 target, CSM signs each scored-set download with that target’s node key:

GET <set_url>[?since=<version>]
X-CSM-Node: <node_id>
X-CSM-Key: <key_id>
X-CSM-Timestamp: <unix seconds, UTC>
X-CSM-Nonce: <32 lowercase hex characters>
X-CSM-Signature: ed25519=<128 hex characters>

The signed message is built as in Signature, with method GET, the decoded path of the download URL, and the SHA-256 of an empty body. The query string is not signed. Each request has a fresh nonce. CSM sends no body, no bearer token, no credentials from the URL and no other X-CSM- headers, and it does not follow redirects. A full download carries no since; a later one carries exactly one.

AnswerWhat CSM does
200Verifies the signed snapshot or diff, then uses it.
304Keeps its current list.
401Keeps its last verified list, logs that the node was refused, and tries again at the next refresh.
Any other status, timeout or network errorKeeps its last verified list and tries again at the next refresh.

Sample collector (Go)

This HMAC signature-verification skeleton is not a complete collector. secretFor returns the shared secret for a node and key ID, or nil when unknown. Before using it, implement the timestamp and replay checks above, body validation, duplicate detection and durable storage at the marked point. Return 2xx only after storing the report. If configured, also verify the bearer token. The body limit rejects oversized requests rather than verifying a truncated prefix.

// signedMessage rebuilds the bytes CSM signed for this request.
func signedMessage(r *http.Request, body []byte) ([]byte, error) {
	ts, err := strconv.ParseInt(r.Header.Get("X-CSM-Timestamp"), 10, 64)
	if err != nil {
		return nil, err
	}
	sum := sha256.Sum256(body)
	var msg []byte
	for _, f := range [][]byte{
		[]byte(r.Header.Get("X-CSM-Node")),
		[]byte(r.Header.Get("X-CSM-Key")),
		[]byte(r.Method),
		[]byte(r.URL.Path), // the path CSM was configured with
		sum[:],
		[]byte(r.Header.Get("X-CSM-Nonce")),
	} {
		msg = binary.BigEndian.AppendUint32(msg, uint32(len(f)))
		msg = append(msg, f...)
	}
	return binary.BigEndian.AppendUint64(msg, uint64(ts)), nil
}

func handleReport(w http.ResponseWriter, r *http.Request) {
	if r.Method != http.MethodPost {
		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
		return
	}
	body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 64<<10))
	if err != nil {
		http.Error(w, "read failed", http.StatusBadRequest)
		return
	}
	secret := secretFor(r.Header.Get("X-CSM-Node"), r.Header.Get("X-CSM-Key"))
	msg, err := signedMessage(r, body)
	scheme, sigHex, _ := strings.Cut(r.Header.Get("X-CSM-Signature"), "=")
	sig, hexErr := hex.DecodeString(sigHex)
	if secret == nil || err != nil || hexErr != nil || scheme != "sha256" {
		http.Error(w, "unauthorized", http.StatusUnauthorized)
		return
	}
	mac := hmac.New(sha256.New, secret)
	mac.Write(msg)
	if !hmac.Equal(sig, mac.Sum(nil)) {
		http.Error(w, "unauthorized", http.StatusUnauthorized)
		return
	}
	// Check the timestamp window and the nonce here, then decode and
	// store the report. Answer 409 for a body this node already sent.
	w.WriteHeader(http.StatusNoContent)
}

For an ed25519 target, check for the ed25519 scheme and replace the HMAC step with ed25519.Verify(publicKey, msg, sig).

Building & Testing

Toolchain and prerequisites

Use the Go version required by go.mod (currently 1.27.4), including its formatter. A newer local formatter can disagree with the pinned CI linter. For an installed Go launcher that supports toolchain selection:

export GOTOOLCHAIN=go1.27.4
export PATH="$(go env GOROOT)/bin:$PATH"
go version

Linux tests need PHP CLI and python3-cryptography for the shipped PHP runtime and release verification regressions. The release-verifier tests skip when the module is absent and fail when CSM_REQUIRE_PYTHON_VERIFIER=1, which the CI test jobs set so a missing module cannot pass as a silent skip. The Web UI JavaScript tests need Node 24 or newer the same way: they skip without it and fail when CSM_REQUIRE_NODE=1, which the CI test job sets. Builds with yara,journal,bpf also need CGO, pkg-config, YARA-X 1.23.0 and the systemd development library. Use the release builder or the documented test images. CI images ship the module toolchain and run with GOTOOLCHAIN=local, so a stale image fails instead of downloading another toolchain in every job.

Build

# Standard build (no YARA-X)
go build ./cmd/csm/

# Build with YARA-X support (requires libyara_x_capi)
CGO_LDFLAGS="$(pkg-config --libs --static yara_x_capi)" go build -tags yara ./cmd/csm/

Test

Some tests keep scratch folders under their package directory, because system temporary directories change the behavior they test. Source inventories skip a folder removed during their walk; a missing root or failed read still fails.

Process-context unit tests stub start times for synthetic PIDs so unrelated processes cannot supply cache identity evidence. UID and command mismatches are checked separately. Atomic replacement fixtures require newer birth times only when testing WAF log replacement; plugin-copy tests also work without filesystem birth-time support.

Admission retry tests repeat evidence with its original observation timestamp and check the refusal reason. Changing that timestamp creates conflicting evidence and bypasses episode placement. Forged-account flood tests keep the host and real account backlogged in each evidence class and check repeated alternating turns. A single account pick can pass through sort order alone.

The default CI job runs every package with the race detector, no extra build tags, and a 30-minute per-package timeout. Run its exact test command on Linux:

go test -race -timeout=30m -covermode=atomic -coverprofile=coverage.out -coverpkg=./internal/... ./...

make test and the test step of make ci use -short for local iteration; they do not reproduce the full CI suite. make test-full runs the default-tag suite with the CI flags (no -short, fresh results, the 30-minute package timeout). Neither default command tests the shipped optional backends. The additional required production job runs:

scripts/production-tests.sh portable

That script selects all packages with yara,journal,bpf, -race, -count=1, -p=2 and -timeout=30m, retains JSON execution evidence, and checks the test inventory. See production and kernel tests for the builder, tagged lint/security commands and required kernel execution.

On macOS, use scripts/go-linux.sh for Linux code. Build the repository’s PHP-enabled test image once with an available container builder:

docker build -f build/Dockerfile.systemd-test -t csm-linux-test .
GO_LINUX_RUNTIME=docker GO_LINUX_IMAGE=csm-linux-test scripts/go-linux.sh \
  bash -ec 'apt-get update -qq && apt-get install -y --no-install-recommends python3-cryptography
    go test -race -timeout=30m -covermode=atomic -coverprofile=coverage.out -coverpkg=./internal/... ./...'

The wrapper derives the default Go version from go.mod, shares persistent caches across worktrees, and grants fanotify/nftables capabilities. Its plain Go image does not include PHP, python3-cryptography or the production CGO libraries. The command above installs Python’s verifier in the disposable test container. An image built with another runtime can be selected through GO_LINUX_RUNTIME and GO_LINUX_IMAGE; local test execution still goes through the wrapper. Kernel firewall regressions use isolated network namespaces:

scripts/go-linux.sh go test -tags nftkernel ./internal/firewall -race -count=1

The separate kernel gate requires an isolated Linux runner with the documented boot capabilities. A successful macOS or default-tag suite does not demonstrate BPF LSM attachment.

Antispam learning benchmarks

Run the learning benchmarks on Linux with repeated iterations:

TMPDIR=/var/tmp/csm-gotest go test ./internal/antispam/learn ./internal/antispam/learnwatch -run '^$' -bench . -benchtime=3x -count=3

Each publication iteration starts with one live anchor and adds one new contributor. Each flood iteration starts from an empty baseline before new messages arrive. Database state is reset between iterations; setup and cleanup are excluded from timings. The flood wait metric is the mean across iterations. The benchmarks check that publication commits, discovery visits the corpus, the second account submits before the flood finishes, and audit coverage completes.

Storage metrics count logical keys and values, including contribution charges, account usage and pending-message indexes. They exclude bbolt page overhead and anchor message bodies. These fixed workloads do not establish maximum database sizes or production latency: message names, contributor mixes, filesystem performance and native rspamd calls change the cost. Runtime allowances remain unchanged.

Fuzz

CSM has fuzz targets for parsers that read attacker-controlled input, including Exim mainlog lines, Dovecot maillog lines, Apache Combined Log Format, /proc/net/tcp rows, wp-config.php bodies, /etc/shadow, auditd comm fields, and finding messages coming back from the WebUI.

Fuzz targets live in *fuzz*test.go files across the internal packages and scripts. Fuzz targets do two things:

  1. Their seed corpus runs as part of the normal test suite. go test ./... executes every seed, so a known-bad input stays a regression test forever.
  2. The actual fuzzer runs with -fuzz=FuzzFoo.

Run a target for a fixed time while investigating:

go test ./internal/checks/... -run=^$ -fuzz=^FuzzExtractPHPDefine$ -fuzztime=30s

Run only the seeds:

go test -run=Fuzz ./...

If the fuzzer finds a crasher it writes the failing input to testdata/fuzz/FuzzFoo/<hash>. Commit that file alongside the fix and the input becomes a permanent seed.

Adding a fuzz target:

func FuzzMyParser(f *testing.F) {
    // Seeds: real-world valid shape, empty, malformed.
    f.Add("valid input")
    f.Add("")
    f.Add("corrupt/truncated")

    f.Fuzz(func(t *testing.T, s string) {
        _ = myParser(s)   // must not panic on any input
    })
}

Keep the target tight: call one function, assert it returns. Output verification belongs in a regular test.

Lint

make lint                        # must pass before push
make fmt-check                   # checks tracked Go files with the pinned formatter

make lint uses repo-local cache directories under .cache/ and the timeout set in .golangci.yml, the same one CI uses. Install the pinned tools with make tools; the golangci-lint version is GOLANGCI_LINT_VERSION in the Makefile, and CI runs the same release. It lints the Linux build, so a macOS host checks the code that ships rather than reporting its linux-only callers as unused. Production-tag lint needs the Linux CGO libraries described in production tests.

make sec, make vuln, and make check-fixtures are the local security, vulnerability and fixture checks. For an aggregate local check use make ci, then run the full default and production commands above. Passing local checks does not replace the required kernel and cloud jobs.

The gosec pin includes the upstream module loader fix. Version 2.30.0 can split large packages in deeply nested checkouts and report false type errors. Keep this pin aligned in the Makefile and both build images until a tagged release includes the fix.

Linter config in .golangci.yml: errcheck, govet, staticcheck, unused, ineffassign, gocritic, misspell, bodyclose, nilerr.

CI/CD

GitLab CI (.gitlab-ci.yml) is the internal build pipeline. It runs lint/test/package jobs, publishes internal packages, mirrors to GitHub, and creates the public GitHub release artifacts.

StageWhat it does
.preManual CI tools and builder image builds (amd64 and arm64) block dependent jobs when their definitions change on a branch.
preflightRelease preflight rejects version tags without a usable cPanel image, after required images are built.
lintPinned golangci-lint and formatter, vet, blocking gosec/govulncheck, fixture privacy, and Prometheus config validation.
testFull default-tag race/coverage suite (30-minute package timeout), shipped-tag lint/security/race gate, required kernel/service gate, four pinned clean-application gates, and the attack replay corpus.
buildamd64 and arm64 release binaries with YARA-X CGO and the yara journal bpf build tags. arm64 builds run under QEMU in rootless BuildKit.
packageRPM + DEB via nFPM
integrationSpin up cloudv-1 AlmaLinux and Ubuntu hosts plus the configured clean cPanel image via phctl, install the pipeline-built amd64 packages, run the integration test binary, collect coverage, and confirm every test server was really deleted. main integration is manual and can omit cPanel; version tags require the cPanel image, baseline-to-candidate package upgrade, installer, WHM, mail watcher and forward guard checks.
signDetached signatures on release artifacts
publishInternal GitLab Generic Package Registry (versioned + latest)
repoPublish RPM/DEB to the public mirrors.pidginhost.com apt/dnf repos
pagesDocs + coverage HTML (GitLab Pages preview)
cleanupRemove old package versions
releaseGitLab release on tags matching v*
githubMirror to GitHub + upload release artifacts (auto on tag push)

Lint and test jobs use the shared Go cache runner. Compiled and analysis caches rotate with the CI tools and builder image tags because Go does not detect changes to installed C libraries. Module downloads remain shared. The default suite executes on every pipeline even when compiled packages are cached. Release build and package jobs stay on separate runners and do not consume these caches.

Preflight, lint, test, build and package jobs retry up to twice when the runner, registry, scheduler or API fails; a failing script is not retried. A newer push to the same ref cancels the older pipeline’s lint and test jobs. Integration, kernel tests and publishing jobs neither retry nor get cancelled. Integration retries the Ubuntu package index update while the mirror syncs; the image repair and the CSM install run once.

The test job uses gocover-cobertura -strict so unresolved package information fails coverage conversion. A successful test job must include every covered package in its report.

When an image definition changes on a branch, run its blocking manual image jobs: ci-tools, or both builder architectures. Preflight waits for those builds, and both release builds wait for preflight while still running alongside lint and tests.

Release builds and preflight run in branch and tag pipelines, including scheduled and manually started runs. They are excluded from merge-request pipelines; merge requests use the checks in the branch pipeline.

Public Releases

To cut a release:

  1. Move the [Unreleased] heading in CHANGELOG.md to the new version (e.g. [2.4.2] - YYYY-MM-DD), commit as release: cut X.Y.Z. CHANGELOG.md holds one block of ten minor versions. The first release of a new block (3.40.0, 3.50.0, 4.0.0) moves the previous block and its link references to docs/changelog/<first>-<last>.md and adds that file to the archive list at the top of CHANGELOG.md.
  2. Tag and push:
    git tag vX.Y.Z
    git push origin main vX.Y.Z
    
  3. Wait. The tag pipeline runs integration, publishes packages to the mirror, creates the GitHub release, and uploads every artifact including the fresh merged-coverage.out. No manual pipeline clicks needed.

Tag pipelines require INTEGRATION_CPANEL_IMAGE to name a clean cPanel CI image, or CSM_RELEASE_WITHOUT_CPANEL to state why no image is available. INTEGRATION_CPANEL_PACKAGE optionally selects its compute package and defaults to cloudv-2. A release taken without cPanel coverage records cpanel_coverage: "absent" in dist/cpanel-release.json; see cPanel release tests.

Tag-specific publish dependencies require preflight, unit tests, fixtures, corpus, production tags, kernel tests, signed artifacts and integration. Repository publication and the GitLab release depend on publish; the GitHub release also directly requires integration and the test gates. A missing csm-kernel runner leaves publication pending. A missing cPanel image fails tag preflight. See cPanel release tests for image acceptance and the roadmap for remaining operational readiness work. These configured dependencies are not evidence of a successful live release run.

The Pages workflow builds the coverage report and badge from merged-coverage.out of the latest GitHub release that carries one (it walks back through releases if the newest is missing the asset), and renders it against that release tag’s sources, so a file removed since the release does not break it. A failed release-list request or response parse stops the workflow. Mirrored pushes to main that change the docs, README, workflow or coverage renderer can run before the new release’s assets exist and show the previous release. The release job requests another Pages run after uploading the assets to refresh the badge; a manual workflow run can retry a missed refresh.

Installs and upgrades on end-user servers come from the GitHub release artifacts or the apt/dnf mirror. The internal GitLab package registry is operational tooling only.

Code Conventions

  • Imports: stdlib, blank line, third-party, blank line, internal. Use goimports -local github.com/pidginhost/csm
  • Errors: Return up the call stack. Wrap with fmt.Errorf("context: %w", err)
  • Store: store.Global() singleton bbolt DB. Always nil-check.
  • State: state.Store handles finding dedup, alert throttling, baseline tracking, latest findings persistence. Passed to subsystems at init
  • Web UI: Vanilla JS, no framework, no build step, and syntax up to ES2019 (a test rejects later syntax). Tabler CSS framework. Each script keeps its helpers in a function scope and adds only to the CSM namespace; the shared runtime is csm-core.js, csm-format.js, csm-page.js and csm-live.js. Use CSM.get() / CSM.post() / CSM.delete() for API calls. Escape string-built markup with CSM.esc(); prefer DOM APIs for attacker-controlled values. Run the script tests with node --test ui/.
  • API responses: answer through writeJSONError, writeOK, writeItems / writeAll / writeCapped and writeJSON, so every route follows the contract in API Reference: JSON errors, ok on actions, lists under items, UTC instants, durations in seconds and severity labels.
  • Logging: New code should use internal/log (wraps log/slog). Legacy fmt.Fprintf(os.Stderr, "[%s] ...", ts()) call sites remain valid until migrated.

Content-bound dismissal state

state.Store.SetBaseline retains a known content hash when a result for the same finding has no hash, including duplicate results and baseline resets. An explicit new hash replaces the earlier one. Findings absent from the new baseline are still removed.

ReconcileDismissedContent accepts fresh content evidence only. Empty hashes and ScanCarryForward snapshots neither adopt content nor revoke a dismissal. Revocation deletes the entry, invalidating pending delivery and resolution callbacks tied to it. Finding keys and fingerprints stay unchanged. The daemon calls it before deduplication, FilterNew and delivery in dispatchBatch, and before filtering and delivery in respondToInitialScan. Every observation in the batch contributes content evidence even when notification deduplication keeps only one finding. For an ended dismissal it returns the batch without that key’s hashless, carried or previously dismissed observations, so deduplication keeps the new content, and it logs the ended dismissal with the key quoted so file names cannot inject log lines or terminal controls. The initial scan retains the full batch in history and audit, including the observations omitted from notification and response. Pending shutdown findings retain ScanCarryForward in their private storage format so replay cannot treat stale content as fresh.

Undo restores the previous content hash only when the dismissal created the baseline or changed its content. Content adopted after a dismissal that left it alone survives the undo.

Response admission primitives

internal/admission is a standard-library-only package that live responses reach only as a preview. Its persisted enum values are fixed by golden tests. Assess sets ReassessBy to the first instant the current class or severity falls, or all roots become stale, without new evidence. Redundant evidence can preserve the tier after another root expires. Each corroborating pair lasts until its local root loses freshness or its support expires; the longest-lived pair determines how long corroboration remains available. Equal deadlines and input order do not affect the result. Direct compromise evidence may expire while corroboration continues to preserve the same class.

SlotLimits.Take adds a scope holding only for one-unit block_ip work. Zero-cost and multi-unit work do not add holdings, even when marked as eviction-capable. Free slots still decrease by the granted cost.

Effect records use an expiry-ordered key followed by the action ID. Decoding requires a canonical value that reproduces the key and all recorded fields. Victim references accept either an admission attempt ID or an import’s eff_ name followed by 32 lowercase hex digits. Record keys remain attempt IDs; an import has no record key. Imports have the fixed tier (C2, High) and occupy general slots. The codecs keep their existing versions: an older build refuses a ledger containing import claims, so such a ledger must stay on a compatible build. A ledger that never executed has no such claims. The effect records’ partition tally rejects negative counts, underflow and overflow; failed additions return the original tally. Combining a tally with set occupancy also returns an error and the original occupancy if either partition’s total is invalid or overflows. A full-set deferral raises the capacity notice for any assessed tier in a live ledger. A preview ledger raises it only for Critical work. A pending block that evicts names its victim in its effect record and its intent; an admission victim’s record names the block that claimed it and stops holding a slot once that block’s application is verified with the victim’s removal proven. A verified block that never removed its victim releases the claim; an unproven removal is an unknown outcome, never a verified one. A definite failure releases the victim claim; unknown outcomes keep it through retirement until the victim expires. An import, a stamped legacy block the firewall lists each turn, is a (C2, High) victim with no record of its own: only the claiming block’s record and intent name it, by its eff_ name, and no other block claims it while that record does. Opening rejects duplicate import claims and requires a general slot above the import tier. Execution uses the current turn’s list; a missing or changed import drops the claim. Scheduling, reservations, observe previews and wake calculations share that list and count at least as many general imports as it names, while preserving a larger occupancy count. An overfull partition funds no replacement until the complete state fits. Observe previews write no claim or effect record; wake calculations write nothing. A preview reports the block whose slot it took, so the owner counts previewed replacements in status; a readback reports none. A member’s sweep also ends its import claim; a delayed outcome may still name the import in its intent without looking for a victim record. A verified eviction marks its victim’s record gone at the evicting block’s outcome. A verified block that two consecutive firewall answers no longer hold is gone from the first of them, whatever removed it: an operator unblock, undo or flush, or a lost kernel element. A held answer in between starts the count again; the count lives in memory, so a restart only delays the mark. Failed answers keep all previous misses and count nothing new. Bounded batches get two answered checks before advancing, so a larger configured capacity cannot hide later records. A record that changed in between is left to what changed it. A gone record holds no slot, so waiting work can take it, and stays until its expiry sweep. The first stored gone record moves the ledger to schema 14, which older builds refuse. Schema 14 requires the live-start marker, effect bucket and tally even after all records expire. A check that marks nothing leaves the schema unchanged. Only a live ledger has effect records, so preview and forecast hosts never write schema 14.

The owner asks at most once a minute, and only while the ledger counts a verified block. The firewall answers with the retirement rule reversed and writes nothing: an element still at the address that no other action’s verified row proves counts as held, even after a forward wall step makes the block’s own row look expired, and so does an address whose family the firewall cannot read. A failed or short answer marks nothing and counts as no miss.

Startup drops the state row of an admission-record victim whose eviction the ledger verified, and of a block the ledger marked gone, so a restart cannot restore an element whose slot the ledger no longer counts. A running owner retires the row of a block it marks gone as it retires an evicted victim’s, so a firewall reload cannot restore it either. A cleanup pass that clears no row is retried twice at once, then after waits that double from two seconds to a minute; a row that leaves ends the wait, and a new eviction goes ahead of the waiting rows and is tried at once. Imports have no durable cleanup record: a restart before retirement restores the import only until its original expiry. Startup readers report an error if the effects bucket disagrees with the schema or an effect is a nested bucket; they do not treat damaged eviction storage as an absent record. Expiry can remove either record first. A late definite outcome still ends the claim using the retained intent. A reservation that expired without running releases its claim. Only records still holding slots contribute to occupancy and expiry wakes. Scheduling retains scope turns while a lane has no usable slot or strictly lower victim. A live ledger on a trusted clock may instead rotate an equal-tier record out of a full partition for fresh work: work whose episode row has had no verified response, never work queued without one. The victim is a verified record of exactly the work’s tier, in the partition of the work’s own lane, whose 30-minute hold passed on both the admission clock and the ceiling’s elapsed time since its verified application. A reboot grants no elapsed time and a degraded reading allows no rotation. Each partition is shared among its active scopes, using the queue’s integer share rule: the scopes whose unexpired, unclaimed records hold its slots, the scopes of queued work that would hold one at the last schedule cycle, and the requester’s. Each live schedule cycle on a trusted reading that reads the victim pool first moves each partition’s remainder cursor to the next active scope and stores it beside the queue cursors, so reservations share the set as their cycle did; a preview ledger never moves it. A zero-share requester waits for a later cursor turn. Work at or over a positive share rotates only its own scope’s records; work below it reclaims only the scope with the greatest excess among those holding a verified record of its tier in that partition at another address, never taking that scope below its share, and waits if that scope has no record whose hold passed. The oldest application, then the lowest action, goes first. Imports never rotate. The claim and its outcome are those of an eviction, opening accepts an equal-tier claim only inside the block’s own lane partition, and a live rotation raises the capacity notice. Notice allocation preserves the new attempt’s recovery hold; insufficient room sends the notice to its fixed overflow record. An older build refuses a ledger holding a rotation claim or a moved set cursor. Saved ceiling turns also keep their earned credits through slot pressure and restart; backoff keeps its existing release behavior. Records and intents without an eviction keep their version 1 encoding.

Policy lookups can overlap across registration, minting and validation, so they must be immutable or safe for concurrent use. Callers must serialize every operation on an inventory generation tracker. Off cPanel, hosting inventory reads account directories without checking mount state: an empty readable root contributes no accounts, while other readable roots still do. An unreadable required root fails the whole snapshot. A changed incarnation token gives an account a new generation, even when the account was replaced between two observations. Every account carries one: cPanel’s recorded creation date, elsewhere the device, inode and birth time of its home directory. An account whose token cannot be read, whose home is not a directory or whose Linux filesystem supplies no birth time fails the snapshot, including a home listed as a symlink. The cPanel creation date must contain only decimal digits and be recorded exactly once. Missing, empty, overlong or overflowing values fail the snapshot. cPanel records zero for its own system entries; recreating one between observations keeps its generation when the token stays the same. An observed absence retires the generation even for a zero-dated entry. Tenant edits do not change the token.

The admission ledger (store.AdmissionLedger) keeps this state durably in the daemon’s state database, in adm: buckets it creates the first time it is opened. The daemon’s admission owner (internal/admissionowner) opens it at startup, settles what a previous process left outstanding after its first clock reading and ceiling, before ingress or admission, and holds the only handle: one goroutine makes every change, records clock readings on a timer, applies the ceiling at startup and, after a reading at the saved limit, on reload or the first tick that sees a changed configured value, refreshes the inventory from complete reads, delivers audit rows to the action log and notices through the queue-health path with their own pacing, and reads status on a timer. Once the ingress has made new decisions, the owner drains it on a short timer. Each group is frozen before its fresh clock reading, so concurrent submissions wait for a reading that includes their observation time. A clean stop drains all held work and checkpoints final decisions before closing the generation. A failed drain, including its clock or publication step, closes admission until a drain succeeds. Its cause survives tick, inventory and reload recovery, so a lasting failure is one stop with its cause. A drain whose only failure is a damaged ledger record is not a failed drain: the arrivals naming that record are discarded and counted lost, the rest commits, admission stays open, and status and csm doctor keep the damage cause until the daemon restarts. If a later write, checkpoint or snapshot fails in that drain, admission closes until a drain succeeds; recovery clears the drain failure but keeps the damage cause for the discarded work. Failed ceiling reloads stay pending until a tick applies and revalidates them; inventory refreshes cannot reopen admission in the meantime. The daemon wires the owner into the response funnels (checks.SetResponseAdmission) while it runs: each automatic response the legacy funnels select is minted as evidence and submitted to the ingress, and derived responses (challenge timeout, central, incident, mail subnet, ASN crawl, netblock, permanent block) carry the root evidence that caused them. Legacy blocking never waits for admission and its outcome never changes a legacy decision. The owner serves its queue as observe previews: a schedule reserves and charges each pick and ends it observed in one transaction, so a crash cannot leave a preview reserved. Each hour it appends counts of its decisions to the action log. One owner serializes every write, and each call is one transaction, so a failed call changes nothing. Admission time comes only from recorded clock readings: a wall clock that steps back never lowers it, a new boot credits no elapsed time, and a reopened ledger admits no new work until it records a fresh reading. Evidence is immutable once published, and later reports of the same observation are kept as bounded links. A later mint that differs only in its owner, because the inventory changed between the two mints, is refused as a stale identity. If the owner and any other field differ, including the finding, the record is invalid. A candidate takes its entry, check and finding link from its primary evidence, coalesces repeated requests without refreshing its queue age, and moves from queued through reserved and executing to one outcome. A proven failure may requeue after a backoff, with at most three attempts in total; an unknown outcome never retries. Reserve and Execute report whether the call granted the step; reading back an attempt that is already reserved or running grants nothing. A new process settles what the previous one left outstanding before it reserves anything: a reservation, which recorded no intent and so dispatched nothing, fails and may retry; a running attempt ends unknown. Nothing is dispatched again. A handle that has granted a reservation refuses to settle, and an unknown outcome needs the intent its attempt recorded until that uncertainty is retired, including when reading the outcome back. A missing intent or intent bucket returns a corrupt-record error without changing the ledger. An empty ledger settles without a clock reading. The owner’s kill-point tests stop it at each step of a block turn and of audit delivery, then start a new owner on the same ledger. After the restart a reservation has failed and retries as its next attempt, a running attempt is unknown and holds its address until it retires, the startup reads hold back every row not finished applied, the consumers see only outcomes finished applied, and no attempt reaches the engine twice. Original attempts keep their charges, and retries consume new charges without renewing the original expiry. Synthetic clocks keep the scenarios independent of runner load.

Candidate records without an existing-effect link keep version 1 and their original encoding. A refused existing-effect candidate can carry the block name and expiry in version 2. Its maximum encoded size includes the link only when present; attempts and audit rows keep their existing formats.

Schema 2 of the ledger adds its queue, and the first open upgrades a schema 1 ledger in the same transaction; legacy work that cannot fit the unassessed general partition refuses the upgrade intact. Ingress, queued and in-flight work share 1000 positions; 200 take only direct compromise or corroborated evidence. Each partition reserves 64 positions for memory transfers, so durable work cannot consume handoff space. In a full partition each verified scope has a fair share: a scope at its share can only replace its own lowest-tier, newest candidate with stronger work, and a scope below its share takes one position from the scope most over its share. A displaced candidate ends as queue overflow. Queued candidates end at their age-out, or at their effect expiry while waiting to retry, and are assessed again when a root’s freshness changes; an inventory refresh ends those whose roots name a retired account. The scheduler serves the reserved lane first, alternating direct and corroborated turns, then the general lane at C3:C2:C1 4:2:1. Inside a class, verified scopes rotate, a scope serves Critical:High:Warning 4:2:1 and a severity serves its oldest candidate; deficit accounting lets a candidate that costs several block units wait its turn without starving. The scheduler’s position persists, and every pick is revalidated before it is returned. Regression tests reopen the ledger during unfinished scope and reserved sub-lane rotations, then check service order across the restart. Overload recovery uses elapsed-time refill and verifies each reservation’s charge, so a fresh cursor or an uncharged pick cannot satisfy the test. Free-slot rotation tests require a granted reservation, its stored effect, and the expected occupancy and charge, including after a limit increase. A missing effect must fail rather than count as proof that no victim was taken. A schedule passes over candidates the caller names as pending, such as one waiting for the panel’s verdict: they are never picked or deferred, their scopes keep their turns, and the next wake waits only for their deadlines. A candidate the caller names as allowed ends without a charge, so it is picked as uncharged work at no ceiling or history cost, even with both spent, and wakes the owner at once. Detectors hand work to a nonblocking ingress that applies the same fair shares against the owner’s latest snapshot of the queue. It judges freshness at that snapshot’s admission time advanced by elapsed monotonic time, so an idle owner does not make new evidence look future-dated. Durable victims remain counted until commit. Reports arriving during a drain remain held for a later report-only commit, and overflow counts persist with their links. Ingress cursor progress and loss counts are checkpointed even when only refusals occurred; publication cannot overwrite later memory decisions. The owner persists held items in groups; an ingress generation that ends without a clean close is recorded as interrupted, since its unpersisted items cannot be counted. A failed snapshot after a committed drain stops new submissions until the owner publishes a snapshot at least as new as that commit, from the same ingress generation. Shared queue damage retains the whole handoff; only arrivals carrying damage are discarded.

Schema 3 of the ledger adds the emergency ceiling; the first open upgrades a schema 1 or 2 ledger in the same transaction, with empty buckets because its recent spend is unknown. The engine sets the hourly ceiling L; a fifth of it, rounded up, is reserved for direct compromise and corroborated work, and the general lane cannot spend it. Each lane refills a token bucket at its hourly rate from elapsed time within a boot, up to ten minutes of that rate with a one-unit floor for a nonzero lane, and the first limit of a new ledger fills both buckets once. A new ledger takes that first limit together with the legacy hourly block count, charged at the end of its hour and subtracted from the fill; a count that cannot be read starts the ledger without credit. The reader rejects null scalar values and uses the last instance of a repeated local hour, even when other hours separate its instances. Retained import reads validate the ceiling totals and the charge links to ordinary attempts. A later limit only clips saved credit; the owner checkpoints elapsed time at the saved rate before changing it. Every reservation, retry included, is charged to the lane its schedule picked, in the reservation’s transaction, after a reserved lane is rechecked against the candidate’s current assessment. A readback grants and charges nothing; its lane must be zero or equal the recorded lane, including when an upgraded attempt has no recorded lane. A charge counts until a full hour of admission time and of elapsed time have both passed, so neither downtime nor a forward clock step releases it early. Schedules serve no more than each lane can charge, and the ledger’s next wake includes when waiting work gains budget. Challenge work spends fair-turn and history budgets without spending ceiling units.

Schema 4 adds storage accounting; the first open upgrades a schema 1, 2 or 3 ledger in the same transaction. Evidence stays stored while a candidate names it as a root. Evidence no candidate names, and candidates that ended before any attempt, each wait in a bounded ring and leave oldest first, so rejected traffic cannot allocate rows without bound. A candidate’s details become history at its first reservation: each reservation charges the growth of their largest possible size to the allowance of its lane, and a fifth of the fixed history budget is reserved for direct compromise and corroborated work. Each allowance admits history at a rate that spreads its size over the seven-day review window and saves at most ten minutes of that rate, so a flood cannot fill a week of history in an hour. Scopes earn history turns in fixed quanta, so large records cannot take more than their share, and a record that has earned its turn holds its lane until the credit covers it. An ended candidate’s details are kept through the review window and, unless the candidate was only previewed, for as long as the firewall row its attempt may have written can last: a failed attempt’s row can outlive a failed restore write, and startup must still know its action while the row could be restored. Endings written before this rule keep their earlier time. They are retired after thirty days, or earlier when an allowance needs the room, oldest first. An unresolved outcome is pinned in a separate recovery reserve until recovery settles it, and new work waits while that reserve is full. Tick refills the allowances and retires history at its target in the clock’s transaction. The next wake runs the scheduler read-only; unassessed or overdue queue entries wake immediately for a sweep. Report links stay readable after a policy change.

Schema 5 adds the outbox; the first open upgrades a schema 1 to 4 ledger in the same transaction. Every admitted step of an attempt (reservation, execution, outcome) writes an audit row in its own transaction, and the reservation holds room for all three in the recovery reserve, which the outbox shares; ended history is retired only after its rows are acknowledged. Response gaps are recorded in the transaction that causes them as coalesced notices, one record per reason, check and action family, with a bounded count of examples, a fixed overflow record per kind and a fixed Critical summary that no flood can refuse. An applied block whose root is a Critical outbound destination (c2_connection) raises a Critical partial-containment record, reason unsupported_containment with outcome applied: the input-chain drop stops replies from the address, but outbound-only traffic to it still leaves. Its response ran, so it is not counted in the Critical summary or the doctor’s gap row. Records may use only a fixed share of the reserve, a key is due for delivery at most once an hour and a summary once a minute after acknowledged delivery. While the owner previews or forecasts, ledger notices go to findings history only, as Warnings starting “Admission preview:”. The independent stopped-admission notice still reaches alert channels as a Warning saying existing blocking is unaffected. While it executes, a gap is an outage of automatic blocking: notices go to the alert channels with their kind’s severity and no preview prefix, and the stopped-admission notice is Critical. A ledger notice keeps the mode it was raised in: a record with an event from while the owner executed, or from a real attempt’s outcome, is delivered live after a dry run starts, and one raised only in a dry run stays a preview after execution returns. Other records from before a restart take the mode the owner starts in. If the startup clock fails, later notice mode boundaries use the first valid clock reading; restored notices keep the initial startup mode. Deliveries before that reading use the saved ledger time to retain the mode of events raised during delivery. Live and preview groups back off separately, so failing previews never delay a live notice. The stopped-admission notice uses the mode at delivery, including a retried one. Stop retries retain their original timestamp, cause and stop identity. Acknowledgements carry the count and first-event time of the record read, so a repeated acknowledgement cannot consume a later record under a reused key. An audit row’s acknowledgement carries the row’s time as well, so one held past the row’s retirement cannot remove the row a re-minted attempt writes later. After the action log write and before the acknowledgement, the owner hands the applied block_ip rows of each batch to Options.Applied; no other row reaches it. A failed call acknowledges nothing, so its consumers see the same row again and must be idempotent by action and transition. A row this process already wrote to the action log is not written again while its consumer or acknowledgement fails, even if new rows sort ahead of it into another batch. Each row’s successful acknowledgement clears only that row’s write receipt; a restart writes pending rows once more. The daemon opens the threat and attack databases before the owner starts, and the consumer refuses rows while either is missing. Delivery errors remain visible in admission.owner.audit_error until a pass succeeds; the independent audit queue reports a stall if a pending batch cannot advance. The daemon marks the address blocked in the attack database, as the legacy path marks a block that applied, and, while the action’s verified firewall row still stands, records a threat database row that ends at the block’s absolute expiry. After that write, a daemon goroutine records those standing blocks where a legacy block is recorded (checks.RecordAdmissionBlocks): a tracker row that netblock escalation counts, a permanent-block strike and the Critical auto_block finding that reaches history, alerts and the block digest. The in-memory backlog uses the existing audit delivery bound; a full consumer leaves the durable audit delivery pending for retry. A block already tracked at its expiry is skipped, so a repeated delivery changes nothing. The tracker batch is saved before strikes and notices: a crash can lose those best-effort side effects but cannot repeat them on replay. An unreadable tracker is left untouched; a failed tracker save emits no strike or notice. The strike promotes nothing, since a permanent block is not an admission response. That goroutine waits for the tracker lock a scan holds; the owner never does. Blocks already queued when the daemon stops are not recorded there; a delivery after the consumer stops is refused and stays pending in the audit outbox, so the next start records it. One durable transaction per delivery batch precedes threat memory publication; read or write errors leave delivery pending and threat memory unchanged. Local evidence survives feed withdrawal; permanent local evidence and longer temporary rows remain unchanged. Local threat writers serialize separately from lookups. The consumer reads the firewall state under the threat writer lock, and unblocks and flush cleanup remove the stored row under that lock after the firewall row, so an unblock that lands during delivery removes what the batch wrote. An applied batch does not hold the lookup mutex during its durable write, so reputation reads and whitelist reload continue while that batch waits. Store transaction callbacks do not call back into threat mutations; loaders finish reading the store before publishing an instance, and reload changes the existing instance’s config whitelist only. Repeats preserve the stored timestamp, counters and undo entries, including after restart. Queue events and attempt outcomes are also counted into five-minute, hourly and daily buckets. Status reads every section in one read transaction without the ledger’s lock or a current clock, each section with its own error, and the pure doctor rules turn it and the ingress’s own health into fixed rows. Missing buckets fail the sections that read them; an unreadable database fails every section. Quiet notice indexes are checked with their records. Health snapshots own copies of the admission view, and a clean ingress generation clears the interruption marker.

Schema 6 adds offense episodes; the first open upgrades a schema 1 to 5 ledger in the same transaction, without inventing episodes for candidates queued before. Each ledger draws episode IDs from its own random nonce and a counter, so IDs never repeat, even in a ledger created again after loss. Opening proves every episode row against the candidates it names. History entries written before episode accounting keep their charges and encoding at upgrade. New entries require the episode allowance; an older entry is exempt only while it has no row for its episode. The ledger, not the caller, assigns each arrival its episode and generation when it persists the arrival; an arrival that names either is refused. Responses of every kind at one target share its episode. An observation made within an hour of the episode’s last one, or while its work is queued or in flight, joins it; a later one opens the next episode, and one older than the previous episode’s end, or already accepted by that episode while its work held it open, is refused as stale. A degraded clock reading never ends an episode, and an episode ends only once the ceiling’s elapsed time has passed its end as well: a forward wall step cannot end it early, and a reboot grants it no time. Each row records that elapsed end, and its quiet hour runs on elapsed time from the reading that places the observation, so a report processed late after a step cannot end it early either; a row written before it existed ends on wall time alone until its next write, including maintenance that leaves its last observation unchanged. Existing elapsed ends survive maintenance, and verified closing replaces the quiet end with the block’s remaining lifetime, read at the block’s execute reading, so a wall step between execution and its outcome cannot shorten it. Execution readings are released when an outcome commits, including startup settlement. A verified episode keeps that boundary: a later verified block of the same episode moves neither end. A legacy verified row gains its missing elapsed end from the original expiry when closed again. An answered block frontier without candidate references retains that verified boundary on both clocks, using schema 15; earlier readers refuse it. Its verified predecessor’s retirement collects it after both ends pass. Address and subnet block histories remain charged while their episode boundary is still active on either clock. Scheduling and wake times count that retained history as unavailable. Bounded cleanup continues past retained histories across ticks, so they cannot hide later expired history. The first row with a positive elapsed end moves the ledger to schema 13 on every host, which older builds refuse. Schema 13 records nothing of the live start or the effect records, so a schema 13 ledger proves them from their own markers. Episode placement refuses an observation dated after the ledger’s current reading, even within assessment’s clock-skew tolerance. This keeps future times out of observation frontiers and prevents an early episode boundary. A queued candidate coalesces later observations before its first attempt. A queued retry remains live but answers new arrivals without coalescing; one that ended before any attempt is followed only by a later observation of that kind. Once a candidate’s effect is verified or previewed, later observations of its kind in the episode are refused as an existing effect, which raises no notice, and still extend the episode. After an unknown outcome they are refused the same way but never extend the episode, which stays open until recovery retires the outcome. Other kinds may queue at that target, but scheduling and admission defer them as pending_recovery until the unknown is retired. An attempt still recorded as running, left by a crash before settlement or by an outcome that failed to commit, holds its target the same way until its outcome is recorded. The hold also covers candidates from earlier episodes. A missing intent bucket refuses the operation without changing the ledger. Each handle keeps the held targets as an index it builds at open from the proven intents and changes only when a transaction that writes or deletes an intent commits, so scheduling and reservation look a target up instead of reading every intent. A final failure or a retired unknown outcome lets a later observation start the next generation. An episode row lives only as long as a candidate it names: ring eviction and history retirement clear that candidate reference but retain its generation, and whether its effect answers its kind, while another candidate holds the row. The last retained candidate takes the row, and each admitted candidate’s history charge covers a row. A target without a row opens a new episode, so a retired candidate’s ID is never minted again. A verified block ends its episode at the block’s original expiry instead of an hour after the last observation, even while work of another kind is queued or in flight; an unknown outcome or a response of another kind does not end it. The expiry is set on the engine’s lease clock, which after a wall clock step back leads admission time; the owner passes that lead to FinishOnLease or, with a victim, FinishEviction, so the episode ends when the kernel element expires in admission time. Its elapsed end is the time left until then, so a forward wall step does not end the episode while the kernel element remains. Verified outcomes use the wall and elapsed values of the last committed clock reading, retained together by the ledger handle and restored at open. An unreadable ceiling or a failed later tick does not prevent recording a result for work that already ran. New work still requires a successful reading. An isolated member’s isolation ends the same way. Retirement and retry staleness stay on the lease frame: late retirement only holds an address longer, and the engine refuses an expired member.

On a live ledger, a verified block whose record is gone stops answering its kind before the episode’s original verified end: a later observation of that kind, made after the removal and with every root observed after it, queues a charged new generation in the same episode. On a live ledger the block line records the generation of its latest verified block, whose history outlives later generations, so queued, ended and forgotten returns keep that evidence gate against the latest removal; such lines also need schema 15. Older lines recover that generation from an unbroken chain of retained predecessors on their next arrival and persist it with the next episode write. A gap leaves the line answered. Opening a ledger does not rewrite older lines, but rejects a reference to an unverified candidate or a block older than another retained verified block of the same kind and episode. Forgetting a refused return keeps the verified boundary until both ends pass. The episode earns no equal-tier rotation. An observation or root at or before the removal is answered or stale. Past the original end, as when a reboot keeps the episode open on elapsed time, the block keeps answering until the episode ends on both clocks. The owner caps a re-admission’s first attempt at the episode’s original verified end, the time left in admission time added to the lease clock; a retry keeps its first expiry. A re-admission with no time left, or whose episode row has been replaced or lost, is refused as expired, or as invalid in a preview. A forecast caps a block returning to an address whose forecast block the session replaced at that block’s expiry. It applies the cap before the firewall snapshot and again before deciding each pick, since an earlier pick can replace that address in the same turn. A shortened expiry must still leave the firewall’s minimum timeout; otherwise the return ends uncharged without changing forecast occupancy or retaining an example.

A candidate refused as an existing effect can name the block that enforces it: the admission action that wrote it, or the firewall’s eff_ name of an operator or earlier block, with its expiry. TerminateExisting ends a queued pick that way, including a retry whose earlier attempt never ran; FinishExisting fails the current attempt, keeping its charge, and ends the pick linked in the same transaction, or leaves it failed on its last attempt. A historical result cannot end a later retry; a repeated linked ending must name the same block and expiry instant. A linked record is candidate version 2 and an admitted candidate’s history pays for its bytes; without room the transaction changes nothing. A linked record requires at least schema 8, with the same buckets; this build opens schemas 7 to 14 without rewriting them, and older builds refuse an unknown schema rather than misread it. An unchanged schema-7 ledger stays readable by older builds until a write needs a newer format. A write never lowers the schema.

A panel verdict that allows a queued candidate ends it refused as verdict_allow, like any other refusal: no attempt, charge or history allocation, one ended count, and the notice that Critical or C3 work raises when it is refused. A retry keeps the charge of its failed attempt. The episode is not answered, so a later observation can queue the next generation. Such a record requires at least schema 9, with the same buckets; a later schema is retained and older readers refuse unknown schemas.

Queued work whose response the configuration no longer allows ends refused as disabled: no attempt, charge or history allocation, one ended count and no notice, since the operator turned the response off. The episode is not answered. The first such record moves the ledger to schema 10, with the same buckets; older builds refuse it. An ingress checkpoint carrying disabled refusal counts also requires schema 10, even without a stored candidate. A full outcome bucket, every key at its largest count, is now 32872 bytes against a bound of 33 KiB.

EndDryRun ends queued work that dry-run mode forecast instead of running: the candidate moves to dry_run without an attempt, ceiling charge, history allocation, audit row or notice. The ending counts one dry_run outcome in the windows and no queue event. A retry keeps its failed attempts, their charges and the history they paid for, which ends; roots it gained while it waited are dropped as on other endings. The work must be ready as a reservation requires: a retry waits out its backoff, an unresolved attempt still holds the target, and stale work is refused. The same ending again changes nothing. The episode is not answered. A dry-run ending requires at least schema 9 and retains any later schema.

SchedulePreview selects queued work for a dry-run forecast. It is not admission: spent live ceiling and history budgets do not stop it, and it reserves, charges and defers nothing for capacity. Every candidate, a challenge included, weighs one fair unit and no history. Until the ledger starts live a challenge’s comparison preview still spends live history, so the selection passes one over, with no wait or fair turn, while the live history budget cannot pay for it, as a schedule does, and serves it on a lane that can. Challenge picks share that live allowance within a selection; an earlier pick leaves less for the next. On a started ledger a challenge preview spends no history and is selected as other work is. Direct and corroborated work alternates within a bounded reserved share of each batch, and the general lane serves the rest by class and severity, where reserved-eligible work competes too. Revalidation, age-out, retry backoff and recovery holds gate it as they gate a schedule, a held target is deferred as pending_recovery, a stale live ceiling or storage_share wait clears as a schedule clears it, pending work is passed over, and the live schedule and the selection share one fair position, which advances once per selection. PreviewWake is the matching wake: now while selectable work waits, otherwise a backoff end, a queued deadline or, until the ledger starts live, when the live history budget can pay for a passed-over challenge, whatever the other live budgets. CeilingSnapshot reads the committed ceiling and its retained charges in one read transaction, proven against each other, for a forecast to start from.

Preview selection and wake calculations check saved ceiling, storage and live candidate history even though live budgets do not limit them. Damaged accounting refuses the operation without advancing the fair position.

RecordDryRunBlocks writes one turn’s forecast examples, at most one batch, into the dry_run_blocks bucket the legacy writer also uses. A record keeps the legacy ip, reason (the registered check) and timeout_sec, and adds candidate_id, session_id, kind and expires_at; its key is a fixed-width UTC time with nanoseconds, the canonical address and the candidate, so two decisions for one target never overwrite each other. The writer validates every record and the encoded key and value bounds before writing; one bad record refuses the batch. A full bucket makes room by evicting earlier sessions’ first keys in key order in the same transaction. It refuses a batch when no room can be made without evicting its active session, or while the bucket remains above its cap. MaintainDryRunBlocks runs one bounded step of a pass over the bucket, trimming an oversized bucket without evicting the active session, removing records whose key time is strictly older than a cutoff, and, for a purge, every record of a session other than the one it keeps, legacy records included. A purge with no active session removes everything; a purge with an active session preserves that session regardless of time. It resumes after the last key it visited, including protected keys, writes nothing when nothing is due, and a failed step changes nothing.

The forecast model keeps charges in admission-time order, including when imported legacy spend is dated in the future. Earlier simulated charges can leave their window while that spend remains. Only block_ip consumes simulated ceiling credit and temporary address space; other kinds return not_modelled without changing capacity. These primitives do not activate owner execution or replace the comparison-only challenge path.

Schema 7 adds execution intents; the first open upgrades a schema 1 to 6 ledger in the same transaction. Execution marks a batch of up to 64 reserved attempts as running and records, in the same transaction and before any host change, what each member may apply: its kind, its target and its absolute expiry. Executing the same members again reads them back; a member that runs in another batch refuses the call. A batch mixing reserved and running members refuses as a transition conflict, regardless of member order or target overlap. An applied or failed outcome removes the member’s intent, while an unknown one keeps it, so a restart knows which effects are unproven. A reservation holds room in the recovery reserve for its intent, and an unknown outcome pins the intent with its history. The upgrade records an intent for any attempt that ran without an outcome or ended unknown and holds each pinned outcome’s intent in the reserve, and opening proves every intent against its attempt. Startup reads the unresolved intents directly, without opening the ledger; a ledger older than intents has none, and one it cannot read is an error. Ledger buckets without schema metadata are an error, including unfamiliar buckets. An unknown outcome also holds the slot of the row that will retire it. Once its absolute expiry has passed on a current, trusted clock, its outcome rows are delivered and the engine finds nothing of it remaining, recovery retires the uncertainty: the outcome stays unknown and counts as nothing new, nothing is dispatched again, the intent goes, the history and intent leave the recovery reserve, and an expired_unproven audit row records the event. Status counts the attempts that run and the unknown ones not yet retired, with the earliest expiry after which one can be retired. A missing or inconsistent intent marks only that section unreadable. Opening checks retired attempts against the stored clock even after their retirement events have been acknowledged.

Firewall startup restores temporary blocks only for their remaining lifetime. Timed elements use the same millisecond encoder in legacy and durable state; rows with less than a millisecond left or an unrepresentable lifetime are skipped. Blocks, operator allows and subnets use one startup clock reading. Durable updates compare absolute expiries before encoding timeouts, so millisecond rounding does not replace unchanged timed blocks. Kernel timeouts come from a lease clock that never runs backward: the later of the wall clock and its last reading advanced by the time elapsed since. Reapplying the rules after the wall clock is set back restores each timed block for the time it had left, never for the step on top. Batch checks and sends read the same clock. Fresh legacy blocks record their deadline on that clock too. Operator undo retains the original deadline and refreshes its remaining timeout after persistence and before each bounded retry. Fresh legacy sends also refresh after persistence. Ruleset installation ages precomputed block timeouts after table discovery and rule construction. Durable action planning, verification, sends, and whole-set retries use the same lease clock for timed blocks. Allow and subnet expiry still use wall time. The first Apply resumes the checkpoint before durable planning, so its evidence and the installed timeouts share one clock. After every ruleset it installs, the engine saves that clock with the boot it was read in and the time since boot (lease.json beside the state file). A timed block saves it before the legacy or durable state intent too, so a new deadline created during a forward clock step cannot gain time after correction and restart. If that save fails, the new block is refused before state or kernel mutation. A save failure after ruleset installation warns without undoing the installed rules; the next Apply retries it. Separate clock samples are ordered so sampling latency can only shorten restoration. A new engine resumes from it before restoring timed blocks: within the same boot at the checkpoint plus the time elapsed since it, whatever the wall clock did meanwhile; on a new boot at the checkpoint itself, crediting no downtime. Before the first Apply the daemon also moves the clock forward to the time the admission clock proves has passed: its saved high-water mark or, within the same boot, its checkpoint plus the time elapsed since. A valid but degraded admission clock still supplies that bound and logs the discrepancy; a damaged checkpoint or an unreadable ledger logs the missing coverage, and without either the engine starts from the wall clock. Forward steps can shorten restored leases. After a step back the lease clock stays ahead unless wall time catches up through a later correction; elapsed time alone does not close the lead. New rows record expiries that far ahead; the engine logs the first lead beyond the readback slack once per engine. The admission owner reads the engine’s lease clock once per batch and sets a first attempt’s expiry on it, so the lead is not lost from the block’s lifetime and a short block is not refused as already expired; a retry keeps the first expiry. The owner already defers while the admission clock is degraded, the only time the lease could trail it.

The engine also provides internal batch entry points for up to 64 timed full-address blocks. The admission owner runs them only when the daemon gives it an engine, and production gives none until cutover. Each member is checked against one state and kernel snapshot over bounded fresh sockets; unreadable state refuses the whole call. Expired rows remain available for kernel cleanup in both the intent and restored state. Rejected members consume no slot and cannot exclude a valid member. An address counts as already blocked only when a verified row and its kernel element agree: an untimed element for a permanent row, or one with the time a timed row has left, accounting for elapsed time across both set reads and all member checks. Apply results also age existing-block proofs through the batch’s writes. A timed proof that can already have expired is not coverage, even if its row has time left within the readback slack. The row must also last as long as the member, within the readback slack: a shorter block holds the address until it ends. Any other row or element holding the address refuses the member as pending recovery. An existing-effect refusal, at check or apply time, names the block that holds the address: the admission action that wrote its row, or eff_ and a digest of an operator or earlier row’s identity, with its expiry. The same row gets the same name on every check and after a restart. Set readback proves membership, not enforcement, so a batch, a member check and a state probe each read the input chain’s rule handles before the set snapshot, up to the last source-address drop of a filtered blocked set. A missing or altered family guard, source load, register flow, lookup or drop verdict refuses them. The check decodes the raw expression list, so an expression unknown to the nftables decoder cannot disappear from it; other rules contribute their handles without their expressions being understood. The batch reads the handles again just before its write and refuses when they changed, as after a ruleset reload or a rule inserted ahead of the drops. That refusal writes and sends nothing and holds execution until a probe finds the drops again, without an apply fault. The namespace generation is not read: any other nftables user changes it. The batch rechecks, writes one intent with expiry instants stored in UTC, refreshes timeouts after the write, and uses one atomic transaction on a fresh socket with a two-second deadline. Exact-family readback proves each applied member’s key and timed lifetime. Capacity is checked only at apply: a check before reservation never answers set_full. Victim checks require retained rows at the address to agree on action, expiry and verification. An unreadable startup ledger disqualifies all victims; members can still use free slots. The admission snapshot lists at most 256 imports, soonest expiry first, then name: timed, verified rows a legacy automatic block stamped, each alone at its kernel address with a proven element and named by its raw row’s eff_ name. Mapped addresses share the canonical address; a retained zoned alias also disqualifies an import. A lone zoned row stays unlisted and unclaimed. The imported occupancy count still includes unlisted stamped occupants, so a bounded victim list cannot free capacity. A member whose victim, the admission block or import the ledger claimed, still has that row, name, expiry and proven element takes the victim’s slot if the resulting set fits the current limit: the same transaction deletes the victim’s element, and only a readback of the victim’s set proves it removed; a member whose removal is unproven is unknown, even with its own add read back. A victim that no longer qualifies is spared and its member needs a free slot. A victim with no more than the batch’s minimum lifetime left at the send is spared and its member fails set_full alone, since its expiry in the kernel would reject the whole transaction. A victim removed by other means before the send rejects the whole transaction with ENOENT, which fails its members like any rejection. The victim’s row stays until the owner retires it after the ledger records the outcome, so a restart before then restores the victim. Retirement removes an evicted import’s row by its eff_ name only while the stamped row still matches it. An operator or unstamped row is never retired under an import name. Definitive kernel rejection restores state; uncertain exchanges retain unknown rows. Unknown rows are marked unverified and excluded from counts and seeding. Expiry retains them until the owner retires the matching attempt. A failed state write is returned as a state-integrity error and retains live suppression until a successful state write; later batches refuse while that error is held. The admission owner must reconcile action-linked rows before startup or any downgrade to a build that ignores their verification fields. The retry marker for a kernel rejection requires a successful state restore. Marking with no action IDs still writes the safe state to release a state hold. RemoveBlockActionRows removes, in one state write, the rows the ledger has settled (failed, retired after expiry, or evicted with proven removal). An error reports no completed removal, including a durability failure after rename. Startup calls it after marking every unproven row, with the settled subset store.ReadSettledBlockActions returns, so a start that stops between the two leaves the rows held back and the next start removes them: it asks the ledger about every saved action, including rows an earlier start held back and other actions sharing their addresses. An expired row no start marked is never restored and is left to expiry, so it holds back no other block at its address. The engine selects those rows on its lease clock, retaining that reading through Apply so a backward wall-clock step cannot restore an unchecked action. Marking preserves those separate rows until cleanup can remove only the settled actions. Removing a hold therefore cannot restore an unchecked action or erase another block. Until a restart proves the saved actions, the engine removes no block for a batch and its admission snapshot says so; the owner then plans no eviction or rotation, a dry run forecasts none, and higher work waits for a free slot. The engine’s kill-point tests stop a batch around its intent write, kernel batch, readback, unknown marking and rejected-state restoration, then start the ledger and a new engine over the same state, after a process restart or after a reboot that empties the kernel. Around an eviction they also stop before the outcome and before the victim’s row is retired: until the ledger records the eviction a restart restores the victim and holds the block back. After a committed eviction an admission victim’s row goes and only the block is restored. An import has no startup cleanup record: after a committed eviction, a restart before row retirement restores both blocks until the import’s original expiry, including after a reboot. An unreadable ledger restores unexpired unmarked rows with degraded coverage; a readable ledger keeps a committed applied outcome and holds the rest. Attempts without a committed outcome are unknown. Their rows are held back, a new block of an address a held row names waits, and once the expiry passes retirement removes the rows and the ledger retires the attempts. A new boot retains the saved lease floor. The test kernel reads emitted transactions, including set families, timeouts and base chains, instead of assuming a successful send installed the requested effects. It rejects incomplete batches and wrong set destinations before modeling loss. Mapped IPv6 keys never become IPv4 blocks. In a real kernel, a batch killed after the kernel took it leaves its element until a new engine’s Apply replaces the table without the marked row. An unconfirmed removal is written again on retry even if its rows already appear absent. Cleanup preserves other actions’ rows, including distinct actions sharing one address. RetireBlockActions checks the kernel for up to 64 unknown attempts under one engine lock: an attempt is clear when its address has no element, or the element matches a verified row of another action, remaining lifetime included, and its own row then goes in one write. An element still present keeps it held. Retirement requires a kernel snapshot of the target’s address family; a missing or disabled IPv6 set cannot prove absence, so it holds that attempt while the rest of the batch is answered. Cleanup resumes and saves the lease checkpoint first; an unreadable clock or checkpoint authorizes no removal. Address matching for recovery accepts legacy mapped, uppercase and zoned spellings within the engine’s namespace; this does not relax the admission parser. Unverified rows cannot suppress alerts or reputation checks through stale legacy tracker entries. They do not count as enforced blocks or select threat evidence for operator flush cleanup. Live deny limits count their kernel elements using kernel timeouts rather than the unresolved rows’ requested lifetimes, including replacement accounting. Permanent promotion requires a verified block row. A row an admission action wrote belongs to the owner: an automatic legacy block of its address ends as already blocked, even when the row is unproven or its element is gone, never evicts it at the temporary cap, and permanent promotion refuses it (ErrAdmissionOwnsBlock). Operator blocks, unblocks, allows, flushes and undo keep their authority over it. The guard also protects action rows already present in restored state. An executed admission block writes no legacy tracker (blocked_ips.json) entry. Reputation, alert suppression and its injected loader, flush cleanup, threat lookups and the blocked list read the verified firewall row instead, so it counts without one.

A read-only admission snapshot answers the same local checks for a bounded batch, each answer naming its member, with the temporary occupancy and limit from the same snapshot. No member takes a slot and no repeat is a duplicate there: the caller checks only the members it is ready to run, in order, for repeats and capacity. The snapshot also counts which of the bounded forecast addresses the sets already hold as a timed or an untimed element. A malformed or repeated forecast address refuses the call before any read, and other platforms report the snapshot unavailable.

With an engine, each owner turn runs its block_ip picks as one batch. The engine checks every member against one snapshot before anything is reserved. An address a row or element holds without proof defers a member; a block the engine proves already holds the address ends it linked to that block, and an invalid or protected address or a spent lifetime ends it, all without a charge. If the ledger refuses that link (no history budget for it), the pick stays queued with the deferral reason and the rest of the batch runs; the next check tries again. Each member carries the victim its reservation claimed, as its intent names it, and the ledger records what the batch did to that victim with the member’s outcome. Whole-batch failures, short replies and delayed outcome writes keep the victim and lease clock information. Once a verified eviction is recorded, the owner marks an admission victim’s row and retires it, as startup does, rotating failed or held rows behind waiting rows and retrying on later turns, including while waiting for an absent engine during dry run. Cleanup follows the retirement policy, respects state-repair probe waits and preserves independent apply faults. Its errors remain visible alongside unknown-attempt retirement errors; a restart removes the row before the firewall restores its state. The turn hands the ledger the imports the set read listed. Clearing that set also clears its imports, including a failed read, an idle turn or entry into dry run. Returning to previews reads a fresh set and import list. The dry-run forecast reads the list from its own snapshot and may replace a listed import, which it remembers by name until the import’s expiry; it never claims one. An import’s row is retired by its eff_ name without a mark. An import-only retirement still retries a shared state repair after its probe wait, without marking any import or clearing an independent apply fault, including during dry run. An import still enforced at retirement (a reload or restart restored its row) is forgotten. An error or short retirement answer keeps the request for retry; no ledger record names an evicted import, so a restart before its retirement restores it until its own expiry. After an unknown outcome the victim’s row stays until the victim expires. A proven block found only when the batch applies is checked again when the outcome is written. With a fresh proof the pick ends refused as an existing effect, linked to the block, keeping its charge. A member with a victim records its eviction outcome before attempting that link; a member without one records its failure and link in one write. Without a fresh proof, or if the link write fails, the attempt still records its failure. If the outcome cannot be saved, execution waits for the pending write; a crash before it is saved leaves an unknown attempt. After a successful write the ordinary bounded retry or the next check decides. On the last attempt the pick already ends failed. The rest are reserved, recorded as running with their intents, applied and finished by their own effect. Failed outcome writes retry before more work is selected and once at shutdown, even without a fresh clock reading; the effect is never dispatched again for that attempt. An outcome the ledger refuses for good (a damaged record or a conflicting state) is not retried: its attempt stays unresolved until recovery. Each turn, before scheduling, the owner retires unknown outcomes whose expiry has passed: it asks the engine, for up to 64 at a time, whether anything of them remains (RetireBlockActions); running attempts are never sent. The engine removes each clear attempt’s own row, and the ledger then retires it (RetireUnproven). An element still present, as after a forward wall step, keeps that attempt for a minute on its own wait, so a held batch cannot starve the intents behind it. Each checked batch advances through action order, wrapping around, even when its retries are due by the next turn. A degraded clock, a missing engine or an engine that cannot check retires nothing; the next pass waits one second, doubling to a minute. An attempt whose retirement the ledger refuses (undelivered outcomes, or a ledger clock that cannot yet prove the expiry) is asked again after one second, doubling to a minute. A damaged record or an attempt that ended another way is asked again a minute later; any other fault, such as a failed write, retries that attempt after one second, doubling to a minute. A missing or unreadable attempt uses the same wait without preventing other attempts from retiring. An unreadable intent list delays the whole pass. Each fault is reported once, in admission.owner.retirement_error and the doctor row admission retirement, until no attempt, list read or engine retirement check is left faulted; no retirement fault fails a turn. An owner without the firewall engine, or under a refused execution policy, retires nothing and keeps each unknown’s address held past its expiry: admission.owner.retirement_held_by names that cause (engine or policy) while unknown outcomes remain, and the doctor row admission unresolved outcomes names it with its fix instead of a retire time. Policy changes publish the retirement cause with the execution hold, preserving the cached ledger sample and its age. Once the cause is gone, each unknown still retires only after its expiry and the engine’s proof under a valid clock. A configured engine that has not started names no cause: retirement retries on its waits and the firewall reports its own startup error. A stop between the row removal and the ledger retirement leaves a row-less unknown that the next pass retires. The turn wakes for the earliest expiry or monotonic retry. With one noted, a turn lists no intents until it is due; with none noted, every turn lists them. A turn that fails after its retirement pass fails with its own step’s error. It never rebases a retry on admission time or adds time spent inside an engine call. Expiry conversion uses a monotonic sample taken before reading the admission clock, so a slow tick or later call cannot postpone another expiry. Running attempts create no retirement wake; retired attempts leave no retry entry. Every other kind is still previewed. A batch the kernel rejected fails every member, charged; each then retries alone, one per turn, until its next outcome or its expiry. A state write the engine could not make, a failed batch check or apply (including per-member transport and readback errors), or a missing engine holds automatic execution: block picks wait, deferred as an unavailable engine, while one probe runs per wait, from one second doubling to a minute. A held state write is probed by writing the safe state again and checking its snapshot. Without an engine, idle scheduling pauses and resumes on the first schedule check that finds an engine, without waiting for the old probe deadline. Other idle holds still probe on their waits; successful snapshots cannot reset a failing member check’s backoff. A failed apply keeps its cause and backoff through state repair, ledger refusals and member refusals until an applied effect proves recovery. Its probe is one member: every pick is checked and refusals end, but only the first accepted pick is reserved and applied, and the rest wait untouched for the batch after it applies. An isolated retry does not narrow these probe checks; it still runs alone if selected. An apply-confirmed firewall window holds the engine’s member checks, batches, probes and retirements (ErrConfirmPending) from before its snapshot until it is confirmed or rolled back, so its rollback cannot erase a block the owner wrote inside it; a batch in flight finishes first and lands in the snapshot. A rollback with saved block state restores it and then gates the rows on the ledger again, as startup does, before the hold ends: a mark or removal the ledger asked for inside the window is not lost with the restore. The restored ruleset also gives each timed block the time it had left when the window opened. When the hold ends, RetimeBlocks uses one strictly read row image and replaces the blocked sets in one transaction. Normal rollback restores configuration first; explicit confirmation accepts the current configuration. If startup with IPv6 disabled left no blocked-set reference, retiming binds the restored IPv6 set after a successful transaction. An unreadable image sends nothing, even with a cache or a successfully preserved corrupt state file. A retiming failure warns and releases the hold. A restart with a window pending holds the same checks and batches before the owner sees the engine, until startup recovery settles the window. A window opened between member checks and apply also holds execution, but is not an apply fault: after it ends, a successful snapshot probe clears that hold even on an idle turn. Any earlier apply fault still needs a successful one-member apply. The read-only admission snapshot of a forecast still answers. A kernel rollback alone does not release the hold when the saved engine configuration is unreadable; the confirmation remains pending until recovery succeeds or the operator confirms it. Keep its state snapshot through partial recovery, so another rollback restores the same state even after intervening legacy writes. Failed marker removal keeps the hold. Once the marker is gone, the hold and recovery fault clear even if snapshot cleanup fails; that error is still reported. Hold changes copy the cached owner status without rereading the ledger. Status (admission.owner.execution_held) and the csm doctor row admission execution name the cause. An owner publishes admission.owner.executing only while it runs block picks: it has Options.Blocks and its policy is not a dry run. admission.DoctorImpact then closes every admission row with ExecutionAffected instead of PreviewUnaffected and turns the outages that stop automatic blocks, held execution included, from warnings into failures. A degraded admission clock cannot bound a lease, so it defers block picks the same way. One batch starts a second at most; with an engine, a drain that queues new work runs the schedule at once instead of waiting for its timer. An owner with an engine also selects, checks or forecasts work once a second at most, including turns that dispatch nothing: work that falls due sooner, from a drain or a panel answer, stays due for the next second, while retirement of earlier attempts keeps its own waits. Before the next live batch may start, a turn selects nothing and leaves the next selection due. While execution is held, a turn before its probe is due selects nothing either, so held work keeps its place in the rotation; probe turns still select. A missing engine has no probe: its hold still selects once per wait, from one second doubling to a minute, including during a clock outage, so work that needs no engine is still previewed. A returning engine selects immediately even after a recent missing-engine preview; batch starts still keep their spacing, and an earlier apply fault still requires a one-member probe. Clock or engine outages still defer waiting blocks without spending fair turns or running early probes; work already deferred for capacity keeps its reason. Dry-run forecasts and previews without an engine keep their own cadence. The policy installed at the turn boundary decides whether the batch wait applies; changing between live execution and dry run does not spend a turn that cannot select work. During a batch wait, opt-outs end silently before any outage is reported. Outage deferrals revalidate waiting work first, so expired evidence and changed ownership keep their own terminal reasons.

The first time an owner executes, its live start runs once per ledger, in one transaction, before anything is drained or served under that policy: every drain and turn reads the clock first, and the reading runs the live start. It charges the legacy hour as the greater of its count and the charges the ledger already retains in that hour, since previews and legacy blocking count the same responses; earlier preview charges stay as handover charges with their lane, cost and release times, so a later forecast cannot refund them. Pending legacy entries stay legacy’s to retry and count nothing: a finding admission takes never enters that queue. Block work the preview period left queued, deferred or not, ends as disabled with no new attempt, charge or notice: legacy blocking answered it. Work still in the ingress drains as live work; the engine’s one row per address ends any whose block legacy applied as an existing effect. It ends the episode answers of earlier observe previews, so the next observation of a previewed address queues a new generation, or a new episode when no other candidate kept the old one; challenge previews and verified blocks keep their answers. It clears the notice records raised while admission previewed, so none goes out as a live alert or delays a key’s first live notice. The sender discards the preview delivery retry delay once it sees that committed start; later live delivery failures keep their own backoff. The start also makes the previews’ history eligible at once. Until it runs, an executing owner delivers no ledger notice; with execution enabled the sender reads the mode and the due notices together, and an acknowledgement taken before the live start never covers a record raised after it. It raises the schema to 11, which every-host schemas 13 and later already exceed, and records the live feature bit in the ledger’s meta bucket; every earlier build refuses that unknown key, so a binary rolled back across the live start does not open the ledger and keeps legacy-only blocking. A feature bit this build does not know is refused as a newer schema. It leaves blocked_ips.json in place. A legacy state file it cannot read keeps no saved credit, and admission.owner.import.error names the error. A failed live start fails the turn, reload or drain that ran it, and a failed tick or drain closes admission. A failed live-start tick reports its cause through tick_error and the stopped-admission notice until a later tick succeeds. The import error describes a committed handover: an aborted transaction does not replace it, and later startup retries do not erase it.

After the live start a preview spends nothing that live work lives on. The owner previews what it does not run: every kind but block_ip, and every pick of an owner without an engine. Such a preview ends as a dry run does, with no attempt, slot, ceiling charge or history, and it leaves its episode open, so the next observation is previewed again; the comparison still counts it as observed. A refused preview counts in the hourly comparison only after its terminal ending commits, in the hour of that ending. Deferrals and failed endings add no refusal count. A live pick the engine’s check or its reservation refuses is counted the same way; one an existing block answers is not a refusal, and a forecast’s picks keep the rows a dry run already writes. The schedule serves those kinds as uncharged picks, so they take no lane’s ceiling budget or history allowance from the blocks that run. A dry-run selection on a started ledger no longer waits for history a challenge preview does not spend, and an owner without an engine selects its previews as a dry run does, outside the lanes, and wakes for them at once.

An owner with an engine binds to one execution policy: the dry-run mode, the verdict callback and its timeout, the hourly ceiling and the response lifetimes, read together from one config publication. Every publication that changes auto_response or the mode gets a new epoch. The owner does not start without a valid policy; it names the refusal instead, even with the callback off, and mode: observe is refused. Each turn and reload installs a newer epoch before it selects work, and the turn uses that policy for every pick; a reload returns once the policy is installed, after any turn already running. An older epoch is ignored. Changed settings or missing required dependencies under an installed epoch hold execution, as does an invalid newer policy, with the refusal in execution_held, until a valid newer epoch arrives; the daemon reports a refusal at reload as a config_reload_error finding. While a refusal holds execution, a selected pick of any firewall kind is deferred as engine_unavailable with nothing previewed, reserved or charged, and the schedule then passes it over, as it passes over work waiting for the panel, until a valid policy is installed; a refused policy also suspends engine probes and repairs. The schedule checks for newer epochs even while idle, so a publication without a reload resumes held work and any probes suspended by dry-run. A challenge keeps its comparison preview. A comparison-only owner never reads the policy.

The policy also carries whether the configuration still allows each queued response: checks.AutomaticResponseAllowed, the switches the legacy funnels test before they act (auto_response.enabled and block_ips for every firewall kind, block_cpanel_logins for the checks that need it, and each derived entry’s own opt-in; challenges follow challenge routing). Under a valid policy, a pick it no longer allows ends refused as disabled before any panel ask, engine check, forecast, preview or reservation, live or dry run. The test reads the configuration live at each turn, not the epoch’s: challenge, incident, central and per-check settings can change without a new epoch, and a saved change applies from the next turn. A queued block of an address the live configuration now names as a trusted proxy, or of a known CDN or website-firewall edge that one of its roots may have seen relaying, also ends as disabled: the mint would refuse it now. Roots are read only for an edge address.

Disabled responses end together before capacity selection, including challenge previews. Spent ceiling or history budgets, retry backoff and another attempt’s recovery hold do not delay their refusal or generate capacity notices. The check visits only the bounded durable queue and leaves allowed work’s scheduling turns and capacity checks intact. Refusing queued work never releases an earlier attempt’s charge or recovery hold. Startup and ceiling reload also refuse disabled work before inventory or queue revalidation can report expired evidence as a withheld response.

Under a dry-run policy the owner forecasts instead of running. It selects work with SchedulePreview, whatever the live budgets, and ends each pick as a dry run with a prediction: no attempt, charge, history, audit row or mutating engine call. A forecast session starts at startup or on entry into dry-run from the committed ceiling and its retained charges (ForecastSeed), then lives in memory apart from it: the same committed clock meters it, a new limit applies without new credit, and live credit is never copied over it. The seed leaves out the charges of observe previews, which are comparison work, not responses: their units leave the window and their credit returns to its lane up to the bucket’s cap (Uncharge); legacy and genuine charges stay. Ceiling, charges and keyed attempts come from one read-only transaction. Each retained attempt’s charge must match its reservation time, lane and response cost; damaged records refuse the seed without changing committed state. While the owner forecasts, the ledger also stops counting a preview as the answer of its block line (PreviewsAnswer), so an address previewed before the session queues its next generation on a new observation and is forecast; a verified block, or its remembered answer, still answers until its expiry, and challenge lines keep following their previews. Leaving dry run turns previews back on. One read-only admission snapshot answers a turn’s block_ip picks, with the real temporary occupancy and the real blocks of forecast addresses. A refused target ends or waits as a live check has it. A real block that covers the address predicts existing_actual, unlinked; a forecast block that lasts as long, within the readback slack a real block is allowed, predicts existing_shadow and a shorter one pending_recovery, and neither is renewed. Other picks are checked against the modelled temporary capacity, the bound on forecast blocks and the modelled ceiling, and only capacity_admit, capacity_evict and capacity_rotate add a forecast block and a simulated charge. The model budgets slots with the ledger’s rule (SetOccupancy.Limits): with no free slot a pick may take the first strictly lower-tier forecast block or listed import in eviction order and predicts capacity_evict. Forecast blocks rank by their pick’s tier and next attempt; one without a tier is never replaced. Each records its pick’s scope and the admission time and session elapsed time of its decision. Once the general partition is full, the session shares each partition as a live ledger does: among the scopes of its forecast blocks the real sets do not hold, the scopes of the turn’s block picks and the requester’s, from session cursors that each such turn moves once before its first pick. A fresh pick may then rotate out a forecast block of exactly its tier by the same rule as the ledger, once that block’s hold passed on the admission clock and the session’s metered elapsed time, and predicts capacity_rotate. Fresh means the ledger’s episode row says so (AdmissionLedger.Fresh, which needs no live start) and no forecast block of the address left the model before that block’s expiry: a replaced or rotated forecast block leaves the model and answers its address until then (spec 5.2). Active forecast blocks and retained answers share the existing modelling bound; when it fills, an otherwise funded pick predicts forecast_limited. Answers expire only at their recorded boundary, never to free space for new predictions. A pick that finds no room while a lane’s partition holds a scoped block of exactly its tier at another address predicts equal_tier_wait; other full sets predict set_full. Real admission blocks count in occupancy but in no scope. A replaced import is remembered by name until its expiry and subtracted only while the engine still lists it. With the verdict callback on, a pick the model would judge is asked about first and waits for its answer with no forecast change; a usable allow ends it as verdict_allow before any modelled capacity is judged, and a covering forecast block, or a full modelled set with nothing lower to replace and nothing to rotate, ends it without a call. Other firewall kinds end as not_modelled, with no call; a challenge keeps its comparison preview. Without an engine or snapshot, or with a degraded clock, picks wait as engine_unavailable and the forecast does not change. A decision is installed only once its ending commits, and only then counted in the hourly comparison summaries as a dry_run decision with its prediction, a field apart from the refusal reason. The prediction names live in the admission package, which the finding stream tool reads too. A forecast neither probes nor repairs the engine: a hold from live work stays in execution_held until the next live turn. A live policy ends the session and never replays a dry-run ending; a later observation of the target is a new generation. A session whose model cannot be metered by a committed clock reading or a new limit ends too: the forecast error is model_error until the next session starts, and that session names model_error as its reset rather than a mode change.

Each committed capacity_admit, capacity_evict or capacity_rotate leaves one example in dry_run_blocks. A turn writes its examples in one batch after its endings, also when a later pick stops the turn; a failed write loses them without undoing the endings or the forecast, and nothing retries it. An owner that runs blocks keeps the bucket alone: one bounded maintenance step a second trims it to its cap and removes examples older than a week. Installing a live policy after a dry run, or at startup, asks for a purge of every example but the active session’s; the purge pass starts over whenever the active session changes, so a dry run that resumes keeps its own examples whatever their times, and the examples are stale until a purge pass completes. The daemon’s legacy prune and live purge leave the bucket to such an owner.

Such an owner reports its forecast as owner.dry_run in status: whether it forecasts, the session and why it started, the model’s coverage, real against predicted temporary occupancy from the turn’s snapshot, the forecast’s own ceiling, each prediction’s session and last-hour counts, lost examples and summaries, failed writes, a pending purge, and a fixed code for why forecasting last failed. A turn that succeeds clears the code; a new session clears its time too. Leaving dry run clears the forecast and keeps the last session’s identity; a dry run whose session has not started shows none. The loss counters cover the process. csm doctor renders it in three rows that warn and never fail.

The forecast’s committed predictions are also summarized in history, never sent to an alert channel: one auto_block Warning, “Dry-run capacity forecast: N responses passed modelled capacity checks”, with the other predictions’ counts, its interval, a session count and a bounded list of distinct examples in committed order, each naming its candidate, finding, target and check. The first summary is due a minute after the first prediction it counts, and each later one a minute after the previous attempt, a failed one included. One summary is pending at a time: the sender takes it whole, later predictions start the next one, and a failed delivery goes back in front of them as one summary, retried with the notice backoff and counted in the notice queue while it waits. A mode change never flushes or relabels a pending summary; Stop sends one its minute allows and counts any other as lost. Executable owners supply a cancellation-aware, history-only summary sink separately from legacy notice delivery. An error must mean the summary was not committed. Stop cancels an active delivery, settles its receipt, and bounds the final attempt outside the owner goroutine. Failed or cancelled summaries remain counted until delivered or reported lost. The status getter returns a deep copy of its cached snapshot.

With the policy’s verdict callback on, a live owner asks the panel about each block_ip pick before anything is reserved, as a dry run does before it forecasts one. One read-only admission snapshot answers the turn’s local checks without giving any pick a slot: a refused, protected or already enforced target ends or waits as a batch check would have it, all without a call. Room is the ledger’s: it schedules a pick only on a free slot or a victim it can claim. Any other pick is asked about, as CSM auto-block: <check> for its canonical address, and stays queued; the schedule passes it over until the answer arrives. Workers, outstanding jobs and cached answers are bounded; a full stage leaves picks queued and is never read as an answer. Each answer is keyed by candidate, attempt and policy epoch and expires from completion, including time awaiting receipt. Admission uses the configured signature and replay validation, including explicit signing exceptions, through the same client as legacy blocking. A valid allow ends the pick as one verdict_allow refusal with no attempt or charge, even with the ceiling spent: the schedule picks it as uncharged work. Block, an empty verdict or a failed exchange continues the local path, and the reservation consumes the answer. Local checks run again before an answer is used. A pick that stops being eligible has its call cancelled. A new epoch, a mode change included, cancels every call, drops every answer and starts new workers only after the old ones return; Stop cancels the calls and joins the workers. A cancellation is never an answer. The owner status carries a verdict object for an owner with an engine provider: stage gauges, process counters and the last failure as a fixed class, never response text. Its outstanding jobs are also the admission.verdicts queue row. Workers count completed calls before delivery; late or duplicate results cannot count another call or consume a different job.

The admission ledger exposes two row-recovery reads. ReadSettledBlockActions names failed attempts, retired unknown outcomes and verified blocks later evicted by a proven removal among the supplied action IDs. ReadUnprovenBlockActions includes every non-verified attempt and those evicted blocks. UnresolvedIntents gives the owner the same ordered intents as the handle-free startup reader. Callers must inspect each attempt’s outcome: an executing intent is unresolved but cannot be retired as unknown.

Attack event storage

Attack events live in attacks:events; attacks:events:ip stores empty values under <ip>/<TimeKey> keys. The writer chooses an unused primary key inside the write transaction because batch counters can repeat for the same timestamp. Primary rows, index entries and the event count are updated atomically, including count-cap pruning.

Address queries walk the index newest-first and resolve primary rows until the requested limit is met. Older index entries can still contain a full event copy; the reader falls back to that copy if the primary row is missing, malformed or belongs to another address. Both forms must match the requested address. The UTC time-key migration preserves index values verbatim, so readers must keep handling both forms until older entries age out.

Structured Logging (slog)

Legacy daemon call sites emit log lines via fmt.Fprintf(os.Stderr, "[%s] ...", ts()). The internal/log package provides a drop-in slog wrapper so operators can opt into JSON output for log-shipping pipelines (Loki, ELK, Datadog) without a big bang migration.

Operator controls

Two environment variables, read once at daemon startup:

VariableValuesDefaultEffect
CSM_LOG_FORMATtext, jsontextOutput handler
CSM_LOG_LEVELdebug, info, warn, errorinfoMinimum log level

Set via systemd drop-in:

# /etc/systemd/system/csm.service.d/logging.conf
[Service]
Environment="CSM_LOG_FORMAT=json"
Environment="CSM_LOG_LEVEL=info"

Then systemctl daemon-reload && systemctl restart csm.

Writing new logging code

import csmlog "github.com/pidginhost/csm/internal/log"

csmlog.Info("scan complete", "findings", len(f), "duration_ms", d.Milliseconds())
csmlog.Warn("log not found, will retry", "path", path, "retry_in", "60s")
csmlog.Error("alert dispatch failed", "err", err, "channel", "email")

Keys should be snake_case. Values should be machine-parseable (numbers, strings, booleans) – avoid formatted strings when you can pass the raw value. Pass paths and errors as they are: the text handler quotes and escapes any value with whitespace, a quote or a control character, so a value quoted beforehand is escaped twice.

Migrating legacy call sites

Migration is incremental and optional. The legacy format stays valid. Start with the hottest subsystems (alert dispatch, firewall operations, WAF handlers) where structured fields provide the most value, then work outward. Do not batch-convert – each subsystem should get a dedicated commit with before/after log samples in the PR description.

Keep the [TIMESTAMP] prefix of journalctl lines readable by humans: slog’s text handler uses time=... level=... msg=... which is also human-parseable, so journalctl viewers still work.

Worker processes

Code that parses attacker-controlled input and can loop, crash or exhaust memory runs in a child process under internal/workerproc. The daemon can kill a child; it cannot stop a stuck goroutine.

The launcher gives the child one Unix socket as both fd 0 and fd 1. A frame is a 4-byte big-endian length followed by a JSON envelope, and it may carry descriptors. The length is checked against a cap before anything is allocated. Passing a descriptor never calls Fd(), so the caller’s pipe keeps its nonblocking mode and deadlines. Request descriptors are duplicated before launch, and the caller may close its files once a call returns.

A request that misses its deadline kills and reaps the worker, and that worker is never used again. Any other round-trip failure does the same, and its error matches workerproc.ErrWorkerFailed with its text unchanged. Three failures in a row pause spawning for 60 seconds, then one trial request is let through. Each supervisor publishes a requests queue-health row.

workerproc.FailureReason classifies counted failures as timeout, protocol, crash, oom, or spawn, including through wrapped errors. Malformed frames and rejected descriptors are protocol failures; a broken socket is a crash unless the process reports an OOM kill. The daemon cannot read a unit worker’s kill from the unit’s own cgroup: by the time it sees the worker die, systemd has removed that cgroup and its counters. On cgroup v2 each supervisor uses its own slice, and the slice’s memory.events, which counts its children’s kills and outlives them, is compared with its value at start. Before that snapshot, startup waits for any previous unit’s cgroup to disappear, bounded by StartTimeout: a reaped systemd-run client alone does not prove its service has exited. Standalone CLI scans include their process ID in the prefix to keep their slices distinct. On v1 that counter is not hierarchical, so an eventfd on memory.oom_control reports the kill as it happens; OOM and removal notifications are counted together even when they arrive in separate reads, and a single notification with the directory gone remains undecided until the watch closes after the client’s exit. Diagnostic text and underlying errors remain unchanged. A failed start matches ErrSpawn, not ErrWorkerFailed.

Config.OnFailure receives each counted failure while the supervisor lock is held. The callback must return promptly and must not call back into the supervisor. Stats.Kills counts these failures by reason, including failed starts, and returns an independent snapshot. Cancellation, stopped or breaker refusals, and local descriptor exhaustion before the round trip do not reach the hook or these counters.

Config.OnJob receives one outcome for every request that reached the breaker: ok with the worker’s memory use after the job (0 when unknown), a failure reason, or refused when the breaker turned the request away. It runs under the same lock and rules as OnFailure. Cancellation, a stopped supervisor and local descriptor exhaustion are not reported. The daemon feeds it into the csm_parse_worker_* metrics.

Some errors are not worker failures and never count toward the breaker:

  • caller cancellation, including during launch; a canceled trial restarts the breaker cooldown;
  • the daemon running out of descriptors (EMFILE or ENFILE) while duplicating request descriptors or starting a child. The request counts as lost work, but the healthy worker stays. Outside load can push the daemon to its descriptor limit, and that must not switch a worker off;
  • PHP parser panics reported in a valid reply, which count as lost requests but do not replace the worker.

Config.BreakOnSpawnOnly narrows the breaker to failed starts. The mail parse workers set it: a refused extraction delivers mail unscanned under fail_mode: open, so crafted mail that kills workers must not open the breaker for unrelated messages. Each message then costs at most one spawn. Every successful start resets the failed-start count, even if that worker then fails on its input or the caller cancels the request.

A known limit: when the daemon is at its descriptor limit, the kernel drops descriptors a reply carries. That looks the same as a worker sending too many, so it counts as a worker failure, classified as a crash rather than a protocol violation.

Config.MaxJobs retires a worker after that many requests, and Process.Retire lets a launcher retire one after a request, for example over a memory threshold. Config.IdleTimeout stops a worker left unused; the idle countdown restarts only after a served request and never stops a worker in the middle of one. Retirement keeps the reply and is not a failure. Both settings default to zero; the PHP taint worker leaves them at those defaults.

workerproc.Handshake sends hello and checks the child’s protocol version before any real request. It honors cancellation and the context deadline, refuses reply descriptors and clears the socket deadline before returning.

workerproc.Strikes remembers inputs that made a worker fail. Two failures within 24 hours block an input, for at most 1,024 inputs; NewStrikesWith sets another limit and window, each failure within the window of the one before. The phishing check keys it by the zip’s SHA-256 and reports a blocked zip with the phishing_kit_archive_unscannable warning instead of retrying it on every scan. The spool watcher keys it by Exim message ID, and counts kills per message source with a limit of three in ten minutes.

Worker diagnostics must go to stderr; stdout carries the protocol socket. The PHP taint worker (csm phptaint-worker) runs through UnitLauncher as csm-phptaint-worker-<generation>, with the parse worker’s lockdown and a 512 MiB memory cap. The YARA workers run through it too; the section below describes them.

Parse worker units

The archive parse worker (csm parse-worker) runs through UnitLauncher: each worker is a transient systemd unit started with systemd-run --pipe, named csm-parse-worker-<slot>-<generation>, with the properties listed in service confinement. There is no fallback to running it directly. The daemon installs one lazy zip lister, using slot zip0, before scans start. It publishes archive_list.requests and is stopped during daemon shutdown. The worker retires after 1,000 requests or ten idle minutes. Each archive RPC has a 30-second timeout; unit startup has a separate ten-second timeout, and reaping can take another ten seconds.

After the hello exchange, before an archive request, the launcher checks that the worker’s main PID is in that unit’s own memory cgroup, reads back the RAM cap, caps swap, and reads current usage. On cgroup v1 it caps memory and swap together, or sets swappiness to zero where there is no swap accounting. On cgroup v2 it disables swap; without swap accounting it requires a host with no active swap. A missing cgroup, a cap that does not read back, unreadable counters, or usage already over the cap fail the start. Process RSS polling cannot replace this: it misses memfd pages and other processes in the unit.

A watchdog reads the cgroup counters every 500 ms and kills a worker over its cap, or whose usage cannot be read. It skips a tick when the daemon itself runs out of descriptors (EMFILE or ENFILE). Memory kills are identified by the watchdog or the unit’s cgroup OOM counter; a missing or unreadable counter alone is not evidence of an OOM kill.

A worker over three quarters of its cap after a request is retired. Units also retire at three quarters of their lifetime cap. The supervisor checks retirement before reusing an idle worker, as well as after a request, so expired workers get no new input. Kill closes the socket at once and asks systemd to kill the unit by name; it never signals a PID and never blocks. The daemon attempts to kill leftover csm-parse-worker-* units at startup, with a 30-second timeout. Cleanup failures are logged; lister initialization still proceeds. Disabling scheduled phishing checks does not disable cleanup or archive listings for requested account scans.

The worker gets an open zip descriptor, never a path, and reads it with positional reads, so the daemon’s file offset stays put. It returns entry names in a memfd it seals against writes, growth and shrinkage. The daemon treats every reply byte as hostile: it checks the seals, then the size, every record and the entry count, before the visitor sees one name. Records are charged against the request’s directory budget, never a size the worker reports. A malformed reply kills the worker and matches workerproc.ErrWorkerFailed. An archive the worker could not read reports itself incomplete, without counting as lost work. A worker can still leave names out; validating the reply does not verify the archive. Name classification stays in internal/checks.

workerproc.Pool holds a fixed number of supervisors, one per slot, and Call names the slot. The mail extractor (parseworker.Extractor, op extract_message) sizes its pool from email_av.scan_concurrency and names units csm-parse-worker-mail<slot>-<generation>. The worker writes each non-empty attachment to a memfd sealed against writes, growth and shrinkage. A reply carries at most 200 parts. Names, content types and archive names are cut to 255 bytes, the partial reason to 512, and From, Subject and each of at most 100 recipients to 998.

The mail extraction API accepts open spool header and body descriptors. Spool event descriptors are close-on-exec so worker launch commands cannot inherit queued messages. Its reply must be one UTF-8 JSON object with a parts field, which may be null for an empty result. Each part must have a readable, sealed descriptor of the claimed size. Individual sizes and their total must fit the request’s limits without integer overflow. Invalid replies discard all parts and kill the worker with reason protocol. The spool watcher extracts every message through it: each scan goroutine owns one worker slot, because parts are charged to the memory cgroup of the worker that wrote them. A failed extraction becomes a partial result naming the failure, never an error. Watcher replacement waits for the previous scan goroutines to finish before reusing their slots. On shutdown, queued spool events release their descriptors without starting extractions. The daemon waits for active scans and watcher restarts to finish before stopping or clearing the parse pools. If a panic unwinds the daemon before shutdown is signalled, cleanup skips that wait so the process can exit. Extractor lookup and removal share the watcher lifecycle lock because a replacement may still be starting.

Unit tests use temporary proc and cgroup trees, never the host’s. The kernel tests in internal/workerproc/unit_kernel_test.go run real units as root on the CI kernel runner.

YARA-X Worker Process

By default CSM never compiles or scans YARA rules in the root daemon. Two kinds of csm yara-worker process run through workerproc.UnitLauncher with the parse worker’s lockdown: no root, no network, no account files, a memory cap and a lifetime cap. Wiring lives in internal/daemon/yara_backend.go; the daemon side is yaraworker.Backend in internal/yaraworker/backend.go and the worker side yaraworker.Serve in internal/yaraworker/worker.go.

  • The compile unit, csm-yara-compile-worker-<generation>, runs for one compile: 1 GiB memory cap, files up to 256 MiB (its reply), five-minute unit lifetime. The daemon allows two minutes for the compile request. The daemon reads the rule files (trust check, disabled-rule stripping) and passes their text in one sealed memfd. The worker compiles with includes disabled, so rule text comes only from that memfd, and returns Rules.WriteTo output in a sealed memfd. The daemon copies it into its own sealed memfd and keeps it.
  • The scan unit, csm-yara-worker-<generation>, is a single worker: memory cap 512 MiB plus the larger of thresholds.full_scan_max_file_mb and, with mail scanning on, email_av.max_attachment_size, read from the live config at each launch; no file writes, 24 hour lifetime. Before the supervisor may send it a scan, the launcher loads the kept rules into it. A restart loads them again and never recompiles: 55 ms for the shipped rules, about 3.4 s for Forge full.
  • Both units run with MemoryDenyWriteExecute=no (UnitConfig.AllowExecutableMemory): wasmtime maps the compiled rules executable, and with the denial the load fails as “invalid YARA-X compiled rules file”. No RLIMIT_AS is set; the cgroup cap is the limit.
  • A reload compiles in a new compile unit. On success the new rules replace the kept ones, the scan worker on the old rules retires before its next request, and a background warm-up starts its replacement. On failure the current rules stay; with none loaded the failure is CompileError(). Compiles run one at a time; a canceled startup retry stops waiting even when a rule update is already compiling.
  • Forge downloads are compile-tested in the compile unit before they are installed. If the merged reload afterwards fails because the compile unit could not run, the new tier stays on disk, its version is not recorded, and a Critical yara_backend_unavailable finding says so; only rules that do not compile, or a collapsed rule count, roll the tier back.
  • At startup the daemon attempts to kill leftover units of both prefixes; cleanup failures are logged without preventing startup.

Failures and findings:

  • The backend is active from boot, before the first compile. While no rules are loaded because a compile failed or could not run, every scan returns an error, never a clean result.
  • The boot compile runs before the daemon reports ready to systemd, so it is bounded to 45 seconds. A compile that cannot start, crashes, times out or outlives that bound is retried in the background with backoff; when the boot attempt and three retries have failed, a Critical yara_backend_unavailable finding fires.
  • Rules that do not compile raise Critical yara_worker_compile_failed and are not retried. So does a rule set the compile unit cannot fit: more than 1024 rule files (refused before a unit starts), an out-of-memory kill under the 1 GiB cap, or compiled rules over 256 MiB. Each message names signatures.yara_worker_enabled: false, which compiles in the daemon.
  • Any counted scan worker failure (failed start, timeout, crash, out of memory, protocol) marks watcher: yara_worker failed and raises Critical yara_worker_crashed, at most once a minute after the last alert; an attached worker is killed. The next request starts a replacement; a served request marks the watcher attached again.
  • Only failed starts count toward the worker breaker (BreakOnSpawnOnly), so content that kills the worker costs one restart per scan and cannot switch YARA off for other files. The backend keeps workerproc.Strikes by SHA-256 of exactly the bytes sent for scanning. Content with two scan worker failures within a day is refused as a scan error without starting another worker. The check and failure recording run under the supervisor’s request lock (Supervisor.CallGuarded), so queued copies see the failures ahead of them. A failure is recorded against the rules the worker actually loaded, even if a reload lands while it starts. Failed starts, cancellation, breaker refusals and descriptor pressure do not strike content. Hashing runs only while an unexpired strike is recorded or to record a new failure. Records expire a day after the last failure and a successful rule reload clears them. The list holds at most 1024 content hashes, evicting the oldest failure when full.

signatures.yara_worker_enabled: false runs YARA-X inside the root daemon instead; csm validate warns about it.

Wire contract (workerproc frames on the unit’s socket pair):

OpRequest descriptorsReply
hello0protocol version
compile1: rule texts back to back, named and sized by sourcesrule_count and a sealed memfd of compiled rules, or compile_error and no descriptor
load1: compiled rulesrule_count
ping0rule_count of the loaded rules
scan_fd1: content, read positionally up to max_bytesmatches, always an array

Malformed requests get an error frame, which kills the worker. A rule compile error is a result, never a worker failure, and its text is cut to fit the frame. Scans need a regular descriptor and every requested byte; the worker gets its own read-only open of a file. Byte scans travel as sealed memfds, so the frame cap does not limit them. With no rules, or every rule disabled, byte scans answer clean without a worker unless compilation failed, and mail-part scans fail.

Testing: internal/yaraworker launches the test binary as the worker with a fake engine (helper_test.go), including fault modes for crashes, hung loads and bad compile replies. Tests tagged yara run the real engine, including backend recovery from invalid rules with disabled rules still excluded. Unit lockdown is verified on the CI kernel runner and on scratch hosts.

Building the Documentation

cd docs
mdbook build              # generates docs/book/
mdbook serve              # local preview at http://localhost:3000

Clean application corpus

The same job measures the shipped rules against long uninterrupted base64, hex and word runs, dense variable calls, and PDF-shaped streams. It discards one warm-up and scores the fastest of two further scans against a 5-second per-file budget. Each attempt has a 10-second engine timeout; a timeout never counts as a completed scan, and a cold timeout alone cannot fail the gate. Both scored attempts timing out fails it. The job shares the heavy-test resource group to avoid competing with other heavy tests in this project.

A rule with no literal atom to match on can still pass match tests while stalling mail delivery. Investigate slow rules with yr scan --profiling. Run the gate and its fixture/timing checks locally with YARA-X installed:

CGO_LDFLAGS="$(pkg-config --libs --static yara_x_capi)" go test -count=1 -v -tags yara ./internal/yara -run 'TestShippedRulesScanWithinBudget|TestRuleScanBudget'

Every pipeline runs the required clean-corpus gate in the production YARA-X builder image. Package publication and GitHub releases depend on its success.

The attack replay corpus gate runs the same way and gates the same jobs.

See cPanel release tests for the required image, upgrade baseline, release dependencies and retained evidence.

Production tag selection, execution artifacts, and the required isolated kernel runner are described in Production build and kernel tests.

make check-fixtures is also a blocking GitLab job and publication dependency. It checks all tracked and unignored files under testdata and fixtures, including files without extensions. Scanner failures stop the check; reports identify the file and line without printing the suspected address. The same run scans every tracked and unignored file for private names when CSM_PRIVATE_TERMS (or -terms) names a file holding one case-insensitive regular expression per line. That file stays outside the repository: the GitLab job receives it as a file-type CI variable and fails without it (-require-terms), while a local run without one prints a skip notice. Reports name the file and line, never the matched text. Filenames are checked too; a matching filename is reported with its path redacted. The fixture sanitisation rules describe the additional manual privacy review.

Attack replay payloads are decoded in memory before checking. Every data file in a sample directory is decoded regardless of its suffix; .b64 files are decoded elsewhere in the attack corpus test data too. Manifests, corpus-root notes and saved Go fuzz seeds are checked as plain text. Invalid base64 fails the check, and diagnostics never print decoded content.

Admission preview ledger

Admission previews reserve, charge history and capacity, and end observed in one transaction. They write both audit steps, never execute a response, never count as applied, and do not end the evidence episode or raise a notice. Repeated evidence is answered until its quiet window closes.

Imported legacy usage retains its accounting-window charge after an attempt is retired. Retirement of old attempt history cannot create new admission capacity before that window closes.

The scheduler records only ceiling, history-budget and recovery-reserve deferrals, once per changed reason. Fair turns, member bounds and retry backoff produce no deferral; a cleared budget reason produces no new gap notice. Challenges spend fair turns and history without spending block credit, and bounded durable holds preserve the earned turn of charged work while challenges proceed.

The owner schedules queued preview work on its timer and uses the next ledger wake even when no new arrival appears. Reloads recompute that wake under the current ceiling. A first attempt lasts the legacy-selected lifetime, or the configured response’s, never longer than auto_response.block_expiry (spec 5.6): an executable owner reads it from the policy of the epoch, an owner without an engine from the current configuration, and a policy without a positive one is refused. Retries keep that absolute expiry. The owner observes each pick transactionally and continues delivery retries independently of scheduling. A refused preview ends only its own queued work, so an invalid expiry cannot starve other picks; a storage failure retries the turn, twice at once and then after waits that double up to a minute, as a held probe is, and shows as admission.owner.turn_error and the doctor row admission turns rather than a stopped owner.

Held responses coalesce by evidence, kind and effective entry, whether the root entry is named explicitly or omitted. Derived entries validate their producer binding before fresh placement, coalescing and answered results, including direct queue requests. Retained queued work also revalidates its entry binding before scheduling or coalescing. The first selected temporary lifetime is durable and cannot be extended by a later report of the same candidate; absent metadata retains the configuration-selected lifetime.

Root preparation and selected submission use the cached admission view and do not wait for ledger writes. Missing observation is an attribution refusal; malformed input remains invalid. Unminted Critical responses refused while the ingress is stopped count in its stopped status. Minting accepts canonical IPv6 evidence, while the selected kind checks containment support: HTTP challenges do not depend on the firewall family setting.

Per-address funnels hand every selected block or challenge to admission before the existing responder runs. A process-local finding flag skips only the second evaluation of the same finding, including its queued block retry; unrelated pending work and fresh observations still run. Admission uses the canonical address selected by the legacy extractor, including its message fallback, without changing the finding or its audit identity. Distinct observations still submit even when their address is already challenged. While admission executes blocks, a block it accepts, or refuses because the client is behind a trusted proxy or is a CDN edge, skips the legacy block: no engine call, tracker row, queued retry or hourly charge. For a block it accepts, the scan also drops every other legacy block of that address in the same pass: its queued retry, findings admission could not take and mail spray findings. Every other mint or synchronous ingress refusal keeps the legacy block. Mail spray blocks, and derived blocks asked to last longer than auto_response.block_expiry, stay legacy: admission would cut them to it. The funnel reads that lease as the scan loop does, with a 24-hour fallback for an empty, unparsable or nonpositive value; the daemon passes a nonpositive value to the policy unchanged, the policy refuses it, and the owner then executes nothing, so legacy keeps every block. Acceptance is not application: a later hold, dry-run change or queue displacement never returns that request to legacy. A derived block admission takes ends before the engine as an expected refusal: timeout escalation and central intel log nothing, and the incident correlator counts it as requested. Previews and challenges never change a responder decision.

Subnet funnels submit every selected response before the existing responder evaluates its available execution budget. Constituent evidence follows the selected mail response; paths without eligible observations retain their designed refusal rather than minting a replacement root.

A challenge entry retains its routed root until timeout escalation hands the selected response to admission. Escalation never creates new root evidence, and permanent selections take the designed rootless policy refusal.

Central selections use the locally corroborating finding as their root and carry its original attribution if preparation refuses. They do not adopt a remote report as local evidence, and preserve the selected temporary lifetime.

Incident event preparation retains the attesting finding’s check and severity, its evidence or its preparation error until a block is actually selected. Roots use the canonical address the legacy block selects while retaining the original finding. Only that selection counts an admission decision; generic and spray paths preserve the original check and cause. A derived response never mints a second root.

Hourly summaries count committed arrivals, memory-coalesced reports, mint refusals, displaced submissions and both observed and terminally refused picks. The displacement callback runs under the ingress mutex on the submitting goroutine and uses the funnel’s clock; drain callbacks use the owner’s fresh clock. Summary time never regresses: event hours are clamped to the latest admission-clock or funnel hour after a backward wall step, including clock readings from an incomplete startup. Immutable write acknowledgments cannot consume newer rows. Summaries continue during ledger or clock outages, and shutdown flushes completed decisions even when a final drain fails. Failed writes retain a bounded chronological set of completed hours, so missing recordings remain explicit lower bounds.

Ingress placement on an ended or in-flight candidate is a counted invalid refusal. An episode that cannot be encoded is ledger damage. Shutdown closes ingress before returning new blocks to legacy, even while an owner turn is still running. Previously accepted work is drained for restart; new legacy-owned blocks cannot enter that queue. Incident background response loops stop before the owner. An owner that executes blocks ends, as disabled, the incident and spray blocks an earlier process queued: the correlator reads its selectors at start, so that work may no longer be selected.

Selected work also clears a stale budget reason without another count or notice. The scheduler preserves deferrals owned by other components. Wake times consider both the next charged turn and the history needed by challenges, while independent history holds retain their earned credit through a ceiling wait.

A batch readback requires every member to be running. Direct queue requests can reserve a retired action ID again. If another member of that batch still runs, execution refuses the mixed batch without writes.

Clean application corpus

The required test:clean-corpus job downloads the public vendor ZIP archives in scripts/clean-corpus/manifest.json. The manifest pins the version, official HTTPS source, SHA-256, license file, and exact regular-file count. Archives are cached under their checksum, so editing metadata or adding a source downloads only new pins. Each archive is verified before extraction into a new private directory. Corrupt cache entries are replaced with a verified download; unverified downloads never enter the reusable cache. Failed downloads, missing licenses, empty or incomplete inventories, unsafe archive entries, and read failures stop the job. No downloaded PHP or JavaScript is executed.

CSM_CORPUS_ARCHIVE_CACHE selects the cache directory. The tool reserves its .csm-corpus-v1 subdirectory for checksum-named archives, temporary downloads, and a persistent lock. Do not put other files in that subdirectory. Cleanup only touches generated archive and temporary names there, never ZIPs elsewhere in the configured directory or symlink targets. A symlinked managed directory is refused. Concurrent preparations sharing a cache fail with a busy error; retry after the active run finishes.

Valid flat cache entries from older versions are verified and copied into the managed cache without downloading. Their originals are left untouched because names alone cannot establish ownership; operators may remove those old inputs once no older tool uses them. Managed archives no longer pinned and interrupted downloads are pruned before fetching, even if preparation later fails. GitLab saves verified downloads and cleanup on failed jobs too, so retries retain progress without accumulating retired pins.

Every source names the CMS it belongs to in a cms field, and the manifest lists each supported CMS that has no pinned source yet under pending, with the roadmap item that blocks it. The manifest is version 2; the validator refuses older versions, a source without a CMS, a supported CMS that is neither sourced nor pending, and one that is both, so a database scanner for a new CMS cannot ship without a corpus decision. A pending entry records absent evidence only: no scanner, admission rule, hit budget or result filter reads it, and files from that CMS are treated exactly as before on a server. When a source for a pending CMS lands, its pending entry is removed in the same change. The manifest, not this page, is the record of which CMS has a false-positive gate.

Ordinary unit tests validate the manifest offline, including with go test -trimpath. Invalid metadata is rejected before cache, extraction or report files are created or changed. Command tests check the full archived manifest and every inventory record; these checks do not replace the detector gate.

The initial corpus contains 10,178 files: WordPress 6.8.2 (GPL-2.0-or-later), WooCommerce 9.9.5 (GPL-3.0-or-later), and Elementor 3.29.2 (GPL-3.0-only). The archives retain their license notices, including notices for bundled components. These are fixed test versions, not installation recommendations. Source URLs and license file locations are in the manifest.

Run the same gate locally with the Go toolchain from go.mod and the matching YARA-X C library installed:

bash scripts/clean-corpus-test.sh

The script compares the installed C library version with both the Go binding and production builder pins. It runs YARA, YAML signatures, PHP taint, and JavaScript taint gates. Ordinary unit tests may omit external corpus inputs; the required job sets CSM_CORPUS_REQUIRED=1, so missing input is fatal. Package publication and GitHub releases explicitly depend on the job.

corpus-results/ is retained for one year, including on failures:

  • manifest.json identifies the authenticated archives and licenses.
  • inventory.json lists every extracted file, its size and SHA-256.
  • engines.txt records the commit, toolchain, YARA-X and parser versions.
  • Engine JSON reports include counts, per-rule hits including zeroes, thresholds, and taint status counts. tests.jsonl records execution.

YARA and both taint engines require zero findings. YAML retains the existing reviewed false-positive budgets in internal/signatures/corpus_fp_gate_test.go; new rules default to zero. The pinned corpus currently produces 10 YAML hits across five rules, all within those existing budgets. This is a regression gate, not a claim that every engine has zero false positives. A second YAML check fails if any regex would be skipped on a corpus file it matches.

The signature engines apply the production default 16 MiB bound and archive admission policy; 10,157 inputs reach each engine. PHP taint reads 10,171 inputs within its source limit, including 163 that complete analysis. JavaScript admission receives all 10,178 files and completes analysis on 289; seven are oversized and twelve have parse errors. These JavaScript gaps have explicit ceilings in the required gate; canceled, resource-limited and panic outcomes have a zero ceiling. Neither oversized nor unparsed inputs count as proof of successful analysis. The required taint gates each need at least 100 completed analyses as well as their corpus inventory checks.

To update the corpus, download a versioned archive from its official source, review its license and file inventory, and commit the new manifest pin with the measurements. Investigate new matches and narrow detectors using content semantics. Do not add scan-path exclusions or raise budgets to pass a run. The gate also compiles deliberately overbroad YARA and YAML rules and verifies that their clean-file matches are rejected.

Attack replay corpus

The required test:attack-corpus job plants known attack samples and near-miss controls in a temporary cPanel account and runs them through the three production paths that read files: realtime analysis, the scheduled deep tier and the operator full account scan. Each path applies its own production responders. The job fails when a path stops catching a sample, catches it at a lower severity, flags or changes a control, remediates differently, or starts catching a recorded gap. Package publication and GitHub releases depend on it.

Unit tests prove that a detector works on the input it was written for. The corpus proves that the whole path still works: admission, path scoping, severity, quarantine eligibility and the responders. A change that breaks a detector end to end fails here even when every unit test passes.

The corpus measures file content only. Log and event replay and expected incidents are later stages.

The three paths

Every sample runs alone, with its own account tree, cPanel domain map, state store, quarantine directory and monitor. The platform is pinned to cPanel, the layout the samples use. The config is the shipped configs/csm.yaml template with auto-response and file quarantine turned on and observe mode off, and nothing else changed. Both signature engines load the shipped rules from configs/. A partial rule load, an empty rule set or a scan error fails the gate.

  • Realtime. The driver builds the file monitor without fanotify, opens each planted file and derives its event path the way the event loop does. The file then goes through the production admission filter and, when admitted, through analyzeFile. A file the filter drops counts as not detected. The driver drains the dropper tracker and the staged package queue at once instead of waiting for their timers. Remediation is the inline quarantine that analyzeFile performs.
  • Deep tier. The driver runs the file checks of the deep tier through the production runner, runParallelWithContext, so the runner’s responders act on the findings as they do in production. Every deep check is classified as a file check or a host check (network, database, mail, packages). Host checks do not run. A test fails when a new deep check has no classification, so it cannot stay out of the gate unnoticed. Before planting, the driver runs the tier once on the empty account tree with the same state store. Findings that name no planted file and also appeared in that empty run are dropped as host noise. No check or path is allowlisted.
  • Full scan. The driver runs the account scan with the options that csm scan <account> --full --quarantine uses, then the scan job’s quarantine loop. That loop ignores the auto-response settings, as it does in production.

The tier and the full scan run PHP taint analysis in-process. Production runs it in a separate worker, which has its own tests.

Offline

The run never reaches the network. Requests through the default HTTP transport are refused. On the two scan paths, database queries get an empty, successful answer for a synthetic site, and host commands fail as a missing program would. The realtime WordPress checksum fetcher is stopped. Refused requests, database queries and wp runs are counted per sample and appear in the report as network_attempts. A sample whose verdict depends on such a lookup says so in its description.

Sample format

Samples live in internal/attackcorpus/testdata/corpus/<category>/<name>/. Each directory holds a sample.json and the base64 content files it names. Content is base64 at rest and decoded only in memory or under a test’s temporary directory, because endpoint antivirus deletes decoded web shells from a checkout.

sample.json fields:

  • name: unique across both halves of the corpus, public and private.
  • category: one of webshell, dropper, malicious_plugin, credential_stealer, injected_js, phishing_kit, spam_script, persistence, binary, archive.
  • kind: malicious or control.
  • source: synthetic, fixture or real.
  • description: one line on what the sample is. It also names any policy, shortfall or offline lookup that explains its expectations.
  • provenance: where the content came from. For a fixture, the test it was taken from and how the planted shape differs; for a synthetic sample, written for the corpus.
  • files: each with path, account-relative (public_html/...) or @crontab for the account’s crontab; exactly one of b64 (a file beside the manifest, ending in .b64) or selftest (a sample from the self-test bundle); and an optional mode. JSON has no octal, so 0755 is written 493.
  • expect: one entry for each of realtime, tier and fullscan:
    • findings: each with check, file (a planted path) and min_severity (warning, high or critical). Every listed finding must arrive at that severity or above. Extra findings are reported, not failed.
    • gap: why this path misses the sample. Exclusive with findings.
    • remediation: per planted file, quarantined, cleaned or untouched. A file left out is expected untouched. cleaned needs the SHA-256 of the cleaned bytes in cleaned_sha256. The deep tier and the full scan clean a WordPress core, plugin or theme file in place, instead of moving it, when a Critical quarantine-eligible finding names it. A plugin or theme file with a known web shell name is quarantined instead, and so is a file that cleaning would leave empty with no page output, or that its core or plugin release does not ship. Whole-file malware expects quarantined; an injected site file expects cleaned, or untouched when the cleaner refuses the removal.

A missing file counts as quarantined only when a quarantine entry names its path, or a parent directory’s, and the retained copy matches the original bytes. A damaged entry is an error. A file that disappears with no entry is reported as removed, which no manifest may expect. A file whose bytes changed is cleaned.

All files of a sample are planted before the first one is replayed. Manifest order is not write order, so a sample whose verdict depends on the order of writes is judged on the final tree.

Every self-test bundle sample is referenced exactly once, so the bundle and the corpus cannot drift apart.

Adding a sample

  1. Write or decode content only in a temporary directory that your antivirus excludes, never in the checkout. Commit only the .b64 file.
  2. Add files by explicit path. Never run git add -A or git add .: an antivirus that deletes a decoded file leaves a deletion for the next commit to pick up.
  3. Public samples follow the public repository rules: documentation IP addresses only (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 2001:db8::/32), example.com, example.net or example.org hosts, and no real names, accounts, domains, mailboxes or credentials. This applies to names, descriptions and provenance as well as content.
  4. make check-fixtures decodes every corpus data file and scans it for addresses and private names. It decodes one layer only. A payload nested inside the sample, such as base64 inside PHP, a compressed blob, an archive member or a String.fromCharCode list, needs a manual check of the same rules before commit.
  5. Write the expectations from what the sample is (see below), then run the gate. TestPublicCorpusLoads in the default test job loads the public corpus and checks category coverage, so a malformed manifest fails there first.

Expectation rules

Judge each expectation from what the sample does. A run may draft the findings, but every expected finding and severity must match what the attack deserves, not what the detector happens to report today, except a recorded severity floor (below).

  • Gaps. A malicious sample that a path misses records a gap with the specific reason: which check misses which shape. A gap is a measurement, not an exemption. When a detector starts catching a recorded gap, the gate fails with gap closed: until the manifest replaces the gap with the finding. Only a finding on a planted file, or one that names no file but quotes an admitted planted file in its text, closes a gap; a finding on a directory or other path is reported as an extra finding. Files rejected by admission cannot close a gap. Closing a gap is a corpus edit in the same change as the detector fix.
  • Never weaken an expectation. Do not delete a sample, lower a min_severity, turn a finding into a gap, or change an expected remediation to make the gate pass. A regression is fixed in the detector or the responder. Never allowlist a path or name to make a sample pass.
  • Explicit policy. Where production deliberately leaves a detected file in place, because the check is not on an explicit quarantine list or the responder is off in the shipped config, the path expects untouched and the description names the policy. When the policy changes, the expectation becomes quarantined in the same change.
  • Severity floors. When a path detects a sample below the severity the attack deserves, min_severity records the observed level and the description names the shortfall. The gate then stops the severity from dropping further, but does not raise it.
  • Remediation shortfalls. When a detector heuristic, not an explicit policy, leaves a file in place that should move, the expectation records the observed remediation and the description names the shortfall.

Controls

Every category needs at least one malicious sample and one control, or the run fails. A control is the legitimate file a detector for that category is most likely to confuse: a cPanel MultiPHP handler in .htaccess, an inline base64 image, a vendored phar, a capability-checked upload handler. A control expects no findings and untouched on every path, and cannot carry a gap.

Any finding on a control’s file fails the gate, including a file the realtime admission filter dropped. A finding that names no planted file but quotes a control’s path is a false positive. Any other unattributed finding on a control is an error.

Failures

Each failure in the report and the test log starts with its kind:

PrefixMeaning
missed:An expected finding did not arrive.
severity:An expected finding arrived below its min_severity.
false positive:A control drew a finding.
remediation:A planted file ended quarantined, cleaned, untouched or removed other than expected.
gap closed:A path caught a sample recorded as a gap on it.
error:The driver did not complete for this sample, or a control drew a finding that names no planted file.

A run also fails before its first sample when a corpus cannot load, a required private half is not set, a private sample reuses a public sample’s name, a category lacks a malicious sample or a control, a self-test sample is not referenced exactly once, or an engine did not load. A test filter that excludes a sample, a driver that stops early, a check that panics, times out or reports partial coverage, and a failed quarantine all fail the sample.

Driver errors appear in the test log by type only, unless CSM_ATTACK_CORPUS_SHOW_ERRORS=1 is set (see below). Reports carry check names, severities, planted paths and verdicts, never sample content or finding text, because either can quote decoded bytes.

Measurement limits

What the corpus does not measure yet:

  • No driver replays program execution, so YARA scans of programs started from a document root (docroot_binary_exec) are not measured.
  • No driver replays the critical tier, so the crontab check is not measured on the scheduled path.
  • The daemon’s batch responders (permission fix, database malware response, companion sweep queueing, WP-Cron fix) are not driven. Realtime remediation is the inline quarantine only, and the tier stops at the runner’s responders. In production, realtime findings also reach the batch process kill and file quarantine. Verdicts agree today because the only realtime check on the batch quarantine list, signature_match_realtime, passes the same high-confidence gate there as inline. A realtime check added to that list would change production remediation without the gate seeing it.
  • supply_chain cannot fire: a fresh install has no advisory file. group_writable_php cannot fire: planted files belong to the test user’s group, not a web server group.
  • cleaned is not exercised: the .htaccess cleaner is off by default, the full scan skips .htaccess and crontab findings, the crontab fix is operator-only, and no sample draws a Critical quarantine-eligible finding on a file both scan paths clean in place (see remediation above).
  • The throttled performance file checks run once per sample, in the second tier run only. A tier run that skips any check fails the sample.
  • The full scan quarantine step is a copy of the scan job’s loop. A change to that loop must be mirrored in the driver.
  • Realtime does not admit .js or .phar files, so those controls are not exercised on that path.

Private half

Real attack samples, scrubbed of customer data, live in a private GitLab project in the same format, with <category>/<name>/ at the project root. The project comes from the CSM_ATTACK_CORPUS_PROJECT CI variable, and it must allow this project’s CI job token to read it.

This repository pins the private commit in internal/attackcorpus/private.commit: one line holding the full 40-character lowercase commit hash. Moving the pin is a reviewed public commit.

In CI the script does the following before any test runs. Any failure stops the job.

  1. It fails when CSM_ATTACK_CORPUS_PROJECT is unset or the pin file is missing or malformed. The pin accepts one full lowercase hash, with or without a single final newline, before any clone or checkout runs.
  2. It clones the project with the CI job token into a new temporary directory, open only to its owner and outside the checkout and its artifacts. A credential helper reads the token from the environment, so the token never appears in a command line, a URL or the clone’s git config, and no configured helper can store it. The helper serves only the exact project on the CI server over HTTPS, and Git refuses redirects. A temporary directory inside the checkout or reports is rejected. The temporary clone is removed on success or failure, including a failed clone.
  3. It checks out the pinned commit and fails unless HEAD is exactly that commit, so a short hash or a ref name fails.
  4. It runs scripts/fixturecheck -corpus -require-terms over the checkout. Every file is scanned for IPv4 addresses outside the documentation ranges and for the private terms in the CSM_PRIVATE_TERMS file variable. Sample data is decoded first, one layer deep, as in the public check.
  5. It sets CSM_ATTACK_CORPUS_PRIVATE to the checkout and CSM_ATTACK_CORPUS_PRIVATE_REQUIRED=1, so the tests fail rather than run without the private half.

engines.txt records the pin as private corpus: <commit>.

Outside CI nothing is cloned and the private half is not required. A local run uses the public half alone unless CSM_ATTACK_CORPUS_PRIVATE names a checkout. engines.txt then says private corpus: local, unpinned, and otherwise private corpus: none.

The coverage rules apply to the public half alone. The private half must load without validation errors and hold at least one sample. Sample names are unique across both halves. Reports and test logs print sample names and planted paths, so private samples follow the public naming rules for both: no real names, accounts, domains or mailboxes.

Running it

Run the gate in the release builder image, the one CSM_BUILDER_IMAGE and CSM_BUILDER_TAG name in .gitlab-ci.yml:

GO_LINUX_RUNTIME=docker GO_LINUX_IMAGE=<release-builder-image> \
  scripts/go-linux.sh bash scripts/attack-corpus-test.sh

The script checks that the YARA-X C library matches the Go module and the active builder pin exactly, then runs every TestAttackCorpus test in the harness, daemon and checks packages with -tags yara. It writes to attack-corpus-results/, which the job keeps for one year, including on failure:

  • realtime.json, tier.json, fullscan.json: per sample results and a per-category summary (detected, gaps, missed, false positives, closed gaps, remediation mismatches, errors). The script fails if any is empty.
  • engines.txt: Go version, YARA-X version, commit, the private corpus line (see Private half) and run time.
  • tests.jsonl: the go test -json event stream.

The output directory and its report files are open only to their owner, including when the script reuses an existing directory. The script does not change the umask, so under the usual 022 the drivers create files with the modes the daemon gives them. CI artifact access still follows the project’s artifact permissions.

Environment:

  • CSM_ATTACK_CORPUS_REPORT_DIR requests a run and names the report directory. Without it the corpus tests skip.
  • CSM_ATTACK_CORPUS_REQUIRED=1 turns that skip into a failure. The script sets both.
  • CSM_ATTACK_CORPUS_PRIVATE adds the private half, a corpus checkout in the same format. It must load and hold at least one sample.
  • CSM_ATTACK_CORPUS_PRIVATE_REQUIRED=1 fails the run when CSM_ATTACK_CORPUS_PRIVATE or CSM_ATTACK_CORPUS_REPORT_DIR is not set, instead of skipping. The script sets it in CI.
  • CSM_ATTACK_CORPUS_SHOW_ERRORS=1 logs the full reason of a driver error instead of its type. Local runs only: a reason can quote decoded sample bytes, and CI keeps test logs as artifacts. The report script unsets it, including when inherited from the caller.

The other TestAttackCorpus tests check the drivers themselves, on their own samples, and need no report directory.

Recorded finding streams

Calibrating correlation (the coordinated-attack threshold, corroboration grading, sequence rules) needs what a real host produced over days: every finding with its check, severity, timestamp, owner and text, including the false-positive floods and the long-lived rows that never clear. CSM already writes that stream: every dispatched finding lands in /var/log/csm/audit.jsonl (rotated copies are gzip files beside it). A recorded stream is a copy of those files with every identity removed.

scripts/finding-stream turns the raw files into an anonymized stream:

# On the operator machine, after copying the files read-only from the host:
go run ./scripts/finding-stream anonymize \
    --salt-file ~/.local/share/csm/finding-streams/salt \
    --out ~/.local/share/csm/finding-streams/host-a.jsonl.gz \
    raw/audit.jsonl raw/audit.jsonl-*.gz

What the tool replaces, in every structured field and in free text:

  • Host names become host-<id>, account names acct-<id>, domains dom-<id>.example, mailboxes user-<id>@dom-<id>.example (a mailbox truncated after the @ keeps the same user-<id>). The <id> is derived from an HMAC of the value under a private salt, so the same salt maps one account to one pseudonym on every host and streams can be joined without knowing who is who. Names are replaced wherever they sit: in /home/<account>/ paths, Account: lines, process context, LiteSpeed vhost tokens, and inside longer tokens such as example.com-ssl_log or web1.log. A name holding an underscore or starting with a dot, such as a bare mail login, is replaced as a whole wherever it is not glued to a letter or digit, ignoring case. Where identities overlap, the leftmost complete one wins, and a full mailbox or home path keeps its usual mapping. A replacement is never rewritten again. A learned name made only of punctuation is not replaced, so it cannot turn separators into identities; the final check refuses the run if one appears. The host’s short name (its first label, when it carries a digit) maps to the same pseudonym as the full name. Any other token that looks like a domain (two or more labels, alphabetic last label that is not a file extension) is mapped too, even if no field named it; that over-reaches on a few dotted names like options.option and is accepted. Domain-shaped names inside filenames are replaced even when no structured field named them. Only pseudonyms actually emitted by this run are exempt; a raw name beginning with host- or acct- is still scrubbed.
  • IPv4 addresses map into 198.18.0.0/15 and IPv6 addresses into 2001:db8::/32, both reserved and never routed; a raw value always maps to the same pseudonym. The IPv4 range holds 131,072 addresses, so distinct addresses can share a pseudonym. A manifest counts distinct addresses and distinct pseudonyms per family, and a replay cannot tell merged addresses apart. Loopback addresses, system users such as root or nobody, and bare numeric uids are kept: they identify nobody and carry meaning. Equivalent IPv6 spellings share one pseudonym across all streams; IPv4 addresses written in IPv6 form use the IPv4 pseudonym. Addresses inside filenames and before numeric rotation suffixes are also replaced. This can replace address-shaped version numbers; privacy takes priority over preserving ambiguous numeric text. New audit rows can carry a cidrs list. Those addresses use the same pseudonyms as addresses in text, keep their prefix lengths, and participate in the output leak check. Older audit rows remain readable without the list.
  • The details of a credential-leak finding are dropped entirely, and generic password=, secret: and token= material is blanked anywhere, including quoted keys and values containing spaces.
  • Finding ids become fid-<id>, a salted id of at least 128 bits that keeps case, so an action row can name its finding without exposing the raw id.
  • Timestamps, check names, severities, path structure below the account, plugin and file names, and process names are kept unless they contain an identity: they are what calibration reads. Paths and mailboxes in process command lines and parent processes also teach the scrubber which identities to remove elsewhere.

Before writing, the tool scans its own output for every identity it learned from structured fields, paths and mail addresses, for domain-shaped names, for raw ids (including inside longer tokens), and for any mailbox or address outside the reserved ranges. It refuses to write if it finds one. The raw-id scan includes check names and severity labels; these fields otherwise keep their original vocabulary. Structured finding ids must be ids this run emitted. A name glued to underscores or file extensions is still found. The summary prints row counts, the number of distinct checks, replacement counts, the time span and a salt fingerprint, never identities, check names, paths or the salt itself. An error names only the stream, the file’s position on the command line, the line number and a fixed reason.

Every row must parse as exactly one JSON object of the known schema, with a supported version and a timestamp field. Rows that older CSM versions wrote with a zero time are kept, counted in the manifest and left out of time spans. Unknown or repeated fields, nulls, data after the object and values over the size limits refuse the whole run. Malformed Unicode, non-JSON whitespace and timestamps that cannot be written back as JSON also refuse the run before a salt is created. Typed rows accept only the exact pseudonym spellings emitted during that run.

Handling rules:

  • Copy the audit files read-only (tar over ssh, or scp) into a local directory with mode 0700, run the tool, then delete the raw copies. Nothing runs on the monitored host.
  • The salt file is created exclusively on first use with mode 0600. Existing salts must be regular files without group or other access; symlinks are refused. A concurrent run that reads an unfinished salt fails without replacing it; retry after the creator finishes. Keep it private and reuse it for every host whose stream should be joinable with the others; losing it makes new recordings unjoinable with old ones.
  • Outputs cannot replace an input, the salt, the input manifest or each other, whether named by path, through a symlinked directory or as a hard link, and an existing output must be a regular file. These checks run before the salt is created, including for missing directories and paths containing ... Case-only and Unicode-equivalent path variants are refused on every platform, as are paths that would need a file to also serve as a directory. Every output is written to a private temporary file beside its destination, and all are complete before any is renamed into place; if publishing fails partway, the outputs already replaced are restored. New directories have mode 0700 and outputs mode 0600, even when replacing a less restricted file.
  • Filesystem cleanup failures are reported without claiming success. If rollback fails, remaining recovery copies are retained. If every output was published but backup cleanup fails, the error explicitly says the outputs were published; check the manifest and remove the leftover backups before sharing the directory. Other cleanup errors report unchanged outputs with temporary files left to remove.
  • Recorded streams stay outside the repository entirely, in a private directory such as ~/.local/share/csm/finding-streams/ with mode 0700. A pseudonymized stream still describes real incidents on a real host, and this repository is public. Do not keep them in an ignored directory inside the checkout: git clean -fdx removes ignored files too, and a recording that took a host weeks to accumulate is not reproducible from anywhere else.

Joining actions and firewall entries

The action log (actions.jsonl) and the firewall audit log (<state>/firewall/audit.jsonl) say what CSM did about the findings. The same run can anonymize both next to the findings, so the three streams share pseudonyms. A joined run needs a manifest, and a manifest needs a build of a known commit without local changes, so build the tool from a clean checkout first:

go build -o /tmp/finding-stream ./scripts/finding-stream
/tmp/finding-stream anonymize \
    --salt-file ~/.local/share/csm/finding-streams/salt \
    --out host-a/findings.jsonl.gz \
    --actions raw/actions.jsonl --actions-out host-a/actions.jsonl.gz \
    --firewall-audit raw/firewall-audit.jsonl --firewall-out host-a/firewall.jsonl.gz \
    --manifest host-a/manifest.json \
    raw/audit.jsonl raw/audit.jsonl-*.gz

Action and firewall rows are rebuilt from a closed list of fields rather than scrubbed:

  • An action row keeps its time, operation and action, actor kind, result, a reason category, whether an error occurred, whether the file existed before and after, a block lease, and salted ids for the finding, incident, action and the action it undoes. Addresses map as in findings, networks keep their prefix length and endpoints their port and protocol. File paths, process ids and other targets become tid-<id>. Command lines, error and reason text, undo commands, recovery paths, digests, sizes, modes and owners are dropped. Account names are always mapped, system users included.
  • A firewall row keeps its time, action, target, reason category, source and lease. Firewall entries carry no ids, so nothing joins them to actions or findings; the manifest says so rather than guessing from times or addresses.
  • Admission rows keep their kind and step, including the retirement of an unknown outcome (expired_unproven). An audit row’s target maps like a firewall target, so it joins the legacy row for the same address, and its reason category names the lane. An hourly summary, which sets count, keeps its check, entry, decision, refusal reason and attribution cause, dry-run prediction, count and epoch and names no target; a check that matches a learned account, host or domain refuses the run. An attribution cause belongs only on a refusal summary; action inspection rows carrying one refuse the run. An epoch belongs only on a summary; any other row carrying one refuses the run.
  • An operation, action, actor, result or source the tool has not been reviewed against refuses the run instead of passing through.

The manifest is written last and is the bundle’s completion marker. It lists every input and output with its stream kind, position, SHA-256 of the exact bytes, record count and time span, the tool’s source revision, the salt fingerprint, join counts, counts of discarded fields, action results by value (admission steps and decisions included, at zero when absent), and coverage for each stream. Check every output’s digest against it; a bundle without a manifest, or with a digest that does not match, is incomplete.

  • An action is matched when it names a finding id present in the recording, and nothing more. Actions naming a finding the recording lacks, and actions naming none, are counted apart.
  • Repeated finding rows are kept and counted. A durable action row repeated exactly is a retransmission and does not count as another outcome; rows that share an action id and version but differ are reported as conflicting, never resolved by taking the latest.
  • A result is what the writer recorded. An applied block or a firewall entry is an observation, not a verified effect or a reviewed correct action.

--input-manifest takes the collector’s inventory of what the host has:

{"v": 1, "streams": [
  {"kind": "findings", "availability": "present", "sha256": "<digest of the copied file>", "records": 1200},
  {"kind": "actions", "availability": "not_recorded"},
  {"kind": "firewall_audit", "availability": "absent"}
]}

Every supplied file must appear as a present entry with its digest and record count, and every present entry must be supplied. absent means the collector looked and found none; not_recorded means the host does not keep that stream. Ledger and review streams can only be stated absent or not recorded. Without an inventory, a stream that was not supplied is reported as not supplied.

Comparing the admission preview

While the admission ledger only previews or forecasts, the legacy path decides every automatic block and the ledger sees the same responses. Compare a dry-run window: once admission executes, the legacy path leaves the blocks admission owns to it. compare reads the outputs of one joined run and sets each legacy automatic block in the preview window against what the preview did with its finding. Older unstamped finding rows still supply their identifiers and checks for joins:

/tmp/finding-stream compare --findings host-a/findings.jsonl.gz --actions host-a/actions.jsonl.gz --status host-a/status.json

--status takes the host’s csm status --json output, taken at the end of the run being judged; compare reads only the ledger’s ingress generation, which names that run, and the admission owner’s block_entries from it. Each hourly summary names the daemon run that counted it (epoch), and compare reports each run apart, since a restart loses the hour in progress. Summaries recorded before they named their run are reported as one unmarked run. A run’s window spans its first to its last hourly admission summary; the hour the first one counts may have begun before the run, so it is not counted. A block is matched by one observed attempt for the same finding, response kind and typed target within an hour; a scan block retried from the pending queue may land later, until its retry ages out two hours after it was queued. Spray blocks match through their own entries: the incident spray block of one address through incident_spray, the mail spray subnet block through mail_subnet. Matching can reassign compatible decisions when their windows overlap; each step explains at most one legacy row across all runs. An unused step from before a restart may still explain a pending retry. The same legacy row in a shared boundary hour needs only one step, even when both run reports include it. Observed steps and coalesced decisions are matched before designed refusals, which explain discrepancies but do not satisfy coverage. Coalesced decisions and designed refusals each spend one summary count from their own run for the same entry, kind and check in the same or an adjacent hour. A block the old process made in the hour of a restart also falls in the next run’s window; when the earlier run’s own summaries explain it, the next run reports it as explained by another run, and it adds no coverage there. When both runs can explain several boundary blocks, matching assigns distinct rows before spending spare counts on rows already explained by the other run. Each run still spends only its own counts and earns coverage only from its own matches. Hourly aggregation cannot prove the exact event delay. A backward wall step clamps counts to the admission hour floor; larger clock disagreement stays unexplained and must be checked through clock status. Attribution explains only scan-pass provenance gaps for checks served through the matching entry; policy explains only netblock, permanent-block escalation, challenge timeout, incident or incident spray paths without a retained root. A rootless derived path may spend its fixed unknown-check policy count without a finding link. Other policy and attribution refusals stay unexplained. Summaries must be stamped at an hour boundary, and only refused decisions may carry a refusal reason. Counts that overflow refuse the comparison, and output errors never repeat file paths or host data. Summary matches are aggregate lower bounds, not proof about one finding. Anything else is listed as unexplained. The report also totals Invalid and queue overflow refusals, and names what only csm status and the metrics show: Critical deferrals, queue evictions and the handoff latency.

Each run ends with its gate; only the run the status names is judged. It passes once its window spans at least 48 hours, nothing is unexplained, both totals are zero, and three blocks are matched for every entry the status lists and for every scan check that blocked, even if a later configuration change removes scan blocking from the status. An entry short of three, or listed but never blocking, holds the gate until the run spans seven days; the gate then passes and names it unproven. An unmarked run, or a comparison without --status, never passes. The report ends with the verdict on the status’s run: an earlier run that passed does not pass it. The operator must also verify the named status and metric criteria. Shape validation cannot authenticate a prior run: compare accepts outputs only from the joined run that already passed its identity verifier. The report describes one host’s responses; keep it private. Summary retention discards whole oldest hours after failed writes exceed its bound. Missing summaries may mean idle hours, failed writes or a crash; they cannot certify coverage. Rootless designed-refusal matches consume aggregate counts and provide no finding join. The operator must prove collector coverage and an undamaged ledger through status and doctor before accepting the comparison as passed.

What a recording does and does not contain

The audit log is the dispatch record: it holds findings that were alerted, after deduplication. It is not the persisted latest-state set, which is larger because every scan re-emits the findings it still sees and the merge refreshes their timestamps. On one production host a single sweep dispatched 49 critical findings while the active set carried 157 refreshed critical rows.

A replay therefore understates how full the persisted correlation window gets. Compare windows against each other, and treat an absolute rate from a replay as a property of the replay, not of the live host.

Replaying a stream through correlation

scripts/correlation-calibrate replays a recording through the production cross-account correlation so its thresholds can be re-derived from what hosts produced rather than from an assumption:

go run ./scripts/correlation-calibrate \
    ~/.local/share/csm/finding-streams/host-a.jsonl.gz

It reports how often account-and-check pairs repeat, which checks dominate the input, and what the aggregates did under both derivations: per dispatch batch, and over the persisted latest-state set. For each it prints a threshold sweep using the same eligible, windowed accounts as the aggregate. --window selects the persisted correlation bound: it defaults to one hour, and --window 0 reproduces the original unbounded correlation. Longer windows are applied directly, without the production default limiting them. The source set keeps the store’s retention and size limit. Batch correlation counts all qualifying rows grouped into that dispatch batch. Repeated pairs can include distinct findings on the same account; this metric does not establish how many rows are re-reports of the same finding.

Batch boundaries are inferred from timestamp gaps. Persisted correlation is recomputed at every recorded arrival, including ignored checks that only advance time. Raised duration is measured between those observations, not against an invented scan schedule. Recordings do not contain empty scans, purges or dismissals, so this replay models latest-state accumulation rather than reconstructing every store transition. Rows with no timestamp are counted and skipped because they have no replay position; runtime correlation still counts unstamped stored rows.

The tool needs no host access and reads nothing but the recording. It uses cPanel account roots supplied directly to correlation, without platform discovery on the replay machine.

Replaying scan admission

scripts/response-replay replays a recording through a model of how automatic scan blocks are admitted today: the hourly block limit, the retry queue and its age limit, and eviction at the temporary deny limit. It reports what the model would have done with the recorded findings, not what the host did, and not whether any block was right.

go build -o /tmp/response-replay ./scripts/response-replay
/tmp/response-replay --findings host-a/findings.jsonl.gz --out replay/host-a.json \
    --max-blocks-per-hour 200 --deny-temp-ip-limit 500 --seed 1 --hour-zone UTC \
    --manifest host-a/manifest.json
  • The live path tries queued candidates in Go map order. The model tries them in a reproducible random order chosen by --seed, so compare several seeds rather than reading one run as the order the host used.
  • --hour-zone is the zone the host’s clock runs in: the hourly limit resets when that zone’s hour changes. Local is refused because it would depend on the machine running the replay.
  • --php-relay-ip-recipient-mode mirrors email_protection.php_relay.ip_recipient_mode and defaults to observe. Use enforce to model form-relay client responses. Observed findings never drive a response in either mode. The report records the selected mode.
  • A recorded block from a challenge timeout, central intel, a credential spray or an incident is applied as recorded: it takes a firewall slot but not the hourly budget. Only a single IP address is accepted as the block target; malformed targets are counted as unclassified rows. A permanent block is counted, not modelled.
  • The audit log does not record a finding’s structured source address, so a reputation finding, which names its address only there and in its message, has no address in a replay by default and is counted as missing one. --reconstruct-reputation-source recovers the address from that message’s fixed form; the report then counts the recovered rows and lists the reconstruction among its assumptions.
  • A finding keeps the severity it was recorded with. A later change in how a check grades severity is not applied, so an older recording can replay as blockable a finding the current build would only challenge. Every report lists this among its assumptions.
  • Batches are inferred from equal timestamps. Recordings hold no empty scans, so queued work drains only when another finding arrives. Rows without a timestamp are counted and left out.
  • The report lists the effective policy with every default it applied, the counts, queue delay and eviction residence distributions, blocks per elapsed hour, and the assumptions and mechanisms the model leaves out: infrastructure and allowlist protection, verdict callbacks, subnet blocks, permanent escalation, retries and failures, and manual unblocks. It carries no address, id, name or text from the recording.
  • Candidates the queue dropped (aged out or overflowed) are reported apart from work still queued when the recording ended, which the end of the recording cut off rather than the queue.
  • The demand is the audit log’s record of dispatched findings, not every finding the admission path was called with.
  • The report counts the recorded subnet, ASN-crawl, netblock, permanent and dry-run block rows the model leaves out, and the findings that feed the subnet paths. ASN-crawl subnets spend the same hourly budget the model gives single addresses, so free scan slots in a replay are an upper bound.
  • Hourly distributions include empty elapsed hours across the full recorded time span, without allocating a sample for each empty hour. The first stamped row establishes the replay clock, including dates before year one.
  • With --manifest, the report checks that the manifest describes this exact recording and carries its coverage and recorded outcomes beside the replay. Without one, the report says the recording has no statement of what was collected with it. Negative outcome and join counts are refused.
  • A report is written only to a new path: it never replaces an existing file, and it cannot alias its recording or manifest through symbolic links, parent-directory traversal or hard links. It is staged beside the resolved destination and published as a private file, refusing the run if another writer creates the destination in the meantime. A temporary file cleanup failure is reported with whether the report was published.
  • Like a manifest, a report needs a build of a known commit without local changes.

Crawl Detector Calibration

The distributed crawl detector judges each site and URL pattern over a window of minutes: it removes the heaviest clients, then asks whether the remaining traffic is both far above the pattern’s usual rate and spread over enough distinct clients. Every one of those settings, and the memory bounds behind them, must come from what real hosts served, not from guesses. Two tools turn copies of a host’s access logs into that evidence without retaining raw visitor identities. Pseudonymous recordings are still private. This revision validates bundles and certified coverage and replays them through chronological sessions, but bounded experiments with executable acceptance limits are not yet implemented, so it is still for synthetic data only. Do not convert private recordings, delete preserved recordings on the strength of these outputs, or use reports to choose detector settings until those contracts pass their tests and independent review. Real-data collection and conversion then require separate operator approval.

scripts/domlog-stream converts local copies of domlogs into an anonymized record stream. The following example is for synthetic fixtures only, from a clean committed checkout; use absolute local log-copy paths in the inventory:

# RECORDING_DIR is a mode-0700 synthetic scratch directory outside the checkout.
# Set WINDOW_MINUTES from the synthetic grid; do not reuse a real recording salt.
go build -o "$RECORDING_DIR/domlog-stream" ./scripts/domlog-stream
"$RECORDING_DIR/domlog-stream" convert \
    --salt-file "$RECORDING_DIR/synthetic-salt" \
    --inventory "$RECORDING_DIR/inventory.json" --labels "$RECORDING_DIR/labels.json" \
    --bot-evidence "$RECORDING_DIR/bots.json" \
    --out "$RECORDING_DIR/host-a.records.jsonl.gz" \
    --volume-out "$RECORDING_DIR/host-a.volume.jsonl.gz" \
    --manifest "$RECORDING_DIR/host-a.manifest.json"

Every line goes through the detector’s own record parser and request identity rules, with the same canonicalization. Deployed log escaping and proxy semantics still need qualification before real recordings can be trusted. A record keeps the logged time and order, status, whether the request is dynamic or an expensive query, the Referer class (none, malformed, cross-site or same-site), a claimed bot identity with its verified-bot proof class, an infrastructure flag and the operator’s label. Sites, accounts, client bindings (the full IPv4 address or the IPv6 /64), episode names and URL patterns become salted pseudonyms that keep equality and the site/pattern hierarchy. Two distinct names that would share a pseudonym stop the conversion instead of being merged. No path, query value, address, user agent or Referer is written. Sites and accounts use the same pseudonyms as recorded finding streams under the same salt, so the two can be joined. A volume stream counts lines and bytes per site and minute, including lines without a usable target or client.

The inventory names the recording period, each site’s verified identity and its log copies. The period is bounded by whole UTC minutes, from inclusive and to exclusive, and must have ended. Site names and aliases must be lowercase DNS names without a terminal dot, at most 253 bytes each, and one host belongs to one site: an alias listed twice or naming another site is refused. Use absolute log paths: the tool also accepts a relative path, but resolves it against the directory it runs in, which ties the inventory to that directory:

{
  "period": {"from": "2026-09-20T00:00:00Z", "to": "2026-09-27T00:00:00Z"},
  "sites": [
    {"name": "example.com", "account": "acct1",
     "aliases": ["example.com", "www.example.com"],
     "logs": ["/srv/crawl-recording/raw/example.com",
              "/srv/crawl-recording/raw/example.com-ssl_log.gz"]}
  ],
  "trusted_proxies": ["198.51.100.9"],
  "infrastructure": ["192.0.2.200"]
}

The inventory, labels and bot evidence files must be regular files, not symlinks, devices or FIFOs, in their exact JSON form: every member spelled exactly and given once, no null values and nothing after the document.

Verified-bot identity comes only from the host’s own verified-bot list. The optional bot evidence file is an export of that list for the recording period: the list’s source revision and configuration digest, then published ranges and reverse-DNS verdicts, each for one identity and valid from from (inclusive) to to (exclusive):

{
  "format_version": 1,
  "d2_revision": "<40 or 64 hex digits>",
  "config_sha256": "<64 hex digits>",
  "proofs": [
    {"bot": "googlebot", "kind": "range", "prefix": "203.0.113.0/24",
     "from": "2026-09-20T00:00:00Z", "to": "2026-09-27T00:00:00Z"},
    {"bot": "googlebot", "kind": "dns", "addr": "192.0.2.60", "verdict": "positive",
     "from": "2026-09-20T00:00:00Z", "to": "2026-09-21T00:00:00Z"}
  ]
}

The proofs list is required; use [] for an explicitly empty export. The converter looks nothing up. A claimed identity is range when a range of that same identity held the client at the logged time, else dns or negative from a verdict for that exact address; a user agent alone, another bot’s range, an expired proof or contradictory verdicts leave the claim unverified. The manifest records the evidence digest and its list revision. Bundle validation refuses a proof without that reference or a client binding. A bot the host verifies only through operator-configured user-agent substrings (reputation.verified_bots) has no claimed identity here: its requests are ordinary traffic in the bundle, and bot-label qualification must account for them separately.

Aliases decide which Referers count as same-site; a www host counts only when it is listed. Behind a trusted proxy the client is the rightmost forwarded address only when the qualified proxy appends the original client there. A proxied line without a usable forwarded address, or whose rightmost address is another trusted proxy (an unqualified second hop), counts as attribution loss; a peer that is not a plain address counts as an invalid client. Neither gets a client binding. The optional labels file marks time ranges of a site as attack or overload (each with an episode name) or healthy, optionally only for a first path segment or parameter-name prefix. Each rule requires nonzero RFC 3339 from (inclusive) and to (exclusive) timestamps; matching preserves fractional seconds even though stream timestamps use whole Unix seconds. Prefixes match canonical query names, with ASCII letters lowercased and non-ASCII case preserved. The first matching rule wins and anything else stays unlabeled. Labels are what the calibration measures detection delay and false positives against.

Handling rules for later qualified real-data use (not authorized by this prototype):

  • The operator copies the logs read-only into a private local directory (mode 0700) after explicit collection approval. Take each copy no earlier than the end of the recording period plus the lateness bound, so requests that complete after the last minute are already written; when a copy was taken earlier, exclude the minutes it cannot vouch for as partial_minute. Run the tool, review coverage against those copies and independently recorded handler/logging liveness, then let the operator delete the approved copies. Nothing runs on the monitored host.
  • The salt is created on first use with mode 0600 and must stay private; reuse the finding-stream salt so the streams join. Keep one identity registry per salt, in the salt’s directory. The tool derives its filename as registry-<salt SHA-256>.json, using the full lowercase hex digest of the salt bytes. Optional --registry FILE must name that same file; another filename or directory is refused, even with --new-registry. Directory symlink aliases are accepted. It records every site, account and episode pseudonym the salt has issued, as keyed digests, and refuses a later bundle whose different name would take one; it is locked while a conversion runs.
  • A run that creates the salt starts its registry. For a salt that already exists but never had a registry, such as the finding-stream salt, pass --new-registry on the first conversion only; it is refused once a registry exists. Without that flag, a missing registry stops the run instead of starting an empty one that would forget issued pseudonyms. Back up the salt, registry and persistent .lock file together, and restore them together. Never remove or replace the lock during conversion. If an earlier tool wrote a custom registry filename, stop conversions and move that registry and its lock to the derived names before reusing the salt. Do not start an empty registry to replace existing history.
  • The tool refuses to run from an unknown or modified build, refuses to replace an existing output, and publishes nothing unless every log was read. Each log copy must be a regular file that stays unchanged while it is read. Reads stop at its initial size; size, modification time and change time must still match afterward, so restoring the modification time cannot hide a rewrite. Snapshot checks support Linux and macOS; other platforms refuse conversion. A second path or hard link to a copy already read, or a copy whose decompressed content repeats another, is refused.
  • The manifest, published last, records the digests of every input (as read and decompressed) and output, the tool revision, the period and, per site, the observed extent and the bytes read. Every line read is either a record or exactly one of: oversized, refused by the parser, invalid time, time in the future, time outside the period, or a final line without a newline, which is incomplete and never a request. Records without a usable target or client are counted too. Record bytes are in the volume stream; every other byte is counted as unplaced. For lines without a usable time, the manifest keeps the logged times of the timed lines around them, which bounds where the lost requests belong.
  • The manifest also lists every site, account and episode pseudonym of the bundle with the full keyed digest it is a prefix of. Bundles replayed together must agree on every digest, so bundles converted against a forked or restored registry cannot merge two names unnoticed.
  • The first and last timed lines describe an observed extent, not proof of complete minutes, and the manifest keeps them apart from coverage. Only a coverage proof certifies minutes. An idle log alone cannot certify zero traffic.
  • Streams, labels, inventories and reports stay outside the repository.

scripts/crawl-calibrate replays an experiment through exact models of the detector. Build it from a clean committed checkout as well:

go build -o "$RECORDING_DIR/crawl-calibrate" ./scripts/crawl-calibrate
"$RECORDING_DIR/crawl-calibrate" --experiment "$RECORDING_DIR/experiment.json" \
    --out "$RECORDING_DIR/experiment.report.json"

The experiment file lists the bundles in chronological order, the parameter sets to replay and the truth of every scored episode. Paths are relative to the experiment file’s directory unless absolute. Like the inventory, it must be a regular file in its exact JSON form:

{
  "format_version": 1,
  "identity_version": 1,
  "window": 10,
  "runs": [
    {"params": {"w": 10, "r": 3, "f": 5, "k": 20, "d": 50, "c": 80,
                "baseline": {"alpha": 0.1, "min_obs": 1, "min_age": 10080, "floor_per_min": 1}},
     "sketch": {"m": 64, "h": 128, "seed": 7}}
  ],
  "fixtures": [],
  "bundles": [
    {"manifest": "week1.manifest.json", "records": "week1.records.jsonl.gz",
     "volume": "week1.volume.jsonl.gz", "coverage": "week1.coverage.json",
     "role": "training",
     "states": [{"from": 29840820, "to": 29850899, "state": "normal"}]},
    {"manifest": "day8.manifest.json", "records": "day8.records.jsonl.gz",
     "volume": "day8.volume.jsonl.gz", "coverage": "day8.coverage.json",
     "role": "scoring", "score": [{"from": 29850900, "to": 29852339}],
     "states": [{"from": 29850900, "to": 29852339, "state": "normal"},
                {"site": "dom-0a1b2c.example", "key": {"level": 2, "key": "k-0123456789abcdef"},
                 "from": 29851500, "to": 29851560, "state": "protected"}]}
  ],
  "truth": [
    {"episode": "e-0123456789abcdef", "label": "attack", "site": "dom-0a1b2c.example",
     "keys": [{"level": 1, "key": "k-1111111111111111", "parent": "k-0123456789abcdef"}]}
  ]
}

Bundles must share one salt and identity contract, follow one another in time without overlap, and agree on every identity digest; a disagreement means their identity registries forked or one was restored from an older copy, and the experiment is refused. Each bundle is read once and refused if its manifest is not exactly what domlog-stream wrote, a site is not a unique pseudonym, or any row, digest or total disagrees with the manifest. Without parameter sets the report holds only volume, silence and shape diagnostics. State declarations and truth definitions are validated even without runs. Replaying needs a coverage proof for every bundle, which the operator builds from evidence independent of the logs: the collection record, handler and logging liveness, and a measured bound on how long after its logged time a request can be written. The proof names the manifest by digest and tiles the recording period for every site with certified spans and reasoned exclusions (partial_minute, collection_gap, liveness_unknown, topology_change). It cites evidence only by digest:

{
  "format_version": 1,
  "manifest_sha256": "<SHA-256 of the manifest file>",
  "lateness_seconds": <measured completion-delay bound>,
  "evidence": [
    {"kind": "collection", "sha256": "<digest>"},
    {"kind": "logging_liveness", "sha256": "<digest>"},
    {"kind": "lateness", "sha256": "<digest>"},
    {"kind": "pre_application", "sha256": "<digest>"}
  ],
  "sites": [
    {"site": "dom-0a1b2c.example",
     "spans": [{"from": 29840821, "to": 29840838}],
     "excluded": [{"from": 29840820, "to": 29840820, "reason": "partial_minute"},
                  {"from": 29840839, "to": 29840839, "reason": "partial_minute"}],
     "rejects": [{"from": 29840827, "to": 29840827, "category": "no_target",
                  "lines": 1, "evidence": "<digest of the pre_application entry>"}]}
  ]
}

A replay uses only certified minutes without unknown loss. A minute with a targetless or unattributed record is removed, and so is every minute a line without a usable time may belong to: the minutes between the timed lines around it, widened by the lateness bound. A rejects entry keeps those minutes only when independent evidence shows that exactly that many targetless or oversized lines were refused before the application ran; the HTTP status alone is not such evidence. Removed minutes break windows like any unknown minute and never train a baseline. These bounds assume each copy lists requests in the order they completed. The manifest records, per copy, how far any logged time trails an earlier one; that disorder is a lower bound on completion delay, so a proof whose lateness bound is smaller is refused. The declared disorder must also cover inversions between the timed neighbours of lost lines, including neighbours outside the recording period. A bundle without a proof contributes diagnostics over each site’s observed extent only, and the report says its coverage is unqualified.

Each parameter set replays one session through all bundles in order. Every minute is judged against the history learned before it and only then learned, so held-out minutes never inform earlier ones; windows run on across adjacent bundles, and a gap between them restarts every window. A minute trains a baseline only when its state is declared normal, no finding is active on the key, and either a complete window judged its traffic or the minute was observed silent. Idle keys learn those covered normal zero minutes on reactivation, once only; unknown minutes never become zeros. A minute with no declared state is unknown and never trains. protected, degraded and recovery_hold minutes are frozen the same way whether the protection was applied or only decided. A key’s own declaration wins; otherwise the nearest ancestor’s applies (an L2 key’s covers its L1 keys, the site key’s covers every key), and then the site’s. When a finding starts, the key’s hour-of-week profile and its trust are pinned until a complete window clears it; a coverage gap leaves the finding active but uncertain, and it never counts as a new finding.

The session snapshot API preserves findings across restores. Changing the identity contract also drops old key declarations and non-finding keys; changing baseline semantics resets history and learning watermarks and re-pins active findings without changing their start minute. Either change, or a changed window, sketch mode, size, seed or hash version, requires a complete new window. Threshold and shuffle changes keep compatible evidence and pinned history. A snapshot made immediately after invalidation can be restored again with the same result.

A High transition is the only finding event. Only transitions in a scoring bundle’s scoring spans are reported; findings still active from training appear as active when scoring began. An episode is detected by the first transition in a scored minute at a key its truth entry names, whose window holds the episode’s requests; ancestor findings that were already active are listed, never credited. Every labeled episode in a scoring bundle needs a truth entry for its site. A window’s majority label is only a suggested class for review.

The report records its provenance: the experiment and calibrator digests, the salt fingerprint and, per bundle, its role and scoring spans, manifest and proof digests, converter revision, period and bot evidence. Per bundle and site it records certified and covered minutes, excluded minutes by reason and every line count. It also holds host lines and bytes per minute over the observed extent within the recording periods; gaps between bundles contribute no zero-traffic samples. It also holds line counts for nonempty site-minutes, in-file timestamp disorder across all validated records (including excluded minutes) and how many patterns and clients are active per window. Diagnostic windows continue across adjacent bundles and restart at coverage gaps. Key churn counts each site key only once across the experiment and combines samples in the same UTC hour.

For every parameter set it lists each scored transition with its margins, scope, window labels, suggested class and credited episodes; per site and UTC day the scored and judged minutes, requests by label, transitions and credited transitions; each truth episode’s outcome, detection delay, margins and scope match; key and client peaks over scored minutes; and, with a sketch, an independent exact session’s outcomes, lost or extra decisions and lost findings, beside how far the bounds fall below the exact values of the same window. These sketch comparisons use only scored minutes; both sessions still learn from training and other unscored minutes. Fixtures add synthetic attacks from one-, three- and twenty-request clients with optional padding and churn, replayed cold and scored by suggestion.

Episode onset remains the first labeled request across the experiment that the detector counts, even when its minute is excluded; infrastructure and static requests inside a labeled range never move it. If a later segment introduces the first counted request, findings active at onset are captured again at that request’s minute. An episode with no counted request keeps its first request as onset and is never scored. An undetected episode that began outside the scoring spans, or none of whose counted requests was replayed in a scored minute, is reported as not_scored, not as a qualified detector miss; one whose later requests were replayed and scored is a miss. Relevant anomalous windows keep their margins even when an already active finding prevents a new detection. Only covered records contribute to windows, baselines and detections.

The silence diagnostic joins adjacent spans but stops at an unknown gap. A silent run alone does not certify zero traffic.

A replay is a hypothesis about the detector, not a record of what the host did: logs hold completed requests, not offered load, backend harm or queued work, and the parser is only as good as the server’s log escaping. After qualification, chosen values and their evidence go to a private calibration ledger. Episode delays are offline event-time delays: they exclude request completion, log flush, lateness watermark and tick delay, so they are not correct-finding deadlines on their own. The packed footprint estimate is not measured memory or disk usage.

Qualification must use explicit complete-minute coverage, independently verified zero-traffic minutes, separate training and scoring intervals, and held-out healthy, attack and legitimate-overload cohorts. Warm seasonal profiles require history and observations for every enabled seasonal slot. A cold-start replay is a separate experiment. Neither a model delay nor logged timestamp disorder measures request completion or log-flush delay; disorder is only a lower bound on it.

Review every transition per site, key and day. Majority labels are a summary, not the definition of a correct finding or permission to ignore mixed traffic. The tools model High findings only; absence of Critical or actions here is not an end-to-end safety test. Full-bound resource measurements and a private ledger approval are required before production implementation. The production components must repeat these measurements with their own allocations and I/O.

A cleanup failure returns a fixed error. If the filesystem refuses removal, private temporary or output files can remain; retain them for operator-approved cleanup and do not retry at those paths or consume them as accepted outputs.

The identity registry reserves site, account and episode pseudonyms across all bundles under the same salt. Key and binding pseudonyms are 64 bits and are checked within each bundle only; registering every client and pattern would grow the registry with all traffic ever converted. The registry and its lock must be private regular files; inconsistent digest prefixes are refused. Registry replacement is synced to its directory before bundle publication. Keep the registry and its lock file together; never replace or remove the lock while a conversion is running. A failed publication may leave conservative identity reservations. Keep those reservations on retry instead of rolling the registry back.

A site with no independently certified minutes remains in the bundle: use an empty certified-span list and reasoned exclusions covering its whole period. It contributes no replay windows, while other qualified sites remain usable.

cPanel release tests

Version tags require either a cPanel integration image or a recorded reason for releasing without one. release-preflight runs before builds, and integration repeats the preflight before allocating servers. The tag-specific publication dependencies require that integration succeeds; package registry publication, repository publication and releases cannot use an AlmaLinux/Ubuntu-only result unless the omission was acknowledged. Main-branch integration remains manual and may run without cPanel when no image is configured.

Releasing without cPanel coverage

CSM has no licensed cPanel image while no cPanel licence is available for disposable CI clones. Set the protected variable CSM_RELEASE_WITHOUT_CPANEL to a sentence stating why, for example no licensed cPanel image available. A reason must have at least 12 characters after trimming, contain at least two words, and fit on one line. Blank values and bare flags such as 1 are rejected. The reason is published in the pipeline log and in dist/cpanel-release.json as cpanel_coverage: "absent".

Do not read such a release as cPanel-tested. WHM plugin installation, mail integration, cPanel platform paths, service confinement under a real cPanel layout, and upgrade behaviour are unvalidated in that pipeline, and the findings that depend on them (F19, F22, F25) stay open. Clear the variable as soon as an image exists.

The CSM release maintainer owns the CI variables and image refresh schedule. The PidginHost cloud image administrator owns capture and publication of the private reusable image. The release maintainer must verify a clone before setting INTEGRATION_CPANEL_IMAGE to its immutable ID or versioned slug. Do not set it to a base AlmaLinux image: the package test requires a working cPanel installation and fails if it is absent.

Image preparation

  1. Allocate a dedicated AlmaLinux 9 x86_64 build VM with the CI SSH key. The default test plan is cloudv-2 (8 GB RAM, 80 GB disk); override with INTEGRATION_CPANEL_PACKAGE if needed. Follow the supported OS, hostname, storage and licensing requirements in the cPanel installation guide.
  2. Install full cPanel/WHM using the official installer. Complete initial setup with EA4 Apache, Exim, Dovecot, rsyslog and a valid license arrangement for disposable clone IPs. DNSOnly is insufficient. Do not install CSM or create tenant accounts on the image.
  3. Verify systemctl is-active exim dovecot, exim -bV, doveconf -n, and the WHM AppConfig tools. /var/log/maillog must exist. Ensure cloud-init preserves a valid cPanel hostname and injects the dedicated CI SSH key into the existing phuser account with passwordless sudo.
  4. Have the cloud image administrator capture the VM using the provider’s image preparation process. Remove build credentials and tenant data; regenerate host keys and machine identity on clones. Record the OS, cPanel version, image build date and responsible maintainer in the image metadata. Never publish an image containing a customer backup.
  5. Publish the private image under an immutable ID or versioned slug and verify that the CI account sees it in phctl compute image list. The current CLI can create VM snapshots but cannot promote them to reusable OS images; that step requires cloud image administration access.
  6. Set the protected GitLab project variable INTEGRATION_CPANEL_IMAGE. Run the manual integration job once these checks are on main before relying on the image for a tag release.

Refresh the image for cPanel/OS security updates and review it at least monthly. License activation, SSH injection, empty CSM state and mail service readiness must work on a new clone, not only on the image-build VM.

What the release job checks

The cPanel VM starts without CSM. The job downloads the previously released RPM pinned in build/integration-upgrade.json and verifies its SHA-256. It installs that version, writes operator configuration and a state sentinel, starts its daemon, then upgrades to the RPM built by the current pipeline. It checks that configuration, drop-ins and state survive and that the installed binary has the exact SHA-256 of the current build artifact.

The candidate’s package-mode installer is exercised on that cPanel host, and the service is restarted using its installed unit. Dedicated tests require:

  • cPanel and Exim platform detection with existing platform-derived paths.
  • Working Exim and Dovecot services and configuration commands.
  • Executable WHM CGI, registration reported by the read-only WHM application-list API, and the expected redirect.
  • A running notify service with strict filesystem protection, healthy state, and an attached mail-log watcher.

The ordinary real-system integration suite then runs on that same host and on the AlmaLinux and Ubuntu hosts. No test sends customer mail; alert delivery and automatic response are disabled in the integration drop-in. The cPanel package test only runs with an explicit disposable-server marker and refuses an image with CSM or its state already present.

Update the baseline pin deliberately after reviewing the released RPM and its published SHA-256. Never resolve a moving latest package during this test.

Evidence and cleanup

Integration retains cPanel package output, service/journal diagnostics, baseline and candidate status snapshots, and cpanel-release.json for one year. The JSON identifies the candidate commit, binary/package hashes, image, plan and upgrade baseline. Its package_checks field covers the package phase; the integration job must also pass the later real-system tests and verified server cleanup before publication is allowed.

Server IDs are recorded immediately after creation. Verified cleanup runs before successful completion, with an after_script retry for errors or cancellation. A failed package check retains diagnostics and still reaches that cleanup. Image preparation and a successful live cPanel job are required operational steps; compiling the integration binary locally does not replace them.

The candidate daemon also enables and removes the forward guard through a test drop-in and service restart. This runs the actual cPanel rebuild through the packaged service sandbox and checks that removal preserves operator configuration.

Production build and kernel tests

The default suite tests the portable stubs. test:production also runs the full repository with yara,journal,bpf, the tags used by shipped Linux binaries. It uses the release builder’s YARA-X C library and verifies that its version matches both the Go module and the builder recipe. PHP 8.2 is installed for runtime tests. The same job runs the pinned linter and security analyzer with all shipped tags; scanning APIs retain documented exceptions for intentional reads of caller-selected local files. Rule trust checks cover every accepted filename extension, regardless of letter case. Run the same gate in a suitable Linux image with:

scripts/production-tests.sh portable

Each gate writes the selected test inventory, Go JSON test events, engine versions, and commit ID under production-results/. The inventory comes from go list and the selected Go test files across all packages. A newly added package or tagged test is included automatically. Verification rejects missing tests, package failures, and skipped required regressions. Other environment skips remain visible in the JSON events. These are not kernel coverage.

Kernel runner

test:kernel is a required job on a dedicated Linux shell runner tagged csm-kernel. Provision an ephemeral VM for each job, locked to this project, with local Docker, cgroup v2, BTF, BPF LSM enabled in the boot-time LSM list, BPF ring buffers, fanotify, and nftables. Give it access to pull the pinned builder image. It must have no unrelated workloads or host credentials. The ordinary Kubernetes runner does not satisfy this contract automatically.

Provision it from the cloud catalogue’s alma9 image (AlmaLinux 9, x86_64), the closest available match to the EL production family; alma10, ubuntu26 and debian13 also satisfy the contract. Enable BPF LSM before registering the runner, since it is not in the default boot-time LSM list:

grubby --update-kernel=ALL --args="lsm=capability,yama,selinux,bpf"
reboot
cat /sys/kernel/security/lsm      # must list bpf
stat -fc %T /sys/fs/cgroup        # must print cgroup2fs

Register with gitlab-runner register --executor shell --tag-list csm-kernel, lock it to this project, and disable it for other projects.

This runner does not reproduce the production kernel. Supported hosts run CloudLinux 8 and EL8 on 4.18; no catalogue image offers that kernel. Treat a passing test:kernel as evidence for a 5.14-or-newer kernel only. Capabilities that differ across those kernels – pidfd_open is present on the runner and absent on 4.18 – are probed at runtime instead and reported by csm doctor and the health status.

The job builds build/Dockerfile.production-test on the release builder and uses scripts/go-linux.sh to boot systemd inside a disposable container. GO_LINUX_PRIVILEGED=1 is an explicit Docker-only test option; the wrapper’s normal capability set remains the default. Use the same setup locally only on a dedicated Linux test VM:

docker build --build-arg BUILDER_IMAGE=<release-builder-image> \
  -f build/Dockerfile.production-test -t csm-production-test .
GO_LINUX_RUNTIME=docker GO_LINUX_PRIVILEGED=1 \
  GO_LINUX_IMAGE=csm-production-test scripts/go-linux.sh \
  bash scripts/systemd-account-roots-test.sh production

Systemd runs the packaged service sandbox regression and a second test service. The second service executes the shipped tags plus nftkernel,kernelintegration. Its inventory selects every test added by those tags across all packages, plus explicitly required attachment tests. New kernel test names need no prefix. Required checks include real nftables transactions in isolated network namespaces, kernel-produced BPF ring events, AF_ALG BPF attachment and shutdown, and journal delivery from two actual services followed by reader cancellation. The journal test covers both empty and existing history, excludes unrelated services, and rejects replayed records. Both the CLI and service test binary use the shipped tags. The kernel test service has an explicit Go workspace and toolchain selection; EL8 system services may start without a home directory. Once tests stop and their results and journals are saved, the collector exits the disposable systemd manager directly with the test result, avoiding EL8 exit-target loops. The first service verifies remediation and restore under the packaged unit, custom account-root grants, doctor checks, and process-handle signaling. It does not exercise the daemon’s complete watcher startup; the cPanel package gate and cloud integration cover that separately.

Missing capabilities fail the kernel gate. A skipped AF_ALG attachment test is not accepted. Both production and kernel jobs retain artifacts for one year and are explicit dependencies of publication, including tag-specific dependencies. A missing kernel runner leaves the job pending and publication blocked.

The separate clean application gate provides pinned vendor corpus measurements. cPanel release tests provide package, upgrade, and primary-platform validation. None substitutes for another.

The journal reader starts after existing matching records and also follows services with no prior records. Package integrity rechecks preserve a reported mismatch for the original finding even if that file is now absent or no longer executable; current file mode is not evidence that the modification was repaired.

Required Queue inventory

Both runner modes validate scripts/queue-inventory.json before test selection and execution. Its reviewed owners cover channel handoffs, durable work, coalesced pending work and kernel buffers. Each owner records its health rows, bounds and required publication and lifecycle regressions. The runner writes the union of those regressions and its existing requirements to production-results/<mode>/queue-required.json, then requires actual passing events for every selected requirement.

The portable baseline in scripts/production-required.json also requires the shared accounting, health reporting, doctor and gate regressions. Keep recovery and concurrency regressions with their owner when they test an owner-specific boundary; keep shared contracts in the baseline. A passing full suite alone does not protect an omitted requirement against a later skip.

PHP relay publication requirements start the Linux wiring with temporary filesystem paths and a real state store. They check live registration, persistence, shutdown evidence and failed watcher attachment. They must also reject removal of either production registration; manually registering a provider in a test only proves that provider’s behavior.

The allocation scanner is available for ownership reviews:

go run ./scripts/queuegate -scan > queue-allocations.json

It scans repository Go source across all build constraints, excluding test files, testdata, hidden directories and vendored dependencies. A directory removed during the scan, such as a test’s scratch folder, is skipped; a missing scan root, unreadable directory or failed source read still fails the gate. It records raw channel allocations and imported queuehealth.NewChannel calls, including renamed and dot imports. Named channel aliases resolve across every package variant in a repository import directory; an alias that is a channel in only some variants fails explicitly. Imported named types use the Go toolchain’s source importer. Unresolved types, reflective allocation and indirect references to the accounted constructor fail explicitly. Supporting a new constructor or generic constraint requires scanner tests and an ownership review first.

Allocation identities use the file, enclosing function, assignment target, constructor kind and ordinal. The descriptor includes the allocation expression, expanded repository constants and assignments or local var initializers for the capacity and its local inputs in the enclosing function. Expression grouping and build-variant values are retained, including array lengths, literal indices and slice bounds. Implicit constant declarations, iota, closures and named composite literals in capacity expressions need explicit scanner support and are rejected, including when nested inside field selections. Import aliases are resolved as package names, including aliases that match predeclared identifiers. Runtime collection lengths stay symbolic; only explicit construction and slice bounds enter the local capacity-input graph. The scanner does not infer arbitrary function bodies or type layouts and is not whole-program data-flow analysis.

A version 1 manifest contains allocations and owners. Each allocation copies its scanner descriptor and adds a reviewed class, rationale, and queue owner where applicable. Classes are work, lifecycle, maintenance, and constructor. A semaphore or completion signal can order data stored elsewhere; trace that data before deciding whether it belongs to a work owner. Every work allocation must reference an owner. Changes, omissions and stale descriptors fail validation.

PHP Shield eval-site admission and result channels belong to the php_shield.eval_sites work owner: the ownership proof determines the grade of a finding held by the event reader. Both modes require publication, timeout, result retention, recovery and fail-closed grading regressions. The advisory health row retains a timed-out filesystem walk and a completed result until the caller and walk both settle. A timeout or refused proof counts as lost grading work while the finding stays High.

Each owner records health rows, reviewed bounds, and separate publication and lifecycle evidence lists. An evidence entry names a package, a top-level Go test and its required portable or kernel mode. Owners without a channel allocation need source anchors: a path, symbol and canonical shape. Supported anchors include struct fields, types, variable or constant declarations, and function signatures. Missing or changed anchors fail validation.

For a reviewed manifest, generate requirements for the existing test verifier:

go run ./scripts/queuegate -manifest scripts/queue-inventory.json \
  -mode portable -base-required scripts/production-required.json \
  -required-out queue-required.json

Use the resulting file as scripts/testgate -required with the selected test inventory and actual Go JSON test events. It combines existing required tests with the owner’s evidence for that mode. A missing, skipped or failed required test is not accepted. A passing scanner or a test name in JSON alone does not prove health publication or lifecycle coverage; the named tests need substantive assertions and execution evidence. Runner regression tests exercise both modes with real test events, including changed capacities, unclassified allocations, missing tests and skipped required evidence.

The scanner cannot discover arbitrary queues held in maps, heaps, durable storage, kernel buffers or dependencies. Those need explicit owner entries and source anchors, plus tests that drive real admission, progress, loss and cleanup. Adding a new non-channel owner remains a code-review responsibility. No syntax inventory substitutes for reviewing the behavior of its required tests.

Release Signing

CSM has two separate signing paths:

  • Package repository signing for the normal APT/DNF install path.
  • Detached Ed25519 artifact signatures for raw binaries, tarballs, and package files downloaded outside the package manager.

Do not reuse keys between these paths. The package repositories use GPG because APT and DNF verify repository metadata that way. Detached release signatures use Ed25519 because the standalone install and deploy scripts verify raw artifact bytes with OpenSSL.

Status

SurfaceKey typeCI variableNotes
APT repository metadataGPGCSM_GPG_SIGNING_KEYPublished by repo:publish; operators install with signed-by=/etc/apt/keyrings/csm.gpg.
RPM packages and repository metadataGPGCSM_GPG_SIGNING_KEYPublished by repo:publish; operators use gpgcheck=1 and repo_gpgcheck=1.
Raw binaries, tarballs, .deb, .rpm siblingsEd25519CSM_SIGNING_KEYDetached .sig files for direct downloads and standalone scripts.
YARA Forge rule ZIPsEd25519CSM_SIGNING_KEYSigned by the yara-forge-mirror job; clients verify via signatures.signing_key.

The preferred operator path is the signed APT/DNF repository documented in Installation. Standalone scripts also verify detached signatures: scripts/install.sh embeds the Ed25519 public key in EMBEDDED_SIGNING_KEY. Override it at runtime with CSM_SIGNING_KEY_PEM. Current releases fail before execution if the key, signature, or compatible verifier is unavailable. Setting CSM_REQUIRE_SIGNATURES=0 does not bypass verification for current releases.

Public Key

The same Ed25519 key signs release artifacts and YARA Forge rule ZIPs.

Hex form, for signatures.signing_key in CSM config:

2d1472b2a1d9728c2717b75111487145a7863f7ce731c1b44181f7a68bb908f7

PEM form, for standalone script verification (EMBEDDED_SIGNING_KEY / CSM_SIGNING_KEY_PEM):

-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEALRRysqHZcownF7dREUhxRaeGP3znMcG0QYH3pou5CPc=
-----END PUBLIC KEY-----

Package Repository Signing

repo:publish runs on version tag pipelines and rebuilds the public package repositories from the current tag plus the retained historical releases.

Required protected CI variables:

VariableTypePurpose
CSM_GPG_SIGNING_KEYFileGPG private key used to sign APT metadata, RPM packages, and RPM repo metadata.
CSM_MIRROR_SSH_KEYFileSSH key used to publish the mirror output.
CSM_MIRROR_KNOWN_HOSTSVariableSSH host keys for the mirror host.

The job exports the public key as csm-signing.gpg and publishes it at the mirror root so install docs can reference:

https://mirrors.pidginhost.com/csm/csm-signing.gpg

APT verifies signed repository metadata through the signed-by= keyring. DNF verifies both RPM package signatures and repository metadata via gpgcheck=1 and repo_gpgcheck=1.

Detached Artifact Signatures

sign:artifacts signs binaries and packages with the Ed25519 private key in CSM_SIGNING_KEY when that variable is present. The publish and GitHub release jobs create and sign csm-assets.tar.gz after assembling it. Each signed file gets a .sig sibling uploaded with the artifact.

Examples:

csm-X.Y.Z-linux-amd64
csm-X.Y.Z-linux-amd64.sha256
csm-X.Y.Z-linux-amd64.sig
csm-assets.tar.gz
csm-assets.tar.gz.sha256
csm-assets.tar.gz.sig
csm_X.Y.Z_amd64.deb
csm_X.Y.Z_amd64.deb.sig
csm-X.Y.Z-1.x86_64.rpm
csm-X.Y.Z-1.x86_64.rpm.sig

The signature covers the raw artifact bytes with no hashing wrapper. Verification uses:

openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
  -sigfile csm-X.Y.Z-linux-amd64.sig -in csm-X.Y.Z-linux-amd64

Detached Signature Setup

On a trusted workstation:

openssl genpkey -algorithm ed25519 -out csm-signing.key
openssl pkey -in csm-signing.key -pubout -out csm-signing.pub

Store the private key in GitLab as a protected CSM_SIGNING_KEY variable. Keep the private key in an offline password manager and a second secure backup location. Do not commit it.

For standalone script verification, either:

  • Embed the public key PEM in EMBEDDED_SIGNING_KEY in scripts/install.sh, scripts/deploy.sh, and scripts/deploy-gitlab.sh.
  • Or pass the public key at runtime with CSM_SIGNING_KEY_PEM.

Signature verification is mandatory for current standalone releases. To also reject missing signatures on historical pre-signing releases:

curl -fsSLo /tmp/csm-install.sh https://raw.githubusercontent.com/pidginhost/csm/main/scripts/install.sh
sudo env CSM_REQUIRE_SIGNATURES=1 bash /tmp/csm-install.sh

Detached verification requires OpenSSL 3.0 or newer because the Ed25519 command uses pkeyutl -rawin. The scripts stop on older OpenSSL, including the default on EL8/CloudLinux 8, Ubuntu 20.04 and Debian 11. Use the signed APT/DNF repository on these supported platforms; its GPG verification does not depend on this OpenSSL command. The scripts never download and execute a verifier to bypass this requirement.

If a .sig file exists but verification fails, the installer aborts regardless of CSM_REQUIRE_SIGNATURES. A missing .sig (HTTP 404) is tolerated only with CSM_REQUIRE_SIGNATURES=0 and an explicitly selected complete release version older than v2.2.0, with no suffix or leading zeroes; for any later release the scripts abort even without CSM_REQUIRE_SIGNATURES, because every such release ships a signature and its absence means the download is incomplete or tampered. An unknown version or latest cannot qualify for the exception. This historical exception is checksum-only installation and is reported as unverified, not signature-verified.

The same policy applies to the csm-assets.tar.gz.sha256 checksum: releases published before checksums existed produce a warning and skip the checksum step, while CSM_REQUIRE_SIGNATURES=1 makes the missing checksum fatal. A checksum that exists but does not match always aborts.

The check action only compares checksums and reports update availability; it does not download or execute a release binary. Installation and upgrade perform the signature checks above.

Key Rotation

Package repository GPG key rotation:

  1. Generate a new GPG signing key.
  2. Replace CSM_GPG_SIGNING_KEY in protected CI variables.
  3. Publish a tag pipeline so repo:publish exports the new public key to the mirror.
  4. Update install docs or automation if the key URL changes.

Detached Ed25519 key rotation:

  1. Generate a new Ed25519 key pair.
  2. Replace CSM_SIGNING_KEY in protected CI variables.
  3. Update the embedded public key in standalone scripts, or rotate the CSM_SIGNING_KEY_PEM value used by automation.
  4. Tag a new release.

Old detached signatures remain verifiable only with the old public key. Archive old public keys alongside release metadata so historical releases can still be checked.

Manual Detached Verification

VERSION=X.Y.Z
curl -LO "https://github.com/pidginhost/csm/releases/download/v${VERSION}/csm-${VERSION}-linux-amd64"
curl -LO "https://github.com/pidginhost/csm/releases/download/v${VERSION}/csm-${VERSION}-linux-amd64.sig"

openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
  -sigfile "csm-${VERSION}-linux-amd64.sig" -in "csm-${VERSION}-linux-amd64"

openssl pkeyutl -rawin needs OpenSSL 3.0 or newer. EL8 and CloudLinux 8 ship OpenSSL 1.1.1, whose command line cannot verify Ed25519 at all. Use an already installed CSM build there, which verifies with Go’s implementation and needs no OpenSSL:

csm verify-release csm-signing.pub \
  "csm-${VERSION}-linux-amd64.sig" "csm-${VERSION}-linux-amd64"

It exits zero only for an artifact signed by the given key. The installer and deploy scripts try OpenSSL 3.0+, then an installed csm verify-release, then python3-cryptography, and refuse the artifact when none is available.

The Go and Python verifiers reject special files and limit input sizes: keys to 64 KiB, signatures to exactly 64 bytes, and nonempty artifacts to 512 MiB. Python verification ignores the working directory, user site packages, and Python environment settings when importing its dependencies. Install python3-cryptography as a system package.

If verification fails, treat the artifact as untrusted. Do not install it.

Architecture direction

Why the host agent stays autonomous, keeps one database owner and grows by slices instead of a rewrite.

Keep the host agent autonomous and self-contained, with bbolt owned by one process. Privilege separation must preserve that ownership: the helper and online CLI clients use scoped requests for reads and writes. They must not open the live database independently, even read-only, because bbolt’s writable owner holds an exclusive file lock. Offline maintenance needs exclusive ownership, with the daemon and helper quiesced before handoff.

A local database replacement needs measured contention, query complexity or recovery requirements that the current design cannot meet. Both bbolt and SQLite WAL serialize writers; a switch alone does not remove that constraint. See the bbolt transaction documentation, SQLite WAL concurrency and bbolt read-only mode for the separate-process lock behaviour.

Use the existing store, action log, privileged-operation inventory, scan jobs and incident correlator as the starting points. Add domain interfaces as each slice needs them; a package renaming campaign or a generic bucket/key API is not a prerequisite. Keep operator configuration in YAML and migrate remaining authoritative operational state by domain, with a rollback contract.

The architectural priorities are privilege isolation, durable host actions and browser credential isolation. They complement the harm-based ordering of the roadmap and never defer a protection failure or a precision and response defect. Browser sessions and their HTTP/domain boundary are implemented. Delivery order for the rest:

  1. A firewall block/unblock slice of the durable action lifecycle, including storage measurements and an explicit state-owner contract. Complete the firewall state migration with recovery proof, then use that action contract for the first privileged-helper verbs.
  2. Expand helper coverage and action recovery by response family; extend the existing job model as each long operation moves behind a service. Drop main-process privileges only when the required reads, descriptors and mutations have verified replacements.
  3. Improve correlation through the shared replay harness, then add outbound fleet ingest. Panel availability must never gate local protection.

No mandatory local broker, database server or orchestration platform is added. The fleet service owns its database choice separately; the agent does not select or build a central database stack.

Auto-response safety model

How automatic actions are tiered, budgeted and paused so that one bad detector cannot damage a protected server.

What exists

auto_response.dry_run defaults to on. The scan block path uses the configurable max_blocks_per_hour budget and service restarts use max_restarts_per_hour. The virtual-patch mode has a safe default. The verdict callback lets a panel downgrade a block. Process signalling goes through pidfd. Quarantine and virtual patching resolve paths with openat2 and RESOLVE_BENEATH. The incident correlator has safety caps and a dry-run mode. Firewall changes record a rollback point. mode: observe refuses to run any of it. See auto-response and observe mode.

Automatic file responses share one admission model: realtime and scheduled quarantine, PHP cleaning and access-file cleaning draw on persistent host and account budgets over a rolling hour. Reservations survive reloads, restarts and interrupted actions. Repeated failures pause these responses, and unavailable safety state refuses changes. Pause warnings are deduplicated while the original detections stay visible. Account identity comes from account-home paths; unknown paths share a budget. Failed cleaning leaves the source and recovery evidence for review. Whole-directory and special-file quarantine require manual review. Actions revalidate the target after budget persistence, and the cleaners receive the same file identity captured before admission. The file-response failure pause is host-wide across automatic quarantine and cleaning paths.

Automatic per-address blocks use the firewall engine’s infrastructure, local-address, operator-allow and verified-crawler guards. Block candidates are admitted through reserved lanes and fair scope shares in internal/admission, so independently supported findings keep bounded capacity when one scope floods.

The firewall’s admission snapshot also estimates how long imported and unclaimed temporary occupants have left. Kernel estimates age through later set reads and member checks using elapsed time. Subnet estimates follow the wall clock. A missing kernel expiry leaves the drain estimate unknown. These estimates are observation only; no host enforces admission from them.

Risk tiers

Every operation in internal/privops carries a risk tier, and a test fails when one is missing or disagrees with the reviewed inventory:

TierMeaningExamples
0alert onlymost findings
1recommendation or dry-run recorddry_run blocks, virtual-patch preview
2low-risk reversiblechallenge, rate limit, mail hold
3quarantine or blockfile quarantine, nftables block, virtual patch
4destructive or process-affectingprocess kill, service restart, config rewrite

Each tier gets a confidence floor, a per-action circuit breaker that follows the rules below, mandatory identity revalidation immediately before tiers 3 and 4 (inode and device for files, pidfd for processes, rule handle for firewall entries), and enough recorded metadata to reverse the action. The durable action lifecycle carries that metadata; there is no parallel policy switch or second set of limits.

Adversarial workloads

Safeguards are designed for attacker-triggered load as well as operational faults. Limits bound both resource use and harm to hosted sites while keeping detection and response health visible. Finite action, queue and storage budgets sit alongside evidence-based failure breakers. Budgets are calibrated against representative response demand, including bursts, and measured host capacity; behaviour under overload is validated and withheld responses are visible.

  • A breaker pauses automatic action only. Detection, findings and alerts continue at their own severity.
  • Budgets and pauses are scoped by verified account and action ownership, with shared bounds for common resources and unknown ownership. A source address or check name alone does not establish an independent actor. Admission is deterministic and fair, with bounded reserved capacity for independently supported findings.
  • Only the affected action scopes pause unless a shared mechanism is unsafe. Verified shared storage or execution failures can require a wider pause at once; failures in several accounts alone do not prove that condition. Unavailable intent or budget persistence still refuses new mutations.
  • Target refusals are recorded separately from execution and storage failures. Admitted attempts stay charged, and uncertain outcomes are reconciled before a retry. Every pause has bounded recovery and revalidation; reloads, restarts and policy changes do not silently reset it.
  • Alternative containment must be supported for the affected protocol, honor existing mode and action opt-ins, and pass admission and identity checks. Resource pressure alone never authorizes broader targets or stronger actions. If no safe action is available, the finding is retained with the deferred or refused outcome and the required operator review.
  • A pause that leaves a Critical finding without its automatic action raises a deduplicated Critical alert and is reported by status and csm doctor. Every deferred, refused or dropped candidate is accounted for without an unbounded alert queue. A dispatched request is not proof of containment.

Firewall feedback

For firewall blocks, correctness feedback supplements resource limits:

  • Hard address exclusions and automatic allow protections stay at the authoritative action boundary, including range overlap checks and the existing explicit operator-command semantics. Source attribution is verified against the connecting peer or an explicitly trusted proxy chain. Any operator-session safeguard needs authenticated provenance, expiry and service scope.
  • A detector-specific breaker needs reviewed correctness evidence linked to the original finding and action. Authentication or an operator unblock alone is not proof of a false positive. Independent attack evidence and other healthy response paths are preserved; corrective feedback does not demote findings.
  • Temporary-block expiry and renewal are specified together with the escalation lifecycle. Escalation evidence is validated with corroboration grading; response records and repeated reports are not new corroboration. Operator exclusions and undo are preserved.
  • Firewall sets and admission stay bounded. Range or ASN-based containment requires separate corroboration, protected-range checks and an explicit policy limiting collateral impact; it is not a substitute for capacity.

One admission authority

The same admission authority covers automatic entry points, retries and recovery. Under privilege separation the executor verifies policy and admission independently of caller-supplied findings or state, while a single owner holds the live database.

Calibration

Budgets are calibrated with recorded finding streams joined to action outcomes and reviewed operator decisions. Raw recordings and operational tuning stay private; only sanitized fixtures and aggregate validation results are published.

Durable action lifecycle

Why every host mutation records its intent before acting, and how recovery and undo are meant to work.

internal/actionlog writes one JSONL record per action to /var/log/csm/actions.jsonl, keyed by the operation IDs from internal/privops. A record carries the actor, the finding_id the SIEM audit log also emits so the two streams join, the exact argv when CSM ran a program, the target file’s digest before and after, the result including refusals, and the command that reverses the change. csm actions reads it. See action log. The wiring pattern is settled: record at the operation chokepoint, not at the entry point, so a CLI-driven and an automatic call produce the same record.

A JSONL outcome is evidence, not durable intent. It cannot alone distinguish a refused request from a mutation applied just before a crash, and some effects, including process termination, cannot be undone.

Decision

Every host mutation goes through a shared lifecycle at its operation boundary. Execution stays in the responsible domain; the lifecycle owns identity, admission, persistence, recovery and audit linkage. It reuses mode: observe, auto_response.dry_run, the existing action-specific switches and the file-response budget and breaker settings. There is no second policy switch and no parallel set of response limits.

The first slice is firewall block and unblock. Before mutation it persists a stable action ID, operation, actor, target identity, finding and incident links when present, intended effect and recovery metadata. States distinguish planned, executing, applied and verified from refused, failed, partial, unknown and rolled back. After a crash, incomplete records are reconciled against actual host state before any retry; an uncertain outcome stays visible until it can be proved. A stable ID supports deduplication, not a promise of exactly-once host effects.

A firewall action can carry a batch: a header and a manifest of single-target members, each with its own charge and the rows it removes to make room. Each member ID belongs to exactly one batch. Admitting a known ID with any change to the members, their order or their effects is refused, and a refused batch charges nothing. A single-target block that removes a row to make room records that row as its victim: the only row naming the address before the action, with none naming it after. Selection requires a verified, unexpired row in its canonical spelling and uses complete committed state. Safety and capacity use the active view of the same snapshot; the saved result preserves unrelated rows. In-memory unverified or failed marks also exclude victims, including for operator blocks, without changing the saved rows. Its audit writes an evict_temp record beside the block’s record in the same durable write. A batch only adds its members’ rows, each starting when the batch is planned: nothing evicts for a batch, so admission refuses one whose result drops a row that has not expired or changes any other entry. Kept rows retain their order. In batch planning, unverified rows remain until their owner settles the outcome, even after expiry, and a batch cannot add a block for an address they hold; an unverified address stays pending even when another spelling of the same address has a verified block. Actions written before batches read as one-member batches and keep their stored format. A record written by a newer build is refused as unsupported, never read or overwritten. A firewall-only import moves the durable journal and its snapshot as one unit, and refuses while either database has an unresolved action or undelivered audit. A refusal, including one for nested firewall buckets on either side, leaves the target database unchanged.

The block batch planner checks each automatic temporary block on its own against one committed snapshot, leaves out refused members and runs the rest as one batch with one kernel transaction; a lone valid member runs as an ordinary single action, and a stored batch ID is observed, never planned again. A member too short to outlive its own verification is refused, since its expiry would leave the batch unknown. A batch has one outcome. A crash or lost write at any step, from admission to audit acknowledgement, leaves recovery to prove an outcome or keep it unknown; the kernel is written at most once and each member is charged once. Storage and audit failures reach the caller even when recovery can later prove the outcome. A kernel holding only some members proves neither outcome, so the batch stays unknown until the kernel proves one, an operator resolves it, or it retires. Audit delivery writes one record per member, with the member’s action ID and the batch’s outcome and version, in one durable write. An unknown block action retires as expired_unproven once all member targets are timed and expired, every row its before and after states disagree on is a verified timed block that has expired by the lease clock, its owner attests a trusted clock, its unknown outcome has reached the audit trail, and the kernel holds exactly the rows both states agree on. Recovery then commits those rows as the next revision in one transaction and frees the pending interlock. The outcome stays unknown, its charges stay spent, its audit records say expired_unproven, and nothing is applied again. A lease clock running ahead of the kernel finds an element the kernel still holds and retires nothing. A permanent, unverified or non-block effect, or any change outside the blocked rows, waits for an operator. A retired action cannot be resolved or undone. A missing or degraded clock attestation leaves the action pending; a failed attestation also reports its error. Saved blocked-set evidence must be present, and a changed set topology prevents retirement. IPv6 proof uses saved evidence even if IPv6 is now disabled. Kernel timeouts are checked against the clock sample taken before each read, and a slow read refuses retirement. The unknown outcome must carry its recorded audit acknowledgement; an empty outbox alone does not prove delivery.

Batch evidence selects each member’s current row by ID, target and planning time; retained rows from an earlier use of that ID are not current evidence. An unresolved batch with an expired member remains unknown: absence does not prove that member was never applied. This also holds during immediate verification after apply. A durable terminal outcome is preserved on replay after expiry. A rejected kernel batch is not retried inside the lifecycle. Its owner must first prove rejection and use a new charged action for any retry. Audit is delivered at least once; consumers deduplicate each member or victim event by action ID, transition version, action and target, so a block and its eviction remain distinct.

Batch replay returns the stored manifest, not a fresh decision on candidates left out of the original action. Nil refusals on replay do not mean every supplied candidate ran. The stored requests must appear as an ordered subsequence; extra candidates, including previously refused duplicate IDs, never execute. The original refused list cannot be reconstructed. A singleton uses its member ID as the durable action ID; its unused header ID is not reserved. Retrying that exact automatic temporary block returns the original action before checking current policy or expiry, and cannot add other members. Two stored singletons in one call are a conflict. Dry run refuses only new work: a retry still returns the stored action. A caller needing a durable header alias for singletons must wait for a stored-format extension.

Replaying a settled batch leaves its charges, committed revision and kernel unchanged. An acknowledged audit is not delivered again. Whole-action undo requires the complete saved after-state and kernel proof. Member undo removes only that member’s unchanged row and preserves other members and later live changes. Another live or unverified block on its address refuses member undo. Member undo checks alias expiry on the engine’s lease clock, so a backward wall-clock step cannot make an expired alias prevent undo. Every undo omits expired blocks, allows and subnets, including expired unverified blocks. Restored rows keep their original absolute expiry.

Batch planning dates member rows and kernel evidence by one lease-clock instant, even when the wall clock moves backward during a lease. Operator IP and port allows come from the same committed planning snapshot; allow expiry uses one wall-clock reading. A changed committed revision refuses admission for both batches and lone members.

If intent or budget persistence fails, new automated mutations are refused and the finding is retained. If outcome persistence fails after mutation, the pending intent is preserved, action health reports degraded, and reconciliation runs before another attempt. A bbolt transaction cannot atomically commit a filesystem or kernel change, so host I/O stays outside the database transaction and target identity is revalidated immediately before execution, including on recovery and undo.

JSONL and csm actions remain the operator-facing audit interfaces, linked by action_id. Durable state and audit delivery need a retry and reconciliation contract so an applied action cannot silently lose its audit outcome. csm action show <id> and typed undo dispatch follow; a stored command string is never executed as the authority to reverse a change. Undo verifies the current target, records its own linked action, and refuses changed or irreversible targets. Retention keeps unresolved intent and required recovery evidence. Backup restore must not resurrect pending actions, jobs or sessions as executable or authenticated live state; what is disarmed and what requires review is defined per record type.

The open work is tracked in the roadmap.

Privilege separation

Why a minimal privileged helper will own the operations that need root, and what it must enforce on its own.

Today the whole daemon, including detection, correlation, parsers, threat intelligence, web UI and API, runs as root inside the systemd confinement described in service confinement: ProtectSystem=strict, syscall filtering, a ReadWritePaths allow-list, and systemd-run for the panel scripts that need the whole filesystem. One helper pattern exists: the mail forward guard executes a fixed helper subcommand of the same binary. A parser bug anywhere is therefore a root compromise of the host.

Decision

A minimal privileged helper owns the operations that require root: nftables and firewall changes, process termination, quarantine and other moves across account boundaries, fanotify and BPF initialisation with the resulting descriptors passed back over the socket, and privileged filesystem and configuration writes. Everything else runs with reduced privileges. Reduced, not none: reading every account’s files still needs CAP_DAC_READ_SEARCH, so the unprivileged side keeps a small capability set and loses the ability to write, signal and reconfigure. The helper exposes a narrow Unix-socket RPC authenticated by peer credentials, with fixed verbs and arguments validated against the same path, user, process and firewall scopes the actions enforce today.

The inventory stage is complete. internal/privops lists every operation that needs privilege beyond reading CSM’s own files, with what it writes, the config key that stops it, its risk tier and what an operator loses by withholding the privilege. csm privileges prints it, the capability matrix ships it, and two gates in internal/ci compare it against the packaged systemd unit in both directions, so a writable-path grant that no operation claims fails the build. That table is the action set the helper has to cover.

mode: observe is the interim posture for operators who will not grant a root daemon the ability to act: detection and alerting run, automatic remediation and integration deployment do not. It reduces what the root process does, not what it could do, so it is a stopgap rather than a substitute.

Staging

Each stage ships on its own: the helper for firewall, then signals and quarantine, then privileged configuration and filesystem writes; then descriptor passing for fanotify and BPF; then dropping capabilities in the main process. Each family first needs its safety classification and durable action contract, not completion of every other family’s migration.

What the helper must enforce

Peer credentials authenticate the caller, not the requested operation. The helper enforces target and account scope, identity, permitted verbs and safety policy even if the main process is compromised. It exposes no arbitrary shell, command execution or unrestricted file-write RPC. Request sizes and execution time are bounded; unknown verbs, malformed requests, stale identities and unauthorized peers are tested.

How the helper verifies persisted intent and safety admission is decided before the first helper verbs ship. A record in a store writable by the main process is caller-controlled too; reading it back through RPC does not make it trusted approval. One process owns the store, and helper-enforced policy, budgets and replay protection survive a compromised caller and a helper restart. If the main process remains the owner, its records are evidence only: it must not be able to reset or forge the helper’s safety admission. Forged intent, replay and attempted budget reset are tested alongside valid requests. Socket ownership, protocol compatibility and unavailable-helper behaviour are part of the first slice. Findings are retained when a mutation cannot safely proceed.

The open work is tracked in the roadmap.

Firewall state migration

Why the firewall engine’s authoritative state moves from state.json into bbolt, and the rules the cutover has to keep.

The lossless firewall state storage contract is implemented and tested independently of runtime callers. It preserves complete ordered state with revision checks and atomic replacement. Firewall buckets and store methods exist, and pending configuration rollback already uses bbolt. The engine still reads and writes its authoritative runtime state in state.json.

This belongs with response correctness. JSON writes already use atomic replacement, and the block path persists intended state before touching the kernel. Those guarantees stay; changing the storage format alone cannot make the database and nftables one transaction.

Engine interface

A domain-owned firewall state interface is injected into the engine. It reuses the existing blocked, allowed, subnet and per-port buckets without exposing bbolt transactions to firewall callers. The original store schema and methods were not a lossless engine backend: subnet rows lacked expiry and used a different creation-time field, and loaders hid read and decode failures. The schema and error contract are extended before cutover; original times and explicit provenance are preserved instead of recreated through add methods. A failed or corrupt read must not become a successful empty or partial ruleset. Each logical state change commits together, and the hot-path cache updates only from committed state. Kernel application and recovery follow the durable action lifecycle.

Cutover

Migration is a one-shot step through the owning daemon, or under an exclusive offline maintenance lock. It validates the entire source, imports transactionally, records a schema and cutover marker, and retains the original JSON for rollback. A crash at any cutover step is recoverable. After cutover only bbolt is authoritative; rollback preserves post-cutover changes instead of silently restoring the now-stale JSON. Desired configuration stays in YAML. Existing exports disarm pending configuration rollback; that property is kept and the treatment of new action intent is defined with it. Migration must not reset existing response budgets or failure pauses when those move to bbolt.

The open work is tracked in the roadmap.

Email Forwarder Protection and Visibility – Design and Roadmap

Status: shipped. This page is the design the forward guard, the held-copy quarantine and its retention prune were built from; the configuration keys below are documented in configuration. Date: 2026-06-07. Owner: CSM mail subsystem.

Problem

cPanel/exim servers run many account forwarders that blindly relay inbound mail – including spam and null-sender bounces – to external free providers (Yahoo, Gmail, Outlook). Forwarding spam to those providers generates complaints and tanks the sending IP reputation. The provider then defers all mail from the IP (observed: Yahoo 421 4.7.0 [TSS04] ... unexpected volume or user complaints, thousands of deferrals/day), the queue fills with retrying forwards and backscatter, and legitimate mail is delayed.

Operators today have no view of which forwarders exist, where they point, how much each contributes to the queue, or which outbound IPs are being throttled by which providers – and no control to stop relaying spam externally.

Goal / non-goals

Goal:

  • Full operator visibility: every forwarder, its destination, owner, queue impact, and the outbound-IP reputation picture.
  • Opt-in protection that holds spam/backscatter before it is relayed to an external provider, protecting outbound IP reputation, with no risk of losing legitimate mail.

Non-goals:

  • Not a general inbound spam filter (SpamAssassin owns local-delivery scoring).
  • Not an MTA replacement. CSM observes, supplies data, manages quarantine + UI.
  • Not auto-deleting or auto-disabling forwarders without operator action.

Approved decisions (from brainstorming)

  1. Enforcement action: hold the external-forward copy in a CSM-owned quarantine, always keep/deliver the local copy. Held mail is recoverable (operator release), so false positives mean review, not loss.
  2. What to hold (layered signals, any match -> hold), each individually toggleable:
    • SpamAssassin spam-flagged,
    • ClamAV / YARA-X malware hit,
    • null-sender <> bounce (backscatter),
    • sender IP present in CSM attack DB / reputation,
    • SPF + DKIM + DMARC all fail.
  3. Rollout: single global toggle, default OFF; dry-run mode (log/account only, hold nothing); reuse existing known_forwarders as a skip-list for trusted relays.
  4. Visibility surfaces (all included): forwarder table, outbound-IP reputation panel, per-provider deferral rollup, queue composition breakdown, per-forwarder spam/volume trend.
  5. Build phased: visibility first (read-only, low risk), enforcement second (opt-in), breadth (postfix/non-cPanel) third.

Core architectural decision: CSM is NOT in the live mail path

The MTA performs the hold natively. CSM only (a) generates MTA config + a reputation lookup file, (b) watches the quarantine, (c) renders visibility and handles release/delete.

Rationale (production safety): if CSM is down or crashes, mail must still flow. CSM must never be a delivery dependency. The MTA evaluates the layered signals itself – exim has native SPF/DKIM/DMARC/spam variables, a null-sender test, and an lsearch/cdb lookup against a CSM-maintained bad-IP file. CSM generates the rule; the MTA enforces it.

Failure posture: any error in the MTA rule path must defer (exim retries), never silently drop. CSM-generated config is idempotent and reversible (Remove() restores normal forwarding) and survives cPanel exim rebuilds by using the cPanel-preserved advanced-configuration include sections.

Components (platform-abstracted, OSS-minded)

New package tree internal/mailfwd/:

  • inventory – enumerate forwarders cross-platform. cPanel: /etc/valiases/* (reuse existing ParseValiasEntries / IsExternalDest from internal/checks/forwarder.go and internal/daemon/forwarder_parse.go). Non-cPanel: /etc/aliases, ~/.forward, postfix virtual. Source selected via internal/platform. Returns a normalized Forwarder list.
  • intel – parse exim_mainlog for: per-destination-provider deferrals with reason codes (TSS04 etc.); per outbound-IP reputation (who is deferring it); queue composition (bounce vs real, per-recipient, oldest, frozen). Reuse eximQueueSize / eximQueueDetails in internal/webui/email_api.go.
  • policy – the single source of truth for the layered verdict. One function Verdict(msg MessageMeta, cfg ForwardGuardConfig) (hold bool, reasons []string) used by BOTH the dry-run accounting and the rendered MTA rule (the MTA rule is generated from the same policy definition so they cannot drift).
  • adapter – interface ForwardGuard { Apply(Policy) error; Remove() error; Status() (GuardStatus, error) }. exim adapter first (renders router + transport + bad-IP lookup, reloads exim, writes to cPanel-preserved include). postfix adapter later. Idempotent, reversible.
  • quarantine – CSM-owned maildir for held forwards; release (re-inject via the platform sendmail) and delete; per-forwarder held counters. Mirror the existing internal/emailav quarantine + webui pattern (apiEmailQuarantineList / apiEmailQuarantineAction).

WebUI: extend the existing /email page (handleEmail, internal/webui/ email_api.go, email_groups.go). New read endpoints under requireRead; mutating actions under requireAuth + requireCSRF (existing pattern).

Config

Extends EmailProtection in internal/config/config.go. Remember the three-places rule: DefaultConfig + installer template + packaging csm.yaml.default (CI drift test enforces). Use presence checks (yamlPathExists) where a literal 0/false must be distinguishable from unset.

email_protection:
  forward_guard:
    enabled: false          # master switch, default off
    dry_run: true           # log/account only, hold nothing
    hold_signals:
      bounce_backscatter: true
      spam_flagged: true
      malware: true
      bad_sender_ip: true
      auth_fail: true
    quarantine_retention_days: 14

On cPanel hosts in enforce mode, CSM applies these settings at startup and on config reload. Failed changes are retried on the next 15-minute lookup refresh, using the latest settings. A change that waits for another CSM change to the Exim configuration, such as an antispam one, is logged as deferred, not as a failure, and retried the same way. CSM waits up to one second for the lock before it records a change and before a worker starts one, so short holds do not defer a change. A reload never waits for a forward-guard change: the change runs in the background, and reloads that arrive while one runs are applied right after it, with the latest settings. Shutdown lets the running change finish and discards queued reloads; startup applies the saved settings. Disabled and dry-run settings start no removal worker when no guard is installed. While another CSM change to the Exim configuration runs, even one started before a daemon restart, the removal is deferred the same way. CSM records each change before its worker starts, and a worker whose change is no longer the latest, such as one started before a restart, changes nothing. Each submission has a distinct identity, even when its settings and sender list match an earlier submission. A delayed worker cannot replace a sender list refreshed since the newer submission. If its request record is missing or unreadable, the worker leaves the installed guard untouched. Request records are readable by root only; the Exim configuration and the bad-IP lookup stay readable by the mail transport user. Observe mode and hosts without cPanel leave the installed guard untouched; in observe mode CSM only cancels pending changes. Cancellation does not wait for another Exim transaction to finish. A worker that has already checked its request under the Exim lock finishes its transaction; later workers cannot apply canceled requests.

WebUI surfaces (email page)

  • Forwarder table: source -> destination(s); provider badge (yahoo/gmail/outlook/external/local); owner (cPanel user); keep-local vs forward-only; queued/stuck count; held-or-would-hold count; status (e.g. “Yahoo throttling”); row actions: disable forwarder, add to skip-list, view held.
  • Outbound-IP reputation panel: per sending IP, provider deferrals + parsed reason codes + throttle state.
  • Per-provider deferral rollup: provider -> queued + throttle + % of queue.
  • Queue composition: bounce vs real, top stuck recipients, oldest/frozen, one-click flush backscatter.
  • Held forwards: quarantine list with release/delete (phase 2).

Safety / OSS

  • Default off; dry-run first; skip-list for trusted relays.
  • MTA-native enforcement -> fail-open (CSM down never blocks mail).
  • MTA rule errors defer, never drop.
  • Cross-platform via adapters; exim/cPanel first; capability flag mail.forward_guard.v1 in internal/health/capabilities.go.
  • All mutating actions audit-logged; fully reversible.
  • No private hostnames/IPs in committed code/tests/docs (use RFC 5737/3849).

Testing

  • policy.Verdict: table-driven over each signal, combinations, skip-list, dry-run on/off. Real assertions.
  • inventory parsers: fixtures per platform + fuzz target (attacker-controlled valias/aliases/.forward content).
  • adapter: render-to-golden exim config; idempotent apply/remove; reload failure -> no-op rollback (no half-applied state).
  • intel parsers: real-format exim deferral lines (TSS04, Gmail, Spamhaus) + fuzz.
  • quarantine: release re-injects correctly; retention prune; per-forwarder counts.
  • No live-mail tests on production. Cross-compile + vet locally; verify on phctl scratch hosts (fresh Ubuntu + AlmaLinux).

Roadmap (each item = its own spec-slice / plan / TDD / codex review / MR)

Phase 1 – Visibility (read-only, ship first):

  1. mailfwd/inventory: cross-platform forwarder enumeration + unit/fuzz tests.
  2. Email-page forwarder table + API (source/dest/provider/owner/keep-local).
  3. mailfwd/intel: outbound-IP reputation + per-provider deferral rollup (exim_mainlog parse) + the two panels.
  4. Queue composition panel + one-click backscatter flush action.
  5. Dry-run policy engine: per-forwarder “would-hold” accounting surfaced in the table (no enforcement yet).

Phase 2 – Enforcement (opt-in): 6. Config schema + settings-page toggles + validation/defaults (three-places). 7. policy -> exim adapter: router + transport + bad-IP lookup; idempotent apply/remove; cPanel-rebuild survival. 8. mailfwd/quarantine for held forwards + release/delete + per-forwarder held counts. 9. Live enforcement wiring (dry-run -> enforce), audit log, findings, capability flag.

Phase 3 – Breadth: 10. postfix adapter (non-cPanel hosts). 11. Reputation aids: provider feedback-loop hints, SRS guidance, delisting checklist surfaced to the operator.

Existing code to reuse (do not reinvent)

  • internal/checks/forwarder.go: ParseValiasEntries, IsExternalDest, IsPipeForwarder, parseLocalDomainsContent.
  • internal/daemon/forwarder_parse.go: parseValiasFileForFindings, checks.IsKnownForwarder; live ForwarderWatcher.
  • internal/webui/email_api.go: apiEmailStats, eximQueueSize, eximQueueDetails, topMailSenders, quarantine list/action handlers.
  • internal/webui/email_groups.go: email-page grouping + sort.
  • internal/config/config.go: EmailProtection struct, KnownForwarders.
  • internal/emailav: quarantine model + spool watcher (ClamAV/YARA).
  • internal/platform: OS/panel/path/MTA abstraction.
  • internal/health/capabilities.go: capability flags.

Conventions (from AGENTS.md)

Production-grade, no shortcuts. TDD for security-relevant logic (test first). go test ./... -race, golangci-lint, go vet, govulncheck clean before push. gosec annotations need a real reason. CHANGELOG entry (1-2 sentences, plain English, no internals) in the same commit as behaviour changes. One conventional-commit subject per commit, no Co-Authored-By. Branch off main; never push/MR without explicit instruction. codex review at each item, then self-review (read each changed file end to end). Platform paths only via internal/platform. Never allowlist scan paths.

PHP form-relay state

The form-relay store keeps the per-client recipient history behind php_relay Path 6, described in PHP-Relay. The ip_recipient_spread and ip_recipient_window_hours settings supply the recipient limit and observation window; the store adds no settings.

Counts saturate at the recipient limit. At capacity the store retains the most recent distinct outside recipients, including one entry per unparsed message. New activity replaces older evidence so a busy client does not lose its recent history when its first recipients expire. Lowering the limit retains the newest evidence within the new bound. Raising it cannot recover previously discarded recipients.

The window ends at the latest time observed by recording, cooldown checks, or sweeping. Delayed messages may contribute only while they remain in that window; replaying expired mail cannot recreate expired history or cooldowns. The time boundary survives removal of idle clients. Expired recipients are removed before capacity eviction, so stale counts cannot displace active clients. Expiry alone does not report capacity loss.

Each account retains its most recently observed staff clients. A delayed staff record cannot replace a newer one merely because it arrived later. Staff history is stored apart from recipient history and expires on its existing schedule.

All state transitions and reads hold the store mutex. The evaluator runs only for a message the store counted. It reads the client’s count, script shares and recipient domains and claims the client cooldown in one call under that lock, so expiry cannot remove evidence between counting it and building the finding. Before counting, that call drops recipients the client reached through an account where it is now staff, or through a script that is now ignored, so a current exemption also covers mail recorded before it applied. Script shares are grouped by the script’s normalized name, the one findings print. Returned script shares and recipient domains are copies.

When that call shows the client would fire, the evaluator first reads the sites’ access logs for staff evidence, outside the lock and without claiming. Each distinct log is read once per crossing, even when several scripts share it. Only successful GET requests to recognized staff pages qualify; public or subscriber pages and ambiguous request paths do not. The latest qualifying request time across those logs sets the staff expiry, so rereading an old log cannot renew an exemption. Invalid, future and expired timestamps do not qualify. It marks any account with evidence as staff, then repeats the call, which claims the cooldown only if the remaining accounts still cross the threshold.

The store also records when each normalized script name was last reported, so a script finding repeats neither for a later client nor after the routine sweep of per-script state within the window.

Client findings carry the triggering Exim message ID as their observation in the spool stream. Admission can trace direct responses and later challenge timeouts to that same mail. Script findings remain advisory and carry no address evidence.