CSM Documentation
CSM is a local security monitoring and response daemon for Linux web servers, with or without a control panel. Platform detection picks the paths and checks for cPanel/WHM, Plesk, DirectAdmin, or no panel, on Apache, Nginx, LiteSpeed, or no web server. cPanel hosts get the deepest account and CMS integration; see supported platforms.
Start here
- New installation: Install the signed package, review configuration, start the daemon, then establish the baseline.
- Daily operation: Use the Web UI,
csm statusandcsm doctor, and the incident response runbook. - Response policy: Read Auto-response and Firewall before enabling automatic actions.
- Coverage: Compare real-time, critical, and deep checks with the services installed on the host.
- Integrations: Start with the API, metrics, and audit log.
How CSM works
Real-time watchers process filesystem, authentication, access-log, mail, PAM, BPF, and ModSecurity events. Scheduled scans cover host integrity, accounts, CMS files and databases, packages, mail configuration, and performance. Findings are stored locally and can feed alerts, incidents, the Web UI, API clients, Prometheus, syslog, webhooks, and SIEM exports.
Platform-specific checks skip when their required panel or service is absent. The documentation calls out cPanel-only behavior instead of treating a skipped integration as a failure.
Safety model
Auto-response is off by default. Automatic IP blocking has a dry-run default, while file, process, mail, and BPF actions have their own gates. Infrastructure and operator-allowed addresses are protected from automatic blocks, quarantine records restoration metadata, and firewall configuration supports confirmation-based rollback.
Use Configuration as the source for current defaults and Upgrading for state, config, and package migration rules.
Installation
Supported Platforms
| Platform | Web server | Package | Notes |
|---|---|---|---|
| AlmaLinux / Rocky / RHEL / CloudLinux / CentOS Stream 8+ | Apache (httpd), Nginx, LiteSpeed, or none | .rpm | Host, web, and mail checks for the detected stack, with rpm -V package integrity. |
| Ubuntu 20.04+ / Debian 11+ | Apache (apache2), Nginx, LiteSpeed, or none | .deb | Same, with debsums/dpkg --verify in place of rpm -V. |
| cPanel/WHM on CloudLinux / AlmaLinux / Rocky / Ubuntu | Apache (EA4) or LiteSpeed, with or without ea-nginx | .rpm or .deb | Adds cPanel account, WordPress, Exim, and WHM plugin coverage. |
Postfix hosts with Dovecot also get mail login detection and optional rspamd spam filtering. On hosts without cPanel, set account_roots for real-time document-root rules, scheduled YARA sweeps and web-root performance checks. Other account and CMS scans still depend on account layouts; see deep check platform support.
The daemon auto-detects the OS, control panel (cPanel/Plesk/DirectAdmin/none), and web server (Apache/Nginx/LiteSpeed) at startup. The detected platform is logged at startup as:
[2026-04-10 08:13:37] platform: os=ubuntu/24.04 panel=none webserver=nginx
Check it with journalctl -u csm.service | grep platform: after starting the daemon.
APT repository (Debian / Ubuntu) – recommended
The package repository at mirrors.pidginhost.com/csm/ is the preferred install method for Debian and Ubuntu. Updates are available through apt upgrade, and APT verifies the repository’s signed metadata before installing packages.
# 1. Install the signing key
curl -fsSL https://mirrors.pidginhost.com/csm/csm-signing.gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/csm.gpg
# 2. Add the repository
echo "deb [signed-by=/etc/apt/keyrings/csm.gpg] https://mirrors.pidginhost.com/csm/deb stable main" | \
sudo tee /etc/apt/sources.list.d/csm.list
# 3. Install
sudo apt update
sudo apt install csm
Works on Ubuntu 20.04+, Debian 11+, and compatible derivatives. The single stable suite serves every supported release. Production binaries dynamically link glibc with a build floor of 2.28, which is available on all supported distributions; YARA-X itself is linked into the executable.
To upgrade later: sudo apt update && sudo apt upgrade csm.
DNF repository (AlmaLinux / Rocky / RHEL / CloudLinux / cPanel) – recommended
# 1. Import the signing key into the RPM keyring
sudo rpm --import https://mirrors.pidginhost.com/csm/csm-signing.gpg
# 2. Add the repository
sudo tee /etc/yum.repos.d/csm.repo >/dev/null <<'EOF'
[csm]
name=CSM - Continuous Security Monitor
baseurl=https://mirrors.pidginhost.com/csm/rpm/el$releasever/$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.pidginhost.com/csm/csm-signing.gpg
EOF
# 3. Install (check the fingerprint below before accepting the key)
sudo dnf install csm
rpm --import populates the RPM keyring, which covers gpgcheck – the
signature on the package. It does not populate the separate keyring dnf keeps
for repo_gpgcheck, the signature on the repository metadata. So the first
transaction against a new repository still prompts, even after a successful
import:
Importing GPG key 0x81CD59B1:
Userid : "CSM Package Signing (CSM Repository Metadata Signing Key) <security@pidginhost.com>"
Fingerprint: 3A70 4D78 3CF2 6055 B2AA 8F49 4E0F 27F5 81CD 59B1
Is this ok [y/N]:
Without -y, an unanswered key prompt defaults to no. A rejected key can
produce an error such as:
Error: Failed to download metadata for repo 'csm':
repomd.xml GPG signature verification error: Bad GPG signature
The error alone does not distinguish an untrusted key from damaged or incorrectly signed metadata. Check the fingerprint against the value above, then verify the metadata signature:
base=https://mirrors.pidginhost.com/csm/rpm/el9/x86_64/repodata
curl -fsSLO $base/repomd.xml
curl -fsSLO $base/repomd.xml.asc
curl -fsSL https://mirrors.pidginhost.com/csm/csm-signing.gpg | gpg --import
gpg --verify repomd.xml.asc repomd.xml
# gpg: Good signature from "CSM Package Signing ... <security@pidginhost.com>"
For unattended installs, approve the repository key in provisioning first,
then use sudo dnf -y install csm. DNF’s
assumeyes option
accepts key-import prompts as well as package prompts. Keep both signature
checks enabled; -y is not a substitute for verifying the expected key.
The $releasever variable auto-selects the matching EL major (8, 9, or 10). Both x86_64 and aarch64 are published. Works on AlmaLinux 8+, Rocky 8+, RHEL 8+, CloudLinux 8+, and cPanel-managed hosts.
To upgrade later: sudo dnf upgrade csm.
Online standalone installer
Use the standalone installer when the host has Internet access but cannot use the APT or DNF repository. It downloads the binary and supporting assets from the latest GitHub release, verifies their checksums, requires successful Ed25519 signature verification, and installs outside the package manager.
curl -fsSLo /tmp/csm-install.sh https://raw.githubusercontent.com/pidginhost/csm/main/scripts/install.sh
less /tmp/csm-install.sh
sudo bash /tmp/csm-install.sh
Standalone verification uses OpenSSL 3.0 or newer, an already installed CSM build providing csm verify-release, or python3-cryptography – in that order. EL8 and CloudLinux 8 have the last of these, so the standalone path works there even though their OpenSSL 1.1.1 cannot verify Ed25519. A missing key, missing current-release signature, absent verifier, or failed verification stops the install before executing the binary. See Release signing for the narrowly scoped historical-release exception.
It auto-detects the hostname and alert email, generates a Web UI token, and prompts before applying. Non-interactive mode:
sudo bash /tmp/csm-install.sh --email admin@example.com --non-interactive
This is not an offline or air-gapped installation path. Mirror the signed packages and repository metadata for disconnected environments.
Manual .rpm / .deb download
If you need a specific version or want to install without adding the repository, verify its detached signature before invoking the package manager. Save the trusted public key from Release signing as csm-signing.pub. These commands require OpenSSL 3.0 or newer; older hosts should use the signed repository above. Installing a local package does not by itself establish the repository signature chain:
# RHEL family
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm-VERSION-1.x86_64.rpm
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm-VERSION-1.x86_64.rpm.sig
openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
-sigfile csm-VERSION-1.x86_64.rpm.sig -in csm-VERSION-1.x86_64.rpm && \
sudo dnf install -y ./csm-VERSION-1.x86_64.rpm
# Debian/Ubuntu
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm_VERSION_amd64.deb
curl -LO https://github.com/pidginhost/csm/releases/latest/download/csm_VERSION_amd64.deb.sig
openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
-sigfile csm_VERSION_amd64.deb.sig -in csm_VERSION_amd64.deb && \
sudo apt install -y ./csm_VERSION_amd64.deb
Replace VERSION with a published version. Both files are also available from the package mirror if you need to pin a release without adding the repository.
Filesystem layout
The package uses FHS paths for config, state, drop-ins, and shipped profiles. Upgrades keep /opt/csm/csm.yaml as a compatibility link for older scripts:
| Concern | Current path |
|---|---|
| Main config | /etc/csm/csm.yaml |
| Legacy config link | /opt/csm/csm.yaml |
| Drop-in fragments | /etc/csm/conf.d/*.yaml |
| State directory | /var/lib/csm/state/ |
| Shipped profiles | /usr/lib/csm/profiles/ |
| Shipped config template | /usr/lib/csm/csm.yaml.default |
| Audit log | /var/log/csm/audit.jsonl |
| Binary | /opt/csm/csm |
| Quarantine | /opt/csm/quarantine/ |
| YARA / signature rules | /opt/csm/rules/ |
On Debian and Ubuntu, removing the package keeps the main config. Purging the package removes the main config and its package-created compatibility link. If the main config is a symlink, purge leaves its target in place.
The systemd unit declares StateDirectory=csm and ConfigurationDirectory=csm so systemd manages permissions for the FHS directories. /var/lib/csm is mode 0711, searchable but not listable, so Exim can reach the email forward guard files below it; /var/lib/csm/state/ stays 0700. On upgrade, the package copies a real legacy main config into /etc/csm/csm.yaml when needed and points /opt/csm/csm.yaml at it. On first start the daemon copies a non-empty legacy /opt/csm/state/ into /var/lib/csm/state/ (only when the new directory is empty), then continues using the FHS state path. See Upgrading - FHS migration for the manual-binary-swap case.
The binary installer and package scripts also keep the shared state root searchable, including on systemd 239, which does not reset an existing state directory’s mode. Applying the forward guard repairs legacy parent permissions without making private directories listable and rejects symlinks in those parents. Install and csm rehash likewise refuse a symlinked /var/lib/csm; relocate state with state_path instead. Lookup refreshes maintain search access to the lookup directory inside the daemon sandbox; full guard application runs outside it.
When refreshing the service unit, install and csm rehash create any missing directory the unit requires before replacing it, preserving existing modes except for the shared state root’s required search bits. Optional paths and systemd-managed directories are left alone unless needed as parents of a required path. If a required directory cannot be created, the refresh fails and the previous unit stays in place.
Post-install (all methods)
RPM/DEB packages are the maintained installation path and include package-manager ownership, integration profiles, UI assets, rules, and the prebuilt PAM module. The standalone installer supplies the runtime assets but does not register them with a package manager. In either case, set infrastructure IPs and confirm the alert address before starting the daemon.
sudo vi /etc/csm/csm.yaml # Set hostname, alert email, infra IPs
sudo csm validate # Check config syntax (--deep adds connectivity probes; validates merged conf.d too)
sudo systemctl enable --now csm.service
sudo csm baseline # Record current state as known-good via the daemon (must be running)
Then open the Web UI at https://<server>:9443/login. The baseline is detailed under Baseline Scan.
Rollback to an older version
Both the APT and DNF repositories retain the last 5 tagged releases at any time. To downgrade:
# Debian/Ubuntu
sudo apt-cache policy csm # Show available versions
sudo apt install csm=<version>-1
# RHEL family
sudo dnf --showduplicates list csm # Show available versions
sudo dnf downgrade csm
Verifying platform auto-detection
After systemctl start csm.service, the first line after “CSM daemon starting” reports what CSM detected:
[2026-04-10 08:13:37] CSM daemon starting
[2026-04-10 08:13:37] platform: os=almalinux/10.0 panel=none webserver=apache
[2026-04-10 08:13:37] Watching: /var/log/secure
[2026-04-10 08:13:37] Watching: /var/log/httpd/error_log
[2026-04-10 08:13:37] Watching: /var/log/httpd/access_log
If any field shows none or unknown when you expect something, the auto-detect missed it. File a bug with the output of cat /etc/os-release, systemctl is-active nginx apache2 httpd, and which nginx apache2 httpd.
Optional system dependencies
The daemon runs as one executable plus packaged UI, rule, profile, and PAM assets. Release binaries require glibc 2.28 or newer and the package depends on systemd. A few host packages enable additional checks:
| Package | Platforms | Enables |
|---|---|---|
auditd | All | Shadow file / SSH key tamper detection via auditd |
debsums | Debian/Ubuntu | Cleaner system binary integrity output vs. dpkg --verify fallback |
logrotate | All | Rotation of /var/log/csm/monitor.log, /var/log/csm/audit.jsonl, and the PHP Shield event log |
wp-cli | Optional | WordPress core integrity check; its absence is reported when WordPress sites are found or discovery is incomplete |
| ModSecurity | All | WAF enforcement checks (see platform-specific install below) |
Installing ModSecurity
CSM detects ModSecurity but doesn’t install it for you. Platform-specific commands:
# Ubuntu/Debian + Nginx
sudo apt install libnginx-mod-http-modsecurity modsecurity-crs
# Ubuntu/Debian + Apache
sudo apt install libapache2-mod-security2 modsecurity-crs && sudo a2enmod security2
# AlmaLinux/Rocky/RHEL + Apache (requires EPEL)
sudo dnf install -y epel-release
sudo dnf install -y mod_security
sudo systemctl restart httpd
# AlmaLinux/Rocky/RHEL + Nginx (requires EPEL)
sudo dnf install -y epel-release
sudo dnf install -y nginx-mod-http-modsecurity
sudo systemctl restart nginx
After installing ModSecurity, run csm check and the waf_status finding should disappear.
Manual (deploy.sh)
/opt/csm/deploy.sh install
vi /etc/csm/csm.yaml # set hostname, alert email, infra IPs
csm validate
systemctl enable --now csm.service
csm baseline
Baseline Scan
The csm baseline command scans the entire server and records the current state for change tracking. This is required on first install so CSM knows what’s “normal” for your server. Findings that should never be silently trusted, such as non-standard MySQL superusers or WHM root API tokens, can still be reported on this first scan.
What it does:
- Scans all cPanel accounts for malware, permissions, and configuration issues
- Records file hashes, email forwarder hashes, and plugin versions
- Stores everything in the bbolt database (
/var/lib/csm/state/csm.db)
How long it takes: Depends on server size. A server with 100+ cPanel accounts and thousands of WordPress sites can take 5-10 minutes. The daemon must be running because the baseline is coordinated through the control socket.
When to re-run:
- After a fresh install
- After restoring from backup
- After an intentional state reset approved by the operator
- You do NOT need to re-run for normal deploys/upgrades – the daemon handles incremental state
Important: Start csm.service before running csm baseline. If existing history would be cleared, rerun with csm baseline --confirm only after verifying that reset is intended.
Configuration
CSM is configured via /etc/csm/csm.yaml, with --config <path> to override. Legacy installs that only have /opt/csm/csm.yaml keep working; packaged upgrades migrate that file into /etc/csm/csm.yaml and leave the old path as a compatibility link. Optional drop-in fragments under /etc/csm/conf.d/*.yaml are merged on top of the main file at startup; see conf.d drop-ins below.
Environment-backed tokens and signing secrets use the environment inherited at daemon startup. After changing an environment file, restart the daemon; configuration reload alone does not import those changes. See credential rotation for the systemd procedure.
Installation defaults
Packages and csm install write the same commented configuration, so a new
host starts with the same settings whichever way it was installed. A key left
out of the file takes the built-in default, which can differ from the value
the shipped file sets. Left out or left blank, backdoor_ports and
firewall.smtp_allow_users take the shipped lists; write [] for none.
Users missing from the host are skipped.
The CLI generates a fresh Web UI admin token and issue time, validates the
result, and writes the new configuration with mode 0600. Invalid templates
stop installation without replacing the destination configuration.
Operating mode
mode declares what CSM may do to the host. enforce (default) leaves every
subsystem under its own switch; observe runs detection and alerting without
automatic host remediation or integration updates. Full reference, including
required switch settings: Observe mode.
mode: enforce
Platform & Web Server
CSM auto-detects the host OS (Ubuntu, Debian, AlmaLinux, Rocky, RHEL, CloudLinux), control panel (cPanel, Plesk, DirectAdmin, or none), and web server (Apache, Nginx, LiteSpeed, or none) at daemon startup. The detected platform is logged as:
[2026-04-10 08:13:37] platform: os=ubuntu/24.04 panel=none webserver=nginx
The daemon then chooses the correct log paths, config candidates, and check set without any configuration from you. Verify with:
journalctl -u csm.service | grep platform:
Web server overrides
For hosts with a custom layout (reverse proxy, non-standard package locations, chroot), add a web_server: section to csm.yaml. Every field is optional – anything left blank falls back to auto-detection.
web_server:
type: "nginx" # apache | nginx | litespeed -- overrides auto-detect
config_dir: "/etc/nginx" # Apache or nginx configuration root
access_logs: # tried in order until one exists
- "/var/log/nginx/access.log"
- "/srv/logs/nginx/access.log"
error_logs: # used by ModSecurity deny watcher
- "/var/log/nginx/error.log"
modsec_audit_logs:
- "/var/log/nginx/modsec_audit.log"
modsec_error_log (legacy single-path override) is still honored and takes precedence over web_server.error_logs for the ModSecurity watcher only:
modsec_error_log: "/opt/myapp/logs/modsec_audit.log"
Account roots (plain Linux web-scan coverage)
By default, web-root performance checks iterate /home/*/public_html, which is the cPanel layout. The WP-Cron check and remediation also use cPanel’s domain map for validated addon and subdomain document roots. On a plain Linux host, point CSM at the actual web roots:
account_roots:
- "/var/www/*/public" # e.g. Laravel/Symfony sites
- "/srv/http/*" # Arch / generic layouts
- "/home/*/public_html" # add if you also have cPanel-style accounts
Each entry is an absolute, normalized glob pattern expanded at scan time. Configured directories also define remediation and restore scope. Custom locations need service write access; see Custom account roots. Non-existent matches are silently dropped. If account_roots is empty and CSM is not on a cPanel host, the account-scan checks return no findings (they run but find nothing, which is the correct behavior for a plain-Linux host with no configured web roots).
The setting covers perf_error_logs, perf_wp_config, perf_wp_transients, perf_wp_cron, real-time php.ini monitoring, and WP-Cron remediation roots. CMS integrity, phishing, .htaccess, and file-index scans use the platform’s account layout: every directory under /home on cPanel, DirectAdmin and plain hosts, under /var/www/vhosts on Plesk.
Minimal Config
hostname: "csm.example.com"
alerts:
email:
enabled: true
to: ["admin@example.com"]
disabled_checks: [] # optional: suppress these checks from email only
smtp: "localhost:25"
webui:
enabled: true
listen: "0.0.0.0:9443"
auth_token: "<fresh random token>"
auth_token_issued_at: "<UTC RFC3339 time when the token was generated>"
infra_ips: ["10.0.0.0/8"]
Full Reference
hostname: "csm.example.com"
# --- Alerts ---
# Email and webhook may both be off; findings then show only in the web UI and API.
alerts:
email:
enabled: true
to: ["admin@example.com"]
from: "csm@csm.example.com"
smtp: "localhost:25"
disabled_checks: [] # check names to keep in web/history but exclude from email
webhook:
enabled: false
url: ""
type: "slack" # slack, discord, generic, phpanel
hmac_secret: "" # phpanel webhook signing secret
hmac_secret_env: "" # env var containing phpanel signing secret
heartbeat:
enabled: false
url: "" # healthchecks.io, cronitor, dead man's switch
max_per_hour: 10 # alert emails/hour; CRITICAL always bypasses and is not counted. Code default 30; the shipped csm.yaml template sets 10
block_digest:
enabled: false # add per-country counts of auto-blocked IPs to the daily summary
countries: [] # empty = trusted countries, then all countries
interval: "1h" # how often counts fold into the summary; live alert dedup window; below 1m runs as 1m and warns
live: false # also send one alert per qualifying block
send_on: "any" # live alerts: any | customer
channel: "" # live alerts only: email | webhook; empty = enabled alert channels
min_block: 1 # watched blocks needed to show the summary section; 0 always shows it
audit_log: # SIEM-friendly per-finding stream
file:
enabled: false
path: /var/log/csm/audit.jsonl # default; logrotate fragment ships with the package
syslog:
enabled: false
network: udp # udp | tcp | unix | unixgram | tls
address: 127.0.0.1:514 # host:port, or filesystem path for unix variants
facility: local0 # default: local0
tls_ca: "" # optional CA cert for tls transport
# --- Integrity ---
integrity:
binary_hash: "" # populated by baseline/rehash
config_hash: "" # populated by baseline/rehash/reload
confd_hash: "" # populated by baseline/rehash/reload
immutable: true # apply chattr +i to the installed binary during install/rehash
# --- conf.d drop-in policy ---
confd:
integrity_exempt: [] # fragments an integration rewrites; left out of confd_hash (bare filenames)
# --- Thresholds ---
thresholds:
mail_queue_warn: 500 # default: 500
mail_queue_crit: 2000 # default: 2000
deep_scan_interval_min: 60 # minutes between deep scans (default: 60)
exposed_file_scan_depth: 2 # directory levels below each web docroot (default: 2, max: 10)
multi_ip_login_threshold: 3 # IPs per account before alert (default: 3)
multi_ip_login_window_min: 60 # time window for multi-IP check (default: 60)
cred_stuffing_distinct_accounts: 5 # failed accounts from one IP before credential_stuffing (default: 5)
pam_bruteforce_threshold: 5 # PAM auth failures from one IP before pam_bruteforce + auto-block (default: 5)
pam_bruteforce_window_min: 10 # window for counting PAM failures per IP (default: 10)
plugin_check_interval_min: 1440 # WordPress plugin check interval (default: 1440)
brute_force_window: 5000 # failed auth attempts window (default: 5000)
domlog_max_files: 500 # per-domain access logs per WP brute-force scan (default: 500)
domlog_tail_lines: 500 # trailing lines tailed from each domlog per scan (default: 500)
domlog_max_age_min: 30 # skip per-domain access logs untouched in this many minutes (default: 30)
mail_log_tail_lines: 500 # trailing lines of /var/log/exim_mainlog read by the mail-per-account scanner (default: 500)
syslog_messages_tail_lines: 200 # legacy direct-check FTP tail only; daemon FTP detection follows the log forward and ignores this setting (default: 200)
ftp_fail_window_min: 30 # sliding window (minutes) for per-IP pure-ftpd auth-failure accumulation before ftp_bruteforce; reported when a scan reads a new failure; diagnostic scans do not consume failures (default: 30)
account_scan_max_files: 10000 # account and mail-domain paths per scanner cycle (default: 10000)
# If this cap clips /home/<account>/ paths, account_scan_truncated names the affected account.
# Rolling content coverage spends this cap once per cycle across the whole host, not once per account.
crontab_base64_blob_max_bytes: 16384 # encoded bytes per crontab base64 candidate before decoded-content matching; must be a multiple of 4 (default: 16384)
# Full-scan subsystem (`csm scan --full`) and rolling content coverage.
full_scan_max_file_mb: 16 # cap on a single file scanned by a full scan, in MiB (default: 16)
scan_job_retention: 20 # completed full-scan job records kept in the store (default: 20)
rolling_coverage: true # tri-state; default on. Normal periodic cycles prioritize new and changed PHP source under discovered document roots, then scan a slice of dormant files. Both share the account_scan_max_files budget; unfinished work resumes across cycles and accounts take turns. Set false to disable
# Cursor write failures keep progress in memory until storage recovers; a restart loses that unpersisted progress.
# Changed scans retry incomplete reads and directory listings, preserving earlier findings until coverage succeeds. Clock corrections rewind their progress when needed.
# Rolling windows also preserve earlier findings when a content read fails. Reboots retain changed-scan progress instead of queuing all old files again.
# Existing files on first discovery remain the rolling window's job. Coverage follows the same PHP extensions and depth limit as rolling scans.
# Periodic forced rescans skip rolling coverage and expire old clean-file records without resetting changed-scan progress.
dropper_detection: true # tri-state; default on. Real-time flag for a PHP/executable created under a web docroot and unlinked before the TTL probe. Set false to disable
dropper_unlink_ttl_sec: 300 # seconds a fresh docroot PHP/executable is tracked before the self-delete probe (default: 300, range 30-3600)
# HTTP request flood, User-Agent spoof, and distributed HTTP detection.
# These detectors scan the same per-vhost access-log stream as the WP
# brute-force scanner; no extra log tailer is needed.
#
# http_flood_threshold: minimum per-IP request count inside the window
# that emits http_request_flood. 0 disables the detector. The detector
# ships disabled so operators can sample local baseline traffic first.
# Adjust up for CDNs or CGNAT-heavy visitor pools before enabling.
http_flood_threshold: 0 # 0 = disabled; set after sampling baseline traffic
http_flood_window_min: 5 # rate window in minutes (default: 5)
# http_ua_spoof_threshold: per-IP per-window count for non-browser UA
# kinds (including claimed search-engine bots such as Googlebot/Bingbot/
# Applebot that fail reverse-DNS confirmation) before http_ua_spoof fires.
# A single bot-like request from a residential or mobile client no longer
# hard-blocks it; sustained activity is required. Raise this for visitor
# pools that legitimately send unusual User-Agents.
http_ua_spoof_threshold: 30 # default: 30
# xmlrpc_threshold: per-IP POST /xmlrpc.php count before xmlrpc_abuse fires
# in access-log based detectors (a hard auto-block). WordPress sites using
# Jetpack, the mobile app, or WooCommerce make many legitimate xmlrpc.php
# calls, so this defaults higher than the legacy value. Set 0 to disable the
# check entirely; an absent key uses the built-in default.
xmlrpc_threshold: 100 # default: 100; 0 = disabled
# http_distributed_min_ips: distinct already-abusive source IPs that hit
# the same vhost in one scan window before a per-vhost distributed flood
# finding fires. 0 disables the rollup for existing configs that do not
# opt in.
http_distributed_min_ips: 10 # sample setting; omit or set 0 to disable
# URL scanner profile: fires http_scanner_profile for source IPs whose
# in-window traffic is almost entirely probe-error responses spread
# across many distinct request paths -- the shape of random-URL
# enumeration hunting for downloadable files, exposed backups, and
# dormant shells. Three gates must all pass: a minimum request volume,
# a minimum error percentage, and a minimum count of distinct error
# paths (query strings stripped, so cache-buster URLs on one missing
# endpoint count once). A visitor following dead links, a broken image
# hammered by one page, and a site migration all stay out of scope.
#
# 301 is deliberately not in the default status set: http->https and
# www redirects make every legitimate visitor 301-heavy, and a site
# migration redirects entire domains. Add 301 to
# http_scanner_status_codes only on hosts where that traffic shape is
# impossible.
http_scanner_min_requests: 0 # volume gate; 0 = disabled (default); 30 is a safe start
http_scanner_error_pct: 90 # min % of requests with probe-error status (default: 90)
http_scanner_min_distinct_paths: 10 # min distinct error paths, max 512 (default: 10)
http_scanner_status_codes: [404, 403] # statuses counted as probe errors (default: 404, 403)
# Distributed expensive-request crawl from one ASN. The detector fires
# only when request breadth, uncacheable volume, account share, and PHP
# worker saturation agree. Set min_ips to 0 to disable.
http_asn_crawl_window_min: 60
http_asn_crawl_min_ips: 25
http_asn_crawl_min_expensive: 250
http_asn_crawl_min_share_pct: 50
http_asn_crawl_high_amp_pct: 50
http_asn_crawl_high_volume_mult: 4
http_asn_crawl_saturation: 0 # 0 = performance.php_process_warn_per_user
http_asn_crawl_max_prefix: 8
http_asn_crawl_16_pref_pct: 60
http_asn_crawl_max_tracked_ips: 20000
http_asn_crawl_allowlist_asns: []
http_asn_crawl_reverse_proxy_asns: [13335, 54113, 20940]
# These three opt-in flags extend UA spoof detection to additional UA
# classes. Leave disabled on busy shared hosts; scripting-language agents
# and headless browsers appear on many legitimate monitoring stacks.
http_ua_scripting_enabled: false # flag curl/wget/python-requests/Go-http style UAs
http_ua_headless_enabled: false # flag Puppeteer/Playwright/PhantomJS UAs
http_ua_empty_enabled: false # flag requests with no UA at all
# SMTP brute-force tracker (Exim mainlog, dovecot SASL on submission ports)
smtp_bruteforce_threshold: 5 # per-IP failed auths before block (default: 5)
smtp_bruteforce_window_min: 10 # sliding window in minutes (default: 10)
smtp_bruteforce_suppress_min: 60 # cooldown between repeat findings (default: 60)
smtp_bruteforce_subnet_threshold: 8 # unique IPs per /24 before subnet block (default: 8)
smtp_account_spray_threshold: 12 # unique IPs targeting one mailbox before visibility finding (default: 12)
smtp_bruteforce_max_tracked: 20000 # soft cap on tracked entries; oldest evicted (default: 20000)
# Slow-brute pair: failed auths from one IP across 3+ distinct mailboxes
# within the long window. Catches attackers pacing below the fast window.
# Also fires on breadth alone: 10+ distinct walked mailboxes trigger the
# block regardless of the failure count. A successful SMTP auth from the
# source inside the window disqualifies it (live office NAT, not a walker).
smtp_bruteforce_slow_threshold: 40 # failures before block (default: 40; 3-1024; explicit 0 disables)
smtp_bruteforce_slow_window_min: 360 # slow sliding window in minutes (default: 360; 1-10080)
# SMTP probe-abuse tracker (raw connect-rate per IP; catches scanners that
# never reach AUTH). Threshold sized well above any legitimate MUA usage.
smtp_probe_threshold: 100 # per-IP connects before block (default: 100; explicit 0 disables)
smtp_probe_window_min: 5 # sliding window in minutes (default: 5)
smtp_probe_suppress_min: 60 # cooldown between repeat findings (default: 60)
smtp_probe_max_tracked: 20000 # soft cap on tracked entries; oldest evicted (default: 20000)
# Mail brute-force tracker (IMAP/POP3/ManageSieve via mail_logs source)
mail_bruteforce_threshold: 5 # per-IP failed auths before block (default: 5)
mail_bruteforce_window_min: 10 # sliding window in minutes (default: 10)
mail_bruteforce_suppress_min: 60 # cooldown between repeat findings (default: 60)
mail_bruteforce_subnet_threshold: 8 # unique IPs per /24 before subnet block (default: 8)
mail_account_spray_threshold: 12 # unique IPs targeting one mailbox before visibility finding (default: 12)
mail_bruteforce_max_tracked: 20000 # soft cap on tracked entries; oldest evicted (default: 20000)
# Slow-brute pair, as above. A recent successful login from the source or
# failures confined to mailboxes it holds established good standing for
# keep the source on the advisory path instead of blocking. Also fires on
# breadth alone: 10+ distinct walked mailboxes trigger regardless of count.
mail_bruteforce_slow_threshold: 40 # failures before block (default: 40; 3-1024; explicit 0 disables)
mail_bruteforce_slow_window_min: 360 # slow sliding window in minutes (default: 360; 1-10080)
mail_brute_account_key: "builtin:dovecot-user" # builtin:dovecot-user | builtin:postfix-sasl | regex:<capture>
# Mail login spray rules (SMTP and IMAP/POP3): long memory of failed and
# clean logins per IP. A clean login earns the IP trust on that mailbox.
# Enforce mode blocks last far longer than other blocks, so a busy host
# holds many at once: size firewall.deny_temp_ip_limit before enabling it.
mail_spray_mode: observe # off | observe (findings only) | enforce (block) (default: observe)
mail_spray_window_min: 10080 # memory of failed and clean logins in minutes (default: 10080; 60-10080)
mail_spray_breadth_mailboxes: 2 # finding once one IP (an IPv6 /64) has failed on this many untrusted mailboxes, the current one included (default: 2; 2-10; explicit 0 disables)
mail_spray_missing_networks: 3 # finding once this many networks have tried one non-existent mailbox, the current one included (default: 3; 1-50; explicit 0 disables)
mail_spray_block_expiry: "720h" # how long an enforce mode spray block lasts; other blocks keep block_expiry (default: 720h; 24h-2160h)
modsec_escalation_hits: 3 # denies from one IP before ModSecurity escalation (default: 3)
modsec_escalation_window_min: 10 # ModSecurity escalation window in minutes (default: 10)
modsec_low_confidence_escalation_hits: 30 # low-confidence-only backstop (default: 30)
# --- Web server overrides ---
# Leave these empty to use auto-detected paths for the running platform.
web_server:
type: "" # apache | nginx | litespeed; empty = detect
config_dir: "" # optional Apache/Nginx config root
access_logs: [] # candidate access logs, replacing detected paths
error_logs: [] # candidate error logs, replacing detected paths
modsec_audit_logs: [] # candidate ModSecurity audit logs
# Override the per-vhost access-log glob patterns. Empty uses the
# auto-detected default for the panel (cPanel, Plesk, DirectAdmin,
# bare Apache, or bare Nginx).
domlog_globs: []
# IPs or CIDRs trusted by periodic access-log scans and the challenge server
# to append or replace X-Forwarded-For with their observed peer. Other
# HTTP readers protect these addresses without reading that header.
# Automatic single-address responses never challenge or block a listed
# address; queued blocks are discarded even while blocking is deferred.
# This protection does not cover subnet blocks or manual operator blocks.
# challenge.trusted_proxies entries count too. Addresses of this host
# are never trusted. CSM skips an empty or unreadable entry and never
# trusts one covering every address; csm validate warns about both,
# and the daemon raises a High finding until a config without them runs.
# Restart after correcting the lists; a reload keeps the running lists.
# Legacy null or non-string scalar entries do not prevent startup,
# reload or plain csm doctor; usable entries beside them still work.
# Leave both lists empty to use the direct peer.
trusted_proxies: []
# --- Infrastructure ---
infra_ips: [] # management IPs/CIDRs/hostnames - never blocked
# --- Mail Logs ---
# Packaged releases include journald support. Custom builds need
# `make JOURNAL=1 build-yara` before `source: journal` can be selected.
mail_logs:
source: auto # auto | file | journal
file: "" # optional path override for file source
units: ["postfix", "postfix@-", "dovecot"] # journal units for source=journal or auto fallback; postfix@- is the Debian/Ubuntu instance
# --- Stalwart ---
# Read only on hosts where the stalwart service is the mail server.
stalwart:
log_dir: /var/log/stalwart # where Stalwart writes stalwart.log.YYYY-MM-DD
log_prefix: stalwart.log # file name before the date suffix
ports: [] # non-standard listeners as PORT:PROTOCOL (smtp, imap, pop3, sieve, http), e.g. ["10465:smtp"]
api_url: http://127.0.0.1:8080 # Stalwart admin API for csm stalwart setup and csm doctor; https unless on this host; file-only. Supply the admin login when running setup
api_key_file: /etc/csm/stalwart-api.key # API key CSM uses to act on Stalwart; a regular file owned by root with no group or other access (0600); file-only
# --- State ---
state_path: "/var/lib/csm/state" # bbolt DB and state files
# --- Suppressions ---
suppressions:
upcp_window_start: "00:30" # cPanel nightly update window start
upcp_window_end: "02:00" # cPanel nightly update window end
known_api_tokens: [] # API tokens to ignore in auth logs (e.g. ["phclient"])
ignore_paths: [] # glob patterns to skip in file scans; keep empty, an exempt directory is where an attacker hides
suppress_webmail_alerts: true # don't alert on webmail logins
suppress_cpanel_login_alerts: false # don't alert on cPanel direct logins
trusted_countries: ["RO"] # requires a loaded GeoLite2-City database; download credentials alone do not supply country data
# --- Auto-Response ---
auto_response:
enabled: false
kill_processes: false # kill malicious processes
quarantine_files: false # move malware to quarantine
max_file_actions_per_hour: 50 # shared quarantine and file-cleaning attempt budget
max_file_actions_per_account_per_hour: 10 # per-account share of the same rolling hour
max_file_action_failures_per_hour: 3 # pause automatic file response after repeated failures
block_ips: false # block attacker IPs via firewall
block_expiry: "24h" # positive temporary block duration; omit for the 24h default; admission caps new temporary lifetimes; retries keep their first expiry
http_asn_crawl_tempban: "24h" # Critical ASN-crawl subnet ban duration
# Shared scan-driven block budget; 0/omitted selects 50.
# Also sets the admission ledger ceiling: 0/omitted selects 2000, capped at 20000.
# A value sets both, so leave it out unless the ceiling should be lower too.
# Outside dry run, with the firewall on, the ledger caps the IP blocks it applies.
# See Automatic response admission in the API reference.
# max_blocks_per_hour: 0
enforce_permissions: false # auto-chmod 644 world/group-writable PHP files
fix_wp_cron: false # on perf_wp_cron findings, auto-disable WP-Cron and install a per-user system cron
http_scanner_action: "challenge" # response for http_scanner_profile: "challenge" (default) routes to the PoW page, "block" bans the IP
block_cpanel_logins: false # block IPs on cPanel/webmail/FTP/API thresholded brute findings (multi-IP login, webmail/API brute, FTP brute). Single direct cPanel form logins stay audit-only regardless of this flag.
netblock: false # auto-block IPv4 /24 or IPv6 /64 subnets
netblock_threshold: 3 # IPs from same IPv4 /24 or IPv6 /64 before subnet block; minimum 2, omit for the default
netblock_window: "168h" # blocked IPs count this far back, expired blocks included; omit for the 168h default
permblock: false # promote temp blocks to permanent
permblock_count: 4 # temp blocks before promotion; minimum 2, omit for the default
permblock_interval: "24h" # positive counting window; omit for the 24h default
clean_database: false # auto-drop confirmed malicious DB objects after backup
clean_htaccess: false # auto-clean .htaccess directives flagged by hardened detectors (backups under /opt/csm/quarantine/pre_clean/)
virtual_patch_exposed_files: "off" # off, manual CLI apply, or dry-run-gated auto apply except sample SQL
disable_enforce_af_alg: false # suspend periodic AF_ALG hardening re-assertion
copy_fail_kill_process: false # kill processes caught opening AF_ALG sockets via the live listener
mail_auth_recovery:
restart_enabled: false # opt-in restart after sustained cPanel mail auth backend outage
down_grace: "10m" # continuously-down duration before restart
max_restarts_per_hour: 3 # hourly restart-attempt cap
restart_command: "/usr/local/cpanel/scripts/restartsrv_dovecot"
dry_run: true # safe default; previews IP blocks and web-exposed-file virtual patches
verdict_callback:
enabled: false # call panel before each auto-block
url: "" # POST target for verdict requests
hmac_secret: "" # signing secret, or use hmac_secret_env
hmac_secret_env: "" # env var read at call time
allow_unsigned: false # true only for staged unsigned rollouts
require_response_signature: true # reject unsigned callback replies
timeout_sec: 2 # callback request timeout
# PHP-relay auto-freeze. Off by default; only kicks in on cPanel hosts
# where email_protection.php_relay.enabled is true. dry_run defaults to
# true even when freeze is true, so an operator who enables freeze
# without thinking gets a dry-run rather than a live exim -Mf storm.
# Override at runtime with `csm phprelay dry-run on|off|reset`.
php_relay:
freeze: false # opt in to wire the exim -Mf hook into the alert pipeline
dry_run: true # safe default; flip with `csm phprelay dry-run off [--persist]`
max_actions_per_minute: 60 # rolling 60s cap on exim -Mf invocations
# --- Detection ---
detection:
# Known-vulnerable WordPress plugin matching is default-on and alert-only.
# Omit for the default, or set false to disable it. Allow entries suppress
# only the exact reviewed slug and installed version.
# vulnerable_plugin_scanning: true
vulnerable_plugin_allow: [] # entries: <slug>@<version>, case-insensitive
# db_object_scanning is tri-state: omit for the default (on),
# `false` to explicitly disable. When off, the MySQL persistence
# scanner emits no findings; the manual `csm db-clean --drop-object`
# CLI keeps working for operator-driven cleanup.
# db_object_scanning: true
db_object_allowlist: [] # entries: <account>:<schema>:<type>:<name> -- suppresses db_unexpected_* warnings only
admin_overlap_min_accounts: 2 # raise only if routine shared-admin accounts are expected on this host
admin_overlap_trusted_emails: [] # exact reviewed admin emails that may manage multiple cPanel accounts
admin_overlap_trusted_domains: [] # exact reviewed email domains for developer or reseller admin accounts
# rescan_on_signature_update: true # tri-state; omit for default-on, false pauses queued rescans
af_alg_backend: "auto" # auto | bpf | auditd | none
connection_tracker_backend: "auto" # auto | bpf | legacy | none
connection_poll_interval: 30s # legacy connection tracker interval
exec_monitor_backend: "auto" # auto | bpf | legacy | none
exec_monitor_poll_interval: 30m # legacy process monitor interval
sensitive_files_backend: "auto" # auto | bpf | legacy | none
sensitive_files_poll_interval: 5m # sensitive-file poll/watchset refresh interval
placement_backend: "auto" # auto | bpf | legacy | none; files renamed or hard-linked into a site; restart required
direct_smtp_egress:
enabled: false # detect non-MTA local processes opening outbound SMTP
backend: "auto" # auto | bpf | legacy | none
dry_run: true # safe default for detector-scoped action
ports: [25, 465, 587] # destination ports to inspect
bad_asn_outbound:
enabled: false # off by default; third leg of the host_takeover chain. Needs the GeoLite2-ASN database and operator-supplied ASN lists
blocked_asns: [] # ASNs always treated as bad (e.g. known bulletproof hosters)
allowed_asns: [] # non-empty switches to allowlist mode: any destination ASN outside this set is treated as bad
# --- BPF Enforcement ---
bpf_enforcement:
enabled: false # master switch for in-kernel denial
dry_run: true # log intended denials, allow the connect
direct_smtp_egress: false # gate enforcement on direct SMTP egress matches
verdict_callback: false # userspace advisory callback after the BPF decision
# --- Challenge Pages ---
challenge:
enabled: false # enable PoW challenge pages instead of hard block
listen_addr: 127.0.0.1 # bind address; 0.0.0.0 for direct redirects, unix:/run/csm/challenge.sock for a proxy on this host
listen_port: 8439 # TCP port; ignored when listen_addr uses unix:
listen_group: "" # group allowed to open the unix: socket; the group your proxy runs as
tls_cert: "" # optional HTTPS cert for direct/public challenge listener
tls_key: "" # optional HTTPS key for direct/public challenge listener
public_url: "" # required by webserver-integration, e.g. https://host:8439/challenge
secret: "" # HMAC secret for tokens (auto-generated if empty)
difficulty: 2 # SHA-256 proof-of-work difficulty 0-5 (default: 2)
trusted_proxies: [] # proxies in front of this listener; HTTP checks trust them too
port_gate:
enabled: false # nftables gate for non-loopback challenge listener
captcha_fallback: # widget for JS-disabled visitors (default off)
provider: "" # "turnstile" | "hcaptcha" | "" (off)
site_key: "" # public key embedded in the widget
secret_key: "" # verified server-side
timeout: 10s
verified_session: # signed-cookie bypass for authenticated operators
enabled: false
cookie_name: csm_admin_session
ttl: 4h
admin_secret: "" # POST'd to /challenge/admin-token to mint cookie
verified_crawlers: # reverse-DNS forward-confirm for search crawlers
enabled: false
providers: [] # names: googlebot | bingbot
cache_ttl: 15m
# --- PHP Shield ---
php_shield:
# On CloudLinux, PHP runs inside a CageFS cage that cannot see the Shield
# event socket. Installing or upgrading the Shield registers its root-owned,
# non-writable directory in /etc/cagefs/cagefs.mp. Existing cages must be
# remounted during a maintenance window: cagefsctl --remount-all recreates
# every LVE on the host and disrupts running cages. A conflicting mount entry
# is left unchanged.
# csm doctor checks the event mount in running tenant cages. CSM's own
# worker services are not cages and are left out of that check.
# Rewritten command-parameter probes are quieted only when the Shield's
# source fingerprint matches CSM's verified CMS content cache. Install the
# updated Shield with csm install --php-shield-only; older events, unverified or
# modified scripts, and incomplete request paths still alert. All recognized
# observations remain in the root-only event archive, including quiet probes.
# This does not verify code included by the entry script; content scanning
# and runtime protection remain necessary.
# A failed eval() is High, except when the reported eval() call sits in a
# root-owned file that no account can change, such as the wp-cli bundled with
# WP Toolkit running "wp eval". That file and every directory above it must be
# root-owned, not symlinks, and not group or world writable when the event
# arrives; such a failure is Warning. Tenants can already disable the
# Shield in their own PHP processes. Every event field can be forged, so
# this verifies only the reported path's ownership and permissions, not
# the sender or evaluated code. A web request reaching such an eval() is
# still reported at Warning. An ownership check that fails or times out
# keeps the event at High, and a hung check does not delay later events.
enabled: false # receive PHP Shield events and emit alerts
# --- Reputation ---
reputation:
abuseipdb_key: "" # AbuseIPDB API key for IP reputation lookups
whitelist: [] # IPs to never flag as malicious
# Async PTR + forward-A verification for IPs that claim search-engine
# bot UAs (Googlebot, Bingbot, Applebot). When an IP claims a bot UA
# but reverse DNS does not confirm it, the request counts toward
# http_ua_spoof. Transient DNS lookup failures fail open and are
# retried later. Set false only if your resolver is unreliable. See
# docs/src/auto-response.md for the always-block behavior.
bot_verify_enabled: true # default: true
verified_bots: [] # optional custom crawler identities
# verified_bots:
# - name: "seranking" # rDNS-verified crawler
# ua_substrings: ["serankingbacklinksbot"]
# rdns_suffixes: ["seranking.com"]
# - name: "perplexitybot" # AI agent verified by published IP ranges
# ua_substrings: ["perplexitybot"]
# ip_ranges: ["18.97.9.96/29", "18.97.1.228/30"]
bot_ranges: # built-in AI-crawler range refresh
auto_update: true # default: true; restart required to change
update_interval: "24h" # default: 24h, minimum: 1h
rspamd:
enabled: false # include rspamd rolling history in IP reputation
url: "http://127.0.0.1:11334" # rspamd controller URL
token: "" # controller password, or use token_env; not needed for a loopback url
token_env: "" # env var read at query time
upstream:
enabled: false # include panel-side threat-intel cache scores
url: "" # HTTPS base URL; HTTP only allowed for loopback
token: "" # bearer token, or use token_env
token_env: "" # env var read at query time
cache_ttl_min: 15 # local cache TTL for upstream scores
timeout_sec: 5 # upstream request timeout
report:
enabled: false # opt-in abuse report delivery; restart required
classes: [] # bruteforce | php_relay | credential_stuffing | bad_asn_egress | spam
spool_path: "" # default: <state_path>/abuse_reports.db
spool_max: 10000 # max queued reports, shared by all targets
targets:
- name: "" # stable target name
url: "" # HTTPS collector URL; HTTP only allowed for loopback
transport: "hmac" # hmac | ed25519 (the central abuse database accepts ed25519 only)
node_id: "" # sender node ID
key_id: "" # receiver key ID
key_env: "" # HMAC secret or Ed25519 private key env var
token_env: "" # optional bearer token env var for HMAC targets
central:
enabled: false # opt-in central scored-set consume; restart required
set_url: "" # HTTPS scored-set endpoint; HTTP only for loopback
pubkey_env: "" # env var with Ed25519 public key hex
auth_target: "" # report target whose node key signs downloads
refresh_interval: 6h # pull interval; default 6h
action: "challenge" # off | challenge | block_if_local_corroborated
block_threshold: 60 # score needed before local corroboration can block
# --- Signatures ---
signatures:
rules_dir: "/opt/csm/rules" # YAML signature rules directory
update_url: "" # remote URL to fetch rule updates
auto_update: false # auto-download rules on schedule
update_interval: "" # how often to check (e.g. "24h")
signing_key: "" # required for any remote rule update path; 64-char hex Ed25519 public key
allow_rule_count_decrease: false # permit a deliberate drop below half the installed YAML rule count
yara_forge:
enabled: false # auto-fetch YARA Forge community rules
tier: "core" # "core", "extended", "full" (default: "core")
update_interval: "168h" # how often to check for updates (default: weekly)
download_url: "" # signed ZIP URL/template; supports {tier} and {version}
disabled_rules: [] # rule names to switch off, in Forge and in the shipped rules
# yara_worker_enabled: true # omit for the default: YARA-X runs in locked-down worker units; `false` runs it inside the root daemon
# signatures.signing_key is mandatory whenever either signatures.update_url
# is set or signatures.yara_forge.enabled is true. It must be the hex
# Ed25519 public key used to verify detached .sig files for rule bundles.
# Remote update URLs must use HTTPS and must not point at localhost,
# loopback, link-local, unspecified, or RFC1918 / ULA private addresses.
#
# YARA Forge upstream GitHub releases do not publish CSM detached signatures.
# To enable automatic Forge updates, mirror the ZIPs, sign each ZIP, publish
# the signature at the ZIP URL plus .sig, and set yara_forge.download_url to
# that signed mirror. Otherwise leave update_url empty and yara_forge.enabled
# false.
# --- Web UI ---
webui:
enabled: true
listen: "0.0.0.0:9443" # address:port for HTTPS server
auth_token: "" # API/login credential (auto-generated on install)
auth_token_issued_at: "" # optional UTC RFC3339 issue time; recorded on install
auth_token_expires_at: "" # optional UTC RFC3339 expiry; empty never expires
session_lifetime: "24h" # browser absolute expiry; maximum 720h; restart required
session_idle_timeout: "30m" # browser idle expiry; restart required
tokens: [] # scoped tokens: name/token/scope, optional issued_at, expires_at and allowed_ips (scope: admin, read or unban)
metrics_token: "" # optional Bearer token for /metrics only
metrics_token_issued_at: "" # optional UTC RFC3339 issue time; rotate with metrics_token, then SIGHUP
metrics_token_expires_at: "" # optional UTC RFC3339 expiry; empty never expires
tls_cert: "" # path to TLS certificate PEM file
tls_key: "" # path to TLS private key PEM file
ui_dir: "" # path to UI files on disk (default: /opt/csm/ui)
allowed_origins: [] # extra browser origins ("https://host[:port]") allowed to call the API; loopback is always allowed
# --- MCP (read-only endpoint for support agents) ---
mcp:
enabled: false # serves /mcp on the Web UI listener; needs webui.enabled: true; restart required
tokens: [] # name/token, optional issued_at, expires_at and allowed_ips; all read-only, never valid on the Web UI
# --- Email AV ---
email_av:
enabled: false
clamd_socket: "/var/run/clamd.scan/clamd.sock" # path to ClamAV daemon socket
scan_timeout: "30s" # per-attachment scan timeout
max_attachment_size: 26214400 # max single attachment size in bytes (25MB)
max_archive_depth: 1 # archive levels unpacked (1: attached archives only); a deeper archive counts as not fully scanned
max_archive_files: 50 # max files extracted per archive; also caps encrypted member names reported per message
max_extraction_size: 104857600 # max total extraction size in bytes (100MB)
quarantine_infected: false # true moves infected messages to quarantine; false delivers them with a Critical email_malware finding
scan_concurrency: 4 # parallel scan workers
fail_mode: "open" # behavior when a scan cannot complete: "open" (default) delivers; "tempfail" defers so Exim retries
# Permission events have a five-second scan hold budget. When it expires,
# the open is allowed ("open") or denied for retry ("tempfail"); an admitted
# scan continues. A full scan queue applies the same policy immediately.
# When messages repeatedly cannot start scanning within the hold budget, or
# find the scan queue full, CSM raises email_av_hold_bypass and stops
# queueing new scans for five minutes. A hold that runs out while the
# message is already being scanned does not count. During this cooldown new
# opens are allowed unscanned ("open") or deferred ("tempfail"). Scanning
# resumes automatically afterwards; bypassed messages are not scanned later.
# One sender's messages normally use at most half the scan_concurrency
# slots (at least one), leaving slots for others when concurrency exceeds
# one. The sender is the authenticated user, else the sending address (an
# IPv6 /64), else the local account. Incomplete sender fields are ignored
# when choosing the sender. Mail relayed through a filtering gateway,
# unauthenticated mail submitted over SMTP from localhost, and bounces Exim
# creates each count as one sender. Further messages wait without taking a slot;
# when their hold budget runs out they follow fail_mode and are still
# scanned afterwards. When too many messages from one sender are waiting,
# further ones are scanned at once without waiting their turn. This overflow
# can use every scan slot; email_av_source_backlog names the sender, at most
# once a minute.
# An archive that lists more than ten times max_archive_files entries, or
# whose decompressed data passes max_extraction_size, counts as not fully
# scanned and follows fail_mode. Nested archives share that budget. So do
# corrupt gzip trailers, data after a tar end marker, legacy sparse tar
# members and zip directory entries that carry data, at any nesting level.
# Empty members and directories remain valid.
# Late malware results can quarantine only the original, unlocked message.
# Messages in active delivery, replaced spool files, and messages with a
# pending delivery journal are left untouched with email_av_quarantine_error.
# email_av_late_verdict means an allowed open later received a scan decision
# to stop delivery; it does not mean a previously deferred message escaped.
# Password-protected archive attachments are outside fail_mode. Their members
# cannot be read without the password, so no retry ever makes them scannable
# and "tempfail" would defer the message until it bounced. CSM delivers them
# and reports email_av_encrypted_archive naming the archive and the member,
# at most once an hour. A full alert queue does not consume this allowance.
# Encrypted member names are capped at max_archive_files per message; the
# warning counts any additional encrypted members without deferring delivery.
# Both encryption schemes are covered:
# legacy ZipCrypto and WinZip AES. Blocking such mail outright is an Exim
# policy decision and is not something CSM does for you.
# Attachments are unpacked in locked-down worker processes, one per
# scan_concurrency slot, started on first use. When a worker cannot finish,
# the message counts as not fully scanned:
# worker deadline reached "parse worker timed out"
# worker memory cap reached "parse worker ran out of memory"
# worker crashed or exited "parse worker crashed"
# reply failed validation "parse worker reply rejected"
# worker unit cannot start "parse isolation unavailable"
# unit keeps failing to start "parse worker failing repeatedly"
# message failed twice before "parse worker failed on this message before"
# The first four and the last are caused by the message. Outside observe
# mode they request deferral whatever fail_mode says, so Exim retries it.
# Deferral requires a permission event that is still holding delivery;
# notification-only mode cannot block it. The two unit failures, and a unit
# still starting when the scan deadline passes, follow fail_mode.
# The first four raise email_av_parse_worker_failed, naming the message and
# its sender, at most once a minute; a rejected reply also raises a Critical
# email_av_parse_worker_rejected. The next two raise a Critical
# email_av_isolation_unavailable. Each Critical finding is raised at most
# once an hour. A message whose worker failed twice within a day is not
# unpacked again on later delivery attempts, including after a spool watcher
# restart. Failed starts, cancellation and local descriptor exhaustion do
# not count against messages or senders. When messages from one sender
# make workers fail three times, each within ten minutes of the one before,
# a High email_av_parse_worker_source names the sender, at most once every
# ten minutes. This advisory reports scan failures, not delivery outcomes.
# It never blocks automatically, and blocking an address does not hide it.
worker_memory_limit: 536870912 # memory and swap cap of each attachment worker in bytes (512 MiB); minimum 134217728 (128 MiB)
# Up to scan_concurrency attachment workers run at once, and each may use up
# to worker_memory_limit. A worker that reaches it is killed and the message
# is deferred as "parse worker ran out of memory"; raise the limit if
# ordinary mail reaches it. The phishing-kit zip worker keeps the default
# cap. Changing it takes effect after a daemon restart.
keep_crash_samples: false # keep copies of messages and tenant zips that crash an attachment worker
# Crash samples are for diagnosing worker crashes. They hold customer mail
# and tenant files: they are readable by root only, stored in parse-crashes
# under the state directory, capped at 20 samples and 500 MiB, and deleted
# after 7 days, also after this setting is turned off. A link in place of
# the parse-crashes directory is refused. Failed or incomplete copies are
# discarded without changing the scan verdict or findings.
# --- Email Protection ---
email_protection:
password_check_interval_min: 1440 # how often to audit email passwords (default: 1440)
high_volume_senders: [] # accounts expected to send high volume (skip rate alerts)
rate_warn_threshold: 50 # emails per window before warning (default: 50)
rate_crit_threshold: 100 # emails per window before critical (default: 100)
rate_window_min: 10 # rate check window in minutes (default: 10)
known_forwarders: [] # expected forwarders and Sieve :copy rules
# PHP-relay detector (cPanel only; gated by platform.IsCPanel at startup).
# Off by default. When enabled, the daemon spawns the inotify spool
# watcher, runs a startup spool walk, and starts the Path 2b retro scan
# on /var/log/exim_mainlog. See docs/src/detection-realtime.md#php-relay
# for what each path actually triggers on.
php_relay:
enabled: false # opt in to start the watcher
rate_window_min: 5 # Path 1 rolling window
header_score_volume_min: 5 # Path 1: don't score until script has emitted N msgs
absolute_volume_per_hour: 30 # Path 2 threshold per script
account_volume_per_hour: 0 # Path 2b operator override; 0 = auto-derive from cpanel.config maxemailsperhour
fanout_distinct_scripts: 3 # Path 4 threshold
fanout_distinct_recipients: 5 # Path 2 + Path 4 recipient-diversity gate; 0 disables only this gate
fanout_window_min: 5 # Path 4 window
ip_recipient_spread: 5 # Path 6 threshold: distinct outside recipients per client IP (1-100); whole number; 0 disables Path 6
ip_recipient_window_hours: 24 # Path 6 window in hours (1-168); whole number; 0 uses the default (24)
ip_recipient_mode: observe # Path 6 response: observe (findings only, default) or enforce (challenge, then block)
policies_dir: "/opt/csm/policies/php_relay" # optional mailer_classes.yaml + http_proxy_ranges.yaml (replaces the built-in proxy list); SIGHUP-reloadable
cloud_relay:
allow_users: [] # full mailbox opt-outs for cloud-relay detection
allow_domains: [] # domain-wide opt-outs for cloud-relay detection
# Email forward guard (cPanel only). Opt-in MTA-native enforcement for
# external forward copies. Enforce mode can hold null-sender backscatter,
# bad-sender-IP and spam-flagged copies before they relay to an external
# provider, while the local mailbox copy still delivers. Spam-flagged means
# the message carries X-Spam-Status: Yes, as antispam adds when it tags.
# Any X-Spam-Status field with a Yes verdict counts, ignoring letter case.
# Other text does not count as a spam verdict.
# Malware and auth-fail signals are accounted in dry-run until Exim content
# scanning is wired. CSM is not in the live mail path; an installed Exim rule
# can keep holding matching copies even if the daemon is down. Held copies can
# be released or deleted from the Email page.
forward_guard:
enabled: false # master switch (default off)
dry_run: true # account/log only, do not actually hold (default true)
quarantine_retention_days: 14 # held copies older than this are deleted; 0 keeps them
hold_signals: # signal toggles, each default true
bounce_backscatter: true # null-sender bounce backscatter (enforceable)
spam_flagged: true # message carries X-Spam-Status: Yes (enforceable)
malware: true # message carries malware (dry-run/accounting only)
bad_sender_ip: true # originating IP has bad reputation (enforceable)
auth_fail: true # sender failed SPF/DKIM/DMARC auth (dry-run/accounting only)
# Inbound spam filtering through rspamd on cPanel and Postfix hosts. Opt-in.
# The operator installs rspamd and adds the scan socket. On cPanel CSM writes
# the Exim hooks and its own rspamd settings, and each account's Spam Filters
# switch decides whether its mail is filtered. On Postfix CSM writes only
# rspamd files, every local domain is filtered and `exempt` lists the
# exceptions. See docs/src/antispam.md.
antispam:
enabled: false # master switch (default off)
dry_run: true # scan and log only; no reject, tag or split (default true)
rspamd_socket: /run/rspamd/csm-normal.sock # rspamd scan socket, not the reputation.rspamd controller
reject_score: 15 # reject at or above this rspamd score (default 15)
tag_score: 6 # add X-Spam-Status: Yes at or above this score (default 6)
feeds:
enabled: [] # optional feeds, e.g. [mailspike, psbl]; see docs/src/antispam.md
spamhaus_dqs_key_file: "" # root-owned file holding the Spamhaus DQS key; needed only for spamhaus_dqs
learning:
enabled: false # learn from mail users move to Spam (cPanel) or Junk (Postfix) and keep one message per cluster; needs antispam; see docs/src/antispam.md
redis_socket: "" # Unix socket of a Redis used only by learning, e.g. /run/redis-csm/redis.sock; Postfix needs one, cPanel uses rspamd's Redis when empty
reporter: domain # Postfix only: one learning vote per mail domain (domain) or per mailbox (mailbox)
exempt: [] # Postfix only: recipient domains and mailboxes never filtered or learned from, e.g. [example.com, postmaster@example.net]
# --- Firewall ---
firewall:
enabled: false
# Open ports (IPv4). SSH (22) is intentionally absent; uncomment in
# the YAML lists if sshd listens on 22. TCP 853 is DNS-over-TLS;
# UDP 853 is DNS-over-QUIC.
# 6277/24441 are DCC/Pyzor network checks used by SpamAssassin.
tcp_in: [20,21,25,26,53,80,110,143,443,465,587,853,993,995,2077,2078,2079,2080,2082,2083,2091,2095,2096]
tcp_out: [20,21,25,26,37,43,53,80,110,113,443,465,587,853,873,993,995,2082,2083,2086,2087,2089,2195,2325,2703]
udp_in: [53,443,853]
udp_out: [53,113,123,443,853,873,6277,24441]
# IPv6
ipv6: false
tcp6_in: [] # if empty, uses tcp_in
tcp6_out: [] # if empty, uses tcp_out
udp6_in: [] # if empty, uses udp_in
udp6_out: [] # if empty, uses udp_out
# Restricted ports (infra IPs only)
restricted_tcp: [2086,2087,2325,9443] # WHM and CSM Web UI ports
# One TCP port opened to listed sources only. See firewall.tcp_in_allow
# below.
tcp_in_allow: []
# tcp_in_allow:
# - port: 3306
# sources: [203.0.113.10, 198.51.100.0/28]
# Outbound ports a service on this host needs. Checked, never added to
# the policy: validation warns when an effective family policy omits one.
# Integrations declare theirs in their own conf.d fragment.
required_tcp_out: []
# Passive FTP data ports. Left unset, 49152-65534 is opened on cPanel hosts
# and hosts with pure-ftpd, proftpd or vsftpd installed, and stays closed
# elsewhere. Explicit values take precedence; set both to 0 to close it.
# passive_ftp_start: 49152
# passive_ftp_end: 65534
# Destination-scoped outbound TCP. tcp_out holds single ports only, so a
# port range can only be expressed here -- needed when this host is an FTP
# *client* and must open passive data connections to a known source.
# dst takes an IP or CIDR. 0.0.0.0/0 means any IPv4 destination; ::/0 means
# any IPv6 destination. Each rule applies to its own address family, and
# validation warns: a wide range to anywhere is the outbound path the
# output chain exists to close. When smtp_block is on, these rules follow
# its guard and ranges that overlap smtp_ports are rejected.
# An IPv6 destination emits no rule unless firewall.ipv6 is true.
# IPv4-mapped prefixes under ::ffff:0:0/96 are treated as IPv4 prefixes.
# Lockout warnings only credit valid rules in the connection's family;
# scoped exceptions still warn because hostname resolution is not checked.
tcp_out_allow: []
# tcp_out_allow:
# - dst: 203.0.113.10/32
# port_start: 49152
# port_end: 65534
# Peer-scoped protocols without ports, such as VRRP between the members of
# a high-availability pair. See firewall.protocol_allow below.
protocol_allow: []
# protocol_allow:
# - protocol: vrrp
# peers: [192.0.2.11, 2001:db8::11]
# Infra IPs/CIDRs/hostnames for firewall rules
infra_ips: []
# Rate limiting. SYN/conn-rate/UDP are dual-stack (IPv6 keyed per /64);
# conn_limit is IPv4-only.
conn_rate_limit: 200 # new connections/min per source (CGNAT-tolerant; IPv6 per /64; 0 = disabled; null = default)
syn_flood_protection: true # per-source SYN flood meter (IPv6 per /64)
conn_limit: 400 # max concurrent connections per IPv4 source, IPv4 only (0 = disabled)
drop_ipv4_options: true # drop inbound IPv4 TCP carrying IP options, ahead of every accept (CVE-2026-98432 trigger)
# Per-port flood protection: rate-limit new connections per source IP and IP family.
# Defaults are sized for a busy mail host: 600/300s = 120 new conns/min/IP,
# which tolerates a Thunderbird/iPhone client opening 5-15 parallel sessions
# while still capping single-IP flood storms.
port_flood:
- port: 25
proto: tcp
hits: 600
seconds: 300
- port: 465
proto: tcp
hits: 600
seconds: 300
- port: 587
proto: tcp
hits: 600
seconds: 300
# DoS-exempt ranges: bypass configured DoS meters and subnet auto-blocks.
# See firewall.dos_exempt_ranges below for full details.
dos_exempt_ranges: [] # CIDRs or single IPs exempt from connection/mail-port meters and subnet auto-blocks; /0 and bare hostnames rejected at load
dos_exempt_known_mail_providers: true # also exempt Google and Microsoft mail-provider egress ranges; dynamic, cached, updated every 12h (default: true)
# UDP flood protection (per-source meter; IPv6 per /64)
udp_flood: true
udp_flood_rate: 100 # packets per second
udp_flood_burst: 500 # burst allowance
# Country blocking
country_block: [] # ISO country codes to block
country_db_path: "" # country CIDR directory (default: <state_path>/geoip, filled by `csm firewall update-geoip`)
# Silent drop (no logging)
drop_nolog: [23,67,68,111,113,135,136,137,138,139,445,500,513,520]
# IP limits
deny_ip_limit: 3000 # max permanent blocked IPs
deny_temp_ip_limit: 500 # max temporary blocked IPs; 0 = unlimited for existing blocking; admission forecasts then plan for 10000; a negative value reads as 0 with a warning
# Outbound SMTP restriction
smtp_block: false # block outgoing mail except allowed users
smtp_allow_users: [cpanel, mailman] # extra outbound SMTP users, also exempt from direct SMTP egress detection and BPF enforcement; root and mailnull are always allowed
smtp_ports: [25,465,587]
# Local port locks: "PORT:user,user"; only these users and root may open
# new connections to the port on this host
local_port_locks: []
# Dynamic DNS
dyndns_hosts: [] # hostnames to resolve and whitelist periodically
# Logging
log_dropped: true # log dropped packets
log_rate: 5 # log entries per minute
# --- GeoIP ---
geoip:
account_id: "" # MaxMind account ID
license_key: "" # MaxMind license key
editions: # MaxMind database editions
- GeoLite2-City
- GeoLite2-ASN
auto_update: true # auto-update GeoIP databases (default: true when credentials set)
update_interval: "24h" # update check interval
# --- ModSecurity ---
modsec_error_log: "" # path to Apache/LiteSpeed error log for ModSec parsing
modsec:
rules_file: "" # path to modsec2.user.conf
overrides_file: "" # path to csm-overrides.conf
reload_command: "" # command to reload web server (e.g. "/usr/sbin/apachectl graceful"); also activates CSM rule updates
# --- Performance ---
performance:
enabled: true
load_high_multiplier: 1.0 # load average / CPU cores multiplier for warning (default: 1.0)
load_critical_multiplier: 2.0 # load average / CPU cores multiplier for critical (default: 2.0)
php_process_warn_per_user: 20 # per-user PHP process count warning (default: 20)
php_process_critical_total_multiplier: 5 # total PHP processes / CPU cores for critical (default: 5)
error_log_warn_size_mb: 50 # error log size warning threshold (default: 50)
error_log_critical_size_mb: 1024 # error log size that raises the finding to High (default: 1024)
mysql_join_buffer_max_mb: 64 # MySQL join_buffer_size warning threshold (default: 64)
mysql_wait_timeout_max: 3600 # MySQL wait_timeout warning threshold (default: 3600)
mysql_max_connections_per_user: 10 # per-user MySQL connections warning (default: 10)
redis_bgsave_min_interval: 900 # minimum seconds between Redis BGSAVE (default: 900)
redis_large_dataset_gb: 4 # Redis dataset size warning threshold in GB (default: 4)
wp_memory_limit_max_mb: 512 # WordPress memory_limit warning threshold (default: 512)
wp_transient_warn_mb: 1 # WordPress transient data warning in MB (default: 1)
wp_transient_critical_mb: 10 # WordPress transient data critical in MB (default: 10)
wp_cron_fix: # tuning for the WP-Cron remediation (manual fix from the Web UI or auto_response.fix_wp_cron)
interval_minutes: 15 # system cron frequency; default 15, clamped to [1, 60]
php_bin: "" # override interpreter; empty = per-vhost PHP (inherit: .htaccess handler), then auto-detect
# --- Cloudflare ---
cloudflare:
enabled: false # auto-whitelist Cloudflare IP ranges
refresh_hours: 6 # how often to refresh Cloudflare IPs (default: 6)
# --- Threat Intel ---
c2_blocklist: [] # known C2 server IPs; connections with them are blocked
backdoor_ports: [4444,5555,55553,55555,31337] # ports indicating backdoor activity
# --- Update check ---
updates:
check_enabled: true # notify only; CSM never downloads or applies updates
interval: "24h" # release check interval
github_api_url: "" # optional release API mirror or test endpoint
package_name: "csm" # apt/dnf package name for package-manager fallback
# --- Incidents ---
incidents:
auto_close:
enabled: true # auto-close idle open/contained incidents
dry_run: false # log decisions without writing status changes
by_kind:
mailbox_takeover: 24h
mailbox_bruteforce: 24h
credential_spray: 24h
web_attack: 24h
web_account_compromise: 168h
spray_suppression:
enabled: false # collapse one-source credential spray into one incident
dry_run: true
distinct_mailboxes: 10
severity_escalate_at: 50
per_check: [email_auth_failure_realtime, pam_bruteforce, credential_stuffing]
max_tracked_ips: 10000
block_at_severity: "" # "" | high | critical
auto_block:
enabled: false # block source IPs from incident correlations
block_at_severity: "" # "" | high | critical
kinds: [] # empty means all non-spray kinds with remote_ip
# --- Disabled checks (skip whole categories per host) ---
# Listed finding names disable the scheduled check runner(s) that emit them,
# including sibling findings from the same runner. Realtime findings are not
# affected. Use for whole categories that don't apply to a host (e.g. WAF/web
# checks on DNS-only cPanel servers, where httpd is installed but no virtual
# hosts serve traffic).
# For email-only suppression, use `alerts.email.disabled_checks` instead.
disabled_checks: [] # e.g. [waf_status, waf_rules, waf_detection_only]
# --- Retention (bbolt growth control) ---
retention:
enabled: false # opt-in; when true, a daily sweep prunes old entries
findings_days: 90 # keep active findings this long (0 disables the findings sweep)
history_days: 30 # keep findings-history entries and proven firewall
# outcomes this long; it also sets how far back a
# durable firewall action can be undone
reputation_days: 180 # keep IP reputation/attack entries this long
sweep_interval: "24h" # how often the retention goroutine runs
compact_min_size_mb: 128 # startup compaction floor; 0 disables auto-compaction
compact_fill_ratio: 0.5 # compact when used_bytes / file_size drops below this
# --- Debug / diagnostics ---
debug:
pprof_listen: "" # e.g. "127.0.0.1:6060"; MUST be loopback. Empty disables.
# Reach it over SSH: go tool pprof http://127.0.0.1:6060/debug/pprof/heap
# While listening, samples mutex and block contention; adds CPU overhead even without a profile request.
# --- Sentry (error reporting) ---
sentry:
enabled: false # ship panics and selected errors to a Sentry server
dsn: "" # Sentry project DSN
environment: "production" # e.g. "production", "staging"
sample_rate: 1.0 # 0.0 -> 1.0 (capture all errors)
debug: false # SDK debug logs to stderr
Alert classes
Every finding CSM reports is immediate or summary class. Summary class covers what CSM already handled on its own:
- an attack from an address CSM blocked, before or in the same batch, including by a subnet block or netblock;
- an attack a challenge answers, while the address is on the challenge list or was routed there in the same batch. A challenge does not answer FTP or mail authentication failures, critical reputation sightings, or checks configured to require a hard block;
- applied block and challenge notices, including permanent, subnet and netblock blocks, dry-run block verdicts, and the periodic count of applied automatic responses;
- mail login advisories that never block, such as spray observe mode;
- PHP relay freeze dry runs and rate-limit deferrals that need no follow-up.
Summary-class findings are left out of alert email and the generic alert webhook and are counted for the daily summary instead. They never spend the hourly alert budget. The findings page, history, incidents, the audit log, the event stream and the phpanel webhook still receive them. Only findings that would otherwise notify are counted: a suppression rule, or a check that never notifies, keeps its findings out of the summary too. Such a finding can still show that CSM blocked or challenged an address. If the summary fails to take its findings, the alert run reports an error and immediate alerts still go out.
Everything else is immediate. That includes attacks whose source CSM did not block: auto-response off or in dry run, a block that failed, was refused or is still queued, and checks that never block. Compromise evidence, successful logins, unauthenticated WHM script requests, suspicious mail content and findings that summarize several sources stay immediate even when the address is blocked, because blocking one address does not resolve them. A block of an address inside a Cloudflare allow range leaves ports 80 and 443 open, so its notice and the web attacks from that address stay immediate. Failed, paused or skipped automatic actions and auto-block queue and accounting notices are immediate too.
The generic webhook follows the same classes as email. alerts.email.disabled_checks
mutes email only: a muted immediate finding still goes to the webhook, and a
muted summary finding is still counted in the summary.
There is no setting for this policy, and the same classes decide what
csm scan --alert sends. The former suppressions.suppress_blocked_alerts key
is ignored; a config that still sets it loads, and csm validate warns so the
key can be deleted.
Daily summary
CSM sends one summary of the findings it handled each day at 08:00 server
local time. It goes by alert email when email alerts are on, and to the
generic alert webhook when that is on, as event daily_summary with the
counts under csm.summary (see the
API reference). A phpanel
webhook does not receive it.
alerts.block_digest.channel selects live block alerts only; the daily
summary uses every enabled alert channel. It goes out even with nothing to
report.
It covers the window since the previous summary:
- summary-class findings by check and severity, and how many of them come from checks muted by
alerts.email.disabled_checks; - blocks applied, by kind, including netblocks and promotions to a permanent block;
- challenges, and mail login advisories that never block, such as spray observe mode;
- the 10 addresses with the most summary findings, with their country when GeoIP knows it;
- automatic blocks that hit a known customer, as their immediate alert named them, so a failed alert still reaches you: up to 50 a day, then a count of further blocks;
- dry-run block verdicts and spray observe-mode verdicts that would have blocked a known customer, listed alongside their finding counts: up to 50 blocked addresses or networks a day, each listed once, then a count of further verdicts;
- with
alerts.block_digest.enabled, watched-country blocks by country and category, once they reachmin_block.
CSM tracks up to 1,000 source addresses a day. Past that, a new address takes the place of the one with the fewest findings, so heavy sources stay listed and a count can be overstated, never understated.
Counts are saved at each block_digest.interval (one hour by default), after
the final shutdown drain and before delivery. A crash can lose counts since
the last save. If the host was down at 08:00, the missed summary goes out at
the next start. A failed destination is retried after one minute, then with
the wait doubling up to one hour; a restart retries it at once. CSM logs its
first failure and again when the wait reaches an hour, and destinations
already saved as successful are not retried. While a delivery is pending, new
findings accumulate in the next window. When the next summary is due and a
destination still has not taken the previous one, CSM drops the previous one
for that destination, logs it once and counts it as one lost notification,
then sends the new summary to every destination. A summary whose state cannot
be saved is still sent; the save warning backs off the same way as a failing
destination.
A crash after a destination accepts a summary but before CSM saves that success can repeat it. Clock corrections are checked every minute. Moving back across one or more morning sends does not resend dates already covered, including after a restart. Sending resumes on the first unsent local date. A send recorded more than two days ahead of the clock comes from a clock fault: CSM logs it once, ignores it and sends the next morning.
Only the daemon counts findings. A one-off csm scan --alert prints every
finding; its summary-class findings are neither emailed nor counted in the
daily summary. The hourly block digest email no longer exists.
Customer login history records the source addresses of authenticated cPanel, webmail, SMTP, IMAP, POP and FTP logins for 30 days. Buffered logs keep their original login time, so reading old logs does not renew expired history. FTP logins retain their authenticated account even when the displayed log record is shortened. FTP command text and failed login names do not count as successful logins. An applied automatic block of a known customer is alerted at once; see the known-customer block alert.
An IPv6 customer is recorded by its /64, so a host that changes privacy addresses stays known. Each account holds a bounded number of addresses per log source; logins without an account name share one budget, and an FTP login counts against the hosting account that owns the FTP account.
History has two fixed halves: logins read from syslog (FTP, IMAP, POP and Postfix SMTP) and logins read from cPanel’s session log and the Exim mainlog. Any local user can write to syslog, so each half trims only itself, and a syslog login never replaces a session log or Exim login for the same address. When a half is full, the accounts holding the most addresses lose their oldest logins first, so a flood of logins evicts only its own accounts. History is saved every minute and after log readers stop during a clean shutdown.
Web UI Exposure
An admin token can change firewall rules, files under account homes and most
settings, so treat the Web UI as privileged access. The shipped
listen: "0.0.0.0:9443" answers on every interface, which the WHM addon needs.
csm validate and csm doctor warn when the UI listens on a non-loopback
address, an admin token has no allowed_ips or allows an entire address family,
and the CSM firewall policy does not restrict that family. These warnings do
not make either command fail.
Enabling the firewall alone does not clear the warning. Keep the UI port out
of public tcp_in and tcp6_in entries, or add it to restricted_tcp (which
includes 9443 by default). Keep it outside the passive FTP range. Wildcard
listeners can accept both address families, so also enable firewall.ipv6.
Except for localhost, hostnames are checked against both families without
resolving DNS.
Alternatively, limit admin tokens with allowed_ips, or listen on
127.0.0.1:9443 and reach the UI through an SSH tunnel. An external firewall
may already restrict access, but configuration validation cannot verify it.
TLS Certificates
The Web UI serves over HTTPS. Configure TLS certificates under webui:
webui:
tls_cert: "/var/cpanel/ssl/cpanel/mycpanel.pem" # certificate PEM file
tls_key: "/var/cpanel/ssl/cpanel/mycpanel.pem" # private key PEM file
On cPanel servers, you can reuse the cPanel self-signed certificate (both cert and key are in the same PEM file). For production, use a proper certificate from Let’s Encrypt or your CA.
The private key may appear before or after the certificates. The first certificate must be the leaf served by the Web UI, followed by any chain certificates. Renewal only replaces an expiring CSM-generated leaf; it preserves the private key and all other contents of a combined file. Certificates issued outside CSM are left unchanged.
If both paths are empty, CSM generates an ECDSA self-signed certificate and key under state_path. Set both paths to use an operator-managed certificate. Configuring only one of the pair is invalid.
Validation
csm validate # syntax check
csm validate --deep # syntax + connectivity probes (SMTP, webhooks)
csm config show # display config with secrets redacted
Editing csm.yaml by hand
CSM stores a sha256 of the main config in integrity.config_hash and
a separate digest of loaded drop-ins in integrity.confd_hash. It
refuses to start if the on-disk files disagree with those values. This
is a tamper-detection feature. It catches edits that were never
recorded, but it does not stop anyone who can write /etc/csm,
because they can record new hashes too. There are two supported edit
workflows depending on which fields you touch.
Fast path: SIGHUP reload (safe fields only)
For fields tagged as hot-reload-safe (alerts, thresholds,
detection, suppressions, auto_response, bpf_enforcement,
reputation, email_protection, disabled_checks, confd), the daemon can
accept the change without a restart:
sudo cp /etc/csm/csm.yaml /etc/csm/csm.yaml.bak-$(date +%s)
# edit /etc/csm/csm.yaml with your favourite editor
sudo systemctl reload csm
sudo journalctl -u csm -n 20 --no-pager
Every save CSM makes itself (the Settings page, csm rehash, the installer)
first copies the current file to csm.yaml.bak next to it, so the previous
version is one cp away. For a symlinked config, the backup sits beside the
resolved target. A first write creates no backup; an unchanged save keeps
the previous backup. Backups have mode 0600.
Retired settings are ignored with validation warnings, including settings in YAML merges and conf.d fragments. Removing them does not require a restart.
A reload after a comment edit or retired-setting removal updates the live
integrity hashes even when no setting changes. This includes edits to conf.d
fragments. If csm rehash already signed the edit, reload adopts the verified
hashes without rewriting the main config. A reload with no edits neither
rewrites the config nor republishes it, so it leaves dry-run history alone.
Signing failures emit a Critical config_reload_error finding and count as
reload errors.
systemctl reload sends SIGHUP (wired via ExecReload= in the unit
file). The daemon re-reads the file, validates it, diffs it against
the running config, and if every change is on a field tagged
hotreload:"safe" it swaps the new values into
the live config and re-records the integrity hashes on disk. The
next check tick sees the new thresholds; fanotify marks are not
dropped.
Reloads and Web UI saves share one config-write lock. If a reload signs the config but cannot open the phpanel delivery queue, it reports a reload error and keeps the previous settings and mail account parser active. The live integrity hashes still follow the signed config, so scheduled checks continue.
The tagged-safe top-level fields are alerts, thresholds,
detection, suppressions, auto_response, bpf_enforcement,
reputation, email_protection, disabled_checks, and confd. The Settings
API derives its restart hints from the same manifest that drives
config.Diff, so UI hints and SIGHUP behavior cannot drift silently. A
save from the Settings page runs the same apply steps as a SIGHUP, so a
changed threshold, reputation whitelist, verified-bot list, forward-guard
setting or response ceiling reaches the live daemon at once.
Changes to their sub-keys are picked up on the next tick by the
periodic scanners, the auto-response helpers
(block/kill/quarantine/challenge/permission-fix), alert dispatch, and
the heartbeat.
reputation.verified_bots is reconciled on reload and the bot-verifier
cache is restamped when the list changes.
reputation.bot_ranges controls a long-lived updater goroutine and is
tagged restart-required. A reload that changes it emits
config_reload_restart_required; restart the daemon to change the
auto-update switch or interval.
Two sub-keys are runtime-state exceptions. They live under a safe-tagged parent and SIGHUP accepts edits to them, but they seed long-lived in-memory state at daemon startup. Restart the daemon if you need the running process to use the new value:
reputation.whitelist– seeded into the threat database at startup and replaced on every successful reload, so editing the list in csm.yaml and sending SIGHUP takes effect at once. The threat database also exposes its own runtime API for adding and removing whitelist entries (via the Threat Intelligence page in the Web UI or the/api/v1/threat/*endpoints); those entries are persisted to disk and are kept separate from the configured list, so a reload never drops them.email_protection.known_forwarders– captured by the forwarder watcher at startup and read by scheduled forwarder and mail-filter checks. No runtime API yet; send a restart if you edit this list.
auto_response.mail_auth_recovery is a restart-required sub-key under
the otherwise safe auto_response section. It is captured by the cPanel
mail auth backend probe at startup, so a reload that changes it emits
config_reload_restart_required and leaves the live config unchanged.
alerts.block_digest is restart-required under the otherwise safe
alerts section. The collector and ticker are built at startup, so a
reload that changes these settings emits
config_reload_restart_required and leaves the live config unchanged.
When countries is empty, the fallback to suppressions.trusted_countries
still follows safe reloads. Delivery uses the current email and webhook
settings. The daily summary section gives by-country and by-category
counts, and live alerts add by-reason counts; WAF categories group
ModSecurity escalations and high-volume WAF attacker blocks together.
If you turn off alerts.block_digest.enabled in Settings, restart before
turning off the last alert channel. Until the restart the watched-country
section and live alerts stay on, and Settings rejects a channel change that
would leave them with no channel. Once the running configuration has the
watch off, email and webhook may both be off, and then no daily summary is
sent.
The rest of the sub-keys in every safe-tagged section are read per-call (inside check functions, auto-response helpers, alert dispatchers) and hot-reload cleanly on the next tick.
Look for one of three log shapes in the journal:
SIGHUP: config reloaded; safe fields updated: [thresholds]– success. The new values are live.config_reload_restart_required: SIGHUP reload: restart-required fields changed: [hostname ...]; live config unchanged– the edit touched a field that cannot be hot-swapped. A Warningconfig_reload_restart_requiredfinding is also emitted. Fall back to the restart path below.config_reload_error: SIGHUP reload: parse failed ...or... validation error ...– the file on disk is not loadable or failscsm validate. A Criticalconfig_reload_errorfinding is emitted. The live config is unchanged; fix the file and repeat.
Restart path: unsafe fields
Fields not tagged hotreload:"safe" (the majority, including
hostname, state_path, webui.listen, firewall.*, email_av.*
and anything that survives only one re-init per daemon lifetime)
require a full restart. The integrity hashes must be re-recorded first:
sudo cp /etc/csm/csm.yaml /etc/csm/csm.yaml.bak-$(date +%s)
# edit /etc/csm/csm.yaml with your favourite editor
sudo /opt/csm/csm rehash # re-records integrity hashes
sudo /opt/csm/csm validate # syntax + value sanity
sudo systemctl restart csm
sudo systemctl status csm # confirm active, no crash-loop
If the restart fails (most commonly because rehash was skipped),
roll back with
sudo cp <backup> /etc/csm/csm.yaml && sudo systemctl restart csm.
The backup carries its own matching hash so no second rehash is
needed.
Config-management tools
Config-management workflows (Ansible, Puppet, Chef) should:
- For safe changes, notify
systemctl reload csminstead ofrestart. The daemon re-records the hash itself; no separatecsm rehashstep is required. - For any change that may touch a restart-required field, run
csm rehashbefore the restart notify fires. Or always sendreloadfirst, read the journal, and promote torestartonly when the reload logsrestart-required.
conf.d drop-ins
Files matching /etc/csm/conf.d/*.yaml are loaded after the main config and deep-merged on top of it. Override with --config-dir <path> or CSM_CONFIG_DIR; the flag wins when both are set.
Explicit operator overrides require an existing directory.
- Order: lexicographic by filename. Scalar keys in
20-overrides.yamloverride the same keys in10-base.yaml. Use a numeric prefix. - Merge semantics: maps merge recursively; scalars replace the value from the main file; lists append in fragment order. All-scalar lists drop duplicate entries while keeping the first occurrence; structured lists such as
webui.tokenskeep every entry. - Trust: override directories must be absolute, must exist, and must be owned by root or the running process. The directory and every loaded fragment must not be group- or world-writable. Safe symlinked fragments are allowed, so packaged profiles can still be linked into
/etc/csm/conf.d/. - Integrity ownership: drop-ins cannot set the
integrityblock. Integrity metadata is stored only in the main config. - Hash:
integrity.config_hashcovers the main file andintegrity.confd_hashcovers loaded drop-ins. After editing a drop-in by hand, runcsm rehashbefore restarting, or usesystemctl reload csmso the daemon can re-hash after validating the merged config. Rehash re-hashes the resolved main file only; when a real legacy copy still exists at/opt/csm/csm.yamlwith the same operator content, rehash replaces it with the compatibility symlink so the two never drift. Web settings saves refuse to bless a drop-in change that has not already been re-hashed. A mismatch is refused at the next daemon start with an error that namescsm rehash. Until then the running daemon keeps its old hashes, but its periodic integrity check raises a tamper finding and skips the scheduled checks on every cycle;csm doctorreports the same mismatch, so it can be fixed before a restart turns it into an outage. - Integration-owned fragments: a fragment its owning package rewrites on its own schedule (phpanel-server-agent rewrites
10-phpanel-runtime.yamlon every bootstrap) cannot be pinned by a static hash without making every one of those rewrites a failed restart. List such fragments by bare filename underconfd.integrity_exemptin the main config, then runcsm rehashonce. Their content is left out ofconfd_hash; every other fragment stays covered. The list lives in the main config, outside theintegrityblock, so it is itself covered byconfig_hash, and a fragment cannot setconfdto exempt itself. - Scoped re-hash: an integration that rewrites a covered fragment of its own runs
csm rehash --fragment <file>(repeat the flag for several files) instead of a fullcsm rehash. It re-records hashes only when the named files are the only change since the last recorded state. If the binary, the main config or any other fragment changed, it records nothing and names what changed. A named file may be new, edited or deleted; a deleted one must have been part of the recorded state. Fragments underconfd.integrity_exemptstay outside the recorded state either way. CSM keeps a per-fragment record incsm.yaml.confd-fragments.jsonbeside the main config. Every hash update writes it, and each daemon start that verifies conf.d writes one when it is missing or out of date, so upgraded hosts need no extra step. Exit codes:0: re-hashed, or nothing had changed.4: refused because something besides the named files changed; stderr lists it.5: no per-fragment record for the current recorded state yet. A fullcsm rehashre-hashes everything and writes one.- Any other non-zero code: an error, including a corrupt record, an incomplete option, an invalid fragment name or a file that never existed. Do not fall back to a full
csm rehashon these errors.
- Record privacy: the per-fragment record retains covered fragment contents to validate them against the recorded digest. It is written with owner-only access beside the resolved main config and must be protected like the config itself. A corrupt record never authorizes a fallback to a full rehash; a successful startup verification can rebuild it from verified fragments.
- Use cases: packaged integration profiles (e.g.
/usr/lib/csm/profiles/phpanel-agent.yamlsymlinked intoconf.d/), per-host automation that should not touch the operator’scsm.yaml, secret material rendered from a vault.
ls /etc/csm/conf.d/
# 10-phpanel-agent.yaml 20-tenant-overrides.yaml
csm validate # validates the merged config
csm config show # prints the merged, redacted config
csm config schema # JSON Schema for editor / CI validation
csm validate and csm config show always operate on the merged config so you can audit the effective state without grepping fragments.
detection.direct_smtp_egress
The direct SMTP detector’s backend accepts auto, bpf, legacy, or none;
ports must contain TCP ports in the 1-65535 range. See
Direct SMTP egress.
bpf_enforcement
BPF enforcement requires a BPF-capable connection tracker at
runtime; auto falls back to legacy detection on older servers. See
BPF enforcement.
firewall.tcp_in_allow
tcp_in_allow opens one TCP port to listed sources only. Each entry names a port and its sources:
firewall:
tcp_in_allow:
- port: 3306
sources:
- 203.0.113.10
- 198.51.100.0/28
- 2001:db8::11
Use it for a database, replication or monitoring port that a few known hosts need and nobody else should reach. The port does not need to be in tcp_in, and it may be in restricted_tcp. infra_ips is the wrong tool for this: it opens every port to the address.
port is a single TCP port. sources is required and takes IPv4 and IPv6 addresses or CIDRs; a bare address means that one host, the same as /32 or /128. A port out of range, an empty list, a malformed address and a world-wide prefix (0.0.0.0/0, ::/0) are validation errors; open a port to everyone in tcp_in instead. IPv6 sources need firewall.ipv6: true; with it off, IPv6 is not filtered at all and validation warns.
The Web UI and sshd lockout warnings treat a port with at least one usable source here as deliberate and stay quiet for it, in both IP families. An entry whose sources render no rule, such as IPv6 sources while firewall.ipv6 is off, does not count. A usable source also keeps a port in restricted_tcp reachable without infra_ips.
The accepts come right after infra_ips and before the blocked address, blocked subnet and country rules, and before the rate limits. A listed source therefore keeps reaching its port while it is blocked, and only that port: every other port still follows the blocklists. csm firewall deny on a listed source does not close the port for it. Remove the source from the entry and run csm firewall restart instead.
Entries from conf.d fragments are added to the ones in csm.yaml. A change takes effect on csm firewall restart. csm firewall status and csm firewall ports list the applied entries, including changes made by a firewall restart or rollback, and the Web UI firewall page shows and counts the entries with its port exceptions. That page reads the config the daemon loaded, so after a firewall restart or rollback it can differ from the applied policy until the daemon reloads; the CLI shows what is applied. Hosts that support the key report the firewall.tcp_in_allow.v1 capability.
For a single address on one host without a config change, csm firewall allow-port still works. That allow is a soft one: it sits after the blocklists, so an operator block overrides it.
firewall.protocol_allow
protocol_allow opens an IP protocol that has no ports to listed peers only. Each entry names a protocol and its peers:
firewall:
protocol_allow:
- protocol: vrrp
peers:
- 192.0.2.11
- 2001:db8::11
The only protocol today is vrrp (IP protocol 112), for keepalived and other VRRP speakers on database or load-balancer pairs. tcp, udp, icmp and icmpv6 have their own keys and are refused here. peers is required and takes IPv4 and IPv6 addresses or CIDRs. An empty list, an unknown protocol, a malformed address and a world-wide prefix (0.0.0.0/0, ::/0) are validation errors.
For each peer the input chain accepts the protocol from that address and the output chain accepts it to that address. For vrrp the output chain also accepts advertisements to the VRRP multicast group of each family that has a peer (224.0.0.18, ff02::12). Advertisements a peer sends to the group match its input rule. IPv6 peers need firewall.ipv6: true; with it off, IPv6 is not filtered at all.
The input accepts come right after infra_ips and before the blocked address, blocked subnet and country rules. A blocked peer therefore still exchanges VRRP with this host, and only VRRP. Without this, a block on a peer makes both members claim the shared address. Blocks of other addresses do not affect these rules.
csm firewall status and csm firewall ports list the applied entries, including changes made by a firewall restart or rollback. Hosts that support the key report the firewall.protocol_allow.v1 capability.
firewall.dos_exempt_ranges
dos_exempt_ranges is a list of CIDRs or single IPs (/32 for IPv4, /128 for IPv6) declared by the operator. Entries are validated at startup; /0 default routes, bare hostnames, and malformed CIDRs are rejected before the daemon starts. Default: empty (no declared ranges).
dos_exempt_known_mail_providers (bool, default true) adds Google and Microsoft outbound mail IP ranges to the effective exempt set automatically. The ranges are sourced dynamically and cached on disk; a built-in snapshot covers startup before the first live refresh. The cache is refreshed every 12 hours.
Sources in the effective exempt set bypass the new-connection rate-limit for their IP family, the IPv4 concurrent connection-limit, and the TCP port 25/465/587 flood meters for their IP family. Subnet auto-block (spray, ASN-crawl, and netblock escalation) skips CIDRs that intersect an exempt range, and exempt IPs do not count toward the netblock threshold.
Exempt sources do not bypass: manual blocks (csm firewall deny or csm firewall deny-subnet), SYN flood protection, UDP flood protection, country blocking, and port policy. A manual block placed inside an exempt range still takes effect because blocked sets are evaluated before the DoS meters. See Firewall - DoS-exempt ranges.
Credential rotation
Environment-backed credentials come from the environment inherited when CSM
starts. Reading that environment before each request does not import edits
to an environment file or exports from another shell. Restart CSM after
changing its launch environment. csm rehash, SIGHUP, and
systemctl daemon-reload do not replace a running process’s environment.
The same rule applies to these integrations:
| Integration | Environment variable name configured in | Static fallback |
|---|---|---|
| Upstream threat intelligence | reputation.upstream.token_env | reputation.upstream.token |
| Rspamd | reputation.rspamd.token_env | reputation.rspamd.token |
| Verdict callback | auto_response.verdict_callback.hmac_secret_env | auto_response.verdict_callback.hmac_secret |
| Phpanel finding webhook | alerts.webhook.hmac_secret_env | alerts.webhook.hmac_secret |
A non-empty environment value takes precedence over the static fallback. An unset or empty variable uses that fallback. Each verdict exchange keeps one secret for both request signing and response verification.
Configure a systemd environment file
Create /etc/csm/credentials.env, owned by root with mode 0600. Put the
actual values in this file through your editor or configuration manager:
CSM_UPSTREAM_TOKEN=replace-with-upstream-token
CSM_RSPAMD_TOKEN=replace-with-controller-password
CSM_VERDICT_SECRET=replace-with-verdict-secret
CSM_WEBHOOK_SECRET=replace-with-webhook-secret
Set the corresponding CSM configuration fields to these variable names.
Then create /etc/systemd/system/csm.service.d/credentials.conf:
[Service]
EnvironmentFile=/etc/csm/credentials.env
Apply the new unit configuration and start CSM with that environment:
systemctl daemon-reload
systemctl restart csm.service
systemctl is-active csm.service
For later rotations, update the existing credentials file and run
systemctl restart csm.service. daemon-reload is needed when the unit or
its drop-ins change, not for a value change in an already referenced file.
For a foreground daemon, stop it and launch a new process with the updated
environment supplied by its launcher.
Coordinate the change with each receiver. Where the receiver supports overlapping credentials, retain the old credential through restarts and in-flight requests. Verify successful authentication with the new credential at the receiver without logging its value, then retire the old one.
Regression coverage
TestEnvironmentCredentialRotation launches a subprocess from an environment
file and captures real HTTP requests from all four clients. Requests before
and after an external file edit use the old credential while the process
stays running. Restarting the subprocess from the edited file switches all
four clients to the new credential. The test does not change the running
process’s environment with os.Setenv.
Custom account roots
account_roots accepts absolute, normalized directory paths or glob patterns.
The same configured content trees are eligible for manual remediation and
quarantine restore. Detected panel account homes and the existing scratch
quarantine locations keep their normal scope. A configured content directory
itself is not a restore target; restore operates beneath it.
Symlinks in a configured root or its ancestors are rejected. Unsafe trees are excluded from remediation without disabling other tenants’ valid roots. Destination directories are pinned during restore, including when a tenant renames a directory while the operation is running.
Service write access
The packaged service uses ProtectSystem=strict. A custom content tree needs a
write grant before the daemon can quarantine, clean, or restore files there.
Keep the account container directory owned by root and not writable by tenants:
account_roots:
- /srv/csm-accounts/*/public
For this layout, /srv/csm-accounts is root-owned; the account directories below
it can belong to their tenants. Provision the directories before generating the
drop-in:
set -e
fragment="$(mktemp)"
trap 'rm -f "$fragment"' EXIT
csm systemd-roots > "$fragment"
cat "$fragment"
install -d -m 0755 /etc/systemd/system/csm.service.d
install -m 0644 "$fragment" /etc/systemd/system/csm.service.d/50-account-roots.conf
systemctl daemon-reload
systemctl restart csm.service
csm doctor
csm systemd-roots only prints the drop-in. It grants existing content
directories or their nearest root-controlled ancestor, and adds mount ordering
for those paths. It refuses symlink roots and broad grants such as /, /srv,
or /etc. It escapes path quoting and systemd specifiers. The generated
fragment appends grants to the packaged service; it does not reset that list.
A tenant-owned directory cannot itself be a grant: its owner could replace it
with a symlink before systemd starts. If the command cannot find a narrow,
root-controlled ancestor, move the accounts beneath a dedicated root-owned
container directory and update account_roots.
Regenerate the fragment after changing roots or provisioning a tree outside
existing grants. Changing account_roots requires a daemon restart. If integrity
verification is enabled, update its baseline for an intentional configuration
change before restarting, as described in Configuration.
Verification
csm doctor checks configured custom roots and detected panel roots outside the
packaged home grant. It reports missing roots, unsafe aliases, missing service
grants, and read-only mounts in the running daemon’s filesystem view. A stopped
service or an unavailable systemd connection is reported as unverified access.
Reloading unit files alone does not change the running service’s mount namespace;
restart the service and run doctor again.
After setup, verify detection, quarantine, listing, and restore on a test account under the custom root. Confirm that the restored file still belongs to its tenant. A sibling tree outside the configured roots and existing scratch locations must remain ineligible for restore.
The repository includes a disposable systemd service test. Build an image from
build/Dockerfile.systemd-test, then use the Linux wrapper:
GO_LINUX_IMAGE=csm-systemd-test scripts/go-linux.sh bash scripts/systemd-account-roots-test.sh
cat .cache/systemd-account-roots/result
The test runs production detection, quarantine, listing, restore, and doctor inside the packaged sandbox, with a generated grant for a separate test volume. It verifies owner, permissions, timestamps, and rejected sibling and symlink destinations. This tests service confinement; it does not start the daemon’s watchers or replace panel integration coverage. Logs and generated grants are saved beside the result file. The harness refuses to run outside its disposable Linux container.
Service configuration writes
The packaged and CLI-installed units use ProtectSystem=strict. The daemon can
write these configuration directories when they exist:
| Directory | Runtime writer |
|---|---|
/etc/csm | CSM configuration and fragments |
/etc/audit | CSM audit rules and augenrules generated configuration |
/etc/modprobe.d | Repair of an existing, opted-in AF_ALG mitigation marker |
/etc/apache2/conf.d | cPanel Apache/LiteSpeed challenge snippet, virtual patches, and default overrides |
/etc/apache2/conf-enabled | Debian Apache challenge snippet |
/etc/httpd/conf.d | RHEL Apache challenge snippet |
/etc/nginx/conf.d | Nginx challenge snippet |
/usr/local/apache/conf | Legacy Apache virtual patches and overrides |
/usr/local/lsws/conf/templates | Standalone LiteSpeed challenge template |
The other state, quarantine, account, spool and cPanel plugin grants remain in
place. Optional directory grants tolerate absent software. Install the relevant
web server before starting CSM; restart CSM after creating a previously absent
granted directory. Custom ModSecurity override locations require a drop-in that
grants only their parent directory, followed by systemctl daemon-reload and
systemctl restart csm.service. Atomic replacement requires a directory grant.
See custom account roots for account grants.
The installer generates the unit for the systemd it finds. systemd 239 (EL8,
CloudLinux 8) does not know ProtectHostname, ProtectKernelLogs or
ProtectClock; it would log “Unknown lvalue” for each at every start and
ignore them, so the installer leaves them out there and prints which ones it
omitted. Newer systemd receives the full unit, identical to the packaged one.
A binary-swap upgrade keeps the existing unit; run csm install to regenerate
it.
The daemon cannot write arbitrary files directly under /etc, SSH configuration,
account databases, or systemd unit files. The AF_ALG seccomp drop-in is installed
by the operator’s csm harden CLI, outside the daemon. Periodic AF_ALG module
removal uses a bounded transient service with fixed module names; the daemon
retains ProtectKernelModules=yes and the module syscall restriction.
Exim transactions
The forward guard sends one bounded JSON request over stdin to its own executable
in a transient service. The helper accepts only apply/remove operations and policy
data. It accepts no paths, command names or shell programs. Applying or removing
the managed Exim block, the cPanel rebuild, and rollback all run outside the
daemon mount namespace. Operator configuration is preserved. Lookup refreshes
remain under /var/lib/csm and do not rebuild Exim.
The helper requires root. An exclusive process lock rejects overlapping config transactions without modifying Exim; the daemon reports the failure, and the operator can reload after the earlier transaction finishes. Its lifetime is bounded, and a failed command is never retried as an unsandboxed duplicate. Existing fixed Exim queue/query operations also use transient services. If the systemd bus is unavailable, the existing wrapper attempts direct execution once; inside the service, restricted writes then fail and are reported. A scope unit is unsuitable because it inherits the calling process’s sandbox.
Parse worker units
The daemon reads tenant zip listings for the phishing-kit check, and unpacks
email attachments when email AV is on, in separate workers. Each starts on
demand as a transient unit with systemd-run: one unit for zip listings, and
one per email_av.scan_concurrency slot for mail. Each unit gets these
properties:
DynamicUser=yes,CapabilityBoundingSet=(empty),NoNewPrivileges=yes;PrivateNetwork=yes,RestrictAddressFamilies=AF_UNIX;ProtectSystem=strict,ProtectHome=yes,PrivateTmp=yes,PrivateDevices=yes,ProtectKernelTunables=yes,ProtectKernelModules=yes,ProtectControlGroups=yes,RestrictNamespaces=yes,RestrictRealtime=yes,MemoryDenyWriteExecute=yes,UMask=0077;SystemCallFilter=@system-serviceandSystemCallFilter=~personality, withSystemCallErrorNumber=EPERM. The personality deny stands in forLockPersonality=yes, whichsystemd-runon systemd 239 rejects;MemoryAccounting=yes,MemoryLimit=andMemoryMax=set to the worker’s memory cap,MemorySwapMax=0;TasksMax=64,LimitNOFILE=256,LimitCORE=0,CPUWeight=50, and a one-hourRuntimeMaxSec.LimitFSIZE=is 64 MiB for zip listings; for mail it is 1 MiB aboveemail_av.max_attachment_size, and never below 64 MiB;OOMScoreAdjust=1000, making parse workers preferred victims under host memory pressure, andSlice=set to a slice of its own per unit name prefix directly undersystem.slice, for examplesystem-csm\x2dparse\x2dworker\x2dmail0.slice. The slices stay after their units end; they contain no processes but retain their accounting metadata.InaccessiblePaths=-for each account home volume thatProtectHome=yesdoes not cover: the account roots outside/home(for example/var/www/vhostson Plesk) and the sibling volumes registered cPanel accounts live on, such as/home2. The list is read once when the daemon sets up its workers, or whencsm scanstarts one.
Before launching a worker, CSM sets its slice’s runtime CPUWeight=50
through systemd. CPU weights compare siblings: the service’s weight alone
would not limit its share against the daemon outside that slice. Slice setup
shares the worker’s startup timeout, and a failure prevents the launch.
Empty slices and their runtime settings remain until reboot.
On cgroup v2, a replacement waits for earlier units in its slice to disappear before taking the slice’s OOM snapshot. If they outlive the startup timeout, the replacement gets no request and reports a startup failure.
The zip worker receives the zip as an open descriptor and returns entry names
in a sealed memfd. A mail worker receives the message’s spool header and body
as open descriptors and returns each attachment in its own sealed memfd; the
daemon scans those and closes them. It cannot create IPv4 or IPv6 sockets. Unix sockets remain
available. /home, /root, /run/user and the account home volumes listed
above are hidden; an account volume created after the daemon started is not
hidden until it restarts. Most host files are read-only. The worker can still read
world-readable host files outside those directories and write in its private
temporary directories. The daemon does not forward its environment to the
unit; systemd supplies the service environment.
A worker’s standard error reaches the daemon’s log through a pipe, never directly. Each line is cut to 512 bytes and quoted, so control characters and newlines cannot forge log lines, and at most 256 lines or 64 KiB of log entries are logged per worker unit; after that one “further output dropped” line is logged and the rest is discarded. Error text in a worker’s reply is quoted the same way. When the launcher exits, final diagnostics drain for at most one second. The log reader then closes even if another process still holds its writer.
MemoryLimit caps RAM only on cgroup v1, so the daemon also writes
memory.memsw.limit_in_bytes in the unit’s own memory cgroup, or
memory.swappiness=0 where there is no swap accounting. On cgroup v2 it sets
memory.swap.max to 0, and refuses to use a worker on a host with swap on but
no swap accounting. The daemon can write those files because csm.service
has ProtectControlGroups=no. A worker whose memory cannot be capped or
measured gets no request: the phishing check reports itself incomplete, and
mail is handled as an incomplete scan under email_av.fail_mode.
Without cgroup v2 swap accounting, the watchdog also checks that swap stays
off throughout the worker’s lifetime.
The unit runs the daemon’s own binary as its dynamic user, so the binary is
installed as mode 0711: others can execute it but not read it, and only root
can replace it. Package, CLI and script installs set this mode under any umask
and make the installation directory searchable by others; its subdirectories
keep their private modes. A rolled-back upgrade restores the previous mode.
When the binary or a directory above it is not executable by others, the
daemon logs one line at startup, raises a Critical
email_av_isolation_unavailable finding, and starts no worker. The phishing
check reports itself incomplete, and mail is handled as an incomplete scan
under email_av.fail_mode. Failed unit starts raise the same finding, at most
once an hour. This includes starts refused because the daemon has exhausted
its descriptors; those failures do not open the worker breaker. Mail and zip
workers share this allowance across spool watcher replacements. A finding
rejected by a full alert queue leaves the allowance available for a retry.
Workers are killed by unit name with systemctl kill, never by a PID. If the
daemon dies, a worker waiting for a request reads end of file and exits. A
worker stuck parsing may remain until the lifetime cap stops it. At startup,
the daemon attempts to kill leftover csm-parse-worker-* units; cleanup
failures are logged and do not prevent startup.
A worker the daemon stops on purpose, such as at shutdown, is logged as
stopped and never as a memory failure.
disabled_checks: [phishing] disables scheduled phishing checks. Requested
account scans can still use the worker, and startup cleanup still runs.
email_av.enabled: false starts no mail workers.
The PHP taint worker runs in the same kind of unit, named
csm-phptaint-worker-<generation>, with a 512 MiB memory cap and no file
writes. At startup the daemon attempts to kill leftover csm-phptaint-worker-*
units. The binary’s permissions are checked before each worker launch. If
the dynamic user cannot execute it, the daemon logs why and candidate files
are reported as not analyzed, just as when a unit fails to start or dies.
These failures appear as worker_failure coverage gaps. disabled_checks: [php_taint_deep] stops new analyses; startup cleanup still runs.
YARA-X runs in two units of the same kind. Each rule compile runs in
csm-yara-compile-worker-<generation> with a 1 GiB memory cap, from rule
text the daemon passes in memory; the compiler cannot open include files.
Scans run in one csm-yara-worker-<generation> with a memory cap of 512 MiB
plus the larger of thresholds.full_scan_max_file_mb and, with mail scanning
on, email_av.max_attachment_size, no file writes and the compiled rules
loaded from memory. A reload that changes the file cap applies to the next
scan worker. Both allow executable memory, which YARA-X needs for its
compiled rules; the other lockdown properties are unchanged. At startup the
daemon attempts to kill leftover units of both kinds. While no worker can
scan, files and mail attachments are reported as not scanned by YARA, never
as clean. signatures.yara_worker_enabled: false runs YARA-X inside the
daemon as root instead; csm validate warns about it.
The in-process csm scan <user> command starts its own worker on demand.
Its unit prefix includes the CLI process ID, so concurrent scans use distinct
slices and cannot attribute another scan’s OOM kill to their own worker.
Its report warns when worker setup, startup or archive listing fails, even
if later archives are checked successfully. A scan with unchecked archives
does not report the account as clean.
Limits and validation
These restrictions reduce accidental and misdirected writes. CSM remains a privileged root daemon: fanotify, firewall administration, BPF, process inspection and remediation require elevated privileges. The unit retains its current capabilities, and root access to the system bus permits transient services. This is not a complete boundary against arbitrary code execution as root. Reducing capabilities or separating privileged operations behind a separately permissioned broker requires its own feature and kernel compatibility review.
scripts/systemd-account-roots-test.sh runs the actual packaged sandbox in a
disposable systemd environment. It checks unrelated /etc writes return EROFS,
managed directory atomic replacement, ModSecurity overrides, and Exim helper
apply/remove/rollback using a fixed command fixture. It also exercises account
quarantine, restore and process signaling. The Exim fixture validates the helper
transport and filesystem behavior; it does not validate a real cPanel rebuild.
The cPanel release tests exercise real daemon restarts and require a licensed image
and a live run to establish that platform’s acceptance.
Upgrading
Use the signed APT/DNF repository for maintained package upgrades. Standalone
deploy scripts require successful detached signature verification for current
releases. They verify with OpenSSL 3.0+ where available, otherwise with the installed
CSM binary’s csm verify-release, otherwise with python3-cryptography, so
upgrades keep working on EL8/CloudLinux 8 (OpenSSL 1.1.1) including the first
upgrade to a build that provides csm verify-release. When no verifier is
present the upgrade stops; disabling CSM_REQUIRE_SIGNATURES does not enable
unsigned current upgrades. Set CSM_VERIFIER_BINARY to select a specific CSM
binary. See
Release signing for the historical-release exception.
Package installations (recommended)
Use the same signed repository that installed CSM:
sudo apt update && sudo apt install --only-upgrade csm # Debian/Ubuntu
sudo dnf upgrade csm # RHEL family
The package preserves the operator config and state, updates runtime assets, re-records integrity hashes, and restarts the daemon when it was already active.
Debian and Ubuntu upgrades keep the local main config without a dpkg prompt. Compare it with the shipped template at /usr/lib/csm/csm.yaml.default. RPM keeps a locally modified config and writes a changed template to /etc/csm/csm.yaml.rpmnew. RPM may update an unmodified config in place.
If the main config is missing on upgrade and there is no regular legacy config to migrate, package setup stops before restarting CSM. Restore your config to /etc/csm/csm.yaml and retry.
Standalone installations
sudo /opt/csm/deploy.sh upgrade
The upgrade refuses to install a release older than the one running. To roll back deliberately, run it with CSM_ALLOW_DOWNGRADE=1.
The helper:
- Downloads and verifies the binary and supporting assets
- Stages the UI, rules, PAM files, and deploy helper before downtime
- Stops the daemon and keeps the previous binary and assets as rollback material
- Activates the staged release and rehashes the config once
- Restarts the daemon, checks sustained liveness, and runs
csm doctor
The health gate waits 20 seconds by default. Set CSM_UPGRADE_HEALTH_SETTLE to an integer from 1 to 3600 seconds for a longer or shorter observation window. Invalid values fail the gate. Doctor warnings do not trigger rollback; failed checks do.
If activation, rehash, startup, or the health gate fails, the helper stops the new daemon, restores the previous binary and assets, re-records the restored binary hash, and starts the previous version. It exits nonzero and reports where it retained recovery material. If recovery itself fails, it reports an incomplete rollback that needs operator attention.
Optional nightly upgrades
Packages ship a disabled sample at /opt/csm/configs/cron/csm-auto-upgrade. It runs only after an operator installs it into /etc/cron.d/:
csm version && /opt/csm/deploy.sh check
sudo install -m 0644 /opt/csm/configs/cron/csm-auto-upgrade /etc/cron.d/csm-auto-upgrade
Confirm the host runs a published release first. The checksum check reports any different build as an available update. The downgrade guard rejects a lower version, but a development build with the same version can still be replaced by the published release.
The job invokes the standalone deploy helper every night between 03:30 and 04:30, with a random delay and a nonblocking lock. This updates runtime files directly; it does not update APT/DNF’s installed package version. Hosts that need package-manager ownership and version tracking should use their package upgrade automation instead.
Output goes to /var/log/csm/auto-upgrade.log. Failures also produce cron mail to root; configure and test root’s mail alias before enabling the job. This notification does not depend on CSM’s alert channels or on successful daemon recovery. Successful runs and an already-held lock produce no mail. Staggering spreads upgrades over an hour but does not stop a bad release from reaching the fleet.
Troubleshooting
“store: opening bbolt: timeout” – Most operator commands that need live state now route through the control socket at /var/run/csm/control.sock. This error should only appear from commands that intentionally open the bbolt file directly, such as csm store compact, csm store import, csm store reset-bot-verify, csm db-clean --drop-object, or a second daemon start while one daemon already owns the database.
Fix: stop the daemon before direct-store maintenance commands, then retry:
systemctl stop csm
csm store compact
systemctl start csm
If systemctl says CSM is stopped but bbolt still times out, find the process holding /var/lib/csm/state/csm.db and stop that process after review. Do not delete csm.lock; it is only the daemon instance guard and does not release bbolt’s file lock.
“csm: daemon not running” – CLI commands that talk to the daemon exit 2 with this message when the control socket is missing. This includes csm run*, csm check*, csm baseline, csm status, csm firewall ..., csm store export, csm export --since, and csm phprelay .... Start the daemon with systemctl start csm. Bootstrap commands that run before the daemon exists (csm install, csm validate, csm config schema, csm verify, csm rehash) do not require it.
Never delete csm.db – it contains all historical findings, firewall state, email forwarder baselines, and per-account data. If you delete it, the web UI will show empty data until the next full scan cycle (up to 60 minutes for deep scan findings). Restore from backup when possible; for an intentional reset, run csm baseline --confirm rather than removing the database by hand.
Config changes require rehash – After editing a restart-required field in csm.yaml or any conf.d drop-in, run csm rehash once, validate, then restart. Hot-reload-safe changes can use systemctl reload csm; the daemon validates the accepted config and re-records its hash itself. A restart that fails with conf.d hash mismatch means a drop-in changed since the hashes were last recorded: rehash if the change was intentional, and list fragments an integration rewrites on its own under confd.integrity_exempt so they stop needing one. csm doctor reports the mismatch while the daemon is still up.
FHS migration (state, config, drop-ins, and profiles)
Current packages use FHS paths for state, config, drop-ins, and shipped profiles. Legacy main configs continue to work during the transition.
| Concern | Legacy path | Current path |
|---|---|---|
| Drop-in fragments | n/a | /etc/csm/conf.d/*.yaml |
| State directory | /opt/csm/state | /var/lib/csm/state |
| Shipped profiles | n/a | /usr/lib/csm/profiles |
| Binary | /opt/csm/csm | /opt/csm/csm (unchanged) |
| Main config | /opt/csm/csm.yaml | /etc/csm/csm.yaml |
| Legacy config path | n/a | /opt/csm/csm.yaml symlink |
The package postinstall creates the FHS directories with the right ownership. If /opt/csm/csm.yaml is a real file and /etc/csm/csm.yaml is absent or still the shipped placeholder, the package copies the legacy config into /etc/csm/csm.yaml and then replaces the old path with a symlink. If both paths are real files with different operator content, CSM refuses the implicit default path until you move one aside or pass --config <path>. Copies that differ only in the integrity hashes CSM writes itself (binary_hash, config_hash, confd_hash) are treated as the same configuration: the daemon starts from /etc/csm/csm.yaml, and the next csm rehash replaces the legacy copy with the symlink.
The daemon copies a non-empty legacy /opt/csm/state/ into the new state directory on first start, but only when the new directory is empty (so a partial migration cannot corrupt it). The legacy directory is left in place; remove it after you have verified the new install.
Operators upgrading by manual binary swap (without re-running the package postinstall) keep the legacy state path if state_path: /opt/csm/state is pinned in the existing csm.yaml. To move state to the FHS layout, either reinstall the package or create the directories by hand and remove the state_path: override.
systemd Type=notify drop-in
The packaged unit file is Type=notify with WatchdogSec=300. The daemon signals READY=1 after watchers attach and pings WATCHDOG=1 on schedule, so systemctl is-active reflects truth and the watchdog kills a hung daemon.
Older units shipped Type=simple. The watchdog still functions because the daemon pings regardless of unit type, but systemctl status only sees the process, not “watchers attached.” If you need the new behavior on an older unit, drop in:
# /etc/systemd/system/csm.service.d/notify.conf
[Service]
Type=notify
NotifyAccess=main
Then systemctl daemon-reload && systemctl restart csm. Verify with systemctl show csm -p Type -p StatusText.
Auto-response dry-run safety default
The first disabled queued response raises admission state to schema 10, which older binaries that read only schema 9 cannot open; they keep blocking with the legacy budget. While admission only previews and forecasts, it creates no genuine charge, attempt or unknown hold beyond the state legacy blocking keeps. Preview accounting is not enforcement state. This is not a downgrade guard for a ledger that has executed real actions. Never delete the state database or replace it with an older copy to get past the format check.
The first positive elapsed episode end this release records raises admission state to schema 13, including on hosts that only preview or forecast. Older binaries cannot open it and keep blocking with the legacy budget, as above. Roll back only to a build that supports both schema 13 and elapsed episode records.
A live admission ledger records each episode’s latest verified block and retains a verified episode’s boundary even after an unattempted return is forgotten. The first verified block raises the ledger to schema 15. Roll back only to a build that reads schema 15, understands the recorded verified generation, and retains and cleans up these rows; older builds refuse this ledger. This writer runs only on hosts where admission executes. Keep the database when changing binaries.
After live re-admissions, roll back only to a build that also keeps the original verified boundary and caps queued returns at it. A build that only reads the format can give an old queued return a full lifetime. Stop the daemon before replacing the binary and keep the database.
auto_response.dry_run defaults to true when the key is absent. Automatic IP blocks are not applied. With the firewall on outside observe mode, the dry-run count holds retained forecast examples, including older entries until pruning; it is not active occupancy or a count of all attempted blocks. If your auto_response: block sets enabled: true and block_ips: true but does not set dry_run, add dry_run: false explicitly before relying on auto-block. Verify with:
csm status --json | jq '.auto_response_dry_run, .dry_run_blocks'
csm firewall status # check that "Recently Blocked" picks up new entries after the restart
Manual csm firewall ... operations bypass dry-run and always apply.
Admission applies automatic IP blocks
From this release, a host with auto_response.dry_run: false, the firewall
on and outside observe mode applies automatic IP blocks through admission,
within its hourly ceiling and the free slots of the temporary set. A host
that already runs with dry_run: false does so from the first start of the
new build. To check the admission comparison first, set dry_run: true
before the upgrade, compare a dry-run window as described in
finding streams, and
set dry_run: false once it passes.
Legacy blocks applied before the upgrade stay until they expire. When
admission wants a longer block for an address such a block already holds,
the work waits for that block to end. For Critical work the wait raises a
capacity notice with reason pending_recovery; the legacy block keeps the
address blocked meanwhile.
The automatic block counter now also counts the challenge, central and incident blocks admission applies, so block rates on dashboards can rise after the upgrade.
To stop admission applying blocks without changing the binary, set
auto_response.dry_run: true and run systemctl reload csm. Dry run stops
every automatic IP block, legacy ones included; blocks already applied stay
until they expire. To return automatic blocking to the legacy path, roll
back the binary instead.
The supported rollback target is the pre-flip 4.14.0 release. Older builds cannot safely read all admission records; some also reject newer config keys. Removing config keys does not make those versions safe rollback targets. Keep the database and firewall state, including action identifiers and absolute expiries. Do not restore an older state snapshot to bypass a format check.
Before upgrading, retain a verified 4.14.0 package or release artifact and its matching assets; also retain the new artifact for recovery. The upgrade helper deletes its previous-binary copy after a successful upgrade. Resolve the active config and state locations, including drop-ins, and keep a restricted backup.
For a rollback:
- Set
auto_response.dry_run: truein the effective configuration, reload, and confirmcsm status --jsonreportsauto_response_dry_run: true. A drop-in may override the main config. Confirm admission has stopped executing (admission.owner.executingis false) before proceeding; a failed reload is not a successful stop. - Note the installed version, the
csm doctorresult and the current timed blocks, then stopcsm.serviceand confirm it is inactive. Keep a consistent backup of the stopped database and firewall state. - Install the retained 4.14.0 package with the package manager, allowing
the downgrade. The standalone helper installs only the latest release:
on a standalone host, put the retained 4.14.0 binary and its assets in
place and run
csm rehashbefore starting. Do not substitute a locally compiled binary. - Start the service if the installer did not, verify the running version,
sustained service health and
csm doctor, and compare remaining timed blocks and expiries with the saved inventory. Keep dry run enabled until these checks pass. Version 4.14.0 only previews admission; with dry run off, the legacy path applies automatic blocks. - To resume legacy automatic blocking, set
auto_response.dry_run: false, reload and verify the effective status. Leaving it true continues to suppress automatic blocking. Existing blocks retain their expiry; this procedure does not flush them or promise to restore already expired blocks.
If installation or health checks fail, keep dry run enabled and restore the retained verified new release and its matching assets through the same workflow. Rehash if required, start and repeat the health and block checks. Keep current state in place; recover a damaged state backup only through the documented backup/restore procedure. Do not re-enable automatic blocks on a failed health check.
Admission starts live once per ledger. Upgrading again after a rollback
does not repeat that start: blocks the legacy path applied in the meantime
stay until they expire, and the legacy path’s hour is not carried over
into admission’s hourly ceiling. To compare a dry-run window again before
admission blocks, keep auto_response.dry_run: true through the upgrade.
CLI Commands
Packages and the standalone installer expose /usr/sbin/csm, which points to /opt/csm/csm. Commands that talk to the control socket require the daemon to be running; direct-store maintenance commands say explicitly when it must be stopped.
Global flags
csm rehash and csm capabilities reject missing option values before recording
hashes or printing output. csm capabilities accepts but does not read config paths.
| Flag | Description |
|---|---|
--config <path> | Override the main config path. Default: /etc/csm/csm.yaml, with fallback to /opt/csm/csm.yaml on legacy installs. |
--config-dir <path> | Override the conf.d directory. Default: /etc/csm/conf.d. Wins over CSM_CONFIG_DIR when both are set. Override paths must be absolute, trusted, and not group- or world-writable; loaded fragments must meet the same write-safety check. |
Daemon
| Command | Description |
|---|---|
csm daemon | Run as persistent daemon (fanotify + inotify + PAM + periodic checks). Signals systemd READY=1 after watchers attach and pings WATCHDOG=1 on the configured interval. The systemd notification variables are read once at startup and removed from the environment, so no command the daemon runs inherits them. |
Notification writes time out if systemd stops reading its socket, so they cannot block the daemon indefinitely. ModSecurity rule actions are checked every five minutes using file contents, including linked rule files. Unchanged rule trees skip parsing; empty or incomplete builds keep retrying, and platform detection resumes when the selected directories no longer yield rules. A temporarily empty tree keeps the last loaded actions until a replacement is available.
Checks
| Command | Description |
|---|---|
csm run | Run all checks now via the daemon, send alerts |
csm run-critical | Critical checks now via the daemon (the daemon also schedules critical checks internally every 10 min) |
csm run-deep | Deep checks now via the daemon (the daemon also schedules deep checks internally every 60 min) |
csm check | Run all checks via the daemon, print findings to stdout, no alerts / auto-response |
csm check-critical | Test critical checks only (dry-run via daemon) |
csm check-deep | Test deep checks only (dry-run via daemon) |
csm scan <user> [--alert] | Scan a single cPanel account (capped quick scan). --alert sends alerts for the findings. An account whose home is on an additional home partition is found through the cPanel account list and its passwd home. |
csm scan <user> --full [--wait] [--json] [--respect-ignores] [--quarantine] | Uncapped deep scan of one account, routed through the daemon (bypasses the per-account file cap). It also runs the YARA rules over the account’s document roots, one file at a time through the YARA worker; files over full_scan_max_file_mb are reported as not scanned, and image files are scanned after everything else so a scan that runs out of time has read the code first. It is report-only by default; --quarantine acts on critical file findings the way the scheduled auto-response does (WordPress core, plugin and theme files are cleaned in place unless cleaning would leave them empty or their release does not ship them; directories, symlinks and lower-severity findings are left for review); --wait polls to completion and prints the report, and exits with status 3 when the scan stopped early, was canceled or failed; --respect-ignores honors ignore_paths. |
csm scan --all --full [--wait] [--json] [--respect-ignores] | Server-wide uncapped full scan across every account. Quarantine per account after review (--quarantine is rejected with --all). |
csm scan --status [id] [--json] | List full-scan jobs, or show one job by id. |
csm scan --report <id> [--json] | Print the stored report for a completed full-scan job. Exits with status 3 when that scan did not cover everything. |
csm scan --cancel <id> [--json] | Cancel a running full-scan job. |
Exit status 3 means the scan did not cover everything: a check timed out or failed, a file was skipped as oversize or unreadable, the account was not found, or not every finding was stored.
Passwd home fallback accepts an account-named directory directly under a sibling
home volume, such as /home2, and rejects symlinked volumes and homes. Arbitrary
passwd locations outside this boundary are not used for account scans.
The full-account YARA pass does not follow tenant symlinks. It reports unreadable
or changed files and directories as coverage gaps, and keeps confirmed matches
if a later read or worker call times out. Large worker requests use a private
temporary snapshot in the daemon state directory, bounded by
full_scan_max_file_mb, which is removed after the request. Failure to create
or scan that snapshot is reported as incomplete coverage. The default account
scan still has no YARA pass.
Management
| Command | Description |
|---|---|
csm install | Deploy config, systemd, auditd rules, logrotate, WHM plugin |
csm uninstall [--purge] | Remove the executable and CSM-owned service, audit, logrotate, webserver, and ModSecurity integrations. Config, drop-ins, state, logs, signature rules, and quarantine data are preserved by default. --purge removes all CSM-owned data. Operator ModSecurity rules are never removed. |
csm baseline | Full server scan via the daemon, records current state for change tracking. Dangerous privileged accounts or WHM root tokens can still be reported on first scan. Takes 5-10 min on large servers. Required on first install. Add --confirm when existing history would be cleared. The daemon must be running. |
csm rehash | Re-record the binary, csm.yaml and conf.d hashes without scanning. Use once after editing restart-required config, changing a drop-in by hand, or replacing the binary; a stale hash makes the next restart refuse to start. It also applies integrity.immutable to the installed binary. csm rehash --fragment <file> re-records hashes only when the named conf.d files are the only change; see conf.d drop-ins for its exit codes. Unknown arguments are refused rather than ignored. |
csm status | Show current state, last run, active findings, and automation rollout state. Add --json for the full health snapshot (watchers, severity counts, store health, blocklist size, capabilities, version, hashes, automation). |
csm verify-release <key.pem> <file.sig> <file> | Verify a release artifact’s Ed25519 signature without OpenSSL; the supported verification path on EL8 and CloudLinux 8. |
csm systemd-roots | Print a validated systemd drop-in granting write access to configured account trees; see custom account roots. |
csm privileges [--json] [--markdown] | Print every operation that needs privilege beyond reading CSM’s own files, with what it writes and the config key that stops it. Reads nothing from the host, so it answers “what would this do to my server” before installing. See capability matrix. |
csm action show <id> | Print, as JSON, what the admission ledger recorded for one automatic response: the candidate with its state, reason and outcome, each attempt, and for a response refused because a block already enforced it, that block. <id> is a candidate ID or the action_id of an admission record. Asks the running daemon. See action log. |
csm actions [--since <when>] [--op <id>] [--limit N] [--json] | Print what CSM did to this host: quarantines, cleans, process kills and firewall changes, with the digest before and after a file change. Reads the log directly, so it answers after a crash. See action log. |
csm selftest [--json] | Scan a bundle of samples with known verdicts and report what the installed rules catch, including the gaps the bundle records. Reads no account data. See self-test. |
csm doctor | Config + integrity + daemon + watchers (including the YARA-X scanning worker, which fails while it is down or keeps crashing) + store sanity check, including service write access for custom account roots. Config checks include the firewall lockout warnings for inbound and outbound policy; the integrity check reports a binary, csm.yaml or conf.d hash mismatch before the next restart refuses to start, with the remedy. On CloudLinux with PHP Shield enabled it also reports whether the Shield event mount has been applied to the cages, names up to five missing cages, and gives per-account remount commands. Unresolved accounts appear as uid:N; resolve their account names before remounting. For more than five missing cages, it also offers a remount of all cages during a maintenance window. On Postfix hosts with Dovecot, the smtp auth logging line warns when Dovecot does not log rejected logins, which SMTP brute-force detection needs, logs their passwords, or its settings cannot be read. The correlation attribution line names the checks whose findings sit in the active set without a hosting owner (WARN) or confirms every eligible finding carries one (OK), with the cumulative count since start either way. The web client addresses line says whether the web server configuration sets no client address rewriting, replaces the logged client address (OK, naming the setting and its file; web findings then get no automatic response from admission), or cannot be read completely (WARN). It applies the loaded web_server: settings before reading the files. The configuration root selects Apache or nginx, including cPanel LiteSpeed’s Apache tree; LiteSpeed XML stays at its standard location. On Stalwart hosts the stalwart API key line says whether Stalwart accepts CSM’s API key from stalwart.api_key_file, and the stalwart setup lines check Stalwart’s settings, read-only, through that key. Without a working key doctor uses an admin login from STALWART_USER and STALWART_PASSWORD when both are set, and otherwise does not check the settings. The key is sent only to an API address on this host whose port belongs to root or Stalwart’s service user; for a remote address doctor uses the admin login only. csm doctor challenge checks challenge public URL, TLS, port gate, webserver snippets, configtest, and the live /challenge/gate endpoint. Add --json for machine-readable output. |
csm validate | Validate config (--deep for connectivity probes) |
csm config show [--no-redact] [--json] | Display the redacted config even when callback secrets are set only in the service environment. With --no-redact, a missing required callback secret stops the command before it prints the config. Use --json for JSON output. |
csm config schema | Print a JSON Schema reflected from the Config struct. Use for CI validation of conf.d drop-ins or panel-side editor schemas. |
csm config apply-immutability | Reapply integrity.immutable to /opt/csm/csm. Unsupported filesystems or missing tooling produce a warning and a successful exit; permission and other unexpected errors fail. Package transaction hooks use this after upgrades. |
csm verify | Verify binary and config integrity |
csm version | Version and build info |
csm capabilities | Print this build’s feature tokens, one per line; --json prints a JSON array. Needs no running daemon and no config, so an orchestrator can check a token before it writes a key that depends on it. Tokens for live BPF monitors appear only in the daemon’s GET /api/v1/capabilities. |
csm incidents ... | List, show, and update correlated security incidents (list, show <id>, status <id> <state>, bulk-status). See Incidents. |
csm forensic-snapshot <account> --out <archive.tar.gz> | Evidence archive for incident handoff (triggers, admins, sessions, file mtimes). |
csm webserver-integration <install|upgrade|status|validate|remove> | Install, upgrade, or remove the challenge reverse-proxy snippets for the detected web server. |
csm pam <install|uninstall|status> | Install or remove the pam_csm.so PAM hook (csm pam --help). |
csm report enroll | Generate an abuse-reporting node key pair. |
csm doctor also lists protection queue health:
waiting and running work, losses and lag. A sustained backlog or drop rate
fails the named queue check and includes recovery guidance. Queues whose work
is best effort warn instead of failing, so they never change the exit status.
While the firewall is disabled, queued automatic blocks pass the check, which
says the blocks are not applied and whether dry run is on.
The same evidence appears in csm status --json and the HTTP status response.
Each started live monitor also gets a backend: <feature> check naming the
backend it runs on, such as backend: placement. A monitor on none passes
when its detection.<feature>_backend is none and warns otherwise, with a
pointer to the startup log.
Backup & restore
| Command | Description |
|---|---|
csm backup <path> | Bundle csm.yaml, /etc/csm/conf.d/, the state directory and the antispam learning directory into a tar.gz at <path>. Runtime lock files, in-progress export staging, and unconfirmed firewall rollback artifacts are omitted. The daemon must be stopped; the command verifies both the control socket and state lock before reading data. |
csm restore <archive> [--force] | Validate and stage the complete archive before atomically replacing the live csm.yaml, conf.d, state directory and antispam learning directory; the replaced learning directory is kept aside and restored learning waits for confirmation. Restore rolls back failed replacements, removes stale state entries, and refuses a running or starting daemon. Stop the daemon first. |
csm store export / csm store import (below) is the lower-level alternative: tar+zstd, sha256-verified, finer-grained --only= flags. csm backup/restore is the convenience wrapper most operators want.
Restore and csm store import refuse an archive whose admission ledger is behind this host’s once the ledger has started live: the archive’s ledger never started live, or its clock is behind. Restoring it would spend an hour’s block budget twice and replay the live start from older blocks. --only=baseline is refused too when it carries the firewall state, because it puts that file beside the newer ledger; --only=firewall is not. A host whose ledger never started live, or that has none, is not subject to this comparison. A host ledger this build cannot read, including a truncated database, does not refuse, so a restore can repair it. Pass --force to either command to bypass this comparison.
csm store import rejects archives that place csm.db among state files; the database belongs only in bbolt.snapshot. A baseline import carrying firewall/state.json also verifies the snapshot’s checksum when the manifest provides one. --force does not bypass either format or checksum validation. Baseline imports without firewall state still ignore the unused snapshot.
Backup, export, import and restore check each database snapshot before reading or changing it and refuse a damaged one, including one with leaked pages. They make consistent backups, not salvage copies of damaged databases.
csm store import --only=firewall also checks the live database. When an archive contains antispam learning and omits state, restore checks the staged copy of the live database before replacing any files. Both operations refuse a damaged database or a broken database symlink and accept leaked pages in the live database.
Backup and restore use the state lock to exclude daemon access. Restore stages all data beside its destination before replacement, so validation failures leave the live installation unchanged. Uninstall is intentionally non-destructive unless --purge is supplied.
Restore accepts the format produced by csm backup: one tar archive in one gzip member, with no data or padding after the tar end markers and no trailing compressed bytes or additional members. It validates the gzip checksum and length before replacing any destination, and rejects corrupt or truncated archives.
Both commands stream file contents and limit the total uncompressed tar archive to 16 GiB by default, including headers, manifest, and padding. There is no separate limit on a state file. Use --max-bytes <positive byte count> on both commands to raise or lower the budget; restore also accepts older large backups when they fit the selected budget. An over-limit backup fails without replacing an existing archive. Restore keeps its manifest read limited to 64 KiB.
Backup checks space before copying the state database snapshot. Restore checks before each staged file and each replacement copy, including copies onto a different destination filesystem. These checks leave 64 MiB free beyond the file being copied. Allow space for the old installation, the full extracted archive, and a second copy of the restored files during replacement. Space checks cannot reserve storage against other writers; write failures still abort staging before live replacement. If a limit or space check fails, retain the original archive, provision enough storage, and retry with the same or a higher explicit budget.
Hardening
Operator-driven mitigations applied to the host. Run csm harden with no arguments to print the available subcommands on the current host (the audit detects kernel build, panel, and existing mitigations and only offers what’s relevant). Background, full list, and live-detection details: CVE Mitigations.
| Command | Description |
|---|---|
csm harden | Print the hardening menu for this host. |
csm harden --copy-fail | Apply the CVE-2026-31431 (Copy Fail) modprobe mitigation: blacklist algif_aead + af_alg, unload them. Refuses on built-in-AF_ALG kernels. |
csm harden --copy-fail-seccomp | Apply the CVE-2026-31431 seccomp mitigation: write systemd RestrictAddressFamilies=~AF_ALG drop-ins for LiteSpeed, Apache/Nginx, every PHP-FPM pool, cron, and mail units. The right path on built-in-AF_ALG kernels (typical cPanel/CloudLinux 8). |
The web UI’s Server Hardening page can also apply fixes one section at a time. A failing check that has a safe, mechanical fix shows an Auto-fix checkbox. Apply selected fixes changes only the checked items, then re-runs the audit. The page waits for the worker to finish. Only one hardening worker can run at a time, including across daemon restarts. It uses the daemon’s main configuration and drop-in directory. Applying fixes requires root and systemd-run.
- SSH
MaxAuthTries 4,X11Forwarding noandUseDNS nogo into a CSM block at the top ofsshd_config. The file is checked withsshd -tbefore sshd reloads, and restored if the check fails. The previous file is kept assshd_config.csm-bak. - Kernel settings are set with
sysctl -wand saved to/etc/sysctl.d/99-zz-csm-hardening.conf. A missing drop-in directory stops the fix before kernel settings change. /tmpand/var/tmpbecome1777owned by root./etc/shadowbecomes0600, or0640when the system shadow group owns it.- WHM tweak settings are changed through
whmapi1 set_tweaksetting, with 200 for max emails per hour. Compilers are restricted to root, and cPanel updates are set to daily. - PHP
expose_php,enable_dl, upload progress and fatal error backtraces are turned off inzz-csm-hardening.iniin each build’s scan directory, and in every SAPI’sconf.don Debian and Ubuntu. If a Debian or Ubuntu web SAPI cannot be updated, the CLI setting stays pending for retry. Linked or unreadable SAPI directories require manual changes. PHP-FPM restarts, Apache reloads when mod_php was changed, and LiteSpeed receives a fresh restart request on each apply.
SSH port, password and root login settings, disable_functions, allow_url_fopen, and the mail, web server and firewall checks stay manual.
Remediation
| Command | Description |
|---|---|
csm clean <path> | Clean infected PHP file (backs up original) |
csm db-clean --option <account> <option_name> [--preview] | Sanitize malicious WordPress option values (e.g. injected siteurl / home) |
csm db-clean --revoke-user <account> <user_id> [--demote] [--preview] | Revoke or demote a compromised WordPress admin and invalidate their sessions |
csm db-clean --delete-spam <account> [--preview] | Delete published posts whose title or content matches a high-confidence spam keyword. Matches are confirmed on a Unicode-aware word boundary, so a post is never removed for containing a keyword inside a longer word. Keywords that also occur in ordinary writing are reported by scans but never deleted. |
csm db-clean --drop-object <account> <schema> <type> <name> [--preview] | Drop a MySQL trigger / event / stored procedure / stored function, capturing its CREATE SQL into the db_object_backups bbolt bucket first. <type> must be trigger, event, procedure, or function. <schema> must match a database discovered for <account>. Daemon must be stopped. |
csm virtual-patch [--apply] | Re-scan web roots and preview reversible access-file deny rules. Requires root and manual or auto mode; a timed-out partial scan applies only findings already confirmed reachable and exits nonzero. |
csm enable --php-shield | Enable PHP runtime protection |
csm disable --php-shield | Disable PHP runtime protection |
Installing or enabling PHP Shield creates /opt/csm/shield.conf.php only
when it is missing, so your edits survive. Update its allowed IPs by hand
after changing infra_ips. Disabling keeps the file; uninstalling removes it.
A PHP pool with open_basedir must allow /opt/csm/, not just the shield
script, or the shield ignores /opt/csm/shield.conf.php and uses its
defaults. A .csm-shield-disable file only works where open_basedir
allows the account’s home folder.
State database
| Command | Description |
|---|---|
csm store compact | Reclaim unused space in the bbolt state file (atomic rename over the live DB). Requires the daemon to be stopped (systemctl stop csm) because bbolt holds an exclusive file lock while running. |
csm store compact --preview | Snapshot into a temp file next to the live DB and print src/dst sizes without replacing anything. Use to estimate reclaim before scheduling a maintenance window. |
csm store export <path> | Write a tar+zstd backup containing the bbolt store, the state directory, and the signature-rules cache. Private in-progress staging files and unconfirmed firewall rollback artifacts are omitted. A sibling <path>.sha256 companion file holds the archive hash for verification. Daemon must be running. |
csm store import <path> | Restore from a backup archive. Daemon must be stopped. Default restores everything; --only=baseline restores only state JSON files (file hashes); --only=firewall merges the firewall buckets into the existing bbolt in one transaction, where a durable firewall journal and its snapshot in the archive replace the target’s, and refuses while either side has a durable firewall action unresolved or its audit undelivered; --force-platform-mismatch allows restoring an archive captured on a different OS / panel / web server; --force allows an admission ledger behind this host’s live one (see Backup & restore). |
csm store reset-bot-verify | Drop cached bot PTR verification results and missing-PTR records so the next scan re-runs reverse DNS checks. Requires the daemon to be stopped because bbolt holds an exclusive file lock while running. |
csm export --since <when> | Dump audit-log events for SIEM backfill. <when> is RFC 3339 (2026-04-01T00:00:00Z) or a duration relative to now (24h, 7d). One JSON event per line on stdout, in the same v=1 schema the live audit_log sinks emit. Pipe to a file or directly into a log shipper. Daemon must be running. |
Updates
| Command | Description |
|---|---|
csm update-rules | Download latest signature rules |
csm update-geoip | Update MaxMind GeoLite2 databases |
csm update-bot-ranges | Refresh built-in AI-crawler IP ranges from vendor feeds |
PHP-relay (mail abuse, cPanel only)
Operator controls for the email PHP-relay detector. Talks to the daemon’s control socket; the daemon must be running. See Real-time detection for what the detector fires on, and Auto-response for the freeze action.
| Command | Description |
|---|---|
csm phprelay status | Print the detector’s current state as JSON: enabled, platform, effective dry-run + source (runtime/bbolt/csm.yaml), Path 2b effective account limit, scripts/IPs/accounts tracked, msgID-index size, active ignores. Use to confirm the watcher is wired on a fresh install. |
csm phprelay ignore-script <scriptKey> [--for-hours N] [--persist] [--reason ...] | Suppress every path for a host:/path scriptKey. Default TTL 168h (7d). --persist writes to the bbolt phprelay:ignore bucket so the suppression survives daemon restarts; without it the entry is in-memory only. <scriptKey> is the value the daemon prints in email_php_relay_abuse and email_php_form_relay_script findings (e.g. shop.example.com:/wp-admin/admin-ajax.php). |
csm phprelay unignore <scriptKey> [--persist] | Remove an active ignore. --persist also deletes the bbolt row. |
csm phprelay ignore-list | List all active ignores as JSON: scriptKey, expiresAt, addedBy, reason. |
csm phprelay dry-run on|off|reset [--persist] | Override the auto-freeze dry-run state at runtime. on = freeze findings emitted but no exim -Mf runs; off = live freezes; reset clears the runtime override and falls back to bbolt or csm.yaml. Precedence: runtime > bbolt > yaml. --persist writes the on/off choice to the bbolt phprelay:settings bucket so it survives restarts; on reset --persist the bbolt row is also deleted. |
csm phprelay thaw <msgID> | Manually thaw a frozen Exim message. Wraps exim -Mt with msgID validation (rejects anything that isn’t [A-Za-z0-9-]{16,32}) and writes a thaw entry to the auto-freeze JSONL audit at /var/log/csm/php_relay_audit.jsonl. |
Antispam learning (cPanel only)
Teaches or corrects learning from Spam folders. Runs as root against the running daemon while learning is enabled; observe mode refuses it.
| Command | Description |
|---|---|
csm antispam learn --spam <file>... | Learn each saved message as spam; every cluster it matches gains one operator mark. |
csm antispam learn --forget <file>... | Remove the learned clusters each message matches, including every account’s contribution to them. |
csm antispam learn --status [<request-id>...] | Show what the daemon recorded for request IDs, including clusters still publishing; without an ID, list unfinished requests. |
csm antispam learn --fresh-namespace | With learning disabled, move the learning keys and stored messages aside and empty the learning state; the next enable starts a new namespace. |
Each file gets a request ID, printed before it is sent; csm actions --op integrate.antispam_learn --json shows its recorded outcome and full digests. A lost or truncated reply is retried once with the same ID and snapshot. The exit status is 0 only when every file was learned, already learned, forgotten or matched nothing.
Stalwart
Prepares a Stalwart mail server for CSM through its admin API. See Preparing Stalwart.
| Command | Description |
|---|---|
csm stalwart setup [--dry-run] [--password-file PATH | --password-stdin] [--url URL] [--account-domain NAME] | Make Stalwart log failed logins and the mail its spam filter rejects or discards, end its bans with CSM’s blocks, and take webmail client addresses from the web server behind CSM’s port lock. Create the role CSM response, the service account csm-response and its API key, which works only from this host and is written to stalwart.api_key_file; the account has no password and can receive mail. A role that lacks CSM’s permissions is reset to CSM’s list, dropping permissions added by hand. Setup refuses an existing csm-response account that has a password or another sign-in credential, has sign-ins stopped, or holds keys without CSM’s role. --account-domain names the account’s domain on a server with several. On a server with tenant administrators, use a domain only the server administrator controls, since whoever can edit the account could give it a password. A key Stalwart accepts is kept while it is the account’s only key; to rotate it, delete the key file and run setup. The key steps run only when the API address is on this host. Also prints whether the spam filter is on and its scores. Adds the spam rule CSM_IP, which scores senders on CSM’s address list so more of their mail goes to Junk, or is rejected or discarded at the scores set in Stalwart; it is the one step that changes how mail is handled, and switching the rule off in Stalwart stops it. An unreadable spam filter status warns without stopping setup. Prints each step as name: state: detail. The admin login comes from STALWART_USER and STALWART_PASSWORD, or the first line of --password-file. --password-stdin reads the password from the first line of stdin instead, with the user from STALWART_USER or admin. On a terminal, setup prompts for missing values and hides password input even with --password-stdin. When you enter both values, the username is hidden too. Press Enter at the user prompt for admin. Ctrl-D at an empty prompt cancels setup. Without a terminal, setup reads stdin only with --password-stdin. --url overrides stalwart.api_url. Before sending the login to an address on this host it checks that root or Stalwart’s service user holds that port and owns each connected peer; unreadable ownership records stop the login. Exits 1 when a setting is unsafe, other Stalwart settings cannot be read, the key file cannot be used or checked, or a change failed. Exits 2 on a usage or config error, an API address that is refused, a missing or refused login, a port held by another user, or a host that is not a Stalwart host. |
Firewall
See Firewall for the full reference.
csm firewall status
csm firewall deny <ip> [reason]
csm firewall allow <ip> [reason]
csm firewall tempban <ip> <dur> [reason]
csm firewall deny-subnet <cidr> [reason]
csm firewall grep <pattern>
csm firewall flush
csm firewall apply-confirmed <minutes>
csm firewall confirm
csm firewall restart
csm firewall rollback status|confirm|revert
# ...
csm firewall apply-confirmed and csm firewall restart refuse a pending
CLI confirmation window and say whether waiting for its rollback still helps.
If startup cannot read its saved configuration, rollback restores the kernel
ruleset and state and the window stays pending; once that restore is
recorded, later restarts keep the blocks added since and apply the
configured settings. csm status and csm doctor show the window. Inspect
the current firewall before csm firewall confirm, which accepts the
current ruleset and engine configuration; it does not recover unreadable
settings. Or repair the saved configuration and restart the daemon, which
restores it and rebuilds the ruleset with the current blocks; removing it
does not end a recorded window, even one whose marker was empty or corrupt.
See CLI confirmation window. The
separate Web UI config timer uses csm firewall rollback.
Real-Time Detection
CSM detects threats in under 2 seconds using kernel and log watchers running inside the daemon: a fanotify file monitor, inotify tailers on auth/access/mail/FTP logs and the Exim spool, and a PAM brute-force listener.
fanotify File Monitor (< 1 second)
Monitors the filesystems containing /home, /tmp, /dev/shm, /var/tmp, configured account_roots, and detected cPanel document roots.
A filesystem mark covers the whole filesystem, including all its bind mounts,
which makes writes inside CloudLinux CageFS visible. Roots on that filesystem
share one mark. A root being a mount point does not restrict a filesystem mark
to that path: symlinks and bind mounts can name only a subtree. On kernels that
fall back to mount scope, the mark covers the whole containing mount and misses
writes through other bind mounts. The startup log describes these scopes and
which roots share a filesystem mark, and says when a watch root sits on the same
filesystem as /, which is when the mark raises an event for every write on the
machine; failures to inspect or mark a root are reported even when other roots
succeed. Missing roots are skipped.
The daemon filters delivered file events before queueing them for analysis. In
the shared temporary trees (/tmp, /var/tmp, /dev/shm) that decision uses
the event descriptor. Non-directory objects with executable permission bits are
queued, as are watched user crontabs, PHP source, configuration files such as
.htaccess, .user.ini or php.ini, known webshell names, files under .config
directories, and images inside hosted trees. Writes needed for self-deleting-file
tracking are also queued. Other writes are dropped without analysis.
Files retained for tracking keep their normal analyzer route,
including suppression and WordPress checksum checks. Staging names written by an
atomic save are resolved first, so a configuration file saved through one is
still analysed. A descriptor that cannot be inspected is analysed rather than
skipped.
csm_fanotify_events_total, csm_fanotify_events_admitted_total, and
csm_fanotify_events_dropped_total distinguish completed admission decisions,
queued events, and events lost to a full analyzer queue. Subtract both admitted
and dropped events from the received total to count events rejected before
queue admission, including those whose file path could not be resolved.
Atomic-save files enter the same bounded worker queue as ordinary writes. The watcher receives close-write events, not create events. Scans read the original event descriptor even if the file has been renamed, replaced, or deleted before analysis. No rename event is needed to inspect those bytes. Repeated findings use the normal alert cooldown, and queue overflow schedules a directory rescan of files that remain on disk. That recovery runs one pass at a time, under a time budget and with a minimum gap between passes. Unfinished directories stay queued and retry even after new drops stop. Recovery resumes within a directory and retains its original scan window so deferred files do not age out while waiting. Kernel notification loss still relies on the next deep scan. Shutdown drains the queue under a fixed time budget: events still waiting when it expires are released without being scanned; files still on disk are covered by the next deep scan. Both budgets are checked between scans, so a scan or directory read already in progress must still finish before shutdown completes. A file renamed or hard-linked into a watched tree, or a directory moved into one, raises no close-write event at its new path. The placement monitor covers those when a placement backend runs; otherwise they are left to the deep scan.
Realtime YARA scans hand the worker a sealed memory-backed copy of the
captured bytes, whatever their size, without reopening the event path for
scanning. Alert payload details describe that snapshot. When it is the whole
file, the finding keeps the snapshot’s fingerprint. For partial reads, CSM
reopens the path to fingerprint the whole file when preparing the finding;
unreadable files and files above the fingerprint limit receive no fingerprint.
Re-check compares whole-file fingerprints, so an unchanged file can clear when
YARA no longer matches it. Content outside the read window remains subject to
the scheduled scan limits.
A failed scan raises yara_realtime_scan_error.
For WordPress atomic saves, the intended basename can select a core or plugin checksum entry. Only a match against the complete event-file content verifies the file; missing checksums, partial content, and modifications proceed through normal detection. The finding retains the actual event path.
A WordPress update is unpacked under wp-content/upgrade/ before it is moved
into place, and each staged PHP file is judged by hash rather than by location.
A file that matches the official wordpress.org checksum for its package version
is stock and skips detection like an installed stock file. A file whose
checksums are still being fetched is content-scanned now and compared once they
land, even if the package header was written late or WordPress has already
moved the tree into place. An installed plugin header can identify the release
only when its directory is the original staging directory moved into place, or
when the installed path is still the very file the staged event wrote, so a
stock file first examined after the move skips detection like any other.
The installed plugin directory must remain the same directory during that
check and cannot itself be a symlink. Its identity also connects earlier
queued files from the same tree, including when the later file is already
verified. Copies and unrelated trees do not establish that connection; every
retained digest still needs to match the official release.
A core update copies files, so the installed version.php identifies the
release only if it was replaced or rewritten after the staged file was seen;
a borrowed release still requires the file’s bytes to be the official ones for
that path. A tree removed before its release could be identified gets a
package warning, not file-by-file comparisons against the old release. This
covers refused updates (for example when the new release needs a newer PHP)
and plugin copies whose staging identity was never captured; the warning does
not establish whether installation succeeded. Reused staging paths follow the
normal alert cooldown rather than suppressing later unidentified uploads. A
file the official package does not ship gets its
own warning. An event identified at its installed location keeps that location
while it exists, even if another update reuses the staging name. Other staged
events name the installed path when it exists. Themes, packages
not published on wordpress.org, a full verification queue, and a package whose
checksums do not arrive within 60 seconds raise one warning per staging
directory. WordPress picks a new staging directory for every upload. Once the
package type and version are known, these warnings are identified by site,
package type, name, version and reason. Uploading the same declared release to
the same site again is a repeat. It is emailed again only after 24 hours without
a report, and history follows the normal 24-hour reminder. This
groups checksum-availability warnings; it does not establish that the package
contents are unchanged or safe. Incomplete headers keep an upload-specific
warning, and a later change of type, version or reason can raise a new warning
in the same directory. Per-file checksum warnings repeat across uploads only
when their package identity, relative path and complete digest match; files
that could not be hashed keep their upload-specific warning. Content findings
inside staging keep their own per-file identity and normal alert cooldowns;
package-warning dismissal does not dismiss them.
Location warnings for PHP in sensitive WordPress directories are suppressed only for recognized inert content, including comment-only stubs and literal translation or version data. PHP attributes are executable syntax, not comments, and do not qualify a file for that suppression. Content scans still run.
The head-window eval/decoder check ignores PHP comments and inline HTML. String boundaries and legacy line endings preserve the code that follows them. Tail windows retain raw content because they can begin inside a literal whose opening delimiter is outside the window.
A Critical obfuscated_php_realtime, webshell_content_realtime or
php_dropper_realtime alert whose file has not already had a file response
queues the scheduled content analysis of that file, on one background worker
so slow storage cannot delay other alerts. A Critical result goes through
normal dispatch and the scheduled response, under quarantine_files and the
file response budget. Links in any path component, special files and a file
that changed after analysis are refused. A full queue is logged; the original
alert stays.
Detects:
- Executables in shared temporary directories are reported even when named
.htaccess,.user.ini, orphp.ini, including atomic-save names. Root-owned files with panel maintenance ancestry, package-manager ancestry during an active package window, or verified live root ancestry of an initramfs build, receive Warning. Initramfs provenance requires both real and effective root credentials from the writer through the builder; cached names or unknown credentials do not qualify. The writer’s ancestry is read when the write is seen. Build tools often exit before then; such a file receives the same Warning only when a writer with proven ancestry recently wrote an executable into the same work directory, or a root process with proven ancestry is working inside it, and the executable and every directory below the temporary root are owned by root and cannot be written by other users. An occupant’s actual working directory and every directory back to the work directory must also be root-owned and not writable by other users, without crossing a mount. Occupant scans are spaced per work directory; when scan capacity is full, findings without other proof stay Critical. The vouch is tied to the open file and its directory placement; symlinks, mounts below the temporary root, renamed or replaced files and directories, and a file that was not owned by root when it was written cannot carry that proof. A file the build itself has already removed keeps it. Its lifetime starts when the proven write was seen, and exited writers cannot extend it. Read errors and a known non-root process in the ancestry do not qualify as an exited root writer. The finding of an exited writer with no such proof says so and stays Critical, as does any writer that is still running and proves nothing. Those warnings are grouped: one finding stands for a work directory while maintenance writes into it. Files directly in a shared temporary directory retain separate warnings. Other temporary executable findings remain Critical and are never grouped; an earlier maintenance warning cannot suppress them. Configuration content checks still run, and operator path suppressions still apply. - Webshell creation (PHP files in web directories)
- Self-deleting droppers: a PHP or executable created under a document root and unlinked within
thresholds.dropper_unlink_ttl_sec(default 300s), the loader technique that creates a rogue admin then erases itself before any scan. A file whose exact bytes WordPress moved or copied into place before removing the original is not reported: plugin, theme and core packages, language packs, and the version file a core update reads first. Language-pack and version-probe copies also require a complete, stable data-only snapshot with no suspicious content or earlier unsafe write. A core update that stops after reading its version file leaves the old release installed, so no installed copy matches; that version file is not reported when its bytes equal the wordpress.org checksum for the release and locale it declares. Core checksum downloads start in the background when capacity permits. Plugin downloads do not consume core checksum capacity or share its admission limit. Concurrent requests for a release reuse its download and cached result; a file still unverified at the deletion probe is reported. Package headers are read only from regular files, never blocking on an account-writable named pipe or device. Other executable files do not qualify for this copy exception. The script Really Simple Security copies into uploads to test whether PHP runs there, then deletes, is not reported when its content is byte for byte the shipped script; any other content under that name is. Elementor Safe Mode copies its loader into mu-plugins while enabled and deletes it when switched off; that removal is not reported when the loader’s bytes equal the wordpress.org checksum for the same file in the Elementor release the same site had installed when the loader was written. When those checksums cannot be consulted, a loader byte for byte identical to a complete, stable snapshot of the installed plugin’s own file, reached without symlinks, is reported as Warning and the finding names that file. The same package proof applies when a different file replaced the loader’s path. Other capability-probe scripts cannot exempt executable content in this loader. A loader the checksums reject, or whose earlier payload was rewritten, stays Critical even when its containing directory was removed. A core update unpacks the whole release but installs only what changed and never overwrites the bundled themes and plugins; a staged file it deletes that way is not reported when its bytes equal the wordpress.org checksum for that path in the release now installed. The checksum must describe the same complete, stable snapshot as the retained content. Earlier nonempty content must also be known and unchanged, including the first close-write snapshot; later official bytes or an identical installed copy cannot erase that evidence. The same history rule applies to every staged core, plugin and theme file and to the Safe Mode loader: a file whose content changed after it was first written is reported even when WordPress then moves it into place, an identical copy is installed, or its last bytes are official. Repeated analysis of the same snapshot does not count as another write. A single snapshot whose complete bytes could not be read can still match a move by file identity; a second nonempty snapshot leaves the unknown earlier bytes unaccounted for. Files too large for a complete snapshot retain their leading bytes for content checks. Other files removed from staging without an identical installed copy are still reported. Upgrade staging, atomic-save temp files, template compile caches, BackWPup job-state files (the running job’s JSON and its folder list, written behind a PHP comment, including a captured head cut inside a CRLF line ending), a path taken over by a newer file, and a file whose original directory was removed are recognized and reported at a lower severity; files removed together with the directory holding them, such as a language pack WordPress unpacked and discarded, are reported once for that directory; a create/delete burst collapses into one lower-severity notice. Grouping is recalculated after ignored paths are removed: remaining files can form directory groups, while a lone remaining file is reported on its own. Off withthresholds.dropper_detection: false. Candidate tracking and findings held for aggregation each accept up to 16,384 entries; additional work is refused without evicting older evidence. Status and doctor report refusals, exhausted probes, overdue work and stalled processing through protection queue health. - Dropper admission and pending findings follow the live
suppressions.ignore_pathslist after reload. Intentionally suppressed candidates consume no tracker capacity; losses of eligible candidates still raise the capacity warning. - PHP in uploads, languages, upgrade directories
- PHP in
.ssh,.cpanel,.gnupgor.cagefsdirectories, or in themaildirectory of an account home, including cPanel accounts on further home partitions such as/home2(critical escalation). Account mail locations ignore redundant separators but keep Linux directory-name case sensitivity. Directories namedmailinside applications or document roots do not count; their PHP content is still scanned. - Executable drops in
.config .htaccessinjection and tampering (auto_prepend, eval/base64 handlers, CGI execution remaps, and ModSecurity disablement). Handler abuse keeps its own finding name. These findings never quarantine the file; automatic cleaning waits for a deep scan and requiresauto_response.clean_htaccess..user.initampering andphp.initampering under configured or detected web roots- Obfuscated PHP (encoded, packed, concatenated)
- Fragmented base64 evasion (
$a="base"; $b="64_decode"– function name split across variables) - Concatenation payloads (hundreds of
$z .= "xxxx"lines with eval at end) - Tail scanning: payloads appended to the end of large legitimate PHP files (beyond the 32KB head window)
- CGI backdoors: Perl, Python, Bash, Ruby scripts in web directories (e.g., LEVIATHAN toolkit)
- SEO spam: gambling/togel dofollow link injection in PHP/HTML files
- Phishing pages and credential harvest logs
- Phishing kit ZIP archives
- PHP carried inside a file served as an image. Image writes under an account or configured document root are inspected, including hosted
.configdirectories and roots located in temporary directories. Images up to 128 KiB are inspected in one piece; larger files get a 64 KiB head and a 64 KiB tail scan. Payloads outside those windows require a deep scan, subject tothresholds.full_scan_max_file_mb. A PHP opening tag alone is not enough – a screenshot quoting one in its description chunk stays quiet – and a file that only wears an image name while holding PHP source is reported the same way. Finding:php_in_image_realtime. - A PHP file that includes or requires an image, archive or other non-executable file while also reading request input. That pairing is the loader half of the technique above: the payload lives in the picture and the one-line loader elsewhere. Literal targets ending an include statement, concatenated paths, paths held in a local, encoded paths and suppressed
@includestatements match in either statement order. Ordinary templating that pulls in.html,.tpl,.txtor.svgpartials is outside the extension-based rule. Encoded targets still count because the source hides their extension. - YAML signature matches (PHP, HTML, .htaccess, .user.ini, php.ini)
- YARA-X rule matches (if built with
-tags yara)
Self-deleting file tracking requires a close-write observation and a recent filesystem birth time. Filesystems that do not provide birth time cannot support this detector; ordinary realtime content scanning still runs. Completed upload execution probes remain tracked until the deletion check. Earlier unsafe or uncertain content remains reportable even when a later close-write contains a harmless probe. Image writes use the existing notification-only fanotify instance, its 16,384 event queue and 4-16 analyzer workers. They create no permission-event holds and do not enter the mail scanner. Thumbnail and WebP bursts fill the same queue as other writes: excess events are closed immediately and their parent directories enter the existing capped recovery tracker. Recovery rescans recent surviving files; kernel queue loss and exhausted recovery coverage still rely on the next deep scan. Queue health and content-read truncation remain visible through the existing metrics. Mail permission holds retain their separate hold budget and watchdog.
Both the real-time and the scheduled WordPress admin-creation signature require an administrator role token plus literal or request-derived credentials, and both accept the same ASCII whitespace, including the vertical tab. An importer that creates users with a generated password does not trigger either one just for reading a login from an import form.
Two divergences between the engines are known and left in place. Go folds the Unicode long s into ASCII s and YARA’s nocase does not, so a token spelled with it can match in real time where a scan declines. Bounded expressions also count differently: the real-time engine counts characters and the scheduled one counts bytes. Neither can occur in real PHP, and both belong in the scanner’s regex compilation rather than in hand-written escapes inside every rule. Strictness parity is not enforced across the rulesets: the parity check compares rule names, not how strict each side is.
Complete blank files are excluded from dropper alerts after a close-write observation. Any content or metadata change during the read, including an unlink, leaves the snapshot uncertain. WordPress data and its checksum are read within the same stat interval. A later complete close-write snapshot cannot rule out code that ran during an earlier unstable read. That uncertainty and code seen in any read remain attached to the tracked file. Concurrent firewall-log writes can therefore still produce a warning when the log is replaced, or a critical finding when it is deleted.
Core-release checksum downloads are non-blocking and bounded, including their retry timers. Repeated misses and streams of distinct release names share an admission budget. Release identifiers longer than 64 bytes are rejected before retention or lookup. When that budget is exhausted, cached checksums remain usable; an uncached file stays unverified and receives the normal dropper assessment. A later observation can request checksums again after the budget recovers.
PHP files are also excluded when their first statement stops the interpreter
(exit, die, or __halt_compiler, with at most a plain literal argument and
no preceding comment). Plugins keep state and firewall data in files of that
shape and rewrite them constantly, and the bytes behind the terminator are
never compiled. The exemption is refused when those trailing bytes are made
only of transport-encoding alphabet: an operator who turns on PHP source
conversion can have a file decoded before it is tokenized, which would make an
encoded tail the program and this header padding. Comment-bearing PHP remains
eligible for the same reason – a comment’s tokens depend on the interpreter’s
source encoding. Executable-mode files are not judged by PHP compilation rules,
because a shell may read them instead. Content or signature findings and
previously observed code always override inert-content filtering.
Features:
- Per-path alert deduplication (30s cooldown)
- Process info enrichment (PID, command, UID)
- Auto-quarantine on high-confidence realtime signature matches (category, size, entropy, and hex/execution validation)
- Auto-quarantine on confirmed PHP request-to-execution flows and executable packed payloads, using the stable file version that was inspected
Files Moved or Linked Into a Site
A file written elsewhere and then renamed or hard-linked into a site raises no
close-write event at its new path. The placement monitor reports these files.
Each one goes through the same admission, analysis and response as a write at
that path, including quarantine of confirmed shells, whatever the file’s age.
The monitor rejects a symlink at the final path component and verifies the
file’s identity before analysis. With bpf, that identity comes from the
kernel event. With legacy, the event identifies the parent directory and
entry name; CSM records the file’s identity when it resolves that entry. A
replacement made before that lookup can be analyzed, but a replacement made
after it fails the identity check. The event keeps the placing process’s PID
for restore detection; legacy cannot prove that process placed a replacement
found during lookup.
detection.placement_backend selects how the monitor sees placements. It
takes effect after a restart.
bpfuses BPF LSM rename and link hooks. It sees renames, including both sides of a rename exchange, hard links, including a temporary file published withlinkat, and directories moved in. It needs a BPF-tagged build and a kernel that can attach BPF LSM programs. Chosen explicitly, it has no fallback.legacyuses a second fanotify group that reports directory entries, on kernel 5.9 or newer. It sees files and directories moved in, and new entries, which include hard links and temporary files published withlinkat. A new file that its own close-write already analyzed is not analyzed twice.auto(default) triesbpf, thenlegacy, then runs without a backend.noneturns the monitor off. Placed files are left to the deep scan.
Without a backend the daemon logs a startup warning that files moved or
linked into watched trees are checked only by the deep scan. The status
response names the selection under backends.placement (bpf, legacy or
none), and csm doctor shows it as the backend: placement check. That
check warns on none unless the config chose it. GET /api/v1/capabilities
lists bpf-file-placement while the BPF backend runs. A failed start raises
bpf_unavailable, as for the other BPF monitors. It is High when an explicit
bpf choice cannot start, or when an explicit legacy choice or auto fails
for an unexpected reason, and a Warning when auto falls back to legacy. A
kernel that supports neither backend under auto, or lacks the legacy
backend when it is chosen explicitly, shows none without a finding.
Placements that do not fit the monitor’s queue go to the same recovery as a full analyzer queue: CSM rescans the directory each file landed in, whatever the files’ age, and queues a moved-in directory for its walk. BPF recovery allows the rename or link to finish before completing a rescan or starting a walk, and covers placements in the watched trees only: a file or directory inside a watch root, or a directory that holds one. Recovery queues remain bounded; evictions and unfinished work count as drops.
When a watched filesystem has several whole-filesystem mounts, BPF checks each watched location separately. Admission uses that location’s name, and a child mount hiding one view does not prevent checks through another view.
A directory moved into a watched tree is walked, because its files raised no
events of their own. Each file gets the same identity check and analysis as
a placed file. The walk never follows symlinks, runs on its own worker, waits
for analyzer room so live writes keep priority, and works through a large
tree in bounded passes. When realtime cannot finish a directory, because it
holds too many entries, too many moved-in directories are waiting, the
analyzers stay busy for a whole pass, or entries cannot be read, it raises
one High placed_dir_incomplete_realtime naming the directory. The next deep
scan covers the files it did not check.
placement_unresolved_realtime reports what the monitor cannot see, with
either backend. It is High for a moved or linked file whose path realtime
could not resolve, such as a path deeper or longer than realtime protection
follows or a directory the legacy backend could not open, and for a watch
root the running backend cannot see. Unresolved files repeat the finding at
most once per alert cooldown for each device. A file or directory removed
before realtime reached it raises nothing. One Warning per start names the
roots on a filesystem this kernel cannot watch for placements. The deep scan
covers those files. A kernel queue overflow in the legacy backend raises
fanotify_kernel_overflow.
A PHP file saved under an atomic-save temporary name and then renamed to its
final name is not scanned twice. A recent clean scan is reused only when the
file’s content, size, modification time, permissions and owner match that
scan, its final name gets the same analysis, and no setting or rule changed
since. Executable files and files too large for a complete read are always
analyzed. csm_fanotify_placement_skipped_total counts those reuses, and
csm_fanotify_placement_unverified_total counts placements that failed the
identity check. Queue health for the monitor appears under bpf.placement and
placement.dir_walk in protection queue health.
Sensitive System File Watcher
Tracks a fixed set of system-configuration paths: the account and credential databases, sudoers and its drop-in directory, the SSH daemon config and its drop-in directory, the system cron drop-in directories, and per-user crontabs. The set is not operator-configurable – a path an attacker knows is excluded is a free landing pad.
The backend is chosen by detection.sensitive_files_backend. The bpf backend attaches an LSM hook that catches writes as they happen, and re-expands the watchset every detection.sensitive_files_poll_interval (default 5m). The legacy backend content-hashes the watchset on the same interval.
Both backends key on the path, not the inode. Most tools replace a config file by writing a temporary file and renaming it over the target, which gives the path a new inode and hides the write from the LSM hook. The BPF refresh therefore compares content and security metadata per path: a rename-over is reported as a change on the existing file, only a path no previous refresh had seen is reported as newly appeared, and a regular-file rewrite that leaves bytes, permissions, and ownership unchanged is not reported. It also tracks symlink targets, so retargeting a watched path remains visible even when both targets contain the same bytes. Non-regular entries are tracked by path identity without opening them for content reads. The legacy poller compares content digests by path.
Findings are sensitive_file_modified. A write finding successfully delivered by the LSM hook wins only while the path still has the exact state captured for that finding. The refresh reports a later rename-over separately, retries a refresh finding if the alert queue was full, and always evaluates the bytes used for its digest rather than a later read. Writes inside a package-manager window, or by a process whose ancestor is a package manager or the control panel’s own maintenance tooling, are demoted to Warning unless a cron payload carries obvious persistence tokens. This veto applies to live writes as well as added and modified cron files found by polling. It includes the shared crontab persistence patterns and their bounded base64 inspection, in addition to suspicious shell fragments. A modified file in /etc/cron.d found by the refresh, the legacy poller or the scheduled cron check is also demoted when the only difference from the version CSM saw last is the minute and hour of jobs that already ran once a day at a fixed time, which is how cPanel reschedules its own jobs every night. A changed command, argument, user, environment line, job list or line order, a job that now runs more often or on other days, and a file CSM has not seen before all keep the finding High, and all three paths also refuse schedule-based demotion when the file’s ownership, mode or symlink target changed in the same interval or could not be checked. Files with randomized calendar fields do not qualify: reloading them can choose new execution days, including for jobs whose bytes stayed unchanged. Persisted schedule evidence must belong to the recorded content baseline; an interrupted update cannot justify demotion. Control-panel ancestry is read from the executable a parent process actually runs, not from the name it reports, and the executable must be a root-owned file inside the panel’s own directory that no other user can write. Cached executable evidence must come from a resolved procfs link; an exec-event filename or script argument does not qualify. Ancestry is available on every Linux build before monitoring starts. A cached control-panel hit avoids a live ancestry walk; otherwise CSM also reads procfs and keeps cached package-manager evidence. CSM’s own managed writes are suppressed by content, not by path.
inotify Log Watchers (~2 seconds)
Tails auth, access, and mail logs in real-time. The exact file paths are chosen per platform at daemon startup – see the platform: ... line in the daemon log.
| Log | Platforms | What it detects |
|---|---|---|
cPanel session log (/usr/local/cpanel/logs/session_log) | cPanel only | Logins from non-infra IPs, password changes, File Manager uploads, sessions ended for security token failures |
cPanel access log (/usr/local/cpanel/logs/access_log) | cPanel only | cPanel-API auth patterns |
| Auth log | All | SSH logins and failures. /var/log/auth.log on Debian/Ubuntu, /var/log/secure on RHEL family and cPanel |
Exim mainlog (/var/log/exim_mainlog) | cPanel; non-cPanel when the file exists | Mail anomalies, queue issues, SMTP brute force, probe abuse, and cloud relay abuse |
| Apache/LiteSpeed/Nginx access log | All | WordPress brute force (wp-login.php, xmlrpc.php), real-time. Paths: /var/log/apache2/access.log (Debian), /var/log/httpd/access_log (RHEL), /var/log/nginx/access.log (Nginx), /usr/local/apache/logs/access_log, then /var/log/apache2/access_log and /etc/apache2/logs/access_log (cPanel, also written by LiteSpeed). Automatic candidates exclude the cPanel access log |
| Mail log (platform file or journal) | All hosts with Postfix/Dovecot logs | IMAP/POP3/ManageSieve account compromise and mail brute-force; SMTP brute force on Postfix hosts with Dovecot |
FTP log (/var/log/messages) | cPanel only | FTP logins and failures |
| ModSecurity error log | All (if ModSec installed) | WAF blocks and attacks. Auto-discovered from the detected web server |
Nginx error log (/var/log/nginx/error.log) | Nginx hosts | General web errors, ModSecurity denies |
web_server.access_logs replaces the automatic web log candidates. The
realtime watcher selects the first existing log at startup. If none exists,
it retries every candidate in order until one can be opened, then watches
that path until restart. The periodic WordPress check uses the first
candidate with lines and excludes its symlink aliases from the per-vhost
scan; other candidates can still be scanned when a per-vhost glob matches
them. Reputation collection scans all listed web logs and deduplicates
addresses, while the cPanel log remains a separate source for panel checks.
cPanel session log readers match fields by position, so login names and address lists cannot supply the service, address, event or account. Scheduled and realtime login alerts skip loopback and non-address origins. API-created sessions never count as direct logins for multi-address or hijack correlation, and a login that names its method twice counts as a direct login. A password change entry with an unreadable header or timestamp cannot suppress a shadow-file alert.
A cPanel API authentication failure is a 401 or 403 answer to a UAPI, JSON
API or XML API call, a WHM DNS cluster call, or another endpoint that accepts
API tokens (WebSocket, account transfer, WP Toolkit, Nova), percent-encoded or
not. Page requests on a session URL are not API calls and are not counted.
A failure on a session URL (/cpsess<token>/...)
waits a few seconds for the session log. A browser tab left open after the
account logged in again elsewhere still sends its old URL token, and cPanel
ends that session after a few token failures. When the session log records
that for the same address, the failures that name the session’s account, and
the tab’s later requests that reuse its URL token, are not reported: nothing
guessed a credential. Any other failure, including one from another client
behind the same address, is reported as before. API calls with a token or
password carry no session URL and are reported at once. The periodic
api_auth_failures check counts failures the same way.
Password-change suppression applies only to confirmed 401 responses. Ambiguous authentication failures remain visible.
The wait covers both matching-window edges and the log watchers’ polling delay. It ends at a fixed deadline. Evidence read after that deadline cannot erase a failure. Correlation evidence expires independently of alert delivery, and queue saturation is reported through the existing queue health counters and daemon warnings.
A frozen Exim message is reported once. Exim logs a still-frozen message again on every queue run, and those lines are not reported again, also after a daemon restart: the daemon keeps the messages it reported in its state database until a day passes without a queue run logging them. A message that froze while the daemon was stopped has not been reported, so the first queue run after the start reports it.
Startup discards expired records and invalid message IDs, and adjusts timestamps ahead of the current clock. These corrections are saved even if no new mail event arrives, so another restart does not extend the corrected retention period.
FTP findings use only the peer field directly after the pure-ftpd program tag,
in both realtime monitoring and scheduled scans. Missing or malformed peers and
hostname peers produce no finding. To record numeric peers, use pure-ftpd’s
existing DontResolve yes setting. Addresses in usernames or later message
fields never identify the client.
Successful FTP logins over loopback do not raise an unfamiliar-address warning. Failed authentication remains reportable over loopback, including through local relays.
The FTP and SSH log watchers read the same files as the periodic ftp_logins
and ssh_logins checks, so both see every login. Each login is reported once:
the watcher and the periodic check build the same finding, and the second one
is recognised as a repeat. When the watcher is not running, on a non-cPanel
host or before the log file appears, the periodic check still reports the
login on its own.
An SSH login is reported only from sshd’s own success record: the sshd program
tag, then Accepted <method> for <user> from <address> port <port>. sshd also
logs the login names that clients offer, so the same words anywhere else in a
line are ignored and cannot name an address to alert on or block. The
reputation check uses the same record parser when collecting SSH addresses.
Traditional syslog, RFC3339 timestamps, and records without timestamps are
supported, with either sshd or sshd-session tags and optional process IDs.
IPv4 and IPv6 addresses are accepted; trailing key information does not change
the account or source address.
The identity uses the complete log record, including its timestamp and session fields, even when the displayed details are shortened. A later session from the same address is a new finding, even within the 24-hour history reminder window.
A successful FTP login and a cPanel File Manager write are not emailed. On shared hosting every customer connects from an address that is not infrastructure, so both fire on ordinary use of a core feature. They stay on the findings page, in history, in incident correlation and in the attack database, where their value is correlation with other evidence on the same account. Failed FTP authentication, FTP brute force, a login from a brute-force source, and SSH logins are all still emailed.
The phpanel webhook and SSE event stream still receive these successful operations; the email and operator-webhook filter runs after data delivery.
Login check names on upgrade
New findings use ftp_login in place of ftp_login_realtime, and
ssh_login_unknown_ip in place of ssh_login_realtime. Update external
phpanel, export and SIEM rules to accept the merged names. Historical records
and already queued deliveries can still contain the old names; the JSON schema
is unchanged.
Existing alerts.email.disabled_checks entries and saved suppression rules for
either retired name also match its replacement. The Settings page displays and
saves the current names. A mute for one former producer now covers the shared
finding from both producers; it does not mute FTP brute-force escalations.
Pending SSH blocks and restored SSH incident evidence retain their blocking policy. Historical successful FTP activity remains excluded from incident blocking, including events carrying its retired name.
cPanel-only log watchers are not registered on non-cPanel hosts, so you will not see “not found, retrying every 60s” warnings for them on plain Ubuntu or AlmaLinux.
The Postfix/Dovecot file reader polls every two seconds. It reads replacement
files from the start and rewinds when the current file shrinks below its read
position. Truncation also clears buffered bytes from the previous file contents.
With copytruncate, a file that regrows past that position between polls can
hide the truncation and lose events. Use rename/create rotation with the log
writer reopening its file, or journal input, when that loss is unacceptable.
Mail records are emitted only after their newline arrives. A partial record survives temporary EOF up to the 64 KiB limit, including its newline. Longer records are discarded through the next newline even when written across several polls. Rotation and detected truncation clear pending record state.
Mail source attachment retries after failures, starting at one second and doubling to a maximum delay of 30 seconds. The watcher remains unhealthy and reports an unavailable-source finding until a reader attaches successfully. Repeated identical errors are not re-emitted. Retries use the current mail source configuration and start at the current tail, so delayed attachment does not count historical authentication failures as new activity.
With mail_logs.source: auto, each retry chooses the configured or platform
file if present, otherwise the configured journal units. A file missing for
90 seconds triggers a new selection. The old reader stops before a replacement
starts. Journal input follows new records from the selected services, including
services with no prior entries; it does not replay older records on attachment.
Explicit file and journal modes retry their selected source without
switching, and a working reader stays attached until it stops or loses its file.
Journal input requires a build with journal support. The default units are
postfix, postfix@- (the Postfix instance on Debian and Ubuntu) and
dovecot. Unit names match exactly, so list any further Postfix instance, such
as postfix@postfix-out, in mail_logs.units. Each journal record reaches
the mail detectors with its program name, process id and time, as a mail log
file records them, so every mail detection works the same with either source.
Journal selection uses the trusted systemd unit, not the caller-supplied syslog
identifier. Keep mail_logs.units limited to mail services. File input has no
sender credentials: a local user who can submit mail-facility syslog messages
can forge failures or successes there. On hosts with untrusted local users,
use mail_logs.source: journal with the mail service units, or restrict the
syslog input that writes the mail log.
SMTP / Dovecot Brute-Force Tracker
Detects credential stuffing, password spray, and raw SMTP probe storms. Runs as part of the Exim mainlog watcher on cPanel hosts and on non-cPanel Exim hosts where /var/log/exim_mainlog exists.
On hosts without that log, such as Postfix servers that check SMTP logins against Dovecot, the tracker counts a failed SMTP login when two lines from the mail log agree:
- Postfix’s
SASL ... authentication failedwarning, which is the final result of the request, and - Dovecot’s rejection for the same client address shortly before it:
Password mismatch,unknown user, or PAM’s authentication failure.
Neither line counts alone. Postfix logs a Dovecot outage the same way as a wrong password, and a Dovecot rejection is one password database’s answer: a later database can still accept the login or fail. A rejection followed by a successful login leaves no Postfix warning, a Dovecot error or warning for the same client within the pairing window withholds the failure, and one Postfix warning counts once however many databases rejected the request. Postfix SASL backend warnings also withhold failures from that SMTP process, including stale replies after a failed reconnect. Replies that blame the authentication server or reject the authentication protocol without checking credentials never count.
Pairing is bounded by both receipt time and, when present, the timestamps in traditional syslog or RFC3339 records. Reading a backlog does not make an old timestamped rejection fresh. The mailbox for smtp_account_spray comes from Dovecot; when pending rejections for an address name different mailboxes, its failures count toward address and subnet detection with no mailbox, and long-memory spray detection counts each rejected name once. Each credential or backend failure consumes one pending rejection, including failures withheld for backend errors. Authentication protocol refusals from Postfix or Dovecot neither count nor consume a pending rejection. Tracking capacity is bounded: new verdicts are dropped when full, and error-mark overflow withholds pairing until the affected window expires. Infrastructure protection is checked again before a paired failure counts.
The pairing bounds apply to every name in an ambiguous batch. Only names eligible when a reply is paired enter that reply’s evidence. A withheld backend reply discards the batch’s pending names, so they cannot return as evidence after recovery. Names already counted or discarded remain consumed for their pairing window; a later rejection can start a new window.
Dovecot mailbox existence is unknown when the logged name is empty, sanitized, abbreviated, or disagrees with another pending rejection. An unknown user answer for a name that is not in the mailbox’s own spelling, such as mixed case or a final dot, is unknown too, because it is about that spelling and not the mailbox. These attempts still count toward address and subnet detection. For one unambiguous mailbox, a password mismatch from any database takes precedence over an unknown-user answer. Forged verdicts inside names or logged passwords do not supply mailbox evidence.
Postfix login names are read only from the username field immediately after the authentication method. Sender and forwarding fields cannot supply a missing login name. Mailbox keys ignore surrounding whitespace and ASCII case, and remain stable when a long name must be shortened. Non-ASCII letters remain distinct and cannot borrow another mailbox’s login trust. Names with invalid UTF-8 count as unnamed failures and cannot earn login trust.
Dovecot writes rejections only with verbose auth logging on:
# /etc/dovecot/conf.d/10-logging.conf
auth_verbose = yes
Leave auth_verbose_passwords at no; lines that carry a logged password are ignored. csm doctor checks both settings on Postfix hosts with Dovecot. Rejections for IMAP, POP3 and ManageSieve logins carry a session id and are counted by the mail tracker below from their login summaries. Those failures carry Dovecot’s unknown user and Password mismatch answers too: CSM pairs each answer with the login summary of the same session, address and name, and the spray rules use it to tell whether the mailbox exists. Answers from several databases combine as they do for SMTP. An answer on its own records nothing, and an authentication error that names the session withholds the session’s answers, so a failing database cannot make a real mailbox look missing.
For IMAP and POP, a cut or unrecognized authentication message that names a session also withholds that session’s answers, and a local mailbox lookup does not replace them. Each mailbox answer expires by its own log timestamp; another name cannot renew it, and a later timestamp cannot answer an earlier login summary. Successful logins consume pending session answers too. Session tracking runs only while the spray rules are on, is held in memory and starts empty after a daemon restart.
A successful SMTP login is read from Postfix’s record of an authenticated client, logged when its first recipient is accepted, or from an authenticated Exim delivery. It counts for the mailbox it names. A login from an address with no recent failures on other mailboxes earns trust for that mailbox only, and the trust counts once it is older than the fast detection window. Failures on just one other mailbox, the shape of an office device with a stale saved password, do not count here unless that mailbox does not exist or the login’s own mailbox also failed. The slow brute-force block skips an address only when it holds such trust for every mailbox it failed on. Any other login that follows failures on other mailboxes earns no trust, unless the address already held trust on that mailbox from before those failures or, for IPv6, another address of the same /64 holds aged trust on it. Trust lasts as long as the spray memory window and never exempts an address from fast brute-force or subnet detection.
Trust earned beside a failing mailbox vouches only for the mailbox that logged in. It cannot excuse missing-mailbox probes or downgrade a compromise while the earlier untrusted failures remain in memory.
Failures without a mailbox prevent the slow brute-force trust exemption until they leave the slow detection window. Exim spray evidence comes from authentication failure records; greetings, subjects and SMTP commands cannot supply a failure record or replace its attempted mailbox.
Long-memory spray detection remembers each address’s failed SMTP logins for days, so a botnet that tries each address only now and then is still caught. It flags an address that fails on several mailboxes it holds no trust for, or that fails on a missing mailbox after enough networks have tried the same name. A mailbox is missing when Dovecot reports it as unknown or when the cPanel lookup below finds no such name. For an address that holds trust on some mailbox, a missing name only counts toward the several-mailboxes rule, so one address typo does not block an office. A login after failures on other mailboxes raises mail_account_compromised, except when they were on just one other mailbox that may exist and the login’s own mailbox did not fail. In observe mode, the default, the spray rules report mail_bruteforce_suspected and block nothing; an address is reported once per memory window, and again sooner only when it fails on more mailboxes. In enforce mode they emit smtp_bruteforce. IPv6 failures are remembered per /64, since one host can rotate through many addresses in it: failures from any address of the /64 count together, and aged login trust any of its addresses earned on a mailbox excuses failures on that mailbox for all of them. The missing-name first-strike exemption and the established-source compromise downgrade also count trust from any address of the /64 that predates its current untrusted failures, and a correct login to a mailbox the /64 already trusts starts trust for a new address. Trust is still earned per address, the slow brute-force exemption still needs the address’s own trust, and each failing address is blocked on its own. In observe mode a /64 is reported as one address would be. An operator unblock, allow or flush clears the address’s spray failure history, for IPv6 that of its /64, and keeps its login trust. Failures during an authentication backend outage, Exim’s temporary authentication failures (a 4xx reply, logged when Dovecot cannot answer), and failures withheld by pairing are never spray evidence. An outage is detected a little late, so a failure during one also discards that address’s spray failures from the fast detection window before it. Successful SMTP logins during an outage leave spray memory unchanged. The memory is saved every few minutes and at shutdown, so a daemon restart keeps it. The thresholds.mail_spray_* keys control the mode, the memory window and both rules.
On cPanel hosts, in observe and enforce mode, CSM checks whether a failed SMTP, IMAP or POP login name is a real mailbox, using cPanel’s own account, domain and mailbox lists. A domain the server does not host, an unknown mailbox on a hosted domain, and a bare name that is not a cPanel account count as missing. A real mailbox or cPanel account counts as existing, and any name CSM cannot tell about stays unknown, so it counts only toward the several-mailboxes rule. A name within two edits of a real mailbox, counting the local part and the domain together, counts as that mailbox: spray memory records the failure against the real mailbox, so aged login trust on it excuses the typo and the typo is never a first strike. Login trust and spray failures treat a valid mailbox domain with or without its final dot as the same name, including saved history. The fast, slow and per-mailbox failure counts use the attempted name with surrounding whitespace removed and ASCII case ignored; they do not use typo matches. CSM holds these lists for a short time, so a mailbox created or deleted just now can get its old answer. When the lists cannot be read every name stays unknown; the daemon log gets one warning, and one line when they can be read again. Only the default thresholds.mail_brute_account_key reads the whole login name, so with any other key IMAP and POP failures skip this check and stay unknown. Each login keeps that setting through the check, even if configuration reloads while it is being handled.
Restored spray memory stays within the configured capacity, and expired entries are discarded. Records dated in the future cannot establish login trust or corroborate a missing mailbox. Saved revocations override conflicting trust records.
Dovecot’s informational and debug authentication messages can contain client text; they never mark the authentication backend as unavailable. Error and warning diagnostics still reach the backend-outage detector. smtp_probe_abuse reads Exim lines only.
Signals:
| Signal | What triggers it | Auto-response |
|---|---|---|
smtp_bruteforce | A single attacker IP exceeds the per-IP failed-auth threshold within the configured window, or, with spray detection in enforce mode, fails on several untrusted mailboxes or on a missing mailbox many networks tried | IP blocked via nftables |
smtp_probe_abuse | A single attacker IP exceeds the raw SMTP connect-rate threshold before AUTH | IP blocked via nftables |
smtp_subnet_spray | Multiple distinct attacker IPs from the same /24 subnet exceed the subnet threshold | Entire /24 subnet blocked via nftables |
smtp_account_spray | Many distinct attacker IPs targeting the same mailbox exceed the account threshold | Visibility finding only. No auto-block, because attackers span many subnets and no single-IP action helps |
mail_bruteforce_suspected | Spray detection in observe mode: the spray rules matched an SMTP source. The message names SMTP and the rule | Visibility finding only |
mail_account_compromised | A successful SMTP login after failures on two or more mailboxes including another one, or on a missing one, or after repeated failures on the same mailbox, the first at least an hour before the login | Critical only in enforce mode when a failed name was a missing mailbox many networks tried, and the IP is then blocked and the mailbox suspended under auto-response. A source trusted on two or more other mailboxes is downgraded. Every other case is a High advisory, visibility only |
Tunable via the thresholds.smtp_bruteforce_*, thresholds.smtp_probe_* and thresholds.mail_spray_* keys in csm.yaml. Infrastructure IPs (from infra_ips) are never counted or blocked.
Cloud-Relay Credential Abuse
Detects authenticated outbound Exim deliveries where the same mailbox is sending through public-cloud relay sources. The realtime Exim mainlog watcher evaluates new accepted deliveries, and a bounded startup replay covers recent lines already on disk.
Startup replay reads the time zone and process ID that Exim’s log_timezone and pid log selector add, and a restart does not repeat an alert for sends already reported.
The finding is email_cloud_relay_abuse. Under the global auto-response and dry-run settings, the abused mailbox’s logins and outgoing mail are suspended, with the account-wide mail hold as the fallback. Operators with legitimate cloud mailers can opt out specific mailboxes or domains under email_protection.cloud_relay, or use email_protection.high_volume_senders for known high-volume senders.
Authenticated Sender Profile
Catches a stolen mailbox password that is used quietly: under the rate limits, from residential addresses with no cloud reverse DNS, using individually addressed messages. Every authenticated Exim submission updates a rolling hourly window and the mailbox’s own history of source addresses, source countries and envelope recipients, grouped by UTC day. Repeated sends keep each source current, and a busy mailbox continues to accept new source and country evidence.
The finding is email_compromised_account. Four signals raise it:
| Signal | Severity |
|---|---|
| Sends from several countries within one hour, the pattern of a credential checker validating a stolen password | Critical |
| Far more distinct source addresses within one hour than the mailbox’s own recent maximum | High, Critical when the sends also come from a country the mailbox never sent from |
| Far more distinct source addresses within one day than the mailbox’s own recent maximum | High, Critical as above |
| Far more distinct recipients within one day than the mailbox’s own recent maximum, each in its own message | High, Critical as above |
Each bar adapts to the mailbox’s own prior maximum with a fixed minimum, so a shared office mailbox or a travelling owner is judged against their own habit rather than a global number. A country listed in suppressions.trusted_countries is never treated as new. A mailbox needs established prior activity before a new country counts. A newly seen country remains relevant if unusual sending develops later in the same UTC day. Repeated findings are suppressed during a cooldown, while escalation from High to Critical remains possible.
CSM uses the time in the Exim record to place a loopback send. It uses the country of the mailbox’s most recent webmail login only if that login preceded the send and was recent enough to locate it. An unlocated login clears the previous location.
Recipient fan-out counts distinct envelope recipients of individually addressed messages; bulk announcements do not contribute. Quoted addresses remain single recipients. Records with ambiguous recipient metadata do not contribute recipient evidence, but verified source evidence still counts.
Daily history covers the recent baseline period and expires even for inactive mailboxes. It is cached and saved periodically, before idle eviction and during normal shutdown. An abrupt termination can lose submissions since the last save; a restart always starts a fresh hourly window. Save failures retain pending history for retry. An unreadable baseline pauses profiling for that mailbox until the store can be read again.
Source addresses are never blocked for this finding: they rotate per message and the owner’s own address can be among them. A Critical finding suspends the mailbox’s logins and outgoing mail under the auto-response and dry-run settings, with the account-wide mail hold as the fallback. Mailboxes listed in email_protection.high_volume_senders are not profiled.
Mailbox Compromise Evidence
A stolen mailbox rarely trips a single loud alarm. It shows a few weak signs, each of which an honest owner also produces now and then. CSM keeps a ledger of these signs per mailbox. Each sign is reported once in the audit log, history and the daily summary, and is not mailed on its own.
The signs cover logins from new countries, forged sender domains, bulk recipients, external redirects, and outbound reputation deferrals. Two new countries seen close together carry more weight. A redirect created soon after a foreign login also carries more weight.
A login from a new country is an email_suspicious_geo finding, at most one per mailbox a day. A mailbox needs a few logins of history first, and a country in suppressions.trusted_countries is never new. Webmail, IMAP and POP3 logins share the same history and use the login time in the log. Sessions opened by the panel or a reseller do not locate the mailbox owner. An authenticated send from a country the mailbox never logged in from is the same sign, as is a webmail send after a webmail login from a new country. A send reports email_suspicious_geo the first time the ledger records its country.
A mail filter or Sieve script that sends a mailbox’s mail to an outside address keeps its own email_filter_exfil finding and is also a sign, dated by the script’s file time. The incident finding names the script and its target; CSM leaves the script in place as evidence, so remove it after review.
When large providers refuse or defer this server’s mail because of an address’s reputation, the address is an email_outbound_reputation finding at most once a day, naming the authenticated mailbox that sent most of the refused mail. A tie or an incomplete count leaves the finding at address level. If another mailbox leads a later burst, its ledger records that evidence even after the daily notice.
A mailing is an email_bulk_recipients finding the first time the ledger records it. An authenticated send whose envelope sender is a domain the login’s hosting account does not own is an email_forged_sender finding the first time the ledger records that domain. A subdomain follows its closest mapped parent, including aliases and subdomains delegated to another hosting account.
Each kind of sign counts once, however often it repeats. Mailboxes in email_protection.high_volume_senders send mailings by design, so forged sender domains and bulk recipients do not count for them. Signs expire a week after their last sighting. The ledger groups older details when many distinct values appear.
When the evidence adds up, CSM suspends the mailbox and raises one Critical email_mailbox_compromise finding. It lists the evidence and the restrictions applied. Release restrictions applied by CSM on the Email page, Suspended tab. With automatic response disabled or dry-run enabled, the finding says CSM would suspend the mailbox. Further signs update the same incident. Release discards its evidence, including delayed observations from before release.
Mail Auth Brute-Force Tracker
Detects credential stuffing and password spray against IMAP, POP3, and ManageSieve. Runs through the mail_logs reader: file source uses /var/log/mail.log on Debian-family hosts and /var/log/maillog on RHEL-family and cPanel hosts, while journal source reads configured Postfix/Dovecot units. The wrapper composes with the existing geo-based login monitor, so email_suspicious_geo keeps reporting successful logins from novel countries as compromise evidence.
Signals:
| Signal | What triggers it | Auto-response |
|---|---|---|
mail_bruteforce | A single attacker IP exceeds the per-IP failed-auth threshold within the configured window without matching successful mailbox activity, or, with spray detection in enforce mode, fails on several mailboxes it holds no trust for or, on cPanel hosts, on a missing mailbox many networks tried | IP blocked via nftables |
mail_bruteforce_suspected | An established good source hits the per-IP failed-auth threshold with a confined stale-password pattern, or spray detection in observe mode matched an IMAP/POP source. The spray message names IMAP/POP and the rule | Visibility finding only. No auto-block |
mail_subnet_spray | Multiple distinct attacker IPs from the same /24 subnet exceed the subnet threshold | Entire /24 subnet blocked via nftables |
mail_account_spray | Many distinct attacker IPs targeting the same mailbox exceed the account threshold | Visibility finding only. No auto-block, because attackers span many subnets and no single-IP action helps |
mail_account_compromised | A successful login comes from an IP that repeatedly failed auth against the same mailbox, or follows failures on two or more mailboxes including another one, or on a missing one, or repeated failures on the same mailbox, the first at least an hour before the login | Critical findings block immediately and, under auto-response on cPanel or on Stalwart with its API key, suspend the mailbox (a cPanel account’s own login holds the account’s outgoing mail instead). A source with established history on at least two other mailboxes from before its current untrusted failures emits a High advisory and is not auto-blocked. A login after failures on other mailboxes is Critical only in enforce mode when one of them was a missing mailbox many networks tried; every other spray case is a High advisory |
Tunable via the thresholds.mail_bruteforce_* keys in csm.yaml. Independent from the SMTP tracker so the Dovecot noise floor can be tuned separately. Infrastructure IPs are never counted or blocked. Established good sources with a confined stale-password pattern emit mail_bruteforce_suspected instead of an IP block, while wider spraying and compromise from a source without established multi-mailbox history still block. A compromise from an IP with established history on at least two other mailboxes from before its current untrusted failures remains visible as a High advisory but does not feed direct, incident, or spray auto-blocking. The mail_bruteforce and mail_bruteforce_suspected alerts name the mailboxes the source was hitting, with per-mailbox failure counts and a count of attempts that named no mailbox, so a real attack on a live mailbox is easy to tell apart from dictionary noise.
IMAP and POP logins share the long-memory spray detection described for SMTP above. One memory holds the failures and login trust of both, so failed SMTP and IMAP logins from one address on two different mailboxes count together toward the several-mailboxes rule. A login that follows failures on other mailboxes earns neither login trust nor good standing, unless the failures were all on one other mailbox that may exist and none on the login’s mailbox, the address already held trust on the login’s mailbox from before those failures or, for IPv6, another address of the same /64 holds aged trust on it. The slow brute-force block downgrades to mail_bruteforce_suspected only when the address holds good standing or aged login trust on every mailbox it failed on; a login to some other mailbox no longer counts. One address gets at most one mail_bruteforce per suppression period, and one login at most one mail_account_compromised. The High advisory for a login after failures on other mailboxes is raised once per address and mailbox per memory window, so a client that polls every few minutes does not repeat it. In observe mode the same limit applies to a login after a missing name that several networks tried.
A login identified as a short-window password crack revokes trust for that address and mailbox for the spray memory window, even when the compromise alert is suppressed; for an IPv6 address it revokes that mailbox’s trust for every address of its /64. Account cache pressure does not hide a crack while its per-address failure history remains. Further SMTP, IMAP or POP logins cannot restore that trust during the window. Suspect logins do not count toward successful activity that exempts later failures. A crack also removes any good standing from a recent login that had not yet aged into established standing.
When the mail authentication backend itself fails (for example dovecot cannot reach cpdoveauthd), every login fails regardless of password. CSM detects a burst of these backend errors, pauses mail_bruteforce and mail_subnet_spray auto-blocking, keeps the failures out of spray evidence, and raises a mail_auth_backend_degraded warning so the outage is visible instead of mass-blocking legitimate users. IMAP and POP successes during this degraded period do not update spray memory, raise spray alerts, or earn good standing. The short-window compromise detector still uses recorded failures. Detection resumes automatically once the backend recovers.
Mail login attribution uses the connection fields outside the raw login name. Names containing field-like text or angle brackets remain one account name. Records without a usable remote address do not count toward IP blocking or login trust. Reputation exemptions require a successful Dovecot login summary; success wording in a failed login name does not qualify.
Stalwart Mail Server
On hosts where the stalwart service is the mail server and cPanel is absent, CSM reads Stalwart’s daily log instead of the Postfix and Dovecot mail log. It follows the newest stalwart.log.YYYY-MM-DD file in stalwart.log_dir (default /var/log/stalwart), starting at the end of the current file. Files dated later than tomorrow in UTC are ignored. When Stalwart starts the next day’s file, CSM reads the old file to its end and then the new one from its start. A missing directory or file raises mail_log_source_unavailable.
If the next day’s file cannot be found or opened for a while, CSM raises mail_log_source_unavailable and keeps retrying. If the current file is removed, CSM does not go back to an older day’s file.
Each event is assigned a protocol from its listener port: 25, 465 and 587 are SMTP; 143 and 993 IMAP; 110 and 995 POP3; 4190 ManageSieve; 80, 443 and 8080 HTTP (JMAP and webmail). List other listeners in stalwart.ports as PORT:PROTOCOL. Logins and bans on an unmapped port are ignored; sends on one, such as a local submission listener, count toward volume and recipients only.
What CSM reads:
- Failed and successful logins on SMTP listeners feed the SMTP brute-force tracker, and those on IMAP, POP3 and ManageSieve listeners feed the mail brute-force tracker. The same thresholds, findings, blocks and spray rules apply as on Postfix and Dovecot. Loopback, private and infrastructure addresses never count.
- Failed logins on the HTTP listener raise
stalwart_webmail_bruteforceat the mail brute-force threshold. Stalwart can log an address from a forwarded header there, so this finding never blocks. Stalwart’s own throttling and bans still apply to it. - Authentication, abuse and scan bans Stalwart logs for an SMTP, IMAP, POP3 or ManageSieve listener raise
stalwart_ban, which blocks the address for the auto-response block length. Stalwart’s ban stays in place. A failed-login ban of an address a customer logged in from within the last 30 days, such as an office network with one stale password, is a High finding and is not blocked. Loiter bans, refused connections and bans on the HTTP listener or an unmapped port are ignored. - Authenticated sends on SMTP listeners and unmapped ports are matched to the login on the same connection and feed the rate, sender profile and cloud-relay checks. Sends from a loopback or private address, such as sites on the same server, and sends on an unmapped port count toward volume and recipients only.
- Unauthenticated messages Stalwart queues are matched by queue id to the lines that record their delivery, so the Email page counts what the spam filter did with each one and the sending address gets spam evidence. Rejected and discarded mail is read from the filter’s own line. Messages on the HTTP listener are ignored. See Stalwart hosts.
Stalwart logs failed logins at debug level by default, so password guessing over SMTP, IMAP and POP3 is invisible until the auth.failed event is raised to info. csm stalwart setup does that (see below). csm doctor reports whether the newest log holds failed logins, whether it is current, and whether its recent lines are in the layout CSM reads.
Preparing Stalwart
csm stalwart setup changes four settings and adds a spam rule through Stalwart’s admin API at stalwart.api_url (default http://127.0.0.1:8080), then reloads Stalwart’s settings once. It also creates the role, service account and API key CSM uses to suspend mailboxes and lift bans. The API address, including a --url override, must not contain a login, query or fragment. Remote addresses require HTTPS and cannot use this host’s port locks as protection. Run it as root with a Stalwart admin login:
STALWART_USER=admin csm stalwart setup --dry-run --password-file /root/stalwart-admin.pw
STALWART_USER=admin csm stalwart setup --password-file /root/stalwart-admin.pw
STALWART_PASSWORD works in place of --password-file, and --password-stdin reads the password from stdin, for example printf '%s\n' "$PW" | csm stalwart setup --password-stdin; the user is then STALWART_USER or admin. On a terminal, setup prompts for missing values. When you enter both values, the username is hidden too. Press Enter at the user prompt for admin. Type the password at the password prompt, where echo is off even with --password-stdin. Ctrl-D at an empty prompt cancels setup. Before sending it to an address on this host, setup and doctor check that the API port and each connected peer belong to root or Stalwart’s service user (stalwart or stalwart-mail). An unreadable ownership record stops the login. This also protects reconnects when Stalwart restarts. --dry-run shows each step and changes nothing; a second run after a successful one changes nothing and does not reload. If a settings step fails, part of it may already be stored: resolve the error and reload settings in Stalwart before running setup again. If one of steps 6 to 8 fails, resolve the error and run setup again. If only the reload failed, restart Stalwart: a second run reads the stored settings as done.
-
Failed-login logging. Sets the
auth.failedevent to info unless it is already logged at info, warn or error. -
Webmail client addresses. Makes Stalwart’s HTTP listener take the client address from
X-Forwarded-For(1 hop), so webmail and JMAP failures name the real client instead of the web server. Anyone who can connect to the listener could forge that header, so setup only does this when the API address names this host on loopback, every HTTP listener has known loopback-only binds, and the running CSM daemon locks each of their ports to the web server and Stalwart. Otherwise it skips the step and prints what is missing, including the line to add underfirewallincsm.yaml, for examplelocal_port_locks: ["8080:www-data,stalwart"]; restart CSM after adding it. If Stalwart already takes forwarded addresses without that protection, setup reports the step as unsafe and exits 1 without changing it. See Local port locks.With one hop Stalwart reads the last
X-Forwarded-Forentry, so the web server in front of it must set that header itself. nginx passes a client’s header through unchanged unless told otherwise; addproxy_set_header X-Forwarded-For $remote_addr;(or$proxy_add_x_forwarded_for) to the Stalwart location. Without it a remote client chooses the address Stalwart logs and bans. Setup cannot check this. The users in the lock line must not run site code: run PHP-FPM pools as their own users, not as the web server’s user. -
Ban expiry. Sets Stalwart’s failed-login, abuse and scan ban periods to
auto_response.block_expiry, so its bans end when CSM’s blocks do. Loiter bans are left alone. -
Spam verdict logging. Sets the
spam.classifyevent to info unless it is already logged at info, warn or error. By default Stalwart logs the mail its spam filter rejects or discards only at debug level. This step changes nothing about how mail is handled. -
Address list. Creates the spam rule
CSM_IPand one scoring tag per kind of listing, so Stalwart adds up to 4 to the spam score of a sender on CSM’s address list and more of their mail goes to Junk, or is rejected or discarded at the scores set in Stalwart. This is the one setup step that changes what happens to mail. Setup creates the rule switched on; to stop it, switch the rule off in Stalwart’s admin interface. Setup never switches it back on and reports a rule that is off as done. It creates missing tags but never changes an existing tag, so a tag’s score is yours to set in Stalwart. When the rule’s condition differs from CSM’s, setup resets the condition and leaves the rule’s other fields alone. A rule namedCSM_IPthat CSM did not make is left alone with a warning. If Stalwart’s spam rules cannot be read, setup warns and continues with the other steps. -
Response role. Creates the role
CSM response: the permissions CSM needs to suspend accounts, lift bans, keep its address list in Stalwart and read the settings setup checks, plus Stalwart’s ownUserrole. IncludingUserlets the key change user accounts. When the role lacks any of these or denies permissions, setup rewrites it to CSM’s list and clears denials. If Stalwart has no role namedUser, setup warns and skips the next two steps. -
Response account. Creates the account
csm-responseholding only that role and no password. It can still receive mail at its address. On a server with several domains, name the domain with--account-domain NAME; otherwise setup skips the step and the next. On a server with tenant administrators, put the account in a domain only the server administrator controls: whoever can edit the account could give it a password. If more than one account is namedcsm-response, setup warns and creates nothing. Setup also refuses an existingcsm-responseaccount that has a password or another sign-in credential, has sign-ins stopped, or holds keys without CSM’s role: it warns, changes nothing and skips the key step. -
Response key. Creates an API key of
csm-responsethat works only from127.0.0.1and::1, writes it tostalwart.api_key_file(default/etc/csm/stalwart-api.key, owned by root, mode 0600), then deletes the account’s other keys. The key is never printed. A key already in the file that Stalwart accepts forcsm-responseis kept while it is the account’s only key; if the account has other keys, setup replaces them all, so a rerun finishes a clean-up that failed. If the file cannot be written, setup deletes the new key and the old one keeps working. A key file that is not a root-owned regular file closed to group and others stops setup with exit 1. To rotate the key, delete the file and run setup again.
Steps 6 to 8 need no settings reload, and they run only when the API address is on this host; otherwise setup skips them and the key is never sent. When one of them warns, setup exits 1: CSM then can neither suspend mailboxes nor lift Stalwart’s bans.
Setup also reports listeners that accept the PROXY protocol, which it does not change, and whether Stalwart’s spam filter is on and at which scores it files spam to Junk, rejects or discards. Doctor prints the same. A filter that is off is a warning; setup never switches it on or off. If its status cannot be read, setup warns and continues with the other steps. A refused admin login still stops setup.
csm doctor checks the same steps read-only through CSM’s API key, and says whether Stalwart accepts the key. Only root can read the key, and a missing key is reported even while Stalwart is down. Without a working key, or for an API address on another host, it uses an admin login from STALWART_USER and STALWART_PASSWORD when both are set. A step setup would change is a warning, as is a response step setup refuses until you clear what it names. Its spam lines say whether Stalwart logs the mail its filter rejects or discards, whether the filter is on and at which scores it acts, and whether its DNS blocklists answer. The blocklist check runs only while the filter is on: it sends Stalwart one test message, which Stalwart never delivers or learns from, and warns when the blocklists refuse this server’s resolver. Blocklist warnings show a bounded, terminal-safe list of refused tags; they do not report the test message’s verdict or score. Stalwart keeps refused answers until it restarts, so restart it after changing the resolver. The same test message comes from the address the daemon keeps on CSM’s address list, so while the address list rule is on, doctor also says whether Stalwart scores senders on that list. The daemon writes the list within 10 minutes of setup, so this check can warn until then; run doctor again after that. If it still warns, check the daemon’s stalwart_address_list finding, and run csm stalwart setup only when that finding says to. Observe mode skips this check, since the daemon then does not write the list to Stalwart. From the newest log alone it says when Stalwart’s spam classifier is still learning and how many ham and spam samples it has. It omits that line if the latest classifier status is malformed or a later training run has started. It also warns when Stalwart’s HTTP listener logs a login from another host on a port CSM does not lock, and when Stalwart bans a CDN or proxy address, which locks out every client behind it.
stalwart_log_degraded warns when an hour of log lines holds none in Stalwart’s layout, as after an upgrade that changes it; when many authenticated SMTP sends in an hour cannot be matched to a login; and when the log stops growing for hours while Stalwart runs.
Known limits:
- Suspending a Stalwart mailbox stops new sign-ins only. Sessions that are already open stay open until they end.
- Webmail logins appear in Stalwart’s log from the proxy address, so a webmail login after failures is not detected as a compromise.
- Sends through webmail or JMAP are not tied to an account, because one proxy connection can carry several users, so the send checks do not see them.
- A listener behind PROXY protocol logs the client address, not the connecting peer. CSM cannot tell such listeners apart from the log alone.
Suspending Stalwart mailboxes
CSM suspends a hacked Stalwart mailbox through the API key that csm stalwart setup writes. The auto-response and dry-run settings apply as on cPanel. After a Critical mail_account_compromised, email_compromised_account or email_cloud_relay_abuse finding, CSM stops the account’s sign-ins through Stalwart’s API. That also stops it sending. A sign-in through a mailbox alias or a domain alias suspends the account behind it. All aliases share that account’s suspension and release history. Buffered sends logged before a release do not suspend it again. Reassigning an address does not transfer a suspension to its new account. The mailbox is listed on the Email page, Suspended tab, where Release allows sign-ins again on the same account. Release does not change the password. Sign-ins someone else had already stopped are left alone and not listed.
CSM does not suspend admin accounts, its own API key account, accounts its key may not change, or account names without a domain. Bare names containing + or % are not converted to mailbox addresses; that login convention applies only to cPanel. The finding then asks staff to change the password in Stalwart. The API key is sent only to loopback addresses, after verifying the connected peer. Without the key file nothing is suspended. The email_compromised_account and email_cloud_relay_abuse findings then say the mailbox stays active; a mail_account_compromised finding gets no mailbox response. Address blocks apply either way.
Admin-Panel Brute-Force Tracker
Counts repeated POST requests to high-value non-WordPress admin login endpoints. Runs as part of the web access-log watcher.
Covered endpoints (tight set to avoid false positives on shared hosting):
- phpMyAdmin:
/phpmyadmin/index.php,/pma/index.php,/phpMyAdmin/index.php - Joomla:
/administrator/index.php
When an IP crosses the POST-rate threshold, admin_panel_bruteforce fires and the attacker IP is auto-blocked.
Drupal /user/login and Tomcat Manager /manager/html are intentionally out of scope here. Drupal’s path is too generic on shared hosting, and Tomcat Manager uses HTTP Basic auth (repeated GET requests with 401 responses), not POST form submissions. Both need different detectors and are tracked as follow-up work.
PHP-Relay (Mail Abuse, cPanel Only)
Real-time inotify watcher on /var/spool/exim/input catches WordPress contact-form spam relays where an attacker uses PHPMailer (or similar) with a spoofed From, an external Reply-To, and a script URL that doesn’t belong to the cPanel account. A contact-form relay incident drove the design: a legitimate site running a vulnerable contact-form plugin became a per-message spam relay through the operator’s own mail account.
The detector runs five paths. Paths 1, 2, 2b and 4 fire email_php_relay_abuse (Critical) when one of them crosses threshold; Path 6 has its own findings, described below. Paths 1 and 2 are scoped per-script, using the host:/path from the X-PHP-Script Exim header. Path 2b is per cPanel user. Path 4 is per HTTP source IP across distinct scripts. Path 6 is per client IP across every script and account. Paths 2 and 4 use a recipient-diversity gate that suppresses only known low-recipient notification mail.
The source address is the final, server-observed address in X-PHP-Script.
Client-supplied forwarding text cannot replace it. Behind a proxy this is the
proxy’s address; CSM does not recover a visitor address from untrusted forwarding
text. A missing or invalid final address leaves IP attribution empty.
The script label ends at the first for separator, keeping forwarding text
out of per-script counts and queue actions. PHP includes PATH_INFO in this
header without escaping separators. Paths containing a literal for therefore
share the label before that separator; the source address is parsed separately
from the header’s end.
| Path | What triggers it | Why it exists |
|---|---|---|
| Path 1: header score | Per-script: From domain not in the account’s authorised domains AND additional signal (PHPMailer / suspicious Reply-To / suspicious User-Agent), evaluated over a rolling 5-min window once the script has emitted at least header_score_volume_min messages | The shape that matched the original incident: spoofed sender, contact-form-style. FromMismatch is a HARD precondition – the score never accumulates without it |
| Path 2: absolute volume per script | A single script emits at least absolute_volume_per_hour messages in the last hour. If Exim envelope recipients are known, fewer than fanout_distinct_recipients distinct recipients suppresses this path; unknown recipients or fanout_distinct_recipients: 0 fail open. | Catches a compromised script even if the headers themselves are legit-shaped, while leaving plugin notification mail to a fixed admin set alone |
| Path 2b: account log-tail volume | Per cPanel user: at least effective_account_limit messages in the last hour that the account’s own processes submitted locally (PHP mail(), sendmail), each with at least one recipient outside this host’s mail domains. The effective limit is auto-derived from /var/cpanel/cpanel.config’s maxemailsperhour (60% of it, clamped to 20-60), capped at 95% of the cPanel limit when an operator override is set | Backstop for when Path 2 misses the window. Reads /var/log/exim_mainlog directly and names the account only from Exim’s own submission fields, so a subject or sender address cannot. Mail to the host’s own mailboxes, bounces, authenticated SMTP and mail arriving from the network never count. If a present mail-domain file cannot be read, or both /etc/localdomains and /etc/virtualdomains are absent, nothing counts |
| Path 4: HTTP-IP fanout | Per HTTP source IP: one source IP appears in at least fanout_distinct_scripts distinct script keys in fanout_window_min minutes, after excluding loaded HTTP-proxy ranges, loopback, and the host’s own interface addresses. If Exim envelope recipients are known, fewer than fanout_distinct_recipients distinct recipients outside each sending account’s explicitly listed domains suppresses this path; subdomains are exempt only when listed for that account. Missing account-domain data, unknown or partially parsed recipients, or fanout_distinct_recipients: 0 fail open. | Catches one client walking many scripts while avoiding CDN/proxy traffic, local cron or panel callbacks, and fixed-admin notification fanout, including one client commenting on several hosted sites |
| Path 6: client recipient spread | Per client IP: hosted sites mail at least ip_recipient_spread distinct recipients outside each sending account’s own domains within ip_recipient_window_hours, counted across every script and account. An IPv6 client counts per /64. A message whose account domains cannot be read is not counted; a message whose recipients cannot be parsed counts as one new recipient. ip_recipient_spread: 0 turns the path off. | Catches bots that submit a site’s forms with strangers’ addresses, so the site sends each stranger an automatic reply. One bot sends too few of these per script or per hour for the other paths to notice |
Path 2b reads recipients only from Exim’s envelope recipient list. Message headers and sender text do not supply recipients. A missing recipient list or a truncated address cannot establish an outbound send. The host’s mail-domain inventory is refreshed once per minute; a missing optional file is allowed when the other file can be read.
Path 5 (behavioural baseline) is deferred to Stage 2.
Per-IP paths skip selected shared addresses: CDN and website firewall edges (Cloudflare, QUIC.cloud, Sucuri) and payment gateway webhooks (Stripe, PayPal, Braintree, PayU, Revolut, Mollie). CSM ships this list built in, taken from each provider’s published list. Membership is not proof that the target site chose the service. Cloudflare Workers can make requests to third-party sites, and Sucuri can proxy requests before the site’s DNS changes. Traffic relayed through these retained providers remains exempt, including abusive requests.
Uptime monitors and cron services are not on the list. Anyone with an account there can schedule POSTs with any form body to any URL, so exempting their addresses would let a bot relay form spam through them, and the per-IP paths would miss it. Self-service CDNs such as Amazon CloudFront, Fastly and BunnyCDN are left out for the same reason: anyone can point a pull zone at any site and post through its edges. If you trust a specific cron service, or your sites sit behind one of those CDNs, you can add its ranges yourself, knowing that anyone with an account at that service can use them.
An http_proxy_ranges.yaml in policies_dir (version: 1 plus a cidrs: list; a bare IP counts as /32 or /128) replaces the built-in list whole and reloads on SIGHUP. Without the file the built-in list applies. A file that cannot be read or parsed, has no cidrs: list, or has no valid entry keeps the last good list, which is the built-in one at startup. Invalid entries, including blank or null list items, are reported and skipped; valid siblings still apply. Only an explicitly empty list (cidrs: []) clears the proxy ranges. Load errors at startup or on SIGHUP raise email_php_relay_policies_reload (Warning). To extend the list, start from the built-in file in the CSM source, internal/emailspool/embed/http_proxy_ranges.yaml. Your file replaces the whole list, so later CSM updates to the built-in entries do not reach you until you copy them in again.
CSM 4.10.0 exempted Amazon CloudFront, Fastly and BunnyCDN; later releases do not. An operator file copied from the 4.10.0 list keeps them until you remove those entries and reload. Legitimate visitors of a site behind these CDNs share their edge’s per-IP counts unless you exempt its ranges. For Path 6, review findings in ip_recipient_mode: observe before enforcing; this setting does not disable Path 4. Cloudflare remains separately exempt from Path 6 through the daemon’s refreshed infrastructure list, even when omitted from the operator proxy file.
Netopia, EuPlatesc, Twispay, JCC and Adyen publish no source address list. If your shops use them, add their ranges yourself. Viva publishes a webhook address list; it is not built in yet, add it yourself.
Path 6 does not count a client on the proxy and payment list, a Cloudflare edge the firewall knows, a loopback or host address, or an infra_ips entry. It also skips mail sent from wp-cron.php, which mails whatever its trigger request queued, and staff: a client that sent mail through a WordPress admin page of the same account in the last 7 days. Every .php page directly in a wp-admin/ directory counts as an admin page except admin-ajax.php, admin-post.php, profile.php and index.php, which logged-out visitors or subscribers can reach. The staff exemption covers only that account. Becoming staff, or an ignore-script for a script, also drops the recipients counted earlier in the window through that account or script.
Staff often mail customers without passing such a page, for example by completing orders with the quick actions of the WooCommerce order list. So when a client is about to cross the threshold, CSM reads the last 4 MiB of each involved site’s current access logs (/usr/local/apache/domlogs/<account>/<site> and <site>-ssl_log). For a site starting with www., it also checks the name with that prefix removed. Each distinct log is read once per crossing.
Access-log evidence requires a GET answered 200 on a recognized WordPress staff page, such as the post editor, the WooCommerce order list, uploads, comments, users, settings, plugins or themes. Both order lists qualify: the legacy one on edit.php and the HPOS one on admin.php?page=wc-orders (or page=wc-orders--<order type>). For HPOS, page must be the first query parameter so PHP’s input limit cannot discard it. Public resources, setup pages, subscriber pages and every other admin.php page do not qualify. Requests with another parameter PHP would interpret as page do not qualify either. Encoded or ambiguous paths and extra path after the PHP filename do not qualify. The request must fall within the last 7 days, and the exemption expires 7 days after that request; rereading a log does not renew it. The client’s address, or its IPv6 /64, becomes staff only on that account, whose mail stops counting; it is challenged only if the other accounts’ mail still crosses the threshold. A missing or unreadable log is no evidence.
Path 6 has known limits:
- A plugin that handles POST requests in its
initorwp_loadedhook without checking for a login, or forDOING_CRON, lets a visitor who is not logged in make it send mail by posting to an admin page or towp-cron.php. That mail is not counted, and a post to an admin page also earns the client the staff exemption on that account. - Behind a proxy or CDN that does not pass the client address to PHP, CSM sees only the proxy’s address. A proxy on the list is skipped, so its visitors are never counted; that includes a bot that sends its requests through a CDN account of its own. A proxy missing from the list is counted as one client for all its visitors, so add its ranges to
http_proxy_ranges.yaml. - Path 6 names a form by its host in lowercase and its path up to the PHP file, so extra path after the file or a change of case does not split it, and one
ignore-scriptwith that name covers every such variant. Other names for the same site, such as with and withoutwww., still count as separate forms. - Only WordPress admin pages count as staff evidence. Staff of other back offices, such as PrestaShop, OpenCart or Joomla, who send many order or account mails can be challenged and then blocked. Put a fixed office address in
infra_ips. - Counts live in memory and are lost when the daemon restarts. The startup walk sees only mail still in the queue, and form replies usually leave it within seconds.
- A bot that rotates addresses and keeps each one below the threshold is never challenged, and its form is not named.
- An IPv6 client is counted per /64 but challenged or blocked only at the one address that crossed the threshold.
- A script finding names the client that triggered it. Later clients through the same form in the window get their own client findings but are not added to the script finding.
- One stranger mailed many times counts once.
When a client crosses the threshold, CSM raises a High finding for it, once per mode per window. ip_recipient_mode sets what follows. In observe mode, the default, the finding is email_php_form_relay_observed: it goes to the daily summary and never challenges or blocks, so you can review it before switching on enforcement. In enforce mode the finding is email_php_form_relay and the client is sent to the challenge page; a client that does not solve it within 30 minutes is blocked while auto_response.enabled and auto_response.block_ips are on. With the challenge off, the client is blocked directly under the usual auto_response settings. auto_response.php_relay.dry_run applies only to the mail freeze and does not hold this response.
After a reload from observe to enforce, the next eligible message can trigger enforcement using recipients still in the window. The earlier observe report does not delay enforcement. Switching back to observe prevents new responses from retained findings, queued blocks, challenge timeouts and central corroboration. Existing challenges and blocks keep their expiry.
CSM also raises email_php_form_relay_script (High) for each script that sent at least two of that client’s outside recipients, once per script per window. It names the account and the form to fix: add a captcha, or turn off the automatic reply to the submitter. For a legitimate bulk sender, use csm phprelay ignore-script. This finding has no source address field and never challenges or blocks. If the client table fills up, CSM drops the clients with the fewest recipients first and reports email_php_form_relay_table_full at most once a day.
The detector starts a one-shot retrospective scan of exim_mainlog at daemon startup so Path 2b can fire on history already on disk. IN_Q_OVERFLOW triggers a bounded recovery walk of the spool (capped at 1000 files; if more were skipped, a email_php_relay_overflow_scan_truncated Critical fires too – Path 2b backstops the missed messages).
Operator suppressions (csm phprelay ignore-script <host:/path>) short-circuit the pipeline before any path scoring runs, so a known-noisy contact form can be opted out individually without disabling the detector. An ignore on a lowercase host:/path ending at the PHP file also covers that file reached with extra path after it or a different host case. See PHP-relay CLI for the full operator surface.
PAM Brute-Force Listener
Real-time authentication monitoring through the PAM module.
- SSH login tracking with geolocation
- SSH failures (
service=sshd) drivepam_bruteforceand credential stuffing / password spray breadth: one source IP failing against many distinct accounts insidethresholds.multi_ip_login_window_min. The finding iscredential_stuffing; tune the account floor withthresholds.cred_stuffing_distinct_accounts(default 5). - Only successful sshd logins clear SSH failure counts.
- Failures from other PAM services (FTP, mail, webmin and similar) raise
pam_auth_failuresonce per window abovethresholds.pam_bruteforce_threshold. It never blocks: CSM reads pure-ftpd, Dovecot and Exim logs separately, and counting one login twice would corroborate it with itself. Services without such a log detector, such as vsftpd, proftpd or webmin, have no blocking signal from PAM. - Blocks IPs within seconds of threshold breach
- Integrates with the nftables firewall for instant blocking
Process Context
Exec and outbound-connection findings carry an optional process object with
PID, PPID, UID, user, cPanel account (when known), comm, exe, sanitized cmdline,
and a parent chain up to depth 5. The chain is materialized from an in-memory
LRU+TTL cache (cap 16384 entries, 30-minute TTL) populated from BPF exec
events. Cache misses trigger a bounded async /proc read, so process-context
enrichment does not add blocking work to the connection event loop. When
neither cache nor enricher has data (e.g., a process that exited before
userspace reads its event), the process field is omitted entirely and the
finding still emits.
Counters exposed at /metrics:
csm_process_context_cache_entriescsm_process_context_cache_evictions_total(LRU)csm_process_context_cache_ttl_purges_totalcsm_process_context_cache_misses_total(includes TTL purges)csm_process_context_enrich_queue_drops_totalcsm_process_context_enrich_reads_totalcsm_process_context_enrich_errors_totalcsm_process_context_enrich_stale_totalcsm_process_context_enrich_latency_seconds
Caveats:
started_atis emitted only when the event source supplies a trustworthy start timestamp. CSM does not infer it from procfs directory metadata.- After daemon restart, the
csm_process_context_enrich_*counters may show a smallenqueued - readsdelta. Pending requests in the enricher queue are dropped on shutdown by design. - Hosts without BPF support fall back to
/proc/net/tcp[6]polling. That path has no PID, so emitted findings do not carry aprocessfield.
HTTP Flood, Scanner Profile, UA Spoof, and Distributed Flood
http_request_flood, http_scanner_profile, http_claimed_bot_unverified, http_ua_spoof, and http_distributed_flood are periodic, not real-time. They run inside the same wp_bruteforce scheduled check that scans per-vhost access logs every 10 minutes. A real-time inotify tailer would need to hold per-IP state across log rotations and is out of scope for the initial release (see the plan non-goals). For attack types where sub-minute response matters, the access-log inotify watcher already covers wp_login_bruteforce and xmlrpc_abuse; the periodic scan adds volume-based rate enforcement, pending claimed-bot handling, scanner-profile detection, and per-vhost distributed attack rollups on top.
Requests that a cPanel proxy subdomain (cpanel., webmail., whm. and similar) passes through the web server to the panel are skipped by the web detectors, realtime and periodic, when cPanel’s own central access log records them as proxied. The panel’s own access log carries those requests, so a busy cPanel or webmail session through a proxy subdomain does not count as web traffic, and one request does not count as two kinds of evidence. Requests in a per-site log, or in a log chosen through web_server.access_logs that is not one of cPanel’s own access logs, always count as website traffic.
Web counters use the request and timestamp written by the web server, including requests with Basic authentication usernames and quoted header values. For periodic client attribution, web_server.trusted_proxies and challenge.trusted_proxies together enable X-Forwarded-For from a configured remote proxy; the log format must record it immediately after User-Agent, optionally preceded by a vhost in host:port form. Trust only proxies that append their observed peer or replace untrusted input with that address; trusted ranges must not include visitors. The client is the first address from the right that is not a trusted proxy. The parser keeps complete tokens at the right end, so discarded client text to the left cannot name a client. If no client survives, including when the retained end contains only trusted proxies, the request is not counted against anyone. A listed address is never counted as a client, even where its header is not trusted.
Realtime checks read the connecting address, not X-Forwarded-For, and act only on a bare address in its canonical form. When that address is a configured trusted proxy, in web_server.trusted_proxies or challenge.trusted_proxies, the WordPress, XML-RPC and admin panel checks skip the request, ModSecurity denies and warnings stay visible without escalating, and cPanel API auth failures are not reported. The scheduled ModSecurity audit scan does not report it either. Other cPanel audit and login findings stay visible without blocking or scoring the proxy, and supply no automatic response forecast. A listed address is skipped even where its forwarded header is not trusted, such as an address of this host. The web server must log the connecting peer, not a forwarded client, in that field.
A verified crawler is dropped from the scan before any counter increments: a source IP whose claimed bot User-Agent passes IP-range or reverse-DNS verification cannot contribute to a flood, scanner, or spoof finding, so verified Googlebot and AI crawler traffic does not produce false positives. Verification is covered in Threat intel.
| Finding | Fires when | Key gates (defaults) |
|---|---|---|
http_request_flood | One source IP makes too many requests inside the rate window | http_flood_threshold requests within http_flood_window_min (5 min). Ships disabled (0) so operators sample baseline traffic first |
http_scanner_profile | One source IP’s in-window traffic is almost all probe-error responses spread across many distinct paths, the shape of URL enumeration hunting for backups, downloadable files, and dormant shells | Three gates must all pass: http_scanner_min_requests volume (ships 0, disabled), at least http_scanner_error_pct (90) of requests on a probe-error status, and http_scanner_min_distinct_paths (10) distinct error paths. Probe-error statuses default to 404 and 403; query strings are stripped so cache-buster URLs on one missing endpoint count once |
http_ua_spoof | One source IP sends non-browser User-Agents | Known scanner agents (nikto, sqlmap, nmap, wpscan, nuclei, and similar) fire on the first hit. Claimed crawler UAs with a cache-confirmed reverse-DNS negative, scripting (curl/python/wget), headless (Puppeteer/Playwright), and empty agents fire at http_ua_spoof_threshold (30); scripting/headless/empty still require their opt-in flags (http_ua_scripting_enabled, http_ua_headless_enabled, http_ua_empty_enabled) |
http_distributed_flood | Many distinct already-abusive source IPs hit one vhost in a single scan window | Opt-in: fires once http_distributed_min_ips distinct IPs (sample 10), each having already crossed a per-IP abuse threshold above, hit the same vhost. Built only from IPs that tripped another finding, so a popular site’s normal visitor spread does not trip it |
Per-IP findings roll up per vhost, so a confirmed scanner that sprays a few probes across many vhosts still feeds the distributed rollup. Full threshold reference and tuning notes are in Configuration.
Direct SMTP Egress
Outbound connections to SMTP ports from non-MTA local processes
emit a direct_smtp_egress finding. See
Direct SMTP egress for the full rule set,
config schema, and metric.
Critical Checks
Critical checks run every 10 minutes. Typical wall-clock cost on a busy shared host is a few seconds; the runner enforces the 10-minute cadence even when a tick takes longer.
The tables below name the finding identifiers CSM emits, grouped by area. Most are standalone scheduled checks; a few (for example the http_* traffic findings and bad_asn_outbound) are produced by a parent scheduled check – the access-log brute-force scan and the periodic connection scan respectively – rather than being independently scheduled.
Process & System
| Check | Description |
|---|---|
fake_kernel_threads | Non-root processes masquerading as kernel threads (rootkit indicator) |
suspicious_processes | Reverse shells, interactive shells, GSocket, suspicious executables |
detached_web_process | A non-root process running for at least five minutes that holds a PHP handler listener (LiteSpeed LSAPI or PHP-FPM) with no PHP or web server runtime in its ancestry: a web request started it and it outlived the request. A shell is Critical, the loop shape a webshell leaves behind; with kill_processes on it is stopped with its shell parents of the same account under the rules for loops that write back a removed file, unless a terminal or an SSH, cron or cPanel session is in its ancestry. BusyBox is treated as a shell only when running a shell applet. Anything else, such as a site’s PHP worker or an image job, is a Warning for review and is never stopped on this evidence. Processes with incomplete ancestry, and listeners in other network namespaces, are not classified |
docroot_binary_exec | A non-root program running from a document root. The live exec monitor judges each start; this check judges programs still running. Critical (or the rule’s severity) when the program matches a YARA rule, whoever started it; Warning when a PHP interpreter started it and no rule matched. CGI programs run by the web server, build tools such as esbuild and other launchers stay quiet unless a rule matches. Program verdicts are cached by file identity and change time, scan-size limit, and detection version. Rule reloads and worker restarts invalidate cached verdicts. Oversized images and scripts receive only a header read. Scripts are not programs here: their interpreter lives outside the document root. Exited processes are inspected only through canonical absolute paths without symlinks; missing processes and files are quiet. Starts that arrive faster than the live monitor can inspect them are counted in the bpf.execution.docroot queue health row |
php_processes | PHP process execution, working dirs, environment variables |
shadow_changes | /etc/shadow modification outside maintenance windows |
uid0_accounts | Unauthorized root (UID 0) accounts |
kernel_modules | Kernel module loading (post-baseline). High. af_alg and algif_hash, which upgrades load on demand, are a Warning when a package was installed or updated since the previous scan; every other module, algif_aead (Copy Fail) included, stays High |
af_alg_socket_use | AF_ALG socket use that may indicate Copy Fail exploit activity |
af_alg_enforcement | AF_ALG hardening policy drift and correction status |
SSH & Access
| Check | Description |
|---|---|
ssh_keys | Unauthorized entries in /root/.ssh/authorized_keys |
sshd_config | SSH hardening (PermitRootLogin, PasswordAuthentication, etc.) |
ssh_logins | SSH access anomalies with geolocation |
api_tokens | cPanel/WHM API token usage |
whm_access | WHM/root login patterns, multi-IP access |
cpanel_logins | cPanel login anomalies, multi-IP correlation |
cpanel_filemanager | File Manager usage for unauthorized access |
Network
| Check | Description |
|---|---|
outbound_connections | Root-level outbound to non-infra IPs (C2, backdoor ports) |
user_outbound | Per-user outbound connections (non-standard ports). System accounts for Knot DNS (TCP 853) and chrony or ntpsec (NTS key exchange, TCP 4460) are exempt on that port only; hosting accounts with those names are still reported |
bad_asn_outbound | Outbound connection whose destination resolves (via GeoLite2-ASN) to a bad or unexpected autonomous system. Config detection.bad_asn_outbound: blocked_asns (always bad) and/or allowed_asns (allowlist mode – anything outside is bad). Classified for every process including root (the periodic connection scan); non-root connections are also flagged in real time by the live BPF tracker. Off by default; the third leg of the host_takeover incident chain |
dns_connections | DNS exfiltration and suspicious queries |
firewall | Firewall status and rule integrity |
Brute Force & Auth
| Check | Description |
|---|---|
wp_bruteforce | WordPress login brute force (wp-login.php, xmlrpc.php) |
http_scanner_profile | Source IPs whose traffic is mostly probe-error responses across many distinct paths in one scheduled scan window |
http_claimed_bot_unverified | High-volume claimed crawler traffic while reverse-DNS verification is still pending; challenge-routed when the challenge subsystem is enabled |
http_ua_spoof | IP hitting the per-IP spoof threshold with a search-engine bot UA that fails reverse-DNS verification, or with scripting/headless/empty UAs when those opt-in flags are enabled |
http_distributed_flood | Many already-abusive HTTP source IPs hitting the same vhost in one scheduled scan window |
http_asn_crawl | A single ASN distributing expensive, uncacheable requests across many addresses while one account’s PHP worker pool is saturated. Request classification is a heuristic, not proof of backend cost. Reverse-proxy ASNs and operator allowlists are excluded; Critical findings can carry the specific source CIDRs eligible for temporary blocking. |
webshell_caller | A client that requested a PHP file CSM flagged as a webshell, one finding per client and account, with request and POST counts, the files and a sample request. Requests keep counting after CSM removes the file. Known site entry points, files the installed WordPress or plugin release ships, and a document root’s own index are excluded from caller attribution. A request alone does not prove the client operated the shell; a compromised page can direct visitors there. cPanel hosts only. It reads the recent tail of each site log, so a single request on a busy site can be missed, and a file detected only in real time counts once a scheduled scan reports it. Alert only: no automatic response |
webshell_path_public | A flagged PHP file requested by many distinct clients across all domains serving it. Infrastructure addresses and verified bots count toward this summary but never receive individual caller findings. Its clients are not reported one by one as webshell_caller |
ftp_logins | FTP access patterns and failed auth |
webmail_logins | Roundcube/Horde access anomalies |
api_auth_failures | API authentication failure patterns |
Webshell caller attribution uses dated requests from the last 24 hours and preserves the full request target before resolving dot segments and PHP path info. Above-root traversal is rejected. URL paths are decoded once, including encoded slashes, before resolving dot segments. For a directory request, a trailing decoded slash permits an index.php lookup. Double-encoded slashes are not decoded again. Attribution counts probes, including 404 responses, and does not prove that PHP ran. This assumes decoded path separators; server settings that preserve them, such as Apache’s NoDecode policy, URL rewrites and custom directory indexes are not inferred. When a client accesses the same file through several domains, its account finding does not name a single domain. These exclusions affect caller attribution only; file scanning is unchanged.
| Check | Description |
|---|---|
mail_queue | Mail queue buildup (spam outbreak indicator). Exim aborts when it cannot write its own log, and the daemon’s sandbox makes /var/log read-only, so the queue query runs as a transient unit forked by PID 1. Hosts without systemd-run query exim directly. |
mail_queue_unavailable | The queue depth could not be read, so buildup detection is inactive. Reported instead of assuming the queue is empty. |
mail_per_account | Per-account email volume spikes |
All Exim queue reads and actions started by the daemon run as transient services outside its read-only filesystem sandbox. This includes queue composition, safe backscatter flushing, and PHP relay freeze or thaw actions. If systemd-run is unavailable, CSM runs them directly; a host without Exim skips the Exim-only queue check. The target command is attempted at most once, and cancellation during the wrapper probe stops it before execution.
Data & Integrity
| Check | Description |
|---|---|
crontabs | Suspicious cron jobs and scheduled commands |
mysql_users | MySQL user accounts and privileges |
database_dumps | Database exfiltration attempts |
exfiltration_paste | Connections to pastebin/code-sharing sites |
The MySQL audit exempts mysql@localhost only when MariaDB confirms its stock
socket authentication setup with password authentication disabled. Modified
authentication, other host literals, and accounts whose setup cannot be verified
remain reportable.
Threat Intelligence
| Check | Description |
|---|---|
ip_reputation | IPs against external threat databases and optional rspamd history. Passive HTTP/cPanel sightings are High; SSH and mail-auth activity is Critical |
modsec_audit | ModSecurity audit log parsing |
The attack database counts a finding only against the source address the
finding carries as a structured field. Addresses that appear only in message
text, such as a connection’s destination, a process name or a file path, never
build a score. Scores that earlier releases built from such text are corrected
automatically when the daemon starts; blocks they already caused are not lifted.
The correction has no setting; auto_response.block_ips controls new automatic
blocks.
Raw mail and SMTP login failures are not counted. The mail and SMTP brute-force detectors decide those blocks, and they account for successful logins and mail authentication backend outages, so a mistyped password or a backend outage never raises an address’s local threat score.
At the upgrade restart, stored attack records are re-scored and retired scoring state is removed on the next successful save, including records whose score is unchanged. Existing counts and event history are retained, and existing blocks stay until expiry or operator removal. Rolling back before that save can restore the retired scoring state; an older daemon also resumes scoring new raw mail failures.
The local threat score ranks addresses on the threat page and feeds the unified
verdict. No scheduled check turns it into findings or blocks; the checks that
provide attack evidence apply their own response policies.
No check produces local_threat_score findings any more: the database
malware response reports the addresses of live WordPress sessions in its
notice instead of blocking them, since the site owner’s sessions are among
them. Findings older versions stored under that name clear after the next
completed critical scan. A legacy disabled_checks: [local_threat_score]
value is still accepted but does not disable ip_reputation or its health
checks. Queued score findings and block retries under that name are discarded;
existing blocks stay until expiry or operator removal.
auto_response.block_ips controls new automatic blocks; there is no setting for
the startup correction.
The local threat score retains evidence attributed to the server itself. Such records can represent forwarded attacks or a compromised local process; firewall protection against blocking the server does not establish that traffic is safe.
Performance
| Check | Description |
|---|---|
perf_load | CPU load average thresholds |
perf_php_processes | PHP process count and memory |
perf_memory | Swap usage and OOM killer activity |
Health
| Check | Description |
|---|---|
health | Daemon health, binary integrity, required services |
Platform Support
Runs on every supported platform unless noted below. The daemon auto-detects OS and panel at startup and silently skips cPanel-specific checks on plain Linux hosts (no “not found” spam).
cPanel-only (skipped on plain Ubuntu/AlmaLinux):
api_tokens,whm_access,cpanel_logins,cpanel_filemanager– read WHM API and cPanel session logswp_bruteforce– iterates/home/*/public_html/*/wp-login.phpand per-domain access logs. The domlog pass ranks recent logs first and honorsthresholds.domlog_max_files,thresholds.domlog_tail_lines, andthresholds.domlog_max_age_min.webmail_logins– parses cPanel Roundcube/Horde logsmail_per_account– reads/var/log/exim_mainlog
Plain Linux equivalents that still provide coverage:
mail_queueruns on any host where Exim is installed and is skipped when there is no Exim queue.- Access log brute-force detection (
wp_login_bruteforce,xmlrpc_abuse) runs against the detected web server’s access log (/var/log/nginx/access.logor/var/log/httpd/access_log), so WordPress brute-force alerts still fire on non-cPanel hosts – they just rely on the live log watcher rather than per-domain domlog scanning. modsec_auditruns on any host with ModSecurity installed.ssh_logins, SSH brute force, PAM listener, firewall, kernel modules, RPM/DEB integrity, and threat intelligence all run on every supported platform.
Deep Checks
Deep checks run every 60 minutes and cover thorough filesystem, CMS, email, and database scans.
Scheduled and account scan checks share one budget within the daemon, sized
from the machine’s core count with a minimum of two and a maximum of five
concurrent checks. Web UI scans and CLI jobs submitted with csm scan --full
share this budget. Standalone CLI scans run in a separate process. A cancelled
or timed-out check retains its slot until it actually exits, while its caller
can return promptly. Waiting for slots occupied by another scan does not count
as stalled dispatch in queue health.
The real-time scanner has a separate pool of two to sixteen workers, also sized from the core count. Its two-worker minimum lets other events progress while one content scan is slow. Both the packaged and installer-generated service units give CSM a lower CPU weight than the default. Under contention, this favors sibling services with default weights while allowing scans to use spare CPU on an idle host; it is not a CPU quota.
Alert delivery
Alert email and the generic alert webhook share repeat state. A successful send through either channel counts as delivered. Findings held by the hourly cap, failed sends, or disabled notifications remain eligible on later reports, without waiting for the daily history reminder. This includes findings derived from current observations. Baselines, dismissals and suppression rules still apply, including to the sources of cross-account summaries. The signed phpanel stream keeps its separate delivery policy.
SMTP probe, self-deleting dropper, sensitive file write, email AV degraded and auto-block notice findings keep one alert identity per condition and severity: the source address, the removed file, directory or document root, the written file with its writer and, when the write can be tied to the file’s content, that content, the scanner condition, or the kind of notice. Counts, process or message IDs, ages, sizes, error text and the files sampled in a burst can change without a new alert. Dismissing one of these findings covers its later reports until it goes unseen for 24 hours. A return after that alerts again. Cross-account summaries alert again when the affected accounts change, including after a dismissal. More reports for the same accounts do not repeat the summary. Summaries are derived per dispatch batch, so different scan tiers or realtime batches can report different subsets of the affected accounts, and each subset is emailed once.
An undismissed finding can alert again after 24 hours without an observation, even when no other batch arrived to clean up old state. A finding that stays on the findings page counts as observed, also while its check skips runs between reports, so a check that reports once a day does not email daily. After at least 24 hours of downtime, an open finding alerts once on its next report. A completed scan that covered the finding and no longer reports it also permits a later return to alert. Scan coverage warnings follow the policy below.
Keeping acknowledged findings listed does not rewrite alert state, except for an acknowledgment that lapses with its condition as described above. Other retained findings use the existing periodic save, keeping their delivery state across restarts without changing the displayed last-report time.
Scan coverage alerts
PHP, JavaScript, YARA, host-wide WordPress database coverage, and unfinished email password verification warnings keep one alert identity while their counts change. Alert email and the alert webhook send them once; history records them again once a day unless acknowledged. When the owning scan returns without that condition, its acknowledgment clears, so a later recurrence can alert again. An unacknowledged condition that returns is emailed again only if it went unreported for 24 hours, so a gap that is missing from one complete scan and back in the next sends no new email. An unrelated scan, throttled check, timeout, or cancellation does not prove recovery. Disabling a scanner clears its current coverage condition. Existing per-install database limit acknowledgments are kept when a partial scan cannot inspect those installations.
JavaScript scans bound the paths retained for coverage gaps. If that list is truncated, the coverage warning says so and still counts every analysis failure. The scan keeps all earlier JavaScript detections and can report new ones. A later complete scan can clear findings for content it has examined or that is gone.
PHP analyzer crashes and stalls are tracked by the failing file paths, contents, and failure statuses, including files outside the displayed examples. Different failing inputs can therefore alert even after an earlier failure was dismissed. Check crash alerts ignore stack-trace churn, keep account scans separate by account, and re-arm after the same check returns successfully.
Filesystem
| Check | Description |
|---|---|
filesystem | Backdoors, hidden executables, suspicious SUID binaries |
webshells | Known webshell patterns (c99, r57, b374k, etc.) |
htaccess | .htaccess injection (auto_prepend_file, eval, base64 handlers) plus ten hardened per-pattern detectors – htaccess_php_in_uploads, htaccess_auto_prepend, htaccess_user_agent_cloak, htaccess_spam_redirect, htaccess_filesmatch_shield, htaccess_rewrite_off_shield, htaccess_header_injection, htaccess_errordocument_hijack, htaccess_cgi_handler_abuse, htaccess_security_disabled. Per-pattern findings use the per-pattern audit for cleaning and re-checks; generic findings use the generic scanner plus the per-pattern detectors for their manual fix and re-check. Realtime detection uses that combined audit. Auto-cleaning is gated by auto_response.clean_htaccess, acts on the per-pattern findings and on htaccess_handler_abuse, whose extensions no site uses, and removes only their directives; other generic htaccess_injection findings keep the manual fix. No .htaccess finding is handled by file quarantine. Every prepend or append target carried in a line is judged, including SetEnv values and RewriteRule environment flags with variable names expanded by Apache. Apache quoting and flag separators are not part of the target. Harmless prepend settings and defensive rewrite patterns are not reported, and handler-abuse findings require a complete extension argument in an active handler directive, with one finding per directive. Disabling CGI suppresses CGI handler mappings only when the active options keep CGI disabled; PHP mappings remain reportable. Automatic cleaning requires removable directives, so retained preludes stay reported without entering the automatic response budget. Manual removal of a malicious rewrite also removes its condition chain so the conditions cannot affect the next rule. Oversized files stay unchanged and require manual inspection; re-checks cannot declare them clean. htaccess_security_disabled covers the mod_security 1.x spellings too, because LiteSpeed obeys them by default; Magento’s stock block that turns ModSecurity off is reported and cleaned like any other. htaccess_rewrite_off_shield reports rewriting switched off beside an access grant whose FilesMatch pattern can match no file name, the unfilled template a dropper kit writes; asset folders that grant named file types stay quiet. Only active access grants count, including quoted and continued directives. Cleaning preserves unrelated bytes and removes an enclosing rewrite module block only when the offending switch is its entire body. A prepend of a security plugin’s own file is not reported. For Really Simple Security the target must be the absolute path of wp-content/advanced-headers.php beside the .htaccess, with no dot segments; a document root reached through a symbolic link counts as the same place. Any other target naming that file is reported wherever it points, and when it is a prepend it is never cleaned automatically. |
file_index | Indexed file listing to detect new/unauthorized files |
php_content | Suspicious PHP functions (exec, eval, system, passthru) |
group_writable_php | World/group-writable PHP files (privilege escalation) |
js_keylogger_dataflow | JavaScript keystroke exfiltration found by AST data-flow analysis: a key-event value that reaches fetch, sendBeacon, XHR, WebSocket, or a resource .src sink, including flows laundered through variables that regex rules cannot express. Runs inside the scheduled deep scan on the same file snapshots as the YARA pass but independent of the YARA backend, with its own rolling cursor. Complete sources up to 2 MiB are analyzed; a file the analyzer could not complete (oversize, parse failure, resource limit) is reported in the Warning-level js_taint_scan_incomplete aggregate and its prior finding is preserved, never silently cleared. Recognized PHP and HTML documents (including HTML-first PHP templates), complete JSON/source maps, CSS stylesheets, and gettext catalogs are skipped rather than counted as JavaScript parse failures. Classification uses content, not file extensions, and valid candidate JavaScript is always analyzed even when comments or strings contain document markers. Embedded JavaScript in templates or data files is not extracted or analyzed; PHP analysis and content rules do not provide equivalent JavaScript data-flow coverage. Ambiguous sources and malformed JavaScript retain their coverage warning, and oversize sources keep the size-limit warning. Disable with js_keylogger_dataflow (or the owner ID js_taint_deep) in disabled_checks; disabling the YARA scan does not disable this analyzer. |
symlink_attacks | Symlink-based privilege escalation attempts |
exposed_files | Web-downloadable sensitive files under document roots: version-control directories (.git/, .svn/, reported as web_exposed_repo_metadata and virtual-patched by denying the whole directory), database dumps, full-site backup archives, config/credential backups, PHP source-code backups, and phpinfo.php diagnostics. Each candidate is reported only after a headers-only reachability probe pinned to the vhost’s configured serving IP confirms the server serves it (HTTP 200/206, non-executed body) – files the server already blocks (403) and shipped samples such as wp-config-sample.php are never flagged. The domain is preserved for HTTP Host and TLS SNI, while DNS and front-end proxies are bypassed. The probe reads status and content type only, never the file body. The one exception is phpinfo.php: a 200 response alone cannot prove a dump, so CSM reads a bounded portion of that one response and reports web_exposed_phpinfo only when it contains real phpinfo output (the PHP Version banner in a dump-sized body); stub responses are not findings. Findings: web_exposed_db_dump, web_exposed_backup_archive, web_exposed_config_leak, web_exposed_source_backup, web_exposed_phpinfo, web_exposed_sample_sql. A plain SQL file with a sample/schema-specific name under framework/vendor scaffolding (examples/, docs/, vendor/, unpacked *-main/ or *-master/ directories) is reported as the lower-severity web_exposed_sample_sql warning. Archived, renamed, customer-named, and other ambiguous dumps stay Critical. Descent depth is bounded by thresholds.exposed_file_scan_depth (default 2, maximum 10). Optionally, auto_response.virtual_patch_exposed_files (off/manual/auto) writes reversible .htaccess Require all denied rules – manual applies confirmed findings via csm virtual-patch --apply, while auto applies every confirmed class except warning-only sample SQL and honors dry_run. Recognized backup storage directories are denied as a unit so regenerated archives stay blocked. A zip whose name carries no backup wording is classified from its central directory instead, so an archive named after the site it holds is reported when it contains a site-root or structured CMS configuration path, a regular Joomla configuration file at any depth whose same case-sensitive folder also holds a regular Joomla core file, a document-root directory, or a database dump at the archive root; plugin and theme bundles offered as ordinary downloads stay quiet. Readable backup evidence is retained even if later directory records are damaged. The complete entry list is inspected within a fixed metadata budget, without extracting or decompressing payloads; an archive that exceeds the budget leaves the scan incomplete instead of silently clearing an earlier finding. Each change records a rollback entry; restore refuses to overwrite a later customer edit. Findings in this family can be re-checked, and the version-triggered startup sweep re-checks existing findings after verifier changes. A finding clears only after a complete vhost map provides an unambiguous current serving address, both origin protocols answer there, and the shared detection rule no longer confirms an exposure; phpinfo re-checks repeat the bounded body confirmation too. A missing local file alone is not enough. The re-check never falls back to DNS, so a domain that has migrated to another host leaves the finding open rather than clearing on a stranger’s answer, and incomplete routing data or an unreachable or half-answered probe leaves it open too. |
The webshell, .htaccess, phishing and filesystem scans run on every deep cycle alongside the file index, including while the realtime monitor is active: the monitor does not report a setuid bit being set, and it sees files renamed or linked into place only where a placement backend runs. The exposed-file scan runs in that tier too, on its own interval, because it confirms each candidate with a live request to the site rather than by reading the file. A cycle that skips it on that interval leaves its existing findings in place. The interval starts only after a complete exposure scan; an incomplete attempt can retry on the next cycle, whether or not the realtime monitor is active. A deleted document root does not make the exposure scan incomplete, even if the panel still lists its domain. A missing root reached through a symbolic link, and a subdirectory that disappears during the walk, keep earlier findings until a later scan completes.
Account discovery failures, unreadable directories or files, and truncated backdoor candidate lists leave the affected scan incomplete. Its earlier findings remain active until a complete scan can replace them; detections from the incomplete attempt are still added, whether or not the realtime monitor is active.
The file index runs on every deep cycle, including while the realtime monitor is active. It checks new paths and rechecks indexed files with active findings; being present in the baseline does not clear an alert. Failed content reads preserve prior findings for that file. After a large deletion, scans rewalk the directories until the smaller baseline is adopted, so cached entries cannot restore removed paths or interrupt the consecutive-scan shrink guard.
The PHP content scan reuses a clean result only when the file’s identity and timestamps still match a stable read. Recently changed files, files that change during inspection, and files without usable identity metadata are read again on the next visit. Older cached results are refreshed as files are visited after upgrading; interrupted scans retain progress. Periodic host scans bypass the cache, and explicit full-content scans always read the files they visit.
PHP execution heuristics distinguish attribute metadata and multiline string contents from executable calls. Literal examples do not establish callable bindings or invoke them, and scanning continues through code after attributes. String boundaries and legacy line endings preserve the executable code that follows them. Attribute declarations do not qualify as comment-only stubs for demotion.
PHP and phishing checks read ${'_POST'} and $GLOBALS['_POST'] as $_POST. The same text inside a quoted string or a comment is ignored.
Phishing title analysis ignores a brand that appears only inside an email address, as on a blog author archive named after its author, even when the page also carries a login form. Elsewhere in the title, password capture or a login title lets a brand glued to other words still count. Malformed address-like text does not hide brand evidence.
Re-verifying Findings
Repeated YARA worker buffer-scan failures are logged at most once per minute per tracked error, with suppressed counts reported when the error recurs after that window. If many distinct failures exhaust the tracking limit, additional errors share a bounded summary instead of resetting suppression. Scan errors still reach callers and preserve incomplete-scan reporting.
A finding records what was true when it was raised. The condition behind it is often resolved by someone else – an operator cleans a file, a virtual patch denies an exposure – and none of that moves CSM’s own rules, so re-verification runs once per deep-scan cycle as well as when the re-check logic changes.
A scan retires a finding it did not raise again only for files it actually examined. Coverage is tracked per file and scanner: a gap that names a file – one past the scan size limit, or one that could not be opened – keeps that scanner’s findings for the file and nothing else, while a gap with no path, such as a directory the walk could not enter or a stat that may hide a subtree, keeps every finding the scanner owns, because the unscanned range is unknown. This matters more than it sounds: a single oversized log that will never fit under the limit is a permanent gap, and treating it as a whole-scanner gap froze every finding on the host indefinitely. On a large server the rolling deep scan covers the files over several runs, so each run retires findings for the files it examined, up to where its time budget ran out. Overlapping and symlinked scan roots are walked in one path order, so an interrupted run never retires findings in a root it has not reached. A finding raised by an older version is not tied to its file this way, so it waits for a complete scan or re-verification.
Each family is re-checked by re-running the test that raised it, giving four outcomes:
- Cleared. The condition is provably gone: the file was removed, or the server no longer serves it as an exposure. If the same detection returns later, it alerts as a new finding.
- Demoted to Warning. The flagged content is gone but the file changed since detection, so the cleanup cannot be proven byte-for-byte. The finding is kept – an attacker must not be able to retire one by editing the file – but it stops ranking beside live threats. Only a replacement proven inert qualifies: an empty file, or a comment-only PHP stub with a valid opening tag and no closing tag. A malformed opening tag can leave the file serving page content, so it does not qualify for demotion. This is an allow-list on purpose. A deny-list of dangerous shapes would make every omitted include, callback or inline script a way to buy a lower severity.
- Restored. A demoted finding whose replacement stops being inert returns to the severity it came from, so a second edit into a detection gap cannot leave live content sitting at Warning.
- Left alone. Anything uncertain: an unreachable or half-answered probe, a domain no longer served by this host, a replacement that changed while it was being read, a file that still matches.
A finding keeps its identity throughout; only its severity changes, because a finding’s key is derived from its details. Every store mutation is conditional on the snapshot verification actually examined, so a scan or realtime alert that refreshes the same key while a re-check is in flight is never overwritten by the older verdict. An unconfirmed demotion also survives a scan that does not raise the finding again, since that scan is weaker evidence than the verifier that reads the file.
PHP Remote-Source Taint Analysis
This analyzer runs as part of the scheduled deep-content scan, alongside the YARA and JavaScript consumers, and reports php_remote_taint. Disable it with php_taint_deep (or php_remote_taint) in disabled_checks.
It runs in a separate process, and that is not an implementation detail. A single file can put the underlying PHP parser into a state it never returns from, which no in-process deadline can interrupt. Analysis therefore happens in a supervised child process that is killed outright when a file exceeds its deadline; that file is reported as reduced coverage and the scan continues. Repeated failures are rate-limited so no single account can consume the scan budget, and the analyzer recovers without operator action. Whenever a file cannot be examined – including when no worker is available – it is reported as unexamined, never as clean. The byte check that decides whether a file could hold such a flow at all needs no parser, so it runs in the daemon itself: a file without a PHP open tag, a code-execution construct and a remote-fetching call is ruled out there and never reaches the worker. Cancellation during this byte check is still reported as unexamined.
CSM includes a PHP analyzer that looks for code fetching content from a remote server and then executing it, even when the fetch and the execution happen in different functions. This is a flow that regular expressions cannot express: matching it requires binding the value a fetch returns to the value handed to execution, which is beyond what YAML pattern rules or YARA-X can do.
The analyzer parses PHP source and tracks whether a value returned by a remote-fetching call reaches a code-execution construct (eval, include, include_once, require, require_once, create_function, or assert given a string argument), following the value through variable assignment, string concatenation, decoding calls, across function and method boundaries, and through a file_put_contents write into an include of the same path expression. curl_exec, curl_multi_getcontent, wp_remote_get, wp_remote_retrieve_body, and fsockopen are always treated as a remote source, whatever their argument. file_get_contents and fopen are dual-use: only these two calls have their argument inspected, and only when that argument carries an HTTP, HTTPS, FTP, FTPS, php://input, or data:// scheme are they classified as a remote source rather than a local read. Not every php:// stream counts as remote: php://input is request-controlled and treated as a source, while php://memory, php://temp, and a local php://filter resource are not – though a filter wrapping a remote resource still carries its nested remote scheme and is classified accordingly. A URL used directly as a sink’s own argument, with no acquiring call in between – include 'http://evil/x.php' – is out of scope: the pre-filter requires a source keyword before parsing is even attempted, and a bare remote include has none, so it is reported not_candidate, not analyzed.
Only two outcomes mean a file was actually examined: analyzed (parsing and the data-flow pass both completed) and not candidate (a fast pre-check proved the file cannot contain a reportable flow, without needing to parse it). Every other outcome – oversize, a parse failure, a parser recovery that produced only a partial tree, an internal resource limit, cancellation, or an internal error – is a coverage gap, not a clean result, and must never be read as “nothing to see here.”
An oversize file is judged once more before it counts. Only a file whose leading bytes could be source of that language is reported. Both analyzers are handed every readable file the walk produces, and their own pre-check rejects the rest instantly – but that pre-check never runs on a file too large to send, so without this step every large image, archive and compiled catalog on the host arrived as source the scan had failed to examine. PHP is judged by an opening tag anywhere in the inspected prefix, including after binary content. Binary content carrying PHP tokens can therefore remain reportable; the prefix alone cannot establish that it is inert. JavaScript has no such marker. A prefix without NUL is admitted; one containing NUL gets a conservative lexical check, since binary characters are legal in JavaScript literals and comments. Incomplete tokens and ambiguous syntax remain reportable. Neither gate looks for taint source or sink keywords in the prefix. A read that fails answers yes either way, because a file the scan could not examine is exactly what the report exists to name.
A finding’s severity follows how firmly the source was shown to be remote: a decoder-confirmed remote fetch reaching execution is Critical, a fetch carrying a remote URL is High, and a dual-use call whose argument could not be resolved either way is a Warning for review. When one file contains several flows, its strongest flow sets the severity. The last group is where legitimate template compilers and cache layers land, so it is kept visible without paging anyone.
Each flow in a finding’s details reads source -> sink (confidence, basis), for example curl_exec -> eval (high, always-remote) or file_get_contents -> include (low, unresolved). The basis says how the source was identified: always-remote (the call can only read over the network), literal (the argument text carries a remote scheme), decoded (the scheme appears only after escape or builtin decoding), request (a requester can supply the start of the path), call-argument (a call site in the same file passes the remote argument), or unresolved (the analyzer could not decide whether the argument is local or remote). The basis describes the strongest proof that reaches the sink, which can come from a different source call than the one the flow names. Some basis values appear only in later versions.
A finding’s identity is its file, its severity and the source and sink pairs of its reported flows. A new version that changes only the details wording or the basis does not show a dismissed finding again. Any change in those flows or severity, a new file, or a change to the file’s content does. When evidence is capped, basis ranking does not change which equal-confidence flows are retained.
The WordPress database scan reports what it could not fully inspect as
db_content_scan_incomplete, counting affected installs against the number
discovered and naming one bounded example config path per reason:
unreadable_config, missing_credentials, unresolved_table_prefix,
query_failed or incomplete_content. Example paths use ASCII escapes for
control characters, non-ASCII bytes, quotes and backslashes, with the existing
length limit applied after escaping. Content gaps include truncated or
unusable query results. A server regular-expression timeout is a statement-local
failure: independent checks continue, while the failed check keeps coverage
incomplete and preserves earlier findings. Hidden-link selection examines only
the leading and trailing parts of each value that the parser reads, so large
values cost no more than the parsed sample. When the parser joins those parts
into a complete value, selection examines them together too. Ordinary CSS
declarations are matched with literal searches; guarded expressions handle
commented and encoded styles without admitting unrelated page text. Comment
bodies are preserved during matching so normalization cannot change their
meaning. If the server stops a regular expression, the selection is repeated
without commented styles, so plain
and encoded styles stay covered while the coverage gap is still reported. Each
install counts once; installs sharing a failed database count as affected without
retrying its queries. Incomplete discovery
is reported separately in the details because additional installs may be
missing from the total. With no attributed reasons, including when discovery
stops before reaching any install, the generic three-cause sentence remains
the fallback. Multisite safety limits keep their own account-specific warning
and do not count again in the summary or hide unrelated failures. Installation
findings carry an opaque database scope; the host-wide coverage summary remains
unscoped. A complete installation scan retires
its resolved findings even when another installation fails. Failed or
undiscovered installations retain their findings in the same atomic store
transaction. Older findings without a database scope stay until reobserved or
until the whole scanner completes; CSM does not guess their database from
message text. A partial multisite scan keeps findings for its entire network.
Incomplete or interrupted scans protect earlier findings from eviction when
new results fill the active list, including when every database or discovery
attempt fails, the scanner times out, or an internal panic stops execution.
That protection applies to findings already in the active list. Newly detected
conditions compete for the remaining space under the normal priority order,
so repeatedly incomplete scans cannot grow the list beyond its cap. Retained
findings can still refresh their details without losing their first observation.
Credential aliases sharing a database scope must all complete before that scope
can retire findings, regardless of scan order. This includes an alias whose
database and table prefix are known but whose login credentials are missing.
Query diagnostics include the detector stage, failure class and numeric MySQL error code. Repeated errors are counted together, with bounded detail when many causes occur. Raw SQL, server error messages and credentials are never included. Known statement errors, such as a missing table or column, leave the scan incomplete but allow independent detectors to continue. Connection, authentication and unknown failures stop further queries for that database.
Coverage the scan could not reach is reported as php_taint_scan_incomplete, which names how many files were affected and why – a per-file status such as a timeout or a worker failure, or a location the walk could not read at all, where the affected files cannot even be listed. Panics and timeouts are reported in a separate aggregate so hard analyzer failures remain visible beside routine coverage limits. A file that had a finding and later becomes unexaminable keeps its previous finding rather than having it cleared.
Separately from the outcome, an analyzed file can still report reduced precision. Constructs that defeat static variable identity – extract(), compact(), variable variables, a call dispatched through a value, an assignment target the analyzer cannot name, or a value a closure or arrow function captures from its enclosing scope – are recorded alongside the result. A recorded loss means tracking stopped at that point and the file may hold a flow that was not followed; it is never left implicit. The capture case is recorded only when the captured value was itself tainted, so it marks a real loss rather than the mere presence of a closure.
The parser supports PHP syntax up to version 8.1. A file written against a newer PHP version may use constructs the parser does not recognize; when that happens, parsing recovers what it can but the result is incomplete, and the file is reported as reduced coverage (a partial parse) rather than analyzed, so an incomplete view is never presented as a complete one.
WordPress
| Check | Description |
|---|---|
wp_core | Core file integrity via official WordPress.org checksums. A command timeout does not mark files as verified, and a timed-out re-check stays unresolved. Repeated verification failures produce wp_core_unverified. A core file proven to differ from the release only in line endings is not reported, and an install with no other difference counts as verified. Standalone PHP deprecation notices do not block this comparison; PHP warnings and ambiguous output still do. For any flagged file not proven so, the daemon log says why at Info level, without copying command output. Incomplete checks stay unresolved. |
wp_plugin_inventory | Reports persistent plugin inventory failures as wp_plugin_inventory_unverified. Shares the existing refresh and interval with the outdated and vulnerable plugin checks. |
nulled_plugins | Cracked/nulled plugin detection |
outdated_plugins | Plugins behind the latest release, graded by version gap |
vulnerable_plugins | Installed plugins whose version matches a curated known-vulnerable feed (CISA-KEV + confirmed in-the-wild CVEs). Fires only from a fresh shared inventory when a parseable version is inside the affected range; patched, stale, and unparseable versions stay silent. Matched versions are High or Critical regardless of version gap, including inactive plugins because their files remain reachable. Alert-only (never disables a plugin). Toggle detection.vulnerable_plugin_scanning; accept one reviewed build via detection.vulnerable_plugin_allow (slug@version, case-insensitive). For an active install, a host engine that is Off or DetectionOnly, or a disabled account or vhost scope, marks the finding unprotected and raises it to Critical: no request filtering applies and no modsec audit record is written. When CSM ships a virtual patch for that CVE (virtual_patch in the feed) the alert says that patch cannot run; otherwise it claims only the filtering gap. An inactive install is never annotated – its files are why it is reported, but WordPress does not load the vulnerable code path, so a missing filter says nothing about reachability. An addon domain is linked only to its exact cPanel-associated subdomain; unrelated parked, main, and addon sites that happen to share a document root do not inherit one another’s disabled flags. Off cPanel, where CSM deploys no virtual patches and no per-vhost ModSecurity state exists, findings are left as the detector graded them. |
vulnerable_timthumb | Bundled TimThumb (timthumb.php / thumb.php) image-resizer scripts, the abandoned library whose remote-code-execution bug (CVE-2011-4106) is a recurring WordPress entry point. Files are confirmed by TimThumb’s own constants (not just filename) so generic thumbnail helpers are never flagged. A version below the last patch (2.8.14), an unparseable version, or an enabled WebShot / external-fetch feature is reported High; a patched-but-deprecated copy is a Warning to remove. The final release is suppressed only when ALLOW_EXTERNAL, ALLOW_ALL_EXTERNAL_SITES, and WEBSHOT_ENABLED are defined as false in executable PHP and none is also defined as true. Alert-only – TimThumb is never auto-quarantined, since deleting it would break the theme. |
db_content | Database injection (an external script loader on an ordinary HTTPS host that carries no attacker marker is reported once, as a Warning db_options_new_external_script, the first time that host appears in an option after the site’s first scan; plugin status options that WordPress renders as dashboard notices – LiteSpeed Cache’s CDN setup summary and its stored message list – are read by name instead and reported Critical as db_options_plugin_notice_injection when they carry markup that executes, because those rows hold plugin state and never site content, so neither host reputation nor the first-seen baseline applies and a payload stored before the site’s first scan is still reported. The named notice rows are read without a script-only SQL filter; complete values up to 64 KiB are inspected, while oversized or malformed query rows mark the scan incomplete and retain prior findings. Notice findings are alert-only; stored JSON and serialized data are not rewritten by this check. If another finding triggers option cleanup, cleanup refuses to write while any executable notice markup remains, even when its host has no reputation marker. A value WordPress stored as JSON has every slash escaped, so script URLs are normalised before a host is extracted, in options, in posts, and in page-builder content), siteurl hijacking, rogue admins, spam, a sudden publishing flood measured against the site’s own history (db_post_volume_burst; only on sites older than 400 days, with at least 50 recent posts and more than 5x everything published before – a genuine content migration looks the same, so it is alert-only), spam categories, tags and other taxonomy terms left behind when spam posts are removed (db_spam_taxonomy; a term named after a URL is High, spam vocabulary alone is Warning), PHP-capable snippets stored in the database by WPCode (db_stored_code_execution, Critical when the snippet is published and therefore running, lower when it is a draft or trashed; stored code is invisible to filesystem scanning), stored snippets that both defeat request caching (true DONOTCACHEPAGE, false WP_CACHE, WordPress no-cache headers, or LiteSpeed’s no-cache response header) and inspect the user agent for a search or SEO crawler (db_stored_cloak_logic, High when published and Warning otherwise; direct, concatenated, and ROT13 crawler names are recognised, while a snippet that already matched a malware signature carries the cloak as a bounded note rather than raising a second finding), outbound links wrapped in a container the page hides from readers (db_hidden_link_injection; an off-canvas container – a large negative left/top/text-indent – is High on its own, while display:none, visibility:hidden and fully transparent opacity need corroboration from a second linked domain in the same container or spam vocabulary, because themes legitimately hide panels that link out. Links back to the site’s own registrable domain are ignored, and the scan is skipped when siteurl/home cannot be read, since every absolute link would then look external. Markup is tokenized rather than parsed into a tree, so nesting the injection past a tree parser’s open-element limit does not hide it), autoloaded options named by a 32-character hex digest whose value base64-decodes to one complete PHP-serialized array (db_hostname_keyed_option; kits key the digest to the site’s own hostname so one payload serves many sites, and both halves are required because plugins do write hashed cache keys and base64 alone is ordinary), rewrite rules pairing an exact sitemap<N>.xml route with feed=xmlsitemap<N> in the same rule (db_doorway_sitemap_routes; sitemap plugins add numbered rewrite rules too, so the matching number is what distinguishes a doorway cluster from paginated sitemaps). Bounded option reads that omit any stored bytes mark the scan incomplete, and that row does not produce a finding. The scan also reports published posts attributed to a user that does not exist (db_phantom_post_author, Critical at 100 posts or more and Warning below). Small orphan groups can result from direct SQL user deletion; large groups can identify doorway content whose invented author IDs are hidden from dashboard queries. Direct WordPress installs are covered at every document root the panel serves, at every addon-domain directory in an account home, and one directory below a document root, so a root the panel has stopped serving is still examined, and every finding states which of the two it came from – served roots are reachable now, unserved ones hold a live database that publishes again as soon as a domain is pointed at them, and the statement is omitted entirely when the panel’s domain map could not be read rather than guessing; account backup, cache, staging and metadata directories are not, and installs deeper than one directory below a document root are not discovered recursively. Discovery is shared with the object, admin-overlap, credential-reuse, core-integrity and plugin checks and with their fixers and re-checks, so a finding raised here can always be re-located by the code that resolves it. A site address is reported when its shape cannot be one – a missing host, an invalid port, a backslash, a query string, a fragment, a script for a path, or a non-web scheme – because WordPress builds every asset URL from it. Invalid-address findings are alert-only; only explicit code injection enters database auto-response. Serving a site under a domain hosted elsewhere is ordinary and is not reported on its own – but a root the panel is serving right now, whose address names a domain absent from that account’s panel domains, is reported once per site as db_siteurl_foreign_host (High). Ownership follows the panel’s most-specific exact or wildcard domain mapping, so a delegated subdomain belongs to its own account rather than the account holding its parent. A migrated site is no longer served here, so the served state is what separates the two; an unserved root, an incomplete domain map, or a value the shape check already reports stays silent rather than saying it twice. Multisite-aware: when wp-config.php declares define('MULTISITE', true), up to 100 active secondary blogs (wp_<N>_options / wp_<N>_posts for blog IDs from wp_blogs) are scanned alongside the unprefixed main-site tables; each blog’s PHP-capable WPCode snippets and taxonomy are scanned, and its posts are checked against the network-wide users table. Larger networks emit db_content_scan_incomplete and retain prior findings. |
db_content_joomla | Joomla database content scanning. Discovers installs via configuration.php containing class JConfig, parses credentials from public $...; assignments. Scans <prefix>extensions params, <prefix>content article bodies, and joins <prefix>users with <prefix>user_usergroup_map for Super User detection (group_id=8). Findings: joomla_extensions_injection, joomla_content_injection, joomla_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too. |
db_content_drupal | Drupal 8+ database content scanning. Discovers installs via sites/default/settings.php plus the core/lib/Drupal.php marker. Credentials parsed from the $databases array. Scans config, node_revision__body, and users_field_data joined with user__roles (administrator role). Findings: drupal_settings_injection, drupal_content_injection, drupal_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too. Drupal 7 not yet covered. |
db_content_magento | Magento 1.x and 2.x database content scanning. Discovers installs via app/etc/env.php (M2, preferred) or app/etc/local.xml (M1). Credentials parsed via encoding/xml for M1 (CDATA-aware) or field-level regex for M2. Scans core_config_data, catalog_product_entity_text, cms_block, cms_page, and admin_user (with the configured db.prefix). Findings: magento_settings_injection, magento_content_injection, magento_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too. |
db_content_opencart | OpenCart database content scanning. Discovers installs via the config.php + admin/config.php pair both containing define('DB_DRIVER'. Credentials parsed from DB_HOSTNAME / DB_USERNAME / DB_PASSWORD / DB_DATABASE / DB_PREFIX defines. Scans <prefix>setting (config_url / config_ssl are canonical hijack targets), <prefix>product_description, <prefix>information_description, and <prefix>user (admin/staff). Findings: opencart_settings_injection, opencart_content_injection, opencart_admin_injection. Administrator rows are baselined on the first pass; only an administrator that appears later is reported, once, as High. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too. |
db_content_prestashop | PrestaShop database content scanning. Discovers installs via config/defines.inc.php containing _PS_ROOT_DIR_. Credentials come from the six database_* entries of app/config/parameters.php (1.7.4 and later, 8 and 9), or from the 1.6 config/settings.inc.php defines when neither parameters.php nor parameters.yml exists; no other configuration value is read. A shop on 1.7.0 to 1.7.3, which keeps its live settings in parameters.yml, leaves the scan incomplete. Staging, backup and cache copies and suspended accounts are skipped. Scans <prefix>configuration and <prefix>configuration_lang values, <prefix>cms_lang content, <prefix>product_lang descriptions and short descriptions, and <prefix>category_lang descriptions in every language and shop, with one finding per setting, page, product or category. Setting values are never shown, because the mail password is stored among them. Findings: prestashop_settings_injection, prestashop_content_injection, prestashop_admin_injection. Back-office employees are baselined on the first pass; an employee that appears later is reported once as High, and a new employee with the SuperAdmin profile, or an existing employee moved to it, is reported once as Critical. Every query is capped at 200 rows and runs under the scan deadline; installs under addon-domain document roots are discovered too. |
db_objects | MySQL persistence mechanisms: triggers, events, stored procedures, stored functions. Critical when the body matches known-malware patterns (sys_+exec, INTO OUTFILE, LOAD_FILE, etc.); Warning when an object exists at all (vanilla CMSes ship none). Toggle with detection.db_object_scanning; suppress Warnings via detection.db_object_allowlist. Manual drop via csm db-clean --drop-object. |
admin_overlap | WordPress administrator email overlap across cPanel accounts. Reports when the same admin email appears on the configured number of accounts, with reviewed emails and domains suppressible in detection. |
credential_reuse | WordPress administrator password-hash reuse across cPanel accounts. Groups identical hashes with an in-memory fingerprint and reports only the affected accounts and count. |
supply_chain | Composer and npm lockfile advisory matching against the local advisory database. Silent when no advisory file is present. |
Core verification and plugin inventory report persistent failures separately from malware or integrity findings. A failed attempt first appears in status; repeated failed attempts in distinct scan cycles raise a Warning naming the installation, last attempt and a bounded reason category. A missing wp-cli or an unreachable checksum service stops every installation on the host at once, so when one reason covers more installations than the per-cause limit a host scan collapses the warnings into a single finding naming that reason, the total and a sample of the paths. The limit applies across accounts for each kind of verification and cause. Account scans keep installation-specific warnings. A host summary belongs to an account only when every affected installation shares it; sampled paths do not determine account attribution. Raw wp-cli output is never stored in this history. These warnings do not enter incidents or automatic remediation. The history survives restarts, and successful checks clear the warnings through the normal completed-scan merge. Overlapping scans preserve attempt order and failure streaks even when they finish out of order. New discovery alone does not discard an attempt still finishing in another scan.
Cache reads do not count as attempts. Plugin coverage uses
thresholds.plugin_check_interval_min; partial refreshes keep the existing
refresh policy. disabled_checks controls these scheduled checks, and plugin
coverage performs no refresh when both inventory consumers are disabled.
Cancellation and incomplete discovery retain earlier evidence. Complete discovery
removes installations that are no longer present, scoped to the account scanned.
A domain the panel still lists after its directory was deleted holds no
installation and does not make discovery incomplete when the original mapped
config path is also confirmed absent. Ambiguous symlink resolution, account
boundary violations, and inspection errors keep discovery incomplete and retain
earlier evidence.
A wp-cli refusal explicitly identifying a non-WordPress directory is recorded as
not applicable rather than an outage. A completed core check that finds modified
or missing files retains its existing integrity findings and is counted separately.
Only the completed checksum summary establishes a negative verification result;
partial warnings without that summary remain unverified.
csm status, csm status --json and /api/v1/status report the last observed
coverage for core verification and plugin inventory. These counts describe
previous attempts, not a fresh scan of every site. See
verification status.
WordPress content findings use the account, database server, database name, and table prefix to stay separate. Changes to the panel’s document-root map do not create duplicate findings. Publishing a previously inactive suspicious snippet, or an orphaned-post group growing into a Critical content farm, produces a fresh alert even when the earlier condition was baselined or dismissed. Ordinary count changes within the same orphaned-post severity tier keep the existing identity.
Hidden-link findings are tracked per account, database host, database, and table prefix. More affected rows or a different row order do not raise another alert for the same destinations and concealment strength. Off-screen concealment raises a new High even after an earlier Warning was baselined or dismissed. Changes to destinations outside the displayed sample also produce a new finding. The corrected identity can show existing hidden-link findings once more after an upgrade; previous dismissals do not transfer to the new identity.
Hidden-link corroboration counts registrable domains within one hidden container, not hostnames across a whole row. Multiple subdomains of one linked domain count as one target, and both the WordPress home and site addresses count as local.
Joomla, Drupal, Magento, OpenCart, and PrestaShop configuration reads accept regular files up to 1 MiB. Configuration symlinks and special files are rejected, including during Joomla, OpenCart, and PrestaShop marker probes. Drupal’s version marker must also be a regular file. Reads use the opened file throughout, reject changes observed during the read, and stop when the scan is canceled. Read failures and missing required credentials mark that CMS scan incomplete; manual re-checks keep the finding unresolved when its configuration cannot be inspected.
Joomla, Drupal, Magento, OpenCart, and PrestaShop scans skip suspended accounts
before reading their configuration or applying the account scan file limit: a
suspended account’s database user is locked, so scanning it would only leave the
check incomplete. Both Magento configuration layouts use this filter. Only
installs with a resolved owner and a cPanel suspension marker are excluded;
unowned paths remain eligible. Active accounts still scan when others are
suspended. PrestaShop also skips copies named staging, backup,
backups, cache and .trash, matched without regard to case.
CMS re-checks look at every install of the account in the layouts the scanner
discovers, the account’s public_html and each folder directly under its home,
using each install’s own database login and endpoint, including custom ports and
sockets. Administrator findings and every PrestaShop finding record their
database host, database name and table prefix, and are re-checked only in that
install. Joomla, Drupal, Magento, and OpenCart settings and content findings do
not record it, so they resolve only when the row is clean or absent in every
install of that CMS on the account. An unreadable install, a recorded database
that no install uses, configuration failures and query errors leave the finding
unchecked. Home directory listing and install discovery errors also leave it
unchecked. Links in the home folder that point nowhere or to regular files are
ignored; symlink loops and permission or I/O errors leave the finding unchecked.
Re-checks inspect every relevant database even after finding an injection, so a
later query failure still leaves the finding unchecked. Linked
folders naming the same database are checked once. PrestaShop settings and
content resolve only when every language, shop and scanned column is clean or
absent. New employees resolve only when deleted; SuperAdmin findings also
resolve after demotion.
Administrator baselines for Joomla, Drupal, Magento, OpenCart, and PrestaShop are scoped to the hosting account, CMS, database host, database name, and table prefix. Two sites under one account keep separate baselines when they use different databases or prefixes. Paths that share the same database and prefix share the same administrator set. Upgrading from account-wide baselines starts a fresh baseline for each installation on its first complete administrator query; later additions produce one High finding per new administrator. PrestaShop also records which employees hold the SuperAdmin profile: a new SuperAdmin, or an employee moved to that profile, produces one Critical finding instead, and an employee demoted and later promoted again is reported again. Finding details identify the affected database and prefix.
Database errors, discovery errors, and configuration or query limits keep the affected CMS check incomplete. Earlier findings remain until that CMS completes a scan; another CMS can still complete and clear its own resolved findings. Queries inspect at most 200 rows and request one extra row to detect overflow. Known statement errors and overflow allow independent queries to continue; connection failures and cancellation stop further queries for that installation. Administrator baselines and recorded IDs change only after a complete result; a successful empty administrator result also establishes a baseline. New IDs in a partial result can still be reported against an existing baseline.
PrestaShop scans employees first, so settings or content limits do not prevent employee baselines or SuperAdmin alerts. An employee overflow keeps coverage incomplete and preserves employee state, while settings and content queries still run within their own row limits and the scan deadline.
CMS Scanner Support Policy
New CMS scanner work targets upstream-supported major versions. EOL versions are best-effort when the existing scanner covers them through the same low-risk layout or schema. Adding a new EOL-only scanner needs operator fleet data and an explicit security reason.
Current scanner scope:
- WordPress single-site and multisite.
- Joomla installs using the common
configuration.php/JConfiglayout and standard content/user tables used by supported Joomla releases. - Drupal 8 and newer. Drupal 7 is not a planned support target.
- Magento 1 and 2.
- OpenCart installs using the standard storefront and admin config pair.
- PrestaShop 8 and 9; 1.7 and 1.6 best-effort through the same schema.
The supported kinds are declared once in internal/cms. The database
scanners and the PHP taint analyzer’s knowledge of each CMS’s bootstrap path
constants are tested against that table. The tests also require every typed
CMS kind constant, including local declarations and aliases, to have exactly
one descriptor. Membership drift fails the tests. The clean-corpus manifest
is validated against the same table: every supported CMS is either pinned as
a source or listed as pending with a reason. Support does not by itself mean
a CMS has clean-corpus false-positive evidence (see
the clean corpus). Database object scanning (db_objects)
discovers WordPress installs only.
Phishing & Malware
| Check | Description |
|---|---|
phishing | 8-layer phishing detection (kit directories, credential harvesting). Kit archives are judged from their file list, read through an isolated worker without decompression. Unread or malformed directory records leave the check incomplete. |
email_content | Outbound email body scanning for credentials and suspicious URLs. Only mail submitted on this server, by an authenticated SMTP client or a local process, is scanned; unauthenticated mail arriving from other servers is not. Messages are found in both flat and split Exim spool layouts. Oversized headers are checked from a bounded prefix; long padding lines do not discard readable evidence or stop body checks. |
Outbound scanning covers any authenticated SMTP client, including logins Exim records without a mailbox, and local processes. Quarantine and re-check act only on the message ID the finding names; the envelope sender is shown in the finding details and mailbox field.
If the ZIP worker is unavailable, the check stays incomplete. ZIP parsing never
falls back into the daemon. The existing phishing entry in disabled_checks
controls the check.
An archive that causes two worker failures within 24 hours is skipped on later
scans with a phishing_kit_archive_unscannable warning naming its path and
SHA-256. That skip alone does not leave the check incomplete. The failure record
expires 24 hours after its last failure; it is held in memory and is cleared by
a daemon restart. Changed content is eligible for scanning, and a file rewritten
while the worker runs cannot acquire a failure for its replacement contents.
Malformed listings, canceled scans, and an open worker breaker leave the check
incomplete without adding failures against the archive. Once all failure records
expire, scans no longer hash each candidate archive before requesting its list.
The phishing and group-writable PHP scans include dependency and source-control folders within their depth budgets. They do not recurse through directory symlinks. Phishing content reads also refuse file symlinks and special files without blocking, including when a regular file is replaced during a scan. The PHP permissions check uses regular-file permissions, so a symlink’s write bits do not produce a finding.
Phishing directory analysis honors suppressions.ignore_paths for the pages it
reads, just as page analysis does. Explicit audits can bypass those exclusions.
Directory brand evidence uses the same title rules as page analysis and checks
visible body content, so a brand appearing only inside an email address in the
title or hidden markup does not establish impersonation. Suspected kit directories
are reported for review; auto_response.quarantine_files applies only to eligible
page findings.
Files Planted Alongside Malware
A webshell kit is usually dropped in one session: a loader no rule matches, an encrypted payload, a binary dropper, an access file. Removing only the member a scanner recognised leaves the rest in place. When CSM proves one file malicious, it looks at the other files in that file’s directory and reports the ones planted with it.
Files are matched by change time, not modification time. Kits backdate modification times, but the kernel sets the change time on every write and an account cannot forge it.
- Trigger. A Critical finding from a content scanner, a YARA or signature rule, or a known webshell or backdoor name, from the realtime monitor or a scheduled scan, including the startup scan. Successful quarantine or cleaning through the daemon, including manual dashboard actions and access-file cleanup, also triggers one. Its metadata comes from before the move or rewrite. Refused actions do not trigger sweeps. An account scan sweeps for the detections in its own results and includes the outcome in its report.
- Scope. The detected file’s own directory, not its subdirectories. Symlinks and special files are never followed or read. Content reads remain bound to the opened directory and selected file version. Only files owned by the detected file’s owner count: an account can only create files it owns.
- Window. Files changed within ten minutes either side of the detected file. A file caught as it is written may be the first of its kit to land, so its sweep waits until the window has closed.
- Scan. A first pass reads metadata only. Files in the window, and the
directory’s
.htaccesswhatever its age, then get a full content scan with YARA, the PHP content heuristics, PHP taint analysis and the.htaccessaudit, regardless of scan caches. A match is an ordinary finding and follows the normal response policy. Account scans retain their requested file-size limit. Failed PHP analysis is reported as incomplete, not as a clean result. - Report. Files in the window that nothing matched are listed in one High
malware_companion_filesfinding when they are PHP, executables, archives, files without an extension or.htaccess, with size, change time and modification time. A modification time more than a day older than the change time is marked as possible backdating; unpacking an archive also keeps old modification times, so treat it as a hint. CSM never quarantines these files on its own: review each one first.
Each plant window of a directory and owner is swept once a day at most, so
another member of the same kit, a repeated alert or the quarantine of the
detected file does not sweep it again. Sweeps run one at a time and share the
scan concurrency limit and per-check execution deadline with periodic checks. Matches are
delivered in batches, releasing the scan slot while waiting for delivery.
Overlapping windows do not repeat content findings for the same file version.
Repeated detections retain one file version, and expired records are removed
even when no new work arrives. Pending windows stay claimed until they run.
A huge directory costs one metadata read per entry; only files written in the
window and .htaccess are opened. A timeout produces a check_timeout warning;
directory or content scan failures produce malware_companion_scan_incomplete.
The access-file audit checks sibling context once, in batches, and stops on
cancellation or when it has enough evidence for its existing sibling gate.
A bulk ownership or permission change, or an account restore, resets the change time of every file it touches. A detection in such a directory lists every file of a reported kind from the same operation. Sweeps still waiting for their window are held in memory, and a restart drops them. Standalone CLI cleanup commands do not run this daemon worker; use an account scan to inspect companions when the action runs outside the daemon.
System Integrity
| Check | Description |
|---|---|
rpm_integrity | System binary verification via rpm -V |
open_basedir | open_basedir restriction validation |
php_config_changes | Security-weakening .user.ini and php.ini files under account web roots |
DNS & SSL
| Check | Description |
|---|---|
dns_zones | Security-sensitive DNS zone changes (delegation, mail, apex, and wildcard records) |
ssl_certs | SSL certificate issuance (subdomain takeover) |
waf_status | WAF mode, staleness, bypass detection. On cPanel, staleness follows vendor configuration files that WHM reports as active and ignores retired trees that remain on disk; the warning means at least one loaded vendor has not refreshed in over a month. Other platforms keep the conservative oldest-artifact check so an unused fresh tree cannot hide stale loaded rules. |
Email Security
| Check | Description |
|---|---|
email_weak_password | Email accounts with weak passwords; in-process verification with supported hash formats and cost limits |
email_password_audit_incomplete | Password verification was interrupted or encountered a hash outside the supported audit formats or limits |
email_forwarder_audit | Forwarders redirecting to external addresses, piping mail to a command, or discarding it. Pipes that run cPanel’s autoresponder, BoxTrapper or Mailman list software are not reported. Scheduled and real-time scans interpret command quoting and escaped destination quotes using Exim’s rules. |
email_mail_filters | Exim mail filters and dovecot/Roundcube Sieve scripts that copy mail to an external address while keeping a local copy, forward externally, pipe to a command, or blackhole all mail. Sieve is what webmail-managed rules actually execute, so both are scanned. A forward that leaves the mailbox its own copy is what a webmail forward rule produces, so on its own it reports as a Warning for review; it is Critical when an independent forwarding layer on the same mailbox, mail the mailbox never receives, or the same destination across accounts corroborates it. The full audit runs on the email audit interval. The next deep scan picks up edits to a Sieve script, including the target of an active script link. Each such forward is also compromise evidence for its mailbox. |
Performance
| Check | Description |
|---|---|
perf_php_handler | PHP handler configuration (DSO vs CGI vs FPM) |
perf_mysql_config | MySQL my.cnf optimization |
perf_redis_config | Redis configuration |
perf_error_logs | Error log file growth (bloat) |
perf_wp_config | WordPress wp-config.php settings |
perf_wp_transients | WordPress database transient bloat |
perf_wp_cron | WordPress cron scheduling (missed crons) |
perf_wp_loopback | WordPress self-requests sustained above once a minute |
perf_wp_myisam | WordPress tables still on the MyISAM storage engine |
Platform Support
The deep checks are the most cPanel-biased part of CSM because they iterate account home directories and per-user public_html trees. On plain Ubuntu/AlmaLinux the account-scan based checks do not run today:
cPanel-only (skipped on plain Linux):
htaccess,file_index,php_content,group_writable_php,symlink_attacks– iterate/home/*/public_html/**wp_core,outdated_plugins,vulnerable_plugins,db_content,db_objects,admin_overlap,credential_reuse– find WordPress installs through the shared discovery: the panel document-root map,/home/*/public_html, one directory below it, and addon-domain directories in an account home. Unresolved document-root aliases retain prior findings and cached plugin inventory instead of treating a partial walk as a clean result.supply_chain– scanscomposer.lockandpackage-lock.jsonunder/home/*and/home/*/public_htmlphishing,email_content– scan user home directories and Exim spooldns_zones,ssl_certs– read cPanel’s DNS zone store and SSL installation recordsemail_weak_password,email_forwarder_audit– read/etc/valiases, Dovecot/Courier auth databasesemail_mail_filters– read per-mailbox Exim filters under/home/*/etc/<domain>/<localpart>/filterand domain filters under/etc/vfiltersopen_basedir– reads EA-PHPphp.iniunder/opt/cpanel/ea-php*/php_config_changes– recursively scans.user.iniandphp.inibelow account web roots; incomplete walks emit a coverage finding and preserve prior findingsperf_wp_config,perf_wp_transients,perf_wp_cron,perf_php_handler– WordPress and PHP handler introspection via cPanel’s EA-PHP layout; the WP-Cron check also uses cPanel’s domain map for addon and subdomain roots
Runs on every platform:
filesystem,webshells– fanotify and file-tree scans over/home,/tmp,/dev/shmrpm_integrity– dispatches torpm -Von RHEL family ordebsums/dpkg --verifyon Debian familywaf_status– detects ModSecurity on Apache, Nginx, and LiteSpeed across all supported distrosperf_mysql_config,perf_redis_config,perf_error_logs– rely on standard service locationsperf_wp_myisam– uses shared WordPress discovery in the deep tier, including account home layouts when no panel map is available
Operators on plain Linux can point perf_error_logs, perf_wp_config, perf_wp_transients, and perf_wp_cron at generic web roots with the account_roots glob list (see configuration.md). The remaining account and CMS scans still assume the cPanel /home/*/public_html layout.
Package integrity rechecks retain a modification that the package verifier still reports for the flagged file. Changing its executable mode or removing the file does not by itself resolve that finding.
Hidden temporary files
The filesystem check reports hidden regular files in the shared temporary directories when they have executable permissions, ELF magic, or a leading shebang, long PHP opening tag, or short echo tag. Aliases of the same inode produce one finding; distinct files remain separate even when their names match. Failed content reads leave the filesystem scan incomplete and preserve earlier alerts.
An inert staged blob without these signals is outside this heuristic. There is no guaranteed alternate detection before it is made executable or interpreted; signature and runtime detection depend on recognizable content and activity. Changing its permissions to executable makes it eligible for the next filesystem scan. This check does not establish that every unreported temporary file is safe.
Auto-Response
When enabled, CSM automatically responds to detected threats. All actions are logged in the audit trail.
Actions
| Action | Description |
|---|---|
| Kill processes | Fake kernel threads, reverse shells, GSocket, loops that write back a file CSM removed, and shell loops a web request left running in the background. Never kills root or system processes. |
| Quarantine files | Moves webshells, backdoors, phishing to /opt/csm/quarantine/ with full metadata (owner, permissions, mtime). Restoreable from the web UI. |
| Block IPs | Adds attacker IPs to the nftables firewall with configurable expiry. Admission uses auto_response.max_blocks_per_hour (default 2000/hour); legacy scan blocking defaults to 50/hour. |
| Clean supported malware | Applies bounded PHP and .htaccess cleaners with pre-clean backups. Database cleanup has a separate opt-in. |
| Drop malicious DB objects | When clean_database is on, confirmed-malicious stored triggers/events/procedures/functions are dropped after a SHOW CREATE backup is recorded, so the drop is reversible. Detection runs regardless; the drop is gated on the operator opt-in. |
| PHP shield | Blocks PHP execution from uploads/tmp directories and inspects directly executed wp-content scripts for request-fed command sinks and packed eval loaders. |
| PAM blocking | Instant IP block when one address breaches thresholds.pam_bruteforce_threshold SSH failures inside pam_bruteforce_window_min minutes, or fails SSH logins against cred_stuffing_distinct_accounts distinct accounts. Only successful SSH logins clear those failures; failures from other PAM services never block. |
| Subnet blocking | Auto-blocks IPv4 /24 or IPv6 /64 when 3+ IPs from the same range were blocked within netblock_window (7 days by default). Currently blocked addresses count regardless of age, including operator and permanent blocks. Ended blocks count while their latest observed block start is inside the window, unless an earlier subnet block already answered them. A returning operator block starts fresh history after its absence is observed. Whitelist and clear actions, including bulk whitelist, forget the address. |
| Permblock escalation | Promotes temporary blocks to permanent after N repeated offenses. |
| Suspend mailbox | After confirmed credential abuse, attempts to suspend that mailbox’s logins and outgoing mail through cPanel. Either success stops escalation; the account-wide outgoing mail hold is the fallback when both fail, no account owns the domain, or the target is a bare domain. A Critical mail_account_compromised (a login after brute force) also suspends the mailbox. When that login is the cPanel account’s own username, the account’s outgoing mail is held instead, because that password unlocks every mailbox. A stolen login never falls back to the account hold: a failed suspension raises a Critical email_mailbox_response finding, as does each suspension or hold it applies. Signs of compromise that add up for one mailbox suspend it with a single Critical email_mailbox_compromise finding; see Mailbox Compromise Evidence. All of these follow auto_response.enabled and dry-run, and each attempted suspension or hold is recorded in the action log with its quoted release command. |
| Auto-freeze (PHP relay) | On cPanel, freezes active Exim messages attributed to a high-confidence PHP-relay finding. It has its own dry-run control and action-rate limit. See PHP-relay CLI. |
Of the processes a web request left running, only shells are stopped; anything else, including a non-shell parent of a shell, stays a Warning for review.
Mailbox suspension runs /usr/local/cpanel/bin/uapi as a transient systemd service outside the daemon’s filesystem sandbox; hosts without a usable systemd-run run it directly. Each call is bounded in time, the JSON result decides success, and mailbox names stay literal arguments on both paths.
Partial suspensions name the restriction that succeeded; they do not claim both logins and sending stopped. Mailbox restrictions and account holds already applied outside CSM are not offered for release. If cPanel cannot report the mailbox state, CSM suspends anyway and records the suspension as its own; sending evidence can still fall back to an account hold.
After resolving the credential abuse, release the mailbox or account on the Email page, Suspended tab. The tab lists every suspension and hold CSM made with its live state in cPanel; one lifted directly in cPanel shows as released outside CSM. Incomplete or unreadable status stays unknown, and a delayed status response cannot release a newer suspension.
Release undoes the recorded restrictions and never changes a password. Evidence captured before the release cannot suspend the mailbox again. Cleanup failures are reported so the release can be retried.
Releasing an account hold delivers the mail cPanel held, so delete spam from the Queue tab first; the confirm says so. The action log also keeps the release commands, quoted for the shell, for example /usr/local/cpanel/bin/uapi '--user=account' Email unsuspend_login 'email=alice%40example.com' and the matching unsuspend_outgoing command. The address is URI-encoded because uapi decodes its values.
WHM holds and releases check the JSON API result before reporting success. Dry-run writes one planned-action record without running a command; disabled auto-response writes none.
Subnet history is pruned hourly and saved only when it changes. If the file cannot be read it is left untouched and escalation counts only the addresses blocked right now until it is repaired or removed; status and doctor report the failure. Clear, whitelist and flush actions report history cleanup failures so operators can retry them.
Mail login spray blocks of IPv4 addresses do not contribute to /24 escalation. IPv6 spray blocks count toward their /64 like any other block, since one /64 is treated as one source. The classification comes from committed firewall state, so a missing or stale scan tracker does not turn IPv4 spray blocks into range evidence. Saved spray blocks absent from the kernel do not count as active blocks. If block metadata cannot be read, subnet escalation waits while individual block attempts continue.
Blocks of an address on c2_blocklist never count toward subnet escalation
or permanent promotion, whatever finding asked for them: a listed
command-and-control server is where local processes connect, not a source
of attacks. Other blocked addresses in its subnet still count.
Process termination
An eligible finding receives one automatic termination evaluation across scan and alert delivery. A refused or failed attempt is not retried when the same finding reaches alert delivery. Realtime findings still receive an evaluation; new detections and findings replayed from storage receive a fresh evaluation.
CSM opens a kernel process handle before verifying ownership, executable, start time, or the file referenced by a process. It checks that the captured process is still alive after verification and sends the signal through that handle. An exited process cannot redirect the signal to a replacement with the same PID. Automatic and manual malware termination reject root credentials, including effective and saved root IDs. The separate opt-in AF_ALG reaction requires the current credentials and executable to match its recorded event.
Safe signaling needs pidfd_send_signal (Linux 5.1). Where pidfd_open
(Linux 5.3) is absent – EL8 and CloudLinux 8 ship 4.18 kernels without it –
CSM pins the target through its /proc/<pid> directory descriptor, which
pidfd_send_signal accepts and which refers to the same kernel process. Both
paths reject a recycled PID; CSM never falls back to numeric PID signaling.
On a kernel without pidfd_send_signal, or when service restrictions deny the
call, termination stays disabled: csm doctor reports process termination supported as failed and the health status becomes degraded whenever
auto_response.kill_processes is enabled, so an inoperative protection is
visible before an incident needs it. Each health snapshot probes this capability
again, so transient resource failures clear once signaling becomes available.
Automatic termination failures are logged while the original detection remains.
Manual kill-and-quarantine reports a termination failure even if the file was
successfully quarantined; inspect both the process and recovery entry before
retrying. Manual request cancellation is checked before sending a signal.
Automatic file response limits
Realtime quarantine, scheduled quarantine, PHP cleaning and automatic .htaccess
cleaning share one rolling-hour budget. auto_response.max_file_actions_per_hour
caps host-wide attempts (default 50), max_file_actions_per_account_per_hour caps
attempts for one account (default 10), and max_file_action_failures_per_hour
pauses these responses after repeated failures (default 3). Zero or an omitted
key uses the default; negative values and values above 10000 are rejected.
Each attempt is reserved before touching the file. Successful, failed and
interrupted attempts all consume capacity. Reservations live in
<state_path>/file-response.json and survive configuration reloads and daemon
restarts. Entries expire one hour after admission. A backwards clock adjustment
keeps future-dated reservations charged until their window has passed.
Account budgets come from the target’s account-home path, not finding text. Paths outside recognized account homes share an unknown-account budget. One account reaching its limit does not stop other accounts unless the host or failure limit is also reached. Automatic directory and special-file quarantine is refused because one directory move can affect an unbounded number of files. Manual remediation remains available after reviewing the original detection.
A brief shared lock protects each safety reservation and outcome update. The
file action runs after reservation without holding that lock, so a slow copy
does not block another available slot. Pending actions hold failure capacity
until their outcome is saved, so concurrent failures cannot pass the configured
ceiling. If the state lock remains busy, the next attempt is refused. Unreadable,
incomplete or unwritable safety state also refuses mutations. The original
detections remain visible, and a deduplicated auto_response_paused warning
reports the cause. Account-limit notices are
grouped at host scope so a fault across many accounts cannot flood the alert
budget. A pause does not create a retry job; new eligible detections can act
after capacity returns. Review outstanding findings and recovery evidence before
manual remediation. Do not delete safety state to clear a pause; repair storage
faults and let reservations expire.
Duplicate detections of one stable file version share a single realtime response
attempt. A different file planted at the same path remains eligible.
Within one daemon run, alert delivery does not repeat a file response already
evaluated by a scan or admitted to the realtime safety gate, including budget
refusals. A realtime detection rejected using sampled content remains eligible
for full-file validation during delivery. The original findings still reach
alerts and history; a new detection can be evaluated again.
Findings still queued at shutdown retain the existing restart replay behavior:
the next daemon run evaluates them again under the same persisted limits.
Parked findings also keep the offending subnets and targeted accounts needed for
subnet response and credential spray correlation. Older parked findings replay
without that data. The subnets also appear in the public cidrs field.
Critical realtime PHP findings that prove request data reaches an execution primitive, or that executable PHP unwraps and runs encoded content, enter this file-response gate immediately. Quarantine holds an exclusive Linux lease on the open file that produced the finding until that file is removed. It refuses while another process has the file open or is running it, and when the content or path identity changes. Whole-file malware identified by the cleaner uses this lease too. Other quarantines do not take this lease, so they can still move a file that is open or running. Comment-only matches, URL-only droppers, filename-only checks, tail-window matches and YARA-only findings remain available for review without an automatic move.
File quarantine copies only the file size it verified and checks the file again before removing it. A file that changed is restored and its copy discarded. If its name was recreated meanwhile, the error names the private directory that holds the changed file.
Parked reputation findings keep their intelligence source and expiry. If an expiry cannot be encoded, only that intelligence metadata is dropped; the finding and the rest of the queued batch are still saved and replayed. This handling is automatic and has no setting.
A failed PHP cleaner leaves the file and any pre-clean backup for manual review. It no longer escalates to whole-file quarantine. A cleaner that recognizes no injection or declines an unsupported target refuses the file instead of failing. Sources that change or disappear before mutation are also refusals, including socket replacements and parent paths replaced after quarantine copying. These attempts still use capacity but do not count toward the failure pause. Read, write, backup and durability errors still count as failures. Quarantine and cleaners retain their descriptor-based identity checks, and automatic actions revalidate the file after saving the reservation. Opening a replacement special file cannot block response processing.
Manual full scans leave cleaner refusals for review instead of reporting a
failed remediation. csm clean also distinguishes a refusal from an action
failure; both return a nonzero exit status when the file was not cleaned.
WordPress core, plugin and theme files are cleaned in place, with three
exceptions. When cleaning would leave nothing of a file that sends no page
output, the file is quarantined whole: a stub would lose the same content,
while quarantine keeps it for a restore. When cleaning would erase a file’s
page output, the file stays untouched for review. When the release of WordPress
core or of a wordpress.org plugin that a site runs does not ship the file, it
was planted and is quarantined. CSM trusts the release a site names only when
its version.php or <slug>.php plugin main file is that release’s own copy,
and only from the checksums the realtime file monitor has cached. A plugin
whose main file has another name stays with the cleaner. Page output is any
text outside PHP code, including short and echo tags. Whole-file quarantine
rechecks the original content under the same protection used for other
verified file responses; a changed file stays in place. csm clean refuses a
file it would leave empty.
Automatic permission fixes dismiss the original finding after a successful fix or after confirming its unsafe permission is already gone. A confirmation alone does not report a permission change.
These limits use the existing enabled, quarantine_files and clean_htaccess
opt-ins. Observe mode still forbids automatic changes. dry_run continues to
control IP blocking and web-exposed-file virtual patches; it does not preview
file quarantine or cleaning. Other response families have their own controls.
Files written back after removal
When CSM quarantines or cleans a file, it clears the alert state it kept for that file. If a detection finds the file there again, the finding is reported as new instead of being held back as a repeat. Dismissals remain in effect.
CSM also remembers, for seven days, the content each quarantine or clean took
away. A process that writes the same bytes back to the same path raises
removed_file_restored_realtime (Critical). With quarantine_files on, CSM
then does what it did before: it quarantines the file again, or cleans it
again. The daemon keeps this memory across restarts. A file cleaned with
csm clean from the command line is not remembered. A restore from the web
UI quarantine page, CSM’s own writes and official WordPress release files are
never treated as the file coming back.
The saved history is read only from a private file the daemon owns. A damaged or unsafe one is logged and ignored, and detection carries on without it.
Before repeating the response, CSM checks that the file still holds the
removed bytes, so a later replacement or a restore from the web UI is never
removed. Removed files are watched whatever their extension. A restored
.htaccess is cleaned only when clean_htaccess is on.
With kill_processes on, CSM also stops the process that wrote the file back,
together with its parent processes of the same account, parents first, so a
loop cannot start another writer. It does so only when those processes trace
back to the web server or run detached under init, the shape of a loop a
webshell leaves behind. A PHP worker that serves the site is not stopped unless
it wrote the file itself. A process with a terminal, or one started from an
SSH, cPanel or cron session, is left running and the file is still handled;
root processes are never stopped. An SSH, cron or cPanel session anywhere
above a PHP worker also prevents the kill, and so does any part of the
ancestry CSM cannot read. The processes are named when the write is seen, and
each is checked again through a kernel process handle just before the signal,
as described above.
Restoring quarantined files
Regular-file quarantine and pre-clean backups write and sync the private content copy, metadata, and directory entries before removing or changing the original. Directory quarantine syncs the tree and metadata before moving it on the same filesystem. A directory move across filesystems is refused and leaves the source in place. Storage failures do not count as successful remediation.
New quarantine and pre-clean sidecars record the original modification time, owner, group, permissions, and size. Restore reapplies the saved attributes to the opened destination before syncing it. Linux preserves modification times at the precision supported by the destination filesystem. An ownership or timestamp failure keeps the recovery evidence and reports an error.
On Linux, restore can verify the destination after saved permissions remove all read access. The restored file keeps those permissions, including a mode with no access granted to anyone.
Older sidecars may use quarantine_at; listing and restore also read that
historical spelling. Entries without a saved quarantine date sort last. Older
entries have no recorded original modification time, so restore leaves the new
file’s modification time in place instead of treating the archive’s timestamp
as the original. Some older access-file cleanup backups also lack trustworthy
ownership and permission data; check those attributes when restoring them.
Configured account roots participate in manual remediation and restore. Set up service write access for roots outside the packaged grants before enabling these operations.
Web UI restore refuses symbolic links in destination directories and does not replace an existing file or directory. If a destination changes during restore, CSM reports a conflict and retains the quarantine entry. Check the original location before retrying; a failed or interrupted file restore can leave a partial file in the directory that was opened for restoration. CSM keeps this file because removing it could discard a concurrent replacement.
Restore syncs the replacement and its containing directory before deleting quarantine evidence. A failure after a move or unlink reports partial completion and retains recovery metadata where possible. Inspect both the original and quarantine locations named in the error before retrying; a copy may already be restored while quarantine cleanup remains incomplete. These guarantees depend on the filesystem and storage device honoring sync requests.
Virtual-patch rollback uses the same directory confinement. It replaces or removes the access file only when the saved content, owner, and permissions still match, preserving later customer edits.
Virtual-patch backups are reused only when their content and saved attributes, including modification time, match. A later edit with identical bytes but a new modification time gets a separate recovery point.
Rollback isolates displaced entries in a private directory. If another writer changes the destination, CSM preserves the live replacement and reports any retained recovery directory in the error. Check that directory beneath the original destination or quarantine location before retrying. If another writer moves a restored directory, CSM cannot recover it from its old name and will not substitute a different entry into quarantine.
Configuration
auto_response:
enabled: true
kill_processes: true
quarantine_files: true
max_file_actions_per_hour: 50
max_file_actions_per_account_per_hour: 10
max_file_action_failures_per_hour: 3
block_ips: true
block_expiry: "24h" # positive temp block duration; omitted defaults to 24h
# max_blocks_per_hour omitted: 50 scan blocks an hour, admission ceiling 2000
netblock: true # enable subnet blocking
netblock_threshold: 3 # IPs from same IPv4 /24 or IPv6 /64 before subnet block; minimum 2
netblock_window: "168h" # blocked IPs count this far back, expired blocks included; omitted defaults to 168h
permblock: true # promote temp blocks to permanent
permblock_count: 4 # temp blocks before promotion; minimum 2
permblock_interval: "24h" # positive counting window; omitted defaults to 24h
# Response to http_scanner_profile findings: "challenge" (default)
# routes the IP to the PoW challenge when challenge.enabled is true,
# falling through to a firewall block when it is not; "block" always
# hard-blocks without offering a challenge.
http_scanner_action: "challenge"
# Deny HTTP access to confirmed exposed files with reversible .htaccess
# rules. off: alert only; manual: `csm virtual-patch --apply`; auto:
# apply confirmed findings except warning-only sample SQL when dry_run is false.
virtual_patch_exposed_files: "off"
# SAFETY DEFAULT: dry_run defaults to TRUE when this key is absent.
# Automatic IP blocks do not touch nftables. Admission keeps bounded
# examples of successful capacity predictions, not every attempted block.
# Manual operator commands (`csm firewall ...`) bypass dry-run and apply.
# Flip to false only after verifying the policy in dry-run.
dry_run: true
# Advisory verdict callback. CSM POSTs each impending auto-block
# to the panel before applying. The panel can downgrade to "allow"
# (audit-only), attach `tenant_id` for downstream correlation, or
# add a reason. CSM fails open on hook errors. Wire contract:
# verdict-callback-contract.md in this book.
verdict_callback:
enabled: false
url: "" # POST target
hmac_secret: "" # signing secret, or use hmac_secret_env
hmac_secret_env: ""
allow_unsigned: false # true only for staged unsigned rollouts
require_response_signature: true # reject unsigned callback replies
timeout_sec: 2
# PHP-relay auto-freeze (cPanel only). Off by default; opt in
# explicitly. dry_run defaults to true even when freeze=true so an
# operator who enables freeze without thinking gets a dry-run.
php_relay:
freeze: true # enable the exim -Mf hook
dry_run: true # safe default; flip with `csm phprelay dry-run off`
max_actions_per_minute: 60 # rolling 60s window cap on exim -Mf invocations
Exposed-file virtual patches
Each applied deny keeps a rollback copy under /opt/csm/quarantine/pre_clean/. Re-applying the same rollback state reuses that copy; content, ownership, permissions, or remove-versus-replace changes keep separate restore points.
All-in-One WP Migration rewrites the access file inside wp-content/ai1wm-backups, so CSM also denies only .wpress filenames from the parent wp-content access file. It does not add parent-wide .zip or .gz rules because those extensions can be legitimate downloads elsewhere under wp-content.
Dry-run safety default
auto_response.dry_run defaults to true when the key is absent. This is deliberate: an operator who turns on block_ips: true without reviewing policy keeps automatic IP blocks unapplied while evaluating policy. The retained-example count surfaces in csm status --json and /api/v1/status; forecast status reports capacity decisions separately. On entry into live mode CSM requests a purge; owner-managed purges run in bounded passes and report stale examples until complete. While dry-run remains enabled, records older than a week age out.
With the firewall on and outside observe mode, admission also forecasts each automatic IP block in dry run: whether it would fit the hourly ceiling and the temporary block set, behind the blocks and forecasts before it. Status, csm doctor and findings history show the forecast, with at most one summary a minute. Admission owns new examples, pruning and purge. On a live start, existing examples also wait for its bounded purge, including while admission startup retries. New examples describe committed capacity predictions; the count also includes retained older examples until pruning or purge. It is not unique IPs or active predicted occupancy. Example loss and incomplete comparisons appear in status. A forecast is not a block; restart or re-entry into dry run starts a new session without restoring hypothetical leases from examples. The forecast fills the temporary block set as automatic admission will: of the slots operator, unknown and subnet blocks leave, a fifth is kept for direct or corroborated attacks, and an unlimited set is modelled as 10000 entries. Forecasts and previews cap each new automatic lifetime, including challenges, at auto_response.block_expiry: longer requests are cut and shorter ones stay short. Retries keep their first absolute expiry, even after a lower cap is configured; existing blocking keeps its own lifetimes. In a full set a forecast block may take the place of a lower-priority one, as live admission would: an earlier forecast block of a lower class or severity, or a legacy automatic block that recorded its origin. A new attack may also rotate out an earlier forecast block of the same class and severity once that block has held its slot for 30 minutes, within the share of the set its scope may use; an address whose forecast block was replaced or rotated out cannot rotate again before that block’s expiry. A forecast block for an address that returns after the forecast replaced its earlier block lasts only until that earlier block would have ended. A return with too little time left for a firewall block is refused without a forecast charge or retained example.
Admission checks how each finding’s address was read against its producer’s declared address sources and the check’s current policy, when it mints evidence and whenever retained evidence is used. A producer without a trusted address source cannot mint evidence. This excludes PHP relay findings, whose client address comes from a mail header a local account can forge. With auto_response.dry_run: true, these findings cannot produce a successful admission forecast. Stored evidence keeps its existing format and identifiers.
Scheduled cPanel, ModSecurity audit, per-site and central access log scans can supply automatic response forecasts from complete records with a verified address source and server time. The scheduled command-and-control connection scan can too, for established IPv4 connections this server opened. It names each one by its socket during the current boot, keeps its first-seen time across restarts when the state save completed, and reports both endpoints and the socket owner. Listener changes and overlapping scans do not refresh a matched socket’s first-seen time. Socket inodes can be reused between scans; those reports retain the earlier time, and lost first-seen state can make a forecast conflict with retained evidence. A connection on a local port that also listens still alerts but supplies no forecast, because the socket table does not show which side connected. Connections still waiting for a reply and those in the IPv6 socket table are not reported. Without a socket identity the alert still shows but supplies no forecast. Blocking the address drops its replies but does not stop data this server sends to it. Rereading a record does not refresh its time or add independent evidence. Unfinished or ambiguous records cannot supply a new observation. HTTP findings whose counts or bot-majority decisions use forwarded addresses, including mixed direct and forwarded traffic, name a client behind a trusted proxy and are refused as unreachable. The client is the first untrusted address reached from the right of the forwarded header. Periodic scans retain a bounded suffix of complete address tokens; padding to the left of a retained client does not suppress its count. A missing or unreadable right end, or a retained suffix containing only trusted proxies, names no client and is not counted. The resolver refuses a header over 4 KiB or a walk that must read more than 32 hops, its client included; hops left of the client never count. An address in web_server.trusted_proxies or challenge.trusted_proxies is never read as the client, including when its header is not trusted because it belongs to this host or the host’s addresses cannot be read; an address of this host is never trusted as a proxy. An HTTP, cPanel or IP reputation finding whose address is a known CDN or website-firewall edge, from the Cloudflare, QUIC.cloud and Sucuri ranges CSM ships (a replacement http_proxy_ranges.yaml does not change them) or, when the ASN database names it, a network in thresholds.http_asn_crawl_reverse_proxy_asns, is refused as proxy_peer, whether or not cloudflare.enabled is set. This edge refusal does not apply to SSH, FTP or mail session peers. When Apache (RemoteIPHeader, RemoteIPProxyProtocol On, a loaded or configured mod_cloudflare including DenyAllButCloudFlare, or mod_rpaf with RPAF_Enable On or RPAFenable On), nginx (set_real_ip_from) or LiteSpeed (Use Client IP in Header, in its own configuration or, on cPanel, the Apache configuration it loads) replaces the logged client address, or CSM cannot read those files completely, new HTTP findings are refused as rewritten_log and access-log lines are not IP reputation sightings; cPanel access lines still are, even when the same address also appears in a rewritten web log. An Apache, nginx or LiteSpeed access log format that writes request data into the first client token counts the same way. Leading whitespace and extra spaces after cPanel’s splitlogs virtual-host field cannot hide that token. A leading quote, brace or other non-address character guards normally escaped output. Brackets cannot guard a request value because scheduled scans accept bracketed IPv6 peers, including peers with ports. Apache format nicknames ignore case, including the TransferLog default. A definition in another virtual host or an unrelated conditional section cannot hide a log’s inline or common-format fallback. With HostnameLookups On in any scope, %h counts as a request value, because it then logs the client’s unverified reverse DNS name. Inside a mod_macro <Macro>, a parameter in a log format, format name, piped log target, module name or directive name leaves the state unknown, including a parameter that spells ServerName. Quoted macro names and parameters retain their boundaries. A parameter later in a log file path or in a server name is harmless. A parameter or ${...} variable that starts a log target leaves the state unknown, except Debian’s ${APACHE_LOG_DIR}/. That exception does not apply when Define or UnDefine can change the variable, including inside a macro. The built-in CLF nickname is also checked after piped logging removes its leading field. An ErrorLogFormat request value before an unconditional message in any output record also counts. A message that shares its field with another placeholder cannot guard later request values because that field may disappear. Unresolved Apache configuration variables in log formats, format names and server names are unknown, and so are those that could make or hide a piped log target. Unsupported format handlers, blank Apache server names and custom time formats that can write a line break are unknown, and so are unproven time formats or process, port and duration arguments in or before the field CSM reads. Empty logging conditions, including whitespace-only expressions, are unknown; a nonempty expr= condition does not change how its format is read. LiteSpeed formats with unsupported escapes, modifiers, handlers or ambiguous brace arguments are unknown, in both native XML and cPanel Apache configuration. A valid format no CustomLog, GlobalLog, TransferLog or nginx access_log uses does not count. nginx checks HTTP and stream format namespaces separately. A log selects the first previously declared format with the same name, ignoring ASCII case. Later case variants do not count as used formats. Unresolved names and malformed log declarations leave the state unknown. Entries inside maps are values, including entries read from included files. With escape=none, any request variable can introduce another log record, even after the normal client field or a quote. Binary address variables have unknown output framing. Empty server-name, hostname and port fields cannot hide a following request value. User-defined variables do not inherit trust from a built-in-looking name. nginx original-peer variable names and stream server names remain unproven because their availability depends on the installed modules and version; captures or scripts can supply request values under those names. This conservative check also applies when those variables come from nginx itself. Stream byte counts, session duration and transport protocol remain server values in the stream namespace. This build does not read a separate original-peer log field. CSM reads the files when admission starts, at the start of every reload attempt and web UI settings save, and every 10 minutes. Settings saves refresh this state before applying changes, including saves that fail or require a restart. web_server.config_dir selects the Apache or nginx root, including the Apache tree used by cPanel LiteSpeed; LiteSpeed XML remains at its standard location. The main file is apache2.conf on Debian Apache, conf/httpd.conf on RHEL or cPanel, and nginx.conf on nginx; a cPanel root ending in conf already names that directory. An absent selected Apache main file is unknown, even if a default tree exists elsewhere. A disabled or merely loaded mod_rpaf does not count as a rewrite. Malformed legacy-module arguments leave the state unknown. Cloudflare trusted-proxy hostnames or variables also leave it unknown when CSM cannot establish another active rewrite, because only Apache can resolve those values. Apache section suffixes are ignored only where Apache accepts them. Malformed closing tags and comments on closing tags for read-time conditions such as IfModule leave the state unknown; a rewrite directive found beside a parsing gap still counts as active. Files on disk can differ from what the web server loaded until it reloads, a setting in any virtual host applies to the whole host, and responses already queued are not checked again. SMTP password sprays read from Postfix mail logs can now supply forecast evidence. ModSecurity observations expire with the denial’s recent-log window. The startup Exim history scan names the newest counted send of the account it reports, using Exim’s full timestamp and zone, as the realtime mail log reader names that line. An invalid zone supplies no observation. At its event cap, the realtime cloud relay window keeps the current send’s address. An IP reputation finding supplies a forecast only with a local sighting: a record its log’s parser read the address from, dated by the server within the last 24 hours. An HTTP or cPanel sighting never uses an address listed in web_server.trusted_proxies or challenge.trusted_proxies. Admission still applies its normal evidence freshness check. The existing auto_response.dry_run setting decides whether admission forecasts these blocks or applies them.
A reader that starts inside an unfinished line still delivers its remainder to existing checks, but that fragment cannot supply admission evidence. Complete records that follow it keep their shared references.
Admission also binds each response target to its evidence. Only the mail subnet and ASN crawl entries may widen an attributed address to a containing prefix. Direct queue requests, retained candidates and reservations repeat this check, so previously queued responses cannot bypass it.
This setting is not a universal simulation mode. It gates automatic firewall and related network enforcement paths plus web-exposed-file virtual patches. File quarantine, cleanup, permission changes, and process termination are controlled by their individual auto_response flags. Leave those flags off while evaluating block policy.
IP auto-blocking still requires firewall.enabled: true. The firewall engine owns live nftables mutations and the dry-run gate; with the firewall disabled there is no engine to call, so csm validate warns on auto_response.enabled: true plus block_ips: true.
Verify dry-run state explicitly:
csm status --json | jq '.severities, .blocklist_size'
csm firewall status # "Recently Blocked" entries with timestamps after the restart confirm live mode
To go live: set dry_run: false, then run systemctl reload csm. The field is hot-reload-safe, and a successful reload validates the config and re-records its hash. For a planned restart instead, run csm rehash once before restarting.
Verdict callback (advisory)
When verdict_callback.enabled: true, every auto-block call POSTs a
signed JSON request to the panel before mutating nftables. CSM refuses
to start without hmac_secret or a non-empty hmac_secret_env value
unless allow_unsigned: true is set for a staged unsigned rollout.
Without that opt-in, an unsigned allow response is rejected and the
default block continues.
When a secret is configured, CSM also requires the panel to sign the
response body unless require_response_signature: false is set for a
staged rollout. With that opt-out, CSM still checks any echoed nonce
or timestamp when a secret is configured; a legacy response that
omits both keeps working. The panel can return {"verdict": "block"}
(apply), {"verdict": "allow"} (audit-only; CSM logs the decision and
skips nftables), or attach metadata (tenant_id, note). The callback
runs after local validation and infra-IP safety checks, and before the
dry-run gate, so panels can observe dry-run decisions too. In dry run
with the firewall on outside observe mode, admission also asks for each
locally eligible block attempt, with CSM auto-block: <check> as the reason, up to four
requests at a time, and uses the configured
allow_unsigned and require_response_signature settings. A full set still
allows a request when a lower-priority automatic block could be replaced or
an equal-priority forecast block rotated out.
Pending requests take no slots or replacement claims; admission checks
capacity again after each answer, so competing requests cannot reuse one
victim. An allow ends that forecast without a block. Until admission decides blocks, a panel can receive an additional request
for the same automatic block. Local refusals, existing effects, capacity
waits, coalescing and retries mean request counts need not be two per block.
The v1 request has no dry-run field. Forecast examples and summaries are
bounded, unsupported promotions are not simulated, and reload discards old
policy answers.
CSM fails open on hook errors (timeout, non-2xx, malformed body): local admission continues, still subject to dry-run and capacity checks. An error proves neither a block nor a forecast example. The failure is written to the daemon log. Full request/response schema: Verdict callback contract.
Infrastructure IP DNS guard
Hostnames listed in top-level infra_ips or firewall.infra_ips are resolved every 5 minutes and their current addresses feed the infra auto-block guard. If a hostname stops resolving, the daemon emits an infra_ips_unresolvable Warning finding and keeps the last known addresses protected during the grace period (default 10 min). The finding auto-clears when resolution recovers.
Findings that always trigger IP block
When auto_response.block_ips: true and the firewall is enabled, qualifying findings in this list block the source IP. Per-row severity and challenge exceptions apply. The dry-run gate still applies if dry_run: true. Suppression rules do not stop these blocks; allowlist an address to exempt it.
Suppression rules also leave incident auto-blocking, credential-spray containment and central threat responses active. With database response enabled, suppression stops database cleanup and session revocation; the notice listing the session addresses is still sent. Suppress the action’s own check type to mute its notification; this does not disable enforcement.
Incidents and central threat responses receive new findings, including suppressed findings and checks that do not notify operators. Duplicate observations within a batch count once. Cross-account correlation uses only unsuppressed sources, and its derived alerts can be muted with their own suppression rules without removing them from enforcement.
| Finding | Description |
|---|---|
wp_login_bruteforce | WordPress login flood via wp-login.php |
xmlrpc_abuse | XML-RPC endpoint flood |
email_php_form_relay | Client that made hosted sites mail many outside addresses through their forms (responses require email_protection.php_relay.ip_recipient_mode: enforce, including retries; routed to the PoW challenge first when challenge is enabled) |
http_request_flood | Per-IP HTTP request volume exceeds threshold (disabled by default; enable by setting thresholds.http_flood_threshold > 0) |
http_scanner_profile | Random-URL probe pattern from one source IP (disabled by default; enable by setting thresholds.http_scanner_min_requests > 0; routed to the PoW challenge first unless auto_response.http_scanner_action: "block") |
http_claimed_bot_unverified | High-volume claimed crawler traffic while reverse-DNS verification is pending (routed to the PoW challenge first when challenge is enabled) |
http_ua_spoof | IP exceeding the UA anomaly threshold, including confirmed search-engine bot UA spoofing (periodic; see configuration.md for opt-in flags) |
ftp_bruteforce | FTP authentication flood |
smtp_bruteforce | SMTP authentication flood |
smtp_probe_abuse | Raw SMTP connect-rate flood before AUTH |
mail_bruteforce | IMAP/POP3/ManageSieve authentication flood without matching successful mailbox activity |
mail_account_compromised | Successful login from an IP that repeatedly failed auth on the same mailbox, or a mail login after failures on two or more mailboxes including another one, or on a missing one. Only Critical findings block; High advisories are visibility only |
admin_panel_bruteforce | phpMyAdmin or Joomla admin POST flood |
ssh_login_unknown_ip | SSH login from an IP with no prior history, whether the realtime watcher or the periodic scan saw it first |
c2_connection | Outbound connection to a known C2 server |
ip_reputation | IP flagged by AbuseIPDB / rspamd / upstream threat-intel |
modsec_block_escalation | ModSecurity deny escalation |
modsec_csm_block_escalation | CSM-internal ModSecurity deny escalation |
waf_attack_blocked | WAF high-volume attacker |
email_compromised_account | Email account compromise indicator |
email_cloud_relay_abuse | Cloud relay abuse |
Successful cPanel, FTP, webmail and PAM login audit events and authenticated File Manager writes do not trigger direct blocks or challenges. Incidents containing only these audit events cannot request a block, even when they retain a higher severity from an older version. Independent attack evidence can still justify a response.
block_cpanel_logins gates cPanel multi-IP logins, API authentication failures,
webmail brute force and realtime FTP authentication failures. This includes
webmail challenge routing, since an unanswered challenge can become a block.
Queued block candidates are checked against the current check policy and login-blocking setting before retrying. Legacy queue entries without a check identity are discarded on upgrade; new eligible findings can queue them again. Existing firewall blocks and permanent evidence are not removed by this change.
Distributed HTTP flood rollups do not trigger a direct IP block because they describe one targeted vhost, not one source IP. The per-IP findings that feed the rollup still drive normal block decisions.
http_asn_crawl is also handled separately because one finding can carry several offending CIDRs rather than one source IP. Only Critical findings with confirmed PHP worker saturation can tempban those CIDRs, using auto_response.http_asn_crawl_tempban. Dry-run and subnet safety guards still apply.
mail_bruteforce_suspected and the High forms of mail_account_compromised
are visibility only. They do not feed direct, incident, or spray
auto-blocking. A separate blockable finding can still block the same source.
mail_bruteforce_suspected also reports SMTP, IMAP and POP password sprays
while spray detection runs in observe mode.
In enforce mode a spray block lasts thresholds.mail_spray_block_expiry (30
days by default) instead of block_expiry; an address already blocked keeps
its current expiry. A block that mixes a spray finding with another finding
for the address, such as a WAF or per-IP brute force finding, is a normal
block: it lasts block_expiry and uses the scan budget. When the temporary
block limit is full, the spray block that ends soonest is evicted first, and
an operator’s temporary block is never evicted for an automatic one. A
blocked address is remembered until one spray memory window after its block
ends. One failed login from it in that time, on any mailbox, blocks it again
for the spray length. That includes a failure logged just before an
authentication outage is detected. On a shared IPv6 /64, an address blocked
on the /64’s shared evidence keeps its own re-block until it is released too.
An operator unblock, or an undo that removes a block, clears the address’s
spray history: its re-block and the failure history of the address, or for
IPv6 of its /64. An allow or temporary allow clears it even when the address
holds no block, and so does the allow CSM adds for an address that a
dyndns_hosts or infra_ips hostname resolves to. Its SMTP, IMAP and POP
failure counts restart across all address spellings. Login trust and
cracked-password records stay. Pending SMTP rejections are discarded so
delayed replies cannot restore old spray evidence. A self-service unban
through the panel API with an unban-scope token lifts the block and its
re-block but keeps the failure history and counts, so renewed spraying is
blocked again on the usual rules. An admin token or browser session on that
route performs a full operator release. Failed firewall releases return an
error without clearing threat state. A flush clears the spray history,
per-address failure counts and pending rejections of every address.
Authentication outage guards remain active after a release or flush.
Every auto-response block source - scan findings, challenge-timeout
escalations, central-intel corroborated blocks, and incident spray
containment - records the same evidence: a temporary threat-DB row, a
blocked-IPs tracker entry, an auto_block finding visible to the block
digest and alerting, and a step toward permanent-block escalation. A block
the admission ledger applied is never promoted to a permanent block while it
stands; its offense still counts toward a later block of the address.
All of them block only while auto_response.enabled and block_ips are
on; with either off, an expired challenge simply leaves the challenge list.
Reloading these switches takes effect before the next block attempt, including
later entries in a challenge-timeout batch. Incident block refusals caused by
these switches keep the incident visible without a failed-block warning.
Admission applies auto_response.max_blocks_per_hour to scan, challenge,
central, and incident IP blocks. On the legacy path that budget applies to
scan-driven blocks only.
Mail login spray blocks use a separate hourly budget of the same size. They reserve credit before the firewall attempt. An uncertain result keeps that credit; a confirmed non-block releases it. Retries wait when the spray budget is full, while other scan blocks keep their own budget. Waiting spray retries hold at most half of the retry queue and yield space to other retries when the queue fills. Deferred retries follow the same limits. One warning an hour names the budgets that are full when it is sent, and says when retries were dropped because the spray share of the queue was full. A failed spray reservation defers spray blocks and reports a warning, without stopping other scan blocks.
The same setting also sets the hourly ceiling of the admission ledger, which
covers every IP block it executes, including challenge, central, and incident
blocks. For the ledger an omitted or zero value selects 2000 and a value above 20000 is
clamped to 20000; the per-IP limit above keeps reading omitted or zero as 50.
An explicit value sets both. Packaged configs used to set 50, which keeps the
ceiling at 50 until the key is removed; csm doctor warns about a configured
ceiling below 2000.
At its first start the ledger imports the current hour’s block count, so a
restart cannot grant a fresh hour; a block count that cannot be read starts
it without saved credit. Status and csm doctor show the ceiling and its
source. A changed ceiling applies on reload, and a web UI settings save
applies at the owner’s next clock tick; a failed change is retried on later
ticks before admission resumes, with elapsed time credited at the saved
ceiling before the new one applies. Every automatic response the existing
blocking paths select is also handed to the ledger. With
auto_response.dry_run false, the firewall on and outside observe mode,
the ledger applies automatic IP blocks itself within this ceiling and the
free slots of the temporary block set, and the existing paths leave those
blocks to it. In dry run it forecasts them. With the firewall disabled or
in observe mode it previews them; neither applies them. While a configured
firewall engine is starting or unavailable, queued live blocks wait and
new responses remain on the legacy path. Work the set has
no room for waits as set_full until a slot frees: the slots operator,
unknown and subnet blocks leave are divided as in the forecast, legacy
automatic blocks occupy general slots, and an unlimited set counts as 10000
entries. A Critical local attack, or one with direct compromise or
independently corroborated evidence, may instead take the general slot of
a legacy automatic block that recorded its origin. With the firewall on, queued responses disabled by the current
policy end as disabled without a withheld alert, including when their evidence expires before a
restart or configuration reload. Turning a response back on requires a new
eligible observation; ended work is not replayed. Status, csm doctor and findings
history show the results. Forecast endings have no new per-attempt row in
the action log.
The resulting auto_block findings are output evidence, not new local
corroboration for central intelligence or incident correlation. They are
deduplicated before the digest, attack database, history, and alert sinks
receive them. Only a block that applied marks its address as auto-blocked in
the attack database; a dry-run “would be blocked” warning does not.
Central intelligence and incident auto-blocking act on an address only when
a finding names it as an attacker. The remote end of an outbound connection,
a customer login, and advisories such as ModSecurity warnings or visitors of
a mail-sending script do not count, even when another finding raised the
incident’s severity. That includes a listed command-and-control server: only
the C2 connection check blocks it, when auto_response.block_ips is on.
Known-customer block alert
When an automatic block applies to an address that logged in successfully as
a customer in the last 30 days (cPanel, webmail, SMTP, IMAP, POP or FTP), CSM
sends an alert at once: one email, and the generic alert webhook as event
customer_block with the blocks under csm.customer_blocks (see the
API reference). It names the
address, the block reason and the check that caused it, and the customer’s
login kind, account or mailbox and last login time. An IPv6 customer is named
by its /64. A subnet block or netblock that covers known customers sends one
message with their count and the five most recent. Every applied block
counts: scan blocks, permanent promotions, challenge timeouts, central
intelligence and incident blocks. A customer whose own login caused the block
is left out, also when the block comes later than its cause: a mail account
compromise or an FTP login after brute force names that address and account
in its own Critical alert. This also applies when earlier address blocks
escalate to a range block: its count and examples include only the remaining
customers. Each block names the command that lifts it:
csm firewall remove <ip> for an address,
csm firewall remove-subnet <cidr> for a range. An allow rule
(csm firewall allow <ip>) trusts the address on every port. It does not
pass a blocked range, but CSM never blocks a range over an allowed address.
The alert is the false-positive check of automatic blocking, never an
exemption. The block stands with its configured length, and its notice still
goes to the daily summary, which also lists the block with its customers, so
a failed alert still reaches you. A login with stolen credentials also makes an
address known, so the alert can show the block of an attacker holding a
customer’s password, which is worth seeing too. Each applied block sends at
most one message; a later new block of the same address sends another. A
promotion to a permanent block is named in its block’s message when both
reach CSM together; a promotion that reaches CSM apart from its block sends
a message of its own. The alert replaces the live watched-country alert for
the same block, and the block still counts in the watched-country section of
the daily summary. It does not spend the hourly alert budget, goes out with
alerts.block_digest off and ignores alerts.block_digest.channel. A phpanel
webhook does not receive it. Dry-run and observe-mode verdicts that would have
blocked a known customer are listed in the
daily summary instead.
Safety Guards
- Never kills root processes, system daemons, or cPanel services
- Infrastructure IPs (
infra_ipsandfirewall.infra_ips) are never blocked or sent to the challenge page - A block of an address that recently logged in as a customer is alerted at once and never exempted
- Subnet blocks refuse the default route and any range that covers infrastructure, local host, allowed, or port-specific allowed IPs
- Quarantined files preserve full metadata for restoration
- Every regular file is copied from its verified open descriptor into a private quarantine inode before the detected name is removed. Other hard links are reported after removal; a file swapped into the detected path is reported as a refused remediation, with the captured copy kept as evidence and the replacement left untouched
- Realtime signature auto-quarantine requires high confidence: category
webshellordropper, file size at least 512 bytes, and either Shannon entropy >= 5.5 or hex density > 20% with an obfuscated-execution signal. This prevents legitimate WordPress plugins from being quarantined. - IP blocks are limited by
auto_response.max_blocks_per_hour: admission defaults to 2000/hour and legacy scan blocking to 50/hour. - CRITICAL alerts and threat-intel reputation sightings always bypass the operator email/webhook rate limit (default 30/hour); lower-severity findings batched with them still count against it
- Attacks CSM blocked or challenged, and its applied block and challenge notices, go to the daily summary instead of alert email and the generic webhook; an attack whose block failed, is still queued or ran in dry run is alerted immediately, as are a block of a Cloudflare-covered address and the web attacks from it
- Trusted countries (
trusted_countries) suppress login alerts from expected geolocations
What CSM Detects in Real-Time
Beyond standard malware patterns, CSM detects advanced evasion techniques:
- Fragmented function names: attackers split
base64_decodeacross variables ($a="base"; $b="64_decode") to evade simple string matching - Appended payloads: malicious code added to the end of large legitimate files, beyond typical scan windows. Realtime PHP checks scan the first and last 32KB, and periodic PHP content analysis scans a larger head window plus the tail.
- Non-PHP backdoors: Perl, Python, Bash CGI scripts in web directories (detects toolkits like LEVIATHAN)
- SEO spam injection: gambling/togel dofollow link injection into theme files
- WordPress brute force: real-time access log monitoring for wp-login.php and xmlrpc.php floods (blocks within seconds, not the 10-minute periodic scan)
- Admin-panel brute force: same access-log path, tracks POSTs to
/phpmyadmin/index.php,/pma/index.php,/phpMyAdmin/index.php, and Joomla/administrator/index.php. Emitsadmin_panel_bruteforceand auto-blocks the IP. Path matcher is intentionally tight to avoid false positives on shared hosting; Drupal and Tomcat Manager use different attack shapes and need separate detectors. - SMTP brute force and probes: tails
/var/log/exim_mainlogon cPanel and non-cPanel Exim hosts where the file exists. Emitssmtp_probe_abuseandsmtp_bruteforce(per-IP, auto-blocks),smtp_subnet_spray(per-/24, auto-blocks the whole subnet), andsmtp_account_spray(per-mailbox, visibility only). Spray detection addsmail_bruteforce_suspectedin observe mode andmail_account_compromisedfor a login after failures on other mailboxes. - Mail brute force: tails
/var/log/maillogfor direct IMAP, POP3, and ManageSieve auth failures. Composes with the existing geo-login monitor soemail_suspicious_geokeeps working. Emitsmail_bruteforce,mail_bruteforce_suspected,mail_subnet_spray,mail_account_spray,mail_account_compromised, andmail_auth_backend_degraded. Established good sources with a confined stale-password pattern emit the suspected advisory without auto-blocking. A compromise finding from an IP established on at least two other mailboxes stays visible as High without auto-blocking. Wider spraying and compromise without that standing still block. When the auth backend is degraded,mail_bruteforceandmail_subnet_sprayauto-blocking pause until backend errors age out. - Mail auth backend probe (cPanel): independently of the log signals above, CSM opens the
cpdoveauthdsocket on a short interval. dovecot keeps answering its IMAP/POP3 ports during a cpdoveauthd outage, so cPanel’s own service checks do not notice, yet every login fails regardless of password. When the probe finds the socket unreachable CSM raisesmail_auth_backend_degradedand pauses both mail and SMTP brute-force auto-block. Withauto_response.mail_auth_recovery.restart_enabled, CSM restarts the mail service once the backend has been continuously down pastdown_grace(default 10m, rate-limited), so a brief blip during maintenance never triggers a needless restart. Changes to mail auth recovery settings require a daemon restart.
Network dry-run precedence
Three settings combine for direct-SMTP BPF enforcement. Any applicable dry-run value that is true wins; live network denial requires all three to be false.
| Layer | Knob | Default | Effect when true |
|---|---|---|---|
| Auto-response | auto_response.dry_run | true | Record automatic block decisions without applying firewall denial |
| Detector | detection.direct_smtp_egress.dry_run | true | Suppress detector-scoped action |
| Kernel | bpf_enforcement.dry_run | true | BPF program emits decision but allows traffic |
The kernel knob is consulted by the BPF program itself; the others gate userspace action paths. All three default to true on a first install so a configuration mistake cannot start blocking traffic.
Observe mode
mode declares what CSM is allowed to do to the host it runs on.
| Value | Meaning |
|---|---|
enforce (default) | Every subsystem acts under its own switch. This is how CSM has always behaved. |
observe | Detection, correlation, alerting and the audit sinks run. Automatic host remediation and integration updates are disabled. |
mode: observe
Use observe mode to evaluate detection quality on a real host before granting CSM the ability to act, or to run CSM permanently as a reporting sensor next to another response tool.
What observe mode stops
Observe mode skips host integration work that otherwise runs automatically:
- The auditd rules file is written and
augenrulesis run, so CSM’s audit layers stay current across package upgrades. - The host integration files are refreshed: the WHM plugin CGI and its AppConfig registration, and the CSM section of the ModSecurity user config.
- Legacy challenge snippets and managed webserver integration snippets are refreshed, with webserver validation and reloads when needed.
- The WAF check refreshes stale vendor rules and deploys custom ModSecurity rules during periodic scans.
- The Exim forward guard is reconciled, including removing an installed guard when its switch is disabled.
- The antispam integration is reconciled, including removing it when its
switch is disabled. An installed integration is reported as
observe_modeand left in place. - The AF_ALG kernel mitigation is re-applied on each critical tick when an opted-in hardening marker is present.
Existing host protections are left in place. Remove or change them explicitly before switching modes if that is the intended posture.
Resolve pending firewall apply/rollback windows in enforce mode before switching to observe. Observe startup refuses a pending settings or ruleset rollback: restoring it would change the host, while ignoring it would abandon the rollback deadline.
Contradictory settings are refused, not rewritten
A config that sets mode: observe and still enables a subsystem that changes
host state is rejected at load, naming every conflicting key at once:
mode: observe forbids changing host state, but these keys still enable it:
auto_response.enabled (set false), firewall.enabled (set false)
The keys checked are auto_response.enabled, firewall.enabled,
php_shield.enabled, bpf_enforcement.enabled,
email_protection.forward_guard.enabled, email_protection.antispam.enabled,
auto_response.php_relay.freeze,
auto_response.mail_auth_recovery.restart_enabled, and
auto_response.virtual_patch_exposed_files set to auto. It also refuses
auto_response.copy_fail_kill_process: true and, when the antivirus scanner is
enabled, email_av.quarantine_infected: true and email_av.fail_mode: tempfail,
all of which act on a host independently of auto_response.enabled.
Mail that makes the attachment parse worker fail is allowed with a finding
instead. In enforce, CSM requests deferral whatever fail_mode says when a
permission event still holds delivery.
CSM refuses rather than silently turning those switches off in memory, because
the config re-hash path marshals the in-memory config back over csm.yaml:
an in-memory override would eventually be written into the operator’s file.
What still runs
Detection is unchanged. Real-time watchers, scheduled checks, correlation,
incidents, alerts, webhooks, the SSE stream and the audit-log sinks all behave
as they do under enforce, except that checks report host drift without fixing
it. CSM still writes its own configured state, log, cache and quarantine trees,
and creates its runtime sockets under /run/csm (/var/run/csm). Explicit
config edits and reloads can update the config integrity hashes.
Manual operator commands still work. csm firewall deny, csm clean,
csm virtual-patch --apply, csm db-clean and csm harden are explicit
actions an operator takes, not daemon behaviour, so observe mode does not block
them. auto_response.virtual_patch_exposed_files: manual is accepted for the
same reason.
Signature and GeoIP updates still run: they write only inside CSM’s own directories.
Two host writes remain, and the capability matrix marks both as not configurable:
- On a BPF build, capability discovery loads and briefly attaches probe programs to find out what the kernel supports.
- The Copy Fail check runs
kcarectl --patch-infoto see whether a livepatch covers the host. kcarectl refreshes its own cache under/var/cache/kcareon every run, including a read-only query.
Neither changes the host’s security configuration, and skipping them would cost the detection they exist for. They are listed here so “observe” is not read as a promise of zero writes.
Confirming the posture
csm doctor # "operating mode: observe (no automatic remediation or integration changes)"
csm status # mode: observe
csm status --json # .mode
curl .../api/v1/status # .mode
The capability string mode.observe.v1 reports that a build understands the
setting.
Changing mode requires a restart (systemctl restart csm), not a reload.
Doctor compares the running mode with the configured mode and warns if they
differ. When the daemon is unreachable, it reports only the configured mode.
Capability matrix
Every operation CSM performs that needs privilege beyond reading its own files, or that writes outside its own directories, with the privilege it needs and the setting that stops it.
Print it on any host, before or after install:
csm privileges # table
csm privileges --json # machine-readable
The command prints a static inventory. It does not load CSM configuration or probe host privileges, so it can be used before installing the service. Output failures produce a nonzero exit status in text, JSON and Markdown formats.
How to read it
Needs is what the operation requires from the kernel. root means uid 0
rather than one capability: the operation reads or writes files owned by many
different accounts, or drives a panel tool that assumes root. Today the daemon
runs as a single root process; splitting it into a small privileged helper plus
a reduced-privilege main process is planned, and this inventory is its first
stage. Capability names describe the privileged kernel interfaces as well as
the root filesystem access. For BPF, tracing and LSM programs need
CAP_BPF plus CAP_PERFMON; cgroup socket programs need CAP_BPF plus
CAP_NET_ADMIN. CAP_SYS_ADMIN is the broader fallback on older kernels.
These requirements follow the kernel program-load checks.
Trigger is automatic when the daemon may start the operation on its own,
and operator when it runs only in response to a command or a button. Not
running the command is how an operator operation is turned off.
Writes lists what the operation writes while it runs, including writes made
by a tool it invokes. Paths starting with / are filesystem paths; everything
else is a resource named <kind>:<name>. “nothing (read-only)” means the
operation only reads. Paths describe default locations; configured account roots,
state, rule, log, spool and policy paths replace those defaults and may require
service drop-ins. Shared CSM state and audit-log writes are grouped in the state
rows. This is not a list of every possible path written by an external panel tool.
Turn it off is the config key and YAML value that stops the named operation.
Dotted keys describe nested YAML mappings; they are not literal top-level keys.
Apply changes before restarting the daemon: several controls are read only at
startup. A row saying “not configurable” has no single config switch that stops
all its callers. disabled_checks takes a list and only affects scheduled checks.
Stopping new work does not undo installed rules, hooks, quarantines or blocks;
use the subsystem removal commands for that. Disabling the forward guard or
antispam removes its existing Exim and rspamd configuration, which is itself a
host write, so both rows name mode: observe.
Action record says whether the operation writes to the action log. “no” does not mean the operation is silent; it means it is not on that stream yet and the daemon log is where it appears.
Without the privilege is what an operator loses by withholding it, so the row reads as a decision rather than a demand.
Relationship to the systemd sandbox
The daemon runs under ProtectSystem=strict with an explicit writable-path
allow-list; see service confinement. The two lists are
compared by tests: every declared in-daemon filesystem write must fit a packaged
grant, and every grant must have an in-daemon writer. Empty inventories and grant
lists fail. Paths are compared on directory boundaries, and an operation outside
the sandbox cannot justify a daemon grant. The unit parser rejects invalid paths
and unsupported syntax, including continuations, specifiers and directory flags,
instead of guessing which locations are writable. Tests cannot discover omitted operations
or prove the privilege and disable claims; those require tracing the runtime code.
Operations marked “outside the systemd sandbox” run as standalone operator
commands or transient services forked by PID 1. The latter use direct execution
only where their helper permits it; the archive parse worker has no direct
fallback when systemd is unavailable. Mixed operations have
separate rows for their in-daemon writes, such as the AF_ALG marker repair and
forward-guard lookup refresh. The service sandbox does not bound external tools.
Archive worker memory limits are also written inside the daemon sandbox.
Their cgroup: resources are kernel controls, writable because the packaged
unit keeps ProtectControlGroups=no, rather than ordinary directory grants.
Turning most of it off at once
mode: observe stops automatic remediation and integration deployment, and
refuses configurations that enable those subsystems. It still maintains CSM state,
loads temporary BPF capability probes in BPF builds, and invokes the KernelCare
probe, which can update its own cache. Archive worker startup cleanup and
on-demand unit launches also remain active. It is not a guarantee of zero host writes. See
observe mode.
The matrix
The risk tier says what can go wrong if an operation acts on the wrong
target: 0 changes nothing outside CSM’s own directories, 1 records a
recommendation or dry-run decision, 2 is a low-risk change such as a probe,
challenge gate or mail hold, 3 quarantines, blocks or denies a target, and 4
signals processes, reloads services, or rewrites content or configuration.
Rows describe maximum live effect, so preview-only tier 1 has no separate
row today. A tier does not promise automatic rollback; see recovery gaps
below and the current contracts in the JSON inventory. The TIER column of
csm privileges and the JSON inventory use the same 0 to 4 numbers.
| Operation | Needs | Trigger | Risk tier | Writes | Turn it off | Action record | Without the privilege |
|---|---|---|---|---|---|---|---|
state.control_socketcreate the private control socket used by operator commands | root | automatic | 0 | /var/run/csm | not configurable | no | CLI commands cannot reach the daemon |
state.mail_relay_policiesload operator-supplied mailer classes and proxy ranges for the PHP-relay detector | none | automatic | 0 | nothing (read-only) | email_protection.php_relay.enabled: false | no | only policy files readable by the daemon uid can be loaded |
state.php_shield_eventsreceive PHP Shield events over a socket and append their local archive | root | automatic | 0 | /var/log/csm-php-shield | php_shield.enabled: false | no | PHP runtime events are not collected |
state.sign_configrewrite CSM’s own integrity hashes into csm.yaml after an approved change | root | operator | 0 | /etc/csm | do not run the command | no | the integrity hashes cannot be re-recorded, so the next restart refuses to start after any config edit |
state.update_forgedownload and verify YARA Forge rules independently of YAML updates | root | automatic | 0 | /opt/csm/rules | signatures.yara_forge.enabled: false | no | YARA Forge rules are not refreshed |
state.update_signaturesdownload and signature-verify YAML malware rule updates | root | automatic | 0 | /opt/csm/rules | signatures.update_url: "" | no | detection freezes at the ruleset shipped with the installed package |
state.write_logswrite the daemon log and the audit-log sinks that feed a SIEM | root | automatic | 0 | /var/log/csm | not configurable | no | no local record of findings or actions |
state.write_storewrite the bbolt state database, baselines, incidents and scan reports | root | automatic | 0 | /var/lib/csm, /opt/csm/state | not configurable | no | CSM cannot run: without state there is no baseline, no dedup and no incident history |
detect.account_databasesread account MySQL credentials and scan databases for injected content and stored objects | root | automatic | 0 | nothing (read-only) | not configurable: db_object_scanning only stops stored-object scans; content and requested scans remain | no | no database detection: injected admin users, poisoned options and malicious triggers stay invisible |
detect.af_alg_socketsdeny AF_ALG sockets through BPF LSM, or observe socket use through the audit-log fallback | CAP_BPF, CAP_PERFMON, root | automatic | 3 | kernel:AF_ALG socket denial | detection.af_alg_backend: none | no | the live monitor is gone; the periodic critical check still reports an exposed kernel |
detect.audit_rulesquery loaded audit rules with auditctl to check detection coverage | CAP_AUDIT_CONTROL, root | automatic | 0 | nothing (read-only) | not configurable | no | audit-rule coverage cannot be verified |
detect.bpf_probeload and briefly attach BPF LSM, tracepoint and cgroup programs to discover kernel support | CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN, root | automatic | 2 | kernel:temporary BPF programs and maps | not configurable: capability discovery runs independently of monitor settings in BPF builds | no | BPF capabilities report unavailable; configured automatic backends use their fallbacks |
detect.file_placement_bpfwatch files renamed or hard-linked into watched trees through BPF LSM rename and link hooks | CAP_BPF, CAP_PERFMON, root | automatic | 2 | kernel:BPF programs and maps | detection.placement_backend: none | no | auto mode falls back to the fanotify placement group where the kernel has it; an explicitly selected BPF backend stays unavailable |
detect.file_placement_fanotifywatch files renamed or hard-linked into watched trees through a fanotify group that reports directory handles, opened with open_by_handle_at | CAP_SYS_ADMIN, CAP_DAC_READ_SEARCH, root | automatic | 0 | nothing (read-only) | detection.placement_backend: none | no | files renamed or hard-linked into a site are checked by the next deep scan instead of at once |
detect.filesystem_eventsfanotify stream over account roots and world-writable temp directories | CAP_SYS_ADMIN | automatic | 0 | nothing (read-only) | not configurable | no | no real-time file detection; scheduled scans still run, so a webshell lives until the next cycle |
detect.kernel_livepatch_proberun kcarectl –patch-info to see whether a KernelCare livepatch covers Copy Fail; kcarectl rewrites its own cache on every run | root | automatic | 2 | /var/cache/kcare | not configurable: the startup kernel probe and hardening audits also invoke kcarectl, including in observe mode | no | a patched kernel reads as vulnerable, so the AF_ALG finding cannot be cleared |
detect.kernel_oomread the restricted kernel message buffer with dmesg to find recent OOM kills | CAP_SYSLOG, root | automatic | 0 | nothing (read-only) | not configurable | no | swap statistics remain available, but OOM kills are not reported |
detect.mail_queue_probequery the Exim queue through a transient unit, because Exim opens its logs even to answer a read | root | automatic | 2 | /var/log/exim_mainlog, /var/log/exim_paniclog, /var/log/exim4 (outside the systemd sandbox) | disabled_checks: [mail_queue] | no | queue depth is unknown, so a queue-size spam outbreak may go undetected |
detect.outbound_connectionswatch outbound connections through BPF cgroup hooks, or by polling /proc/net/tcp | CAP_BPF, CAP_NET_ADMIN, root | automatic | 2 | kernel:BPF programs and maps | detection.connection_tracker_backend: none | no | auto mode uses polling; an explicitly selected BPF backend stays unavailable |
detect.pam_eventsreceive authentication attempts from the pam_csm.so hook over a private socket | root | automatic | 0 | /var/run/csm | not configurable | no | no immediate brute-force or credential-stuffing block; log parsing lags by up to a minute |
detect.parse_worker_memorycap the memory and swap of the archive and mail parse workers through kernel cgroup controls inside the daemon sandbox | root | automatic | 2 | cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dparse\x2dworker*.slice/csm-parse-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dparse\x2dworker.slice/csm-parse-worker-*.service | not configurable: disabled_checks only suppresses scheduled phishing checks and email_av.enabled: false stops the mail workers; requested account scans remain | no | a worker gets no request if its memory and swap cannot be capped: the phishing check reports itself incomplete and email attachments are handled as incomplete scans under email_av.fail_mode |
detect.parse_worker_unitsstart the archive and mail parse workers as locked-down transient units and attempt to kill leftover parse worker units at startup | root | automatic | 2 | systemd:transient units, systemd:runtime slice CPU weights (outside the systemd sandbox) | not configurable: disabled_checks only suppresses scheduled phishing checks and email_av.enabled: false stops the mail workers; requested account scans and startup cleanup remain | no | tenant zip archives are not checked for phishing kits, the phishing check reports itself incomplete, and email attachments are handled as incomplete scans under email_av.fail_mode |
detect.phptaint_worker_memorycap the memory and swap of the PHP taint worker through kernel cgroup controls inside the daemon sandbox | root | automatic | 2 | cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dphptaint\x2dworker.slice/csm-phptaint-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dphptaint\x2dworker.slice/csm-phptaint-worker-.service | not configurable: disabled_checks with php_taint_deep stops new analyses; startup cleanup of leftover units remains | no | PHP files are not analyzed for remote-input taint |
detect.phptaint_worker_unitsstart the PHP taint worker as a locked-down transient unit and attempt to kill leftover PHP taint worker units at startup | root | automatic | 2 | systemd:transient units, systemd:runtime slice CPU weights (outside the systemd sandbox) | not configurable: disabled_checks with php_taint_deep stops new analyses; startup cleanup of leftover units remains | no | PHP files are not analyzed for remote-input taint |
detect.process_execwatch process execution through a BPF tracepoint, or by walking /proc | CAP_BPF, CAP_PERFMON, root | automatic | 2 | kernel:BPF programs and maps | detection.exec_monitor_backend: none | no | auto mode uses periodic /proc sampling; an explicitly selected BPF backend stays unavailable |
detect.read_service_logsread mail, authentication, web server and ModSecurity logs | root | automatic | 0 | nothing (read-only) | not configurable | no | no mail abuse, brute-force or WAF detection: these logs are root-readable only |
detect.scan_account_filesread every account’s files for scheduled, real-time and on-demand scans | CAP_DAC_READ_SEARCH, root | automatic | 0 | nothing (read-only) | not configurable: disabled_checks only suppresses scheduled checks; realtime and requested scans remain | no | only files readable by CSM’s own uid are scanned, which on a shared host is close to nothing |
detect.sensitive_file_writeswatch writes to /etc/shadow and comparable files through a BPF LSM hook | CAP_BPF, CAP_PERFMON, root | automatic | 2 | kernel:BPF programs and maps | detection.sensitive_files_backend: none | no | sensitive-file changes are found by periodic hashing instead of at the moment of the write |
detect.yara_worker_memorycap the memory and swap of the YARA scan and compile workers through kernel cgroup controls inside the daemon sandbox | root | automatic | 2 | cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dyara\x2dworker.slice/csm-yara-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dyara\x2dworker.slice/csm-yara-worker-.service, cgroup:/sys/fs/cgroup/memory/system.slice/system-csm\x2dyara\x2dcompile\x2dworker.slice/csm-yara-compile-worker-.service, cgroup:/sys/fs/cgroup/system.slice/system-csm\x2dyara\x2dcompile\x2dworker.slice/csm-yara-compile-worker-.service | signatures.yara_worker_enabled: false | no | files and mail attachments are reported as not scanned by YARA |
detect.yara_worker_unitsstart the YARA scan and compile workers as locked-down transient units and attempt to kill leftover YARA worker units at startup | root | automatic | 2 | systemd:transient units, systemd:runtime slice CPU weights (outside the systemd sandbox) | signatures.yara_worker_enabled: false | no | files and mail attachments are reported as not scanned by YARA |
integrate.antispaminstall, update or remove CSM’s rspamd settings, optional feeds, learning files, the webmail Junk-button include and Exim spam-filter hooks, rebuild the Exim configuration and reload Exim and rspamd; on Postfix hosts only CSM’s rspamd files and an rspamd reload | root | automatic | 4 | /etc/rspamd/local.d, /etc/rspamd/override.d, /etc/rspamd/lua.local.d, /usr/local/cpanel/base/3rdparty/roundcube/config/inc.d, /usr/local/cpanel/etc/exim/acls, /etc/exim.conf.local, /etc/exim.conf, /var/lib/csm, exim:rebuild artifacts and service reload, service:rspamd reload (outside the systemd sandbox) | mode: observe | no | CSM’s spam-filter settings and hooks cannot be installed, updated or removed; existing mail filtering keeps its current configuration |
integrate.antispam_ip_mapwrite the CSM IP map rspamd reads, readable only by root and rspamd’s group, and delete it once no rule reads it | root | automatic | 2 | /var/lib/csm-rspamd | mode: observe | no | rspamd gets no scores from CSM’s own evidence; other filtering is unchanged |
integrate.antispam_learnlearn an operator’s file as spam in rspamd’s local fuzzy storage, or forget the clusters it matches, keeping one anchor message per cluster | root | operator | 2 | /var/lib/csm/antispam/learning, rspamd:fuzzy storage | email_protection.antispam.learning.enabled: false | yes | operators cannot teach or correct antispam learning; rspamd’s other scoring is unchanged |
integrate.antispam_recordinvalidate pending antispam workers before reconciliation and record each new request under the shared Exim lock inside the daemon sandbox | root | automatic | 0 | /var/lib/csm/forward_guard | not configurable: pending workers must be invalidated even when antispam is disabled or mode is observe | no | pending antispam workers cannot be invalidated and new antispam changes are not handed to the root worker |
integrate.auditd_ruleswrite CSM’s auditd rules and reload them, so audit-backed detection survives package upgrades | CAP_AUDIT_CONTROL, root | automatic | 4 | /etc/audit, kernel:audit rules | mode: observe | no | audit-backed detection layers stay inactive after an upgrade |
integrate.challenge_port_gateinstall the separate nftables port gate for the public challenge listener | CAP_NET_ADMIN, root | automatic | 2 | nftables:challenge port gate | challenge.port_gate.enabled: false | no | the listener remains reachable without the challenge port filter |
integrate.challenge_snippetrefresh the web server snippet and rewrite maps that route challenged visitors to CSM’s proof-of-work listener, and reload the web server when the snippet changed | root | automatic | 4 | /etc/apache2/conf.d, /etc/apache2/conf-enabled, /etc/httpd/conf.d, /etc/nginx/conf.d, /usr/local/lsws/conf/templates, /var/cache/csm, service:web server reload | mode: observe | no | suspicious visitors are blocked outright instead of being offered a challenge |
integrate.firewall_rulesetbuild and load CSM’s nftables table, including the operator’s port policy and rate limits | CAP_NET_ADMIN, root | automatic | 4 | nftables:csm table | firewall.enabled: false | yes | no firewall management; another tool owns the host’s packet policy |
integrate.modsec_sectionrefresh the managed ModSecurity section at startup and during periodic WAF checks, preserving operator configuration | root | automatic | 4 | /etc/apache2/conf.d, /usr/local/apache/conf | mode: observe | no | CSM’s virtual-patch WAF rules are not installed |
integrate.panel_plugindeploy the WHM plugin CGI and its AppConfig entry, and register it with the panel | root | automatic | 4 | /usr/local/cpanel/whostmgr/docroot/cgi, /var/cpanel | mode: observe | no | no panel plugin; the web UI is still reachable on its own port |
integrate.php_shieldinstall the PHP runtime hook and register its shared event directory for an operator-scheduled CageFS remount | root | operator | 4 | /opt/csm, /var/log/csm-php-shield, /etc/cagefs, /etc/csm, /opt/cpanel, /opt/alt, /usr/local/lsws, php:runtime configuration (outside the systemd sandbox) | do not run the command | no | no PHP runtime blocking of uploads and temp-directory execution |
integrate.waf_vendor_rulesask the panel to update stale ModSecurity vendor rulesets when a periodic check finds them out of date | root | automatic | 4 | /etc/apache2/conf.d, /usr/local/apache/conf, /var/cpanel, service:web server reload | mode: observe | no | stale WAF rules are reported but not refreshed |
operate.apply_hardeningapply the hardening audit’s mechanical fixes for one section on an admin’s web UI request, in a root worker started outside the sandbox, then reload the services that read them | root | operator | 4 | /etc/ssh/sshd_config, /etc/ssh/sshd_config.csm-bak, /etc/sysctl.d, kernel:sysctl values, /etc/shadow, /tmp, /var/tmp, /etc/cpupdate.conf, /usr/bin, cpanel:tweak settings, /opt/cpanel, /etc/php, /etc/php.d, /usr/local/lsws/admin/tmp, service:sshd, PHP-FPM and Apache reload (outside the systemd sandbox) | do not run the command | no | each fix is applied by hand from the audit’s suggested fix |
operate.clear_stalwart_banlift Stalwart’s own automatic bans of an address unblocked, allowed or unbanned in CSM or allowed through a DynDNS name, through the Stalwart API; other Stalwart bans stay | root | operator | 2 | stalwart:blocked address list (outside the systemd sandbox) | do not run the command | yes | Stalwart keeps refusing the address until its ban ends or is lifted in Stalwart |
operate.delete_forwarderdelete one destination of a mail forwarder through uapi, as the account that owns it, on an admin’s web UI request | root | operator | 4 | cpanel:mail forwarders (outside the systemd sandbox) | do not run the command | no | forwarders are deleted in cPanel |
operate.export_archivesread protected configuration, state or account evidence and export backup or forensic archives, temporary snapshots and checksum sidecars | root | operator | 4 | filesystem:operator-selected archive destinations (outside the systemd sandbox) | do not run the command | no | protected configuration, state and account evidence cannot be exported completely |
operate.harden_hostapply a supported CVE mitigation: a modprobe blacklist, or seccomp drop-ins for the services that need one | CAP_SYS_MODULE, root | operator | 4 | /etc/modprobe.d, /etc/systemd/system, service:restart, kernel:modules (outside the systemd sandbox) | do not run the command | no | the mitigation is applied by hand from the command the audit prints |
operate.install_serviceinstall or remove CSM itself: the systemd unit, the PAM hook, logrotate and the panel integrations | CAP_AUDIT_CONTROL, CAP_LINUX_IMMUTABLE, root | operator | 4 | /opt/csm, /etc/csm, /var/lib/csm, /var/log/csm, /etc/systemd/system, /etc/pam.d, /etc/logrotate.d, /etc/audit, /usr/sbin/csm, /lib64/security, /usr/lib64/security, /lib/security, /lib/x86_64-linux-gnu/security, /usr/lib/x86_64-linux-gnu/security, /lib/aarch64-linux-gnu/security, /usr/lib/aarch64-linux-gnu/security, /usr/local/cpanel/whostmgr/docroot/cgi, /var/cpanel, service:csm and panel integrations, /etc/cron.d, /var/cache/csm, /opt/cpanel, /var/run/csm, /etc/apache2/conf.d, /etc/apache2/conf-enabled, /etc/httpd/conf.d, /etc/nginx/conf.d, /usr/local/apache/conf, /usr/local/lsws/conf/templates (outside the systemd sandbox) | do not run the command | no | CSM cannot be installed as a service |
operate.manual_firewallblock, allow, tempban or flush addresses on request, and roll a firewall apply back | CAP_NET_ADMIN, root | operator | 3 | nftables:csm sets | do not run the command | yes | firewall changes are made with nft or the panel’s own tooling |
operate.manual_remediationclean or quarantine files and spool messages, truncate malicious crontabs, kill verified malware processes, or change database objects on request | CAP_KILL, root | operator | 4 | /home, /tmp, /var/tmp, /dev/shm, /var/spool/cron, /var/spool/exim/input, /var/spool/exim4/input, /opt/csm/quarantine, mysql:account databases, process:signal | do not run the command | no | remediation is done by hand over SSH |
operate.rehashre-hash configuration, converge legacy config copies, set binary immutability and refresh the launcher, service and log rotation | CAP_LINUX_IMMUTABLE, root | operator | 4 | /opt/csm, /etc/csm, /etc/systemd/system, /etc/logrotate.d, /usr/sbin/csm, service:daemon reload (outside the systemd sandbox) | do not run the command | no | hash recording and upgrade integration refresh cannot complete |
operate.restore_backupstage and restore protected configuration, drop-ins and state from a backup while the daemon is stopped | root | operator | 4 | /etc/csm, /var/lib/csm, /opt/csm, /tmp (outside the systemd sandbox) | do not run the command | no | configuration and state cannot be restored |
operate.stalwart_setupprepare Stalwart through its admin API: log failed logins, log rejected spam, trust the webmail client address behind CSM’s port lock, match ban length to CSM’s blocks, and create the role, service account and API key CSM acts through | root | operator | 2 | stalwart:failed-login and spam-verdict log levels, HTTP client address source, ban periods, settings reload, stalwart:response role, service account and API key, /etc/csm/stalwart-api.key (outside the systemd sandbox) | do not run the command | no | Stalwart is prepared by hand in its admin interface |
operate.truncate_error_logempty an account error log that has grown large enough to threaten the filesystem | root | operator | 4 | /home | do not run the command | no | a bloated log is reported and truncated by hand |
operate.release_mailboxrelease a mailbox suspension or account outgoing-mail hold CSM made, on an operator’s request | root | operator | 2 | cpanel:mailbox login and outgoing suspension, cpanel:account outgoing mail hold, stalwart:account sign-in permission (outside the systemd sandbox) | do not run the command | yes | the suspension stays until it is released in cPanel or Stalwart |
respond.af_alg_enforceunload the AF_ALG kernel modules again when an opted-in mitigation marker is present | CAP_SYS_MODULE, root | automatic | 4 | kernel:modules (outside the systemd sandbox) | auto_response.disable_enforce_af_alg: true | no | a module reload silently reopens the Copy Fail exposure |
respond.af_alg_killkill a verified AF_ALG socket caller through the separate Copy Fail response setting | CAP_KILL, root | automatic | 4 | process:signal | auto_response.copy_fail_kill_process: false | no | the process is reported but remains running; this path is independent of kill_processes |
respond.af_alg_markerrestore a changed AF_ALG mitigation marker after the operator has opted in | root | automatic | 4 | /etc/modprobe.d | auto_response.disable_enforce_af_alg: true | no | a changed module blacklist is reported but cannot be repaired |
respond.block_ipadd an attacker address or subnet to the firewall’s deny sets | CAP_NET_ADMIN, root | automatic | 3 | nftables:csm sets | auto_response.block_ips: false | yes | attacks are reported but not stopped; dry-run records what would have been blocked |
respond.bpf_deny_egressdeny matched outbound connections in the kernel through a BPF cgroup hook | CAP_BPF, CAP_NET_ADMIN, root | automatic | 3 | kernel:bpf cgroup program | bpf_enforcement.enabled: false | no | direct-to-MX spam egress is detected but not stopped at the source |
respond.clean_filestrip injected code from a PHP or access file, keeping a pre-clean backup | root | automatic | 4 | /home, /tmp, /var/tmp, /dev/shm, /opt/csm/quarantine | auto_response.enabled: false | yes | injected files are reported and left in place |
respond.database_cleanuprevoke a rogue CMS admin, sanitize poisoned options, and drop confirmed malicious stored objects after recording their definition | root | automatic | 4 | mysql:account databases | auto_response.clean_database: false | no | database persistence survives file cleanup and re-infects the account |
respond.enforce_permissionschmod a world-writable or group-writable PHP file back to 644 | root | automatic | 3 | /home | auto_response.enforce_permissions: false | no | writable code files are reported only |
respond.fix_wp_crondisable WP-Cron for an account and install a per-user system cron entry instead | root | automatic | 4 | /home, /var/spool/cron, /tmp | auto_response.fix_wp_cron: false | no | runaway WP-Cron load is reported only |
respond.forward_guardinstall or remove the managed Exim block that holds forwarded spam and backscatter, and rebuild the Exim configuration | root | automatic | 4 | /etc/exim.conf.local, /etc/exim.conf, /var/lib/csm, exim:rebuild artifacts and service reload (outside the systemd sandbox) | mode: observe | no | forwarded spam keeps damaging the host’s sending reputation |
respond.forward_guard_checkrecord the latest forward-guard request and check whether a rule is installed under the shared Exim lock; cancel pending requests in observe mode without waiting for the lock, inside the daemon sandbox | root | automatic | 0 | /var/lib/csm/forward_guard | not configurable: pending forward-guard workers must be canceled even in observe mode | no | forward-guard changes are not handed to the root worker, pending workers cannot be canceled, and the installed guard stays as it is |
respond.forward_guard_lookuprefresh the forward-guard bad-sender lookup inside the daemon sandbox | root | automatic | 0 | /var/lib/csm/forward_guard | email_protection.forward_guard.enabled: false | no | Exim continues using the last successfully written sender lookup |
respond.freeze_mailfreeze queued Exim messages attributed to a confirmed PHP-relay finding | root | automatic | 2 | /var/spool/exim/input, /var/spool/exim4/input, /var/log/exim_mainlog, /var/log/exim_paniclog, /var/log/exim4 (outside the systemd sandbox) | auto_response.php_relay.freeze: false | no | a compromised script keeps sending until an operator freezes the queue |
respond.hold_outgoing_mailrequest a cPanel account outgoing-mail hold through whmapi1 after sustained mail abuse | root | automatic | 2 | cpanel:account outgoing mail hold | auto_response.enabled: false | yes | the mail abuse finding is reported but the account can continue sending |
respond.kill_processsignal a malicious process through a kernel process handle, never a recycled PID, never root | CAP_KILL, root | automatic | 4 | process:signal | auto_response.kill_processes: false | yes | reverse shells and miners keep running until an operator kills them |
respond.mail_delivery_gatedefer Exim delivery with fanotify permission responses when tempfail policy requires it or a message makes its parse worker fail | CAP_SYS_ADMIN, root | automatic | 2 | fanotify:mail delivery decisions | email_av.enabled: false | no | mail AV falls back to notifications and cannot defer delivery on scan failures |
respond.quarantine_filemove a confirmed malicious file out of an account tree into CSM’s quarantine, preserving owner, permissions and mtime | CAP_LEASE, root | automatic | 3 | /home, /tmp, /var/tmp, /dev/shm, /opt/csm/quarantine | auto_response.quarantine_files: false | yes | malware is reported and left in place |
respond.quarantine_mailmove an infected message out of the Exim spool after an antivirus match | root | automatic | 3 | /var/spool/exim/input, /var/spool/exim4/input, /opt/csm/quarantine | email_av.quarantine_infected: false | no | infected mail is reported and delivered |
respond.restart_mail_authrestart the panel’s mail authentication service after a sustained outage | root | automatic | 4 | service:mail authentication | auto_response.mail_auth_recovery.restart_enabled: false | no | an authentication backend outage is alerted but not repaired |
respond.suspend_mailboxsuspend one mailbox’s logins and outgoing mail through uapi, or its Stalwart sign-ins through the Stalwart API, after its credentials are abused | root | automatic | 2 | cpanel:mailbox login and outgoing suspension, stalwart:account sign-in permission (outside the systemd sandbox) | auto_response.enabled: false | yes | the credential abuse finding is reported but the mailbox keeps logging in and sending |
respond.virtual_patchwrite a reversible deny rule into an account access file to close an exposed file | root | automatic | 3 | /home, /opt/csm/quarantine | auto_response.virtual_patch_exposed_files: off | no | exposed backups, dumps and configs stay reachable until the account owner fixes them |
Regenerate the table with go run ./cmd/csm privileges --markdown. A gate in
internal/ci fails when the page and the inventory disagree.
Risk tiers and recovery coverage
csm privileges --json reports each operation’s risk tier as a number from 0
to 4. A row with no tier would report -1; the test suite refuses such an
inventory, so a release never ships one. The tier is the operation’s maximum
live effect on a wrong target. It is not the severity of the finding that
triggered the operation.
Eight operations carry a safety contract in the JSON inventory:
respond.block_ip, operate.manual_firewall, integrate.firewall_ruleset,
integrate.challenge_port_gate, integrate.challenge_snippet,
integrate.antispam_ip_map, respond.quarantine_file and
respond.clean_file. Each states the authority it
needs, how the target is revalidated before the change, how the change is
reversed, and the limit that bounds it. The contracts describe current
behaviour, including what it does not cover. Every other operation that changes
the host states the recovery this inventory does not yet cover:
| Operations | Recovery gap |
|---|---|
detect.bpf_probe, detect.outbound_connections, detect.process_exec, detect.sensitive_file_writes, detect.file_placement_bpf, detect.af_alg_sockets, respond.bpf_deny_egress | This inventory does not yet specify verified detach, map restoration and crash recovery for these kernel hooks. |
detect.kernel_livepatch_probe, detect.mail_queue_probe | This inventory does not specify recovery of incidental external cache or log writes by probe commands. |
detect.parse_worker_memory, detect.phptaint_worker_memory, detect.yara_worker_memory | Memory and swap limits end with the worker cgroup; this inventory specifies no separate cleanup contract. |
detect.parse_worker_units, detect.phptaint_worker_units, detect.yara_worker_units | Worker units and their cgroups end with the unit; empty slices and their runtime CPU weights remain until reboot. Startup attempts to kill leftover workers and logs failures. |
respond.mail_delivery_gate, respond.hold_outgoing_mail, respond.suspend_mailbox, respond.freeze_mail, respond.quarantine_mail, operate.release_mailbox | This inventory does not yet specify identity checks for releasing or restoring mail, or recovery after a restart. |
respond.af_alg_enforce, respond.af_alg_marker, respond.forward_guard, respond.fix_wp_cron, integrate.auditd_rules, integrate.modsec_section, integrate.panel_plugin, integrate.php_shield, integrate.waf_vendor_rules, operate.harden_host, operate.install_service, operate.rehash, operate.restore_backup | This inventory does not specify an action-wide snapshot and verified rollback of configuration, service and external-tool side effects. |
respond.af_alg_kill, respond.kill_process, respond.restart_mail_auth | Process termination and restart cannot restore lost process state; this inventory does not yet specify a full recovery contract. |
respond.database_cleanup, respond.virtual_patch, respond.enforce_permissions, operate.manual_remediation, operate.truncate_error_log | Current remediation may retain local recovery evidence, but per-operation identity-checked undo and partial-failure recovery are not specified by this inventory. |
integrate.antispam | Rollback cannot change existing SMTP sessions; failed rollback or an interrupted worker needs another apply or remove transaction. |
integrate.antispam_learn | Forget removes CSM’s contribution but cannot restore the near-match coverage other learned messages lost to it, and a lost reply is settled by reconciliation, not undone. |
operate.export_archives | Export can replace an existing operator-selected archive; removing the new archive does not restore overwritten bytes. |
operate.delete_forwarder | The web UI audit log names the deleted destination and its account; nothing re-creates the forwarder. |
operate.apply_hardening | sshd_config keeps a .csm-bak copy and is restored when sshd -t rejects the change; sysctl, permission, WHM tweak, cPanel update and PHP drop-in changes are undone by hand. |
operate.stalwart_setup | Setup prints the previous values; undoing a change is done by hand in Stalwart’s admin interface. |
operate.clear_stalwart_ban | Stalwart bans the address again on new abuse; a cleared ban is not restored. |
Self-test
csm selftest scans a bundle of samples whose verdicts are known and reports
what the installed rules catch. It reads no account data and writes nothing, so
it is safe to run on a production host, and it answers the question an
evaluator actually has before pointing a scanner at real sites.
csm selftest
csm selftest --json
Exit status is 0 when every sample matched what the bundle records for each available engine, and 1 when anything needs attention. Incomplete or empty rule loads, scan errors, invalid samples and an empty bundle fail the run. A build without YARA-X can pass its realtime checks, but explicitly reports that YARA coverage was not tested.
The command uses the configured rules directory and disabled-rule list for
both engines. Disabling a detection can turn a sample into MISSED. A missing
or unreadable configuration fails the command rather than measuring a full
packaged ruleset that the host may not run. Pass --config and --config-dir
when the daemon uses custom paths.
What the bundle contains
Adversarial samples and benign controls, in one list. The controls are the half that decides whether a rule set is usable: a scanner that flags an ordinary WordPress plugin is worse than one that misses a shell.
Samples are stored base64-encoded and decoded only in memory. Endpoint antivirus deletes files that look like web shells, and a decoded copy on disk would make the bundle disappear from the machine it is meant to test.
The split-assert sample represents legacy PHP string assertions, not PHP 8 behaviour. The chr-built sample constructs a callable command-execution function; the test suite checks that it is callable without running its payload.
Reading the output
| Verdict | Meaning |
|---|---|
detected | An adversarial sample the rules caught. |
clean | A benign control the rules left alone. |
known gap | An adversarial sample the shipped signature rules do not catch, recorded in the bundle. |
MISSED | An adversarial sample that should have been caught and was not. A regression. |
FALSE POSITIVE | A benign control the rules fired on. |
GAP CLOSED | A recorded gap that now fires. Good news; the bundle needs updating. |
ERROR | The sample could not be decoded or scanned. The error is printed below it. |
The summary counts missed samples, false positives, closed gaps and errors
separately. A skipped engine has no sample results; JSON reports the reason in
its skipped field.
If configuration disables every rule in an engine, the command still measures its samples, reports zero loaded rules and the resulting misses, and exits with failure. Missing or malformed rule files remain load errors.
Known gaps are recorded, not hidden
A gap is a measurement, and the test suite fails when one closes as well as when one opens. A rule change that starts catching a recorded gap breaks the build until the bundle is updated, so no gap can quietly become permanent and no improvement goes unnoticed.
A gap is a gap in the signature engines. CSM’s other layers – PHP taint analysis, the behavioural checks, PHP Shield, and the correlation that turns findings into incidents – are not measured by this bundle. A sample listed as a known gap may still be caught in production by one of them.
Engines
CSM ships two rule sets, and the command reports each separately:
realtime, the YAML rules the real-time watchers use.yara, the YARA-X rules used by scheduled and email scanning. Present only in builds compiled with YARA-X; a build without it reportsSKIPPEDwith a reason. A compiled engine whose rules cannot load fails the command.
The regular CI test job runs the realtime gate. The required test:production
job runs scripts/production-tests.sh portable, which includes all packages
with the yara,journal,bpf tags and the pinned YARA-X library, so it runs the
YARA bundle gate too. Tagged builds and lint alone do not execute that test.
To measure the bundle locally, install PHP CLI for the fixture validity check, then run:
go test -count=1 ./internal/selftest
go test -count=1 -tags yara ./internal/selftest
The second command requires the pinned YARA-X library. The untagged command does not measure YARA detection.
Incidents
CSM groups related findings into Incident objects so operators see one escalating story per account, mailbox, or process instead of a stream of unrelated findings. Original findings are not mutated or suppressed – the Incident is layered on top.
Classification
Each incident is assigned a kind at create time. Inbound attacks are keyed
on the attacker source IP, not on the victim they name:
web_attack– a WAF block, scanner probe, or login brute-force. When the finding names a victim domain or account, that is the attack target, not evidence the account is compromised, so the incident keys on the attacker IP and one attacker’s hits across many vhosts collapse into one incident.mailbox_bruteforce– failed mailbox logins, mail brute-force bursts, and pre-auth SMTP probes from one source. A failed login is an attack attempt, not a takeover, so it keys on the attacker IP rather than the targeted mailbox. A threat-intelligence match on an address seen failing IMAP or POP3 logins opens this kind too, including alerts queued during shutdown and delivered after restart.
Compromise kinds are reserved for evidence that an attack succeeded:
web_account_compromise (on-disk or behavioural signals such as a webshell or
suspicious PHP) and mailbox_takeover (post-authentication abuse such as
outbound spam, cloud relay, or a compromised-account signal). Inbound-attack
kinds carry short attacker-grade retention; compromise kinds get the longer
review window.
Two further classes of finding never open an incident at all. Findings that record an action CSM took, or how well it can see, are excluded so CSM’s own output cannot re-enter its decision path. Findings that report a standing weakness in installed software – a known-vulnerable or outdated plugin – are excluded because nothing has happened yet and there is nothing to contain. Known-vulnerable plugin findings remain alertable. Routine outdated-plugin notices remain informational, and both classes appear in the findings list.
Lifecycle
| Status | Meaning |
|---|---|
open | Active. New findings for the same correlation key keep merging in. |
contained | Operator marked under control. Findings still merge in window. |
resolved | Closed. Future findings start a new incident. |
dismissed | False positive. Future findings start a new incident. |
Resolved and dismissed incidents are pruned 30 days after their last
update when an operator closed them, and 7 days after their last update
when the daemon closed them (closed_by starting with auto:). Older records
without close attribution keep the 30-day period. Confirming an automatic
closure, changing a closed status, or recording a block on a closed incident
gives that decision 30 days of retention. Recorded operator decisions on older rows with
stale automatic attribution also keep this longer period. Reopening clears
close attribution; the next closure determines retention again.
Open and contained incidents are never auto-pruned by the retention loop, but they may be auto-resolved by the per-kind idle threshold described under “Auto-close” below. Retention sweeps use bounded transactions so a large backlog does not hold the store writer for the entire cleanup. Pruning frees space for reuse; it does not shrink the state file. Finding history has its own retention, so a retained incident can refer to findings already evicted from history.
Auto-close
To stop the open-incident backlog from growing without bound on busy
hosts, the daemon scans Open / Contained incidents shortly after startup
and then once an hour, auto-resolving any whose updated_at exceeds the
per-kind idle threshold. A live sweep closes at most 1000 stale incidents
at a time; if more stale incidents remain, follow-up sweeps run every 30
seconds until the backlog drains. Dry-run sweeps still scan the full set
so the counters show every would-close decision. Auto-resolved incidents
carry closed_by: "auto:stale" and an incident_auto_closed action in
their timeline so reporting can distinguish them from operator closes.
Defaults (configurable in csm.yaml):
incidents:
auto_close:
enabled: true # set false to disable
dry_run: false # set true to log decisions without writing back
by_kind:
mailbox_takeover: 24h
mailbox_bruteforce: 24h
credential_spray: 24h
web_attack: 24h
web_account_compromise: 168h
Kinds absent from by_kind are never auto-closed. The default map
omits host_integrity_risk, host_takeover, and post_exploit_process
because those host-level incidents should stay open until an operator
reviews them. host_takeover is the compound escalation raised when any
two of three host-takeover legs (a new uid-0 account, a planted suid
binary, an outbound connection to a bad ASN) are correlated for the same
host inside the merge window.
If a fresh finding for the same correlation key arrives after the auto-close, the merge-window stale-binding logic creates a new open incident – nothing about auto-close blocks re-detection. History is preserved on the closed record.
Tuning on high-volume hosts. Each by_kind threshold is the idle
time before a kind auto-resolves; they are independent and operator-set.
A host under sustained brute-force keeps a large open set mostly from the
longer-lived kinds (web_account_compromise defaults to 168h). If the
open-incident count is higher than you want to triage, shorten the
relevant by_kind entry (e.g. web_account_compromise: 72h) rather than
disabling auto-close. Untouched auto-resolved records are retained 7 days,
measured from when the incident resolves, so shortening the threshold also
moves the eventual prune point earlier relative to the last finding.
Auto-close still keeps a resolved record for follow-up instead of deleting
history at close time.
Metrics: csm_incidents_auto_closed_total and
csm_incidents_auto_close_dry_run_total.
Credential-spray suppression
Failed mailbox logins already collapse onto the attacker IP as a single
mailbox_bruteforce incident (see “Classification”). Spray suppression
extends that across protocols: it tracks the distinct-mailbox/account set
per source IP for the configured per_check detectors (mail and PAM)
across the merge window and, once an IP exceeds distinct_mailboxes,
opens a single credential_spray super-incident keyed on the IP with
breadth-based severity escalation. Subsequent findings from that IP
attach to the spray incident’s timeline.
If per_check includes mail_subnet_spray or smtp_subnet_spray, grouping
uses their single cidrs subnet. Older findings with the subnet in source_ip
keep the same grouping, including when replayed or joining restored incidents.
Defaults (configurable in csm.yaml):
incidents:
spray_suppression:
enabled: false # default OFF; opt-in
dry_run: true # default ON; counters move, routing unchanged
distinct_mailboxes: 10 # threshold to trip
severity_escalate_at: 50 # bump severity to CRITICAL at this many
per_check:
- email_auth_failure_realtime
- pam_bruteforce
- credential_stuffing
max_tracked_ips: 10000
block_at_severity: "" # "" detection-only, "high" block on open,
# "critical" block on escalation
Setting block_at_severity hands the source IP to the firewall as soon
as the spray detector trips at the chosen tier, once
spray_suppression.dry_run is false. The detector also requires
auto_response.enabled and auto_response.block_ips; the firewall still
honors auto_response.dry_run, so a dry-run host logs the would-be block
without applying nftables rules. Live accepted requests are recorded on
the incident timeline as a credential_spray_block_requested action.
Non-live outcomes (dry-run, verdict-allow, already blocked) and failed
attempts do not latch the incident, so a later finding can retry after
blocking is live again. Concurrent findings for the same incident share
one in-flight firewall call, and resolved or dismissed spray incidents do
not make new block decisions.
A spray incident blocks only an address that a finding with address
evidence attests, the same gate every incident block uses. Raw mailbox
failures such as email_auth_failure_realtime count mailboxes and open the
incident, but cannot block the source alone; pam_bruteforce or
credential_stuffing from the same address can. Visibility-only findings,
including mail_bruteforce_suspected and the High mail_account_compromised
advisory for an established multi-mailbox source, never make a spray
incident blockable either.
Incident blocks name the finding that supplied address evidence. The
attesting check and severity survive trimming and restart and are checked
again at startup. Retained active proof in older incidents is migrated;
missing or retired proof cannot authorize another block until a fresh
qualifying finding arrives. Existing installed blocks keep their expiry,
and incidents.auto_block and incidents.spray_suppression still control
responses. At startup, older incidents learn
the link from an attesting timeline event even when their evidence flag was
already set. If an older timeline has no finding identity or already lost
that event, the original link cannot be recovered; it is never made from
message text. The next finding that attests the key’s address records its own
identity when that link is missing, and the new link survives trimming and
restart. Existing blocks are unchanged.
Automatic incident blocks use the address-evidence gate without a separate
exclusion list. Advisory events alone never attest an address, but can raise
an incident’s severity after an earlier attestation, even when that event
left a trimmed timeline. An ip_reputation match never attests either: an
incident it opens blocks only after a local attack finding from the same
address joins it. Retained evidence in existing incidents follows
this rule on the next finding after upgrade. The existing incident and spray
blocking settings still govern the action.
Whitelisted IPs (entries in reputation.whitelist and the live bbolt
whitelist updated via the Web UI) are skipped from spray detection so
internal mail relays, NAT egresses, and known-good infrastructure
never produce a spray incident. Changes to reputation.whitelist apply
on config reload.
Choosing block_at_severity:
""(default) – detection-only. Spray incidents open, no firewall hand-off. Use during dry-run validation and on hosts where blocking is owned by a separate system.high– block at thedistinct_mailboxestrip. Recommended once the dry-run counter looks clean. Trips on the first sustained burst before the source IP goes idle for longer than the merge window.critical– block only after severity escalates, i.e. one IP hitsseverity_escalate_atdistinct mailboxes before the source IP is idle for more than the merge window. A low-and-slow attacker that stays below that count before each idle reset never escalates and never blocks. Pick this only when you have strong shared-NAT exposure and accept that slow sprayers evade the gate.
Rollout:
- Ship the daemon with
enabled: false, dry_run: true. The detector tracks per-IP mailbox sets and incrementscsm_credential_spray_dry_run_totalwhenever the threshold would have tripped, but routing stays on the legacy per-mailbox path. - Validate the counter on your own infrastructure for 24h. If a
trusted IP shows up in the dry-run trips, add it to
reputation.whitelist. - Flip
enabled: true, dry_run: false. New attacker IPs route through the spray path; existing per-mailbox backlog drains via the auto-close path. - After another 24h, set
block_at_severity: high. The firewall hand-off runs on every spray decision (open + merge), so an incident opened before the flag was armed still blocks on the next finding from the same IP.
Metrics: csm_credential_spray_opened_total,
csm_credential_spray_suppressed_mailbox_takeover_total,
csm_credential_spray_dry_run_total,
csm_credential_spray_tracked_ips.
Incident auto-block
spray_suppression only handles the credential_spray super-incident
kind. Low-and-slow scanners that never trip a per-detector window
(modsec escalation, mail brute-force, smtp probe) still produce
web_attack, mailbox_bruteforce, mailbox_takeover, or
web_account_compromise incidents but never get firewalled. The
incidents.auto_block block adds a generic incident-driven firewall
hand-off:
incidents:
auto_block:
enabled: false # default OFF; opt-in
block_at_severity: "" # "" / "high" / "critical"
kinds: [] # empty = any non-spray kind with one source IP
When the gate trips, the correlator hands the source IP to the firewall
through the same dry-run / block_ips gate as the spray path. A live
accepted request records incident_block_requested; non-live outcomes
(dry-run, verdict-allow, already blocked) do not latch the incident, so
an operator who arms auto_block AFTER an incident has already crossed
the gate still gets a block on the next finding while the incident is
open or contained. Incidents with multiple source IPs are left for manual
review.
If a long-running incident’s timeline was truncated and the source IP is
not part of the incident key, auto-block also stays off because the
remaining visible timeline may not contain every source IP.
The incident blocks an address only when a finding that names it as an
attacker joined the incident. An outbound connection’s remote end, a
customer login or an advisory never qualifies, whatever raised the
incident’s severity; see Auto-Response.
Qualifying address evidence is retained when findings join an incident and across restarts, including for incidents saved by earlier versions. Reopening an incident keeps its evidence; a new incident starts with its own evidence.
The same visibility-only findings are excluded from generic incident
auto-blocking. In particular, setting block_at_severity: high does not turn
an established multi-mailbox mail_account_compromised advisory into firewall
evidence. A Critical compromise finding remains blockable.
credential_spray is explicitly excluded from this path; the dedicated
spray hand-off owns it. Set kinds to narrow the surface (e.g. only
web_account_compromise) if you do not want every CRITICAL
mailbox_takeover incident to block its source IP.
This pairs naturally with the ModSecurity escalation thresholds
(thresholds.modsec_escalation_hits,
thresholds.modsec_escalation_window_min) – raising the window from
the shipped default of 10 minutes to e.g. 4 hours lets the modsec
detector promote paced scanners to a Critical escalation finding,
which then trips the generic auto_block gate.
ModSecurity escalation is confidence-gated. Each deny is classified as
high-confidence (a specific attack/probe rule – SQLi, RCE, traversal,
URL-encoding abuse, CSM custom, or an OWASP CRS rule from an
APPLICATION-ATTACK rule file, which is how LiteSpeed logs, since it
omits the rule message and tags), low-confidence (policy/anomaly scoring
rules such as COMODO content-type 210710 or anomaly-points 214930,
and OWASP CRS anomaly-score rules), or unknown. Message and tag evidence comes
from the rule configuration CSM reads, with macro references removed, so text a
request puts into a logged message cannot change confidence. Without that
configuration, known rule IDs and the rule file decide; other rules stay unknown.
A burst escalates to a
firewall ban at the normal hit count only when it contains a
high-confidence deny or an unknown deny; a low-confidence-only burst
emits one non-actioned modsec_low_confidence_burst finding for
visibility instead of banning, then stays quiet until that active
low-confidence window drains. This stops false bans of legitimate
traffic (e.g. an unusual checkout that only trips content-type/anomaly
rules) without blinding CSM to real attacks, which trip the
high-confidence rules. A determined source that floods only
low-confidence rules is still banned once it reaches the
thresholds.modsec_low_confidence_escalation_hits backstop (default
30). Unknown blocking rules are escalation-eligible (fail-secure) and
raise a modsec_classifier_gap finding once per rule for the whole host
(repeated daily while the rule stays unclassified) so a new vendor rule
pack is noticed rather than silently given a no-ban path. A deny whose
request text imitates the rule fields ModSecurity writes counts as an unknown
rule, so a request cannot borrow another rule’s confidence or escalation
setting. A few CSM rules that match one specific exploit’s request shape (for example the PHP2Shell
upload progress field or the wp2shell tool User-Agent) escalate on their
first deny, whatever the hit threshold; turning firewall escalation off for
that rule on the ModSecurity rules page opts out, including its retained
evidence in later decisions. Upload progress requests carrying browser headers
are denied by a separate rule whose escalation is off by default. Requests
lacking those headers retain first-deny escalation. See
ModSecurity Integration for the browser-header limitation and
behavior when saved exclusions cannot be read. Existing bans keep their
configured expiry.
Warnings remain non-blocking even when request fields contain text resembling
a deny message.
While qualifying evidence remains in the window, duplicate escalations stay
suppressed until auto_response.block_expiry allows fresh activity to renew
the ban.
Kinds
web_account_compromise– findings attributable to a hosted account or script (PHP relay, webshell, account-scoped login bruteforce, etc.).web_attack– an inbound web attack or remote-IP reputation/threat-score signal not tied to failed mailbox logins. When the finding includes a victim domain or account, that field is recorded as target context, not as the correlation key. Keyed on the source IP and given attacker-grade retention (default 24h) so defended probes and attacker-IP reputation hits do not inflate the account-compromise count or its longer review window.mailbox_bruteforce– failed mailbox authentication, mail brute-force bursts, account-spray signals, pre-auth SMTP probes, and remote-IP reputation matches on addresses seen failing IMAP or POP3 logins. Keyed on the attacker source IP with attacker-grade retention (default 24h).mailbox_takeover– post-authentication mailbox abuse such as suspicious geo, credential leaks, cloud relay abuse, spam outbreaks, and confirmed compromised-account signals.post_exploit_process– process exec from/tmp,/var/tmp,/dev/shm.host_integrity_risk– daemon/kernel-level signals (sensitive file changes, fake kernel threads, binary/config tampering). Periodic binary/config tamper findings join the local host incident even without account or IP attribution. Startup verification still alerts and refuses to start before normal incident processing is available.host_takeover– any two of a new uid-0 account, a planted suid binary, and an outbound connection to a bad ASN, seen for the same host inside the merge window.credential_spray– one source IP brute-forcing many distinct mailboxes/accounts inside the merge window. Keyed on the source IP rather than per-mailbox, so a scanner spraying thousands of usernames produces one super-incident instead of thousands of mailbox_bruteforce rows. Findings from the same IP after the trip attach to this incident’s timeline. See “Credential-spray suppression” below.
The host-integrity set, all five compound sets, kind selectors and identity
exclusions are checked against the detector registry. The independent test
fixture internal/incident/testdata/check-policy.json records an explicit
classification role and selecting-set membership for every registered check,
including checks with no named override. Tests reject unknown names, missing
eligible members, new tables without a contract and new checks without a
policy decision. They also exercise each check with account, mailbox, process
and source-IP attribution to test classification precedence.
These tests live in the external incident_test package, which can import the
check registry without adding a production dependency from incident to
checks. That dependency would cycle through checks -> control -> incident.
Changes to the fixture require a review of the detector’s emitted evidence;
do not regenerate it from the selecting tables. Classification coverage does
not calibrate correlation weights or prove that broader compound membership
is safe.
Severity policy
Severity escalates only. Each incident keeps the highest severity any
joined finding has carried. Findings themselves are never re-emitted at
a higher severity. The audit trail records an
incident_severity_changed action when an incident’s severity bumps.
Correlation window
15 minutes by default. Findings outside the window for the same key start a new incident. The window is a named constant in code; not yet exposed via config.
Open threshold
Non-Critical findings normally need at least two correlated sightings inside the merge window before an incident opens. The first sighting is held in a pending bucket and counted toward the threshold; the second promotes both into a new incident with a two-event timeline. A mail-filter exfil finding remains a first-hit incident even when an uncorroborated copy-forward is graded Warning, so the operator review path is not lost. Stale pending entries are pruned by the daily retention sweep.
Critical-severity findings (account compromise, cloud-relay abuse, modsec rule escalations) bypass the threshold and open immediately so escalations still page on first hit.
The threshold suppresses one-shot scanner noise (a single modsec
deny from a wandering scanner, an isolated mistyped password) without
hiding sustained activity. The current pending-bucket size is exposed
as the csm_incidents_pending gauge.
The stored incident includes the full correlation key, including process PID/UID and remote IP when those are the only available dimensions, so active incidents keep merging after daemon restart.
Cross-account correlation of findings
Separately from incidents, the scan runner, the latest-state merge and the realtime dispatcher derive two findings from the findings they see:
coordinated_attack(Critical) when at least three distinct hosting accounts each carry at least one Critical finding from a check classified as a security event or malware artifact. The checks may differ between accounts. Repeated findings or several installs inside one account never raise the count.cross_account_malware(Critical) when the same malware-artifact check (webshell,new_webshell_file,backdoor_binary,new_executable_in_config) is present on two or more accounts at any severity. Different malware checks on different accounts do not combine.
Over the persisted latest-state set, both aggregates only combine findings
whose condition was first observed in the last hour. A scan re-emits every
finding it still sees and the merge refreshes its report time, so judging by
that would let a months-old condition re-enter the window on every cycle; the
merge therefore carries each finding’s first observation across re-reports and
correlation reads that instead, including when a completed scan replaces its
owned findings. A condition removed by a completed scan starts a new observation
if it is found again later. The window is relative to the merge time, so
the next completed scan clears expired aggregates even if it produces no
findings. Source rows are retained when their aggregates expire, a finding
carrying no first observation falls back to its report time, and a finding
carrying no timestamp at all still counts. Derived timestamps never affect the
window. JSON output omits first_seen when no first observation was recorded.
The first observation is not retroactive. On upgrade, a stored finding adopts its existing report time, because nothing recorded when that condition actually started. A long-lived finding with a recent report can therefore count once after upgrading; it ages out one hour after that saved report time, and the next completed scan clears expired aggregates even if it reports the same conditions again.
The realtime and scan batch paths keep their existing grouping and count all qualifying rows in the batch. A scan can carry forward an older finding for a file it could not examine; its original timestamp does not exclude it from batch correlation or get refreshed by correlation.
Every registered check is classified as a security event, a malware
artifact, ignored with a stated reason, or derived. Derived findings are never
inputs. A nonempty TenantID wins verbatim, including case, over the file
path and text. It must identify the host-local hosting account. Existing
stored values and verdict callbacks use the same precedence; callbacks must
supply the same owner keys as producers. External identifiers can split one
owner or combine distinct owners, and correlation does not validate them.
If that field is empty, correlation uses the account home containing the
cleaned absolute FilePath. Relative paths, root/home-only paths and paths
that escape the home after cleaning do not identify that home. This is a
lexical mapping through the platform’s account roots, including Plesk roots;
it does not resolve symlinks or prove that directory aliases are distinct
owners. Nonstandard content roots need producer-supplied identity.
The final fallback scans Message before Details for an account-root path.
It recognizes <account-root>/<user>/, not (account: user) labels. Within
each field it searches roots in configured order, can match an embedded root
substring, and can select an incidental path. These compatibility limits are
why account-aware producers should supply identity.
Qualifying rows with no identity from any source do not count toward either aggregate. They are counted once per call and logged once per check name per process, using only that call’s row count and no finding text. Ignored, derived, unknown and below-threshold security-event rows produce no diagnostic count. A check with a declared attribution gap is still eligible: when its producer does supply an authoritative owner, that Critical counts.
The health snapshot (csm status --json, /api/v1/status) carries a
correlation_attribution block with two views: current is the per-check
count of unattributed qualifying rows retained in the active set and inside
the correlation window at its latest merge, including any eviction caused by
the size limit. Unstamped legacy rows also count. It clears when a later merge
attributes those rows or they age out of the window; cumulative
sums every unattributed row since the daemon started, across active-set
merges and per-batch derivations, so a producer that recovered stays visible
as having failed. Counters are published together in merge order. The block
is absent until the first merge. csm doctor
reports the same state as correlation attribution: OK when current is
empty, WARN naming the checks and their counts otherwise, with the history
in both cases.
The two per-batch derivations (scan runner, realtime dispatcher) see only
their batch and produce alerts. The latest-state merge derives from the
merged, deduplicated, capped persisted active set under the same lock as the purge,
so evidence from separate scans combines: three accounts compromised in three
different scans still produce a persisted coordinated_attack, and it clears
only when fewer than three accounts still carry a qualifying Critical there.
Each completed runner snapshot replaces only its own checks’ rows; rows kept
for an unscanned owner or a file coverage gap stay as inputs; the previous
derived findings are dropped and recomputed from the merged set. Demotion,
dismissal or re-verification of a contributing row takes effect at the next
nonempty scan merge, and only if fewer qualifying accounts remain; there is
no immediate recompute and no promise that every file is re-verified.
Demoting a malware artifact below Critical cannot clear the all-severity
malware aggregate by itself. There is no time window, no shared-signature or
causal requirement, and no precision multiplier: the widened Critical inputs
accumulate across scans and can include unrelated events and false
positives, which is why the threshold calibration stays open on the roadmap.
Callers initialize platform detection before correlation; the latest-state caller does so before taking the store lock. Correlation reads cached roots, and attribution warnings are reported after the merge releases that lock.
Class membership does not mean an emitter currently reaches Critical: the non-WordPress administrator checks emit High with a stored baseline and Warning without one, and the outbound backdoor-port and bad-ASN variants emit High, so all of them are eligible but contribute nothing until a Critical variant exists.
Eligible producers supply verified identity when available. The test inventory names the producer test for each check, including unresolved branches:
- Database and CMS scanners stamp the owner of the install’s configuration path, resolved through the account roots. An install outside every root keeps its display label but no owner.
- Mail producers (rate windows, mail holds, credential leaks, bulk-service logins, forwarders, filters, mail brute force, cloud relay, geo logins, PHP relay volume) resolve a mailbox or domain to its owning account through the panel’s domain ownership table. Without that table (any panel other than cPanel) the owner stays empty and the row is reported, not counted. A bare account name must resolve to a passwd home directly under an account root. Credential and bulk-service findings use the authenticated identity, never the envelope sender. A sender-domain volume aggregate carries an owner only when every counted arrival proves the same local account through authentication or local submission. Arrivals with a remote ident username after the connecting address stay unattributed: that unquoted text can imitate later authentication metadata. Records whose greeting hides the connecting address also remain unattributed and supply no address. Mixed or unverified aggregates stay unattributed without reducing the volume count. Owner lookups run after tracker locks are released. Mail hold and governor findings require a local mail-server permission decision. Submission identities are read only from reception metadata before the message size; message IDs, subjects, addresses and login names cannot supply, replace or remove an owner or connecting address. Address literals in message IDs and recipients remain message data, and local arrivals do not acquire a peer from later metadata. An optional MAIL AUTH value follows the authenticated identity and does not change that identity or the peer. Subject credential alerts read the arrival’s subject, with Exim’s quote and backslash escapes decoded; escapes for non-printing characters stay as written. Literal tabs remain as logged. Sender addresses, greetings, message IDs and recipients cannot supply that subject. Records with remote ident or optional MAIL AUTH text are skipped for subject alerts because those unquoted values can imitate later fields. Subject text remains sender-controlled and is not proof of authentication. TCP Fast Open connections retain the same peer and ownership checks. The submission boundary follows Exim’s reception log fields and assumes Exim’s default greeting syntax check.
- Process, login and crontab producers accept a system user as owner only
when its home directory sits directly under an account root, so root,
service users and unknown uids never become an account. Direct SMTP findings
apply this validation to both socket users and enriched process accounts.
SSH logins from unknown addresses use this validated account as their tenant
only when the sshd record confirms successful authentication. Root and
service-user logins group by source address after upgrade, and their
findings carry no
tenant_idand no attack database account. The attack database uses only this validated tenant for SSH attribution; certificate text in the log cannot supply an account. Existing incidents and attack records keep their attribution. No setting changes this behavior. - File families (content, phishing, htaccess, file index, core integrity, realtime file events, PHP shield events, self-deleting droppers) carry the judged file’s path, which resolves as described above. The collapsed core-integrity finding has no single path and carries the install owner.
- Socket checks resolve the kernel UID through the same passwd and direct account-home validation. Bad-ASN events also retain that owner when realtime process enrichment misses. Root, service and unknown UIDs stay unattributed. An attributed socket or mail finding includes its owner in the message so different accounts retain separate dispatch and audit identities.
- A sender-domain mail aggregate with mixed or unverified submitters remains a declared attribution gap. The owning account for a mailbox still requires the cPanel domain table. Any unattributed qualifying finding reaches the diagnostic count rather than contributing an invented account.
Correlation policy table
The table below is generated from the check registry: every registered
check with its class, its ignore reason when it is excluded, and its declared
attribution gap. A test compares it with the registry byte for byte and fails
when a row is stale, missing or duplicated; regenerate it with
go test ./internal/checks -run '^TestCorrelationDocumentation$' -args -update-correlation-docs
rather than editing rows by hand. A class says how correlation treats a check
when it fires; it does not say the check currently reaches Critical, that its
owner is available on every panel, or how precise it is.
Regeneration preserves the surrounding prose and marker line endings, even when they use CRLF. The generated block itself uses LF line endings.
Ignore reasons:
account-aggregate: already summarizes several accounts without a single victim identityattacker-side: attacker activity or attempted access, not evidence of compromise of the named victimhost-scope: host-wide condition with no account to attribute; a cross-account count cannot use it even when it is a real compromiseinformational: audit trail or inventory event with no compromise claimperformance: resource usageposture: static configuration, hardening or hygiene state; a Critical means a misconfiguration, not an attack on the accountresponse: record of an automatic action already taken; feeding it back would double countself-health: CSM’s own health, capacity or coverage state
Attribution gaps:
envelope-sender: sender-domain volume aggregate is unattributed when contributing submissions are unverified or belong to different accounts
| Check | Class | Ignore reason | Attribution gap |
|---|---|---|---|
account_scan | ignored | self-health | |
account_scan_error | ignored | self-health | |
account_scan_truncated | ignored | self-health | |
admin_cross_account_overlap | ignored | account-aggregate | |
admin_panel_bruteforce | ignored | attacker-side | |
af_alg_enforcement_corrected | ignored | self-health | |
af_alg_socket_use | security event | ||
antispam_apply_failed | ignored | self-health | |
antispam_controller_open | ignored | self-health | |
antispam_feed | ignored | self-health | |
antispam_file_replaced | ignored | self-health | |
antispam_ip_map | ignored | self-health | |
antispam_learning | ignored | self-health | |
antispam_precondition | ignored | self-health | |
antispam_rspamd_down | ignored | self-health | |
antispam_rspamd_recovered | ignored | self-health | |
antispam_scan_error | ignored | self-health | |
api_auth_failure | ignored | attacker-side | |
api_auth_failure_realtime | ignored | attacker-side | |
api_tokens | ignored | informational | |
auto_block | ignored | response | |
auto_response | ignored | response | |
auto_response_paused | ignored | response | |
auto_response_withheld | ignored | self-health | |
backdoor_binary | malware artifact | ||
backdoor_port | security event | ||
backdoor_port_outbound | security event | ||
bad_asn_outbound | security event | ||
bpf_ringbuf_error | ignored | self-health | |
bpf_unavailable | ignored | self-health | |
bulk_password_change | ignored | account-aggregate | |
c2_connection | security event | ||
cgi_backdoor_realtime | security event | ||
cgi_suspicious_location_realtime | security event | ||
challenge_route | ignored | response | |
check_panic | ignored | self-health | |
check_timeout | ignored | self-health | |
config_reload_error | ignored | self-health | |
config_reload_restart_required | ignored | self-health | |
coordinated_attack | derived | ||
cpanel_file_upload | security event | ||
cpanel_file_upload_realtime | security event | ||
cpanel_login | ignored | informational | |
cpanel_login_realtime | ignored | informational | |
cpanel_multi_ip_login | security event | ||
cpanel_password_purge | ignored | informational | |
cpanel_password_purge_realtime | ignored | informational | |
credential_log_realtime | security event | ||
credential_reuse | ignored | posture | |
credential_stuffing | ignored | attacker-side | |
crond_change | ignored | host-scope | |
crontab_change | ignored | informational | |
cross_account_malware | derived | ||
csm_health | ignored | self-health | |
database_dump | ignored | informational | |
db_content_scan_incomplete | ignored | self-health | |
db_doorway_sitemap_routes | security event | ||
db_hidden_link_injection | security event | ||
db_hostname_keyed_option | security event | ||
db_magic_token_user | security event | ||
db_malicious_event | security event | ||
db_malicious_function | security event | ||
db_malicious_procedure | security event | ||
db_malicious_trigger | security event | ||
db_options_injection | security event | ||
db_options_new_external_script | security event | ||
db_options_plugin_notice_injection | security event | ||
db_phantom_post_author | security event | ||
db_post_injection | security event | ||
db_post_volume_burst | security event | ||
db_rogue_admin | security event | ||
db_siteurl_foreign_host | security event | ||
db_siteurl_hijack | security event | ||
db_siteurl_invalid | ignored | posture | |
db_spam_injection | security event | ||
db_spam_taxonomy | security event | ||
db_stored_cloak_logic | security event | ||
db_stored_code_execution | security event | ||
db_suspicious_admin_email | security event | ||
db_unexpected_event | ignored | informational | |
db_unexpected_function | ignored | informational | |
db_unexpected_procedure | ignored | informational | |
db_unexpected_trigger | ignored | informational | |
detached_web_process | security event | ||
direct_smtp_egress | security event | ||
dns_connection | ignored | host-scope | |
dns_zone_change | ignored | informational | |
docroot_binary_exec | security event | ||
dpkg_integrity | ignored | host-scope | |
drupal_admin_injection | security event | ||
drupal_content_injection | security event | ||
drupal_settings_injection | security event | ||
email_auth_failure_realtime | ignored | attacker-side | |
email_av_degraded | ignored | self-health | |
email_av_encrypted_archive | ignored | self-health | |
email_av_hold_bypass | ignored | self-health | |
email_av_isolation_unavailable | ignored | self-health | |
email_av_late_verdict | ignored | self-health | |
email_av_parse_error | ignored | self-health | |
email_av_parse_worker_failed | ignored | self-health | |
email_av_parse_worker_rejected | ignored | self-health | |
email_av_parse_worker_source | ignored | attacker-side | |
email_av_quarantine_error | ignored | self-health | |
email_av_queue_overflow | ignored | self-health | |
email_av_scan_error | ignored | self-health | |
email_av_scanner_panic | ignored | self-health | |
email_av_source_backlog | ignored | self-health | |
email_av_timeout | ignored | self-health | |
email_bulk_recipients | security event | ||
email_cloud_relay_abuse | security event | ||
email_compromised_account | security event | ||
email_credential_leak | security event | ||
email_defer_fail_governor | ignored | informational | |
email_dkim_failure | ignored | posture | |
email_filter_blackhole | security event | ||
email_filter_exfil | security event | ||
email_filter_forwarder | security event | ||
email_filter_pipe | security event | ||
email_forged_sender | security event | ||
email_mail_filters | ignored | self-health | |
email_mailbox_compromise | security event | ||
email_mailbox_response | ignored | response | |
email_malware | ignored | attacker-side | |
email_outbound_reputation | security event | ||
email_password_audit_incomplete | ignored | self-health | |
email_phishing_content | ignored | attacker-side | |
email_php_form_relay | ignored | attacker-side | |
email_php_form_relay_observed | ignored | attacker-side | |
email_php_form_relay_script | security event | ||
email_php_form_relay_table_full | ignored | self-health | |
email_php_relay_abuse | security event | ||
email_php_relay_account_volume_capped | ignored | self-health | |
email_php_relay_action_dry_run | ignored | response | |
email_php_relay_action_failed | ignored | response | |
email_php_relay_action_skipped | ignored | response | |
email_php_relay_cpanel_limit_unreadable | ignored | self-health | |
email_php_relay_disabled | ignored | self-health | |
email_php_relay_inotify_overflow | ignored | self-health | |
email_php_relay_inotify_overflow_recovered | ignored | self-health | |
email_php_relay_msgindex_persist_failed | ignored | self-health | |
email_php_relay_no_exim | ignored | self-health | |
email_php_relay_overflow_scan_truncated | ignored | self-health | |
email_php_relay_path2b_disabled | ignored | self-health | |
email_php_relay_policies_reload | ignored | self-health | |
email_php_relay_rate_limit_hit | ignored | response | |
email_php_relay_sweep_failed | ignored | self-health | |
email_php_relay_watcher_failed | ignored | self-health | |
email_pipe_forwarder | security event | ||
email_rate_critical | security event | ||
email_rate_warning | security event | ||
email_spam_outbreak | security event | ||
email_spf_rejection | ignored | posture | |
email_suspicious_forwarder | security event | ||
email_suspicious_geo | security event | ||
email_weak_password | ignored | posture | |
executable_in_config_realtime | security event | ||
executable_in_tmp_realtime | security event | ||
exfiltration_paste_site | security event | ||
exim_frozen_realtime | ignored | host-scope | |
fake_kernel_thread | security event | ||
fanotify_kernel_overflow | ignored | self-health | |
fanotify_overflow | ignored | self-health | |
firewall | ignored | host-scope | |
firewall_ipv6_unmanaged | ignored | host-scope | |
firewall_ports | ignored | host-scope | |
firewall_state_unreadable | ignored | self-health | |
ftp_auth_failure_realtime | ignored | attacker-side | |
ftp_bruteforce | ignored | attacker-side | |
ftp_login | ignored | informational | |
ftp_login_after_bruteforce | security event | ||
full_scan_file_too_large | ignored | self-health | |
group_writable_php | ignored | posture | |
htaccess_auto_prepend | security event | ||
htaccess_cgi_handler_abuse | security event | ||
htaccess_errordocument_hijack | security event | ||
htaccess_filesmatch_shield | security event | ||
htaccess_handler_abuse | security event | ||
htaccess_header_injection | security event | ||
htaccess_injection | security event | ||
htaccess_injection_realtime | security event | ||
htaccess_php_in_uploads | security event | ||
htaccess_rewrite_off_shield | security event | ||
htaccess_security_disabled | security event | ||
htaccess_spam_redirect | security event | ||
htaccess_user_agent_cloak | security event | ||
http_asn_crawl | ignored | attacker-side | |
http_claimed_bot_unverified | ignored | attacker-side | |
http_distributed_flood | ignored | attacker-side | |
http_request_flood | ignored | attacker-side | |
http_scanner_profile | ignored | attacker-side | |
http_ua_spoof | ignored | attacker-side | |
infra_ips_unresolvable | ignored | self-health | |
integrity | ignored | host-scope | |
ip_reputation | ignored | attacker-side | |
joomla_admin_injection | security event | ||
joomla_content_injection | security event | ||
joomla_extensions_injection | security event | ||
js_keylogger_dataflow | security event | ||
js_taint_scan_incomplete | ignored | self-health | |
kernel_module | ignored | host-scope | |
local_threat_score | ignored | attacker-side | |
magento_admin_injection | security event | ||
magento_content_injection | security event | ||
magento_settings_injection | security event | ||
mail_account_compromised | security event | ||
mail_account_spray | ignored | attacker-side | |
mail_auth_backend_degraded | ignored | self-health | |
mail_bruteforce | ignored | attacker-side | |
mail_bruteforce_suspected | ignored | attacker-side | |
mail_log_source_unavailable | ignored | self-health | |
mail_per_account | security event | envelope-sender | |
mail_queue | ignored | host-scope | |
mail_queue_unavailable | ignored | self-health | |
mail_spam_source | ignored | attacker-side | |
mail_subnet_spray | ignored | attacker-side | |
malware_companion_files | security event | ||
malware_companion_scan_incomplete | ignored | self-health | |
modsec_block_escalation | ignored | attacker-side | |
modsec_block_realtime | ignored | attacker-side | |
modsec_classifier_gap | ignored | self-health | |
modsec_csm_block_escalation | ignored | attacker-side | |
modsec_disabled_vhost | ignored | posture | |
modsec_low_confidence_burst | ignored | attacker-side | |
modsec_warning_realtime | ignored | attacker-side | |
mysql_superuser | ignored | host-scope | |
new_executable_in_config | malware artifact | ||
new_php_in_languages | security event | ||
new_php_in_sensitive_dir | security event | ||
new_php_in_sensitive_dir_clean | ignored | informational | |
new_php_in_upgrade | security event | ||
new_php_in_uploads | security event | ||
new_php_in_uploads_clean | ignored | informational | |
new_suspicious_php | security event | ||
new_webshell_file | malware artifact | ||
nulled_plugin | ignored | posture | |
obfuscated_php | security event | ||
obfuscated_php_realtime | security event | ||
open_basedir | ignored | posture | |
opencart_admin_injection | security event | ||
opencart_content_injection | security event | ||
opencart_settings_injection | security event | ||
outdated_plugins | ignored | posture | |
pam_auth_failures | ignored | informational | |
pam_bruteforce | ignored | attacker-side | |
pam_login | ignored | informational | |
password_hijack_confirmed | security event | ||
perf_error_logs | ignored | performance | |
perf_load | ignored | performance | |
perf_memory | ignored | performance | |
perf_mysql_config | ignored | performance | |
perf_php_handler | ignored | performance | |
perf_php_processes | ignored | performance | |
perf_redis_config | ignored | performance | |
perf_wp_config | ignored | performance | |
perf_wp_cron | ignored | performance | |
perf_wp_loopback | ignored | performance | |
perf_wp_myisam | ignored | performance | |
perf_wp_transients | ignored | performance | |
phishing_credential_log | security event | ||
phishing_directory | security event | ||
phishing_iframe | security event | ||
phishing_kit_archive | security event | ||
phishing_kit_archive_unscannable | security event | ||
phishing_kit_realtime | security event | ||
phishing_page | security event | ||
phishing_php | security event | ||
phishing_realtime | security event | ||
phishing_redirector | security event | ||
php_config_change | ignored | posture | |
php_config_realtime | ignored | posture | |
php_config_scan_incomplete | ignored | self-health | |
php_dropper_realtime | security event | ||
php_in_image_realtime | security event | ||
php_in_sensitive_dir_realtime | security event | ||
php_in_uploads_realtime | security event | ||
php_remote_taint | security event | ||
php_shield_block | security event | ||
php_shield_eval | security event | ||
php_shield_webshell | security event | ||
php_suspicious_execution | security event | ||
php_taint_scan_incomplete | ignored | self-health | |
placed_dir_incomplete_realtime | ignored | self-health | |
placement_unresolved_realtime | ignored | self-health | |
prestashop_admin_injection | security event | ||
prestashop_content_injection | security event | ||
prestashop_settings_injection | security event | ||
protection_queue_degraded | ignored | self-health | |
protection_queue_recovered | ignored | self-health | |
realtime_scanner_panic | ignored | self-health | |
removed_file_restored_realtime | security event | ||
reputation_quota_exhausted | ignored | self-health | |
response_capacity_exhausted | ignored | self-health | |
root_password_change | ignored | host-scope | |
rpm_integrity | ignored | host-scope | |
self_deleting_dropper_overflow | ignored | self-health | |
self_deleting_dropper_realtime | security event | ||
sensitive_file_modified | ignored | host-scope | |
shadow_change | ignored | host-scope | |
shared_senders_stale | ignored | self-health | |
signature_match_realtime | security event | ||
signature_update_rescan_queued | ignored | self-health | |
signature_update_rollback | ignored | self-health | |
smtp_account_spray | ignored | attacker-side | |
smtp_bruteforce | ignored | attacker-side | |
smtp_probe_abuse | ignored | attacker-side | |
smtp_subnet_spray | ignored | attacker-side | |
spam_evidence_evicted | ignored | self-health | |
ssh_keys | ignored | host-scope | |
ssh_login_unknown_ip | ignored | informational | |
sshd_config_change | ignored | host-scope | |
ssl_cert_issued | ignored | informational | |
stalwart_address_list | ignored | self-health | |
stalwart_ban | ignored | attacker-side | |
stalwart_log_degraded | ignored | self-health | |
stalwart_spam_filter_off | ignored | self-health | |
stalwart_webmail_bruteforce | ignored | attacker-side | |
suid_binary | security event | ||
supply_chain_vuln | ignored | posture | |
suspicious_crontab | security event | ||
suspicious_file | ignored | host-scope | |
suspicious_php_content | security event | ||
suspicious_process | security event | ||
symlink_attack | security event | ||
test_alert | ignored | informational | |
threat_feed_stale | ignored | self-health | |
trusted_proxy_unusable | ignored | self-health | |
uid0_account | ignored | host-scope | |
user_outbound_connection | ignored | informational | |
vulnerable_plugins | ignored | posture | |
vulnerable_timthumb | ignored | posture | |
waf_attack_blocked | ignored | attacker-side | |
waf_bypass | ignored | posture | |
waf_detection_only | ignored | posture | |
waf_rules | ignored | posture | |
waf_rules_stale | ignored | posture | |
waf_status | ignored | posture | |
web_exposed_backup_archive | ignored | posture | |
web_exposed_config_leak | ignored | posture | |
web_exposed_db_dump | ignored | posture | |
web_exposed_phpinfo | ignored | posture | |
web_exposed_repo_metadata | ignored | posture | |
web_exposed_sample_sql | ignored | posture | |
web_exposed_source_backup | ignored | posture | |
webmail_bruteforce | ignored | attacker-side | |
webmail_login_realtime | ignored | informational | |
webshell | malware artifact | ||
webshell_caller | ignored | attacker-side | |
webshell_content_realtime | security event | ||
webshell_path_public | ignored | informational | |
webshell_realtime | security event | ||
whm_account_action | ignored | informational | |
whm_login_realtime | ignored | informational | |
whm_password_change | ignored | informational | |
whm_password_change_noninfra | security event | ||
whm_unauth_scripts_realtime | ignored | attacker-side | |
world_writable_php | ignored | posture | |
wp_core_integrity | security event | ||
wp_core_unverified | ignored | self-health | |
wp_login_bruteforce | ignored | attacker-side | |
wp_plugin_inventory_unverified | ignored | self-health | |
wp_user_enumeration | ignored | attacker-side | |
xmlrpc_abuse | ignored | attacker-side | |
yara_backend_unavailable | ignored | self-health | |
yara_forge_rollback | ignored | self-health | |
yara_match_realtime | security event | ||
yara_match_scheduled | security event | ||
yara_realtime_scan_error | ignored | self-health | |
yara_scan_incomplete | ignored | self-health | |
yara_worker_crashed | ignored | self-health |
Findings from retired checks
A check name that no version of CSM emits any more stays registered while
older installations can still hold findings under it, because a finding is
only ever cleared when its name appears in the owning runner’s purge list.
Two file-index names, new_php_in_languages and new_php_in_upgrade, are
in that state: findings written by releases before the content-first file
index are cleared by the next completed file_index scan, are kept while
that scan is incomplete, and are kept per file while the scan reports a
coverage gap for that file. Nothing emits them again. php_dropper was
never emitted by any release, is not registered, and no response table lists
it: the manual, automatic and full-scan quarantine sets and the attack
database mapping are each declared once and tested against the registry, so a
renamed or never-emitted name cannot sit inert in a response table. The same
guard removed modsec_block and waf_block from the attack database mapping;
neither was ever emitted, so WAF blocks have never fed local reputation
scoring through that database. Whether the emitted ModSecurity block names
should is an open scoring decision, not something the guard decides.
Directory enumeration and PHP handler-configuration read errors make the file-index scan incomplete, including when only one account root is unreadable. The scanner keeps its previous index and directory cache, preserves its active findings, and still reports new findings from readable directories. A later completed scan clears the retired names; absent optional directories do not prevent completion. The first scan after startup and every retry after an incomplete or interrupted walk enumerate directories again, even if their cached modification times still match.
API
GET /api/v1/incidents– list, newest first. Without query parameters the response is a bare JSON array (compat with the existing wire shape phpanel/SIEM consumers decode against). When?limit=,?offset=, or?status=is present the response switches to an envelope:{"items":[...], "total":N, "offset":N, "limit":N, "status":"..."}. Status accepts the four spec values plusactive(open + contained, the default web UI filter). Limit is capped server-side at a safe maximum.GET /api/v1/incidents/<id>– one incident.POST /api/v1/incidents/<id>/status– transition status.
See api.md for endpoint detail.
Web UI
Open Monitor -> Incidents. The page has three tabs:
- Correlated – the default flat list of incidents with status filter, page size, and detail panel. The detail panel shows the current firewall block state for the incident’s source IP (permanent, temporary, cphulk, or not blocked) when an IP is known.
- Grouped – rolls up incidents by
(kind, source)so a credential spray that produced thousands of mailbox_bruteforce incidents shows as one row per attacker IP. Pageable with the same page-size selector as Correlated. Click a group to see member incidents in the detail panel, which also surfaces the source IP’s firewall block state; clicking a sample id jumps back to the Correlated tab focused on that incident. - Timeline Search – the older IP/account history search across the audit log.
Admin tokens can transition incident status (open / contained / resolved / dismissed); read-scope tokens can browse all three tabs.
Control socket
csm incidents list [--status all|active|open|contained|resolved|dismissed] [--limit N] [--offset N] [--all]
csm incidents show <id>
csm incidents status <id> <open|contained|resolved|dismissed> [details]
csm incidents bulk-status --older-than 24h [--last-seen-before RFC3339] [--status active|open|contained] [--kind K] [--domain D] [--account A] [--mailbox M] [--limit N] [--to resolved|dismissed] [--apply --confirm]
csm incidents list returns the first 100 incidents by default. Use
--offset for the next page, --status active for open + contained
incidents, or --all for an explicit full dump.
csm incidents bulk-status defaults to dry-run. It prints the total
match count and a bounded preview of the incidents that would change.
At least one age guard is required: --older-than, --last-seen-before,
or both. To mutate incidents, pass both --apply and --confirm.
Metrics
csm_incidents_open– gauge of currently open + contained incidents.csm_incidents_created_totalcsm_incidents_severity_changed_totalcsm_incidents_status_changed_totalcsm_incidents_findings_merged_totalcsm_incidents_compacted_totalcsm_incidents_pending– gauge of findings held in the threshold gate, awaiting a second correlated sighting.
Incident Response Runbook
Use this flow when CSM flags account compromise, mailbox takeover, malicious database triggers, or outbound spam on a production cPanel host.
Safety rules
- Do not delete customer files during first response.
- Do not thaw, release, or purge queued mail until affected credentials are rotated or an operator approves the specific queue action.
- Do not close incidents until the account was reviewed, credentials were rotated or explicitly deferred, and a fresh scan is clean.
- Take a CSM backup before upgrading CSM or changing incident state.
1. Verify the deployed binary
Deploy only after the required GitLab pipeline passed and the package was published.
/root/deploy-csm.sh check
/root/deploy-csm.sh upgrade
/opt/csm/csm version
/opt/csm/csm doctor --json
2. Take a backup
mkdir -p /root/csm-backups
systemctl stop csm.service
/opt/csm/csm backup /root/csm-backups/csm-pre-response-$(date +%Y%m%d%H%M%S).tar.gz
systemctl start csm.service
/opt/csm/csm doctor --json
sha256sum /root/csm-backups/csm-pre-response-*.tar.gz
Backup refuses a live or starting daemon and verifies the state lock before reading data. Restart the service immediately after the archive is created. If backup fails, start the service before investigating the failure.
Confirm the archive is readable:
gzip -t /root/csm-backups/csm-pre-response-*.tar.gz
tar -tzf /root/csm-backups/csm-pre-response-*.tar.gz | sed -n '1,80p'
3. Preserve evidence
mkdir -p /root/csm-forensics
/opt/csm/csm forensic-snapshot <account> --out /root/csm-forensics/<account>-$(date +%Y%m%d%H%M%S).tar.gz
sha256sum -c /root/csm-forensics/<account>-*.sha256
tar -xOzf /root/csm-forensics/<account>-*.tar.gz manifest.txt
Check the manifest for private-path exclusions, schema count, capture
errors, and recent_mtimes_status=ok.
4. Map affected accounts
Map incident domains and queued local senders to cPanel users before rotating credentials or changing mail queue state.
/opt/csm/csm incidents list --status open --all
exim -bpc
exim -bp | exiqsumm
grep -E '^example.com:' /etc/trueuserdomains /etc/userdomains
whmapi1 listaccts searchtype=user search=<account> --output=json
Use native cPanel APIs for inventory:
uapi --user=<account> Email list_pops --output=json
uapi --user=<account> Ftp list_ftp --output=json
uapi --user=<account> Mysql list_users --output=json
5. Rotate credentials
Rotate the cPanel account password, FTP accounts, affected mailboxes, WordPress administrator users, database users, and application secrets for the affected account. Prefer WHM and UAPI calls or the control panel over direct file edits.
Do this before releasing mail or marking incidents resolved unless the operator explicitly defers rotation for a documented reason.
6. Review queued mail
Start with read-only summaries:
exim -bpc
exim -bp | exiqsumm
exim -bp
Review headers before any queue action:
exim -Mvh <message-id>
Group messages into:
- safe to remove: frozen bounces, obvious backscatter, duplicate failed delivery notices with no customer value
- do not touch: current customer conversations, invoices, form leads, or any message where the business value is unclear
- needs review: suspicious local sender messages, mixed external bulk mail, or messages tied to an account whose credentials are not rotated
Only remove or thaw message IDs that were reviewed:
exim -Mrm <message-id>
exim -Mt <message-id>
7. Review stale incidents
Preview first:
/opt/csm/csm incidents bulk-status --older-than 72h --status active --kind web_account_compromise --limit 20
/opt/csm/csm incidents bulk-status --older-than 24h --status active --kind mailbox_takeover --limit 20
Apply in bounded batches only after review:
/opt/csm/csm incidents bulk-status --older-than 72h --status active --kind web_account_compromise --limit 100 --apply --confirm --details "operator cleanup after review"
For mailbox incidents, confirm mailbox rotation or explicit operator deferral before applying status changes.
8. Confirm recovery
/opt/csm/csm status --json
/opt/csm/csm doctor --json
exim -bpc
Keep the forensic archives, CSM backup, command notes, and queue decisions with the incident record.
Direct SMTP egress
CSM watches the local mail stack via spool + log scanning. Non-MTA processes that open outbound SMTP connections directly bypass that path. The direct SMTP egress detector catches that at connect time and feeds the incident correlator.
What fires
A finding with check: "direct_smtp_egress" is emitted when:
- A non-root process opens an outbound TCP connection.
- Destination port is one of the configured SMTP ports (default 25, 465, 587).
- Destination IP is not loopback, infra, or in the operator’s
infra_ipslist. - The process user is NOT a known MTA user (mail, mailnull, postfix, dovecot, dovenull, mailman, plus exim on cPanel). On non-cPanel hosts, Stalwart’s stalwart and stalwart-mail users are also recognized when their UID is below 1000. Hosted UIDs do not receive this automatic exemption, even when they use a Stalwart username.
- The process user is NOT listed under
firewall.smtp_allow_users. Those users may send throughfirewall.smtp_block, so their connections are expected. List any other mail server’s service user there.
Process names are never a standalone allow condition. A hosted account
renaming malware to smtp or smtpd still emits a finding.
The detector always emits findings when enabled. The dry_run knob does not suppress findings; it participates in the BPF enforcement gate, where any dry_run=true layer keeps kernel denial in observe-only mode.
Configuration
detection:
direct_smtp_egress:
enabled: true
backend: auto # auto / bpf / legacy / none
dry_run: true # safety default for detector-scoped action
ports: # each value must be 1-65535
- 25
- 465
- 587
Backends
auto– allow both BPF and legacy scan paths. Live backend choice still followsdetection.connection_tracker_backend.bpf– emit only from the cgroup/connect4,6 consumer.legacy– emit only from the/proc/net/tcp[6]polling path (live poller or scheduled critical scan). This path lacks PID/comm; MTA matching uses the username and UID.none– detector disabled even whenenabled: trueis set elsewhere; useful for staged rollout.
The generic outbound connection tracker is still governed by
detection.connection_tracker_backend; this setting only gates
direct_smtp_egress findings.
Metric
csm_direct_smtp_egress_findings_total – monotonic counter,
incremented per finding emitted by the BPF connection consumer. The
legacy poller does not bump this counter today; operators who run
backend=legacy should track findings via the audit log.
rDNS enrichment
When the BPF backend is active, finding details include a Domain field populated from a TTL-cached reverse lookup (30 min TTL, 1 second per-lookup deadline). The lookup runs only after the cheap direct-SMTP filters match. On resolver miss or timeout the field is omitted; the finding still fires.
Caveats
2525is intentionally NOT in the default port list. Many operators run unrelated services on it. Add it toportsif your infra uses it for submission.- The detector emits regardless of the dry_run knob. Kernel denial
requires
auto_response.dry_run, this dry_run key, andbpf_enforcement.dry_runto all be explicitly false.
BPF cgroup-deny enforcement
Optional in-kernel denial of outbound connections that match an enabled userspace detector. Direct SMTP egress is currently the only enforcement gate. Every layer starts in dry-run so operators can review telemetry before allowing live denial.
What it does
When bpf_enforcement.enabled=true, direct_smtp_egress=true, the
connection tracker is running on BPF, and all dry-run layers are false:
- The cgroup/connect4 + cgroup/connect6 BPF program inspects each outbound TCP connect.
- If destination port is in the protected set AND the source UID is not in the safe-UID map AND the gated detector matches, the program returns 0 (kernel denies the connect).
- Userspace observes the decision via the
decisionfield on the ringbuf event and emits an audit-log entry.
When any dry-run layer is true (the default), the program emits the decision but always returns 1 (allow). Operators can run dry-run for as long as they need to gather telemetry before flipping to live denial.
The safe-UID map includes system UIDs below 1000, traditional MTA
accounts, and accounts listed in firewall.smtp_allow_users. Stalwart
usernames are exempt only as system UIDs; a hosted account cannot gain
an exemption by choosing a Stalwart name. CSM rereads /etc/passwd every
5 minutes and drops the exemption of hosted accounts that were deleted or
renamed, so a new account reusing such a UID is exempt for at most one
refresh.
What it does NOT do
- It does NOT wait on remote verdict callbacks in-kernel. That would add HTTP latency to every connect. The verdict callback (if enabled) runs in userspace after the BPF decision and enriches the emitted finding; it cannot undo a kernel denial.
- It does NOT enforce on UDP, ICMP, or non-cgroup paths.
- It does NOT replace detection. Findings still emit regardless; enforcement is a separate, layered control.
Configuration
bpf_enforcement:
enabled: false # master switch; default off
dry_run: true # safety default; flip after telemetry review
direct_smtp_egress: false # gate enforcement on the direct SMTP detector
verdict_callback: false # userspace post-decision callback
bpf_enforcement.enabled=true requires at least one feature gate.
Today the only gate is direct_smtp_egress, which itself requires
detection.direct_smtp_egress.enabled=true. The connection tracker
backend must be auto or bpf, and the direct SMTP backend must be
auto or bpf.
Kernel requirements
- Linux >= 4.10 with
CONFIG_CGROUP_BPF=y. cgroup/connect4andcgroup/connect6BPF program types.- The capability surface
bpf_enforcement.available.v1is the wire signal that the binary supports the feature; combined withbpf_enforcement_activeon the health snapshot, operators can detect both feature presence and runtime state.
On older kernels or default builds without the BPF tag,
detection.connection_tracker_backend: auto falls back to the legacy
/proc/net/tcp[6] poller. In that state direct SMTP findings still
work when detection.direct_smtp_egress.backend is auto or
legacy, but BPF enforcement is inactive.
When CSM attempts BPF and cannot start it, it emits a
bpf_unavailable finding. The message reports whether the daemon is
running on a fallback backend or has no live fallback active.
Metrics
csm_bpf_enforcement_decisions_total{decision="allow|dry_run|deny"}csm_bpf_enforcement_uid_map_refresh_total– successful periodic refreshes of the safe-UID BPF map.csm_bpf_enforcement_uid_map_refresh_failures_total– failed refreshes (e.g. /etc/passwd unreadable).
Dry-run precedence
Three independent dry_run knobs interact:
auto_response.dry_run: keeps automatic network response in observe-only mode.detection.direct_smtp_egress.dry_run: detector-scoped action knob.bpf_enforcement.dry_run: kernel-side denial knob.
Rule: any dry_run=true wins. Live denial requires all three to be false at the layer they apply, plus a BPF runtime backend. Defaults are dry_run=true everywhere on first install.
Rollout recipe
- Enable the direct SMTP detector without BPF enforcement. Watch
csm_direct_smtp_egress_findings_totalfor a week. - Enable BPF enforcement with
dry_run: true. Watchcsm_bpf_enforcement_decisions_total{decision="dry_run"}and confirm dry-run denials track expected hosted-account egress. - Set BPF
dry_run: falseon a single canary host. Audit incidents for false positives. - Roll out to fleet.
CVE Mitigations
CSM treats CVEs as a three-layer problem:
- Operator-driven hardening via
csm harden ...- applies the right preventive control for the host (modprobe blacklist, seccomp drop-ins, sysctl tweaks). - Continuous enforcement by the daemon - re-asserts the control on every startup and as a periodic check, so a package upgrade or manual
modprobedoes not silently undo the mitigation. - Live detection - auditd/BPF listeners flag exploit signatures the moment they fire, even on hosts where the kernel itself cannot be patched.
The hardening audit detects what the host actually has (kernel build, KernelCare livepatches, seccomp coverage) and refuses to claim protection it cannot deliver. Run csm harden with no arguments for the full list of available mitigations on the current host.
Active mitigations
CVE-2026-31431 - “Copy Fail” (Linux kernel AF_ALG)
Two operator paths depending on whether AF_ALG is loadable on the kernel:
csm harden --copy-fail- writes/etc/modprobe.d/csm-copy-fail-mitigation.confblacklistingalgif_aeadandaf_alg, then unloads them. Refuses on kernels where AF_ALG is built in (typical cPanel / CloudLinux 8), since the blacklist would have no effect there.csm harden --copy-fail-seccomp- writes systemdRestrictAddressFamilies=~AF_ALGdrop-ins for the units that spawn untrusted code: LiteSpeed, Apache/Nginx, every PHP-FPM pool, cron, mail. The right path on built-in-AF_ALG kernels.
The audit recognises KernelCare CVE-2026-31431 livepatches via kcarectl --patch-info and reports pass only when the host is genuinely mitigated (module blacklisted, seccomp drop-ins applied, or KernelCare livepatch active).
Live detection and BPF blocking
BPF-tagged builds (make BPF=1 or go build -tags bpf) load an LSM socket_create program on kernels with BPF LSM and ringbuf support. That program refuses socket(AF_ALG, ...) from non-root UIDs before the AF_ALG socket is allocated, returns EPERM, emits a ringbuf event, and feeds the existing Critical af_alg_socket_use finding path. UID 0 keeps AF_ALG access for system crypto users. There are no UID-range or alert-only BPF tunables today; use detection.af_alg_backend to select auditd or none if a host needs to avoid kernel-side refusal.
Default builds, BPF-tagged builds on unsupported kernels, and hosts forced to detection.af_alg_backend: auditd keep the audit-log listener. The audit path catches non-system AF_ALG socket attempts at Critical within about 500 ms but cannot stop the syscall before it reaches the kernel. Hosts that are not exploitable skip the live listener.
If CSM attempts the AF_ALG BPF path and cannot start it, it emits a
bpf_unavailable finding. The finding says whether the audit fallback
is active or no live fallback is available.
Auto-response
auto_response.copy_fail_kill_process: true- SIGKILL the offending process when the live listener fires. Default off (alert-only).auto_response.disable_enforce_af_alg: true- suspend the periodic re-assertion of the module blacklist without removing the hardening marker. For triage only.
The daemon self-heals its auditd rule file on startup if it has drifted from the embedded copy, closing the upgrade gap where a new binary shipped without re-running postinstall would leave detection silently inactive.
Configuration knob
detection.af_alg_backend-auto(default) |bpf|auditd|none.auditdis the kill switch for a misbehaving BPF-tagged release.bpfis strict mode (no fallback).nonedisables the live listener entirely.
The csm_af_alg_backend{kind="bpf-lsm"|"auditd-tail"|"none"} Prometheus gauge surfaces which backend the coordinator selected at startup, so dashboards can see the active path without parsing logs.
BPF validation
On a BPF LSM host with a BPF-tagged CSM build:
-
Set
detection.af_alg_backend: bpffor strict validation, or leave it asautoand confirm BPF was selected. -
Start the daemon and check metrics:
curl -k -H "Authorization: Bearer $CSM_TOKEN" https://127.0.0.1:9443/metrics \ | grep -E 'csm_af_alg_backend|csm_bpf_backend'Expected selected series:
csm_af_alg_backend{kind="bpf-lsm"} 1 csm_bpf_backend{feature="af_alg",kind="bpf"} 1 -
As a non-root user, attempt an AF_ALG socket:
sudo -u nobody python3 - <<'PY' import errno import socket import sys try: socket.socket(socket.AF_ALG, socket.SOCK_SEQPACKET, 0) except OSError as exc: print(exc.errno) sys.exit(0 if exc.errno == errno.EPERM else 1) raise SystemExit("AF_ALG socket unexpectedly succeeded") PY -
Confirm the command prints
1(EPERM) and CSM emits a Criticalaf_alg_socket_usefinding. The finding details should say the call was refused by the BPF LSM program.
CVE-2026-41940 - cPanel/WHM auth-bypass
Detection in the access-log path:
- Non-infra WHM login attempts surface at Warning (suppressible alongside other cPanel-login alerts) so an operator can see brute force traffic against the bypass surface.
- The tokenless WHM-script request the published exploit uses for cache promotion surfaces at Critical, always-on, and feeds auto-block.
No operator hardening command is required. The host fix is to apply the cPanel patched build. CSM provides the detection layer for windows where patching has not yet rolled out.
CVE-2026-109618 and CVE-2026-109619 - “PHP2Shell” (PHP core)
Two critical flaws in PHP itself, reachable over the network without a login (advisories GHSA-w8hw-w9fx-j79w and GHSA-6wmf-4rv5-p3rp). They are fixed in PHP 8.2.35, 8.3.36, 8.4.27 and 8.5.12, and updating is the only complete fix.
- CVE-2026-109618 - when memory runs out at the wrong moment, PHP leaves request memory in an invalid state that request shutdown then uses. Crafted multipart bodies can force that during request startup, before any script runs. Apache mod_php is exposed without any application involvement; PHP-FPM and LiteSpeed LSAPI need an application-specific path.
- CVE-2026-109619 - a fatal error or memory exhaustion can leave worker state pointing at freed request memory, and some operations in the next request use it before normal initialization. Session upload progress is one of them. It runs on any multipart POST carrying the upload-progress field, whether or not the site uses sessions.
CSM provides a request filter and configuration audits until patched PHP is installed:
- ModSecurity filter - rule
900133denies requests other than GET and HEAD whose form field names contain the defaultPHP_SESSION_UPLOAD_PROGRESSfield name, including the single-quoted spelling PHP accepts. Requests carrying browserOriginorSec-Fetch-Siteheaders are denied by900134instead, whose firewall escalation is off by default. Keep that exclusion to prevent browser-steered bans. Requests carrying neither header retain first-deny escalation, including a browser whose extension or proxy stripsOriginwhen posting to a plain-HTTP site. A customsession.upload_progress.nameis outside this filter’s coverage. It needsSecRequestBodyAccess On, and it sees only what ModSecurity’s request body limits let it inspect, so the PHP setting below is the stronger stopgap. - Hardening audit -
php_upload_progress_<version>warns whilesession.upload_progress.enabledis on or cannot be confirmed off. For cPanel EA4 PHP the audit readsphp.iniand thenphp.din PHP’s own order, so a drop-in that turns the setting off is recognised. Conditional site settings and FPM pools are checked separately; disabling one does not hide another that is on. Unreadable overrides and unresolved values retain the warning. On bare servers, the audit follows the additional files reported by the system PHP CLI in their load order. A pass covers only inspected settings, not running workers, per-user CageFS copies,.user.inifiles, or a separate web SAPI configuration. Confirm the effective value through each site’s web handler after restarting. Thephp_version_<version>item marks a branch end of life the day after its upstream security support ends. - PHP 8.5 backtraces -
php_fatal_error_backtraces_<version>warns on PHP 8.5 and later whilefatal_error_backtracesis on or cannot be confirmed off, using the same scope and pool rules. Scripts can still turn it on at runtime, so a pass covers the configured default only.
Both configuration audits recognise quoted FPM setting names. Within each pool, admin values override ordinary values; the first duplicate at the same level wins, matching PHP-FPM’s application order. Remove conflicting pool entries instead of relying on a later value to override an earlier one. The backtrace audit does not determine the installed patch level and still checks PHP 8.5 and later after an update.
To turn the handler off on cPanel EA4, add
session.upload_progress.enabled = Off to a file in each
/opt/cpanel/ea-phpXX/root/etc/php.d/ directory. For CloudLinux alt-php, add
the same line to /etc/cl.selector/global_php.ini and run
selectorctl --apply-global-php-ini. With CageFS, PHP inside each cage reads its
own copy of the EA4 configuration directory, so run cagefsctl --force-update
after adding the file. Restart PHP workers afterwards and confirm the value from
inside a cage (cagefs_enter_user <user> <php binary> -r '...') or from a web
request, not only from the host CLI.
On PHP 8.5, also set fatal_error_backtraces = Off the same way; the advisory
names it as the mitigation for its strongest confirmed case. The advisory’s setting for
CVE-2026-109618, memory_limit = -1 behind a separate memory cap, is partial.
Do not cap PHP with an address-space limit such as RLIMIT_AS or a CloudLinux
VMEM limit: failed allocations reach the vulnerable path.
None of these replaces the PHP update. PHP published source backports for 7.4, 8.0 and 8.1 but no releases, and nothing for older branches. Vendors such as CloudLinux may ship fixes in their builds, so check the package changelog for the CVE IDs.
NGINX2Root - nginx worker to root through kernel SCTP
NGINX2Root is five nginx flaws (CVE-2026-123007, CVE-2026-123633, CVE-2026-123634, CVE-2026-121810, CVE-2026-113425) that run code as the nginx worker. Upstream fixed them in 1.30.7 and 1.31.9; see the nginx security advisories. Update and restart nginx, since a reload keeps the old binary. CSM does not check nginx versions: distributions backport the fix without changing them.
The published chain reaches root through a separate Linux kernel SCTP bug.
Any local foothold can use it, a PHP webshell included, once the sctp module
loads, and an unprivileged user can get it loaded on demand. Web hosts do not
use SCTP, so the hardening check os_sctp_blocked asks kmod for the module’s
effective load plan and passes only when:
- the kernel has no SCTP support, or
- the module is unloaded, modprobe reports it missing, and CSM confirms its
absence in the running kernel’s module tree (EL8 ships it in
kernel-modules-extra), or - the module is not loaded and kmod would run a no-op install command, or
- the module is not loaded and the kernel has module loading disabled.
blacklist sctp is not enough: another module can still load SCTP as a
dependency. Block it with an install directive instead:
echo 'install sctp /bin/false' > /etc/modprobe.d/csm-disable-sctp.conf
modprobe --show-depends sctp # must print "install /bin/false"
If SCTP is already loaded, stop what uses it and run modprobe -r sctp, or
reboot. A kernel with SCTP built in fails until it is replaced. The check
warns when it cannot read the kernel config, /proc/modules, the modprobe
configuration, or the load plan. A missing-module report also warns if CSM
cannot inspect the module tree and its index, or finds an SCTP module there.
Future CVEs
New mitigations land here as they ship. The bar is:
- The host can be measurably hardened (modprobe / seccomp / sysctl / config), and/or
- An exploit-signature detector can fire reliably without false positives.
CVEs that are purely “patch the package”, with no preventive control we can apply and no signature we can detect, do not get a CSM mitigation; the right answer is the vendor patch. The daemon’s package-integrity check (rpm -V / debsums) only shows that installed files still match their package; it cannot tell whether that package contains a fix.
Firewall (nftables)
CSM includes a native nftables firewall engine that replaces LFD and fail2ban. It writes rules through the kernel netlink API directly via google/nftables. Integrity monitoring reads the installed structure with the nft command.
Features
- Atomic ruleset - single netlink transaction, no partial application
- Named IP sets with per-element timeouts (blocked, allowed, infra, country)
- IPv4 options drop - inbound IPv4 TCP packets with IP options are dropped before conntrack, NAT to containers and every accept, closing the CVE-2026-98432 trigger on unpatched kernels; set
drop_ipv4_options: falseonly for workloads that need IP options, such as CIPSO labels - Rate limiting - SYN flood, UDP flood, and per-IP connection rate are dual-stack (IPv6 metered per /64); per-port flood meters are dual-stack per source address; the concurrent connection limit is IPv4-only
- Country blocking via MaxMind GeoIP CIDR ranges
- Outbound SMTP restriction by UID (prevent spam from compromised accounts)
- Local port locks - only listed users and root may open new connections to a port on this host
- Subnet/CIDR blocking with auto-escalation from individual IPs and safety guards for infra, local, and allowed addresses
- Permanent block escalation after repeated temp blocks
- Dynamic DNS hostname resolution (updated every 5 min) with grace-period guard against transient resolver failures
- IPv6 dual-stack with separate sets
- Commit-confirmed safety - Juniper-style auto-rollback timer
- Infra IP protection - refuses to block infrastructure IPs
- Auto-response dry-run - safety default that forecasts eligible IP blocks and retains bounded examples without applying them
- Verdict callback - optional advisory hook to the panel before each auto-block (allow / block / attach metadata)
- cphulk integration - unblock flushes cphulk too
- Mail spray history - unblock, allow, undoing a block and flush also clear the released addresses’ mail login spray history and failure counts
- Audit trail - JSONL log with 10MB rotation
- State persistence with atomic writes
Storage contract preparation
The durable firewall action service is implemented and tested through engine injection. Production activation, reader cutover, migration, restore, downgrade and operator recovery interfaces remain open.
Firewall actions record the actor, source, linked finding or incident, and complete before and after state before changing the kernel. Pending intent is separate from committed state. Recovery verifies target identity and expiry before recording an outcome; it does not blindly replay a mutation. Repeated request IDs reuse the original action and admission accounting. Audit delivery retries use the same action ID. Typed undo checks that the affected targets still match the recorded result.
Mail login spray blocks keep their eviction priority through restart, durable batches, and undo. Undo restores the original expiry. An operator replacement does not inherit the spray classification. Undoing an unblock restores the block, not the spray history the unblock cleared.
Proven outcomes are retained for undo and review, bounded two ways. The retention sweep drops delivered outcomes older than the findings-history setting, and a hard cap on retained outcomes and on their total size applies even when sweeps stay off. The size cap includes retained audit evidence. Existing journals are indexed on their first retention operation. Pending actions and outcomes whose audit has not been delivered are never dropped. The newest outcome also survives the hard cap even if it alone exceeds it. Hourly scan counters keep only the newest windows. Scan admission refuses pruned windows after a backward clock correction, preserving budget safety. Deleting an outcome ends the undo window for that action.
When an action cannot be proven, for example because the kernel did not answer
in time, its outcome stays uncertain and the engine refuses further firewall
changes rather than guess. The daemon retries recovery at startup and on its
maintenance tick, which settles the outcome as soon as the kernel can answer.
csm firewall actions shows what is waiting and why. If the kernel can never
prove it, an operator inspects the host and records the answer with
csm firewall actions resolve, which takes applied or rejected and an
optional note. Kernel evidence wins over the operator: if recovery can prove
the outcome at that moment, the proven one is recorded and the operator’s
answer is not used. The decision and its note reach the action log.
Startup recovery runs before applying the firewall. If startup fails, pending actions remain available for automatic recovery and operator resolution. After resolving them, restart the daemon to retry firewall setup. A storage error while saving kernel evidence stops resolution; it never permits an operator assertion to replace that evidence. Failure to refresh committed state is also reported, even when the outcome was saved.
Applying an action writes only the entries that change, so the kernel write for one block does not grow with the size of the blocked set. Large removals use bounded messages within the same atomic update. Ranged sets are written whole because their start and end markers move together. If the kernel already expired an entry the action meant to remove, the set is rewritten instead, and both paths end at the same state.
Firewall state storage provides complete snapshot reads and revision-checked replacement using the existing database. Reads preserve expired entries, original timestamps, explicit provenance, duplicate allow entries and collection order. An uninitialized store, corrupt data or a failed read returns an error without usable state. Rejected or rolled-back writes preserve the previous snapshot and revision. An error after the transaction body succeeds reports an uncertain commit; callers must reconcile stored state before retrying and must not assume rollback or confirmed durability. Legacy bucket edits invalidate a committed snapshot instead of silently changing its revision. The contract does not activate runtime cutover.
Firewall storage metrics separate write wait from transaction duration and expose read time, pending writes, failures and snapshot batch size using fixed labels. Repeatable benchmarks exercise competing writers and local snapshot copying. Backup restore and downgrade compatibility require the later migration stage.
Mutation failures
Single-address blocks refuse loopback, unspecified, link-local, multicast and
the limited broadcast address. The kernel drops multicast and limited-broadcast
sources before the input chain. Subnet blocks refuse ranges overlapping
loopback, link-local or multicast ranges, infrastructure, host interface
addresses, or full-IP and port-specific allows, plus unspecified or broadcast
individual addresses and default routes. Other ranges containing those addresses,
such as 0.0.0.0/8 or 240.0.0.0/4, remain blockable. All these safety refusals
are recorded as refused in the action log, like single-address refusals. Invalid
targets and storage or kernel errors remain failures. Refusing a permanent
promotion leaves the prior temporary block and its expiry unchanged.
Protected targets are refused before timed-block lifetime checks. Subnet dry-runs validate the target before reporting a proposed block. Manual requests also record safety refusals when durable action tracking is enabled. Legacy entries for protected addresses survive reloads and can still be unblocked.
WAF attacker reports for link-local addresses stay visible but advise investigating the traffic source instead of a block CSM would refuse.
Integrity checks compare live rule structure with the snapshot captured after
CSM last applied the firewall. Editing or rehashing configuration alone never
approves a changed ruleset. A failed snapshot capture reports a monitoring gap;
after correcting nft availability, re-apply the firewall to restore its
baseline. Dynamic set membership does not affect this comparison.
Firewall changes persist their intent before changing kernel rules. A failed atomic kernel transaction restores the previous state. Failed writes and rollbacks return errors without success audit records; expiry cleanup logs the failure and retries on a later pass. DNS refreshes also retry failed removals. Removing one allow source preserves any other active source for that address.
An unconfirmed-durability error means the replacement is visible on disk but
storage did not confirm its survival across power loss. For kernel changes,
CSM attempts to restore the previous state before returning the error. A
failed rollback is reported as well. Correct the storage problem, inspect the
saved state and live rules, and repeat the intended operation or run
csm firewall restart to apply the saved state. Do not treat an error as a
successful rule change.
Port-specific allow additions and removals update saved state only. Run
csm firewall restart to apply them to the kernel; the command acknowledgement
states that a reload is required.
Clearing a stale local threat score
Run csm firewall forget <ip> as root against the running daemon to clear one
address’s accumulated local threat score. It accepts exactly one IPv4 or IPv6
address, including equivalent IPv6 spellings; CIDRs and extra arguments are
rejected. There is no dry-run option.
The command clears equivalent stored spellings together and reports their highest score and total event count. Forgetting an IPv6 address also forgets the spam evidence of its /64 network, and the reply names that network. Blocks, allow lists, whitelists and raw event history remain intact. New findings immediately start a fresh scoring record, even if they arrive while the command runs. This does not suppress findings about the host’s own address.
Persistence is attempted immediately but remains best effort: a success reply confirms removal from memory, not durability across a restart. Check the daemon logs for attack database write failures if an old score returns after restart.
Attack statistics and event queries use the daemon’s state database or its configured attack database directory. If neither is available, they do not read event files from the working directory.
Stalwart bans
On a host whose mail server is Stalwart, Stalwart keeps its own list of banned
addresses. When one address is unblocked, allowed or undone in CSM, unbanned
from a hosting panel, or allowed through a hostname in dyndns_hosts or
infra_ips, CSM also lifts Stalwart’s automatic bans of it through the
Stalwart API key from csm stalwart setup. An allow through a hostname counts
the same as an operator allow. This
runs in the background, so the unblock never waits on Stalwart. Only the bans
Stalwart makes on its own, for failed logins, failed recipients, loitering and
port scans, are lifted. Blocks made by hand in Stalwart and bans with any other
reason stay, and the daemon log says so. A flush and a subnet removal leave
Stalwart’s list alone. Each lifted ban is written to the
action log as operate.clear_stalwart_ban.
A lift that fails is written to the daemon log, which names the address Stalwart may still refuse, and to the action log when it got as far as the delete. Every unblock ends with Stalwart reloading its ban list, also when no ban was left to lift. So unblocking the address again retries a ban that Stalwart still lists, and finishes a lift whose reload did not happen. Until then Stalwart can keep enforcing a ban it no longer lists.
Startup failures
Overlapping, nested, duplicate, and adjacent ranges are merged for the kernel, including IPv4 and IPv6 ranges ending at the last address. This applies to infrastructure, country, Cloudflare, DoS exemption, and blocked subnet sets. Stored subnet entries keep their own source and expiry; removing or expiring one entry rebuilds the remaining coverage in an atomic transaction. Default routes remain forbidden as subnet blocks to prevent operator lockout.
CSM tries to initialize and apply an enabled firewall up to three times, with one-second and two-second delays between attempts. Each attempt builds a fresh atomic transaction. A failed apply keeps the previous kernel rules in place. The retry delays stop when the daemon shuts down.
If all attempts fail, CSM continues monitoring but reports degraded health.
/api/v1/status and csm status --json expose
automation.firewall_enabled: true, firewall_managed: false, and
firewall_startup_error. The error remains available for the lifetime of that
process. csm doctor reports a failed firewall check and a recovery step.
Inspect journalctl -u csm.service, correct the reported configuration or
nftables permissions problem, and restart csm.service. A successful startup
clears the error and enables the firewall-dependent services. Disabling the
firewall deliberately does not degrade health.
If the firewall state file cannot be read at startup, CSM renames it to
state.json.unreadable-<UTC time> (with a numbered suffix if needed) in the
same directory before anything can
replace it, then starts without the blocked and allowed addresses it held. It
raises a Critical firewall_state_unreadable finding, and csm doctor fails
its firewall state check, naming the kept file, until the next restart.
automation.firewall_state_set_aside carries the same path. Review the kept
file and add back the entries that must stay. This startup alert bypasses the
finding queue and finding suppression rules. If the file cannot be renamed
or synced, startup fails without retrying, even if installing the base rules
also fails. A running CSM ruleset stays in place with its blocks; after a
reboot, CSM installs its base rules without saved blocks, allows or port
exceptions. The failed engine does not adopt replacement state or resume
management when storage becomes readable. Correct the storage problem before
restarting CSM. A running engine keeps its known state across an unreadable
rewrite, including failed saves, and writes that state back when storage
accepts a later save.
Before restoring blocks at startup, CSM asks the admission ledger about every
block an automatic response attempt wrote. Blocks whose attempt is still
running, ended unknown or failed are marked unverified and held back from the
kernel. Firewall views omit them once the mark is saved; if the mark cannot be
saved, the block still stays out of the kernel, but views can list it until a
later save succeeds. Blocks whose attempt failed, or ended unknown and was
later found expired and absent, are then removed from the saved state; if that
removal cannot be saved, they stay held back and the next start removes them.
If the ledger cannot be read, eligible blocks are restored
as before and existing unverified marks are kept; none of those automatic
blocks is evicted to make room for a new one, and csm doctor warns until a
restart reads the ledger. automation.firewall_ledger_error carries the
reason. A ledger missing its admission queue state counts as unreadable.
The same check runs after a confirmation window restores its saved blocks.
With a readable ledger such blocks are never evicted either: an
operator block that finds the temporary limit reached only because of them
is added above the limit. New timed automatic blocks record their origin
explicitly; operator blocks and operator undo do not acquire that origin
from their reason text. Older unstamped rows keep unknown origin.
Automatic admission may replace a block that recorded its origin with a
higher-priority one; a block of unknown origin is never replaced.
An automatic block replaces only blocks that recorded an automatic origin:
at the temporary limit it never replaces an operator’s temporary block or a
block of unknown origin, and is refused when only those remain. An automatic
alias of an operator block keeps that operator protection and the longer
lifetime. An operator block may still replace them.
Rows of an address family without a blocked set neither fill the temporary limit nor become replacement victims. Turning off IPv6 filtering therefore does not make saved IPv6 blocks refuse a new temporary block. Checking the limit does not remove blocks. Dry runs and allowed verdicts keep existing blocks, even when the limit would require a replacement.
At the temporary limit, replacement uses the same live set image as the limit count. A saved row without an element cannot free a slot. When no replaceable row has an element, the new block is refused; if a complete image cannot be read, the saved rows remain the fallback. The element must belong to the address’s own family: an IPv4-mapped key in the IPv6 set does not prove that an IPv4 block is present.
Temporary IP blocks use a firewall clock that counts elapsed time and never moves backward. Setting the system clock back does not add that step to their lifetime; a forward step can shorten restored blocks. While the firewall clock leads system time, new blocks show expiry times that much later. Elapsed time alone does not close the lead, but a later clock correction can. The journal reports the first lead greater than one second once per firewall start.
lease.json, beside the state file, stores the firewall clock reading, boot
identity and time since boot. CSM attempts to save it after each successful
ruleset install and saves it before a new temporary block changes state or
kernel rules. A restart on the same boot resumes no earlier than the saved
time plus elapsed time since that checkpoint. A reboot supplies no proof of
elapsed time from the previous boot, so the saved time alone provides that
bound. The current wall clock and the admission ledger’s saved clock can move
the resumed time forward.
An unreadable or damaged checkpoint logs a warning; restoration uses the remaining clock bounds. A save failure after ruleset installation warns without removing those rules. A save failure before a new temporary block refuses that block with an error. Older CSM versions ignore the checkpoint; if system time still trails the firewall clock at downgrade, saved temporary blocks can be restored for longer by that remaining lead.
Firewall and threat views omit unverified addresses, including duplicate spellings. When firewall state exists, threat views use it alone for block status; legacy tracking entries cannot make held-back blocks appear enforced.
CLI Commands
# Status
csm firewall status # Show status and statistics
csm firewall ports # Show configured port rules
# Block / Allow
csm firewall deny <ip> [reason] # Block IP permanently
csm firewall allow <ip> [reason] # Allow IP (all ports)
csm firewall allow-port <ip> <port> [reason] # Allow IP on specific port
csm firewall remove <ip> # Remove from blocked and allowed
csm firewall remove-port <ip> <port> # Remove port-specific allow
# Temporary
csm firewall tempban <ip> <dur> [reason] # Temporary block
csm firewall tempallow <ip> <dur> [reason] # Temporary allow
# Subnets
csm firewall deny-subnet <cidr> [reason] # Block subnet
csm firewall remove-subnet <cidr> # Remove subnet block
# Search
csm firewall grep <pattern> # Search blocked/allowed IPs
csm firewall lookup <ip> # GeoIP + block status lookup
# Bulk operations
csm firewall deny-file <path> # Bulk block from file
csm firewall allow-file <path> # Bulk allow from file
csm firewall flush # Clear all blocked IPs (subnet blocks kept)
# Safety
csm firewall apply-confirmed <minutes> # Apply the firewall block from csm.yaml with auto-rollback timer
csm firewall confirm # Confirm applied changes
csm firewall rollback status|confirm|revert # Manage pending config rollback
csm firewall restart # Reapply full ruleset
# Profiles
csm firewall profile save|list|restore <name> # Profile management
# Audit
csm firewall audit [limit] # View audit log
# GeoIP
csm firewall update-geoip # Download country IP blocks
# Cloudflare
csm firewall cf-status # Show Cloudflare IP whitelist status
CLI confirmation window
csm firewall apply-confirmed <minutes> applies the firewall settings from
disk with a rollback timer. csm firewall confirm accepts the current
ruleset and engine configuration and cancels that timer. While confirmation
remains pending, csm firewall restart and another apply-confirmed are refused.
A rollback restores the ruleset and the saved blocks of the moment the window
opened. When the window ends, CSM retimes temporary blocks from readable
saved state, including IPv6 blocks when a restart during the window had
disabled IPv6 filtering. Normal rollback restores the saved engine configuration
first; confirmation after an unreadable saved configuration accepts the current
configuration. A retiming failure logs a warning and releases the hold, so
restored timers may still outlast the displayed expiry. Older windows with
no saved block snapshot keep their restored timers.
If startup cannot read the saved engine configuration, rollback still
restores the kernel ruleset and state, but the window stays pending. Once
that restore is recorded, later restarts keep the blocks added since instead
of restoring the window’s start again; a missing, truncated or mismatched
record makes the next restart restore again, and an empty or corrupt marker
is recorded like any other. After such a ruleset-only rollback a restart
applies the configured settings, so the old kernel ruleset does not survive
a reboot. csm status and csm doctor show a pending window, when it rolls
back and why it cannot settle by itself. Inspect the current ruleset and
configuration before confirming: confirmation accepts the current state; it
does not recover the unreadable settings. To recover them instead, repair
the saved configuration and restart the daemon: the settings are restored
and the ruleset is rebuilt with the current blocks, and the window ends only
once that rebuild applies. Removing the saved configuration does not end a
recorded window. While a window is pending, a second apply-confirmed or a
restart is refused with what settles it; waiting helps only while its
rollback is scheduled and can restore the saved configuration. Confirmation
or rollback can report a leftover snapshot cleanup error after ending the
window. Fix that cleanup error before starting another window.
This CLI window is separate from the Web UI config rollback below, which
uses csm firewall rollback status|confirm|revert.
Configuration
Firewall defaults can be edited in two places:
- Web UI: Settings -> Firewall section. Port lists, rate limits, flood protection, deny caps, country block, and outbound SMTP restriction are all editable. Changes are restart-class. The save endpoint warns if the WebUI listen port is missing from
tcp_in. Theport_floodper-port rule list is YAML-only for now. - YAML: edit
/etc/csm/csm.yamldirectly. Runcsm rehashthensystemctl restart csm.
Tentative apply (rollback timer)
The Firewall section in the Web UI offers two save buttons. Save writes
the new config and prompts you to restart. Apply with rollback timer
writes the new config, restarts the daemon, and starts a timer (default 5
minutes, range 1-30). If you do not click Confirm before the timer
expires, the daemon restores the previous config and restarts again. This
protects against locking yourself out by, for example, removing the WebUI
port from tcp_in.
When the Web UI is unreachable (firewall mistuned, daemon broken), use the CLI escape hatch:
csm firewall rollback status
csm firewall rollback confirm
csm firewall rollback revert
Rollback state survives daemon restarts (the snapshot and its firewall configuration are persisted in the state directory). On startup the daemon checks for a pending rollback: if the deadline has already passed it restores the previous config and restarts; otherwise it restores the running firewall configuration and rearms the timer for the remaining window. Backup and store export omit this transient state, so restoring an archive cannot re-arm an old confirmation window.
firewall:
enabled: true
ipv6: false # false = ALL IPv6 traffic bypasses the firewall; CSM raises a finding on dual-stack hosts
conn_rate_limit: 200 # new connections per minute per source (IPv6 per /64; 0 = disabled; null = default)
syn_flood_protection: true # per-source SYN flood meter (IPv6 per /64)
conn_limit: 400 # max concurrent connections per IPv4 source (0 = disabled)
drop_ipv4_options: true # drop inbound IPv4 TCP carrying IP options (default: true)
smtp_block: false # restrict outbound SMTP
log_dropped: true
dyndns_hosts: # resolved every 5 min and whitelisted
- "monitoring.example.com"
Passive FTP range
When passive_ftp_start and passive_ftp_end are both unset, CSM opens TCP
49152-65534 for passive FTP data connections only on cPanel hosts and on hosts
with pure-ftpd, proftpd or vsftpd installed. This includes Plesk’s ProFTPD and
Debian’s MySQL, PostgreSQL and LDAP Pure-FTPd packages. Other hosts keep the
range closed. Explicit values take precedence; set both endpoints to 0 to
keep the range closed.
Full firewall reference: Configuration - Firewall.
Auto-response interaction
Auto-block calls require firewall.enabled: true because they go through the firewall engine. The engine consults two policy hooks first:
-
auto_response.verdict_callback- when enabled, the engine POSTs a signed JSON request to the panel after local validation and infra-IP safety checks. When a secret is configured, CSM rejects unsigned callback replies by default. The panel can downgrade toallow(audit-only), attachtenant_idfor downstream correlation, or add a note. CSM fails open on hook errors. In dry run, admission also asks the panel about each block it forecasts. Wire contract: verdict callback contract. -
auto_response.dry_run- when true (or absent; safety default),BlockIP()skips the automatic IP mutation. With the forecast owner active, only committed predictions that pass capacity checks create new bounded examples; a legacy dry-run outcome alone does not guarantee an example. Manualcsm firewall ...operator commands bypass viaBlockIPForceand always apply. Verify withcsm firewall statusafter policy changes; “Recently Blocked” timestamps newer than the last restart confirm live mode. See Auto-response - Dry-run safety default.
Subnet blocks refuse the default route and any range that contains an infrastructure IP, a resolved infra hostname, a local host address, a full-IP allow, or a port-specific allow. Remove the allow or narrow the CIDR before applying the block.
Allowlist precedence
Inbound IPv4 TCP packets with IP options are dropped in prerouting before
any of the rules below, infra included, while drop_ipv4_options is on.
Hosts that support the key report the firewall.drop_ipv4_options.v1
capability.
The nftables input chain accepts infra_ips first, then drops
blocked_ips, then accepts allowed_ips. Because the drop is evaluated
before the allowed_ips accept, an allowlisted IP that lands in
blocked_ips would still be dropped. The same applies to port-specific
allows, because those rules are evaluated after blocked_ips too. To keep
operator allows effective, the auto-block path refuses to add an IP to
blocked_ips when it is on allowed_ips (set by csm firewall allow), has
a port-specific allow (csm firewall allow-port), or is in a verified-bot
range (built-in or reputation.verified_bots). Precedence:
infra_ips- hard protect. Never blocked by anything, auto or manual; subnet blocks containing one are refused.tcp_in_allowsources - firm, one port. The accept for a listed source on its declared port sits right afterinfra_ips, ahead of the blocked address, blocked subnet and country rules. A block of that source, automatic or manual, takes effect on every other port and never on the declared one. To cut a source off, remove it from the entry. See firewall.tcp_in_allow.allowed_ips, port-specific allows, and verified-bot ranges - soft allow. The auto-block path skips them, but an explicit operator deny (csm firewall deny, Web UI manual block) still applies, because operator commands go throughBlockIPForceand bypass the soft-allow gate.
An operator block carries the lifetime the operator chose. csm firewall deny and the Web UI permanent block never expire; the Web UI 24 hour block
expires in the firewall and takes its threat-database evidence with it, so
the address does not keep scoring as malicious after the block is gone.
The Web UI 24 hour action refuses to shorten a permanent or longer block; unblock explicitly before changing its lifetime. Undo of a bulk block or unblock restores each original deadline, including permanent blocks, and never extends an expired block. A later operator decision supersedes the older undo action.
Lockout warnings
Config validation warns when an enabled firewall would cut off the management
plane. csm doctor, daemon startup, and the Web UI save path all run the same
checks, and the Web UI returns each warning once:
- The enabled Web UI port is missing from
tcp_in, or from an explicittcp6_inoverride when IPv6 filtering is enabled. An emptytcp6_ininheritstcp_in. A port opened to at least one usable source intcp_in_allowdoes not warn. - A
restricted_tcpentry also appears in an effective public TCP allow list, but noinfra_ipsare configured. The restricted list only filters public accepts; it does not open ports itself. Matching ports are therefore reachable only through the port-agnostic infrastructure-IP accept rule, and with no infrastructure addresses they are reachable from nowhere. A port with a usabletcp_in_allowsource stays reachable from that source and is left out of the warning.
These stay warnings and never block a save or a start: fronting the Web UI with
a reverse proxy or reaching it over a VPN are legitimate reasons to leave the
port out of tcp_in.
One more warning compares the policy against the host instead of against the
config, so it runs in csm doctor, csm validate --deep, and the Web UI
firewall save rather than on every load:
- sshd listens on a port that
tcp_in(or an explicittcp6_in) does not allow. The shippedtcp_inleaves 22 out, because many hosts move sshd, so a host that never moved it loses SSH on the first apply. EveryPortdirective counts, including ones inIncluded drop-ins under/etc/ssh/sshd_config.d/, and a port named inrestricted_tcpwithinfra_ipsset is treated as a deliberate infra-only listener. A port opened to listed sources intcp_in_allowis treated as deliberate too, unless none of its sources renders a rule.AddressFamilyandListenAddresslimit the check to the IP families sshd exposes, and loopback-only listeners do not count because the inbound firewall always accepts loopback traffic. Hosts with no sshd config get no warning.
Egress
tcp_out is default-drop as well and ends in a TCP reset, so a host whose
policy omits a port it dials does not lock an operator out; it goes silent.
Every heartbeat, finding delivery or intel lookup fails at once with
“connection refused”, which reads like the far end being down, while the host
looks healthy locally. The same validation pass therefore warns when an
enabled firewall’s outbound policy would refuse a connection the daemon
itself needs:
- The port of every enabled outbound endpoint in the config is checked
against
tcp_out:alerts.email.smtp,alerts.webhook.url,alerts.heartbeat.url,alerts.audit_log.syslog.address(tcp and tls transports),auto_response.verdict_callback.url,reputation.rspamd.url,reputation.upstream.url,reputation.report.targets[].url,reputation.central.set_url,signatures.update_url,signatures.yara_forge.download_url,sentry.dsnandupdates.github_api_url. HTTP endpoints use their explicit numeric port, or the scheme default when the port is omitted. SMTP and TCP/TLS syslog addresses also accept TCP service names, matching their dialers. Disabled features are skipped, and so are loopback destinations, which the output chain accepts ahead of any port rule. - Port 443 is checked once for the built-in HTTPS endpoints (threat feeds, AbuseIPDB, MaxMind, YARA Forge, AI-crawler range feeds, release check), because dropping it silences all of them at once.
- Every port under
firewall.required_tcp_outis checked. That list is a declaration, never added to the policy: a conf.d fragment owned by an integration can state the ports its service needs, andcsm doctorreports when the effective policy drops one instead of the operator discovering it from a silent node. The check runs against the mergedtcp_out, so any config layer that permits the port satisfies the declaration.
On a restricted output chain, smtp_block installs per-user accepts for the
mail ports ahead of the port rules, and those ports never get a port rule of
their own. The daemon runs as root, so its alert mail is not warned about when
tcp_out omits a port that smtp_block still lets root reach. A port
declared for another service still warns under smtp_block, because a port
declaration cannot prove that service’s user is allowed. When IPv6 is managed,
an explicit tcp6_out is checked separately; an empty one inherits tcp_out,
and the single warning covers both families. When only the IPv6 lists are set,
IPv4 egress is accepted wholesale and the warning names tcp6_out. A literal
IPv4 or IPv6 destination is checked only against its own family.
This catches the daemon’s own egress and whatever has been declared. It does
not see what an arbitrary third-party process on the host dials; an agent
that ships its own conf.d fragment should declare its ports under
required_tcp_out there.
Value validation
Unlike the warnings above, these are errors, because the value cannot do what the operator meant:
- Ports outside 1-65535 in any port list, including
drop_nologand the passive FTP range, and a passive FTP range that ends before it starts. country_blockentries that are not two-letter ISO codes.port_floodentries with an out-of-range port, a protocol other thantcporudp, or a non-positive hit count or window.
Two enums elsewhere get the same treatment because their consumers fall back to
a default branch rather than failing: reputation.central.action (an
unrecognised value became a challenge policy) and
incidents.*.block_at_severity, which accepts only high or critical and
silently disabled incident blocking on anything else.
Incident auto-block escalation
An incident-driven block used to be requested once per incident and never
again. The block it applied expired after auto_response.block_expiry, but the
marker saying “already blocked” did not, so an attacker who kept going past the
expiry was never blocked a second time while the incident stayed open and kept
collecting evidence.
A block is now re-requested whenever the previous one has lapsed and the incident is still active and still receiving qualifying findings, and each request lasts longer than the last:
| block | lifetime |
|---|---|
| first | auto_response.block_expiry (24h by default) |
| second | 7 days |
| third and later | permanent |
Admission caps new temporary lifetimes at auto_response.block_expiry, live
and in dry-run forecasts; shorter requests stay short and retries keep
an earlier attempt’s absolute expiry, even after the cap is lowered.
Admission does not turn a non-positive lifetime into a permanent block.
The lifetimes in the table are what existing blocking applies. A block
admission applied is never promoted to a permanent block while it stands.
A permanent block is never re-requested. Closing an incident, by an operator or by the stale-incident sweep, resets the ladder, so a later recurrence starts at the bottom rather than inheriting a months-old episode. Concurrent findings still collapse into one firewall call, and a declined or dry-run request is not recorded, so it can retry.
The ladder survives restarts and quiet intervals while the incident remains active. Closing the incident, manually or automatically, resets it; a pending block callback cannot restore the old ladder after that close.
The incident view carries a Block button whenever the incident has one
unambiguous source address. Mixed-source or truncated timelines without an
address in the correlation key do not offer a block target. The button asks
for confirmation, blocks permanently, notes the block on the incident timeline as
operator_block, and settles the ladder so the automatic hand-off does not
re-request a block for an address the operator just blocked.
The block API accepts an optional incident_id. An invalid, unknown, or
address-mismatched incident ID does not prevent the firewall block, but does
not change the incident. Refreshing an existing temporary block does not
advance the escalation rung. Blocks recorded on closed incidents remain audit
actions without restarting the ladder.
Local port locks
firewall.local_port_locks limits which local users may open new TCP
connections to a port on this host. Each entry is PORT:user,user; root is
always allowed. Other users get a connection reset. Connections to other hosts
on the same port and connections already open are not affected.
firewall:
local_port_locks:
- "8080:www-data,stalwart"
Use it for a service that trusts what local clients tell it, such as a mail
server that takes the webmail client’s address from a forwarded header. List
only users that run no site code: a PHP pool running as a listed user can
connect too. A user
name that does not exist is reported at startup and skipped; the port stays
locked to the others. The status page and csm doctor read the locked ports
from the running daemon after successful installation. Unapplied changes, failed replacements, and live rules that no longer match the installation do not count as protection.
Infrastructure IP DNS guard
Hostnames listed in top-level infra_ips or firewall.infra_ips are resolved every 5 minutes and their current addresses feed the infra auto-block guard. If a hostname stops resolving, the daemon emits an infra_ips_unresolvable Warning finding and keeps the last known addresses protected during the grace period (default 10 min). This prevents a transient DNS outage from deprotecting the management plane. The finding auto-clears when resolution recovers.
DoS-exempt ranges
Operators can declare IP ranges that bypass the new-connection rate meter for their IP family, the IPv4 concurrent connection-limit, and mail-port flood meters, preventing false-positive throttling and subnet auto-blocks for carrier CGNAT pools or mail-provider egress. Configure under firewall.dos_exempt_ranges (your own CIDRs) and firewall.dos_exempt_known_mail_providers (adds Google and Microsoft mail ranges, on by default). See Configuration - firewall.dos_exempt_ranges.
What exempt sources bypass
Sources in the exempt set skip three categories of metering:
- Connection rate-limit - the new-connection rate meter (configured via
conn_rate_limit) does not apply for the source’s IP family. - Concurrent connection-limit - the IPv4 concurrent connection cap (
conn_limit) does not apply. - Mail-port flood meters - the
port_floodrules on TCP 25, 465, and 587 do not apply for the source’s IP family.
Subnet auto-block (spray, ASN-crawl, and netblock escalation) also skips any subnet block whose CIDR intersects an exempt range, and exempt IPs are excluded from the per-subnet threshold count so they cannot push a subnet over the netblock limit. Auto-response subnet blocks whose range falls inside an exempt range are removed automatically at daemon startup and at the start of each auto-block cycle, but only while IP auto-response is live; dry run leaves them in place. Manually created IP and subnet blocks are never pruned, even if they fall inside an exempt range.
What exempt sources do not bypass
The following protections remain in force regardless of exempt status:
- Manual blocks -
csm firewall deny <ip>andcsm firewall deny-subnet <cidr>go throughBlockIPForce, which bypasses the exempt check. An IP or range that is both exempt and manually blocked is still dropped. - SYN flood protection - the SYN flood meter is not affected by the exempt set.
- UDP flood protection - the UDP flood meter is independent of the exempt set.
- Country blocking - country CIDR blocks apply unconditionally.
- Port policy -
tcp_in,tcp_out, andrestricted_tcpport rules are not modified.
The rule ordering that makes this work: the nftables input chain evaluates blocked_ips (and subnet blocks) before the DoS-meter rules. So a manual block inside an exempt range still drops the traffic – the block is hit before the meter that exempt sources bypass.
Dynamic mail-provider ranges
When dos_exempt_known_mail_providers is true (the default), the daemon resolves Google and Microsoft outbound mail ranges at startup and pushes them into the firewall exempt sets before the first rule application. The ranges are discovered from the providers’ published SPF records (the Google and Microsoft mail SPF roots), so they track provider changes without a CSM update. They are cached on disk so the previous set is available immediately on subsequent starts. A built-in snapshot is used if the cache is missing or the first live refresh has not completed. The cache is refreshed every 12 hours; if a refresh fails or the nftables reapply fails, the previous overlay is preserved unchanged.
ModSecurity Integration
CSM detects ModSecurity (WAF) on Apache, Nginx, and LiteSpeed across cPanel and plain Linux hosts. Custom rule deployment, override writes, and reload management are currently cPanel-only; other platforms still receive status, staleness, and event detection.
Supported Web Servers
| Web server | Config candidates | Status check | Custom rule deployment |
|---|---|---|---|
| Apache on cPanel EA4 | /usr/local/apache/conf/*, /etc/apache2/conf.d/modsec*, whmapi1 modsec_is_installed | Yes | Yes (via cPanel modsec user conf) |
| Apache on Debian/Ubuntu | /etc/apache2/mods-enabled/security2.conf, /etc/apache2/conf-enabled/*, /etc/apache2/conf.d/modsec2.conf | Yes | No |
| Apache on RHEL/Alma/Rocky | /etc/httpd/conf.d/mod_security.conf, /etc/httpd/conf.modules.d/* | Yes | No |
| Nginx on any distro | /etc/nginx/nginx.conf, /etc/nginx/modules-enabled/50-mod-http-modsecurity.conf, /etc/nginx/modsec/main.conf | Yes | No |
| LiteSpeed | /usr/local/lsws/conf/httpd_config.xml, /usr/local/lsws/conf/modsec2.conf | Yes | cPanel only |
When ModSecurity is not installed, the waf_status check emits a platform-specific install hint:
# On Ubuntu + Nginx:
Install: apt install libnginx-mod-http-modsecurity modsecurity-crs
# On Ubuntu + Apache:
Install: apt install libapache2-mod-security2 modsecurity-crs && a2enmod security2
# On AlmaLinux + Apache:
Install (requires EPEL): dnf install -y epel-release && dnf install -y mod_security
# On AlmaLinux + Nginx:
Install (requires EPEL): dnf install -y epel-release && dnf install -y nginx-mod-http-modsecurity
# On cPanel:
Install: WHM > Security Center > ModSecurity
Rule-staleness alerts scan both the flat CRS layout (/usr/share/modsecurity-crs/rules/*.conf) used by distro packages and cPanel vendor trees, including nested layouts such as modsec_vendor_configs/VENDOR/rules/*.conf. On cPanel, CSM maps WHM’s active configuration files to their vendor trees and checks the newest artifact in each loaded tree. This honors individual configuration overrides, keeps retired trees out of the result, and prevents a fresh unloaded vendor from hiding a stale loaded one. LiteSpeed also keeps the on-disk check as a backstop while cPanel rebuilds the active configuration list and rule tree. If WHM cannot provide the mapping, and on other platforms, the check keeps the conservative oldest-artifact behavior. Update instructions are platform-specific (apt update && apt upgrade modsecurity-crs, dnf upgrade modsecurity-crs, or WHM on cPanel).
Features
- Custom CSM rules - IDs 900000-900999 in
configs/csm_modsec_custom.conf(cPanel only today) - Rule override management -
SecRuleRemoveByIddirectives for false positive suppression - Escalation control - change rule severity or action per-rule
- Live deny escalation - repeated ModSecurity deny events from one IP emit an escalation finding that feeds auto-response blocking. CSM-owned rules keep their existing per-rule escalation controls. CSM rules that match one specific exploit escalate on the first deny.
- Disabled-scope detection - reports domains and accounts with the engine switched off, covering both the userdata flag and the per-account and per-domain config includes used by Apache and LiteSpeed in the std and ssl trees. A malware finding from a scheduled or account scan on such an account, or on a file served by such a domain, gets a context line naming the scope and the file that switches the engine off. It shows in alerts, the web UI, scan reports, webhooks and the audit log. Only the accounts and sites with malware findings are read for this. The line is not part of the finding’s identity: the finding keeps its severity, and turning the engine off or on neither re-alerts it nor undoes a dismissal
- WAF event log parsing - correlates events by IP, URI, and rule ID
- Hot-reload - apply changes without Apache restart (cPanel only)
- Rule activation - ModSecurity reads rules only when the web server starts or reloads. When CSM’s installed rule sections change, for example after an upgrade or
csm install, the daemon runsmodsec.reload_commandat startup or during its WAF check. Each rule change reloads once. Standalonecsm checkruns never reload. A failed reload raises awaf_statuswarning and is retried. Without a command, CSM only warns at startup and the rules wait for the next web server restart.
First-deny escalation requires a blocking action for an exact CSM rule ID. Request text cannot turn a warning into a deny. Disabling a rule’s firewall escalation excludes both new events and retained evidence from later blocking decisions. Existing firewall bans keep their configured expiry.
ModSecurity error-log detections take the client address only from the
structured prefix before ModSecurity: or [MODSEC]. Fields later in the same
record cannot supply a client address. Live monitoring, scheduled WAF checks,
and the Web UI use the same parser. The module marker must follow the server’s
structured header directly. Markers inside ordinary file errors, PHP errors, or other
message text do not create WAF detections. Standard Apache, Nginx and LiteSpeed
headers are accepted; custom headers outside those formats are ignored.
Apache’s module-qualified error prefix, including [:error], identifies its
unbracketed IPv6 connection port. Legacy Apache and libmodsecurity client fields
keep the complete IPv6 address. Bracketed addresses with a port are also accepted.
Custom Apache logs must retain the module field to distinguish unbracketed IPv6
addresses with ports from bare addresses.
LiteSpeed IPv6 connection formats are unsupported: events keep an unknown
client and cannot trigger an IP escalation. The Web UI uses the finding’s
recorded source address, with a prefix-only raw-log fallback for older findings;
it leaves the client unknown when neither is available.
On Apache 2.4 a line must carry ModSecurity’s own module tag (security2 or
security3). Lines that another module tags, such as PHP or CGI output, and
LiteSpeed PHP stderr lines are never ModSecurity events, even when their text
holds a ModSecurity message. A LiteSpeed line must have its standard header,
with the connection field directly before [MODSEC].
Apache 2.2-style headers without a module tag remain accepted for compatibility. They cannot distinguish ModSecurity output from a hosted site’s message that starts with the same marker, so false alerts and visitor escalation remain possible with that format. Use module-tagged Apache logs to separate writers.
Header validation does not authenticate file-based records. Nginx can write literal line breaks from decoded filenames into its error log, allowing request text to create an entire apparent record on a new line. File-based real-time escalation and scheduled error-log scans remain exposed to that case. Native ModSecurity audit logs provide transaction framing for scheduled scans; fixing real-time ingestion requires a source that preserves authenticated record boundaries rather than splitting arbitrary log bytes into lines.
Scheduled audit scans read native serial logs and concurrent indexes from
ModSecurity 2 and libmodsecurity 3, plus libmodsecurity 3 JSON logs. Counts use
the recorded client address and denial action or response status. Request and
response content cannot select another client or cancel another client’s counts.
The existing web_server.modsec_audit_logs setting selects the candidate logs.
To scan error-log records instead, include the source in that candidate list
and identify it with modsec_error_log or web_server.error_logs. Detected
platform error logs are also recognized. These sources count only messages
starting with Access denied with code after the module marker. Request text
cannot switch their interpretation to an audit index or suppress a denial.
Other audit candidates accept native audit records only. Custom error-log
candidates must be identified this way when upgrading. Changing a source’s
type restarts its scan so previous counts cannot carry across formats.
Only complete transactions within the recent line window contribute to these counts. The first scan and scans after log rotation read from the file’s start to establish transaction boundaries. Later scans resume from the saved position. Large existing logs can take longer on the first scan. Memory use stays bounded, and incomplete writes wait for the next scan.
CSM adds the default escalation exclusion for browser-marked upload progress requests once, including on upgrade, and keeps later operator changes. If the saved exclusions cannot be read, CSM keeps running and blocking requests but pauses ModSecurity firewall escalation, and the rules page reports the error instead of overwriting them. The next start retries the default update.
LiteSpeed logs every rule match as triggered!, whether or not the rule denied
the request, so CSM looks up each rule’s action in the installed rule files.
Apache logs say whether a request was denied and do not need the lookup. A rule
whose action is block does what the SecDefaultAction for its phase says, and
pass when there is none, as in ModSecurity. OWASP CRS in anomaly scoring mode
sets pass, so its detection rules count as warnings, not denies. CSM does not
follow Include order: a SecDefaultAction applies to the rules in its own
directory and below, so a vendor pack’s setup file governs that pack and no
other.
CSM reads rule IDs, phases and actions from the rule’s action argument. Text
inside an operator, message or log data cannot override them. Explicit
disruptive actions take precedence over inherited block actions. When setup
files in the applicable directory cannot be read or parsed, inherited block
actions remain blocking until CSM can resolve them. Explicit actions are still
usable, and rules without a disruptive action remain unknown.
The directory walk follows symlinked packs and rule directories. Defaults follow the configured directory layout, including nested overrides. Directory cycles are reported as incomplete reads and prevent caching the result.
CSM checks the rule-action registry every five minutes and rebuilds it when rule or setup file contents change. Read failures leave the build uncached so the next check retries. Files that were read in full but exceeded the parser’s line limit are reported without forcing unchanged files to be parsed again. Rules appended after parsing reaches the end of a file are picked up on the next check.
The reload command runs inside CSM’s systemd sandbox. Use a service-manager
command such as systemctl reload lsws for LiteSpeed, so the web server’s service
performs the reload. Direct reload scripts inherit CSM’s filesystem restrictions
and may fail. A LiteSpeed reload restarts workers and can briefly raise load.
The LiteSpeed Cache role-simulation filter covers privileged routes and writes, including WordPress REST method overrides, when requests carry simulation cookies with a weak hash. Ordinary public GET/HEAD crawling remains allowed. This is a request-scoped mitigation: public reads still run under the simulated identity, so upgrading the vulnerable plugin remains necessary.
The usual hash discriminator is 1-16 alphanumeric characters versus the fixed plugin’s 32-character hashes. Numeric equivalents are also filtered because the vulnerable plugin compares hashes loosely; padding or exponent notation must not turn a weak hash into an exempt one. These checks also keep public crawler reads allowed.
The WordPress user enumeration filter blocks anonymous requests for the REST
users route, whether the route follows wp-json/ in the path or starts at
wp/v2/users in the rest_route query parameter, in any letter case. Other
page paths and REST namespaces are left alone. Query values are matched as
already decoded by the query parser. Requests that carry an Authorization
header or a WordPress logged-in cookie pass, so admin screens, the editor and
Application Password clients keep working. Both are presence checks: the filter
turns away anonymous scanners, and WordPress still decides who is signed in.
The filter runs in phase 1 and does not inspect request bodies. A route supplied
only in a POST body, including with a REST method override, is outside its scope.
Sites that need to restrict the public users endpoint must enforce that policy
in WordPress. Disabling rule 900112 disables this filter for both route forms;
its helper rules only set transaction-local flags and do not block requests.
The PHP session upload-progress filter denies requests whose form field names
contain PHP’s default PHP_SESSION_UPLOAD_PROGRESS field name, matched
case-insensitively so quoted spellings are covered. A custom
session.upload_progress.name is outside this filter’s coverage.
GET and HEAD pass. Other methods use the first-deny escalation rule when both
Origin and Sec-Fetch-Site are absent; browser-marked requests use the separate
rule whose escalation is off by default. Keep that exclusion to avoid
browser-steered bans. A browser whose extension or proxy strips Origin on a
plain-HTTP request can still reach the first-deny rule.
PHP handles that field before any script runs, which
leaves the web server as the only layer that can drop the request. See
CVE Mitigations for the PHP-side setting that removes the
handler.
For Apache ModSecurity v2 regression validation, run
python3 scripts/test-litespeed-modsec.py in a disposable Debian Linux environment
with apache2, libapache2-mod-security2, libapache2-mod-php, and python3
installed. The test loads the complete shipped configuration and exercises HTTP
requests through the actual engine and PHP parser. It does not establish
LiteSpeed runtime compatibility; verify changed rules on the supported LiteSpeed
engine before deployment.
Web UI Pages
ModSecurity (/modsec) - WAF status overview, event log, active block list, filterable by time range, minimum severity, and source country
ModSec Rules (/modsec/rules) - per-rule management:
- View the CSM rules with descriptions and hits in the last 24 hours
- Enable or disable individual rules; changes are staged and applied with one reload
- Turn firewall escalation off for a rule: the rule still denies the request, but CSM does not block the IP in the firewall
- Escalation exclusions are listed and edited here even when rule management (
modsec.rules_file,modsec.overrides_file,modsec.reload_command) is not configured, since the daemon applies them either way
API Endpoints
GET /api/v1/modsec/stats WAF statistics
GET /api/v1/modsec/blocks Blocked request log
GET /api/v1/modsec/events WAF event details
GET /api/v1/modsec/rules Loaded rules list
POST /api/v1/modsec/rules/apply Apply the set of disabled rules and reload
GET /api/v1/modsec/rules/escalation Rule IDs excluded from firewall escalation
POST /api/v1/modsec/rules/escalation Exclude one rule from escalation or turn it back on
Signature Rules
CSM uses YAML rules for real-time scanning and finding re-checks. Optional
YARA-X rules also run during deep scans and email attachment scanning. Rules
are stored in /opt/csm/rules/. An engine that loads zero rules (a mistyped
signatures.rules_dir, an empty rule sync) raises a realtime_rules_missing
finding at startup and after every reload, because both engines otherwise
treat an empty directory as a successful load and scan every write against
nothing.
Deep scans are rolling: each scheduled run resumes from a persisted cursor and scans as much as fits in its time budget, so the whole content set is covered across runs even when a single run cannot finish it. A warning finding is raised if no full pass has completed within 30 days.
The same rolling walk also feeds the JavaScript keystroke taint analyzer (js_keylogger_dataflow) and the PHP remote-source taint analyzer (php_remote_taint); see the deep checks reference for both. Each consumer keeps its own cursor and completion record, so none of them stalls the others: a missing or failed YARA backend does not hold up either taint analyzer, and the PHP analyzer being unavailable does not affect YARA or JavaScript coverage.
A rule declaring file_types: [".php"] is applied to every extension a stock PHP handler executes (.php2 through .php8, .phtml, .pht) and to .phps, because that source-view extension still contains PHP source. .phps stays outside the set of extensions CSM treats as executable, so this does not change which files the real-time dropper tracker considers runnable.
Both engines skip ZIP, gzip, bzip2, xz, 7z, and RAR containers, but only when the file name carries an archive extension as well as the archive signature. Matching compressed bytes or stored filenames produces false positives without inspecting the archived file, so real containers are scanned when they are extracted onto monitored storage instead. A file whose name a web server would execute is always scanned whatever its first bytes look like, because PHP echoes any leading bytes and runs the rest. Uncompressed tar files and executable PHP archives (PHAR) remain scannable, and filename-based phishing-kit archive detection is unchanged.
YAML Rules
Encoded spam-link detection applies to real-time scanning and finding re-checks only. It looks for external-link attributes in PHP that directly prints a base64 literal. Embedded tracking pixels, local image references and inline logos without external links do not supply that evidence. An encoded external link is a heuristic, not proof of spam or proof that the destination is outside the site’s own domain. Scheduled YARA scans do not use this rule.
The WordPress REST API exploit signature is a YAML-only heuristic. It requires a users-endpoint URL literal and a password query parameter or a nearby PHP, JSON or form-encoded password field, including payloads prepared before the URL. Endpoint prose alone does not qualify. This is bounded textual evidence, not PHP data-flow analysis or proof that a request is unauthorized.
rules:
- name: webshell_c99
severity: critical
category: webshell
file_types: [".php"]
patterns: ["c99shell", "c99_buff_prepare"]
min_match: 1
- name: phishing_login
severity: high
category: phishing
file_types: [".html", ".php"]
patterns: ["password.*submit", "credit.*card.*number"]
exclude_patterns: ["legitimate_form_handler"]
min_match: 2
Fields:
name- unique rule identifierseverity- critical, high, or warningcategory- webshell, backdoor, phishing, dropper, exploitfile_types- file extensions to match (or["*"]for all)patterns- case-insensitive literal stringsregexes- regex patterns, compiled case-insensitivelyregex_pairs- entries withfirst,secondandmax_distance; each pair counts as one regex match when its match starts are within that many bytes, in either orderexclude_patterns- literal patterns that suppress a match (false positive reduction)exclude_regexes- regex patterns that suppress a matchmin_match- minimum total number of matching literal and regex entries; each entry counts oncerequire_regex- require at least one regex among the matches counted towardmin_matchmax_file_bytes- skip this rule when the complete scanned file is larger than the byte limit; omitted or0is unboundedmax_file_bytes_exempt_regexes- high-confidence regexes that let the rule continue normal evaluation abovemax_file_bytes
A regex only runs on a file that contains the fixed text every match of it
must contain, compared without regard to case: for eval\s*\(\s*base64_decode
that is both eval and base64_decode. Write regexes around distinctive words
such as function names: a regex with no fixed text of two or more characters
runs over every file of its types, and realtime pays that cost on each write.
Required text can include escaped bytes such as \x00; these remain part of
the literal when checking whether a file could match.
Regex pairs compare the ordered, non-overlapping matches of two expressions.
Both expressions must be non-empty and compile; max_distance must be
non-negative. A distance of zero requires both matches to start at the same
byte. Use pairs for nearby evidence instead of joining expressions with a
wide wildcard gap, which can slow finding re-checks on large files.
When a regex includes a literal listed in patterns, the same content can
satisfy both entries. Use independent entries when a rule needs multiple pieces
of evidence. The bundled HTTP tunnel rule requires both socket creation and a
CONNECT request. The legacy PHP callback rule uses the same narrow signature in
YAML and YARA-X: a direct function call with a quoted parameter list, a variable,
null, a simple array lookup or a short helper call as its first argument.
The second argument is a decoder or request lookup, optionally preceded by one
concatenated literal. Quoted lists can contain commas, semicolons and escaped
quotes. Array indices accept a single quoted key or an unquoted scalar. Helper
arguments accept at most one literal among unquoted scalar operands, including
implode(',', $args). These bounded forms consume quoted operands whole and
exclude comments, interpolation and nested expressions, so delimiters inside
data cannot supply the body-source evidence. Double-quoted array keys, helper
literals and body prefixes must escape dollar signs; unescaped dollars require
interpolation analysis.
Shared positive and benign fixtures check both engines. Generated socket and
funchand wrappers and ordinary legacy callbacks stay silent under these rules.
The cron downloader signature shares its expression between both engines. It
recognizes numeric five-field schedules and the standard @ shortcuts,
including lists, range steps and randomized ranges. The month and weekday
fields also accept a single three-letter ASCII name in either case. Unicode
lookalikes are not schedule names. Name lists and ranges are not
matched, because they made crafted lines too costly to scan. Schedule tokens
must be complete and separated by spaces or tabs. Fields are limited to 64
characters. Root entries that disable cron logging and scripts
that install a quoted cron entry remain covered. It requires a download followed
by a complete shell, eval or exec command on the same line. Quoted shells,
command groups, common execution wrappers and shell lookup commands are
recognized. Bounded spans keep crafted lines cheap to scan;
unrelated lines and shell words inside longer command names are not evidence.
The download-and-pipe, miner download, miner cron and shell startup file
signatures also look for their parts within bounded spans of one line, in both
engines. A download command is found wherever it sits on a long line, and a
# shortly before it still marks a commented-out usage line. The YARA-X rules
treat a command as documentation only inside a fenced code block, whose opening
fence starts a line and whose closing fence stands on its own line, or as the
text of a Markdown link. The shell startup file signature judges a file with
more Markdown blocks and links than any published document carries without that
suppression, so its scan time stays linear. The download-and-pipe and shell
startup file signatures need a whole shell or interpreter name after the pipe,
so a download piped into a checksum tool or another program whose name starts
the same way is not an execution. Versioned Perl and Python names, Python
build variants and standard Windows executable names retain detection. Both
engines use the same shell name boundaries, including commands embedded in
strings and binary data. The miner cron signature ignores letter case in both
engines, accepts only spaces and tabs between schedule fields, and counts its
bounded command span in characters so non-ASCII paths have the same coverage.
The other bounded spans are measured in bytes by the YARA-X rules and in
characters by the YAML rules, so the two can disagree on a line with non-ASCII
text in the span. Measuring characters in YARA-X slowed scans of crafted lines
too much. A finding is re-checked by the engine that raised it.
Both engines report download-and-pipe commands even when a file begins with HTML or has an HTML extension. This includes install commands printed on web pages. Leading markup does not prove that later lines cannot run as shell code.
The unauthenticated uploader signature skips a framework helper class: a file holding only a direct-access guard that exits and one class declaration, which writes the upload through the object and creates no object itself. Code after the class, or syntax the check cannot read, keeps the alert.
The PHP goto-obfuscation rule requires three independent signals in both engines: a PHP opening tag, at least nine jumps to digit-bearing generated labels or eleven to alphabetic labels, and a decode call, execution call, dynamic include, or request input. Fixed-path bootstrap includes do not supply this evidence. Variable and array callback calls count even when the function name is constructed and the argument is a literal. Comments between a callable and its opening parenthesis do not hide the call. Line breaks and keyword case do not change the label counts. Long encoded strings and data URIs alone are not execution evidence. These are source-text heuristics, not PHP dataflow analysis; they cannot resolve arbitrary dynamically generated code or distinguish every benign use of these operations.
The PHP content heuristic that runs during scans applies the same evidence
rule: generated goto labels and descriptive goto labels both need a decode
call, execution call, dynamic include, or request input before they count as
an obfuscation indicator. call_user_func is deliberately not evidence in
either place, because plugin loaders dispatch their own callables through it.
The content heuristic uses the signature’s evidence expression, including
comment-separated calls, grouped and array callbacks, and case-sensitive PHP
superglobal names. A regression check guards against expression drift.
The archive loader signature reports PHP that runs code read out of an
archive, in both engines. A ZipArchive member read, or a zip:// or phar://
path, must sit inside an eval, assert or create_function call, lie
within 1000 bytes of an eval that leaves PHP mode first, before or after it
(measured between match starts in both engines),
or follow a password set on the archive and precede eval of a variable.
Extracting archives to disk, parsing data members and template engines
evaluating local text are not evidence, nor is a bundled tool such as a
.phar whose stub names an archive path far from a template engine’s eval.
Including a zip:// path is reported with or without parentheses. These are
source-text heuristics, not PHP dataflow analysis. Execution arguments stop
at the end of their call while allowing balanced prefix calls. Assertions
that parse or compare archive data are not execution evidence. Both strings
passed to create_function remain source text, as described in the
PHP manual.
Regression fixtures run through both engines, including mixed-case calls and
ordinary archive and template code.
Three signatures report code an injection kit leaves in WordPress files, in
both engines. The first-line stub signature matches only a file that opens
with an array holding an empty string followed by an isset test with an
empty body. The line does nothing, outlives cleaners that strip only the
payload, and no generator writes it. The visitor logger signature needs a site
or host hash, cookie handling, an append-mode fopen and a visitor address
read, each starting within 1200 bytes of a lookup of the upload temp
directory. The hook injector signature needs add_action or add_filter
called with a hook name returned by a call on an encoded literal, starting
within 4096 bytes of both base64 decoding and a byte-wise XOR loop. Plugins that name hooks in plain
text, keep caches in the temp directory or mask licence keys with XOR are not
evidence.
Legacy callback parser follow-up
The callback signature does not inspect quoted function bodies. Doing so needs
PHP string decoding, tokenization and expression analysis: for example,
assert($x > 0) is an ordinary boolean check, and "eval($x)" can be data.
The rules scan source text, so they do not promise general PHP comment or string
awareness, nor complete coverage of dynamically generated code.
The following cases were covered by the expanded regex and are deliberately outside the narrowed signature. They remain acceptance cases for a parser follow-up, not claims of current detection by this rule:
| Deferred case | Examples to restore |
|---|---|
| Constructed parameter lists | Concatenation, nested array indices, helper calls with multiple quoted operands, and chr(100/(1+1)) as the first argument; only the bounded simple forms above are covered |
| Comments inside constructed parameters | Comments in array indices or helper arguments, especially those containing closing delimiters or body-source names |
| Interpolated parameter operands | Double-quoted array keys or helper operands containing unescaped dollars, including interpolation with nested quoted keys |
| Comments between arguments | Block comments with embedded commas, and line comments before the body source |
| Constructed body expressions | Grouped concatenation, parenthesized decoders and trim(base64_decode($payload)); a single literal concatenated onto request input is covered |
| Interpolated bodies | A double-quoted body such as "return {$_POST['code']};", or a concatenated double-quoted prefix containing unescaped dollars |
| Literal executable bodies | eval($x) or string-capable assert($x), with statements, strings or comments before them; both outer quote styles and escaped quotes |
| Literal expression contexts | return, or, do, case, include, include_once, require, require_once, clone, yield from, comparisons, shifts and inequality before an execution sink |
| Literal lexical edges | Global assert, comment backslashes before * or */, and quoted operands before a comparison |
| Outer call contexts | Calls immediately following a ternary colon, case-label colon or comparison operator |
This work belongs in internal/phptaint, which already uses VKCOM/php-parser
and records the second argument of create_function as a sink. Extend that
analysis to decode and parse statically known callback bodies under its existing
budgets, distinguish boolean assertions from string execution, and report
unresolved dynamic bodies as analysis gaps. It currently feeds a separate
scheduled check; it is not a post-filter for YAML or YARA. Realtime coverage
would need explicit integration and latency tests, and standalone YARA would
still have the narrower coverage.
Acceptance requires restoring the deferred cases as positive parser fixtures, retaining the shared benign fixtures, checking both quote styles and comment forms, and passing the clean-corpus gate without new baseline entries. Do not expand another regex into a PHP tokenizer to recover these cases.
YARA-X Rules (Optional)
Build CSM with YARA-X support:
CGO_LDFLAGS="$(pkg-config --libs --static yara_x_capi)" go build -tags yara ./cmd/csm/
The rules directory and rule files must be owned by root or the scanner user and
must not be group- or world-writable. The standard service runs as root and
reads the rule files; the YARA workers get the rule text from it and never open
them. The installer uses 0750 for the directory and 0640 for shipped rules.
Custom rules must also remain readable by the service. With the default worker
enabled, rule files cannot include other files.
Place .yar or .yara files alongside YAML rules in /opt/csm/rules/. CSM compiles them at startup and uses them for:
- Real-time fanotify file scanning
- Scheduled deep-scan filesystem sweeps
- Email attachment scanning
Scans hand every file to YARA-X regardless of its name, so a rule may decide on the leading bytes rather than the extension. The bundled rule for PHP carried inside an image does exactly that: it requires a genuine PNG, JPEG, GIF, WebP, ICO, BMP or TIFF container, a PHP opening tag, and an execution or remote-fetch construct beside it. A picture is a working payload store, and nothing about its name says so.
Its other half is the loader: a PHP file that includes or requires a
non-executable file while reading request input. The extension list covers
images, archives and opaque data files; source partials such as .html,
.tpl, .txt and .svg are deliberately absent, because including those
can be ordinary templating. Literal targets must end the include statement;
an image named later in a rendered HTML expression is not an include target.
Encoded targets remain suspicious regardless of the hidden extension.
Findings include up to five distinct non-executable path literals near an
include or require. These are candidate payload references: proximity does
not prove variable identity, and concatenated paths may be partial. Extraction
advances through the content once and stops when the output cap is reached.
Native Linux programs are judged by content too. A program that carries a
wget or curl download command with an open output path, together with
either a URL to a PHP script or .htaccess file (including at the URL root),
or PHP source that writes files, is reported as a Critical dropper wherever
it sits. The ELF header decides what counts as a program: non-PIE, PIE and
static PIE executables do, shared libraries (native Node.js modules, Python
extensions) do not. The executable bit is not consulted. A program in a web root without those traits
is not reported: vendor download binaries, bundled build tools such as
esbuild, image optimizer binaries and old CGI programs are normal there.
Scheduled YARA sweeps scan regular, non-empty files under configured
account_roots, or cPanel /home/*/public_html roots when no override is
configured. Files larger than thresholds.full_scan_max_file_mb, symlinks,
and special files are skipped. An unreadable or oversized file, or a scanner
backend error, emits yara_scan_incomplete and preserves findings from the
previous complete sweep. Scheduled findings and real-time fanotify findings
have separate ownership, so one path cannot purge the other’s results.
A real-time scan that cannot inspect a changed file emits
yara_realtime_scan_error instead, so a scanning outage stays separable from
the scheduled coverage report. The dashboard’s Components matrix shows this
finding as Fanotify’s last event; scheduled coverage reports do not advance
that event time. This error finding is suppressed while the file monitor is
stopping: the YARA backend is stopped before the file monitor has finished
draining, and a clean restart is not an outage.
With signatures.yara_worker_enabled: true (the default), files are reported
as not scanned until the first rule load succeeds, including while a failed
startup compile is retried. An empty rules directory counts as loaded for
files, but mail parts are reported as not scanned until at least one rule
loads. A failed reload keeps the rules already loaded. A file or mail part
whose content made the scan worker fail twice within a day is temporarily
reported as not scanned without another attempt, including when copies are
already queued. A successful rule load clears these failures; they also
expire a day after the last failure. The oldest failures may be forgotten
earlier if many different inputs fail.
Without the yara build tag, YARA rules are not loaded or evaluated, and the
worker backend reports files as not scanned.
Updating Rules
Before merging bundled YARA changes, run the rules against an unpacked corpus of clean WordPress core and plugin files:
YARA_FP_CORPUS=/path/to/corpus go test -tags yara ./internal/yara/ -run TestRepositoryRulesAgainstCleanCorpus -v
The same corpus measures the YAML rules that only realtime and finding re-check run:
YARA_FP_CORPUS=/path/to/corpus go test ./internal/signatures/ -run TestRepositoryYAMLRulesAgainstCleanCorpus -v
The same corpus also checks that no YAML regex is skipped on a file it matches:
YARA_FP_CORPUS=/path/to/corpus go test ./internal/signatures/ -run TestYAMLGatesSoundOnCleanCorpus -v
Both gates require at least 5,000 non-empty files within the default scheduled scan size limit. Rule-load, traversal, and read failures fail the relevant run instead of counting as clean; YARA backend errors do too.
The measured YARA baseline is empty. The YAML baseline records rules that already fire on clean plugin and core code and are named in the realtime-rule porting backlog. Tighten a noisy rule rather than excluding paths or filenames.
csm update-rules # download latest rules and reload the running daemon
csm update-rules now asks the daemon to reload through the control socket once the download completes. If the daemon is not running, the next start picks the files up automatically. kill -HUP $(pidof csm) still works.
Or from the web UI: Rules page > Reload Rules button.
Remote rule updates are now signature-verified. Any configuration that enables signatures.update_url or signatures.yara_forge.enabled must also set signatures.signing_key to the 64-character hex-encoded Ed25519 public key that verifies the downloaded .sig files. Both URLs must use https; a plain-http URL fails validation.
A valid signature proves who published a rules file, not that it is the current one. The YAML updater therefore also refuses a download whose version is lower than the installed file’s, or that carries fewer than half as many rules, and keeps the installed rules. The daemon emits a Critical finding for either refusal so a stale mirror or a replayed old release is visible instead of silently stripping detection. A missing or unparsable installed file is not compared, so a signed update remains the way to recover from a corrupt rules file.
For an intentional reduction below half the installed YAML rule count, set signatures.allow_rule_count_decrease: true, restart CSM, and run the signed update. Then return the setting to false and restart CSM again; this exception never permits a version downgrade.
Remote update URLs must use https and must not point at localhost, loopback, link-local, unspecified, or RFC1918 / ULA private addresses.
YARA Forge Integration
CSM can automatically fetch curated YARA rules from YARA Forge, which aggregates and quality-tests rules from 40+ public sources including signature-base, Elastic, Malpedia, and ESET.
Configuration
signatures:
signing_key: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
yara_forge:
enabled: true
tier: "core" # core (5K rules, low FP), extended (10K), full (12K)
update_interval: "168h" # weekly
download_url: "https://mirrors.pidginhost.com/csm/yara-forge/{version}/yara-forge-rules-{tier}.zip"
disabled_rules: # rule names to switch off, in Forge and in the shipped rules
- SUSP_Example_Rule
The project operates the signed mirror shown above. A ready-to-use drop-in is shipped at /usr/lib/csm/profiles/yara-forge.example.yaml; copy or include it under /etc/csm/conf.d/ to enable Forge without editing the main csm.yaml. The matching signing_key is the project Ed25519 public key (hex), published on the release signing page.
signing_key must be a hex string for the Ed25519 public key that matches the private key used to sign the remote Forge artifact. It is not a PEM block and not a file path.
YARA Forge’s upstream GitHub releases publish ZIP files, but not CSM detached signatures. CSM therefore requires yara_forge.download_url to point at a mirror you operate. The URL may contain {tier} and {version} placeholders. The detached signature must be available at the resolved ZIP URL plus .sig.
When the URL has a standalone {version} path segment, CSM first reads a plain-text latest pointer from that version directory’s parent, for example https://mirrors.pidginhost.com/csm/yara-forge/latest. If the pointer is not published, CSM falls back to the upstream GitHub release tag. Mirror network errors and server errors fail the update instead of falling back, because GitHub may name a release the mirror has not signed yet. CSM accepts only short tag tokens from the pointer; the ZIP signature still gates installation.
If you do not have a signed update source yet, disable remote updates instead:
signatures:
signing_key: ""
update_url: ""
yara_forge:
enabled: false
Tiers
| Tier | Rules | Size | False Positive Risk |
|---|---|---|---|
| core | ~5,000 | 1.6 MB | Low (quality >= 70, score >= 65) |
| extended | ~10,500 | 3.3 MB | Medium |
| full | ~11,600 | 3.7 MB | Higher (includes score >= 40) |
Update Flow
- CSM resolves the latest YARA Forge version from the mirror pointer, falling back to GitHub only when no pointer is published
- If different from the installed version, downloads the ZIP for the configured tier from
yara_forge.download_urland its detached signature - Verifies the download against
signatures.signing_key - Filters out any rules listed in
disabled_rules - Compile-tests the rules with YARA-X before installing, in the compile unit when worker mode is enabled
- Atomically replaces the previous Forge rules file
- Reloads the YARA scanner
Custom rules in malware.yar are never overwritten by the Forge fetcher.
Disabling Rules
If a rule produces false positives, add its name to disabled_rules in the config and restart the daemon:
signatures:
disabled_rules:
- SUSP_XOR_Encoded_URL
- php_goto_obfuscation
The list covers every rule CSM loads, not only YARA Forge: Forge rules are
stripped from the download, rules shipped in malware.yml are skipped when
the real-time engine loads them, and rules shipped in malware.yar are
stripped before the scheduled engine compiles them. The self-test measures
the ruleset that is left, so disabling a rule shows up as the coverage it
costs.
Names are matched in full, ignoring surrounding whitespace and letter case; prefixes and substrings do not match. Removing a YARA rule preserves neighboring rules, including when declarations share a line or literals contain braces. The daemon strips disabled rules before every compile, so rule reloads and worker restarts keep honoring the list.
A valid ruleset whose rules are all disabled loads as an empty set, including on reload; stale rules are not retained. The self-test reports the resulting misses. An empty or invalid replacement still reports a load error.
csm validate warns about a name that matches no rule, because a typo here
otherwise reads as “that rule is off” while the rule keeps firing. Disabling
a rule is a last resort and a standing gap in coverage; prefer fixing the
rule.
Validation recognizes a disabled YAML rule even if its regular expression is invalid, and counts repeated names only once.
Signature settings require a daemon restart. SIGHUP does not apply a changed disabled list; rule-file reloads keep the current list.
How Rules Avoid False Positives
Signature rules require structural nesting, not co-presence of strings. Two dangerous function calls appearing in the same file but in unrelated code paths won’t trigger a rule. The call must directly wrap or chain with the other for a match.
YARA-X rules cannot express nesting, so multi-string rules state how their evidence is related. A bounded distance is used when closeness is part of the malicious shape. When valid malware can carry padding between its signals, PHP diagnostic records are kept as separate contexts so one record cannot borrow evidence from another. Anchoring works the same way: a CGI webshell rule requires its shebang at offset 0, because that is where the web server needs it. None of these controls refers to a file’s name or path.
Realtime signature auto-quarantine adds a safety gate: only webshell and dropper matches are eligible, and the file must be at least 512 bytes and either have Shannon entropy >= 5.5 or hex density > 20% plus an obfuscated-execution signal. Legitimate plugin code (well below 5.5 entropy) passes through; obfuscated malware (5.8+) is caught.
Alert Rate Limiting
Default: 30 operator alert dispatches/hour (configurable via max_per_hour). CRITICAL findings and threat-intel reputation sightings always get through by email or generic webhook regardless of the rate limit, and they do not count against it. Other lower-severity alerts are rate-limited, including when they are batched with a CRITICAL finding: once the budget is spent, only the urgent findings in that batch are sent.
A dispatch uses one slot only when email or a generic webhook successfully delivers routine findings. If only urgent findings reach a channel and routine delivery fails, the slot remains available for a later dispatch. Email-disabled findings do not reserve a slot unless a generic webhook will carry them. The phpanel webhook stream bypasses this budget.
Summary-class findings never reach the cap: attacks CSM already blocked or challenged, its applied block and challenge notices, and the other findings listed under Alert classes in the configuration reference go to the daily summary instead of email or the generic webhook.
Suppressions
Create suppression rules to silence known false positives:
- From the Findings page: click the suppress button on any finding. The path pattern is a glob; the dialog pre-fills the finding’s own file with
[,],*,?and\escaped, so the rule matches that file only - From the Findings page with several findings selected: bulk Suppress creates one rule per selected file
- From the Rules page: manage suppression rules directly
- Via API:
POST /api/v1/suppressions
A suppression rule hides matching findings from the Findings page, stops their email and webhook alerts, and stops file, process and account remediation for them. It does not stop IP blocking, challenge routing or attack scoring: a rule that mutes a whole check would otherwise leave every attacker that check reports unblocked. To exempt an address that was blocked by mistake, whitelist it on the Firewall page under Allow Rules or with csm firewall allow.
Incident auto-blocking, credential-spray containment and central threat intelligence also use suppressed findings. Database response never blocks session IPs; a suppressed database finding cannot trigger cleanup or session revocation. IP action notifications are separate findings; suppressing their check type mutes those notifications without stopping the action. These rules apply to startup, scheduled and real-time scans, control-socket runs with alerts enabled, and replay after restart.
To suppress email alerts for specific checks while keeping them visible in the web UI, use disabled_checks in your config:
alerts:
email:
disabled_checks:
- "email_spam_outbreak"
- "perf_memory"
Email AV
CSM scans email attachments in real-time using ClamAV and YARA-X on the Exim mail spool.
How It Works
- fanotify watches the Exim spool directory for new messages, including every
split_spool_directoryhash subdirectory (the cPanel default layout); hash directories Exim creates later are picked up within a minute - Attachments are extracted and scanned by ClamAV (socket) and YARA-X (if available) Base64 and quoted-printable decoding tolerates stray characters, missing padding and malformed escapes. Encoded multipart bodies are decoded at every level. Ambiguous base64 is scanned under quartet and alphabet-only interpretations, and a valid padded prefix is scanned separately when data follows it. At most 16 additional interpretations are scanned per message. If decoding or multipart parsing reports an error, recovered content and later attachments are still scanned and the message is reported as incompletely scanned. Extraction size and MIME nesting limits still apply
- Zip and tar.gz attachments are unpacked within configured size and file limits
- Attachments are unpacked in a separate locked-down process with no network access, one per concurrent scan (
scan_concurrency), started on first use. Extracted parts reach the scanners in memory and are never written to disk; each scan holds at mostmax_extraction_sizeof them. A process that times out, runs out of memory, crashes or sends an invalid reply makes the message an incomplete scan. Outside observe mode, CSM requests deferral whateverfail_modesays. Deferral requires a fanotify permission event that is still holding delivery; notification-only mode cannot block it - Attachment names written to logs and the UI use sanitized, shortened base names. Archive detection uses the full name, so shortening a displayed name does not prevent unpacking
- Infected messages raise a Critical finding; with
quarantine_infected: truethey are also quarantined with full metadata instead of delivered - Sender, recipient, and message ID are logged
Quarantine and release preserve each queue file’s ownership, permissions, and
modification time, including when the move crosses filesystems. Release returns
the message to the queue directory it came from, including the hash directory
Exim uses when split_spool_directory is on.
Release accepts the flat queue root or the message’s hash directory, named for the sixth character in both 16- and 23-character Exim IDs. Other subdirectories and hash-directory symlinks outside the trusted queue roots are rejected.
Quarantine persists recovery metadata before moving spool files and syncs file contents and directory changes. A failed rollback retains the remaining quarantine files. Release restores the body before publishing the queue header; storage failures retain metadata and report the affected paths. Inspect both the spool and quarantine locations before retrying a partially completed operation.
Web UI
The Email page (/email) shows:
- AV watcher status (active, engine health)
- Scan statistics (scanned, infected, quarantined)
- Quarantined email list with release/delete actions
API Endpoints
GET /api/v1/email/stats Scan statistics
GET /api/v1/email/quarantine Quarantined email list
GET /api/v1/email/av/status AV watcher status
POST /api/v1/email/quarantine/ Release or delete quarantined email
Related Checks
email_content- scans outbound email body for credentials and suspicious URLs; every base64 body and part, attachments included, is decoded with the same tolerance as attachment scanning, using live MIME headers and the declared multipart boundariesemail_weak_password- detects email accounts with weak passwordsemail_forwarder_audit- audits forwarders for exfiltration redirectsmail_queue- alerts on queue buildup (spam outbreak indicator)mail_per_account- per-account sending volume spikes
Email password audit
The deep scan checks mailbox passwords against account-derived candidates and the bundled weak-password list. Verification runs inside CSM; passwords and stored hashes are never passed to subprocesses or included in findings. Confirmed matches can still use the HIBP range API for breach counts, sending only a SHA1 prefix.
Supported stored formats follow Dovecot password schemes:
| Scheme | Accepted format and audit limit |
|---|---|
| CRYPT, SHA512-CRYPT, SHA256-CRYPT | SHA crypt with a nonempty salt of up to 16 characters; 1,000 to 1,000,000 rounds, or the standard 5,000 when omitted |
| CRYPT, MD5-CRYPT | MD5 crypt with a nonempty salt of up to 8 characters |
| CRYPT, BLF-CRYPT | bcrypt 2a, 2b, or 2y; cost 4 through 14 |
| ARGON2I, ARGON2ID | Version 19; at most 64 MiB memory, 4 passes, and 4 lanes; salt 8 to 64 bytes, digest 16 to 64 bytes |
| PLAIN | Plaintext |
| PLAIN-MD5, LDAP-MD5, SMD5 | MD5 digests; salted variants allow 1 to 64 salt bytes |
| SHA, SHA1, SSHA, SHA256, SSHA256, SHA512, SSHA512 | SHA digests; salted variants allow 1 to 64 salt bytes |
Unprefixed hashes use CRYPT. cPanel shadow files, which add password aging
fields after the hash, are read the same way. Scheme names and .hex, .b64, and .base64
encoding suffixes are case-insensitive. Unsalted digests also accept Dovecot’s
hex/base64 autodetection. DES crypt, bcrypt 2x, yescrypt, PBKDF2, and
mechanism-specific formats such as SCRAM are not audited.
Stored values are limited to 4 KiB and candidates to 256 bytes without embedded NULs. At most three hash calculations run concurrently. Cancellation returns promptly; a calculation already running keeps its worker slot until its bounded work finishes.
An unsupported, malformed, or over-budget hash produces
email_password_audit_incomplete. CSM keeps earlier password findings and does
not record that mailbox as successfully audited. Other mailboxes continue to be
checked. When these are the only failures, the audit waits for
password_check_interval_min before trying again; temporary verification failures
remain eligible for retry on the next deep scan. Cycles skipped by the interval
preserve both earlier password findings and the audit warning. A completed audit
replaces them with its current results. Upgrading
starts a fresh audit even when an older CSM version recorded the same hash.
Inbound spam filtering
CSM can filter inbound mail through rspamd on cPanel hosts, where each cPanel account decides for itself whether its mail is filtered, and on Postfix hosts, where every local domain is filtered. It is off by default.
The operator installs rspamd. CSM configures it, wires it into Exim on cPanel, keeps the integration current and reports when filtering is not working. CSM is never in the live mail path: Exim or Postfix talks to rspamd directly, and stopping CSM leaves the installed filtering in place.
Most of this page describes cPanel hosts. Postfix hosts lists what differs there.
What each account gets
The switch is cPanel’s own Spam Filters page, “Process New Emails and Mark
them as Spam” (the ~/.spamassassinenable file). New accounts start opted in.
Who may change the switch is controlled by the spamassassin feature in cPanel
feature lists; CSM adds no setting for it.
With dry_run: false, inbound mail is scored by rspamd and Exim applies the
account’s choice. For an opted-in account, Exim:
- rejects it with 550 when the score is at or above
reject_score; - adds
X-Spam-Status: Yeswhen the score is at or abovetag_scoreand belowreject_score. If the user turned on cPanel’s spam box, the mail is filed in the Spam folder; otherwise it is delivered normally with the header. When the forward guard enforcesspam_flagged, a copy forwarded to an outside address is held instead of relayed.
Mail for an opted-out account is delivered with no CSM rejection and no spam verdict header. Opting out never affects antivirus, outbound checks, SMTP-time controls such as RBLs, rate limits and anti-spoofing, or CSM’s detections.
A change to the switch applies from the next recipient Exim accepts. CSM does nothing and Exim does not reload.
The Spam Filters page’s own score threshold and allow and deny lists are not
used: reject_score and tag_score apply to every opted-in account.
Who decides for a message
Policy follows the account that owns the addressed domain, including aliases, forwarders and catch-alls. It is not re-evaluated for the mailbox a forwarder finally delivers to:
- a forwarder in an opted-out account delivers unfiltered into an opted-in mailbox it points at;
- an address in an opted-in account can reject mail its forwarder would have passed to an opted-out mailbox.
When one message has recipients with different choices, Exim accepts the first recipient and answers 452 for each later recipient whose choice differs. The sending server retries those recipients in a separate delivery, which follows their own choice.
If /etc/global_spamassassin_enable exists, cPanel treats every account as
opted in and no account can opt out. CSM reports this as a warning.
Mail that is never scanned
Only unauthenticated mail from remote hosts to the host’s local domains is
scanned. Authenticated submissions, local mail and these peers are not:
127.0.0.0/8, ::1, trustedmailhosts and cpanel_mail_netblocks, the same
set cPanel exempts from its own spam scan.
cPanel also accepts some recipients before the scan hook runs, so they are
never filtered: mail to a domain from that domain’s remote MX hosts, hosts in
skipsmtpcheck_hosts, always-relay and pop-before-SMTP hosts, Mailman
-bounces addresses and suspended recipients.
Before you enable it
This section is for cPanel hosts. Postfix hosts have their own requirements.
Install rspamd from the upstream rspamd.com
repository and keep its systemd unit’s User= setting (the upstream package
runs as _rspamd); CSM checks the scan socket’s owner against it. CSM does not
install rspamd or Redis. A first install waits for the conditions below; a
definite failure after installation removes owned integration content when safe.
-
SpamAssassin is off. Disable spamd in WHM so
/etc/spamddisableexists. Otherwise cPanel’s own scan uses CSM’s scanner and adds its own headers and subject tag whatever the account chose. -
A scan socket for Exim. Add a Unix socket to rspamd’s normal worker, in
/etc/rspamd/local.d/worker-normal.inc:bind_socket = "/run/rspamd/csm-normal.sock mode=0666";This replaces the normal worker’s TCP listener. The socket and every parent directory must be owned by root or the rspamd user. Parent directories must not be writable by group or others, so a hosting account cannot replace the listener. Extra ACL entries on the socket or any parent directory block installation. Exim’s runtime user must be able to search every parent directory and write to the socket;
mode=0666grants socket write access. -
rspamd accepts every message Exim accepts. rspamd’s default
max_message(50 MiB) equals cPanel’s default Exim limit, and Exim adds headers before scanning. Setmax_messageto Exim’smessage_size_limitplus at least 1 MiB in/etc/rspamd/local.d/options.inc, for examplemax_message = 60mb;with cPanel’s 50M default. -
Validate and reload rspamd (
rspamadm configtest, thensystemctl reload rspamd). -
Exim is running. CSM probes an Exim listener before and after each change.
CSM also refuses to install while any of these hold, and names the cause:
- Exim settings, ACLs or filters that already act on spam scores, such as
deny_spam_score_over_*,ratelimit_spam_score_over_*or automatic spam deletion and forwarded-spam restrictions; - another
spamd_addressin/etc/exim.conf.local; - cutthrough delivery, PRDR or CHUNKING enabled;
- an Exim main log other than the one CSM reads (
log_file_path); - an Exim or rspamd unit that cannot reload;
- disabled custom ACL hooks or invalid named lists used by the hooks;
- unsafe mutation targets or malformed CSM markers in the locations CSM manages;
- an rspamd override that defeats CSM’s settings below.
CSM does not change WHM options, account toggles, Redis or rspamd worker settings to satisfy these conditions.
Configuration
email_protection:
antispam:
enabled: false
dry_run: true
rspamd_socket: /run/rspamd/csm-normal.sock
reject_score: 15
tag_score: 6
feeds:
enabled: []
spamhaus_dqs_key_file: ""
exempt: []
enabledturns the integration on. Setting it back tofalserequests removal of CSM’s owned content; conflicts or failures can prevent removal and produce findings. On a host that is neither a cPanel host running Exim nor a Postfix host it stays inactive; the daemon logs a warning at startup and whenever a configuration change is reloaded. Once such a host runs Postfix, for example after Postfix was installed or started after CSM, the next reload (systemctl reload csm) starts antispam without a restart.dry_runscans and logs what would happen without rejecting, tagging or splitting recipients. On a Postfix host CSM’s rspamd files already apply in dry run, and the host’s own action thresholds decide on CSM’s scores; see Dry run and enforce.rspamd_socketis the scan socket above: an absolute path of at most 107 characters using letters, digits,/,.,_and-.reject_scoreandtag_scoreare whole rspamd scores with0 < tag_score < reject_score <= 1000.feedsselects optional blocklists; see Feeds.exemptlists recipient domains and mailboxes that are never filtered on a Postfix host, such asexample.comorpostmaster@example.net, and whose Junk folders are never learned from. Entries are lower-cased, and only plain domains and mailboxes are accepted. cPanel hosts ignore it: there each account’s Spam Filters switch decides. See Which mail is filtered.
All of these are hot-reloadable: after antispam starts, systemctl reload csm or a
Settings save updates health tracking and requests a reconcile pass before
forward-guard reconciliation. Settings saved before antispam starts are picked
up when it initializes, after the startup forward-guard reconciliation.
enabled: true is refused in observe mode.
rspamd_socket is not the reputation.rspamd controller URL. That optional
setting reads rspamd’s scan history for IP reputation and is unrelated to
filtering.
The capabilities API lists mail.antispam.v1 when a build supports this
integration, and mail.antispam.postfix.v1 when it also supports Postfix
hosts; neither means filtering is installed on the host.
mail.antispam.learning.v1 means the build can learn from Spam folders,
and mail.antispam.learning.postfix.v1 that it can also learn from Junk
folders on Postfix hosts; while learning runs, csm status --json carries
an antispam_learning block.
Feeds
Feeds and map redirects need rspamd 4 or later. On an older or unreadable rspamd version CSM leaves rspamd’s own lists and maps alone and reports it.
Every feed is off until you name it:
email_protection:
antispam:
feeds:
enabled: [psbl, mailspike]
spamhaus_dqs_key_file: "" # only for spamhaus_dqs
enabled is a list of feed names; an unknown or repeated name, or
spamhaus_dqs without a clean absolute spamhaus_dqs_key_file, makes the
configuration invalid. Check each list’s terms against your use before you
enable it:
| Name | Terms |
|---|---|
mailspike | Free for sites under 100,000 messages and 100,000 DNS queries a day; contact Mailspike above that. |
psbl | Free for anyone. |
uceprotect1 | Free for private and commercial use; above 100,000 requests a day UCEPROTECT asks for its rsync feed. Level 1 also lists some forwarders that break SPF. |
nordspam | Free for commercial and non-commercial use; contact Nordspam above 10,000 queries a day. |
sem | Free; for scoring only, never to reject mail on its own. Tell Spam Eating Monkey before going above 100,000 queries a day. |
spamcop | Free; SpamCop calls its list aggressive and recommends tagging, not rejecting. |
barracuda | Free after registering the addresses of the resolvers that query it with Barracuda. |
dnswl | Free through DNS under 100,000 queries a day; its pages conflict on non-commercial use. A listing lowers the score slightly. |
uribl | Free for low-volume use only; URIBL blocks public and large shared resolvers. |
surbl | Free only for organisations under 1,000 users or 250,000 messages a day, and never inside a product or service sold for a fee; other use needs SURBL’s paid data service. See the SURBL usage policy. |
spamhaus_dqs | Needs a DQS key. The free key covers non-commercial use under 100,000 queries a day; commercial use needs a paid key. |
csm_ip_map | Local data only: CSM’s block history, its threat feeds and the spam evidence it records. |
rspamd_maps | Keeps rspamd’s own map downloads from maps.rspamd.com; free use is non-commercial and under 5,000 queries a day, other use needs an agreement with rspamd. See rspamd’s usage policy. |
Maps without any feed
Without rspamd_maps, CSM redirects the rspamd.com maps it manages to
local files: the copies the rspamd package installs in /etc/rspamd/maps.d,
refreshed by package updates. The freemail and disposable-address rules stay
quiet until you put your own lists in
/etc/rspamd/local.d/maps.d/freemail.map and
/etc/rspamd/local.d/maps.d/disposable.map. Phishing exceptions read
/etc/rspamd/local.d/maps.d/phishing_whitelist.inc when it exists.
This does not block network access to rspamd.com. Operator-owned files and additional maps can keep downloads active; CSM reports those it finds in enabled modules after the integration is installed and current.
DNS feeds need a local resolver
Some blocklists refuse queries from public resolvers. CSM enables a DNS feed
only when every nameserver rspamd uses is loopback, private, or inside
the top-level infra_ips. rspamd uses dns.nameserver from its options, or
/etc/resolv.conf when that is unset.
When your own resolvers sit on public addresses and refuse other networks,
list their addresses or CIDRs in infra_ips; hostnames do not count for this
check. infra_ips changes need a CSM restart, not a reload. Otherwise run a
recursive resolver on the host and add this line to
/etc/rspamd/local.d/options.inc:
dns { nameserver = ["127.0.0.1"]; }
cPanel’s local BIND usually recurses for localhost; cPanel’s PowerDNS does
not. When rspamd uses the systemd-resolved stub (127.0.0.53 or 127.0.0.54
on port 53), CSM asks systemd-resolved for every server it can forward to,
including its fallback servers when no other server is configured, and
judges those. If CSM cannot read that list, DNS feeds stay off with a
warning. Other ports on the stub addresses count as separate local
resolvers. When PowerDNS holds port 53, run the resolver
on another port and name it with that port, for example
dns { nameserver = ["127.0.0.1:5353"]; }. A loopback or private address passes CSM’s
address check, but a resolver that forwards to a public resolver can still
be refused. CSM never edits options.inc. After changing it, run
rspamadm configtest, reload rspamd, then reload CSM to request a check.
Resolver lists accept master-slave:, round-robin:, random:, and hash:
prefixes without regard to case. Rspamd 4.2.2 does not support sequential:;
CSM leaves DNS feeds off for that setting. A warning about an unreadable
/etc/resolv.conf applies only when no explicit nameserver setting exists;
invalid explicit settings get their own warning.
Spamhaus DQS
Put the key alone in a root-owned file, mode 600, and name the file in
spamhaus_dqs_key_file. CSM reads it on every check, never logs it, and
writes it only into /etc/rspamd/override.d/rbl.conf, mode 640, owned by root
and rspamd’s group. CSM must know that group: set User= (and optionally
Group=) in the rspamd unit. A root-started rspamd that changes user itself
leaves the group unknown, and the feed stays off.
rspamd writes the name of every failed lookup, which contains the key, to its
log. CSM keeps spamhaus_dqs off, with a feed warning, while that log file
can be read by other users, which is the rspamd package’s default. Make it
private, for example chmod 750 /var/log/rspamd and chmod 640 on the files
in it, and set rspamd’s log rotation to create new files with mode 640. The
DQS rules also run without rspamd’s own list monitoring. CSM probes ZEN’s
test points and reports failures without stopping lookups. Turn the feed off
when a critical warning reports listings for a never-listed test point.
The log and its directories must be owned by root or rspamd, with no extra
ACLs or write access for other users. A private file or enclosing directory
must restrict readers to root and rspamd, including group access. Symlinks
and paths with dot components cannot establish privacy. CSM cannot verify
syslog or console routing; use logging.type = file with a private log.
The worker rechecks log privacy and rspamd version before applying feeds.
CSM IP map
csm_ip_map gives rspamd CSM’s own evidence about sender addresses. CSM
rebuilds the map at start and every 10 minutes into
/var/lib/csm-rspamd/csm_ip.map, and rspamd scores listed senders through
one rule, CSM_IP:
| Symbol | Listed when | Score |
|---|---|---|
CSM_IP_DROP | the address is in a Spamhaus DROP network from CSM’s threat feeds | 4 |
CSM_IP_BLOCKED | CSM blocked the address permanently | 3 |
CSM_IP_MAIL_AUTH | the address attacked SMTP or mail logins in the last 30 days | 3 |
CSM_IP_SPAM | its spam evidence score is 50 or more, see Spam evidence | 3 |
CSM_IP_BLOCKLISTDE | blocklist.de lists it in CSM’s threat feeds | 2 |
CSM_IP_CINS | CINS Army lists it in CSM’s threat feeds | 2 |
Each address or network gets one line with its strongest symbol, so the map
adds at most 4 to a message. Whitelisted addresses (operator, temporary and
reputation.whitelist), infra_ips, private, loopback and link-local
addresses and the server’s own addresses are never listed, and neither is a
network that contains one of them.
IPv4-mapped IPv6 addresses and CIDRs in infra_ips protect the same senders
as their IPv4 forms. Infrastructure hostnames use the last successful DNS
answers from CSM’s firewall resolver. This requires a running CSM firewall;
without it, use literal addresses or CIDRs. DNS changes affect the map on
its next rebuild and spam evidence on the next scan record. Entries only in
firewall.infra_ips grant firewall access without suppressing spam evidence
or map listings.
The feed stays off with a warning until the map exists as a regular root-owned file, mode 0640, in rspamd’s service group, without extra ACLs. Every parent must be a root-owned directory with no group or other write access or extra ACLs, and rspamd must be able to traverse it. Symlinks cannot establish safe map access.
CSM must know rspamd’s group, as for DQS above. The worker rechecks map access and rspamd version before applying a queued request. A failed check leaves the installed configuration intact. The next regular check retries with the current host facts.
The feed needs CSM to own /etc/rspamd/local.d/multimap.conf, even with
rspamd_maps selected. CSM replaces an operator-owned file there, see
Replaced files; one that is not safe to replace leaves the
feed off.
Like the other feeds, it needs rspamd 4 or later.
CSM checks that rspamd uses the current map by scanning a canary message
from 192.0.2.255 every 30 seconds while the rule is in use. The scan stays
out of rspamd’s history and statistics. antispam_ip_map reports a map CSM
could not write, or two consecutive probes showing rspamd still not using
the current map after its 5-minute load grace; a matching answer clears it.
Earlier probe failures are forgotten when the map is removed, monitoring is
disabled, observe mode is entered, or the scan socket changes.
Dismissing the warning acknowledges the current outage; recovery clears the
dismissal so a later failure can notify again.
A failed write keeps the previous map, which rspamd keeps using. A missing
map, a changed file mode or a changed
rspamd group makes CSM write the map again even when its content is the
same. Once no installed rule reads the map, CSM deletes it; a failed removal
raises antispam_apply_failed and is retried on the next pass.
If removal fails and a later apply reuses the surviving map, CSM resumes
monitoring it and subtracts its score from new spam evidence.
Spam evidence
While antispam is enabled, CSM reads each scan result Exim logs and records
a verdict about the sending address: spam when the score reaches the reject
score, ham when it stays below the tag score, and nothing in between or when
the scan failed. The score the CSM IP map gave the sender is subtracted
first, so the map never feeds its own evidence. Dry run and enforce record
the same way, and so do recipients with filtering off: the verdict is about
the sender. Whitelisted, infra_ips, private and the server’s own addresses
are skipped. Postfix hosts record no verdicts; see Scan records.
This works with or without csm_ip_map, so you can review the data before
turning the map on. After a CSM restart with csm_ip_map enabled, verdicts
are not recorded until CSM has written the map again, because the map’s
share of a score is unknown until then.
A spam verdict creates or updates the sender’s record on the Threat page.
Ham only updates a record that already exists, so busy legitimate senders do
not fill the attack database. The spam score is the share of spam among
recent verdicts, each verdict losing half its weight every 7 days; two recent
spam verdicts and no ham reach 50, which lists the sender as CSM_IP_SPAM.
Spam evidence never blocks an address, never changes its threat score and is
never reported to AbuseIPDB.
CSM forgets a record that holds only spam evidence once its spam and ham verdicts have both lost nearly all their weight: about 47 days after a single spam message, later for a sender with more verdicts. Ham on the record keeps it, so the next spam from a mostly clean sender does not list it. A record that also has attack findings, a block or a mail login attack in the last 30 days is kept. Every record leaves 90 days after its last spam verdict or finding; ham does not extend that limit. Stored evidence without a decay timestamp keeps its weight, as it does when scoring the sender. CSM checks at start and with every deep scan, hourly by default. The spam evidence budget can drop a record that holds only spam evidence sooner.
An IPv6 sender is recorded and listed per /64 network, for example
2001:db8:1:2::/64, so a sender that rotates addresses builds one score.
IPv4 senders, including IPv4-mapped IPv6 addresses, stay per address.
Clearing or whitelisting an address inside the network forgets the network’s
evidence, and the CSM IP map does not list a network that holds an excluded
address. Evidence recorded per IPv6 address by an earlier release stays
until it expires or the spam evidence budget drops
it.
With spam in reputation.report.classes, a sender that rspamd keeps
rejecting, and that has sent no clean mail recently, is reported to the
central abuse database. Senders of the shared mail services are never
reported. Dry-run nodes count their verdicts too. See
Abuse Reporting.
The Threat page’s Spam column shows spam and ham counts and the spam score, with a badge once the score reaches the listing threshold of 50. The badge does not confirm a listing: the feed may be off, and the map still skips excluded addresses. The “Spam evidence only” filter lists senders by spam score. The IP lookup shows mail login attacks and spam verdicts.
Spam evidence budget
The spam evidence budget is 50,000 records, about 22 MB of memory for records held only for spam evidence. It is fixed and has no setting. Each time CSM records a verdict, it keeps those records at or under that budget, counted at the verdict’s observation time. A missing time uses the current time. Invalid addresses and ham from an untracked sender are ignored and do not trigger the budget.
Startup loads unexpired evidence without trimming it to the budget. The count can therefore exceed the budget until a verdict is recorded. It can also grow as login evidence ages: a record with a mail login attack counts toward the budget only when that attack is more than 30 days old at the verdict’s time and the record has no attack findings or block. Records with attack findings, a block or still-protected login evidence never count toward the budget and are never dropped for it. A delayed verdict can leave more than 50,000 such records at the current time because it counts them at its own earlier time.
When a verdict is recorded and the count is over the budget, CSM drops records until nine tenths of the budget remain, rounded down. Each record has a level: the weight its verdicts still have, rounded up to whole messages, and at least 2 once it has two or more verdicts. Records of the lowest level go first. Inside a level, records in the network that holds the most spam-only records go first, counted per IPv6 /48 and per IPv4 /16 at the start of the pass. When those counts tie, the record with the least weight goes first.
A flood of single messages from one network, one /48 or /16, loses its own records first once that network holds more spam-only records than any other. A flood spread over many networks competes with the other records of the same level: the most crowded networks go first, then the records with the least remaining weight when network counts tie, whoever sent them. A flood of single messages cannot remove a sender with two or more verdicts, a listed sender or a sender’s ham history while records with two or more verdicts fill less than nine tenths of the budget; above that, one pass trims the weakest of them once. Higher levels are reached only after every lower-level record has been dropped; they can still be dropped when they alone exceed the trim target. A flood that sends as many messages from each address or /64 as a level competes with every record of that level, listed senders included. Senders that share a crowded /16 or /48 with a flood can lose records of the same level with it.
CSM writes a warning to the daemon log at the first drop, then at most once per day of recorded verdict time while drops continue. It also queues the warning for alerts; a full queue can refuse it. The warning gives the number of records dropped, names the most crowded network and says how many records that network held. The count in a later warning includes everything dropped since the previous one. Drops made after a warning are reported by the next warning. It comes with a later drop or with a deep scan, once a day has passed since the previous warning. A restart forgets drops not yet reported. Evicted records are removed from storage on the next successful flush; they can return after a crash before that flush. The API threat statistics include the total dropped since the daemon started.
Look at the named network and its count. The count is how many records holding only spam evidence the network had when the drop began, not how many of the dropped records were its own. The count alone does not show whether the network sent the flood. Inspect its senders; if the network is hostile, block it in the firewall. Records with attack findings, a block or a mail login attack in the last 30 days are never dropped by this budget. Records with two or more verdicts are dropped only once such records fill nine tenths of the budget.
Scores
No feed symbol weighs more than 5, and the symbols of one feed add up to at most 14, so no feed alone reaches the default reject score of 15. Each symbol counts once per message. SURBL’s four scoring categories weigh 3.5 each; Spamhaus DQS gives 5 for ZEN, 3 and 2 for the DBL categories and 3 and 1 for newly seen domains. URIBL’s grey and red lists weigh 0.5 because they also list domains that legitimate bulk senders use.
Updated feed weights apply on the next successful reconcile while antispam is enabled and observe mode is off.
Files
CSM owns these files while a feed or map redirect needs them:
/etc/rspamd/override.d/rbl.confand/etc/rspamd/override.d/rbl_group.conf, with a DNS feed. They replace rspamd’s own lists and their weights, so only the feeds you named run;forged_recipients.conf,mid.conf,mime_types.conf,whitelist.confandmultimap.confin/etc/rspamd/local.d, withoutrspamd_maps. The local IP feed also needsmultimap.confwhenrspamd_mapsis selected.
CSM replaces an operator-owned file at one of these locations while it
needs the file, see Replaced files. A file that is not
safe to replace makes CSM omit the DNS feeds or the affected map redirect
instead. An operator file CSM does not need stays, and its contents can
still conflict with required settings and block core filtering. In
particular, an override.d/rbl.conf that enables the RBL module defeats
CSM’s attempt to turn it off when no DNS feeds are usable. Resolve that
conflict before enabling antispam.
Feed findings
Each antispam_feed finding names the feed, rspamd_maps for maps, or
rbl for loaded DNS-rule checks:
- feed off: the reason is in the details, such as a public resolver, a key file with the wrong owner or mode, or a file CSM cannot own. Fix it; the feed returns on the next check.
- map redirect skipped: CSM could not manage that optional file. Check the operator’s map settings; they may already use local copies. Move those settings out of CSM’s target file if CSM should manage the redirect.
- rspamd.com paths still loaded: rspamd still fetches those maps from
rspamd.com. Redirect the named maps
yourself, or enable
rspamd_mapsonly if your use meets rspamd.com’s terms. - loaded rules differ during a routine check: rspamd runs a different set of blocklist rules from those CSM wrote, usually through an operator override. This is a warning only; CSM leaves the installed filtering in place. Correct the overrides, validate and reload rspamd, then reload CSM.
- IP-map rule differs or cannot be verified: with
csm_ip_mapenabled, CSM checks that multimap is enabled and thatCSM_IPuses the managed map, expected score and complete symbol set. Symbol order does not matter. Extra rule options that can change matching or force an action are conflicts. Failed configuration dumps also produce a warning. Routine checks leave core filtering installed; correct the override or command failure before retrying an apply. - refuses CSM’s queries: the list answered with an error code, such as a refused resolver, a query limit or a rejected key. Those error-code answers add no score.
- does not answer: three checks in a row got no answer through rspamd’s resolver. Check that the resolver recurses and can reach the list.
- unexpected answer: the test point returned an unrecognised answer. Check the provider and resolver; only the configured listing codes can score. A listed test point returning both listing and refusal codes also raises this warning; it does not prove that the list stopped scoring.
- answers for addresses it never lists (Critical): turn the feed off and reload CSM. CSM’s probes only report findings; they do not disable feeds. A listing code still triggers this warning when the reply also contains a refusal code, since rspamd can score the listing code independently. rspamd stops querying a list only after its own monitoring detects it. DQS monitoring is disabled, so its lookups continue until you turn it off.
An apply that fails at feeds_check is reported as antispam_apply_failed
and attempts to roll back. CSM omits DNS feeds and csm_ip_map from subsequent
applies until the enabled feed names or DQS key-file path changes, or CSM
restarts, and does not rewrite the IP map meanwhile. Fix the override before
retrying. Reloading unchanged settings or replacing the key’s contents does not
clear this hold.
CSM probes only while antispam is enabled outside observe mode and the installed integration is current without blocking preconditions. It checks two test names per supported zone through rspamd’s first nameserver, on the port rspamd uses, looking up A records. It does not probe SEM’s IPv6 zone or Spamhaus DBL and ZRD. A healthy probe result does not establish the health of those zones or of other configured resolvers.
What CSM writes
- three Exim ACL hooks under
/usr/local/cpanel/etc/exim/acls(custom_begin_spam_scan_check,custom_begin_spam_scanandcustom_begin_check_message_post), inside# CSM-ANTISPAM BEGINand# CSM-ANTISPAM ENDmarkers; - one
spamd_addressline in the@CONFIG@section of/etc/exim.conf.local; actions.conf,greylist.conf,rbl.conf,emails.conf,asn.conf,fuzzy_check.conf,phishing.confandregexp.confin/etc/rspamd/local.d, each starting with# csm-managed-version: 1. Greylisting, the emails module, sender-IP lookups to rspamd.com, phishtank URL lookups to rspamd.com and public fuzzy storage are off. DNS block lists stay off unless a DNS feed is enabled.regexp.confholds theCSM_INVISIBLE_CJK_SUBJECTrule: a Subject with an invisible character, such as a zero-width space, next to a Chinese, Japanese or Korean letter. It checks the decoded Subject, including MIME-encoded ones, and covers kana, Han and Hangul letters with their half-width and supplementary forms. Punctuation does not trigger it. Spam uses invisible characters to split words so filters miss them. On its own the rule lifts a message above the defaulttag_scorebut not to the defaultreject_score. Keep your own regexp rules in/etc/rspamd/override.d/regexp.conf;- the map redirect files and, with a DNS feed, the blocklist files listed under Feeds;
groups.confin/etc/rspamd/local.d, which lowers rspamd’sHIDDEN_TEXTweight from 5 to 2. Newsletters hide their preview line, and at the defaulttag_scorethat rule plus one minor symbol tagged them. It also countsURL_VERY_LONGonce per message instead of once per long link, so newsletters full of tracking links are not tagged for that. It raisesDMARC_POLICY_REJECTfrom 2 to 3, so forged mail from a domain whose DMARC policy is reject reaches the tag score sooner. A symbol you define in any*_group.confin/etc/rspamd/local.dkeeps your weight: CSM leaves it out of its file. If one of those files cannot be read or uses includes, macros or multiline strings, CSM leaves all three weights to rspamd. Youroverride.dgroup files already take precedence. CSM replaces an operator-ownedgroups.conf. One that is not safe to replace is left alone: rspamd keeps its own weights and CSM reports anrspamd_scoreswarning;- with learning on, the learning files listed under Learning from Spam folders;
- a completion record in
/var/lib/csm/antispam.
Content outside CSM’s markers, including forward-guard blocks, is preserved. CSM regenerates its own blocks and files, so manual edits to them are overwritten. A file of the same name without CSM’s header is replaced, see Replaced files; removal leaves it alone.
The loaded phishtank_enabled setting must be the boolean false. Missing,
duplicate or non-boolean values are configuration conflicts, just like true.
CSM reports them and removes its integration when safe. Failed commands or
malformed command output leave the integration in place for a later check.
Each change runs as one transaction in a short-lived root helper started by
systemd outside CSM’s sandbox, under the same lock as the forward guard. It
rebuilds the Exim configuration, checks it, reloads Exim and rspamd only when
needed, and restores the previous state if a step fails. If systemd-run is
not available, CSM changes nothing.
CSM checks the integration at startup, after each config reload and every 15
minutes, and repairs drift, such as a cPanel rebuild that dropped the hooks.
After fixing a condition that blocked it, run systemctl reload csm to request
another check, even when csm.yaml is unchanged. A check already running finishes
first; the reload does not wait for it.
When another CSM change holds the lock, it tries again 20 seconds later.
Before each check, CSM cancels antispam changes still waiting to run from an earlier check or an earlier daemon, also while antispam is disabled or in observe mode. A change that is already running finishes first.
If cancelling fails, CSM raises a warning and retries: after 20 seconds while antispam is enabled, at the next regular check while it is disabled.
Existing Exim SMTP sessions retain their old hooks; new sessions use the updated hooks. Account toggles are still read at each recipient. Reloading rspamd can change scoring for scans from either generation of Exim hooks.
Replaced files
CSM owns its rspamd files and the Roundcube include outright. When antispam
is enabled and one of those paths holds your own file, CSM copies it to
/var/lib/csm/antispam/replaced before writing its own. Each copy is
readable by root only. Its name is the UTC time followed by the escaped
original path, for example
20261101T120000Z_%2Fetc%2Frspamd%2Flocal.d%2Factions.conf.
CSM replaces only a regular file owned by root, with no group or other write
access and no extra ACL entries, in directories protected the same way. Any
other file at one of these paths blocks the install, or leaves the optional
feature that needs it off. CSM repeats these checks before copying the file
and before replacing it. The Exim hook files and /etc/exim.conf.local
are shared: CSM edits its own block there and never replaces them.
Each retained copy raises one antispam_file_replaced warning naming the
file and the copy in the same check that runs the update. Move any settings
you still need into a file CSM does not manage, then delete the copy to clear
the warning.
When a change fails, CSM restores your file with its original owner, group and permissions. It deletes that copy only after the restore is durable, even if another file could not be restored. Copies made for replacements that never happened are removed too. If a restore cannot be confirmed, its copy remains for recovery.
On a cPanel host, disabling antispam deletes CSM’s files and does not
restore the copies. On a Postfix host, disabling antispam puts the newest
copy of each file back over CSM’s own file and deletes that copy, and
returning to dry run does the same for actions.conf, see
Dry run and enforce. Older copies of the same file
stay, and CSM never restores over a file it did not write. Restores require
root-owned backups with mode 0600 in a root-only directory. A symlink,
unsafe backup or unsafe destination permissions stop the transaction
before mail configuration changes. Correct the permissions and retry.
If a backup changes before completion, CSM rolls back and keeps that edit.
Restores and their rollbacks never follow directory symlinks.
A restored file gets the owner and mode checked before the restore, which
after a takeover are your original ones. /etc/rspamd/override.d/rbl.conf
written while a DQS key is set is the exception: CSM’s file is owned by root,
in rspamd’s group, with mode 0640, and the restored file keeps that.
Each warning says what CSM will do with its copy. The warnings describe
past replacements, including
after antispam is disabled.
Rolling it out
- Meet the conditions above and enable with
dry_run: true. - Leave dry run on for at least 7 days. Review each would-reject decision in
the Exim main log (
CSM-ANTISPAM event=scan ... decision=reject), and watch for scan-error findings. Do not enforce until all predicted rejects are classified, none rejects legitimate mail, and no scan or configuration failure finding remains. - Review which accounts are opted out today, and decide whether users may use
the spam box (WHM Tweak Settings,
skipspambox). That setting only shows or hides the choice: accounts that already have~/.spamassassinboxenableget tagged mail in their Spam folder either way, and turning the spam box off in WHM deletes that file from every account. - Set
dry_run: false. Check every 550 for a week. Return to dry run on a confirmed false positive.
To roll back, set enabled: false while CSM is in enforce mode. Successful
removal reloads Exim, and new sessions no longer use CSM filtering. Check for
removal failures before assuming rollback completed. Observe mode leaves the
installed integration untouched.
Sessions opened before removal retain their old hooks and may still scan. Those scans use operator or default rspamd settings after CSM’s settings are removed; scores can change and external lists can become active again.
Postfix hosts
On a host without cPanel that runs Postfix, CSM manages rspamd only. Postfix already passes mail to rspamd’s proxy worker as a milter, and rspamd applies the decision. CSM never edits Postfix’s configuration. The sections above about cPanel accounts, Exim and Roundcube do not apply.
Requirements
-
Postfix passes mail to rspamd’s proxy worker.
smtpd_miltersmust name the address of an rspamd proxy worker that hasmilter = yes. CSM checks this read-only withpostconfand rspamd’s worker settings. When smtpd runs chrooted, aunix:milter path is resolved inside Postfix’s queue directory, as Postfix does. Without this wiring CSM reports a definitepostfix_milterprecondition and installs nothing.milter_mail_macrosmust also include{auth_authen}, with the braces, as Postfix’s default does: rspamd knows authenticated mail by it. Without it CSM reports the same precondition, because customers’ outbound mail would be filtered as remote mail. CSM reads both settings frommain.cfonly; per-service-o smtpd_miltersoverrides inmaster.cfare not read. -
rspamd refuses the mail it rejects. With
discard_on_rejecton the proxy worker, rspamd’s milter drops a message it rejects; withquarantine_on_rejectit accepts the message into Postfix’s hold queue, where it stays until someone releases or deletes it. Either way the sender is never refused. CSM does not enforce while either is set on a proxy worker Postfix passes mail to, and removes an enforcing install, with a definitemilter_rejectsprecondition. In dry run, where the host’s own rejects go the same way, it is a warning. An option given more than once counts as set unless every value is false, and so does any value that is not a boolean, such as null. Set both to false, once each, for example in/etc/rspamd/local.d/worker-proxy.inc. -
rspamd 4.2 is recommended. CSM accepts rspamd 3 or later, but its scope rule is tested with rspamd 4.2.3.
-
/etc/rspamd/lua.local.dexists, owned by root and not writable by group or others. CSM’s scope rule goes there. -
A scan socket for CSM. CSM uses it to check that rspamd answers. Add a Unix socket to rspamd’s normal worker in
/etc/rspamd/local.d/worker-normal.inc:bind_socket = "localhost:11333"; bind_socket = "/run/rspamd/csm-normal.sock mode=0600";Only CSM uses this socket, as root, so mode 0600 is enough. Keep the first line if your proxy worker passes mail on to the normal worker instead of scanning it itself. The socket and every parent directory must be owned by root or the rspamd user and not writable by group or others. Run
rspamadm configtest, thensystemctl reload rspamd. -
rspamd logs to a file at
infolevel. CSM reads its scan records from rspamd’s log file, for example/var/log/rspamd/rspamd.log. When rspamd logs to the journal or syslog, CSM warns and sees no scan records.infois rspamd’s default. Atnotice,warningorerrorlevel rspamd does not write them, and CSM does not warn; setlevel = "info";in/etc/rspamd/local.d/logging.inc. -
rspamd accepts every message Postfix accepts. CSM warns when rspamd’s
max_messageis below Postfix’smessage_size_limit. -
Postfix and rspamd are running.
Which mail is filtered
All unauthenticated mail from remote hosts is filtered, whatever domain it
is addressed to. That includes mail Postfix relays for hosts in
mynetworks on public addresses; list those in infra_ips. CSM never
rejects or tags mail:
- from a sender who authenticated to Postfix;
- from rspamd’s local networks (
local_addrs), which include loopback and the private address ranges by default; - from an address in
infra_ips.
Mail whose every recipient is in exempt is scored and logged, but never
rejected or tagged. A domain entry matches that domain only, not its
subdomains. One recipient outside the list is enough: the whole message is
then filtered.
Your own rspamd settings rules, for example in
/etc/rspamd/local.d/settings.conf, keep working. For mail CSM does not
filter, CSM only raises the action thresholds on top of what your rule set.
A rule that limits which symbols run (symbols_enabled or groups_enabled)
also skips CSM’s scope rule: mail it matches keeps the normal thresholds, even
when it is authenticated, local or exempt, and gets no scan record.
Dry run and enforce
A dry run on Postfix already changes how rspamd scores mail. rspamd is in the mail path, and CSM installs its scoring files from the first dry run, so it can predict its own decisions. From then on:
- greylisting, the emails module and public fuzzy storage are off;
- rspamd’s own DNS block lists are off, unless you name DNS feeds in
feeds.enabled; - sender-IP and phishtank lookups to rspamd.com are off, and without the
rspamd_mapsfeed the rspamd.com maps CSM manages are read from local copies, so the freemail and disposable-address lists stay empty until you fill them, see Maps without any feed; - CSM’s symbol weights and its invisible-character Subject rule apply;
- CSM’s
milter_headers.confreplaces yours, so other header routines you enabled there stop; - your own files at those paths are replaced, with a copy kept and an
antispam_file_replacedwarning, see Replaced files.
What CSM leaves to the host in dry run is actions.conf: rspamd keeps
rejecting and tagging at the host’s own thresholds, now applied to CSM’s
scores, for all mail, including authenticated, local and exempt mail. Mail
can therefore score differently than before CSM, in both directions. CSM
logs what it would have done with each scored message: decision=reject,
tag or none, and unfiltered for exempt mail.
When the host’s own reject threshold is at or below reject_score, as with
rspamd’s built-in actions when there is no actions.conf, the dry run still
rejects mail. CSM raises a dry_run_rejects precondition warning, for
example “dry run on this host still rejects: rspamd rejects at 15, at or
below CSM’s reject score of 15”.
With dry_run: false CSM takes over /etc/rspamd/local.d/actions.conf. An
existing file is copied first, see Replaced files. rspamd
then rejects at reject_score and adds headers at tag_score. CSM’s
thresholds never reject or tag authenticated, local, infra_ips or exempt
mail; other rspamd modules, such as antivirus, can still reject it.
Going back to dry run puts your actions.conf back from its newest copy,
deletes that copy and clears its warning. The file gets the owner and mode
CSM’s file had, which after a takeover are your original ones, or root
ownership and mode 0644 when CSM’s file was loosened.
Your thresholds decide again. Without a copy CSM deletes its own file, and
rspamd’s built-in actions apply, which reject too.
Headers and the Junk folder
CSM owns /etc/rspamd/local.d/milter_headers.conf. rspamd removes any
X-Spam-Status header a message arrives with and adds its own:
X-Spam-Status: Yes, score=... at the add header threshold, No below it.
For authenticated and local mail, which rspamd’s milter headers skip, CSM’s
scope rule removes the inbound copy itself. In dry run the header follows
the host’s own add header threshold.
rspamd also adds its own Authentication-Results header with the SPF, DKIM,
DMARC and ARC results, and removes any copy the message arrived with, since a
sender could forge one. Authenticated and local mail get no new
Authentication-Results header and keep any existing copies, so rspamd does
not add login names to outbound mail. Their inbound X-Spam-Status is still
removed. Other header routines, such as X-Spam-Level or rspamd’s extended
headers, are off.
CSM leaves the X-Spam: Yes header of rspamd’s proxy worker alone. Filing
tagged mail into the Junk folder stays your Sieve rule, for example one that
files X-Spam: Yes into Junk. CSM does not manage Sieve.
Scan records
CSM’s scope rule writes one CSM-ANTISPAM event=scan line per scored message
into rspamd’s log, at info level: rspamd logging at notice or above
hides them. Authenticated, local and infra_ips mail get none. These
records count toward scan health and the Email Security page only. They do
not feed spam evidence, the CSM IP map, the Threat page or abuse reports:
any local process can make rspamd log such a line through its local scan and
milter listeners.
Failed scans come from Postfix’s mail log: Postfix logs a warning each time
its milter call to rspamd fails, and CSM raises antispam_scan_error. What
happens to that mail depends on Postfix’s milter_default_action: accept
delivers it unfiltered, tempfail defers it, reject rejects it and
quarantine holds it in Postfix’s hold queue.
Files and learning
CSM writes the rspamd files and the completion record listed under
What CSM writes, actions.conf only while it enforces,
plus milter_headers.conf in /etc/rspamd/local.d and csm_scope.lua in
/etc/rspamd/lua.local.d. It writes no Exim or Roundcube file. Each change
reloads rspamd only. Disabling antispam removes these files and puts back
the newest copy of each file CSM replaced, see
Replaced files.
Learning installs the same rspamd files as on cPanel, without the webmail include; see Learning from Junk folders.
If CSM cannot tell which mail server runs, for example while Postfix
restarts on a host that also has Exim installed, it leaves an installed
integration in place and reports a temporary platform precondition.
Learning from Junk folders
With learning.enabled: true CSM learns from mail users move into their
Dovecot Junk folder. Scores, limits, retention, stored messages, operator
commands and findings work as described under
Learning from Spam folders, with these
differences:
- Every mailbox Dovecot lists takes part. There is no per-mailbox switch
and no opt-in file. Mailboxes and domains in
exemptare not watched: CSM does not read their Junk folders. - CSM reads the
newandcurdirectories of each Junk folder. A report must be owned by the uid Dovecot stores that mailbox’s mail as. - Mail already in a Junk folder is never learned. The first pass records what is there, and only messages moved in later are reports.
- No webmail Junk-button include is installed.
Learning requirements
The learning prerequisites apply, and also:
-
Dovecot 2.3. CSM lists users with
doveadm user '*'and reads mail settings withdoveconf. A userdb must be able to list its users. A passwd or passwd-file userdb can, but not a passwd-file whose path holds%variables, such as one file per domain. An SQL or LDAP userdb needs its iterate settings. A static, checkpassword or prefetch userdb cannot. When no userdb can list users, and on Dovecot 2.4, whose mail settings changed, CSM reportsunsupportedand leaves learning off. When a userdb that can list users fails, for example while its SQL database is down, CSM reportsbackend_unavailable: installed learning stays and new reports wait for a later check. A temporary failure looking up an individual user also pauses discovery, without cancelling reservations or pruning state. Unusable or case-colliding user names pause discovery because their identities and mail owners cannot be established safely. -
Junk marked as
\Junk. CSM finds Junk folders byspecial_use = \Junkin Dovecot’s namespace mailbox settings, as Dovecot’s example15-mailboxes.confsets forJunk. A single configured mailbox is supported too. A folder named Junk without the flag is not watched, and with no\Junkmailbox at all every mailbox is skipped. -
Maildir in its default layout. CSM skips a mailbox stored another way: another driver such as
mdboxormbox,LAYOUT=fs, or thezlibormail_cryptplugin. It also skips a mailbox whose uid is 0, whose home or mail location it cannot use, such as a location withDIRNAME,MAILBOXDIRor a%{if}condition, or whose lookup confirms the user does not exist. A%{if}inmail_locationor a private namespace location skips every mailbox; CSM still reads their mail owners for isolation checks. Known owners remain part of those checks even when their mailboxes are skipped for another reason. Two users whose mail directories are the same, including a symlink alias, or one inside the other, are both skipped: CSM cannot tell their moves apart. Private namespaces without Junk also count. An unresolved mail location holds other mailboxes sharing its uid until discovery can distinguish them. Skipped mailboxes are counted by reason in onemailboxes_skippedwarning that names none of them; learning goes on for the others. -
A Redis of its own. Learning never uses rspamd’s Redis on Postfix, because other rspamd modules share it. Run a second Redis instance and set
learning.redis_socketto its socket; without it CSM reportsunsupported. On Debian and Ubuntu the packagedredis-server@.servicetemplate runs an instance from/etc/redis/redis-<name>.conf. For an instance namedcsm, create its data directory:install -d -o redis -g redis -m 0750 /var/lib/redis/csmThen write
/etc/redis/redis-csm.conf:port 0 unixsocket /run/redis-csm/redis.sock unixsocketperm 770 pidfile /run/redis-csm/redis-server.pid dir /var/lib/redis/csm maxmemory 256mb maxmemory-policy noeviction appendonly yesand start it with
systemctl enable --now redis-server@csm. The template’s sandbox lets Redis write below/var/lib/redis,/var/log/redis, its run directory and, for cluster mode,/etc/redis;systemctl cat redis-server@csmshows the exact list on your system.dirmust be in one of them for persistence to work. Use the data directory created above, not/var/lib/redisitself, where the main instance keeps its files. Sizemaxmemoryfor the hashes you expect to learn.Give rspamd’s user access through the socket’s group, for example with
usermod -aG redis _rspamd; no other user may be in that group. Then restart rspamd: a reload keeps the groups it started with, and its fuzzy worker would be refused by the socket. The Redis checks of the prerequisites apply to this socket. rspamd’s globalredissettings must not use sentinels, TLS or a named user, because rspamd would apply them to the learning Redis too. -
A private controller socket for CSM, as on cPanel; see rspamd admin interface.
-
A mail user that is not a service user. Users’ Sieve scripts and Dovecot’s mail processes run as the uid Dovecot stores mail as, such as
vmail, so CSM treats every name of that uid as a user that can run tenant code: none may be in the Redis socket’s group. Mail stored as rspamd’s or Redis’s uid is refused withunsafe_prerequisite.
CSM checks these on every reconciliation pass while learning is requested. The learner and its Junk folder watcher start only once none of them fails.
Reporters
learning.reporter decides who casts one learning vote:
domain, the default: each mail domain is one reporter. Its mailboxes share one vote per cluster and one learning allowance. A domain that reaches its allowance pauses only its own new reports.mailbox: each mailbox is a reporter of its own, as each cPanel account is.
A user without a domain, such as a system user, reports under its own uid in either mode.
Changing the mode, or exempt, restarts the Junk folder watcher. Votes
already cast stay until their cluster expires, so a cluster reported both
before and after a mode change can count the same domain or mailbox one
extra time. Reports the old reporters had not sent yet are dropped, and
the Junk folders of domain mailboxes start new baselines.
Enabling learning on Postfix
- Set up the Redis instance and the private controller socket above, then
run
rspamadm configtestand restart rspamd. A reload is not enough after adding rspamd’s user to the socket’s group. - Set
learning.enabled: trueandlearning.redis_socket, then runcsm validateandsystemctl reload csm. - Check
antispam_learningfindings andcsm doctor. Once learning runs,csm status --jsoncarries anantispam_learningblock. - Move a test message into a Junk folder. The next pass, within five minutes, picks it up.
Rolling it out on Postfix
- Meet the requirements above. Choose
feeds.enabledfirst: the dry run already turns off rspamd’s own DNS block lists and greylisting. Enable withdry_run: true. Leaveexemptempty unless a recipient must never be filtered. - Check that rspamd’s loaded actions are still the host’s
(
rspamadm configdump actions) and that nodry_run_rejectsormilter_rejectswarning appears. Review eachantispam_file_replacedcopy, and check that greylisting and DNS block lists are as you intend. Then check thatCSM-ANTISPAMlines appear in rspamd’s log and that mail at the host’s add header threshold still carriesX-Spam: Yes. - Leave dry run on for at least 7 days. Review each would-reject decision
in rspamd’s log (
CSM-ANTISPAM event=scan ... decision=reject) and watch for scan-error findings. - Set
dry_run: false. Check every reject for a week. On a confirmed false positive, return to dry run: CSM puts your ownactions.confback.
To roll back, set enabled: false. CSM removes its rspamd files, puts back
the files it replaced and reloads rspamd.
Stalwart hosts
On a host whose mail server is Stalwart, CSM does not install or manage
rspamd and puts nothing in the mail path. Stalwart’s own spam filter and
its own scores decide what happens to each message. csm stalwart setup
makes Stalwart log the mail its filter rejects or discards and reports
whether the filter is on and at which scores it acts; csm doctor checks
the same and whether the filter’s DNS blocklists answer. Neither switches
the filter on or off or changes the scores at which it files mail to Junk,
rejects or discards it. Setup also adds a spam rule that adds up
to 4 to the spam score of senders on CSM’s address list, so more of their
mail goes to Junk, or is rejected or discarded at the scores set in
Stalwart. It is the one setup step that changes what happens to mail;
switch the rule CSM_IP off in Stalwart to stop it. Setup keeps existing
tags and reports scores only for scoring tags; it lists other tags as
non-scoring. See
Preparing Stalwart.
Filter verdicts
CSM reads Stalwart’s log and counts one verdict per message on the Email page: delivered, Junk, rejected or discarded. A message to several recipients that went to Junk for any of them counts as Junk. Delivered and Junk messages are counted two minutes after CSM reads them, so the page lags by that much. A message with no local delivery by then, because it was relayed, deferred or failed, is not counted. Mail from logged-in senders, local mail between mailboxes included, is not counted, and neither is mail Stalwart logs without a client address. When more messages wait than CSM can hold, as under a flood, the extra ones count as “not matched”. If a held queue ID names conflicting messages, CSM counts that ambiguous join once as “not matched” and records no sender evidence. Repeated copies of the same queued line neither add a verdict nor extend the hold.
Delivered counts every delivered message, also while the filter is off. It also counts mail a mailbox’s own Sieve rule filed into a folder, Junk included, because Stalwart logs such mail as not spam. Rejected and discarded show a dash until Stalwart has logged one.
Spam evidence on Stalwart
Each counted message, except one not matched, also records a verdict about
the sending address, as on cPanel (see Spam evidence):
Junk, rejected and discarded count as spam, delivered as ham. Junk is spam
at Stalwart’s Junk score, not at a CSM score. The same addresses are
skipped: whitelisted, infra_ips, private and the server’s own. IPv6
senders are recorded and listed per /64 network, and clearing or
whitelisting an address inside forgets the network’s evidence. Spam
evidence never blocks an address, never changes its threat score and is
never reported. It does not need email_protection.antispam.
The spam evidence budget applies here too.
A sender on CSM’s address list gets no
verdict unless CSM sees that the rule CSM_IP is off. The rule raised the
score of its mail, so Stalwart’s verdict on it says more about the list
than about the sender. The list that counts is the one CSM builds now, not
what Stalwart holds. While Stalwart’s spam filter is off, no verdict is
recorded at all. The Email page counts these messages as usual.
CSM’s address list on Stalwart
CSM keeps its address list in Stalwart’s lookup lists, where the rule
CSM_IP reads it. The list holds the same addresses as the
CSM IP map, from the same sources and with the same
exceptions. CSM checks it at start and every 10 minutes, writes only what
changed, and then has Stalwart load it. If loading fails twice, CSM keeps
retrying on later checks, even when no keys need changing.
The lookup namespaces whose names start with csm-ip belong to CSM. CSM
deletes any key there that it did not add, so edits made there in Stalwart
are undone within 10 minutes. To stop the scoring, switch the rule CSM_IP
off in Stalwart: CSM then empties these namespaces. A missing rule or a
rule whose type is not Any also empties them. CSM fills them again once
the rule exists, has type Any, and is enabled. In observe mode CSM reads
Stalwart but writes nothing, so the namespaces keep what they hold, even
with the rule off. An unfinished reload keeps its warning in observe mode.
stalwart_spam_filter_off warns while Stalwart’s spam filter is off, since
Stalwart then scores no mail. A failed filter read leaves an existing
warning in place until CSM can confirm the filter is on.
stalwart_address_list warns when CSM cannot inspect or read its key file,
read the rule or list, write the list, or have Stalwart load it. When
Stalwart refuses CSM’s key, its message says to run csm stalwart setup.
These warnings clear once recovery is confirmed.
CSM’s key needs more access to keep the list, so run csm stalwart setup
again after upgrading. Without a key file CSM writes nothing to Stalwart and
raises neither finding; setup and csm doctor report the missing key. An
unreadable key file instead raises stalwart_address_list.
Not done on Stalwart
Apart from the rule for its address list, CSM does not change Stalwart’s scores or rules, set up its DNS blocklists or check outgoing mail.
rspamd admin interface
rspamd’s controller answers admin and statistics requests. By default it
trusts every connection from 127.0.0.1 and ::1 without a password, and
every process on the server can connect there, including customer scripts.
Such a process could read rspamd’s history, which lists senders and
recipients of scanned mail, or change scores and actions at runtime.
While antispam is enabled, CSM reports each controller listener that other
users may reach as an antispam_controller_open finding. The finding names
the listener and the reason, including cases where CSM cannot prove isolation:
- a trusted address that local processes can use (loopback, a wildcard, or one of the server’s own addresses);
- a trust entry CSM cannot resolve, such as a map file or a hostname;
- no controller password at all, only an enable password, or an empty password;
- a controller configuration CSM cannot read in full, such as an unreadable
worker type, an unsupported
enabledorcountvalue, a systemd socket, or a second controller worker; - a Unix socket, or one of its parent directories, that other users can write to, or whose access CSM cannot verify.
For symlinked Unix sockets, a private directory protects only its own descendants. CSM also checks the resolved socket path: a private directory on a detour does not hide a socket reachable through its real path.
These warnings also update in observe mode and when an apply is busy. When
rspamadm cannot run, CSM keeps the previous warnings until it can.
Turning antispam off clears them even if CSM cannot cancel pending changes.
To close TCP access, keep a controller password set and put this in
/etc/rspamd/override.d/worker-controller.inc:
secure_ip = [];
rspamd ships with the placeholder password q1, which it rejects as client
authentication, so this also works without setting your own. With both
password and enable_password unset, an empty secure_ip opens every
command to everyone. Setting only enable_password leaves read-only access
open.
For local administration add a private Unix socket in the same file. Direct Unix socket connections get admin access without a password. Forwarded headers trigger additional trust and password checks, so local clients must send no forwarded headers. Keep the socket and its parent directories owned by root or rspamd, with no extended ACLs. Use socket mode 0600 and non-symlink directories that other users cannot write to. Socket mode alone does not protect a replaceable parent:
bind_socket = "/run/rspamd/csm-controller.sock mode=0600";
bind_socket = "localhost:11334";
secure_ip = [];
Reload rspamd after the change. CSM never edits the controller settings.
The Email Security page reads rspamd’s counters through a Unix controller
socket owned by rspamd, such as the private socket above, or else a loopback
listener that secure_ip trusts, and shows “Stats unavailable” without
either; CSM never uses the controller password.
Learning from Spam folders
email_protection.antispam.learning.enabled: true makes rspamd learn from
mail users move to their Spam folder. When several accounts report the same
content, later copies of it score higher. Learning is off by default and
requires antispam to be enabled. Enabling it is refused in
observe mode. This section describes cPanel hosts; for
Postfix hosts see Learning from Junk folders.
Enabling learning is consent to two things: CSM reads new messages in the Spam folders of participating accounts, and it keeps one full message per learned cluster. Clusters normally retire 90 days after their last report, but pauses and cleanup failures can retain messages longer (see Stored messages).
How reports are found
Every five minutes CSM checks the main mailbox and each domain mailbox of
every account whose Spam Filters switch is on, or of every account when
/etc/global_spamassassin_enable exists. It looks at the new and cur
directories of .spam, the folder spam-box delivery and the webmail Junk
button use, and .Junk, which many mail clients use. Folders are read
without following links; CSM never creates folders or changes permissions.
The exception is a .spam or .Junk that links to another folder of the
same mailbox, such as .spambox on some older cPanel accounts, by its bare
name or its exact path under the account’s home. CSM reads that folder, and
pointing the link elsewhere starts a fresh baseline.
The first pass over a folder records what is already there without reading it. Only messages that arrive after that are reports, so existing mail is never learned retroactively. Moving a message within the folder or changing its flags does not report it again. A pass reads at most 20 reports per account, takes turns between accounts, and stops before the next one is due; the rest wait for the next pass.
What counts as a report
A report is a regular file owned by the account, at most 10 MiB, that did
not change while CSM read it. A message still linked from another folder,
such as one delivery shared by several mailboxes, counts once it and its
links have stayed unchanged for a full pass, so a move into or out of Spam
can finish first. Messages are skipped when any X-Spam-Status header
says Yes, when that header cannot be parsed, and when the header block
is malformed or too large, linked or not. Only the text parts of a report
are learned, attached text included; images and other attachments never
are. A message whose MIME structure CSM cannot read is skipped as
no_digest, for operator requests too.
Reports are not authenticated. Tenants control their own mail: they can
move any message they received, or deliver one locally, and a forged
X-Spam-Status: Yes header keeps a message from being learned. In dry run
senders’ headers are not removed, so forged headers have this effect on
any message.
Scores and limits
Each reporting account adds a fixed weight to the cluster. One, two, three
and four accounts give CSM_FUZZY_SPAM scores of about 2.9, 5.4, 7.2 and
8.4, so with the default tag score of 6 a message needs reports from about
three accounts to be tagged on that evidence alone. An operator mark from
csm antispam learn --spam weighs as much as nine accounts.
Copies in multiple Spam folders or mailboxes of the same account still
contribute only once to that cluster.
Messages that share most of their text join the same cluster, so later variants of a campaign score too. A new message whose plain text and HTML versions differ starts its cluster from one of them, normally the larger; attached messages are never used. Later copies score through that version. Newsletters built from one template can also be close enough to join. Lowering or correcting a learned value removes and re-adds it, which can leave a short gap in which the message scores lower.
Learning writes scores with either value of antispam’s dry_run setting.
In dry run they affect the measured score; what happens to the message
still follows the dry-run policy.
Opt-out, disable and retention
Unless /etc/global_spamassassin_enable exists, turning an account’s Spam
Filters switch off stops CSM reading its folders at the next check and
drops its unsent reports, including reports the learner accepted but has
not yet sent to rspamd. Each cluster of a report is handled on its own: a
part whose update rspamd may already have received still completes. If the
learner cannot cancel, CSM reports it and retries at the next check while
the account stays opted out. Turning the switch back on starts new
baselines.
The global override prevents per-account opt-out, including learning.
Removing a cPanel account drops its learner reports the same way, once its
UID belongs to no remaining account. An account whose home directory CSM
cannot use keeps them. While the account list is changing or cannot be
read, every account keeps them and CSM retries at the next check.
What the account already contributed stays until the cluster expires or an
operator forgets it; opting out or removal does not delete it sooner.
Turning learning off keeps stored messages, keys and the learning ledger, but clears folder baselines and unsent reports; re-enabling starts new baselines. Observe mode keeps everything and resumes where it stopped.
Folders absent for 90 days are forgotten with their pending reports.
Unresolved accounts, unreadable homes, incomplete mailbox lists and unsafe
or unreadable folders keep their state until they can be checked again.
Links in an account’s etc directory, such as cPanel’s Roundcube database
link, are not mail domains: CSM does not follow them and they do not hold
back this cleanup.
A backend error stops further submissions for that pass; the next pass
starts with the accounts after the one that failed. Reports stay pending
while rspamd’s fuzzy storage answers no lookup, however long that lasts. A
failure counts against a message only when storage answered a lookup for
another report earlier in the same pass, or when rspamd cannot process the
message; such a message does not stop other accounts’ reports. When rspamd
answers a message’s inspection with an error, CSM inspects a fixed test
message: if storage answers it, the failure counts against the message and
the pass goes on with the next report. If storage does not answer it,
submissions stop and the report stays pending without a counted failure,
even if storage answered earlier in the pass. A message with three counted failures,
including unclassified learner errors, is skipped as failed. Canceled or
expired passes, reconciliation pauses, unsent controller requests, a wrong
controller peer and local state failures never count. Unreadable learning
state holds reports even when the learner cannot finish startup; the
reports resume once the state is available.
Prerequisites
While learning is requested, CSM checks these during reconciliation.
Definite failures request removal of the learning configuration. Temporary
probe failures keep an existing installation. Learning findings do not
disable core filtering; antispam_learning names the reason.
- rspamd 4.2.x and Redis 5.0.x, 6.2.x or 7.x; other versions are refused;
- exactly one private controller Unix socket owned by rspamd, mode 0600, in directories only root and rspamd can write to, with no extended ACLs, and no other controller listener granting other users admin or read-only access (see rspamd admin interface);
- rspamd’s control socket private to root and rspamd, with protected
directories and no extended ACLs. Check the effective
control_socketpath and its permissions as well as the controller socket; - one Redis server for fuzzy storage, reached only through a Unix socket:
port 0andtls-port 0, a socket owned by root, rspamd or theredisservice user, no access for other users or extended ACLs, and any group access limited toroot,redisand rspamd’s service user. Service users must not be hosting accounts. Parent directories must protect the socket from replacement. Redis needs a positivemaxmemoryandmaxmemory-policy noeviction. A Redis build without TLS needs notls-portsetting; - with
learning.redis_socketset, that Redis instead of rspamd’s: CSM applies the checks above to it, refuses it while any other rspamd module or rspamd’s global Redis settings name the same socket, or while those global settings use sentinels, TLS or a named user, and points the fuzzy worker at it with no password. Learning moves to a new socket only after it passes these checks; until then it keeps its current Redis; - rspamd’s uid used by rspamd alone: no cPanel account, and on Postfix no user but the service users, may share it. If CSM cannot read every account’s uid, learning waits: an installation stays, and nothing new is published until a later pass can check;
- rspamd’s log unreadable to other users, by file permissions or by CloudLinux
CageFS. CageFS qualifies only for a file log under
/var/log/rspamd, with every account confined and that directory absent from its skeleton; - no local fuzzy rule and no enabled fuzzy worker of your own. CSM’s files replace them, so CSM refuses instead. The stock rule for rspamd.com is replaced on purpose.
CSM sends the Redis password only to a socket it has verified, and only on a connection served by that socket’s owner.
A cPanel account record with no system user, such as cPanel’s own system
record, does not count as an account sharing the Redis owner when
/etc/passwd does not list it either. If /etc/passwd cannot be read in
full, or an account lookup fails for another reason, CSM does not trust a
non-root Redis owner. On Postfix these account read failures pause learning
and retain its installed configuration. The worker rechecks Dovecot mail
owners before using a dedicated Redis, including owners of skipped mailboxes.
Redis persistence is recommended. CSM warns only when both append-only
persistence (appendonly yes) and snapshots (save) are disabled. While
learning is active, CSM rebuilds missing learned values from stored messages.
What learning installs
fuzzy_check.confandworker-fuzzy.incin/etc/rspamd/override.d, mode 0640 for root and rspamd’s group: local fuzzy scoring and storage;fuzzy_group.confin/etc/rspamd/local.d: theCSM_FUZZY_SPAMscore and its group cap;csm_learning.luain/etc/rspamd/lua.local.d: a controller plugin for message inspection;/usr/local/cpanel/base/3rdparty/roundcube/config/inc.d/zz-csm-antispam.inc.php, which points Roundcube’s Junk button at the Spam folder. This changes the Junk button for every webmail user on the server, regardless of the account’s Spam Filters switch. CSM replaces an operator-owned file at that path, see Replaced files. Themarkasjunkplugin must already be enabled; CSM does not enable it.
CSM checks the include by loading Roundcube’s configuration with cPanel’s
PHP as the nobody user, never as root. If Roundcube does not load the Junk
settings, or the check fails, CSM restores the previous include and reports
roundcube_unavailable.
The rest of the learning configuration stays installed. Later passes omit
CSM’s include until the learning configuration changes or CSM restarts.
A later include can override the settings.
After writing these files CSM checks what rspamd loaded, including the
effective CSM_FUZZY_SPAM weight the controller reports, which can differ
from the files after a runtime weight change. A failed learning check
attempts to roll back the change. Later passes omit learning until its
configuration changes or CSM restarts. Fix the reported cause before
restarting; reloading unchanged settings does not clear the hold.
On an otherwise current, completed installation, changing only learning reloads rspamd without rebuilding Exim. Changing only the Roundcube include reloads neither service. An incomplete installation or Exim configuration drift can still require an Exim rebuild.
Keys live in /var/lib/csm/antispam/learning/keys, root only: directories
are mode 0700 and key files mode 0600. CSM creates a complete set when none
exists. It refuses a partial or invalid set with state_mismatch, leaving
learning off. Do not delete the surviving keys to start over: new keys
cannot match hashes made with the old ones.
The learning ledger is bound to the hash keys and hash settings. A changed
generation stops learning instead of reusing the previous learning state.
The Redis capacity warning starts at 75% of maxmemory and clears below
70%. It leaves installed learning configuration in place and pauses the
learner, including pending changes, until usage falls.
Turning learning off requests removal of CSM’s learning files and Roundcube include, while keeping the keys. Conflicts or transaction failures can prevent removal. Check findings before assuming learned scoring is gone. Disabling learning or entering observe mode stops native learning calls before configuration reconciliation. Calls already in flight are canceled and drained within their deadlines. Reconciliation pauses the learner until a completed pass confirms that learning can resume.
Stored messages
Each learned cluster keeps one message, the first one reported, so the
learner can restore its value if Redis loses it. CSM keeps its headers and
text parts, without images or other attachments, under
/var/lib/csm/antispam/learning/anchors, readable only by root. A cluster
started from one version of a message keeps only that version. CSM deletes
it when the cluster retires, 90 days after its last report, or when an
operator forgets it. Backups and file system snapshots can keep a copy
longer. Messages saved by older versions keep their attachments until
their cluster retires.
If deleting a retained message fails, CSM still removes its retired score
and reports the cleanup error. New reports wait until cleanup succeeds.
Disabling learning, observe mode or unmet prerequisites can delay retirement
and message deletion. Turning learning off keeps stored messages and state.
rspamd can hash the same message differently from one check to the next,
because the words it hashes depend on the language it detects. When a
stored message no longer gives its cluster’s value, CSM retires that
cluster, removes its score and logs a warning; other clusters and new
reports are not affected. A failed inspection leaves the cluster intact.
CSM can still publish the message if its text-only form reproduces the
learned digest; otherwise publication waits for a later retry.
Reports joined to a retired cluster contribute no vote;
operator spam requests report skipped:superseded.
Retained messages consume their learning allowance until deletion succeeds. The record that prevents a retired score from returning remains charged, including when a new cluster retires before its first contribution completes.
CSM’s state archive (csm store export) leaves learning state out;
csm backup includes it, see
Backup, restore and a fresh namespace.
Teaching and correcting learning
Root can learn a saved message as spam, or remove what one taught:
csm antispam learn --spam <file>...
csm antispam learn --forget <file>...
csm antispam learn --status [<request-id>...]
csm antispam learn --fresh-namespace
The daemon must be running. --spam and --forget require learning enabled
and refuse observe mode. Each file must be regular with one link, at most
10 MiB, and unchanged during reading. Symlinks anywhere in its path are refused. The
snapshot is sent to the daemon as bytes. --spam gives each matching cluster one
operator mark, which weighs as much as nine reporting accounts. --forget
removes the clusters a file matches, with every account’s contribution to
them, and deletes their stored message. Other learned messages may still
score similar mail.
Each file gets a request ID, printed before it is sent. If printing fails, that file is not sent. A lost or truncated reply is retried once with the same ID and snapshot. Refusals and completed replies are not retried.
A file reported as pending was reserved; the daemon keeps publishing it
in the background. A file reported as deferred changed nothing, because
learning was busy or waiting on another change; run the command again
later.
csm antispam learn --status <request-id> shows what the daemon recorded
for a request, including clusters still publishing and their phase.
Status only reads the ledger and works with learning disabled or in observe
mode. Without an ID it lists the requests that are not finished. If the
command is interrupted, look its ID up there, or in the action log with
csm actions --op integrate.antispam_learn --json, which records the
request ID, escaped basename, outcome and full digests, never message
content or headers. A missing record does not prove that no change
occurred. Running the command again makes a new request.
Forgetting works for the whole server: the clusters a file matches lose every account’s contribution. CSM keeps a zero-value record for each forgotten value and keeps auditing it, so a late write cannot bring the score back unnoticed. Forget is not a permanent allowlist; new reports of the same content start a new cluster.
| Exit status | Meaning |
|---|---|
| 0 | Every file was learned, already learned, forgotten or matched nothing; --status and --fresh-namespace succeeded. |
| 1 | At least one file was refused, failed or is still pending, or the daemon could not complete the request. |
| 2 | The command line is not valid. |
Backup, restore and a fresh namespace
csm backup stores the learning state as one bundle: the ledger in the
state database and /var/lib/csm/antispam/learning, which holds the hash
keys, transport keys and stored messages. csm restore brings the bundle
back and moves any learning directory it replaces to
learning.old-<UTC time>-<suffix>. Installation failures trigger rollback,
but rollback itself can fail. Cleanup errors can also occur after the new
bundle is installed. After any restore error, check the reported paths and
resolve it before restarting CSM.
Restored learning state holds new reports until the daemon has run with learning disabled outside observe mode; enable learning again only when the backup is the current bundle. CSM cannot tell an old backup from the current one by comparing scores.
If the backup is older, or storage was lost along with the keys, start a fresh namespace instead. With learning disabled, run:
csm antispam learn --fresh-namespace
It waits for an active reconciliation pass, moves
/var/lib/csm/antispam/learning to learning.old-<UTC time> and empties
the learning and Spam folder state. If emptying fails, CSM attempts to move
the directory back and reports any failure to do so. Resolve that failure
before enabling learning. After a successful reset, the next enable creates
new keys and a new storage prefix, and every folder starts a new baseline. Old
reports are not replayed. The old directory still holds full messages;
remove it once you no longer need it. CSM never flushes Redis; old hashes
stay under the old prefix until they expire.
Learning findings
Each antispam_learning finding names one reason. Each reconcile pass
reports server readiness; the running learner adds the reasons from
anchor_drift on. On Postfix the learner also reports a Dovecot listing
that failed in its last pass, until a later pass reads it.
| Reason | Meaning |
|---|---|
unsupported | An rspamd or Redis version is outside the supported versions above, a Postfix host has no learning.redis_socket, or Dovecot runs 2.4 or has no userdb that can list its users; learning is disabled. |
unsafe_prerequisite | Required isolation or Redis settings are unsafe or cannot be established; learning is disabled. |
state_mismatch | Learning keys could not be created or loaded safely, restored learning state awaits confirmation, or the learner found its state out of step with its storage; learning is disabled and nothing is adopted or deleted. The CSM log names the disagreeing fuzzy digests when the learner finds it. |
config_conflict | Operator configuration conflicts with CSM’s learning setup; learning is disabled. |
apply_failed | A learning check failed during apply; subsequent passes hold learning off until its configuration changes or CSM restarts. |
backend_unavailable | A probe failed temporarily, including a Dovecot listing that failed or timed out; existing learning configuration is retained unless a definite failure also requires removal. A first installation waits. |
worker_absent | The installed fuzzy worker is not listening. Restart rspamd after fixing the cause; CSM leaves the configuration installed. |
redis_capacity | Redis has reached the capacity guard above; CSM leaves the configuration installed. |
redis_persistence | Redis has neither append-only persistence nor snapshots enabled; CSM warns without disabling learning. |
roundcube_unavailable | The Junk-button include could not be installed, verified or removed cleanly. The other learning files are unaffected. |
removal_incomplete | rspamd still loads CSM’s learned scoring while learning is disabled; resolve the reported conflict or removal failure. |
anchor_drift | A learned value moved away from its target more than once. CSM keeps correcting it; repeated drift means something else writes to the learning storage. |
mutation_uncertain | A write to the learning storage got no answer. CSM reads the stored value before it sends anything else. |
anchor_body_invalid | A stored message is missing or changed. That cluster’s value is held until it is forgotten. |
state_io | Learning state could not be written, for example on a full disk. Nothing unrecorded is sent and installed scores stay. |
audit_overdue | The check of every learned value has not finished a round within 24 hours. |
request_errors | Learning requests to rspamd failed three times in a row. |
backend_rebuilding | Learned values are missing from storage, for example after Redis lost its data. CSM restores them from the stored messages and holds new clusters until it is done. |
mailboxes_skipped | On Postfix, Dovecot mailboxes CSM does not watch, counted by reason; see Learning requirements. Learning goes on for the others. |
account_capacity | An account, or the operator, reached its learning allowance, or an account’s Spam folder state reached its limit of 65536 messages. Its new reports are skipped or wait; existing learning stays. Confirmed absent folders are forgotten after 90 days, which frees their share. |
Findings
| Check | Severity | Meaning |
|---|---|---|
antispam_precondition | Warning | A condition above is not met. The message names the cause, such as rspamd_socket or spamassassin_enabled. A temporary cause keeps an installed integration; a definite one removes it when that is safe. |
antispam_apply_failed | Warning, or Critical when the previous state could not be confirmed | An install, update or removal failed at the named step, or pending changes could not be canceled. |
antispam_file_replaced | Warning | CSM replaced your own file at one of its paths and kept a copy. Deleting the copy clears it, and so does CSM putting the copy back on a Postfix host. See Replaced files. |
antispam_rspamd_down | Warning | Three probes of the scan socket in a row failed. CSM probes every 20 seconds while antispam is enabled. |
antispam_rspamd_recovered | Warning | Two probes in a row succeeded after an outage. |
antispam_scan_error | Warning | The mail server logged a failed scan. One finding covers every failure until a scan succeeds. |
antispam_controller_open | Warning | Other users may reach rspamd’s admin or read-only interface, or CSM cannot verify isolation. See rspamd admin interface. |
antispam_feed | Warning, or Critical when a list answers for everything | A feed or map redirect is off, loaded rules differ, a list misbehaves, or rspamd still fetches maps from rspamd.com. These findings alone do not disable filtering. See Feeds. |
antispam_ip_map | Warning | The IP map could not be written, or rspamd has not confirmed the current map 5 minutes after a write. Core filtering is unaffected. See CSM IP map. |
antispam_learning | Warning | One finding per learning readiness or removal problem. Core filtering is unaffected. See Learning findings. |
stalwart_spam_filter_off | Warning | Stalwart’s spam filter is off, so Stalwart scores no mail. See CSM’s address list on Stalwart. |
stalwart_address_list | Warning | CSM could not inspect or read its key file, read its rule or list, update the list, or have Stalwart load it. Run csm stalwart setup when the message says so. See CSM’s address list on Stalwart. |
spam_evidence_evicted | Warning | The spam evidence budget was reached and CSM dropped the weakest records. The message names the most crowded network and how many records it held. See Spam evidence budget. |
While rspamd is down or scans fail on a cPanel host, mail is delivered
without spam filtering. Exim adds no rejection for a scanner failure, and
other mail checks keep running. On a Postfix host, Postfix’s
milter_default_action decides, see Scan records.
Disabling antispam clears the down and scan-error findings without a
recovery notice. Preconditions or removal failures can still be reported.
Scan records in the Exim main log include the connecting sender’s address as
ip=. Scan health ignores records with a malformed non-empty address. Older
records without that field, and records with an empty address, still count.
Dismissing a scan-error finding acknowledges the current episode. A successful scan clears that dismissal immediately, even between health probes. Disabling antispam or restarting CSM also resets it. Undismissed scan errors keep the normal daily notification deduplication so intermittent failures do not send one alert per message.
Rspamd outage and recovery notices do not consume the routine hourly alert budget. They honor notification suppressions and email check exclusions. Suppressed notices remain in history, the audit log and event streams.
In observe mode CSM changes no mail configuration; it only cancels changes
still waiting to run. An installed integration is reported as observe_mode.
rspamd timeouts
When a scan reaches rspamd’s task_timeout, rspamd answers with the score it
had so far. A slow rule that would have lowered the score is lost, so a
timed-out scan can tag or reject mail a complete scan would pass. Exim cannot
tell the difference; rspamd’s log records the timeout. If it shows timeouts,
raise task_timeout or remove the slow rule.
Tested versions
cPanel 11.138 with cPanel’s Exim 4.100.1: on AlmaLinux 8 with rspamd 4.2.1 and 4.2.2, and on AlmaLinux 9 with rspamd 4.2.2. CSM needs Exim 4.97 or later with content scanning and rspamd 3 or later.
Threat Intelligence
CSM tracks, scores, and correlates attacks using a local attack database enriched with external feeds and GeoIP data.
cPanel service-subdomain alerts retain the logged address as unverified context, separate from per-IP reputation and attack history.
Attack Database
- Per-IP event tracking (brute force, reconnaissance, reputation, authenticated activity)
- Local scoring from attack volume, types and targeted accounts, used to rank addresses and in the unified verdict; the score never raises a finding or a block on its own
- Top attackers leaderboard
Successful cPanel, FTP, webmail and PAM login audit events, and authenticated
File Manager writes, remain in event history and account counts as
auth_success (Authenticated Activity). They add no event-volume or
multi-account score. Multi-IP login and authentication-failure signals keep
their existing scoring.
file_upload remains readable for historical records; no current check
produces it. Existing scores recorded under older classifications are not
rewritten because aggregated login counts cannot distinguish ordinary logins
from multi-IP alerts. For a confirmed false positive, follow
Clearing a stale local threat score.
That operation preserves historical events and does not remove firewall or
permanent-blocklist entries; review those separately.
Mail evidence
The attack database keeps mail login attacks and spam/ham verdicts in a separate mail section. Equivalent IPv4 and IPv6 spellings share one record; invalid sender addresses are ignored. Spam and ham do not change the local threat score or add attack events. Ham only updates an existing record.
Spam and ham verdicts about an IPv6 sender are recorded per /64 network, not per address. A record that holds only spam evidence is dropped once that evidence has faded, or sooner to keep the 50,000-record spam evidence budget. The budget is checked when a verdict is recorded, using that verdict’s observation time; startup can load more records until then. See Spam evidence for the details.
Verdict weights decay from their observation times, including when older verdicts arrive after newer ones. A missing observation time uses the current time. Spam refreshes the record’s activity time. Remaining ham weight keeps a spam-only record until both spam and ham evidence have faded, but never past the 90-day limit since the last spam verdict or finding. The spam evidence budget can drop it sooner. Retained mail evidence survives a restart with its original counts and observation times.
IP Intelligence
Combines multiple sources into a unified verdict:
| Source | Data |
|---|---|
| Local attack DB | Event count, types, score |
| AbuseIPDB | External reputation (if API key configured) |
| Rspamd | Per-IP rolling history (if controller access configured) |
| Upstream HTTP cache | Panel-side shared score (if reputation.upstream configured) |
| Permanent blocklist | Operator-managed persistent blocks |
| Free public feeds | Spamhaus DROP, blocklist.de, CINS Army |
| Firewall state | Currently blocked/allowed status |
| GeoIP | Country, city, ASN, ISP |
| RDAP | Network name, organization (cached 24h) |
Verdicts: clean, suspicious, malicious, blocked
Free public feeds
CSM loads cached Spamhaus DROP, blocklist.de, and CINS Army data at startup and refreshes it daily. A failed download keeps that feed’s previous data. These lists supplement local evidence without requiring an AbuseIPDB key.
Spamhaus merged EDROP into DROP, so CSM no longer fetches EDROP. An old
<state_path>/threat_db/spamhaus-edrop.txt cache is ignored and can be deleted.
Existing firewall blocks, whitelists and findings are unchanged.
Pluggable sources
Threat-intel sources implement a small Source interface (lookup-by-IP returning a score + reason). The aggregator queries every enabled source in parallel, applies per-source weighting, and produces the unified verdict above. Adding a new source means implementing the interface and registering it; no existing source code changes.
Currently shipped:
- AbuseIPDB (
reputation.abuseipdb_key) - external IP reputation feed. CSM caps uncached lookups per cycle and reserves store-backed daily quota before sending requests. While the quota is exhausted (daily budget or an API 429/402 backoff) CSM emits areputation_quota_exhaustedWarning so the degraded coverage is visible; athreat_feed_staleWarning fires when previously downloaded free threat feeds have not refreshed in over 7 days. Persistent conditions remain visible in Findings; alert email and the webhook send each one once, and history records a reminder at most once a day, including across daemon restarts; these coverage warnings never trigger reputation scoring or blocks. - Rspamd (
reputation.rspamd.*) - per-IP rolling-history signals from the local rspamd controller. Delivered ham dilutes the score, temporary deferrals are neutral, and definitive spam actions count against the sender. Token resolution readstoken_envfrom the process environment at query time. Changing the external environment requires a daemon restart; see credential rotation. - Upstream HTTP cache (
reputation.upstream.*) - shared panel-side cache of AbuseIPDB or proprietary scores. Useful in fleets: agents pay a bounded local cache hit (cache_ttl_min, default 15 m) instead of hammering the upstream once per agent. CSM temporarily opens a fail-open circuit breaker after repeated upstream failures and lets only one cooldown probe through at a time. Use HTTPS for remote panels; plain HTTP is accepted only for loopback. Wire contract: Upstream threat-intel contract.
Verified crawlers
reputation.bot_verify_enabled verifies claimed crawler User-Agents
with static IP ranges first, then strict forward-confirmed reverse DNS.
reputation.verified_bots adds operator-defined crawler identities with
name, ua_substrings, and one verification method: rdns_suffixes or
ip_ranges. With rdns_suffixes the source IP must forward-confirm under
a registrable domain (public suffixes and shared-hosting suffixes are
rejected; a PTR-only match is not trusted). With ip_ranges the source IP
must fall in one of the published CIDRs – this is for crawlers such as
GPTBot and PerplexityBot that publish address ranges instead of crawler
reverse DNS. Over-broad or non-public ranges are rejected. All
checks run at config load and on reload.
Built-in rDNS verification covers Googlebot, Bingbot, Applebot, DuckDuckBot, Amazonbot, the Facebook and Meta crawlers, Brave, and the SERanking backlink bot. Googlebot, Bingbot, and Applebot also match a shipped IP-range snapshot first and fall back to reverse DNS; DuckDuckBot, Amazonbot, Facebook/Meta, Brave, and SERanking are rDNS-only.
Reverse-DNS verification is asynchronous. A newly admitted verification job receives a short, bounded pending window, including time spent waiting in the queue. High-volume traffic in that window can route to the proof-of-work challenge when it is enabled. A full queue, stopped worker, unsupported identity, or expired pending window uses the ordinary flood and scanner controls instead.
DNS failures, missing reverse DNS, and failed cache writes do not prove a
spoofed identity. They leave verification unresolved, delay retries, and do
not renew pending treatment on each retry. Attempt history is bounded. When
it is full, new sources can replace completed entries after their initial
cooldown; retries cannot extend that reservation. Live jobs and newly granted
pending windows keep their history. If no entry can be replaced, admission is
refused until capacity becomes available instead of running untracked lookups.
Evicted or expired sources without a persisted missing-PTR record can receive
another pending window, but the initial cooldown prevents continuous renewal
under churn. Expiry preserves live jobs and their retry delay after completion.
A missing PTR suppresses further DNS lookups for one fixed hour in the state
database, including across restarts. Scans do not extend that hour. After it
lapses, DNS retries resume, but the record remains as attempt history for a
day, so a restart or in-memory eviction cannot grant fresh pending treatment.
Records older than that are removed as new ones are written. Cleanup is
attempted at most once an hour, including after a failed cleanup. A definitive
verification result replaces that history. The record is not a verdict: it
grants neither the verified-crawler exemption nor pending treatment, and it
never counts as a spoof. csm store reset-bot-verify and a verified_bots
change clear these records together with the cached results.
A confirmed negative remains eligible for spoof detection. A cached positive
receives the normal verified-crawler exemption.
GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot and ClaudeBot are recognized
out of the box: their published IP ranges ship as an embedded snapshot and are
refreshed from the vendor endpoints by an auto-updater (reputation.bot_ranges,
default on, outbound HTTPS, configurable interval; restart required for setting
changes). Fetched ranges are validated with the same over-broad and non-public
guards as operator entries, and the embedded snapshot is the trusted fallback
when a fetch fails. Anthropic publishes one combined feed for ClaudeBot,
Claude-User and Claude-SearchBot and documents IP-list verification rather than
reverse DNS, so CSM verifies ClaudeBot by address from that feed; the legacy
anthropic.com reverse-DNS suffix is kept only as a fallback.
csm update-bot-ranges refreshes these ranges on demand (mirroring
csm update-geoip): it fetches the vendor feeds, writes the on-disk snapshot,
and asks the running daemon to apply them without a restart. The auto-updater
and the manual command both export metrics – refresh success/failure, prefix
count per crawler, and the last successful refresh time – under the
csm_botranges_* names.
Abuse Reporting
reputation.report can send minimized confirmed-abuse reports to a central
database or private collector. It is off by default. Remote targets must use
HTTPS; plain HTTP is accepted only for loopback collectors. Keys and target
wiring are read at daemon startup, so changes to this block require a restart.
The central abuse database accepts ed25519 targets only; its report URL ends
in /report and its scored-set URL in /decisions. The request format is in
the Abuse Report Contract.
Reported findings
Each class covers these checks. Only Critical findings are reported, except
bad_asn_egress and spam, which report High findings too.
| Class | Checks |
|---|---|
bruteforce | pam_bruteforce, wp_login_bruteforce, xmlrpc_abuse, ftp_bruteforce, smtp_bruteforce, mail_bruteforce, admin_panel_bruteforce |
credential_stuffing | credential_stuffing |
php_relay | email_php_relay_abuse |
bad_asn_egress | bad_asn_outbound |
spam | mail_spam_source |
Nodes accept spam entries in downloaded central lists. With spam in
reputation.report.classes, a node raises mail_spam_source for a sender
that rspamd keeps rejecting and that has sent it no clean mail recently, and
reports that sender. Dry-run antispam nodes count their verdicts too. The
finding never blocks or challenges the sender on the reporting node.
Shared mail services are never reported: Google, Microsoft, Amazon SES,
SendGrid, Mailgun, Brevo, Yahoo and iCloud. CSM reads their sending hosts
from each provider’s published SPF records and refreshes the list in the
background. Spam reports wait while that list cannot be loaded. When a
refresh has failed and the list is not current, CSM raises a
shared_senders_stale Warning. It stays in Findings until the list is current
again. Alert email and the webhook send it once; history records a reminder at
most once a day, including across daemon restarts.
CSM keeps sender history across restarts. If that history cannot be read at startup, spam source detection stays off for that run and the daemon log says why. A restart turns it back on once the state database can be read.
The class needs email_protection.antispam.enabled. Turning either on or off
takes a restart.
A report carries only the source address, the class, a count and the time.
List the classes to send in reputation.report.classes. An empty list keeps
reporting off.
Set up the central abuse database
-
On the node, generate a key pair:
csm report enrollIt prints a 64-byte private key as 128 hex characters and a 32-byte public key as 64 hex characters. Put the private key in the CSM environment file described in credential rotation, for example as
CSM_REPORT_KEY. The private key is a secret: never paste it into a ticket or chat. Give only the public key to the central operator. -
The central operator approves the node and returns its
node_idandkey_id, the report and scored-set URLs, and the central public key. Add the central public key, encoded as 64 hex characters, to the same environment file, for example asCSM_CENTRAL_PUBKEY. -
Configure the target. The
centralblock is optional; it is described below.reputation: report: enabled: true classes: [bruteforce, credential_stuffing, php_relay, bad_asn_egress] targets: - name: central url: https://abuse.example.net/report transport: ed25519 node_id: "<node_id from the operator>" key_id: "<key_id from the operator>" key_env: CSM_REPORT_KEY central: enabled: true set_url: https://abuse.example.net/decisions pubkey_env: CSM_CENTRAL_PUBKEY auth_target: central action: challenge -
Restart CSM:
systemctl restart csm.service.
reputation.central pulls the database’s signed list of scored addresses.
With action: challenge, a qualifying local attacker finding for an
unprotected listed address adds it to the challenge list. This requires
challenge.enabled: true, an active firewall and a working
challenge setup.
With action: block_if_local_corroborated, CSM attempts a block instead
once the central score reaches reputation.central.block_threshold;
below that score it still uses the challenge path. A live block also
requires an active firewall, auto_response.enabled: true,
auto_response.block_ips: true and auto_response.dry_run: false, and
remains subject to the normal block protections. Dry-run records what
would have been blocked. Central data alone never blocks. An
ip_reputation match is outside intelligence too, so it never counts as
the local finding for either action. action: off pulls the list without
taking central-driven actions.
auth_target names the report target whose node key signs each download.
The central abuse database refuses unsigned downloads. The target must use
transport: ed25519, and its URL must share scheme, host and port with
set_url, so a signed request never goes to another host. Leave
auth_target unset only for a central service that serves the list
without signatures.
Signed downloads work with reputation.report.enabled: false; keep the
target configured even when the node does not send reports. csm validate
refuses an auth_target that names no target or several, a target that is
not Ed25519 or lacks node_id, key_id or key_env, or a set_url on
another origin, and CSM will not start with that config. The settings page
refuses the same changes. If the target’s key is unset or invalid, CSM
starts with the central consumer off and logs one line naming
reputation.central.auth_target. It never falls back to unsigned
downloads while auth_target is set.
If the service refuses a download, CSM keeps its last verified list, tries
again at the next refresh, and logs
central-intel: pull unauthorized; check node enrollment, node clock and reputation.central.auth_target.
Each successful refresh logs central-intel: refresh complete (version=N).
To rotate the node key, run csm report enroll again and have the central
operator register the new public key. Then update key_id and the
environment file, and restart CSM.
Send to a private collector
Use transport: hmac with a shared secret. You choose node_id and
key_id; the collector uses them to find the secret. Set the named secret
and optional token in CSM’s
environment file
and restart CSM. The secret is used as raw string bytes, not hex-decoded.
reputation:
report:
enabled: true
classes: [bruteforce, credential_stuffing]
targets:
- name: collector
url: https://collector.example.net/csm/report
transport: hmac
node_id: web-01
key_id: k1
key_env: CSM_COLLECTOR_SECRET
token_env: CSM_COLLECTOR_TOKEN # optional bearer token
The collector must implement the
Abuse Report Contract. CSM queues one copy per
target, so one node can report to the central database and a private
collector at the same time. Give each target a unique, stable name:
duplicate names select only the last usable target, and removing or
renaming a target, or a key error that makes it unusable at startup,
discards its queued reports on the next delivery pass.
The spool_max capacity is shared across all targets, not reserved per
collector. Overflow drops the oldest entries across that shared queue.
Check delivery
When reporting starts successfully, the log shows
abuse-reporting: enabled for 1 target(s), 4 class(es) with your counts.
A target with a missing field, a non-HTTPS remote URL, an unknown transport
or an unusable key is skipped with a log line. Reporting stays off when
no usable targets or valid classes remain, or when its spool cannot be
opened. Failed deliveries are logged as
reporting: delivery incomplete. The abuse_reporting.ingress and
abuse_reporting.spool entries in
protection queue health show waiting and
lost reports.
Delivery and retries
Reports wait in a durable queue and are delivered once a minute. When a target
is down, throttling, or refuses the node (for example before the node is
approved, or when the node’s clock is wrong), that target’s reports stay
queued in order and it is retried a minute later; other targets keep
receiving theirs. A target that answers HTTP 429 with Retry-After, in
seconds or as an HTTP date, is retried after that wait instead, up to an
hour. Throttling while the target still accepts reports is not logged. If it
keeps throttling without accepting any, the delay doubles on each attempt up
to a minute, or the requested wait if longer, with a warning at most once a
minute per target. A report the collector refuses as invalid (HTTP 422)
would be refused again, so it is dropped and counted as lost.
Reporting and central intelligence actions share the address protection checks. These cover literal infrastructure IPs/CIDRs and crawler IP ranges, not infrastructure hostname resolution or rDNS crawler results. For NAT64 and 6to4 sources, protection of either the original IPv6 address or its embedded IPv4 address applies.
Web UI
The Threat Intel page (/threat) provides:
- IP lookup with composite scoring
- Two separate block actions per IP: a 24 hour block and a confirmed permanent block, singly or over a selection of attackers
- Top attackers with GeoIP enrichment
- Attack type breakdown chart
- Hourly trend chart
- Whitelist management (permanent and temporary)
A 24 hour block records threat evidence that expires with the firewall block, so a mistaken block of a customer address stops counting against it once the block lapses. A permanent block records evidence that stays until an operator clears it. When an address is no longer blocked but still holds permanent evidence, the lookup says so, because that address is flagged again the next time it is seen.
The 24 hour action refuses to shorten an existing permanent or longer firewall block. Unblock explicitly before changing that lifetime. Bulk requests skip those addresses and report warnings. Existing permanent threat evidence remains until cleared; feed updates do not remove a timed operator record before its expiry.
Bulk undo restores each prior firewall lifetime, using the original deadline for timed blocks. Expired blocks stay expired, and a later block, clear, or whitelist decision invalidates the older undo action. Only successful firewall reversals restore the corresponding threat evidence.
When CSM starts it drops the attack records of addresses inside the
top-level infra_ips list, so an internal host listed after it was recorded
stops ranking as an attacker. Only addresses and CIDRs count; hostnames and
firewall.infra_ips do not. Event history is kept.
Unblock & Clear accepts private addresses for an internal host that is not in
infra_ips. Clear and whitelist remove the attack record under every
equivalent spelling of the address, such as an IPv4-mapped form. Whitelist and
allow actions refuse private addresses and point to infra_ips instead.
API Endpoints
GET /api/v1/threat/stats Attack stats and type breakdown
GET /api/v1/threat/top-attackers Top attacking IPs with GeoIP
GET /api/v1/threat/ip IP threat lookup
GET /api/v1/threat/events IP event history
GET /api/v1/threat/whitelist Whitelisted IPs
GET /api/v1/threat/db-stats Attack database statistics
POST /api/v1/threat/block-ip Block IP for 24 hours
POST /api/v1/threat/block-ip-permanent Block IP with no expiry
POST /api/v1/threat/whitelist-ip Permanent whitelist
POST /api/v1/threat/temp-whitelist-ip Temporary whitelist
POST /api/v1/threat/clear-ip Clear from attack DB
POST /api/v1/threat/unwhitelist-ip Remove from whitelist
GeoIP
MaxMind GeoLite2 integration for IP geolocation and ASN enrichment.
Features
- City database - country, city, latitude/longitude
- ASN database - ISP, organization, autonomous system number
- Auto-download on first use
- Auto-update every 24 hours (configurable)
- RDAP fallback for detailed ISP/org info (cached 24h)
Where It’s Used
- Threat intel page (top attackers, IP lookup)
- Firewall audit log (country flags)
- Login alerts (geographic context)
- Country-based login suppression (
trusted_countries) - Country blocking (firewall CIDR ranges)
Configuration
geoip:
account_id: "YOUR_MAXMIND_ACCOUNT_ID"
license_key: "YOUR_MAXMIND_LICENSE_KEY"
editions:
- GeoLite2-City
- GeoLite2-ASN
auto_update: true
update_interval: 24h
Free account: maxmind.com/en/geolite2/signup
CLI
csm update-geoip # Manual database update
csm firewall update-geoip # Download country CIDR blocks
csm firewall lookup <ip> # GeoIP + block status lookup
Address lookups report country, city, ASN, organization, and network when the local databases supply them. Country-block matches retain all matching countries and do not prevent ASN or city enrichment. Missing fields are omitted.
API
GET /api/v1/geoip IP geolocation (?ip=&detail=1)
POST /api/v1/geoip/batch Batch lookup (array of IPs)
Challenge Pages
JavaScript proof-of-work challenge pages - a CAPTCHA alternative for suspicious IPs.
How It Works
- Suspicious IP hits a protected resource
- CSM serves a challenge page requiring client-side SHA-256 proof-of-work
- Browser computes the proof (shows progress bar)
- On valid solution, CSM issues an HMAC-verified token
- Subsequent requests pass through automatically
Features
- SHA-256 based difficulty - configurable 0-5 levels
- Client-side computation - no server load
- HMAC token verification - prevents replay attacks
- Nonce-based anti-replay
- User-friendly - progress bar, instant feedback
- Bot filtering - clients that do not run JavaScript never pass. Headless browsers and scripts written for the puzzle can solve it, and a higher difficulty costs real visitors more time than such clients
Use Cases
- Gray-listing alternative to hard IP blocks
- Protecting WordPress login pages
- Rate limiting without blocking legitimate users
- DDoS mitigation layer
Routing Behavior
When challenge.enabled: true, CSM routes eligible IPs to the challenge page instead of hard-blocking them. This works independently of auto_response settings.
Addresses the firewall refuses to block are never routed: entries in
infra_ips or firewall.infra_ips and, while the CSM firewall runs, the
current addresses of infra hostnames and the server’s own addresses. Entries
only in firewall.infra_ips still produce alerts and spam evidence.
Challenge-Eligible Checks
Pre-auth, browser-visible attack signals only: wp_login_bruteforce,
xmlrpc_abuse, wp_user_enumeration, webmail_bruteforce,
http_scanner_profile, http_claimed_bot_unverified, ip_reputation,
email_php_form_relay (routing and timeout blocks require ip_recipient_mode: enforce).
Post-auth audit events (cPanel, webmail, file upload, WHM
logins), WAF high-volume attacker findings, and non-browser protocols (SSH,
FTP, DNS recursion, outbound traffic, API auth) are excluded - their IPs have
no useful challenge step or no browser session to render the PoW page.
Local threat scores no longer route addresses to the challenge page, and no
check produces local_threat_score findings any more.
http_scanner_profile routing is operator-selectable:
auto_response.http_scanner_action: "challenge" (default) routes the IP here;
"block" skips the challenge and hard-blocks directly. With the challenge
subsystem disabled, both values block.
Always Hard-Blocked
Confirmed malware (webshells, YARA/signature matches), WAF high-volume attackers, C2 connections, backdoor ports, phishing pages, database injections, and spam outbreaks are hard-block candidates immediately, even when challenge is enabled.
Timeout Escalation
If an IP doesn’t solve the PoW challenge within 30 minutes, it is
automatically escalated to a hard firewall block while
auto_response.enabled and auto_response.block_ips are on; otherwise the
entry only expires. The pending claimed-bot path
(http_claimed_bot_unverified) is the exception: it expires without timeout
escalation because the reverse-DNS verifier decides the next action. A confirmed
spoof is hard-blocked by the later http_ua_spoof finding after it reaches the
UA threshold, while a real crawler is removed from the challenge list once
verification succeeds. A client named only by a trusted proxy’s
X-Forwarded-For header is the other exception: its requests reach this host
from the proxy, so neither the challenge page nor a firewall block reaches
it, and its entry expires without escalation.
Counts that include direct requests still allow timeout escalation, and so
does a direct finding for the same address in the same scan. An existing
non-escalating listing stays until it expires.
A timeout block of an address that recently logged in as a customer sends the known-customer block alert at once. The block still applies.
Passed challenges
Each pass is logged as CHALLENGE-PASS: <ip> passed (<method>) and counted
in csm_challenge_passed_total and the passed field of the challenge stats.
CSM remembers passes for 7 days, across restarts, and expires stored passes
during normal operation:
- A challenge that times out within 4 hours of the address’s last pass expires without a block. A returning browser is waved through by its verification cookie for the same 4 hours.
- WordPress login, XML-RPC and user enumeration counts read each site’s last log lines, which on a quiet site can be days old. Lines an address logged before or at its last pass no longer count against it or attribute a distributed attack to that site; newer ones do. While the challenge service is active, these counts also ignore lines older than the pass retention period, so forgotten passes cannot revive answered requests. Lines without a readable timestamp still count because CSM cannot establish their age.
A pass does not exempt an address from new challenges or hard-block checks. A client without a valid verification cookie must solve a new challenge when routed again, including during the timeout grace period.
Bind address
The listener binds to 127.0.0.1 by default, so enabling the challenge
server alone does not expose a new public port. The webserver integration
uses direct redirects to challenge.public_url; installed direct mode
therefore needs a non-loopback listener and a public URL ending in
/challenge. For your own proxy on this host, listen on a Unix socket
instead (see Proxy on the same host).
challenge:
enabled: true
listen_addr: 0.0.0.0
listen_port: 8439
public_url: https://cpanel.example.com:8439/challenge
tls_cert: /var/cpanel/ssl/cpanel/mycpanel.pem
tls_key: /var/cpanel/ssl/cpanel/mycpanel.pem
When CSM’s firewall is enabled and challenge.port_gate.enabled is true,
the daemon also opens challenge.listen_port in the main firewall rules.
The port-gate chain still drops traffic to that port unless the source is
loopback, an infra_ips entry, or an IP currently on the challenge list.
Port-gate rules follow the configured listener address family. An IPv6-only
listener gates only IPv6 clients; IPv4 challenge entries stay in the
webserver map but are ignored by the IPv6 nftables set.
Run csm doctor challenge after changing these fields. The command checks
the public URL shape, TLS files, port-gate setting, installed webserver
snippet version, webserver configtest, and the live /challenge/gate
endpoint. Add --json for automation.
TLS
The challenge listener serves HTTPS when challenge-specific TLS material is configured. Loopback listeners stay on plain HTTP by default. Direct/public listeners can reuse the Web UI cert.
Resolution order:
challenge.tls_cert+challenge.tls_key(explicit per-service).webui.tls_cert+webui.tls_key(shared cert; cPanelmycpanel.pemcovers both webui and the challenge port without extra config) only whenchallenge.listen_addris not loopback or a Unix socket.- Plain HTTP. This is expected for the default loopback-only path.
Public listeners without TLS log a startup warning.
HSTS-pinned parent domains (cPanel, phpanel, customer apex) will
fail with
ERR_SSL_PROTOCOL_ERRORbecause the browser auto- upgrades the URL to https; ship TLS material in production.
challenge:
tls_cert: /var/cpanel/ssl/cpanel/mycpanel.pem
tls_key: /var/cpanel/ssl/cpanel/mycpanel.pem
Trusted Proxies
By default, the challenge server uses RemoteAddr to identify clients.
The shipped webserver integration redirects browsers directly to
challenge.public_url, so it does not need trusted_proxies. Configure
trusted proxies only for a custom proxy deployment where CSM receives
traffic from a proxy on another machine and must trust X-Forwarded-For
from that proxy.
challenge:
enabled: true
trusted_proxies:
- "192.0.2.10"
- "198.51.100.0/24"
Entries are addresses or CIDR ranges. Legacy zero-padded prefix lengths
remain accepted. CSM skips other entries and never trusts a catch-all,
such as 0.0.0.0/0. A peer covered only by a refused catch-all names no
visitor when it forwards a request, even if truncation leaves an empty
header. This includes native IPv6 peers covered by a broad IPv4-mapped
range. A direct request from such a peer still names its own address.
csm validate and plain csm doctor warn about unusable entries; they do
not reject the configuration. Saving the challenge section in Settings
rejects them.
Trust only proxies you control, not ranges that also contain visitors. Each proxy must append its observed peer
address to X-Forwarded-For or replace the header with that address. The
challenge server also trusts the proxies in web_server.trusted_proxies,
the list the HTTP checks use.
CSM reads all X-Forwarded-For field lines as one ordered chain, from the right, past other trusted proxies, and takes the first other address as the visitor. An entry that is not an address ends the read.
If no visitor can be identified because the chain is missing, malformed, over 4 KiB, more than 31 trusted hops before the visitor, or contains only trusted proxies, challenge endpoints return HTTP 400. They do not issue a puzzle or cookie, accept a bypass, or clear a pending challenge for the proxy. This also applies to the gate and admin-token endpoints.
CSM rejects proxy trust for loopback, the unspecified
addresses, and every address on the host’s interfaces, link-local ones
included. Any local account can connect from those and write
X-Forwarded-For. An entry containing one of them does not prevent the
daemon or challenge listener from starting or reloading. Requests attributed
to a listed local address receive HTTP 400. Remote proxies in the same
range remain usable. csm validate and plain csm doctor warn;
csm doctor challenge reports the unusable local proxy as a failure, and
the webserver integration installer refuses it. This applies to both TCP
and Unix socket listeners. For a proxy on this host, use the
socket and remove host-local entries from
trusted_proxies.
While serving requests, CSM uses a cached interface address list, refreshed
on demand after five minutes. If a refresh fails, it keeps the last
successful list. An address added while the daemon runs can still be
trusted until a successful refresh discovers it. Before assigning this
host an address covered by trusted_proxies, remove or narrow the covering
entries to remote proxies and restart CSM. Only then assign the address.
Run csm doctor challenge to check the resulting configuration.
With trusted_proxies set, point challenge.public_url at the proxy.
listen_addr holds a host only; the port goes in listen_port.
csm doctor challenge leaves TLS to the proxy and does not ask for the
port gate. It does not check the proxy itself, so confirm HTTPS works at
public_url and keep listen_port closed to everything except the proxy.
A public_url that names the listener’s own address and port is direct
mode, and the usual TLS and port-gate checks apply.
With both challenge.trusted_proxies and web_server.trusted_proxies
empty, X-Forwarded-For from a TCP peer is ignored to prevent IP spoofing.
Only challenge.trusted_proxies says the listener itself sits behind a
proxy: the installer and csm doctor challenge’s TLS and port gate rows
read it alone, so a site CDN in web_server.trusted_proxies
does not change them. A listed address is never a visitor in either mode.
Proxy on the same host
A proxy on the same host reaches the challenge server through a Unix
socket. Only root and members of challenge.listen_group can open it, so
local accounts cannot pose as the proxy. Shared hosts that do not need a
proxy should use direct mode (see Bind address), which
trusts nothing.
challenge:
enabled: true
listen_addr: unix:/run/csm/challenge.sock
listen_group: nginx # the group your proxy runs as; nobody on cPanel
public_url: https://challenge.example.com/challenge
CSM creates the socket with mode 0660, owned by root and listen_group,
and removes it on shutdown. Its directory must be owned by root, not
writable by anyone else, and writable by the CSM service; with the shipped
systemd unit that means /run/csm. The path must be absolute, without
repeated separators, dot components or a trailing slash. It may use at
most 107 bytes, and its directory at most 91 bytes, since CSM first binds
under a temporary name that needs up to 16 more bytes. listen_port, the port gate
and the Web UI TLS fallback do not apply. The proxy must send
X-Forwarded-For; a request without a readable visitor address gets HTTP
400. trusted_proxies stays optional and lists any proxies in front of
the local one.
nginx:
location /challenge {
proxy_pass http://unix:/run/csm/challenge.sock;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
With a CDN or another proxy in front of nginx, list those proxies in
trusted_proxies and use proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; so the visitor address they add is kept.
Apache 2.4.7 or later, with mod_proxy and mod_proxy_http (it adds X-Forwarded-For itself):
ProxyPass "/challenge" "unix:/run/csm/challenge.sock|http://localhost/challenge"
The socket proves only that a request came through a process in
listen_group. Two things outside CSM decide whether that process can be
trusted, and CSM cannot check them:
- Tenant code must not run as the proxy’s user or group. On cPanel the web
servers run as
nobody, which is safe when PHP and CGI run as each account’s own user (suEXEC, LSAPI with suEXEC, per-account PHP-FPM). It is not safe with mod_php (DSO), CGI without suEXEC, mod_ruid2 or mpm-itk. Use direct mode on those hosts. - The proxy must take the visitor’s address from its own connection. It
must not accept forwarded-address headers from addresses on this host,
such as nginx
set_real_ip_from 127.0.0.1or ApacheRemoteIPInternalProxy 127.0.0.1(set up for ea-nginx in front of Apache). The proxy browsers connect to must be the one that talks to the socket.
LiteSpeed’s proxy emulation does not support this setup, and SELinux in enforcing mode may stop a confined web server from opening the socket. Use direct mode on those hosts.
csm doctor challenge checks the group and the socket directory, then
probes the gate through the socket.
Successful Verification
When a client passes the challenge:
- The IP is removed from the challenge list so the webserver stops sending that visitor to the challenge flow
- A verification cookie is set, so a visitor who is challenged again within its lifetime passes without seeing the puzzle
Passing the challenge changes nothing in the firewall. The visitor is treated like any other client from then on: normal rate limits and blocks still apply, and a later attack signal can list the IP again. The challenge page, the verify endpoint and the CAPTCHA fallback answer only for IPs currently on the challenge list; anyone else gets 404.
Webserver Integration
The webserver integration redirects challenge-listed IPs to
challenge.public_url. The installer refuses to run until that URL is
an absolute http or https URL ending in /challenge, and the
configured challenge listener uses non-loopback TCP or a Unix socket.
For a socket, challenge.listen_addr must use unix:/absolute/path and
challenge.listen_group must name the proxy’s group. Invalid socket settings
are rejected before the installer writes runtime maps or changes the webserver.
csm webserver-integration install # initial wire-up
csm webserver-integration upgrade # re-apply after a CSM upgrade
csm webserver-integration status # show detected stack + version drift
csm webserver-integration validate # run the webserver's configtest
csm webserver-integration remove # uninstall the snippet
The installer auto-detects the active webserver via
internal/platform. Supported stacks and snippet paths:
| Stack | Snippet path |
|---|---|
| cPanel + Apache (EasyApache) | /etc/apache2/conf.d/csm-challenge.conf |
| Debian/Ubuntu Apache | /etc/apache2/conf-enabled/csm-challenge.conf + per-site include |
| RHEL family Apache (httpd) | /etc/httpd/conf.d/csm-challenge.conf + per-site include |
| LiteSpeed (LSWS) | /usr/local/lsws/conf/templates/csm-challenge.conf |
| Nginx (plain + Engintron + phpanel) | /etc/nginx/conf.d/csm-challenge.conf |
Apache without cPanel applies the server-wide snippet inside no
<VirtualHost> on its own, HTTP or HTTPS. The installer also writes the
rule to /var/cache/csm/apache-challenge-site.conf, next to the maps, and
prints the one line each site needs. Add it inside every <VirtualHost>:
<VirtualHost *:443>
...
Include /var/cache/csm/apache-challenge-site.conf
</VirtualHost>
Then run apachectl configtest and reload Apache. CSM never edits site
files; csm doctor challenge warns while any <VirtualHost> in the files
Apache loads lacks the line. The check follows loaded includes through shared
configuration fragments and ignores includes in inactive conditions. If Apache
does not report its loaded configuration, doctor warns that coverage is unknown.
Uninstall leaves a disabled per-site include so existing site configuration
remains valid. Remove the Include lines before deleting that retained file.
On cPanel, Apache and LiteSpeed alike, HTTPS site blocks do not inherit the
server-wide snippet, so the installer also writes the rule to the per-site
include /etc/apache2/conf.d/userdata/csm-challenge.conf. It skips
/.well-known/ paths so certificate validation keeps working. cPanel adds
that include to every site block when it rebuilds httpd.conf with a file
present. csm webserver-integration install and upgrade run that rebuild
once when needed, inside the same configtest-then-reload run. The daemon’s
startup refresh writes the file but never rebuilds httpd.conf; HTTPS sites
are then covered after cPanel’s next rebuild, and csm doctor challenge
warns until they are. Coverage checks inspect each HTTPS site block and
require the per-site snippet to be present.
If a rebuild, configtest, or reload fails, the installer restores the
previous snippets and the saved httpd.conf, including its permissions.
It does not run a second rebuild during recovery. A failed restore or
recovery configtest prevents a recovery reload and reports the failure.
csm webserver-integration remove disables both redirects even if one
snippet is missing. It leaves a managed, comment-only per-site snippet:
cPanel’s required wildcard include must still match a file for the web
server to start. Removal never rebuilds httpd.conf. Installing again
replaces the placeholder with the redirect rule. Automatic startup
refreshes leave both snippets alone if either has a newer template version
than the running binary.
The snippets are rendered from the effective CSM config. Apache and LSWS
read their RewriteMap from /var/cache/csm/challenge_ips.txt; Nginx reads
a native map include from /var/cache/csm/challenge_ips.nginx.map. The
directory is the service’s systemd cache directory: it is world-readable so
the webserver user can read the maps, and unlike the runtime directory it
survives systemctl stop csm, package upgrades and reboots. That matters
because Apache and LSWS validate the map at config-parse time and Nginx
fails on a missing include, so a map that vanished with the daemon would
take every site down at the next webserver reload. CSM rewrites the Nginx
include on challenge-list changes and reloads Nginx only when the file
content changes.
Older installers also deployed a legacy proxy-mode snippet on cPanel
Apache and LiteSpeed (/etc/apache2/conf.d/csm_challenge.conf). Installs
no longer do. On those stacks csm webserver-integration install and
upgrade remove the legacy snippet in the same configtest-then-reload run
that writes the integration snippet, and restore it if that run fails.
If either file cannot be restored, the error reports the incomplete rollback
and no recovery reload runs with that incomplete configuration.
While both are present and the integration template is not newer than the
binary, status reports the integration as stale, so
the daemon’s startup refresh below removes the legacy snippet on hosts
that already run the integration. Newer templates are left alone at startup,
including after a binary rollback. Observe mode skips this refresh.
At startup the daemon creates the default maps if they are absent,
including when challenge mode is disabled. It also checks the legacy
snippet when one is still present. If
its RewriteMap points at a map file the daemon does not maintain – which
makes webserver config validation fail host-wide once that file goes
missing – the daemon re-deploys the shipped template. A CSM-managed
integration snippet from an older template version is refreshed the same
way, through the configtest-then-reload flow below. Files without the
directive, operator-edited snippets, and snippets already pointing at the
daemon maps are never touched. While such a snippet still references a map
under the old runtime directory, the daemon keeps that file present until
the snippet is refreshed with csm webserver-integration upgrade.
csm uninstall and the package post-removal hook remove the snippets
before the maps, and keep the maps whenever a snippet cannot be removed
(for example an Nginx server {} block that still uses $csm_challenged).
On every run, the installer:
- Writes the new snippet to a sibling temp file and renames it into place atomically.
- Runs the webserver’s own configtest (
apachectl configtest,nginx -t,lswsctrl conftest). - On pass: reloads the webserver gracefully and exits 0.
- On fail: restores the previous snippet bytes (or removes the file if it did not exist before) and exits non-zero with the captured configtest output. The webserver is never reloaded with a broken config.
Nginx config validation runs outside the daemon’s read-only sandbox via
systemd-run, so it can open its logs. The availability probe and the
configtest each get a separate 15-second command timeout. If the probe
fails or times out, or systemd-run is absent, CSM tries nginx -t
directly. A failed configtest is reported without retrying it.
The snippet header carries a version marker; upgrade is a no-op when
the on-disk content matches the rendered template and no legacy snippet
remains to retire. Hand-edited files
(missing or mismatched marker) trip a “modified” status and the
installer refuses to overwrite them - remove or rename first.
Hosts with no detectable webserver exit with status=skipped so
package post-install hooks succeed cleanly on, e.g., a plain phpanel
worker that doesn’t run nginx locally.
Bypass Paths
Three opt-in bypass mechanisms let legitimate traffic skip the PoW page entirely. All default off; an upgraded csm.yaml with no new blocks behaves exactly as before.
CAPTCHA Fallback (JS-Disabled Visitors)
The PoW solver requires JavaScript. Visitors with JS off (older mobile browsers, accessibility tooling, text browsers, scripted integrations) would otherwise be locked out. When configured, CSM renders a Cloudflare Turnstile or hCaptcha widget inside a <noscript> block; on completion the form posts to /challenge/captcha-verify and CSM validates the token server-side against the provider’s siteverify endpoint.
Provider rejections do not spend the page nonce, so a visitor can retry the
same challenge page after a mistyped, expired, or failed widget response.
challenge:
captcha_fallback:
provider: turnstile # turnstile | hcaptcha | "" (off)
site_key: "0xAAAA..." # public key embedded in the widget
secret_key: "0xBBBB..." # verified server-side; never sent to client
timeout: 10s
Verified Operator Sessions
Operators who repeatedly hit the challenge during normal admin work can mint a signed cookie that bypasses PoW for the cookie’s TTL. The signing key is generated at daemon startup and rotates on every restart – old cookies stop working automatically.
challenge:
verified_session:
enabled: true
cookie_name: csm_admin_session # default
ttl: 4h # default
admin_secret: "long-shared-secret" # required
To issue a cookie, POST the secret to the challenge server:
curl -i -X POST -d "secret=long-shared-secret" \
https://your-host:8439/challenge/admin-token
# 204 No Content
# Set-Cookie: csm_admin_session=...; Path=/; HttpOnly; Secure; SameSite=Lax
The cookie binds to the requester’s IP, so a stolen cookie does not work from a different network.
Requests from this server’s own addresses, loopback included, get HTTP 403 without a check of the secret, because every local account can send from them. Request the cookie from the network you browse from.
Verified Search Crawlers
Googlebot and Bingbot can be allow-passed by reverse-DNS forward-confirm. CSM looks up the visitor’s PTR, checks it ends in a known crawler suffix (e.g. .googlebot.com), then forward-resolves that name to confirm the original IP appears in the result. A spoofed User-Agent: Googlebot from a residential IP fails forward-confirm and falls through to PoW.
challenge:
verified_crawlers:
enabled: true
providers: [googlebot, bingbot]
cache_ttl: 15m
Positive results cache for cache_ttl; negative results cache for one-fifth that long so a transiently-broken resolver does not lock out a real crawler for the full window.
Operational
Backups
csm store export and csm store import capture the bbolt store, state JSON files (baseline file hashes), and signature-rules cache into a single tar+zstd archive. Use these for re-provisioning, cluster cloning, and disaster recovery rather than re-baselining a 200k-file account tree from scratch. The daemon writes the archive under its state directory (the only tree its systemd sandbox can write) and the CLI moves it to the path you give, so any destination the CLI can write works, including /var/backups. The destination directory and every directory above it must be owned by root or by the account running the command, and must not be writable by other accounts unless they carry the sticky bit as /tmp does; the destination itself must not already exist as a symlink or as another account’s file. Otherwise another account could swap the archive out after the CLI has verified it, and the export is refused before the daemon writes anything. In-progress staging files and unconfirmed firewall rollback artifacts are excluded from backups, so restoring an archive cannot re-arm an old confirmation window.
csm store export /var/backups/csm-$(date +%F).csmbak
sha256sum -c /var/backups/csm-$(date +%F).csmbak.sha256
# transfer the .csmbak + .sha256 to the target host
systemctl stop csm
csm store import /var/backups/csm-2026-04-27.csmbak
systemctl start csm
Partial restore: --only=baseline restores only the file-hash state (useful after a full re-install where firewall and history should stay fresh); --only=firewall merges the firewall buckets into an existing daemon (useful for cloning blocklists across a cluster); a durable firewall journal and its snapshot in the archive replace the target’s. It refuses while either side has a durable firewall action unresolved or its audit undelivered.
Performance Monitor
CSM monitors server performance metrics and generates findings when thresholds are exceeded.
A successful Fix clears only the finding for that file, and the finding alerts again if the problem returns.
Critical Checks (every 10 min)
| Check | What it monitors |
|---|---|
perf_load | CPU load average vs core count (critical/high/warning thresholds) |
perf_php_processes | PHP process count and total memory usage |
perf_memory | Swap usage percentage and OOM killer activity |
Host-wide OOM events are Critical; memory-cgroup limit events are Warning. Both are reported when present in the last hour, with separate deduplication identities for each scope and victim process.
Deep Checks (default every 60 min, thresholds.deep_scan_interval_min)
| Check | What it monitors |
|---|---|
perf_php_handler | PHP handler type (DSO vs CGI vs FPM) and configuration |
perf_mysql_config | MySQL my.cnf settings (buffer pool, connections, query cache) |
perf_redis_config | Redis memory limits, persistence, eviction policy |
perf_error_logs | Bloated error_log files in every document root, with growth rate |
perf_wp_config | WordPress wp-config.php hardening and debug settings |
perf_wp_transients | WordPress database transient bloat |
perf_wp_cron | WordPress cron scheduling (missed crons, excessive events) |
perf_wp_loopback | WordPress sites calling themselves faster than WordPress’s own schedulers, hour after hour |
perf_wp_myisam | WordPress database tables still on the MyISAM storage engine |
Bloated error logs
perf_error_logs scans the configured web roots plus every document root in
cPanel’s domain map, so addon and subdomain sites are covered as well as
public_html. It walks three levels deep. Heavy trees such as wp-content,
wp-admin and vendor are not descended, but an error_log directly inside
one is checked: PHP writes the log next to the running script, so
admin-ajax.php errors land in wp-admin/error_log.
A log over performance.error_log_warn_size_mb (default 50) is a Warning and
only shows on this page. A log over performance.error_log_critical_size_mb
(default 1024) is High and alerts. Once a log has been seen for at least an
hour, the finding also shows how fast it grows per day. A log that shrinks or
is replaced by a new file, as on truncation or rotation, starts a new
baseline. A cancelled scan changes nothing; a scan that could not read part
of the tree keeps the logs it did not reach and updates the ones it did.
The finding keeps the same identity while the log grows, so alert email and the webhook send it once, history records it at most once a day, and a dismissal sticks until the log crosses into the higher tier. When more than 20 logs are bloated, the largest are reported.
WordPress loopback requests
A WordPress site runs background work by sending a request to itself, a loopback. WordPress’s cron spawn and Action Scheduler’s admin dispatch each hold a 60-second lock, so one job reaches its own site at most once a minute.
perf_wp_loopback counts the last three complete local clock hours in every
active vhost log. It counts POST requests with WordPress’s own User-Agent
from the loopback interface or one of the server’s addresses. The count is
kept per job, meaning the full logged path plus the admin-ajax action;
shortened display names do not merge jobs. A job over 60 an hour in each of
the three hours is reported as a Warning on this page. The finding shows the
hourly counts, how many runs failed with a 5xx error, and the User-Agent.
Request timestamps record when requests started, but log lines are written when they finish, so no part of a log can be skipped by its timestamps. The check follows each active log instead: every run reads only what the log gained since the previous run, and the hourly counts are kept in the scan state. A rotated, truncated or replaced log is read from its start, and the first run reads at most the last 8 MB of each log. Memory for distinct jobs is bounded, and the scan stops at its deadline, including inside oversized lines. A log that cannot be read, or exceeds the job budget, keeps its saved position and its prior findings until a later run succeeds.
There are two usual causes:
- A plugin schedules its job for “now” and fires it from every page view. The job then runs about as often as the site is visited, crawlers included.
- A background queue such as Action Scheduler keeps re-dispatching itself because its backlog never drains.
The hour in progress is not counted. A site that reaches itself through a proxy such as Cloudflare is missed when its log records the proxy’s address rather than the server’s. Other tenants can also make requests from the server’s addresses, so this Warning is advisory: it does not establish which site initiated a request, send alerts, or trigger a response action.
WordPress MyISAM tables
MyISAM locks a whole table for every write. While one request writes, every other request that reads or writes that table waits. On a busy site a burst of uncached requests queues behind those locks until it holds every database connection the account may open, and new visitors get database errors. InnoDB locks single rows, so the same traffic keeps flowing.
perf_wp_myisam reads MySQL’s table catalogue once per run, as root and
without site credentials, and matches the MyISAM tables against the
WordPress installs found by the shared discovery. Each install’s database and
table prefix come from explicit string literals in its wp-config.php,
including an explicitly empty prefix. PHP expressions are not evaluated.
A table belongs to the install whose prefix matches it most closely, so two
sites sharing one database are reported separately. Dormant and suspended
installs still reserve their prefixes. Each database and prefix gets one
Warning on this page, listing the tables largest first with their total size.
Not reported:
- installs whose
DB_HOSTis another server; accepted hosts arelocalhost, loopback addresses and IP addresses bound to this server’s interfaces - TCP connections whose port differs from the catalogue server’s port, and explicit local sockets whose path differs from that server’s socket path
- hostname aliases other than
localhost; DNS is not used to infer ownership - document roots the panel no longer serves, and suspended accounts
An unreadable or unresolved configuration, or a failed catalogue query, keeps prior findings until a later run succeeds. If a sibling’s database or prefix cannot be established, affected scopes are withheld rather than assigning its tables to a shorter prefix. Configurations using dynamic or conditional database settings need explicit literals for this check to resolve their scope.
To convert a site without losing data:
- Put the site in maintenance mode so nothing writes during the change.
- Take a full database dump and confirm it completed.
- Run
ALTER TABLE <table> ENGINE=InnoDBfor each listed table, one at a time, and stop at the first error. Each statement copies the table and blocks writes to it while it runs. - Compare row counts before and after, then end maintenance mode.
Keep the server’s strict SQL mode for the conversion. A table holding values InnoDB would reject then fails with an error instead of having those values silently adjusted.
Web UI
The Performance page (/performance) shows real-time metrics:
- Server load and CPU usage
- PHP process and memory charts
- MySQL and Redis health
- WordPress performance indicators
PHP worker counts include LiteSpeed and PHP-FPM pool workers. The PHP-FPM master is excluded from these request-worker counts; security process checks continue to inspect it. Database memory comes from the MySQL or MariaDB server process, independent of its PID-file location or wrapper-related arguments.
The findings list also exposes admin-only fixes, per-row and as a Bulk fix dropdown that applies one fix to every matching finding at once:
-
perf_error_logs: truncate a bloatederror_login place. The inode is preserved so running PHP processes keep writing to the same file. -
perf_wp_config: disabledisplay_errorsin.user.ini,php.ini, or.htaccessby commenting the matched line and appending an Off override. -
perf_wp_cron: adddefine('DISABLE_WP_CRON', true)towp-config.phpand install a per-user system cron that runswp-cron.phpon a fixed interval. Disabling WP-Cron alone would stop scheduled WordPress tasks, so the cron is installed in the account owner’s own crontab (visible and editable by the customer). The cron is installed before the define is written, so a crontab failure leaves WordPress scheduling unchanged. The define is inserted before the “stop editing” marker (or thewp-settings.phprequire); insertion points inside multiline comments or heredocs are ignored, and the fix refuses awp-config.phpwith no safe insertion point rather than corrupt it.The installed schedule is staggered per account and docroot (for example
7-59/15) instead of a wall-clock-aligned*/15, so many managed sites do not all fire in the same second and spike the host load. Non-divisor intervals use a shifted minute list so the gap stays within the configured interval. The command also runs underflock -nwith a per-docroot lock file in the account home, so a slow pass skips the next run instead of overlapping it. When bothauto_response.enabledandauto_response.fix_wp_cronare enabled, managed crontab lines installed by older releases are upgraded on daemon start to this format. Their PHP interpreter changes only when an override is configured or the site’s PHP version can be resolved unambiguously; otherwise, they keep their existing interpreter. Only lines under the# CSM WP-Cronmarker are touched; customer-authored cron entries are never rewritten.
These actions are limited to configured account roots, reject symlinks anywhere in the path and unsupported file types, refuse a file that changes before the edit lands, and remove the fixed row from the active findings view after a successful edit.
WP-Cron fix settings
Tune the WP-Cron remediation under Settings -> Performance:
performance.wp_cron_fix.interval_minutes(default15, range 1-60): how often the installed system cron runswp-cron.php. The interval only bounds task latency – WordPress keeps its own event schedule and a cron pass with nothing due is a wasted full bootstrap – so 15 minutes is right for most sites. Lower it per host only when busy stores need tighter Action Scheduler latency.performance.wp_cron_fix.php_bin(default empty): overrides the PHP interpreter for the cron line. Leave it empty and each site runs under the PHP version its own vhost is set to when cPanel’s domain map provides an unambiguous version. A vhost set to inherit runs the version named by the nearest cPanel-generated handler block that maps.phpin its.htaccesschain, up to the account home. Empty blocks leave the parent mapping in effect. The lookup refuses symlinked directories and files, special files, and oversized files; an unreadable nearer file stops inheritance. CSM falls back to the detected CLI interpreter when neither gives a usable, installed version; an existing managed job keeps its interpreter while the map is unavailable. Setting a value pins that one interpreter for every managed site. CLI php is used instead of an HTTP request so the job never ties up a web worker.
To let the daemon apply this fix automatically on every WP-Cron finding, set
auto_response.fix_wp_cron: true (default false; requires
auto_response.enabled: true). It is opt-in because it edits customer
wp-config.php files and crontabs.
MySQL telemetry auth
The MySQL panel runs mysql -e "SHOW STATUS LIKE 'Threads_connected'" from
the csm process. The client needs to authenticate against the local server,
and csm supports two setups out of the box:
-
A
~/.my.cnffor the csm runtime user with credentials for a MySQL account that holds at least thePROCESSprivilege. cPanel and CloudLinux ship/root/.my.cnffor the root user; csm running as root picks it up automatically. -
A unix-socket grant for the csm OS user, e.g. on Debian/Ubuntu MariaDB:
CREATE USER 'root'@'localhost' IDENTIFIED VIA unix_socket; GRANT PROCESS ON *.* TO 'root'@'localhost';
If neither is configured, the MYSQL card renders n/a / n/a instead of a
misleading 0 conn. csm makes no attempt to connect over TCP or store
credentials on its own.
Redis telemetry auth
The Redis panel connects to local Redis at 127.0.0.1:6379. If Redis
requires a password, set REDISCLI_AUTH in the csm daemon environment.
The dashboard uses that password for its in-process Redis client.
If csm cannot read Redis, the REDIS card renders n/a instead of an empty
Redis. A Redis with no memory limit shows no limit with neutral styling.
API
GET /api/v1/performance Current performance metrics snapshot
POST /api/v1/perf/fix-error-log
POST /api/v1/perf/fix-display-errors
POST /api/v1/perf/fix-wp-cron
See the API reference for which metrics
fields can be null.
Web UI
HTTPS dashboard that updates live from the finding event stream (see Live updates). Dark/light theme toggle.
Static assets use content-versioned URLs. Only a URL matching the served file receives immutable caching; older or unversioned URLs must revalidate. Replacing a file changes its version even when its size and modification time are preserved. Text assets use gzip when accepted by the client, except for byte-range requests.
Navigation
The sidebar groups pages by operator workflow. URLs are stable; the groups only reorder visibility:
- Overview - Dashboard
- Triage - Incidents, Findings (Active and History tabs)
- Response - Firewall, Quarantine, Cleanup History, Email Security, ModSecurity, Threat Intelligence
- Operations - Performance, Server Hardening, Audit Log
- Configuration - Rules, ModSecurity Rules, Verified Bots, Settings
A page has the same name in the sidebar, the browser tab and its heading.
The old /blocked address redirects to /firewall.
Sidebar group expand/collapse state is saved in the browser. On
viewports under 992px the sidebar collapses into a top-bar drawer
toggled from the hamburger button. Account detail (/account?name=<account>) is
not in the sidebar; the finding detail, account groups on Findings, incident
detail, the accounts targeted in a Threat Intel lookup and the dashboard’s
accounts-at-risk card link to it, and the command palette opens an account
typed by name. The palette also lists Sessions. Browser logins require administrator
scope. The header links to session management.
Pages
| Page | URL | Purpose |
|---|---|---|
| Dashboard | /dashboard | Triage queue, daemon status strip, Components matrix, system posture, 24h stats, accounts at risk, auto-response summary, brute-force summary, timeline charts. A queued finding opens its own detail, and each 24h severity count opens the History tab for the last 24 hours at that severity |
| Findings | /findings | Active findings with search, check/account filters, header grouping toggle, detail panel, fix/dismiss/suppress actions, a permanent Block for findings that report an attacker address, sticky bulk operations (fix, dismiss, suppress), modal account scan. The open finding is kept in the URL as ?key=<finding key>, so the link reopens it |
| Findings > History | /findings?tab=history | Paginated archive of all findings, newest first, with date range and severity filters, 25 to 200 rows per page (hperpage), CSV export; window=24h (1 to 720 hours) shows a rolling window instead of calendar days |
| Quarantine | /quarantine | Every file backup: quarantined files and cleaners’ pre-clean backups, with type, live state of the original path, content preview, restore and delete; filters by account, detector, type and date |
| Cleanup History | /cleanup-history | DB-object backups with preview and restore controls; file backups are on the Quarantine page |
| Firewall | /firewall | Subview-tabbed page (?view=overview/blocks/allow/config/audit/danger; ?ip=<address> opens the lookup for that address): blocked IPs/subnets with GeoIP, bulk unblock of selected rows (with undo), the whitelist and allow rules (Allow Rules tab), search, audit log; the lookup links to Threat Intel for the same address; destructive actions live under the Danger tab |
| ModSecurity | /modsec | WAF workbench: status strip, Active WAF pressure summary list (top attackers by hits), top rules / domains side panel, Blocked IPs / Events tabs, and a Manage Rules link to ModSecurity Rules. Block detail panels show first-seen, top URIs, sample events, and direct links to Threat Intel, Firewall lookup, and rule management |
| ModSecurity Rules | /modsec/rules | Enable or disable CSM rules (applied with one reload) and firewall escalation exclusions; the exclusion list works even when rule management is not configured |
| Email Security | /email | Headline numbers for the selected range, each opening the tab that explains it, then job tabs with counts: Overview (mail decisions chart, rspamd, groups needing attention, protection), Queue (why each message is stuck, with retry, return to sender, delete and thaw), Held mail (quarantined malware and held forward copies, to release or delete), Suspended (mailboxes and accounts CSM suspended, with live status and Release; releasing never changes the password), Rejects, Sending accounts (mail per hosting account, or per mailbox login on a Postfix host, and abuse events), Forwarders (each forwarder and where it sends, with delete per destination; a domain’s default address is changed in cPanel), Provider delays (providers and sending IPs deferring mail, and why) and Findings. ?tab= names the open tab. |
| Verified Bots | /verified-bots | Editor for the verified-crawler allowlist (reputation.verified_bots): UA, reverse-DNS suffix, and IP-range identities, plus auto-update posture, with apply-and-reload. Admin scope |
| Threat Intelligence | /threat | IP lookup with scoring/GeoIP/ASN (?ip=<address> runs it on load), 24 hour and permanent block and whitelist actions (single and bulk), top attackers, attack type charts, trends; the lookup links to Firewall for the same address, and the whitelist itself is kept under Firewall > Allow Rules |
| Server Hardening | /hardening | On-demand hardening audit, stored report, score, and remediation guidance. Web server directive checks follow Apache quoting, section comments, continuations, and shell include patterns. Quotes in ignored section suffixes do not cause warnings. Malformed section endings, unreadable includes, unresolved directive names, and unsupported wildcard classes produce warnings. On Apache or cPanel hosts, a configured Apache root takes precedence over default locations; a missing selected main configuration never falls back to another tree. Other servers skip the Apache audit when that main file is absent. |
| Incidents | /incident | Correlated incident list and grouped view, both filterable by every status, with detail panel and bulk status changes (contained, resolved, dismissed) for the selected incidents on the page, plus forensic timeline search by IP or account |
| Rules | /rules | YAML/YARA rule management, suppressions, state export/import, test alerts |
| Account | /account | Per-account analysis: findings, quarantine and history. An on-demand account scan runs from the Findings page |
| Audit Log | /audit | Every operator action in the Web UI and API, including logins, logouts and session revocations, with the credential that acted, search, action and date filters, URL state, and export. Failed logins go to the daemon log instead |
| Performance | /performance | Server load, PHP processes, MySQL, Redis, WordPress metrics |
| Settings | /settings | Searchable config editor with grouped large sections, field-level validation errors, restart notices, redacted secret updates, and firewall tentative apply with rollback timer. Commands, file paths, sockets and environment variable names are shown read-only and change only in csm.yaml; changing the rspamd or upstream address requires entering its credential again |
| Sessions | /sessions | Active browser logins, individual revocation and logout of every session (confirmed first) |
Audit attribution is captured when the action is authorized and remains available if the browser session expires or is revoked while the action runs.
Account views use the recorded finding owner when available, with account paths as a fallback. They also recognize resolved paths under linked account roots, including files already moved into quarantine.
Threat intelligence spam evidence
The Spam column shows spam and ham verdict counts and the current spam score. Click its heading to sort by score. The Spam evidence only filter requests senders with spam verdicts, including those with no attack events. The filter is kept in the URL for bookmarks and history navigation. When switching filters, previous rows stay unavailable for actions and export until a load succeeds.
Spam evidence about IPv6 senders is listed per /64 network. Such a row is named by the network and carries a Network badge. It has no checkbox and no block or whitelist buttons, and clicking it does not open the IP lookup. Looking up an address inside the network shows whichever spam evidence is stronger: the address’s own or the network’s. Ties use the network, which is named whenever its evidence is shown.
The Listing threshold badge means the spam score qualifies for the CSM IP map. It does not confirm an active listing: the feed must be configured and working, and address exclusions still apply. Spam scores remain separate from the unified threat score and firewall block status.
Dates and time zones
Every page shows dates in the time zone chosen under Preferences: the browser’s, the server’s, or a named zone. Date filters pick whole days in that zone. Changing the zone reloads the page so dates already on screen follow it, provided the browser can save and read back the preference. If browser storage is unavailable, new renders use the preference without forcing a reload. Days with a midnight clock change start at the first valid time of that day; a repeated midnight uses its first occurrence.
Email chart hours also follow partial-hour daylight saving changes. Custom date links include both selected days, even when supplied in reverse order. Long custom ranges group adjacent days into labeled spans without dropping counts. Changing the range discards pending results for the previous range.
Live updates
The Web UI keeps an event stream open (/api/v1/events) and shows a Live
indicator while it is connected. Findings are batched for up to a couple
of seconds before pages update:
- Findings shows the “new findings” banner at once instead of on its next check
- Dashboard sends desktop notifications and refreshes the 24h counts and the triage queue
- Incidents reloads the current list, unless incidents are selected for a bulk change
While the stream is connected, the checks those pages run on a timer slow down to once a minute as a safety net; when it drops they return to their normal pace. Pausing auto-refresh also pauses live updates.
Desktop notifications handle findings that arrive out of order within a batch or share a timestamp, without repeating them on the next history poll.
Refresh
The header shows when the page’s data was last loaded (“Updated N ago”). It moves when the page loads its data, on each automatic refresh and on the Refresh button; an action or a detail lookup does not change it. The pause button appears only on pages that refresh on a timer, and pauses that refreshing in this browser.
Refresh reloads the page’s data in place, keeping filters, scroll and open panels. On a page with unsaved edits (Settings, Verified Bots, staged ModSec rule changes) it asks before discarding them. Hardening’s Refresh reloads the stored report and does not run a new audit. Pages rendered by the server, such as Sessions, reload.
A panel that fails to load says what failed and why, with a Retry button that loads only that panel again. The panel’s earlier content returns once a later load succeeds.
The dashboard shows only what the host supports. Email Security appears when Exim, Postfix or Stalwart is detected; a mail server started after CSM shows up within five minutes. Its Queue, Held mail, Forwarders and Provider delays tabs, the antivirus lines of the Protection card, cPHulk actions on the Firewall page, the dashboard’s Email AV chip, and the Email AV, cPanel and webmail login, and known API token settings need cPanel and are not shown elsewhere.
On a host with no PHP interpreter, the PHP Shield, WordPress, WP-Cron, XML-RPC
and PHP process settings and the Performance page’s PHP cards are hidden; PHP
installed after CSM starts shows them within five minutes. PHP-FPM or lsphp in a
container counts as PHP. A container running only Apache with mod_php has no
separate PHP process and is not found. The PHP Shield settings show for
container PHP only when that container has the shield script at its usual
path. Hidden settings keep their values in csm.yaml, including when you save
another field in the same section.
The Suspended tab shows on cPanel and Stalwart hosts. On a Stalwart host it lists the mailboxes whose sign-ins CSM stopped, and Release allows them again. On a Stalwart host the first three Email Security cards show the mail Stalwart’s spam filter did not file to Junk, the mail it rejected or discarded, and the mail it filed to the Junk folder. The state line and mail decisions chart use Stalwart’s filter verdicts, independently of CSM’s antispam mode. The Spam rejected card opens Overview and has no mode badge. It shows a dash and “none logged yet” until CSM has a retained rejection or discard, including events outside the selected range. After that, it shows the range’s rejection count and the discarded count below it. The Mail delivered card also names unmatched verdicts. Missing verdict counts from an older daemon display as zero; they never fall back to rspamd counts. Large card counts are shortened, with the full count available in the value’s tooltip. The rspamd card and the Antispam line of the Protection card are not shown, as they describe CSM’s own antispam.
Rejects and Sending accounts read the Exim main log, or the Postfix mail log on a host without Exim. On Postfix a send is mail accepted from a client that logged in, counted under that login; mail relayed for a trusted host without a login is not counted. A Postfix reject whose reason cannot be told apart from text the client sent is listed under Other. Both tabs stay empty on a Stalwart host.
On Email Security, failed refreshes mark protection status as unavailable and hide stale chart data. An older response cannot overwrite newer health status or the selected outbound abuse range. Headline counts refresh every 30 seconds. Queue messages and composition, held mail, rejects, sending accounts, provider delays and Findings refresh every minute while their tab is open. Hidden tabs do not poll their lists; the Held mail, Sending accounts and Provider delays counts refresh every minute on any tab. On hosts without Stalwart’s filter, the rspamd state in the header also refreshes every minute on any tab. Unchanged results keep their rows; changed results retain paging, sorting, selection, open details and focus for items still present. Pending message, bulk and firewall actions stay disabled across refreshes. Queue actions wait up to a minute; an unreadable outcome is reported as unknown. Held mail custom ranges include only items held during the selected days; the summary also identifies held items outside that range. The other ranges have no end, so a server clock ahead of the browser cannot hide mail just held. Rejects, Sending accounts and Suspended use the Held mail layout, with a search box, a row count and 25 rows per page. Search rejects by sender, recipient, IP or detail, sending accounts by name, and suspensions by mailbox, owner account or reason. The address keeps the search. A new search starts on the first page. A refresh keeps the current page while the search and filters stay the same. Rejects lists up to the newest 200 rejects in the range and says when the list reaches that limit.
The visible History tab refreshes with its current filters and page size. Editors wait for an ongoing save or load before accepting another refresh; Verified Bots keeps edits made while a reload is pending. ModSecurity exclusion controls wait for the exclusion list to load, and a failed load stays visible instead of appearing as an empty list. Late responses cannot replace a newer account tab, settings section, history page, or finding detail, or reopen a finding detail that was closed.
Notifications
Success and information notices fade after five seconds. An error stays until you close it, and the same error is not shown twice while it is on screen.
Confirmations
A confirmation for an action that deletes data, blocks traffic, turns protection off or ends sessions names the action on a red button, such as Delete or Block, and starts with Cancel focused, so pressing Enter does not carry it out. Logging out every browser session asks first. Cancel keeps focus after the dialog finishes opening. A later text prompt uses its own normal OK button, and bulk actions keep one confirmation open at a time.
Bulk file actions
Select-all and every bulk action reach only the rows the table currently shows. Rows on other pages or hidden by a search or filter are never selected or acted on; set the page size to All to act on every row. Quarantine selection counts and buttons are refreshed whenever the visible rows change.
A failed file restore cleans up its own destination copy while retaining the quarantined evidence. A replacement created by another writer is preserved.
A mail message quarantined from a phishing finding is listed as two files, its header and its body. Restoring either one returns the whole message to the mail queue, and Exim delivers it on its next queue run. The complete body is returned before the header becomes visible to Exim. Restore refuses an incomplete message or a conflicting queue entry and keeps the quarantine copy. If restore or quarantine cleanup fails, retry the same entry. Progress survives a daemon restart, and a cleanup retry does not requeue an already delivered message. A header quarantined on its own can be restored if its body remains in the queue.
Quarantine deletes large file selections in sequential batches. If a request fails, later batches are not sent; the page reports the confirmed deletion count and refreshes the list. File restore and delete controls stay disabled until the operation and refresh finish, and Refresh waits for them.
Threat Intel bulk block and whitelist actions accept up to 100 selected IPs and retain one undo action. Larger selections must be narrowed before sending. Findings bulk fix also asks for a smaller selection when the request would exceed the API body-size limit, which includes finding details.
Findings bulk suppress creates one path rule per selected file, up to 100 files, with the file name matched literally. Selected findings that name no file are skipped; a rule for a whole check is made from one finding. Rules are saved one at a time, and a failure stops the rest and reports how many were saved.
Security
- Authentication - API bearer tokens in the header; opaque server-side browser sessions in HttpOnly/Secure/SameSite=Strict cookies
- CSRF - HMAC-derived token bound to the browser session on cookie-authenticated POST, PUT, PATCH, and DELETE requests; a form sends it in the body, never the query string
- Headers - X-Frame-Options DENY, Content-Security-Policy (scripts, styles and forms from the Web UI only; no plugins,
<base>or framing), HSTS, nosniff, and the legacy XSS auditor turned off - TLS - Auto-generated self-signed certificate, renewed automatically within 30 days of expiry and picked up without a restart; a certificate you install is never replaced, and replacing its files takes effect on the next connection. Renewal keeps the existing private key, so a failed certificate write leaves the working pair intact; explicitly configured certificate and key files must already exist
- Rate limiting - 5 credential attempts/min per IPv4 address or IPv6 /64, shared by login submissions and unknown bearer tokens on all authentication routes, including login GETs and unknown API paths. Concurrent requests share the limit; valid UI and metrics tokens used outside their scope do not consume it. Once exhausted, the budget rejects bearer requests before checking credentials. API and
/metricstraffic also has a 600 requests/min limit. - Token expiry - dashboard, API and metrics credentials expire at their configured
expires_atand never expire without one; expired credentials, malformed dates and future issue times are refused - Token sources - a token with
allowed_ipsis accepted only from those addresses and CIDR ranges, matched against the TCP peer - Token length - tokens shorter than 32 characters are reported as warnings at startup and by
csm validateandcsm doctor; they keep working - Bearer auth skips CSRF (for API-to-API calls)
Credential rotation
A credential expires only at its expires_at, a UTC RFC3339 time: set it
on a webui.tokens entry, or use webui.auth_token_expires_at and
webui.metrics_token_expires_at for the legacy and metrics tokens. A
credential without expires_at never expires, so configurations written
before these settings existed keep working. The optional issued_at
(webui.auth_token_issued_at, webui.metrics_token_issued_at) records when
the token was generated; expires_at must be later. Integrations should
narrow the scope (unban for customer self-unblock), list their source
addresses in allowed_ips, and set an expires_at that matches how often
they rotate. Reloads and restarts preserve the deadline. Active event
streams recheck the credential before sending each event or heartbeat.
Browser sessions and their form tokens use the credential that created the
session. Expiry of another administrator’s credential does not interrupt them.
To rotate, replace the token with a fresh random value of at least 32 characters, update its dates and distribute it to clients through your secret manager. Restart CSM after changing login or API credentials. Metrics token and date changes can be applied together with SIGHUP. Changing a date alone is not rotation.
Package installation keeps newly generated credentials out of subprocess arguments. Rotate credentials generated by earlier packages because local users could have observed them in process listings during installation.
An expired credential or an invalid date disables that credential and
produces a warning at startup and through csm validate and csm doctor. Detection and
alerting continue. If all admin credentials expire, update the root-owned
configuration locally and restart CSM to recover dashboard access. Use
token.expires_at.v1 in the capabilities API to detect this policy.
Browser sessions
Log in with an administrator credential from webui.tokens (or the migrated
legacy webui.auth_token). The cookie contains a new random session secret;
the reusable API credential never appears in it. Old token-valued cookies are
rejected, so an upgrade requires a fresh login. Read-scope API tokens cannot
create browser sessions.
Use Sessions in the header (/sessions) to see login names, client address,
browser, creation time, last activity and absolute expiry. Revoke one session
or log out every browser, including your own. These operations do not rotate
API credentials. Logout uses a CSRF-protected POST. Logout and revocation
accept the same browser origins as API writes (webui.allowed_origins); the
login form does not check the origin.
webui:
session_lifetime: "24h"
session_idle_timeout: "30m"
Both durations require a restart. Lifetime must be between one second and 30 days; idle timeout must be at least one second and no longer than lifetime. Zero does not disable expiry. Idle time means time without operator activity: page loads and API requests made within a minute of keyboard, pointer or scroll input. Background polling by an open page does not count, including metrics scrapes, HTML page fetches and event-stream connections. Browser navigation to a page counts as a page load; fetching that page on a timer does not. The activity marker is recalculated when each API request is sent, so a dashboard left open still logs out after the idle timeout. Activity is committed at bounded intervals, so idle expiry can occur slightly early, never late. Passive event-stream heartbeats do not extend the session; streams check revocation and expiry before each event and heartbeat.
A browser session also loses access when its login credential expires. Every daemon restart invalidates all browser sessions. Token removal, rotation, name or scope changes take effect after the required restart; log in again with a current administrator credential. Reauthentication creates a new session and revokes the previous one. Operator preferences remain tied to the login credential, so a new session does not reset them.
The local transactional store keeps session verifiers, never raw cookie secrets.
Failed persistence cannot issue a login or claim successful revocation. If the
previous session cannot be read during reauthentication, login fails without
issuing a replacement cookie or changing that session. Concurrent requests
preserve the latest committed activity even when they arrive out of order.
If the session store is unavailable, browser authentication fails closed; API bearer
authentication remains independent. Session admission is bounded and refuses new
logins at capacity instead of evicting active sessions. The login response says
so; wait for idle sessions to expire, or revoke sessions from a logged-in
browser or with an admin API token (DELETE /api/v1/sessions). Expired sessions are removed during admission, and startup clears
the session records. Backup exports exclude live session records, and full
restores discard any session records from older archives.
MFA is a separate planned feature. See the session management guidance for the security principles behind opaque identifiers, expiry and revocation.
Keyboard Shortcuts
General
| Key | Action |
|---|---|
? | Show shortcut help |
/ | Focus search input |
Ctrl-K / Cmd-K | Open command palette |
Navigate
| Key | Action |
|---|---|
g d | Go to Dashboard |
g f | Go to Findings |
g h | Go to Findings > History tab |
g t | Go to Threat Intel |
g r | Go to Rules |
g b | Go to Blocked IPs (Firewall) |
Findings page
| Key | Action |
|---|---|
j / k | Move selection down/up (focus moves to the row) |
o / Enter | Open selected finding |
d | Dismiss selected finding |
f | Fix selected finding |
Finding and incident rows, finding group headers and sortable table headers also work from the keyboard: Tab to them and press Enter or Space. A sorted header reports its order to screen readers. Closing the detail panel or a dialog returns focus to what opened it, and a dialog opened from the detail panel keeps Tab and Escape to itself. If a refresh removes the opener, focus returns to the open detail panel or main content. Shortcuts act on the focused finding and stay inactive while a dialog or detail panel is open.
Each finding row offers up to four actions: Fix (apply the automated remediation, shown only when one exists), Re-check (re-evaluate the finding against the live filesystem and clear it if the condition is gone, useful after fixing something by hand instead of waiting for the next scan), Dismiss (stop alerts for it while it stays unchanged; a later scan that still finds it lists it again, and undo is offered for 30 seconds), and Suppress (create a rule to hide similar findings for good). A finding cleared by Fix or Re-check is new again when a later scan finds it, so a reinfection alerts. A detection or dismissal that arrives while Fix or Re-check runs stays in place.
Dismissing a finding about a file’s content, such as a malware match, covers that content only. When the file changes, the finding alerts again as a new detection, and automatic responses can act on it. A scan that could not read the file does not end its dismissal. The “new findings” banner appears when a listed file’s content changes. If the file changed after the page loaded, the dismissal is refused and the list refreshes with a warning. A bulk dismissal still dismisses the unchanged findings, and undo covers only those.
Dismissal undo preserves later dismissals, successful re-checks and baseline resets. Findings first received in real time stop alerting when dismissed, even before the next scheduled scan records them.
Re-check appears only when CSM can test a current condition again. Supported
targets include file permissions and content, phishing and .htaccess files,
selected accounts and system integrity checks, WordPress core/plugins, CMS
database rows, administrator accounts, and database objects. Re-check uses the
stored finding identity and current host state; the browser cannot substitute a
different path, row, account, or object.
YARA Re-check compares the complete scanned content with the finding’s recorded
fingerprint, including large files admitted by thresholds.full_scan_max_file_mb.
If the content is unchanged and YARA no longer flags it, the finding clears.
The operation fails closed. Missing or unreadable evidence, package-manager errors, failed CMS discovery, and failed database queries leave the finding active. A file whose bytes changed since detection is never cleared either, because a partial clean and an evasion edit look alike. When the replacement can be proven inert, an empty file or a comment-only stub, the finding drops to Warning instead of clearing, and the original severity comes back if the file stops being inert. Historical events such as login attempts, WAF blocks, and IP reputation cannot be re-evaluated and therefore have no Re-check action. Broad aggregates and dependency findings require a new account or full scan.
WHM Plugin
CSM installs a WHM plugin (addon_csm.cgi) that redirects operators from WHM to the daemon Web UI. After the redirect, API calls are same-origin requests to the daemon.
API requests that carry a browser Origin header are accepted from https://<hostname>:<port> (the configured hostname and webui.listen port), from an https loopback origin such as https://localhost:9443 over an SSH tunnel on any local port, and from every origin listed in webui.allowed_origins (bare https://host[:port] entries, hot-reloadable). A loopback origin is accepted only when it is the origin the request was sent to (its Host): another local service in the same browser shares the Web UI’s cookies, which ignore the port, and is refused. Any other origin gets 403 Cross-origin request blocked, which shows up as a read-only UI: pages load but every action fails. The Host header is used only for this loopback comparison; it never admits a non-loopback origin. A preflight (OPTIONS) from an accepted origin is answered with the methods and request headers the API takes, so a credentialed JSON write from a listed origin goes through.
API Reference
Machine-readable HTTPS API. All endpoints require token authentication. State-changing POST, PUT, PATCH, and DELETE requests require CSRF protection for browser cookie sessions.
Authentication
# Bearer token (header)
curl -H "Authorization: Bearer YOUR_TOKEN" https://server:9443/api/v1/status
# Cookie-based (after login)
curl -b "csm_auth=SESSION_COOKIE_FROM_LOGIN" https://server:9443/api/v1/status
Cookie-authenticated state-changing requests require the X-CSRF-Token header (obtained from the authenticated page meta tag). The token belongs to the browser session that loaded the page; another session’s token is refused, and a form field is read from the request body only. Admin-scope Bearer requests are CSRF-exempt because the Authorization header is the write credential.
Browser session management
A successful browser login exchanges an admin token for an opaque session cookie. API tokens are never valid cookies. Sessions expire on idle or absolute deadlines and all end on daemon restart. The header’s Sessions page provides the same management operations as these admin-only endpoints:
| Method | Path | Result |
|---|---|---|
| GET | /api/v1/sessions | items list of sessions with id, name, created, last_seen, expires, remote_ip, user_agent, current |
| DELETE | /api/v1/sessions/<id> | Revoke that session; an unknown ID answers 404 |
| DELETE | /api/v1/sessions | Revoke all browser sessions, including the caller’s |
Revocation returns {"ok":true} only after committing to the store. Failure
returns 503; invalid session IDs return 404. Lists contain no credential hashes
or session verifiers. Cookie-authenticated DELETE requests require CSRF; admin
bearer callers are exempt. Read-scope bearers cannot list or revoke sessions.
POST /logout requires authentication and CSRF for browser callers; it revokes
the current session and clears the cookie. GET /logout cannot log users out.
Token scopes
Configure tokens under webui.tokens: with a scope of admin, read or unban:
webui:
tokens:
- name: "operator"
token: "..."
scope: admin # full read+write
issued_at: "<UTC RFC3339 time when this token was generated>"
- name: "panel-readonly"
token: "..."
issued_at: "<UTC RFC3339 time when this token was generated>"
scope: read # status, findings, history, stats, challenge stats, blocked IPs, scan jobs, health, components, capabilities, SSE
- name: "panel-unban"
token: "..."
scope: unban # GET /api/v1/firewall/check and POST /api/v1/firewall/unban only
issued_at: "<UTC RFC3339 time when this token was generated>"
expires_at: "<UTC RFC3339 time when this token stops working>"
allowed_ips: ["203.0.113.10", "198.51.100.0/24"]
A credential expires at its expires_at and never expires without one. The optional issued_at records when the token was generated. Replace the placeholders above with fresh random tokens and real times. A malformed or future issued_at, an expires_at that is malformed or not after issued_at, and a past expires_at refuse authentication. See credential rotation.
allowed_ips limits a token to the listed addresses and CIDR ranges, for API calls, browser login and the browser sessions it starts. CSM matches the TCP peer address and ignores forwarded headers. A token without allowed_ips is accepted from any address. An entry that is not an address or CIDR fails configuration loading.
IPv4-mapped addresses and CIDRs within the mapped range match IPv4 peers: ::ffff:192.0.2.0/120 matches the same peers as 192.0.2.0/24. CIDR host bits are masked. Entries with interface zones, such as 2001:db8::1%eth0, fail configuration loading; use an address without a zone. A peer’s interface zone does not affect address matching.
Unban-scope tokens are meant for hosting panels that let customers lift their own blocks. They cannot log in to the dashboard, read status or reach any other route, and they are exempt from CSRF only on those two routes.
On POST /api/v1/firewall/unban, an unban-scope token removes the address’s
block and its mail spray re-block marker. Spray failure history and SMTP,
IMAP and POP failure counts remain, including shared IPv6 /64 evidence and
other addresses’ markers. Renewed spraying can still trigger a block.
An admin token or browser session on the same route performs a full operator
release, clearing the address’s mail failure history and counts as described
in automatic blocking.
Either credential also removes subnet blocks CSM added automatically that
cover the address (subnet_removed in the response). A subnet an operator
blocked is kept and named in subnet_kept: the address stays dropped until
that subnet is unblocked.
Operator descriptions do not grant automatic-unban eligibility. If a subnet
changes during the request, CSM leaves the replacement intact and returns
HTTP 500 with a retry message.
Older subnet entries without recorded automatic ownership are also kept,
even if their source label says automatic; they expire normally or can be
removed explicitly through the admin-only subnet removal endpoint.
The credential accepted at authorization selects the release mode; expiry
while reading the request body cannot change it. A valid browser session
takes precedence over a bearer token and still needs CSRF with an unban bearer.
A failed individual-IP removal returns HTTP 500 without clearing threat state or reporting a successful unban. A failed subnet removal also returns HTTP 500; the individual-IP release and any earlier subnet removals have already taken effect. A firewall backend without self-service unban support returns HTTP 501 for an unban-scope caller; it never falls back to an operator release. A backend without conditional subnet removal returns HTTP 501 when a covering automatic subnet needs removal. The cPHulk flush remains best effort and runs only after the firewall removals succeed.
The legacy single-token webui.auth_token:, with its optional webui.auth_token_issued_at and webui.auth_token_expires_at, is loaded into a legacy-auth-token admin entry. Read-scope tokens are intended for orchestrators and dashboards that consume status, findings, history, stats, challenge stats, blocked-IP summaries, scan jobs, health, components, capabilities, and SSE events. Admin scope is still required for write routes and for sensitive reads such as quarantine, settings, firewall internals, threat-intel detail, rules, account detail, exports, incident timelines, and audit history. (ModSecurity stats/blocks/events are read scope; only the ModSecurity rules and escalation routes need admin.) metrics_token: is a separate, read-only credential for /metrics only.
Read-only MCP endpoint
Support agents that speak the Model Context Protocol get these reads at
/mcp on the same listener, with their own credentials; see
MCP endpoint. It cannot reach any route that changes state.
Errors
Every failure answers a non-2xx status with a JSON body that has an error
message. That includes CSRF, origin, rate-limit and wrong-method refusals.
Some failures add detail next to it; a settings save that fails validation
adds errors, a list of fields and messages. An unknown path under /api/
answers 404. No 2xx response reports a failure.
{"error": "invalid IP address"}
Actions
A request that changes state answers "ok": true with the action’s own
fields, such as undo_token or warning. Work that continues after the
answer, such as a daemon restart, a firewall rollback or a scan job, answers
202. A batch where some items failed answers 200 and lists the failures; a
batch where nothing changed answers an error status. A read-only route
answers 405 to any method but GET.
An empty fix batch is rejected. Threat actions report firewall failures, and bulk actions list invalid addresses as well as failed changes. A failed action can have applied some steps before the error; inspect the current state before retrying it. A response that cannot be encoded answers 500 with an error body.
/api/v1/firewall/check and /api/v1/firewall/unban also send
"success": true for callers written against the older API. It will be
removed; use the status code and ok.
Lists
Every GET route that returns a list answers a JSON object, never a bare
array. The list is under items. An empty list anywhere in a response is
[] and an empty map {}, never null; null means a value is not known.
totalis the number of matches the server counted. It is larger than the length ofitemswhen the route pages or cuts the list.offsetandlimitcome with routes that page or cap the list. Capped routes that do not support paging reportoffset: 0.truncatedis true when matches were left out: past the page, past the limit, or past a scan cap. When a scan cap stopped the count,totalcounts only what was scanned. Incident groups also sendscan_truncatedto distinguish a scan cap from a page limit; theirtotalis exact whenscan_truncatedis false.- Other keys next to
itemsdescribe the whole list, such ascheck_typeson/findings/enrichedorsummaryon/email/forwarders.
/audit, /threat/top-attackers and /threat/events do not count every
match. They send limit and truncated without total.
{"items": [{"ip": "203.0.113.9", "reason": "wp_login_bruteforce"}], "total": 1}
Times and durations
Every time is an RFC 3339 instant in UTC with sub-second precision, such as
2026-09-22T10:04:05.123456Z. The same applies to the event stream. A time
that is not set is left out, never sent as 0001-01-01T00:00:00Z.
The API does not send times it formatted for reading, such as “5m ago”,
“1h2m” or a clock time without a date. Clients format instants in their
own time zone and count down to an expires_at themselves.
Durations are numbers of seconds in keys that end in _seconds, such as
uptime_seconds, elapsed_seconds, duration_seconds,
oldest_age_seconds and update_interval_seconds.
Temporary whitelist responses use duration_seconds, rollback status uses
remaining_seconds, and incident timelines use window_seconds. Request
parameters such as hours and editable configuration values retain their
documented units.
Two values keep a text form. started_at_token is an opaque token for
restart polling. The temporary reason on /api/v1/firewall/check keeps its
“(expires in …)” text for existing callers; expires_at carries the
instant.
Severity
A severity is its label: CRITICAL, HIGH or WARNING. That covers
severity, severity_max, demoted_from and the attack event sev.
Clients derive colours and sort order from the label. A severity filter
takes the label in any case, or the older 0, 1 or 2 level. The findings
inside an email quarantine entry keep the scanning engine’s own rating.
Status & Data
GET /api/v1/status Full health snapshot: version, uptime, watchers, severity counts,
store health, blocklist size, capabilities[], config_hash, binary_hash,
automation rollout state, challenge pending count, rollback state.
`started_at_token` changes after a daemon restart and is suitable
for restart polling.
`security_posture` is `healthy`, `warning`, or `critical` after
combining daemon faults with active high/critical incidents.
`incidents_open_by_severity` breaks open and contained incidents
down by `critical`, `high`, and `warning`.
`correlation_attribution` (present after the first active-set
merge) lists per check the findings correlation could not
attribute to an account: `current` for the active set now,
`cumulative` since daemon start.
`queues` reports named protection queues, including depth,
capacity, running work, recent and cumulative drops, and lag.
A degraded queue changes `status` and `security_posture`.
`backends` names the backend each started live monitor
selected (`af_alg`, `connection_tracker`, `exec_monitor`,
`sensitive_files`, `placement`): `bpf`, `legacy` or `none`.
`latest_scan` is the canonical last-scan timestamp; `last_scan_time`
is a legacy alias kept for older clients and will be removed.
`uptime_seconds` is the time since the daemon started.
GET /api/v1/challenge/stats Challenge-routing activity for the UI: `pending`, `escalated`
(timeouts that became hard blocks), `passed` (challenges solved
or bypassed, since restart), `routed_by_check` (per source
check, since restart), and `recent` routes. Read scope.
GET /api/v1/capabilities Static feature list (e.g. `confd.dropins.v1`, `events.sse.v1`,
`webhook.phpanel.v1`, `webui.prefs.v1`, `webui.undo.v1`,
`mail.queue.composition.v1`, `mail.forwarders.delete.v1`,
`detect.http_scanner_profile.v1`, `challenge.stats.v1`,
`firewall.rollback.v1`, `firewall.dos_exempt.v1` on Linux builds).
Use for orchestrator feature-detect. `csm capabilities` prints
the same build tokens without a running daemon.
GET /api/v1/components Watcher/component matrix with attachment, event, and upstream freshness state.
GET /api/v1/events Server-Sent Events stream of findings as they dispatch.
Read-scope token sufficient. One JSON event per `data:` line.
Writes and flushes have a three-second deadline. A failed
write or flush closes the stream and frees its subscriber slot.
GET /api/v1/health Daemon health (fanotify, watchers, engines)
GET /api/v1/findings Current active findings
GET /api/v1/findings/enriched Enriched findings with GeoIP, accounts, fix info, and a list version.
?limit=N orders by severity, then newest, even when all rows fit;
returns at most N rows with limit and truncated, while total and
the severity counts cover all.
?fields=version returns only {version, total}, for change polling.
block_ip is set only for checks that report an attacker address,
the same evidence auto-block acts on
GET /api/v1/finding-detail Finding detail with action history (?check=&message=)
GET /api/v1/history Paginated history (?limit=&offset=&from=&to=&severity=&search=&checks=&scope=).
checks is a comma-separated list of check names to include.
scope=email limits results to the /api/v1/email/checks list when checks is absent.
total counts every match; truncated is true when matches exist past the page
GET /api/v1/history/csv CSV export of the newest 5,000 entries matching the /history filters
GET /api/v1/stats 24h severity counts and auto-response summary, accounts with active high or critical findings
GET /api/v1/stats/trend 30-day daily severity counts; each `date` is a calendar day in the server's time zone
GET /api/v1/stats/timeline Hourly severity counts for the last 24 hours; each bucket names its `start` instant
GET /api/v1/quarantine Quarantined files with metadata (incl. htaccess pre_clean backups)
GET /api/v1/quarantine-preview Preview quarantined file content (?id=)
GET /api/v1/db-object-backups db_object_backups bucket (MySQL trigger/event/procedure/function drops)
GET /api/v1/db-object-backup-preview Preview captured CREATE SQL (?key=)
GET /api/v1/blocked-ips Blocked IPs with reason and expiry
GET /api/v1/accounts Accounts a server-wide scan covers
GET /api/v1/account Per-account findings, quarantine, history (?name=)
GET /api/v1/audit Newest 200 UI audit log entries; truncated marks older ones. Each entry names the credential that acted (actor) and whether it came as an API token or a browser login (via)
GET /api/v1/export Export state (suppressions, whitelist)
GET /api/v1/incident Incident timeline (?ip=&account=&hours=)
GET /api/v1/performance Performance metrics snapshot (admin scope)
POST /api/v1/perf/fix-error-log Truncate a fixed-row error_log finding (admin scope, CSRF)
POST /api/v1/perf/fix-display-errors
Disable display_errors for a fixed-row config finding (admin scope, CSRF)
POST /api/v1/perf/fix-wp-cron Disable WP-Cron and install a system cron for a perf_wp_cron finding (admin scope, CSRF)
GET /api/v1/hardening Last stored hardening audit report (admin scope)
Performance telemetry
GET /api/v1/performance returns metrics and findings. In metrics,
mysql_mem_mb, mysql_conns, redis_mem_mb, redis_maxmem_mb and
redis_keys are null when csm could not read them; zero is a measured zero.
The two Redis memory fields are read together, redis_keys separately. A
redis_maxmem_mb of 0 means Redis has no memory limit. The csm_performance
MCP tool returns the same response.
Accounts at risk
accounts_at_risk in GET /api/v1/stats lists accounts with a current high or
critical finding, whatever its age. Suppressed findings and internal records are
left out. findings counts all of the account’s current findings, warnings
included, and severity is the highest. At most 50 accounts, ordered by
severity, then finding count, then name. An empty list is [].
Date ranges
/api/v1/history, /api/v1/email/groups, /api/v1/email/overview,
/api/v1/email/rejects, /api/v1/email/senders and
/api/v1/email/relay-abuse take from and to as
a calendar date (YYYY-MM-DD) or an RFC 3339 time. A date is
a day in the server’s time zone, and to includes the whole day. An RFC 3339
to is exclusive. The web UI sends RFC 3339 times so a day follows the
operator’s time zone preference. A value in neither form is rejected with 400.
An empty or whitespace-only bound is treated as absent. Calendar days start at
their first valid local time, including when midnight is skipped or repeated;
a skipped calendar date covers an empty range. Fractional seconds are preserved
when filtering RFC 3339 bounds. On the /api/v1/email/ endpoints a to
without from covers the 24 hours before to.
WordPress verification coverage
wordpress_verification contains core and plugins coverage when installation
history is available. The same fields appear in csm status --json; human status
shows the counts. Each entry reports verified, modified, unverified, unknown,
not_wordpress and last_attempt. unknown means discovery found a directory
but no completed attempt is recorded. modified means core verification
completed with integrity differences, including missing core files. A verified
plugin inventory can still contain vulnerable versions. The timestamp is the
latest recorded attempt in that group.
Counts retain the last observed results across restarts and cached scan cycles.
Overlapping scans retain the latest attempt and its consecutive failure history
in scan order. Discovery without an attempt does not replace verification evidence.
An absent group means there is no retained installation history, not proof of
coverage. An error field reports unreadable history instead of clean counts.
The first failed attempt increases unverified; repeated failures also produce
an installation-specific Warning. In host scans, when one cause stops more
installations than the per-cause limit, the warnings collapse into a single
finding that names the cause, the total and a sample, so a host-wide fault cannot spend the whole
alerts.max_per_hour budget. The limit applies across accounts for each kind of
verification and cause. Account scans keep installation-specific warnings.
A summary has no installation path and names an account only when all affected
installations share it. These findings are separate from queue health:
a command that completed with a refusal did not lose queue work. Existing queue
loss accounting remains responsible for interrupted or missing command work.
Firewall confirmation
automation.firewall_confirmation describes a pending CLI apply-confirmed
window, also when the firewall is disabled or its engine failed to start; it
is absent when none is pending. deadline is the saved deadline in UTC when
it can be read, and error says why the window cannot settle without
csm firewall confirm or repair of the reported recovery problem. It is
separate from the Web UI configuration rollback timer.
Automatic response admission
status.admission.v1 advertises the admission object on status responses.
The same view appears under admission in csm status --json and as
admission rows in csm doctor. The daemon reads it on a timer, so
checked_at says how old it is. Every automatic response the blocking
paths select is also handed to the ledger, which refuses disabled
responses. With auto_response.dry_run false, the firewall on and outside
observe mode, the ledger applies the automatic IP blocks it is handed
(owner.executing) and the blocking paths leave those blocks to it.
Otherwise it previews or forecasts eligible work and never applies it, and
the view is not an enforcement record.
ledgerholds the stored admission time, queue, lifetime queue counters (including newly checkpointed ingress drops), hourly and daily outcomes, ingress generations, the hourly ceiling with its general and reserved lanes, storage, the outbox, notice records andunresolvedoutcomes (runningandunknowncounts and when the next unknown one can retire), andeffects: the temporary set slots the ledger’s own automatic blocks hold, pending and verified, ingeneralandreserved, withnextwhen the first of them expires. Theeffectssection reports anerrorif its records, tally or schema disagree, including damage beyond the first expiry. Each section carries its ownerror, so one damaged record does not hide the rest. Ingress drops checkpointed before this accounting change are not added retroactively to lifetime counters. On an upgraded host, retained outcome windows can therefore exceed the corresponding lifetime totals. A restart does not fill that historical gap. Once the ledger has started live, a preview ends as a dry run does: the outcome windows and the candidate’s state show it asdry_run, while the hourly comparison counts it asobserve.ingresssays whether automatic responses are admitted and, while they are not, since when and how many Critical arrivals were refused.ownersays whether admission runs automatic blocks (executing), whether it has completed a turn since the daemon started (settled), and names why the ledger owner is not running (error), why its last turn failed while it runs (turn_error), the last fault met while retiring uncertain blocks (retirement_error, kept until none is left faulted), what keeps unknown outcomes from retiring while the ledger holds some (retirement_held_by:enginefor an owner without the firewall engine, as with the firewall disabled or in observe mode,policyfor a refused execution policy; their addresses stay held). It is omitted otherwise, including while a configured firewall engine has not started. Policy refusals and recovery update this cause with the execution hold, without waiting for the next ledger sample. Once the cause is gone, each block still retires only after it has expired and the firewall shows nothing of it. While the owner has the firewall engine,block_entriesnames the entries whose automatic IP blocks the live configuration lets admission run on this host (scan,challenge_timeout,incident,incident_spray,central); the comparison gate expects each to be seen. The owner also reports why the last clock reading or its live start failed (tick_error), whether the clock is degraded, where the ceiling came from (ceiling_source:default,configuredorclamped), the legacy hourly count imported at the first start (import, which counts at least until an hour afterat, or names why the count could not be read at the first start or when execution began; a committed read error survives startup retries), the last inventory read and why responses naming a damaged record were discarded (damage_error). The first completed turn publishessettled: truewith any earlierturn_errorcleared. Later failed turns and policy reloads do not resetsettled; a daemon restart does.preview_replacements,preview_replacements_last_hour,preview_rotationsandpreview_rotations_last_hourremain in the API for compatibility and read zero in this build. Live blocks execute and dry-run blocks forecast; neither increments these preview counters.- An owner that runs on the firewall engine, with the firewall on and
outside observe mode, adds
execution_heldwithexecution_held_by(policyfor a refused execution policy,firewallfor the firewall engine), averdictobject for the panel verdict callback, whose counters count this process and restart at zero, adry_runobject for its dry-run forecast and asetobject for the temporary block set outside a dry run, both described below, withset_errorwhen the last turn could not read the set.verdict.enabledis true while the owner itself asks the panel: with the callback configured, during a dry-run forecast and while it runs automatic blocks itself (executing). This staging route does not advertiseadmission.dry_run_verdict.v1; that capability requires the complete admission-gates contract and activation evidence.
owner.set combines the non-admission occupants from the last scheduling
turn with the current ledger holds, divided as automatic admission divides it: limit,
unlimited, capacity, unclaimed, imported and drain_seconds as in
the forecast’s occupancy below, and general and reserved with each
partition’s size, the legacy and admission blocks it holds (used) and its
free slots. A schedule serves block work, previewed or run, into free
slots of the partition its lane occupies, or into the slot of a
lower-priority automatic block it may replace, including a legacy automatic
block that recorded its origin; other block work waits as
set_full and is checked again when a held slot expires or after a minute.
The object is absent when the last scheduling turn found no ready block
work, in dry run and when the set could not be read. A maintenance-only
turn keeps the last non-admission snapshot unless the owner enters dry run;
status still refreshes the ledger holds. Entering dry run, including through
a reload, clears both set and set_error before the next clock reading or
forecast is attempted. Engine admission counts are not
added again when the ledger holds their slots. Any excess counts as
unclaimed and makes drain_seconds unknown (-1), since the engine’s
drain estimate excludes admission rows. This view is not a fresh kernel inventory.
owner.dry_run is a capacity forecast, not a record of blocks. active is
true while the owner forecasts instead of blocking. session_id, since
and reset_reason (startup, mode_change or model_error) name the
forecast session and are null while none has started, including after a
failed start; a session starts at
startup, on entry into dry run or after its model failed, and leaving dry
run keeps the last session’s identity. modelled_kinds and
prediction_scope say what the model covers: block_ip against the hourly
ceiling and the temporary block capacity. occupancy gives the real
temporary occupancy (actual, addresses plus two elements per timed subnet),
the forecast addresses the real sets do not hold
(shadow_only), the listed legacy automatic blocks the forecast replaced
(replaced), their sum minus replaced (predicted), the limit, whether the set is
unlimited and whether retained forecast state fills its modelling bound (limited);
it is null while no snapshot is available. actual and imported count at
least the listed imports, matching the partition counts. A replaced import
is subtracted only while listed, until its own expiry, even if its replacing
forecast block expires or is replaced first. Unlisted imports count again;
a renewal with a new name is a new import. capacity is the finite capacity
automatic admission is modelled within: a positive limit, or 10000 for an
unlimited set. Until validation rejects negative limits, they use the same
fallback as zero and retain their configured value in limit. unclaimed counts the real timed entries it never evicts (operator
blocks, blocks of unknown origin, two per timed subnet and admission
blocks no ledger record holds), imported the
real verified stamped legacy automatic blocks, and drain_seconds estimates how long the last of
them has left at the snapshot, or -1 when a kernel expiry is unavailable.
Zero means no remaining known whole seconds. Treat -1 as unknown, not as
expired; a combined horizon is unknown if any contributing horizon is unknown.
New blocks, renewals and subnet wall-clock
changes can extend that estimate; it is not an automatic removal deadline. general and reserved divide the slots the unclaimed
entries leave, a fifth rounded up reserved for direct or corroborated work,
each with its size, the real and forecast automatic blocks it holds
(used; after a live period the slots the ledger’s records still hold
count in their own partition, read each turn) and its free slots, also
bounded by total remaining capacity. The drain is unknown while admission
blocks have no ledger record. Real blocks are never forecast
replacement victims and their scopes do not share a partition, so a
forecast scope may be due more than live admission would give it.
shares gives, for general and reserved, the scopes that share the
partition (those holding its forecast blocks the real sets do not hold and
those the last turn’s picks wait in) and the fewest slots each is due
(share, the size divided among them; the remainder goes one each to the
scopes after the session’s cursor), both zero without a scope. All share
keys remain present, including zeros; without a snapshot, occupancy is
null instead of an object with missing share keys.
Existing occupants keep their partition when limits change. In a full set,
a forecast block may replace a strictly lower-priority automatic block: a forecast block of a lower class or
severity, or a listed legacy automatic block that recorded its origin,
which the forecast counts as gone while still listed, until the import’s
own expiry. A pick of a new attack may instead rotate out a forecast block
of exactly its class and severity in the partition of its own lane once
that block’s 30-minute hold has passed on both the admission clock and the
session’s elapsed time (capacity_rotate). The set’s scopes share each
partition: a pick at or over its share rotates only its own scope’s
blocks, one below it only the scope that borrows most, and one without a
share waits. A pick that waits while such a block holds its partition
predicts equal_tier_wait, not set_full. A rotated or replaced
forecast block’s address cannot rotate again until that block would have
expired. An overfull set admits none, and entries counted in unclaimed
are never replaced.
Real coverage replaces forecast occupancy: a
manual timed block reduces available capacity, an automatic block counts in
general until ledger partitions are wired, and permanent coverage takes no
temporary slot. ceiling is the forecast’s own
copy of the hourly ceiling: limit, general_used, reserved_used,
general_budget and reserved_budget, null while no forecast runs. picks
counts the session’s predictions by name and last_hour the last sixty
one-minute buckets of process monotonic time, an approximation that a
wall-clock change cannot age; every prediction name is present, and both
are zero while no forecast runs. examples_lost, summaries_lost and
record_errors count this process’s losses and failed writes,
examples_stale is true while a purge of earlier sessions’ examples is
pending, and comparison_incomplete is true after any loss in this
process; a restart alone does not set it, since each run’s hourly
summaries carry its epoch.
forecast_error names why forecasting last failed (engine_unavailable,
clock_degraded, ledger_unavailable, snapshot_mismatch,
session_id_unavailable, invalid_policy or model_error) until a turn
succeeds, and last_error_at keeps its time until the next session starts.
Absent values are null, never missing keys. The durable outcome windows in
ledger keep counting dry-run endings across sessions and restarts.
The hourly ceiling follows auto_response.max_blocks_per_hour, capped at 20000.
An omitted or zero value selects 2000 for the ledger and 50 for the shared
scan-driven block budget. An explicit value is used by both, and csm doctor
warns when it puts the ceiling below 2000. After editing the configuration, run
systemctl reload csm; web UI settings saves apply at the next ledger tick.
See automatic response for the scope of the current
budget.
While admission previews or forecasts, existing blocking paths still decide
every block. Admission rows say existing blocking is unaffected, and
admission execution warns while execution is held. While the owner
executes, the rows say admission decides automatic blocks, and the outages
that stop automatic blocks, held execution included, fail. The dry-run rows of an
owner that forecasts (admission dry run, admission dry-run capacity and admission dry-run examples) show the forecast’s session,
real against predicted capacity and lost examples. The capacity row also
shows the automatic slots of each partition and how long the entries
automatic admission never evicts take to drain, the legacy automatic
blocks the forecast counts as replaced, the forecast blocks of the
last hour that would have replaced a lower-priority automatic block and
how many scopes share each partition; a
replacement is an admission, not a wait. A dry run alone is not a
failure; a held forecast, model limits, waits the model predicts
(equal_tier_wait among them), forecast blocks of the last hour that
would have rotated out an equal-priority block, a set
those entries fill, no free automatic slot, an unknown drain time and lost
or stale examples warn. Recent rotation and wait warnings remain visible
when the forecast has no current capacity snapshot; current occupancy and
scope shares are then unavailable. These warnings clear when they leave
the last-hour window or the forecast session ends.
Outside a dry run, admission set shows the slots the last turn budgeted
block work by. It warns about unavailable capacity or an unreadable set.
The retained preview replacement and rotation counters remain zero in
this build and do not describe live replacements or rotations. Rows warn when
the ledger owner is not running, its clock reading
is refused, an owner turn fails (admission turns), retiring an uncertain
block fails (admission retirement), the ledger is damaged, admission has stopped or a Critical
finding would not have received its response in the last hour; a
partial-containment notice, whose block ran, does not count. They also
show a degraded clock, a clamped ceiling, a ceiling of 1 (only the reserved
lane runs), a legacy hourly count that could not be read, a failed
inventory read and automatic blocks with an unproven outcome, which hold
their addresses until they expire and the firewall shows nothing of them.
Once owner.executing is set, rows say admission decides automatic blocks,
and held execution, a stopped owner, a refused clock reading, a failing
turn, a damaged ledger, stopped admission, discarded responses, a Critical
gap in the last hour and a ledger the firewall could not read at startup
fail. Until owner.settled is set, and for at most two minutes after the
daemon starts, those rows warn and say so: the check that follows an
upgrade runs before the first turn, and its failure would roll the upgrade
back. A Critical gap recorded before the daemon started only warns, and
says so: a flood of the hourly ceiling could otherwise fail every check
after a restart for an hour. The admission execution fix names the
configuration for a refused policy and the firewall otherwise.
admission.audit and admission.notices in queues report the delivery of
admission records to the action log and of admission notices.
admission.owner.audit_error names the last failed audit delivery, including
an applied-block consumer’s state read or threat-store write. The batch stays
pending and later rows can wait behind it; queues["admission.audit"] reports
no-progress stalls. Successful delivery clears the error. Keep the database
and pending rows; repair the reported storage failure before retrying.
A ledger that cannot start, or admission that stops, raises one
auto_response_withheld alert through the same path as protection queue
alerts, a Warning while admission previews or forecasts and Critical while
it executes; it follows the notification channel settings but bypasses
alerts.max_per_hour. Ledger notices (withheld responses, exhausted
capacity, partial containment of an outbound destination and their
summaries) raised while the owner previews or forecasts describe what the
ledger would have done, so they are recorded in findings history as
Warnings starting with “Admission preview:” and sent to no alert channel.
Those raised while it executes go to the alert channels with their kind’s
severity and no prefix. The ledger paces them per notice key and summary. A preview is acknowledged only after its history write succeeds.
Delivery errors leave notices pending for retry:
the next two cycles retry at once, then each attempt waits twice as long as
the one before, up to an hour. Ledger notices and the stopped-admission
alert retry independently, so a failing path does not defer the other;
a successful delivery resets only its path’s wait.
Protection queue health
status.queue_health.v1 advertises the queues map on status responses.
The same measurements appear under snapshot.queues in csm status --json
and as named checks in csm doctor.
findings.ingest, fanotify.analyzer, fanotify.staged_packages,
fanotify.dropper, fanotify.dropper_findings and spool.scanner report
waiting work, capacity, running work, cumulative losses,
losses during the last minute, the oldest waiting item’s age and the oldest
running item’s processing time.
Waiting work includes producers blocked on admission. Ingest work remains
running while the dispatcher holds or processes its batch, including startup.
Ages pause while the startup hold is in place and resume from its release, so
a long baseline scan is not reported as a stall; work lost during the hold
still counts. Releasing a hold preserves any future eligibility time for
deliberately deferred work.
Loss totals include the undelivered tail of a batch canceled during shutdown;
scan warnings intentionally excluded from alerts do not count as lost work.
Recovered file and spool scanner panics count as lost scan work. The workers
continue processing later events, but repeated failures still degrade health.
fanotify.reconcile reports the recovery queue in directory tasks, with
1,024 waiting slots. Repeated drops in a directory retain its original queue
age. A detached scan batch remains running while new drops queue separately,
including new work for a directory that is already being scanned.
Eviction, work older than the recovery scan window, unreadable directories or
candidate files, interrupted batches and unfinished shutdown work count as
failed recovery tasks. An incomplete task may still have scanned some files;
each directory task counts at most once. Directories and files that no longer
exist when recovery runs hold nothing left to scan and complete the task,
which keeps bulk extraction and package restores out of the loss counters.
The recovery window includes its cutoff; work completed exactly at that
boundary does not count as expired.
Staged package verification reserves capacity for its whole running batch.
Its full-queue timer starts when admission fills the queue and continues while
the verifier retains those slots, including between retries.
Files awaiting another attempt retain their original waiting age; retrying
does not reset lag. Shutdown counts files still awaiting verification after
the analyzer workers have stopped. Package metadata I/O cannot block health
polling for this queue.
Dropper candidates and held findings each have 16,384 waiting slots. Their
lag starts after the configured unlink TTL or the finding’s 45-second grace
period, respectively; intentional waiting does not count as overdue work.
Detached probes and emission batches remain visible as running work.
Processing time starts when each batch leaves its waiting queue, independently
of the timestamp used to decide which work is eligible.
Retries and refreshed observations retain earlier eligibility, and exhausted
probes, capacity refusals and unfinished shutdown work count as losses.
mail.delivery reports the selected mail-log reader’s 64-slot delivery queue.
Its running time includes parsing and delivery of resulting findings. Losses
include oversized complete file records, unreadable journal entries, canceled
admission, consumer failure and buffered records abandoned on shutdown.
Counters survive reader replacement and changes between file and journal
sources. They count records already read, including complete records abandoned
in file read-ahead, not unread source history or partial file records that have
not reached a newline.
mail.file_source reports sampled untransferred bytes after the first file
attachment, with depth_unit: bytes and capacity_unavailable: true. This
includes disk backlog, buffered read-ahead and raw partial-line bytes, even when
the reader retains only a bounded prefix of an oversized line. Transfer to
mail.delivery removes the source bytes before output admission can block.
These byte and record measurements are separate stages and must not be added.
A reader-owned sampler checks the current descriptor every two seconds, so
appends remain visible while delivery is blocked. Health reads memory only.
lag_basis: consumer_progress measures time without consumption while unread
work remains. Sampling or new arrivals do not reset this clock. A partial line
at EOF waits for more input and does not report a stalled consumer. Reader and
sampler operations also report processing_lag after one minute without
progress, including blocked reads, metadata calls and cleanup.
File I/O failures report source_io. A successful read cannot clear a failed
cursor check; each operation must recover. Failed metadata sampling makes depth
unavailable until a successful sample. Rotation and observed truncation retire
the old generation, and late samples cannot overwrite its replacement. The
initial attachment still starts at EOF; replacements start at the beginning.
Copytruncate detection retains its existing limit: truncation followed by growth
past the old descriptor offset between checks may be indistinguishable from an
append.
Complete records already held in read-ahead count once in delivery loss when
discarded. Unread disk data and unterminated fragments have no measured record
count; generation changes, source failures and shutdown preserve
dropped_lower_bound in the source row. A working file or journal replacement
clears the retired source’s current error while retaining that history. Source
path disappearance still follows the watcher’s existing grace period.
mail.journal_source reports the journal reader after its first successful
attachment. The cursor exposes no exact unread-record count or queue capacity,
so depth_unavailable and capacity_unavailable are always set, and
lag_basis: unavailable marks the missing backlog age rather than reporting a
waiting time of zero. The row measures the current operation instead. A selected
entry remains in flight until delivery acquires it; known unreadable or abandoned
selected entries count once in mail.delivery. The two stages must not be added
together.
Journal progress follows cursor advancement, entry reads, bounded idle waits,
output admission and close. One minute without operation progress reports
processing_lag, including a stuck cursor with no known selected entry. Cursor,
entry, wait and close failures report source_io; known abnormal exits remain
visible through actual cleanup. An unknown cursor outcome or unread shutdown
sets dropped_lower_bound without inventing a record count. Successful reads
restore current health; historical uncertainty survives reader replacement.
Successful attachment of a file source clears the retired journal’s current
error. Failed attachment leaves it visible. Health reads memory only and does
not call the journal or change its tail positioning, retry or delivery policy.
When their BPF backends are active, bpf.af_alg.output,
bpf.connection.output, bpf.execution.output and
bpf.sensitive_files.output report the 256-slot userspace delivery queues.
Their losses include decoding failures, admission refusals, consumer panics
and buffered output left after the reader stops. A received event remains
running until evaluation and delivery to the finding queue finish, including
events intentionally filtered during evaluation. Kernel ring occupancy and
reservation failures are separate from these userspace measurements.
bpf.placement.output, bpf.placement.kernel and
bpf.placement.unresolved report whichever
placement backend
runs: placements waiting for a worker, placements the kernel lost, and
placements whose path could not be resolved. The legacy backend adds
bpf.placement.create_settle for new files waiting for their own
close-write. When this queue fills, its oldest entry is sent for early analysis
and counted as a drop from the settle queue. Pending entries discarded at
shutdown also count as drops. A full output queue counts the placements it
refuses as drops and hands them to overflow recovery, which rescans the
directory a file landed in and walks a moved-in directory.
placement.dir_walk reports directories moved into a watched tree that wait
for their walk, counted in directories.
Evicted directories and walks cut short count as lost and raise
placed_dir_incomplete_realtime.
bpf.execution.docroot reports up to 1,024 non-root process starts waiting
for a check of a program in a document root. One worker resolves the running
image before reading and scanning it. All non-root starts enter this queue
because a launch path outside a document root can be an alias into one.
Starts refused by a full queue, starts left waiting at shutdown, and findings
suppressed during shutdown count as lost. The scheduled deep scan still reads
programs in its scope, and the periodic process check judges running programs.
bpf.connection.verdict reports the advisory annotation pool’s 256 waiting
slots and up to four running callbacks. Repeated findings for a pending
destination, reason and severity share one request and retain its original age.
Capacity refusal, callback failure and abandoned shutdown work count as lost
annotations, separately from lost findings. Callback errors release the pending
key and allow a later finding to retry; a successful retry preserves earlier
loss totals. Shutdown closes admission, waits for running callbacks and counts
the queued requests left behind. Cached answers remain available without new
work. Findings continue immediately when an annotation is unavailable.
processctx.enrichment reports 1,024 waiting requests and up to two running
process-context workers. It appears only after a BPF consumer initializes the
shared pool; reading health does not start workers. Overflow replaces the oldest
queued request and preserves the waiting age of the remaining requests.
Running time includes the process read, latency observer, account resolution and
cache write. Read errors and interrupted processing count as lost enrichment;
vanished processes and stale or unverified identities are completed filtering.
The queue dropped metric counts refused, evicted and abandoned requests;
queue health also includes failures after a worker receives the request.
The daemon stops the pool after BPF producers have joined, lets running reads
finish and counts buffered requests discarded at shutdown. Final loss evidence
remains available. This row measures enrichment requests, not each individual
deadline-based file read inside a request.
processctx.proc_reads reports the 64 shared slots for deadline-limited process
file and symlink reads, including process-start captures on the BPF path.
Waiting and running reads share this capacity. A slot remains occupied until
both the underlying read and its caller have finished; an undelivered result
still occupies its slot. A timeout counts the lost result immediately and keeps
a blocked syscall visible as running work until it returns. Timeout and a later
read failure count as one loss. Refusals at capacity and read failures count as
losses; missing process files are expected churn. Synchronous reads without a
deadline consume no slots. The initial process-directory check and the cached
boot-time read are synchronous and are not measured by this row.
smtp_rdns.resolves reports the 64 reverse-DNS lookup slots used by direct SMTP
egress detection. A slot remains occupied until its resolver and caller finish.
The one-second caller deadline also bounds healthy processing age; timing out
counts a lost result once while keeping a resolver still running visible.
Capacity refusals and resolver failures count as losses, while NXDOMAIN and
successful empty responses are normal negative results. Cache hits perform no
queued work. The bounded result cache is retained data, not waiting lookups.
Status can initialize the empty cache but never performs DNS or waits for its
cache lock. Synchronous lookups without a deadline use no slots or queue row.
email_password.mailboxes reports waiting mailboxes and the five concurrent
audits per scan, retaining each audit through finding collection and cache writes.
Concurrent scans have no fixed combined waiting capacity. A busy pool uses each
audit’s five-minute budget or shorter scan deadline; an idle slot with no dispatch
progress for one minute reports backlog lag. Admission and post-audit work have
their own one-minute budgets. Cancellation removes waiting demand without adding
losses, while actual audits remain visible until they return. Expired work, failed
verification, cache write failures and abnormal exits count once per mailbox.
Unsupported, malformed and over-budget hashes remain normal incomplete-audit
outcomes. Health retains loss counts after the batch drains and reads only memory.
Deadline losses are recorded where an audit actually stops unfinished. A deadline
arriving after a successful cache write does not turn that mailbox into a loss,
and context evaluation during drain never holds the health lock.
email_password.hashes reports the three shared password-verification slots.
Each slot remains occupied until its KDF and caller have both finished, including
after scan cancellation. email_password.waiting reports callers awaiting a
slot; it sets capacity_unavailable because concurrent scans have no fixed
global waiting limit. Both rows use the password audit’s five-minute check budget
for lag. The occupied pool also reports sustained full capacity. A deadline while
waiting counts one admission loss; a deadline after admission counts one lost
verification result. A later KDF error or abnormal exit cannot count it twice.
Explicit cancellation withdraws demand without loss; actual verification errors
still count. Successful matches and nonmatches are normal results. Rejected
input and already canceled callers start no queued work. These rows contain no
password, hash, mailbox or account data, and status never runs a verification.
The outer mailbox scan’s discovery, network enrichment and persistence are
separate work from these hash slots.
checks.executions reports dispatched check functions across host and account
scans. An execution remains present until both its function and caller finish,
including while its result awaits consumption or its function outlives a timeout.
Lag is evaluated against each call’s original deadline, including a shorter parent
deadline; a delayed function start does not reset it. Timeouts, panics and
abandoned results count once per execution. Explicit cancellation withdraws
demand without counting a loss, but a later function failure still counts.
This aggregate sets capacity_unavailable: separate scans have their own wrapper
limits, and timed-out functions can outlive those slots. It does not measure
checks awaiting dispatch, scan-job persistence or later automatic actions.
Status reads memory without waiting for a check or accessing the state database.
checks.plugin_inventory reports sites waiting for the five inventory workers
per refresh, retaining each site through command execution, result collection and
cache storage or failed-result cleanup. Concurrent refreshes have no fixed combined
capacity. Inventory uses a four-minute budget for its two bounded commands, or
the shorter check deadline; admission and result handling each have one minute.
A full pool stays healthy within those budgets. A free slot with no dispatch
progress for one minute reports backlog lag. Command, decoding and storage
failures count once per site, including when cleanup also fails. A command that
ran and returned output (on stdout or stderr) with an error, such as a tree
wp-cli will not inventory, answered the check and counts no loss. Missing command output still counts as
lost work, including when the command exited with an error. Cancellation
adds no losses; deadline withdrawal counts unfinished sites. Buffered sites stay
visible until the original workers stop consuming and the refresh joins them.
Actual commands remain in flight until they return. Health reads memory only and
retains loss evidence after recovery. Shared-refresh waiters remain owned by
checks.executions; they do not create another set of site jobs. Optional domain
lookup fallback and plugin metadata enrichment keep their existing behavior.
checks.wordpress_core reports installations waiting for the five checksum
workers per scan, retaining each through its command, integrity findings and
verified-file caching. The unit is installations; concurrent scans have no
fixed combined capacity. Command lag uses the two-minute command budget or a
shorter parent deadline. Result and cache work use one minute without progress.
A full finite batch stays healthy within those budgets; a free worker with no
dispatch progress for one minute reports backlog lag. Returned operational
failures and abandoned work count once per installation. A command killed by a
signal counts as failed work while retaining any partial integrity findings.
Recognized integrity results from a completed command, including deliberately
filtered output, complete without a queue loss. So does a command that ran and
refused the tree, such as a directory that is not a WordPress installation or
one whose configuration fails to load.
Missing command output still counts as lost work.
Cancellation withdraws unfinished demand without loss, while deadline withdrawal
counts unfinished installations. Commands ignoring cancellation remain in flight
until they return. A deadline during caching cannot undo completed verification,
and an ordinary cache read failure does not turn a verified site into lost work.
Health reads only metadata, including while result or cache locks are occupied,
and retains confirmed losses after recovery. Discovery is separate check work.
auto_block.waiting reports scan, direct-block, firewall-flush and startup
observation calls waiting for shared state, with no fixed waiting capacity.
auto_block.active reports the single state owner through firewall operations,
state writes and cleanup.
lag_basis: operation_progress times the current operation within a batch;
advancing batches do not degrade solely because their total duration exceeds a
minute. One minute without progress degrades the active row and any waiting
callers. A free state slot with no admission for one minute also reports lag.
Returned direct-block or flush errors, failed state reads and writes, and abnormal
exits count once per call; protected-address refusals do not. Known write failures
are recorded before readback and diagnostic output. Known errors remain visible
during later cleanup, and the common loss threshold and recovery policy apply.
These rows count state-lock callers, independently of persisted per-IP retry records.
Health snapshots use memory only and cannot wait for the state lock or I/O.
auto_block.pending reports durable retry records, including records loaded at
startup when automatic blocking is disabled. Its capacity is the retry admission
limit of 1,000 records. Records selected for a cycle remain in flight until their
state-file outcome is known; newly requeued records also remain in flight until
persistence finishes. auto_block.candidates reports distinct IPs admitted to the
current cycle, with no fixed capacity. Candidate and record counts describe
different stages and must not be added together. A candidate remains visible
through its firewall callback, subsequent bookkeeping and durable requeue.
Pending age uses its original queue timestamp, or first observation when that
timestamp is absent. An eligible record receives a timestamp on its first requeue;
records without check identity are withdrawn under the current block policy.
Refreshing the reason, check or severity does not reset the original queue age.
Normal quota waits stay healthy within the existing two-hour
retry lifetime; older waiting records report backlog_lag. Active record and
candidate work use one minute without operation progress, so advancing batches
can run longer without a false warning. These measurements add no retry scheduler
and do not change the hourly quota, expiry or overflow policy.
An unsuccessful attempt whose retry survives reports retry_failed without a
loss. Successful blocks, dry runs and expected refusals remain completed even
if later bookkeeping fails. Withdrawal under the current check policy or
login-blocking setting is an expected refusal and adds no loss. Persisted record
identities include check and severity, so a refused record cannot acknowledge a
different eligible retry after a failed write. Confirmed removal of expired,
invalid or overflowed records counts as pending loss; a fresh candidate that neither completes nor
survives on disk counts as candidate loss. Existing duplicate coalescence adds no
loss while a retry survives. The common loss threshold and recovery policy apply.
State read or write failures report state_io. Failed writes are read back
because an error after rename can leave the new state in place. If that read also
fails, depth_unavailable and dropped_lower_bound expose the uncertainty. A later
successful read restores measured depth; lifetime loss totals remain lower bounds
when earlier outcomes could not be established. New work with known outcomes
still contributes exact losses. Completed candidate history is released after
settlement, and health snapshots never read files or wait for firewall callbacks.
auto_block.cleanup reports distinct IPs awaiting bookkeeping cleanup after a
successful firewall flush. It observes saved cleanup retries at startup and admits
the flushed engine entries before reading the tracker. Cleanup also includes
tracked blocks, with duplicate IPs counted once. Work remains in flight through
store removal, threat-record cleanup and the final state-file outcome. This row
has no fixed capacity and is independent of the block-candidate stages.
Cleanup waiting age starts at first observation with lag_basis: deferred_checkpoint. Age alone does not degrade health: cleanup retries run on
the next explicit flush, without a background retry deadline. One minute without
active operation progress reports processing_lag. Failed cleanup whose retry
survives reports retry_failed; an old tracker block can retain a retry even
when saving its cleanup marker failed. Completed cleanup stays acknowledged
across failed saves, while a newly blocked IP creates fresh cleanup demand.
Recovery first establishes the known block baseline. Cleanup of a later block
has an exact outcome even when older cleanup history remains uncertain.
Cleanup state and snapshot failures report state_io. An unreadable tracker
sets depth_unavailable and dropped_lower_bound; speculative waiting records
are not reported as measured depth. The last bounded batch retains its identity
for later read recovery. A subsequent unreadable flush replaces that history
with its own batch. A readable state restores measured depth and counts confirmed
unfinished work that no longer has a retry source. An incomplete pre-flush engine
snapshot marks lifetime losses as a lower bound without inventing a missing count.
Health reads memory only. These measurements preserve the existing flush policy,
returned errors and cleanup retry behavior.
attackdb.events reports events awaiting persistence, including historical imports.
The buffer has no fixed capacity. Waiting age starts at admission, independently
of an event’s timestamp. A detached batch remains in flight through writes, close
and error logging; newly arriving events stay in the waiting count. Processing
age uses lag_basis: operation_progress and resets when a write returns, so a
progressing batch does not appear stalled solely because of its total duration.
Waiting age or absent write progress of one minute degrades health.
Completed JSONL records accepted by the file writer and successful database
writes count as persisted. Buffered data alone does not. Confirmed unwritten
events count as losses, including interrupted work left buffered or unencoded, and
returned encoding failures. These losses are visible before cleanup; close errors
or abnormal I/O exits preserve uncertainty
with dropped_lower_bound and report persistence_uncertain for one minute,
ahead of a backlog or a stalled write, as the record queue already does.
The common loss threshold and recovery policy apply to confirmed losses.
These measurements preserve the existing write, retention and shutdown policy;
they do not add retries or promise storage durability beyond the writer’s result.
Memory-only databases have no persistence backlog. Health reads queue memory,
independently of database state locks and I/O.
attackdb.records reports distinct IP records awaiting a saved update or deletion.
Repeated changes to one IP coalesce before the snapshot; a mutation during an
active write remains separate pending work. A successful deletion can also
satisfy a repeated deletion. Loaded records needing normalization and expired
records needing removal enter the same queue. Retained scoring records are not
counted as pending persistence. There is no fixed capacity.
Waiting age starts at the first pending change. The active snapshot stays in
flight through writes, error logging and retry bookkeeping. Processing age uses
lag_basis: operation_progress and resets at each returned store write or delete,
at the commit of their batch, or at the complete flat-file write result. The
store commits one flush as one batch, so a failed commit retries every record
and deletion in it. A refusal affects only that operation; a successful save
cannot clear a failed deletion for the same address, or vice versa. A returned
commit refreshes processing age even if every operation was refused.
One minute without progress or with waiting work degrades health. Returned
failures report retry_failed while the retry remains queued, preserving its
original age without counting it as lost.
Successful retries clear that condition. A failed shutdown flush retains the
in-memory retry; the stopped background saver does not schedule another attempt.
An interrupted operation counts confirmed abandoned demand only when no pending
update or deletion survives. Unreturned write outcomes report
persistence_uncertain for one minute and retain dropped_lower_bound afterward.
This includes a transaction interrupted before its callback starts or after all
its operations were refused.
Known losses remain counted through recovery. Health reads queue metadata only,
without waiting for database state or storage locks. These measurements preserve
the existing persistence, retry and shutdown behavior; they do not make retained
in-memory retry state durable across process exit.
block_digest.records reports the actual buffer of watched-country block
records, bounded to 5,000 with drop-oldest overflow. The buffer folds into the
daily summary at each configured interval, before each summary send and at
shutdown. Age starts when a record enters the buffer, independently of the
block timestamp. Normal batching and a full buffer alone do not report a stall:
waiting becomes overdue one minute after the configured interval. A detached
fold remains in flight; one minute without progress reports processing_lag.
The row uses lag_basis: operation_progress. Records arriving during a fold
belong to the next one. Expected per-IP coalescing is successful completion,
while an abandoned fold counts its discarded records as losses.
block_digest.email and block_digest.webhook report configured destinations
in notifications, with capacity_unavailable. Each eligible destination owns
one notification before delivery starts, including live alerts and the daily
summary, which is sent even when empty. Default delivery follows current alert
settings at admission, attempt and interrupted cleanup. Disabled destinations
add no new waiting work or loss; destinations enabled during an earlier send
are still handled. Explicit destination selection applies only to live alerts
and keeps its error when that alert channel is disabled. Daily summaries use
all enabled alert channels except phpanel.
One minute waiting or active reports lag. A returned sink error
counts one lost notification and stays in flight through error logging. A
daily summary sink, or the summary rendering, that panics fails the same way
and is logged with the panic, so it cannot stop the daemon. If an
operation exits without returning, its attempted delivery reports
delivery_uncertain for one minute and retains dropped_lower_bound; a remaining
destination that was never attempted counts one confirmed loss. Known losses
use the common warning threshold and survive recovery in lifetime totals.
Live delivery remains best effort. Daily summaries save their window before
delivery and retry failed destinations; successful destinations are saved
separately. A failed destination retries after one minute, doubling up to one
hour, and counts one lost notification for that summary however often it is
retried. Between attempts that destination keeps one waiting notification, so
its row reports backlog_lag while the outage lasts; switching that
destination off releases it within a minute. When the next summary is due, a
still unfinished one is dropped, logged once and counted once, and the new one
goes to every destination. A crash between remote acceptance and saving success
can repeat a delivery. Records retained after the collector stops report
consumer_stopped. Health reads metadata independently of collector state
locks, country lookups and delivery callbacks. The rows are always present:
every daemon runs the collector for the daily summary, and without
alerts.block_digest.enabled the records buffer stays empty. The summary and
customer block payloads are described under
Summary and customer block webhooks.
state.pending reports findings parked for the next startup, bounded to the
newest 10,000 findings. The store observes existing parked work when opened.
Depth is the last confirmed file contents; in-flight findings include incoming
appends and cleared batches still in replay. The bound applies to the stored
batch, not concurrent callers or an already detached replay. Overflow and new
findings that fail to reach storage count as losses. Duplicate occurrences
remain separate findings.
Waiting age starts at admission or startup observation, independently of the
finding timestamp. Surviving findings keep their age through later appends;
evicted findings no longer determine it. Waiting uses lag_basis: deferred_checkpoint: a parked or full batch awaiting restart does not itself
report a stall. state.pending_operations measures callers waiting for the
state lock and active persistence or replay, in operations with no fixed bound.
One minute waiting or without operation progress reports lag. Actual read,
write and clear results advance progress; replay stays active through dispatch.
Read, write and clear errors report state_io. Failed mutations are read back:
a returned error after replacement does not imply the new findings were lost.
Readback compares the recovered JSON payload, including internal subnet and
spray-target data and its normal repair of invalid UTF-8, so repaired log text
does not invent uncertainty or hide overflow.
Unreadable outcomes set depth_unavailable and dropped_lower_bound. A later
read restores measured depth, while lifetime losses remain lower bounds. Known
encoding failures and overflow are counted even when other outcomes are unknown.
Indistinguishable retained payloads keep the earlier age after a failed write.
An interrupted write or clear marks disk state unknown before releasing the
state lock. Interrupted replay retains measured disk depth but cannot claim an
exact loss for a partially dispatched batch; uncertainty reports
persistence_uncertain for one minute. Confirmed losses use the common warning
threshold and remain in lifetime totals after recovery.
Parked findings add optional storage-only keys to the public finding fields so
subnet response and credential spray correlation keep their inputs at restart.
Subnets are in the public cidrs field and also keep their storage-only copy,
so a daemon rolled back after a restart still replays them. Older parked
records still replay, and older daemons ignore the added keys. Process-local
delivery markers are cleared on read, so stored findings receive a fresh
evaluation.
Append order, returned errors and clear-before-replay policy are unchanged. These observations add no retry or extra replay. Health snapshots read metadata only and do not wait for state locks, files or dispatch callbacks.
incident.persist.waiting reports immutable incident snapshots waiting for the
ordered writer, with no fixed waiting capacity. incident.persist.active reports
the single occupied writer. A writer or free-slot admission stalled for one minute
reports lag. Failed writes and abnormal callback exits count once; abandoned bulk
snapshots count as waiting losses and release their ordering slots for later writes.
Callbacks remain owned through cleanup and error logging. Store failures retain
in-memory transitions and the existing warning log.
incident.persist.deferred reports coalesced bookkeeping waiting for a later
mutation or explicit flush, including shutdown. Its oldest age uses
lag_basis: deferred_checkpoint; age alone does not degrade health because there
is no periodic flush deadline. Full snapshots supersede earlier bookkeeping;
restoration and retention discard the affected markers. Memory-only correlators
have no persistence work. All three rows read memory independently of state locks
and database I/O. The common loss threshold and recovery policy apply to writes.
checks.file_index.waiting reports live scans waiting for the shared baseline
slot, with no fixed waiting capacity. Each wait keeps its shorter parent deadline
or the file-index check budget; a free slot with no admission for one minute also
reports backlog lag. checks.file_index.active reports the single occupied slot.
Walking and file analysis use the check’s original execution deadline (normally
15 minutes) or the shorter parent deadline. Setup, persistence and cleanup each
have a separate one-minute budget. A healthy long scan alone does not report a
full queue. Actual filesystem work remains visible after cancellation until it
returns. Deadline withdrawal, incomplete walks, unreadable PHP content, failed
executable metadata or state reads and writes, and abnormal exits count once per
live scan. Executable entries disappearing during enumeration and explicit
cancellation alone add no loss.
The common three-loss warning threshold applies, and recovery retains total losses.
A successful late baseline commit adds no loss. Findings and shrink protection
keep their existing behavior. Force-file-index audits bypass this stateful slot
and remain owned by the check execution row. Snapshots read memory only.
checks.reputation_queries publishes accepted lookups immediately after quota
reservation, including while refused lookups receive fallback scoring. It follows
the accepted work through HTTP,
response cleanup, buffered results, supplemental scoring and cache or quota-backoff
storage. Each scan has at most five requests; concurrent scans have no fixed
combined capacity. HTTP uses the client’s timeout, while cleanup, local result
handling and persistence have separate one-minute budgets. Supplemental scoring
uses the combined timeout of its enabled sources or the shorter parent deadline.
Buffered results follow their own active consumer’s deadline; another scan cannot
hide their delay. Query failures, failed storage and abnormal exits count once per
result. Quota responses and refusal before dispatch remain expected outcomes with
the existing quota health warning. Parent cancellation does not cancel the existing
HTTP requests; they remain visible until response handling and storage finish.
Successful late results add no loss. Health reads memory only and retains losses
after recovery. Local feed matches, cache hits and serial pre-query discovery do
not create query jobs. A failed cache write retains the findings already produced.
checks.dispatch reports pending checks and occupied runner wrappers across host
and account scan batches. Concurrent batches have no fixed global waiting limit,
so capacity is unavailable. Lag uses consumer_progress within each batch:
pending work degrades after one minute without progress when that batch has a
free worker slot. A busy pool may keep working within each check’s own deadline.
Setup and result handling have a separate one-minute budget; they cannot borrow
a heavy check’s longer deadline. A wrapper panic or abnormal exit counts a loss,
as does a parent deadline before a check can start. Explicit cancellation
withdraws queued demand without loss. Execution failures belong to
checks.executions; dispatch tracks the surrounding scheduling operation.
Reading status never takes a scan, context or database lock.
scans.jobs reports eight waiting full-scan jobs and one worker. Work remains
visible through enumeration, scanning, remediation, persistence and cleanup.
Its consumer_progress lag follows that worker and its own check batches:
healthy check deadlines allow long scans to proceed, while an overdue child
cannot be hidden by another child’s progress. Enumeration, result handling,
individual file actions and database operations each have a one-minute progress
budget. Thirty seconds at full waiting capacity also degrades health.
scans.admission separately reports callers waiting for admission and writing
their initial job record, with a one-minute lag budget and unavailable capacity.
Status reads only memory; operator polling does not advance the worker’s clock.
Queue refusal, failed job operations and abandoned work count once per job.
Configured finding-history truncation, explicit cancellation and successful
shutdown draining do not count as loss. A terminated worker refuses new jobs.
At startup, both queued and running records from the previous process become
errors with reason daemon_restarted; their lost requests count in job health.
email_av.scans reports antivirus engine work through execution and both result
handoffs. Timed-out engines remain in flight until they return; buffered results
remain queued until the message scan consumes them. Execution lag uses the
original per-part deadline, while result waiting and processing each have a
one-minute budget. Moving a result between buffers preserves its waiting age.
Concurrent messages and engines outliving timeouts have no fixed global limit,
so capacity is unavailable. Timeouts, errors and abandoned work count once per
engine scan; later failures cannot count the same scan twice. Detections and
unavailable engines are normal outcomes, and mail verdict behavior is unchanged.
Status reads only memory, and watcher restarts reuse the same engine health.
php_taint.requests follows callers waiting for the worker lock, active requests
and pipe operations that outlive their callers. Waiting, setup, reply handling
and cleanup each have a one-minute lag budget. Active worker communication uses
its configured timeout or the shorter caller deadline; a buffered reply cannot
borrow a long worker timeout. The request stays in flight through reply decoding
and cleanup, and until any outstanding pipe operation finishes. Capacity is
unavailable because callers have no fixed global waiting limit. Worker failures,
timeouts, breaker refusals and abnormal exits count once per request; known
failures are recorded before cleanup. Oversize input, caller cancellation and
refusal after an intentional stop do not add losses. Status reads memory without
the worker lock, and shutdown retains the supervisor’s health evidence.
Recovered analyzer panics also count as failed work when delivered in a valid
worker reply. The report and worker reuse policy remain unchanged.
yara.requests follows scans waiting for the single YARA scan worker, the
active scan and the rules load of a freshly started worker. Capacity is
unavailable because callers have no fixed global waiting limit. Worker
failures and breaker refusals count once per request.
central.actions reports 1,024 waiting central-intelligence actions and one
running action. Backlog remains visible while the signed feed refreshes;
processing time includes the action handler and its evidence delivery.
Overflow, action failure and abandoned shutdown work count as losses. A
protected-address refusal or absent firewall engine completes the queue task
without claiming that a block happened. Challenge tasks measure delivery to
the challenge list; they do not measure its later file or firewall writes.
Shutdown cancels feed refreshes, waits for an already-running action and
discards the remaining queue. Captured dispatch hooks refuse later work and
preserve its loss count after shutdown. Logging does not reset loss totals.
bot_verification.requests reports 256 waiting bot-identity requests and one
running verification. Duplicate requests share their original waiting age.
Running time includes DNS lookups and the cache write. Queue overflow, DNS
timeouts or transient failures, failed cache or missing-PTR record writes and
abandoned shutdown requests count as losses. Missing PTR records, unknown bot
identities and definitive positive or negative answers remain expected
outcomes. A source with a recorded missing PTR is not queued again until its
one-hour suppression lapses; suppressed requests do not count as losses.
For a day after the lookup, lapsed records prevent renewed pending grace
without blocking DNS retries.
Queue health does not turn a resolver failure into a spoof finding. Shutdown
cancels DNS, waits for any active cache write and discards waiting work. Late submissions
are refused, pending keys are released and loss totals remain available.
abuse_reporting.ingress measures reports awaiting durable storage, with a
capacity of 10,000 or the configured spool limit when smaller. Running time
includes persistence to every configured target. Memory overflow, incomplete
persistence and work abandoned by a worker failure count as losses; one source
report counts once even when several targets fail. Outbound delivery can delay
memory work, and that backlog remains visible. The queue stays open through
the daemon’s final finding flush. Reporter shutdown then closes admission and
persists every accepted report before closing the spool; a failed write does
not discard unrelated waiting reports. Captured hooks refuse later reports.
This row measures the memory queue, not reports already retained on disk for
retry; normal shutdown does not count those durable reports as lost.
abuse_reporting.spool measures durable reports per destination, with the
configured spool capacity. Reports already being sent remain in flight through
the database acknowledgment. Failed admission, discarded records and reports
the collector refuses as invalid count as losses; a delivery retry, failed
acknowledgment or normal shutdown retains the report and does not count it as
lost. A failing destination pauses only its own reports for the rest of a
pass. An evicted or refused record counts as lost only if no send was
acknowledged during this process. An active send settles that count when it
finishes; a failed retry cannot undo a prior acknowledgment.
lag_basis: observed_age means waiting age starts when this process first sees
the record. Existing records start at spool open; retries keep that age. Doctor
labels this as observed_lag, since time spent waiting before restart is unknown.
Waiting or running work degrades after two minutes, allowing for the normal
one-minute delivery interval. The common full-queue and recent-loss thresholds
also apply. spool_io names failed admission, read or acknowledgment operations;
delivery_failed names a failed or panicking sender and stays set while any
destination failed in the last pass or is waiting for its retry deadline.
Throttling without progress remains visible here even between log warnings.
These states clear when the affected operation recovers. Health reads remain
independent of database writes and outbound requests. Reopening with a smaller cap preserves existing
records until the next admission applies the configured overflow policy.
phpanel.spool measures the durable panel webhook queue, capped at 100,000
records per state directory. In-flight work includes writes waiting to commit
and deliveries awaiting database acknowledgment. Retried findings keep their
original waiting age; a failed attempt is not a lost finding while its record
remains durable. An overflowed record counts as lost only when no send was
acknowledged during this process; an active send settles that count when it
finishes. Malformed findings, and findings the panel refuses with HTTP 400,
413 or 422, count once when removed from delivery, and the next record is
sent at once. Their bounded diagnostic archive is retained history, not
pending work. A long run of refusals with nothing accepted between them
looks like a panel refusing every request, so from then on refused findings
stay queued and are retried as a delivery failure until the panel accepts
one. This hold survives config reloads and daemon restarts. Only a 2xx
response from the configured endpoint acknowledges a finding; redirects
are not followed. Every other failure is retried in order with backoff.
Waiting age uses the persisted enqueue timestamp after restart. Boundary
records absent from the accounting rebuilt at open are adopted with the same
timestamp, so retrying them does not reset their age. Missing, damaged or future
timestamps are timed from queue open or adoption. A minute of waiting
or processing degrades the row; delivery and database errors remain visible
until the affected operation succeeds. Health reads use memory only, so a
stalled database or collector cannot block status. Disabling delivery preserves
the durable backlog and cumulative loss; enabling it resumes the stored work.
Stopped queue instances refuse late admissions.
actionlog.writes measures the 64 process-wide action-log write slots. A sink
write that outlives the caller’s 250ms wait budget stays in flight until the
sink and any panic reporting finish; a batch of actions serialising behind one
file lock is normal, and only a minute without the write returning degrades the
row. A caller deadline after
admission does not count as loss, since the sink may still record the action.
Refused admission, sink errors and panics count as lost records, once per record.
Changing or disabling the sink preserves outstanding work and cumulative loss.
Health reads do not wait for the sink. Normal writes still complete before the
caller returns; saturated or stalled recording keeps the existing caller budget.
events.deliveries aggregates live event-stream subscribers in one row without
client identities. Capacity sums their buffers (64 findings per daemon stream);
depth counts waiting deliveries and in-flight work includes encoding, writing
and flushing. One full subscriber can degrade this row even while others drain.
The common one-minute lag, 30-second fullness and recent-loss thresholds apply.
Overflow, encoding errors and failed streams count as losses; a failed stream
also counts its abandoned buffered events. Normal request cancellation and
server shutdown withdraw pending demand without adding losses. Cumulative loss
survives subscriber removal, and an outstanding write remains visible until it
returns. Closing the bus preserves buffered work for consumers still draining it.
HTTP writes retain their three-second deadline; successful delivery here means
the write and flush returned, not that the remote application acknowledged it.
Each active BPF backend also exposes a .kernel row. depth_unit: bytes
labels ring occupancy and capacity. lag_basis: consumer_progress means
lag_seconds measures time without observed consumption while data remains,
starting at the first sample that sees pending data. It is not an event age.
One minute without progress degrades the row; recently observed reservation
failures use the same loss threshold as userspace queues. Kernel loss counters
are separate from decode failures and userspace admission loss.
After a reader unmaps its ring, depth_unavailable: true and
lag_basis: unavailable prevent zero fields from claiming an empty live ring.
The last counter sample adds submitted records the reader never consumed.
Submission accounting precedes publication, so a fast reader cannot consume
an event before it has been counted.
dropped_lower_bound: true marks this final shutdown total: kernel detachment
can leave callbacks finishing after the sample. Doctor prints dropped>=...
for that bound. A counter lookup failure or an incoherent occupancy reading
marks the measurement unavailable and degrades the row as
measurement_unavailable once it lasts half a minute, so one artefact of
reading a live ring raises nothing; a failed final sample degrades at once and
remains degraded. A stopped reader is reported instead of the measurement
artefacts it causes. Invalid occupancy cannot inherit fullness or consumer
stall alarms from an earlier sample; independently counted losses remain visible.
The required kernel suite fills the shipped connection program’s ring with
real non-root connect calls and checks reservation loss and retained output.
fanotify.kernel and spool.kernel report pending notification records and
use the same consumer-progress lag measurement. Their group capacity is not
exposed by the kernel: capacity_unavailable: true marks it as unknown, and
doctor prints depth=N/unknown records. The current system queue limit may
differ from the limit captured when the watcher was created.
Their loss totals are lower bounds: each overflow record proves at least one
loss, and shutdown adds the records known to be unread before closing the
descriptor. Events can still arrive between that sample and close.
An unavailable pending-record measurement degrades the row once it lasts half
a minute; the reading taken at close degrades at once and remains degraded.
Records the kernel has already dropped are reported ahead of an unreadable
depth. Closing a descriptor leaves known zero occupancy.
fanotify.reader and spool.reader track batches after a kernel read, until
all records have been dispatched or filtered. A stalled batch remains visible
even when the kernel queue is empty. Reader losses count batches interrupted
by consumer failure, separately from kernel-record losses.
Spool replacements retain loss and running-batch evidence, while the new
descriptor starts its own occupancy and progress measurements. Health reads,
event reads and permission responses cannot use a descriptor after close.
The required kernel suite verifies pending records, stalls, drain recovery
and unread shutdown loss using real fanotify events.
forwarder.kernel and phprelay.kernel report inotify backlog in bytes,
because records include variable-length filenames. Capacity is unknown and
lag follows observed consumer progress. Overflow markers each prove at least
one lost event. Closing with unread bytes adds one further known loss; the
byte count does not reveal the number of discarded events. These totals use
dropped_lower_bound: true.
forwarder.reader and phprelay.reader track the running read batch, including
synchronous callbacks. Failed callbacks count as interrupted batches. PHP
relay replacements retain earlier losses with fresh occupancy measurements.
Descriptor reads, watch changes, polling and close share one lifetime guard.
phprelay.index.persistence reports the message attribution writer’s 4,096
waiting slots. Writes remain running from channel receipt through the pending
batch and its database transaction. Batches contain at most 256 writes, also
during an explicit flush. Each flush covers the backlog present when the
writer accepts it; concurrent arrivals cannot extend that flush indefinitely.
Refused submissions, encoding failures and writes in a failed transaction count
as losses. Failed writes are settled before emitting their error finding, so a
blocked reporter cannot hide the loss. A failed transaction counts each of its
writes once and does not prevent subsequent batches from committing.
Shutdown closes admission and drains accepted writes; later submissions count
as refused work. The existing persistence dropped metric counts refused
submissions, while queue health also includes writes that fail after admission.
Some rows carry advisory. Their work is best effort: a client that stops
reading its event stream, an unreachable panel asked for an optional
annotation, and expired process context reads all lose detail around findings
that are still detected, stored and delivered. An advisory row reports its own
degradation with the same evidence, but leaves the host status and security
posture unchanged, warns instead of failing csm doctor, and raises no
notification.
php_shield.eval_sites is advisory grading work. It reports running ownership
walks and completed results still held for callers. A timeout or refused
lookup counts as a lost proof, while the finding stays High. A completed walk
releases its single admission slot before publishing, so later events do not
wait for that caller. Concurrent callers can hold completed results, making
the aggregate capacity unavailable. Health reads perform no filesystem lookup.
Each degraded row names its condition: backlog_lag for the oldest waiting
item past its budget, processing_lag for the oldest running item,
queue_full for capacity held continuously, dropped_work for recent losses,
consumer_stalled for a sampled queue whose consumer stopped advancing,
reader_stopped for a released kernel descriptor, measurement_unavailable
for a reading that stayed unavailable, and persistence_uncertain for a write
whose outcome is unknown. The owner-specific conditions retry_failed,
source_io, spool_io, state_io, delivery_failed, delivery_uncertain
and consumer_stopped are described with their rows above.
Doctor names each age by what it measures: observed_lag for an age taken at
observation, consumer_stall for consumer progress, operation_stall for the
current operation, deferred_age for work parked until a restart, and
lag=unavailable where no age exists.
A queue becomes degraded after three losses in a minute, thirty seconds
continuously full, or a minute waiting or processing. These are operational
alert budgets, not measured throughput guarantees. Health is computed directly
from the counters, independently of finding delivery. The daemon records and
dispatches protection_queue_degraded at most once per queue every five
minutes while pressure remains, then one protection_queue_recovered event.
The five-minute bound spans recoveries, so a queue that clears and degrades
again inside the window stays degraded in status without a second event, and
no recovery event follows a degradation the bound suppressed.
These are CSM health events and do not feed account-compromise correlation or
automatic response. Recovery preserves cumulative loss evidence; restarting a
spool watcher also preserves it. Restarting the daemon resets the counters.
Inspect worker errors and CPU, memory and I/O pressure when a queue degrades. Reduce competing bulk work and confirm the queue drains and recent losses stop. This surface currently covers finding delivery, file and spool kernel readers and scanners, recovery scans, staged package verification, dropper processing, BPF queues, process context enrichment and deadline reads, mail-log delivery, forwarder and PHP relay notification queues, and PHP relay index persistence; other bounded queues remain in the roadmap.
Summary and customer block webhooks
The generic alert webhook gets two events of CSM’s own: the
daily summary and the
known-customer block alert.
A phpanel webhook gets neither. Each is one JSON object: text holds the
email body, for chat tools, and csm holds the data.
{
"text": "[CSM] host.example.com - daily summary 2026-06-10\n...",
"csm": {
"event": "daily_summary",
"host": "host.example.com",
"version": "X.Y.Z",
"summary": {
"since": "2026-06-10T08:00:00Z",
"due": "2026-06-11T08:00:00Z",
"checks": {"wp_login_bruteforce": {"CRITICAL": 12}},
"blocks": 3,
"block_kinds": {"wp-bruteforce": 2, "netblock": 1},
"muted": 4,
"sources": [{"ip": "203.0.113.5", "count": 9, "country": "NL"}],
"watched": {"total": 0, "by_country": {}, "by_category": {}}
}
}
}
daily_summary fields:
csm.eventisdaily_summary;csm.hostandcsm.versionname the host and the CSM version.summary.sinceis the start of the window andsummary.dueits scheduled send time, both RFC 3339.summary.checkscounts summary findings per check, then per severity (CRITICAL,HIGH,WARNING).summary.blockscounts applied blocks;summary.block_kindscounts them by kind, such aswp-bruteforce,mail-bruteforce,netblock,permanentorother.summary.mutedcounts summary findings of checks inalerts.email.disabled_checks.summary.sourceslists the 10 addresses with the most summary findings:ip,count, andcountrywhen GeoIP knows it.summary.watchedcounts watched-country blocks:total,by_countryandby_category. It stays empty withoutalerts.block_digest.enabled.summary.customer_blockslists the applied blocks that hit known customers, as customer block objects (below), at most 50;summary.customer_blocks_morecounts further blocks. Both are left out when empty.summary.would_blocklists dry-run and observe-mode verdicts that would have blocked known customers, as customer block objects, at most 50;summary.would_block_morecounts further verdicts. Both are left out when empty.
Each count map holds at most 512 keys; later keys count under other.
{
"text": "[CSM] host.example.com - automatic block hit a known customer: 203.0.113.5\n...",
"csm": {
"event": "customer_block",
"host": "host.example.com",
"version": "X.Y.Z",
"customer_blocks": [{
"target": "203.0.113.5",
"reason": "SMTP brute force from 203.0.113.5",
"check": "smtp_bruteforce",
"customers": [{"ip": "203.0.113.5", "kind": "cpanel", "account": "alice", "at": "2026-06-10T08:15:00Z"}]
}]
}
}
customer_block fields:
csm.eventiscustomer_block;csm.hostandcsm.versionas above.csm.customer_blockslists the blocks of the alert. Each hastarget, the blocked address or network;reason, the block reason or, for a notice without one, its message;check, the check that caused the block, left out when unknown;customers; andmore, the covered customers not listed, left out when 0.- Each
customersentry is one known customer, the most recent first:ip(an IPv6 customer as its /64),kind(cpanel,webmail,smtp,imap,poporftp),account(left out when the login named none) andat, the last successful login in RFC 3339.
The watched-country live alert, event block_live, keeps its own shape:
there csm.blocks lists blocked addresses.
GeoIP
GET /api/v1/geoip IP geolocation (?ip=&detail=1)
POST /api/v1/geoip/batch Batch GeoIP lookup (body: {"ips":["192.0.2.1"]}, maximum 500)
Threat Intelligence
GET /api/v1/threat/stats Attack stats, type breakdown, hourly trend
GET /api/v1/threat/top-attackers Top attacking IPs with GeoIP (?limit=&spam_only=1)
GET /api/v1/threat/ip IP threat lookup (?ip=)
GET /api/v1/threat/events IP event history (?ip=&limit=)
GET /api/v1/threat/whitelist Whitelisted IPs
GET /api/v1/threat/db-stats Attack database statistics
POST /api/v1/threat/block-ip Block IP for 24 hours
POST /api/v1/threat/block-ip-permanent Block IP with no expiry
POST /api/v1/threat/whitelist-ip Permanent whitelist
POST /api/v1/threat/temp-whitelist-ip Temporary whitelist (with expiry)
POST /api/v1/threat/clear-ip Clear IP from attack database
POST /api/v1/threat/unwhitelist-ip Remove from whitelist
POST /api/v1/threat/bulk-action Bulk block (24h or permanent) / whitelist across many IPs
/api/v1/threat/stats includes mail_only_ips, the attack database records
that hold only spam evidence, mail_only_new_24h, those first seen in the
last 24 hours, and mail_evicted, those dropped to keep the
spam evidence budget since the daemon
started.
In /api/v1/threat/top-attackers, ip is an IPv6 /64 in CIDR form, such as
2001:db8:1:2::/64, for a record that holds the spam evidence of a network.
/api/v1/threat/ip takes an address only. Its attack_record.spam_network
names the /64 when the spam evidence shown was recorded for the network.
Equivalent IPv6 spellings use the same address record and return the
canonical address. The stronger spam evidence wins; ties use the network.
The 24 hour endpoint returns 409 Conflict when the address already has a
permanent or longer firewall block. Unblock it explicitly before shortening
its lifetime. Bulk actions use action: "block", "block_permanent", or
"whitelist"; refused blocks are excluded from count and explained in
warnings. Repeated canonical IPs count once. Permanence cannot be selected
through an extra request field.
Bulk undo restores the original per-IP firewall deadlines. A later change to any target invalidates that undo action; it cannot restore dismissed evidence or replace the later decision.
Firewall
GET /api/v1/firewall/status Config, blocked/allowed counts
GET /api/v1/firewall/allowed Whitelisted IPs
GET /api/v1/firewall/subnets Blocked subnets
GET /api/v1/firewall/audit Firewall audit log
GET /api/v1/firewall/check Check if IP is blocked (?ip=); admin or unban scope
POST /api/v1/block-ip Block an IP
POST /api/v1/unblock-ip Unblock an IP
POST /api/v1/unblock-bulk Bulk unblock IPs
POST /api/v1/firewall/allow-ip Allow an IP
POST /api/v1/firewall/remove-allow Remove IP from allow list
POST /api/v1/firewall/deny-subnet Block subnet
POST /api/v1/firewall/remove-subnet Remove subnet block
POST /api/v1/firewall/flush Clear all blocks
POST /api/v1/firewall/unban Unblock IP + flush cphulk; admin or unban scope
POST /api/v1/firewall/cphulk-clear Flush cphulk bans only
The audit log reports each timestamp as an RFC 3339 instant in UTC and
the block or allow lifetime as duration_seconds. Blocked addresses,
subnets and allow rules carry expires_at, left out for a permanent entry.
ModSecurity
GET /api/v1/modsec/stats WAF statistics (read scope). Accepts ?window=1h|6h|24h, ?severity=warning|high|critical.
GET /api/v1/modsec/blocks Blocked requests log, aggregated per IP, with resolved source country (read scope). Accepts ?window=1h|6h|24h, ?severity=warning|high|critical.
GET /api/v1/modsec/events WAF event details with resolved source country (read scope). Accepts ?window=1h|6h|24h, ?severity=warning|high|critical.
GET /api/v1/modsec/rules Loaded rules list
POST /api/v1/modsec/rules/apply Apply the set of disabled rules and reload
GET /api/v1/modsec/rules/escalation Rule IDs excluded from firewall escalation, sorted
POST /api/v1/modsec/rules/escalation Exclude one rule from escalation or turn it back on
POST /api/v1/modsec/rules/escalation takes {"rule_id": 900112, "escalate": false}. The rule ID must be a CSM rule (900000-900999). escalate: false adds the exclusion and escalate: true removes it; other excluded rules are left as they are.
A failed rules reload reports rolled_back: true only when restoring the previous
overrides succeeds. A rollback failure is reported in the response and audit log;
inspect the overrides before trying another reload.
Rules & Suppressions
GET /api/v1/rules/status YAML/YARA rule counts, version
GET /api/v1/rules/list Rule files
GET /api/v1/suppressions Suppression rules
POST /api/v1/rules/reload Reload signature rules from disk
POST /api/v1/suppressions Add a suppression rule
DELETE /api/v1/suppressions Delete a suppression rule by id
POST /api/v1/suppressions takes {"check", "path_pattern", "reason"}. The path pattern is a glob and must be valid. A rule that covers every path of a check, hiding all its findings and stopping their remediation, needs "all_paths": true and no path_pattern; an empty pattern without it returns 400. check must be a check name, not a pattern: letters, digits, _, ., : and -. A name that is neither a known check nor the check of a current finding is saved, and the response carries a warning saying the rule matches nothing yet.
GET /api/v1/email/stats Email scanning statistics
GET /api/v1/email/forwarders Mail forwarder inventory with destination providers and local-copy flags (read scope)
POST /api/v1/email/forwarders/delete Delete one destination of a forwarder on cPanel hosts through uapi, as the account that owns it; takes {"source", "destination"} as the inventory lists them; 404 when the inventory has no such pair, 500 when cPanel or command execution fails, 503 when unavailable (admin scope, CSRF)
GET /api/v1/email/deferrals Outbound deferral rollup by provider and sending IP with reason codes and last_seen, parsed from exim_mainlog; each provider has a sample error text; lines is the number of log lines read and window_start/window_end the times of the first and last of them; one read is reused for 60 s (read scope)
GET /api/v1/email/queue-composition Mail queue makeup: real vs null-sender bounce backscatter, frozen count, unreadable count, oldest age, top stuck recipients; 500 when exim could not list the queue, rather than an empty queue (read scope)
POST /api/v1/email/queue/flush-backscatter Request removal of frozen null-sender messages from the exim queue on cPanel hosts; returns the count of targeted messages no longer queued, reports incomplete verification as 500, or returns 503 when unavailable (admin scope, CSRF)
GET /api/v1/email/queue One page of the exim queue on cPanel hosts, oldest first, each message with why it is stuck; page (default 1), limit (default 25, max 100), filter=all|frozen|backscatter, q (a substring of the sender or of one of its first 50 undelivered recipients) (admin scope)
GET /api/v1/email/queue/{id} One queued message with all undelivered recipients (up to 1,000), why it is stuck and the last 20 lines of its msglog; 404 when it is not queued (admin scope)
POST /api/v1/email/queue/{id}/{action} retry, bounce (return to sender), delete or thaw one queued message; answers the outcome the queue shows afterwards (admin scope, CSRF)
GET /api/v1/email/held Forward copies held by the forward guard, each with its decoded subject and size (admin scope)
POST /api/v1/email/held/{id}/release Re-inject a held forward copy to its external recipient (admin scope, CSRF)
DELETE /api/v1/email/held/{id} Discard a held forward copy (admin scope, CSRF)
GET /api/v1/email/checks Email checks the Email page shows, each with its group kind ("" = never grouped) (read scope)
GET /api/v1/email/groups Server-grouped action rows (kind=compromised_account|spam_outbreak|auth_failure|queue_alert|malware) with from/to/limit (read scope)
GET /api/v1/email/overview Mail decision totals and 15-minute chart series for from/to (default the last 24 h), antispam mode, live queue and mail-related block counts, and Stalwart's filter verdicts: totals delivered, junk, filter_rejected, discarded and unmatched, series delivered, junk, filter_rejected and discarded, and filter_rejects_seen (read scope)
GET /api/v1/email/rejects Recent rejected mail (time, class, sender, recipient, source IP, detail), newest first, kept 7 days; class=content (spam and malware, default)|all|<class>[,<class>...], from/to (default the last 24 h), limit (default 50, max 200); totals has the permanent rejects of every class in the range (admin scope)
GET /api/v1/email/senders Messages each hosting account sent in from/to (default the last 24 h), busiest first, up to 100 rows, with total, unresolved and abuse_events, the outbound abuse events relay-abuse lists for the range, counted up to 5,000 like relay-abuse, with abuse_events_truncated true when that cap cut the count, and abuse_by_account, those events counted per account they name and per kind, before any list limit (admin scope)
GET /api/v1/email/relay-abuse Outbound PHP-mail abuse detections (spam outbreaks, high-volume scripts/accounts) with per-site script breakdown; from/to/limit; total counts up to 5,000 and total_truncated is true when that cap cut it, while truncated is also set when only limit cut the list (read scope)
GET /api/v1/email/rspamd rspamd status (up|down|off), antispam mode, version and rspamd's counters since it started (scanned, learned, actions, top 8 rules), read from its controller without a password and cached 60 s; available false with a reason when they cannot be read (read scope)
GET /api/v1/email/quarantine Quarantined email list, each with the size of its spooled body and its subject, decoded and cleaned as on held forward copies
GET /api/v1/email/av/status Email AV watcher status
GET /api/v1/email/quarantine/{id} One quarantined message
POST /api/v1/email/quarantine/{id}/release Release a quarantined message (admin scope, CSRF)
DELETE /api/v1/email/quarantine/{id} Delete a quarantined message (admin scope, CSRF)
For Stalwart, the overview’s totals.delivered, totals.junk,
totals.filter_rejected, totals.discarded and totals.unmatched count
its own filter verdicts in the selected range. unmatched counts messages
whose verdict CSM could not tie to a sending address. Chart points use
delivered, junk, filter_rejected and discarded, omitting these keys
when zero. totals.rejected and the chart’s rejected key count rspamd
decisions, not Stalwart filter rejects.
The top-level filter_rejects_seen is true when any retained bucket contains
a Stalwart filter rejection or discard, even outside the selected range.
False means no such event is retained, not that the filter rejected no mail.
The Email page shows a dash and “none logged yet” for rejected mail until
this flag is true. Older responses may omit the flag or verdict counts;
the page treats the flag as false and missing counts as zero.
The mail.forwarders.delete.v1 capability advertises the forwarder delete
endpoint. A 500 can follow a delete that partly ran, so read the inventory
again before retrying. A domain’s default address (*@domain) is changed in
cPanel.
The senders abuse count and relay abuse list apply the exact from and to
bounds, including fractional seconds, even when requests share cached
history. from is inclusive and to is exclusive. A cached history limit
cannot let findings outside those bounds hide matching findings.
Email groups report handled: true only for auth failures whose complete,
current history is answered by active blocks on every attacker address or
covering subnet. Missing addresses, truncated or stale history, successful
logins, and account sprays stay unhandled. Unreadable firewall state sets
blocks_unavailable: true and leaves every group unhandled.
The overview’s counting_since begins when mail counting starts and survives
restarts. Stops appear in gaps, ending when counting resumes even if loading
saved counts needs a retry. Gap bounds round outward to whole seconds so
short stops remain visible. uncounted_minutes rounds the total gap duration
to the nearest minute, with a minimum of one for any positive duration.
The overview reads queue depth on every request. Queue detail is refreshed
at most once per minute, even if the queue empties and refills. An empty queue
discards its previous detail; new mail has detail_unavailable: true until
the next listing. Failed listings can reuse detail for at most ten minutes.
The queue list and the overview share one queue listing, refreshed at most
once per minute and after any queue action. total counts the messages that
match filter and q; queued, frozen, backscatter (null-sender
messages) and oldest_seconds describe the whole queue. listed_at is when
that listing ran, so these counts can trail the overview’s live queue size. When no listing has succeeded in
the last ten minutes the list answers unavailable: true with no items and no
counts. A failed refresh cannot reuse an empty listing, and a positive queue
count invalidates any earlier empty result. The next list request then lists
the queue at once rather than waiting out the minute, once for each such
refill. A message exim cannot read is listed with unreadable holding
exim’s error and reason class unreadable; without its envelope it has no
sender, recipients, size or frozen state, and only delete runs on it. An
exim spool error without an immediately preceding message line makes the
listing unavailable, including a missing-file error. Truncated entries and
unrecognized output also fail instead of reporting an empty queue. The
composition counts messages without envelopes only in total and
unreadable, not in real or bounce. Backscatter flushing excludes every
unreadable message and validates the whole listing before and after
removal; a targeted message that becomes unreadable still counts as queued.
Quoted recipient local parts remain intact, including spaces. Each
message’s reason (class, text, raw) comes from its msglog tail; a
partial first line is discarded. Up to 100 reasons are cached for a minute.
The current page takes priority when older entries must be evicted. A
message thawed since it froze shows retry_wait until its next delivery
attempt.
Message reads and actions run exim as a systemd transient unit and never
inside the daemon’s sandbox: without systemd-run the message route and the
actions answer 503 and exim does not run. Each action first checks that the message is
still queued; a missing-file reply must name the requested message. When
it is not queued, the answer is 409 with outcome: "gone". Nothing
ran, unless the message left during the action. retry answers
attempt_started as soon as the delivery attempt has started. bounce and
delete answer bounced and removed when the message left the queue and
exim reported no error. When exim reported an error and the message left
anyway, as when exim -Mrm finds it locked by a delivery that then completes,
the answer is 409 with outcome: "gone". An unreadable message answers
removed despite an error exit only when exim confirms its removal and
the message is no longer queued. A failed or interrupted delete without
that confirmation answers gone when another caller takes the message.
thaw answers thawed when the queue shows the change, even if exim reported
an error. Otherwise the answer
is unchanged with an error. A thaw on a message that is not frozen answers unchanged
without running exim -Mt. Every attempt is written to the UI audit log as
email_queue_<action> with the message ID, the outcome and any error exim
reported.
Held forward release requires a recorded sender and recipient, each a single mailbox without commas, whitespace or a leading dash. An explicitly empty sender is allowed for bounces. Ambiguous or missing envelope metadata causes a release error and leaves the message held without invoking sendmail. Release runs sendmail as a systemd transient unit with the message on its stdin, never inside the daemon’s sandbox: without systemd-run the release fails and the message stays held. The unit uses KillMode=process, so the delivery exim forks before it exits is not stopped with the unit and the message leaves at once. A release or delete first claims the held copy, so of concurrent requests one acts and the others answer that the copy is not held; a failed release puts the copy back. A copy whose release or delete the daemon never finished, as when it stopped mid-release, is held again on the next 15-minute refresh once the claim is at least five minutes old; such a release may already have been delivered. Recovery never replaces another held copy in either Maildir folder and makes each claim visible only once.
Deferrals are grouped under the provider whose MX host deferred them (Google, Microsoft and Yahoo hosts), so a business domain those providers host counts under them. The recipient’s domain decides when the log line names no MX or an unknown one.
The rspamd status follows live health probes. Dismissing an outage finding
does not change it to up. Version display strings are bounded whether
they come from the controller or the installed program. A read of rspamd’s
controller finishes even when the request that started it goes away, so its
result is cached for the other requests; a failed read is logged once per
cache period.
Hardening
GET /api/v1/hardening Load last hardening audit report (admin scope)
POST /api/v1/hardening/run Run hardening audit and save report (admin scope, CSRF)
POST /api/v1/hardening/apply Apply automatic fixes for one section, then re-run the audit (admin scope, CSRF)
Scan Jobs
csm scan --full enqueues full-scan jobs that run inside the daemon and persist to the store. Jobs are report-only unless an account-scope request sets quarantine: true; server-wide jobs reject quarantine.
GET /api/v1/scan-jobs List full-scan jobs (read scope)
GET /api/v1/scan-jobs/{id} Job status and stored report (read scope)
GET /api/v1/scan-jobs/{id}/findings
Paginated findings for one job (?offset=&limit=, limit 500 by
default and at most 5000; truncated marks more pages) (read scope)
POST /api/v1/scan-jobs Enqueue a full-scan job (admin scope, CSRF)
POST /api/v1/scan-jobs/{id}/cancel Cancel a queued or running job (admin scope, CSRF)
Verified Bots
GET /api/v1/verified-bots Configured verified-crawler allowlist plus live verification state (admin scope)
POST /api/v1/verified-bots/apply Validate, apply, and reload an edited verified-bots list (admin scope, CSRF)
Actions
POST /api/v1/fix Apply fix for a finding
POST /api/v1/fix-bulk Bulk fix multiple findings
POST /api/v1/dismiss Dismiss one finding {key} or up to 500 {keys}, optional {contents}; returns undo_token if any were dismissed
POST /api/v1/scan-account On-demand account scan
POST /api/v1/verify-finding Re-check a single finding on demand (admin scope, CSRF)
POST /api/v1/quarantine-restore Restore quarantined file
POST /api/v1/quarantine/bulk-delete Bulk-delete quarantined files; returns count and the ids it could not delete
POST /api/v1/db-object-backup-restore Restore a dropped MySQL object from its db_object_backups record
POST /api/v1/test-alert Send test alert through all channels
POST /api/v1/import Import state bundle (suppressions, whitelist)
fix and fix-bulk act on the file the stored finding names. A request may
repeat that path in file_path, but a different path is refused, and a target
is never a remediation root itself (/home, /tmp, /var/tmp, /dev/shm)
or an account’s home directory.
verify-finding returns the verifier verdict in checked, resolved,
demote, and detail. When that verdict also changes the stored finding,
severity_change is demoted or restored. A demote verdict without
severity_change means no stored severity changed; for example, the finding
was already demoted or a scan replaced its snapshot while verification was
running. Callers must not report a state change from the verdict alone.
Settings
GET /api/v1/settings List editable config sections
GET /api/v1/settings/<section> Read a config section (secrets redacted)
POST /api/v1/settings/<section> Update a config section (safe fields reload, restart fields queue)
POST /api/v1/settings/restart Request a daemon restart. Returns 202 with `started_at_token`
for polling until the restarted daemon reports a new marker.
POST /api/v1/settings/firewall/tentative-apply Save firewall config, restart, and arm rollback timer
GET /api/v1/settings/firewall/rollback Read pending rollback state
POST /api/v1/settings/firewall/confirm Confirm tentative firewall changes
POST /api/v1/settings/firewall/revert Revert tentative firewall changes now
Sections map to top-level config keys: alerts, auto_response, challenge, reputation, performance, infra_ips, sentry, etc. Writes persist to csm.yaml, re-record the integrity hash, and hot-reload where possible; restart-required changes are queued for /api/v1/settings/restart. Invalid field values return 422 and do not touch disk.
Fields marked file_only in the schema are shown but refused on write with a 422: anything that names a command, an executable, a file path, a socket or an environment variable the daemon acts on as root can only be changed in csm.yaml. Changing reputation.upstream.url or reputation.rspamd.url also returns 422 unless the same request enters the effective credential for that address again, and is refused outright when the credential comes from an environment variable. A credential entered in the request must match the resulting configuration after conf.d merging; an overridden replacement cannot authorize sending the stored credential to another address. Firewall tentative apply is restart-class by design: it snapshots the previous config, writes the new one, restarts the daemon, and auto-reverts unless the operator confirms before the timer expires.
Operator preferences
Per-operator state (UI density, timestamp display, default auto-refresh,
saved filter views) is keyed server-side by SHA-256 of the auth token,
so preferences follow the operator across browsers and devices without
the daemon ever storing the raw credential. Capability flag:
webui.prefs.v1. These endpoints require admin scope because they read
or mutate operator-private UI state.
GET /api/v1/prefs/user Read this operator's UI preferences
PUT /api/v1/prefs/user Replace the prefs blob (CSRF on cookie sessions)
GET /api/v1/prefs/views List saved views; `?page=findings` filters by page
PUT /api/v1/prefs/views Upsert one view {page, name, params} (CSRF on cookie sessions)
DELETE /api/v1/prefs/views Delete one view {page, name} (CSRF on cookie sessions)
Response shape for GET /api/v1/prefs/user:
{
"density": "comfortable",
"timezone": "local",
"auto_refresh": "on",
"table_columns": { "findings-table": ["check","severity","when"] }
}
density is comfortable or compact. timezone is server, local,
or an IANA-shaped zone string (e.g. Europe/Bucharest). auto_refresh
is on or off. Server-side sanitisation drops any other value. Unset
prefs encode as empty strings; the UI applies comfortable, local, and
on defaults.
Response shape for GET /api/v1/prefs/views:
{
"items": [
{
"name": "Critical SSH",
"page": "findings",
"params": { "severity": "critical", "check": "smtp_bruteforce" },
"updated": "2026-05-25T21:47:35Z"
}
],
"total": 1
}
Saved views are operator-scoped and capped at 200 per operator. The saved
view collection is stored as one 64 KiB preference blob. page and
params keys must be simple identifiers: ASCII letters, digits,
underscore, hyphen, or dot, up to 64 bytes. Each view has at most 32
params, and param string values are capped at 256 bytes. name must be
1-80 bytes with no control characters. PUT and DELETE return
"ok": true on success.
Bulk-action undo
Finding dismissals, bulk threat block / whitelist and bulk firewall unblock responses return
an undo_token when the daemon queues an inverse operation server-side
for 30 seconds. The UI surfaces a banner with the same TTL; CLI callers
can act on the token through the endpoints below. Each successful undo
writes an undo_<original_action> audit entry. Capability flag:
webui.undo.v1. These endpoints require admin scope because they read
or mutate operator-private action state.
GET /api/v1/undo/pending Latest pending undo entry for this operator (empty object if none)
POST /api/v1/undo/run Consume an entry and dispatch its inverse {id}; empty id uses latest
Non-empty response shape for GET /api/v1/undo/pending:
{
"id": "188d1f2a6c8b0000",
"action": "threat_bulk_block",
"inverse": "threat_bulk_unblock",
"summary": "Blocked 2 IPs",
"recorded_at": "2026-05-26T00:07:09Z",
"expires_at": "2026-05-26T00:07:39Z"
}
POST /api/v1/undo/run returns {ok, action, inverse, count} on
success, or 410 Gone when the entry is missing, already consumed, or
past its 30-second TTL. Recognised inverse action keys are
threat_bulk_unblock, threat_bulk_block, threat_bulk_unwhitelist,
threat_bulk_whitelist, firewall_bulk_reblock, and finding_undismiss. Other bulk actions
(quarantine delete, generic fix) do not surface an undo token because
they have no clean inverse.
Dismissals accept at most 500 keys per request and deduplicate repeated keys.
contents maps a key to the content_sha256 its row showed: an empty string for
a finding without one, or a string of 64 lowercase hex characters. All other
values, including null, answer 400 before any findings are dismissed. A key
whose listed finding now carries different content is not dismissed: a single
key answers 409, and a bulk request dismisses the rest,
returns the refused keys in refused, and counts only the applied dismissals in
count. A request with every key refused returns no undo_token. A key
missing from contents is dismissed as before and covers the listed content.
Undo skips findings changed by a later dismissal, successful re-check or baseline
reset, and count reports only the dismissals it actually reversed. Newer scan
copies remain in the current findings list.
Finding fields
Every finding in /api/v1/findings, /api/v1/events, and the JSONL audit log carries optional correlation fields when CSM can attribute them:
| Field | Meaning |
|---|---|
tenant_id | Tenant attribution from the verdict callback or panel-side webhook reply |
domain | Domain associated with the event (e.g. PHP-relay scriptKey host, mailbox domain) |
mailbox | Mailbox attribution (e.g. mail brute-force target, PHP-relay envelope-from) |
cidrs | Offending subnets the finding names (e.g. a distributed crawl’s collapsed subnets, or a mail or SMTP password spray’s /24; sprays leave source_ip empty) |
relay_total | PHP-relay trigger count for the path that fired |
relay_breakdown | PHP-relay script samples that contributed to the alert, with script key, hit count, last seen time, and a bounded sample subject when available |
Fields are omitted when the daemon could not attribute them. Orchestrators should treat absence as “unknown,” not “global.”
JSONL and syslog audit events include cidrs for new findings and replayed
parked findings after upgrade. Existing audit records are not backfilled.
Cleanup fields
GET /api/v1/quarantine also powers the Cleanup page’s file-backup list. Entries include:
| Field | Meaning |
|---|---|
kind | quarantine or pre_clean |
live_state | original_missing, live_differs, original_not_file, archive_missing, archive_not_file, or unknown. Byte-identical restored entries are hidden. |
GET /api/v1/db-object-backups returns restored and restored_at when a captured MySQL trigger/event/procedure/function backup has already been replayed.
Incidents
GET /api/v1/incidents/groups is a read-scope rollup of active incidents by kind and source. It accepts status=active|all|open|contained|resolved|dismissed, kind, limit and offset, allowing a credential spray to render as one row per attacker rather than one row per target. total counts the groups and scanned_incidents the incidents they came from; truncated is true when the scan cap left incidents out.
GET /api/v1/incidents
Returns one page of incidents sorted by updated_at descending, with
items, total, offset, limit and status. limit defaults to 50
and is at most 500. status filters by open, contained, resolved,
dismissed, or active for open and contained; empty means all.
GET /api/v1/incidents/<id>
Returns one incident by id. 404 if not found.
POST /api/v1/incidents/<id>/status
Body:
{"status": "resolved", "details": "operator-marked"}
Status values: open, contained, resolved, dismissed. Closing an
incident (resolved/dismissed) means future findings for the same
correlation key start a fresh incident. Reopening an incident binds the
same key again. Incident JSON includes correlation_key when CSM has a
stored account, mailbox, domain, process, or remote-IP key.
MCP endpoint (read-only)
CSM can expose what it knows about a server to an LLM support agent through
the Model Context Protocol. The endpoint
is off by default, answers at /mcp on the Web UI listener with credentials
of its own, and is read-only by construction: every tool is answered by
dispatching a GET to one route from a fixed allowlist of API reads,
in-process, so no tool can block, unblock, fix, scan, quarantine or change
settings. It is meant for agents that verify support tickets about the
server (“is my IP blocked”, “why does this account have findings”, “is mail
stuck”).
Enable it
webui:
enabled: true # the endpoint is served by the Web UI listener
mcp:
enabled: true
tokens:
- name: "support-agent"
token: "<64 random hex characters>"
issued_at: "2026-11-02T10:00:00Z"
allowed_ips: ["198.51.100.0/24"]
Restart the daemon (systemctl restart csm); the endpoint is wired at
startup. csm validate checks the block: it refuses mcp.enabled without
webui.enabled, a configuration without a usable token, and a token that is
also a Web UI or metrics token, and warns about short tokens.
The endpoint shares the Web UI’s TLS certificate, port and firewall
exposure. Limit who reaches it with the token’s allowed_ips.
Tokens are bearer credentials with the same dates and source restriction as
webui.tokens, without a scope: there is only one, read-only. A Web UI token
or browser session never opens /mcp, and an MCP token opens nothing else.
A known credential sent to the wrong endpoint, or used outside its dates
or addresses, is refused without spending the guess budget below.
Unknown tokens draw on the same budget as Web UI login and API token
guesses: five failed attempts per minute per client (IPv4 address or IPv6
/64), after which the client is refused for a minute. /mcp also counts
toward the API request rate limit. Up to four tool calls run at once and
each request, a whole batch included, has 30 seconds. A read still running
at the deadline keeps its slot until it finishes.
Transport
The MCP Streamable HTTP transport, JSON-RPC 2.0, stateless:
curl -sk https://server.example.com:9443/mcp \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"csm_lookup_ip","arguments":{"ip":"203.0.113.9"}}}'
POST /mcponly.GETanswers 405 (no server-initiated stream); there is no session id to keep.- Protocol versions
2025-06-18,2025-03-26and2024-11-05;initializeanswers the client’s version when it is one of those. - Methods:
initialize,ping,tools/list,tools/call,notifications/initialized(202, no body).resources/listandprompts/listanswer empty lists. Batches of up to 20 requests are answered as a batch. - A request body is limited to 1 MiB. A tool answer larger than 4 MiB is
refused with a message asking for
limit,offsetor a filter. - A browser
Originheader must pass the same check as on the API: the configured Web UI origin, an entry inwebui.allowed_origins, or a loopback origin that matches the request host.
A client configuration for an agent runtime that takes a URL and a header:
{
"mcpServers": {
"csm-web01": {
"url": "https://web01.example.com:9443/mcp",
"headers": { "Authorization": "Bearer <token>" }
}
}
}
Tools
Every tool carries readOnlyHint: true. A tool answers the route’s JSON as
text content; the route’s own error (400, 404, 503) is returned as a
tool error with the API message, so the agent can read it. Lists keep the
API shape {items, total, truncated}; a tool takes only the arguments its
route reads.
Composite tools:
| Tool | Reads |
|---|---|
csm_overview | hostname, platform, version, status, health, capabilities |
csm_lookup_ip | firewall/check, the blocked-ips entry, threat/ip, geoip for one address |
csm_config_summary | non-secret configuration facts from an explicit field list: mode, platform, Web UI listen address, enabled protections, auto-response limits, brute-force thresholds. Token values, keys, DSNs and URLs are never read |
Route tools, one per API read (the description names the route):
| Area | Tools |
|---|---|
| Daemon | csm_status, csm_health, csm_capabilities, csm_components, csm_challenge_stats, csm_scan_jobs, csm_scan_job |
| Findings | csm_findings, csm_findings_enriched, csm_finding_detail, csm_history, csm_stats, csm_stats_trend, csm_stats_timeline |
| Response | csm_blocked_ips, csm_quarantine, csm_db_object_backups, csm_audit |
| Accounts and incidents | csm_accounts, csm_account, csm_incident_timeline, csm_incidents, csm_incident_groups, csm_incident |
csm_email_stats, csm_email_quarantine, csm_email_av_status, csm_email_checks, csm_email_groups, csm_email_overview, csm_email_rejects, csm_email_senders, csm_email_rspamd, csm_email_relay_abuse, csm_email_forwarders, csm_email_deferrals, csm_email_queue_composition, csm_email_queue, csm_email_held | |
| Host | csm_performance, csm_hardening |
| Threat and rules | csm_threat_stats, csm_threat_top_attackers, csm_threat_ip, csm_threat_events, csm_threat_db_stats, csm_threat_whitelist, csm_rules_status, csm_rules_list |
| WAF and firewall | csm_modsec_stats, csm_modsec_blocks, csm_modsec_events, csm_modsec_rules, csm_firewall_status, csm_firewall_allowed, csm_firewall_audit, csm_firewall_subnets, csm_firewall_check, csm_firewall_rollback, csm_geoip |
Not reachable, on purpose: the findings event stream, CSV and state exports,
settings, sessions, preferences, undo, the verified-bots editor, every
POST, PUT, PATCH and DELETE route, and the reads that return content
rather than metadata: quarantined file previews, queued message headers and
the SQL of backed-up database objects. The allowlist is tested against
the public route registrations, so a route added to it must be a documented
GET read.
Some reads touch the live host rather than CSM’s store: csm_email_stats
and csm_email_queue run the MTA queue tools, csm_firewall_check asks
cPHulk on cPanel, csm_accounts walks account directories and
csm_performance samples /proc.
Logging
Each tool call writes one daemon log line, mcp tool call, with the token
name, tool, peer, HTTP status of the underlying route, outcome and duration,
and the arguments cut to one line. Authentication failures log
mcp: refused token with the credential name, when known, and peer. Reads
are not written to the Web UI audit log, which records actions only.
The capability mcp.readonly.v1 in GET /api/v1/capabilities advertises
that the build carries the endpoint; whether it is enabled is a matter of
configuration.
Metrics (Prometheus)
CSM exposes a /metrics endpoint on its HTTPS web UI port
(default 9443). The endpoint serves the Prometheus text exposition
format (Content-Type: text/plain; version=0.0.4) and is safe to
scrape every 15 seconds.
“Available metrics” below is the shipped set. New call sites are
instrumented in ongoing releases; check CHANGELOG.md under
## [Unreleased] for the latest additions.
Enabling
Metrics are on whenever webui.enabled: true is set in csm.yaml.
The endpoint has its own auth knob:
webui:
enabled: true
auth_token: "<UI login token>"
auth_token_issued_at: "<UTC RFC3339 time when the UI token was generated>"
metrics_token: "<long random string for Prometheus scraper>"
metrics_token_issued_at: "<UTC RFC3339 time when the scrape token was generated>"
metrics_token_expires_at: "<optional UTC RFC3339 expiry; leave empty for none>"
metrics_token is optional. When set, a Bearer header containing
this exact value grants access to /metrics. The UI auth_token or a valid
UI session cookie is also accepted so the dashboard can self-scrape,
but keeping the two tokens separate is recommended: rotating
auth_token does not then break Prometheus scraping, and giving
your monitoring stack the scrape token does not also give it UI
access.
Each credential expires at its expires_at setting and never expires
without one. Rotate the metrics token and its dates together, update the
scraper’s secret, then send SIGHUP to CSM. A malformed or future issue time,
or a malformed or past expiry, refuses the metrics credential. See
credential rotation for recovery instructions.
Prometheus scrape config
scrape_configs:
- job_name: csm
scheme: https
tls_config:
# CSM serves a self-signed cert by default; either skip
# verification here or pin the CA you chose.
insecure_skip_verify: true
authorization:
type: Bearer
credentials: "<metrics_token from csm.yaml>"
static_configs:
- targets:
- csm-host-1.example.internal:9443
- csm-host-2.example.internal:9443
A complete, validated version of this snippet (with global: block)
ships as docs/src/examples/prometheus-scrape.yml. The CI pipeline
runs promtool check config against that file in the promtool-check
job; if the example ever stops validating, the pipeline fails.
Quick check
curl -sk -H "Authorization: Bearer $METRICS_TOKEN" \
https://localhost:9443/metrics | head
Available metrics
Build / process
csm_build_info{version}(gauge, always 1): build metadata. Scrape once to discover the running version. Join on it in queries viagroup_left(version).
Go runtime
Always exposed. Use these to watch daemon memory over time and tell live
growth from GC headroom (rising heap_alloc = real growth; large
heap_idle = retained-but-freed memory).
go_memstats_heap_alloc_bytes(gauge): heap bytes allocated and still in use (live).go_memstats_heap_inuse_bytes/go_memstats_heap_idle_bytes/go_memstats_heap_released_bytes(gauges): in-use spans, idle spans, and bytes returned to the OS.go_memstats_heap_sys_bytes/go_memstats_sys_bytes(gauges): heap and total bytes obtained from the OS.go_memstats_next_gc_bytes(gauge): heap-size target for the next GC.go_memstats_gc_cpu_fraction(gauge): fraction of CPU time spent in GC since start.go_goroutines(gauge): live goroutine count.
For a deeper leak hunt, enable the loopback pprof endpoint (debug.pprof_listen)
and run go tool pprof http://127.0.0.1:<port>/debug/pprof/heap over an SSH tunnel.
The same endpoint serves profile (CPU, 30s by default), goroutine, mutex
and block. Contention sampling for the last two starts with the listener and
stops when the last listener exits, including on a listener failure. Failed or
refused binds do not enable sampling.
While enabled, mutex profiling samples one in 100 contention events. Block
profiling samples waits of at least 10 microseconds and a proportion of shorter
waits. Timing and collecting these samples adds CPU overhead even when nobody
fetches a profile, so use this endpoint for diagnostics and leave
debug.pprof_listen empty when it is not needed. Profiles accumulate for the
process lifetime; disabling sampling does not erase previously recorded data.
For a CPU investigation, collect the CPU profile first and the goroutine dump alongside it:
curl -o cpu.pb.gz 'http://127.0.0.1:<port>/debug/pprof/profile?seconds=60'
curl -o goroutine.txt 'http://127.0.0.1:<port>/debug/pprof/goroutine?debug=2'
YARA-X worker (default-on; off only if signatures.yara_worker_enabled: false)
csm_yara_worker_restarts_total(counter): cumulative number of counted YARA scan worker failures (failed start, timeout, crash, out of memory or protocol), including launches that never create a worker. Rule reloads and lifetime retirement do not count. Alert on sustained growth: a steady climb means scans keep failing and are reported as not scanned.
Findings
csm_findings_total{severity}(counter): every finding CSM records is counted here. Severities areCRITICAL,HIGH, andWARNING(matching thealert.Severityenum). Userate(...)for arrival velocity; watch for sudden CRITICAL spikes.
Alert delivery
-
csm_alert_dispatch_failures_total(counter): alert channel sends that failed after CSM detected findings. Counts email, webhook, and phpanel delivery failures. Sustained growth means findings are not reaching operators; check SMTP, webhook reachability, and credentials. -
csm_audit_sink_degraded{sink}(gauge): one when an enabled JSONL or syslog destination failed to open or write, zero when healthy or disabled. Values appear after the audit pipeline first runs and remain degraded during retry backoff. Syslog reports local write results, not receiver acknowledgement. -
csm_audit_events_dropped_total{sink}(counter): events whose destination was unavailable or whose write failed. Usecsm export --since <when>for backfill; failed live deliveries are not replayed automatically.
State
csm_store_size_bytes(gauge): on-disk size of the bbolt state database (/var/lib/csm/state/csm.dbby default). Retention sweeps bound logical growth. Startup compaction reclaims freelisted pages automatically when the file is large and mostly slack;csm store compactdoes the same immediately with the daemon stopped. The retention sweep logs a restart hint only under that same condition. It includes deleted pages still held by active readers, which become reclaimable when the daemon stops. Later writes can change whether compaction is due at startup. History and attack-event writes pack pages densely when their timestamps arrive mostly in order. A write uses balanced page splits when events older than the newest stored one are most of its records or at least half its inserted data. Space savings depend on record sizes and arrival order. Archive imports keep the snapshot’s page layout; compaction repacks an existing file.
Fanotify realtime monitor
csm_fanotify_queue_depth(gauge): current number of queued events waiting for the analyzer pool. The queue capacity is 4000; sustained values near that cap mean drops are imminent. Alert target:max_over_time(csm_fanotify_queue_depth[5m]) > 3500.csm_fanotify_events_total(counter): delivered fanotify file events whose admission decision has completed. Filesystem marks receive writes anywhere on their superblocks, including other bind mounts. Kernel queue overflow notifications are counted separately.csm_fanotify_events_admitted_total(counter): events queued for analysis, including dropper tracking. Subtract this counter andcsm_fanotify_events_dropped_totalfromcsm_fanotify_events_totalto count events rejected before queue admission. This includes path-resolution failures as well as filter rejections. Use rates over the same interval to compare these independently scraped counters.csm_fanotify_events_dropped_total(counter): cumulative events dropped because the analyzer queue was full. The reconcile pass still rescans drop-affected directories 60 s later, so dropped events do not disappear from detection – they arrive delayed. Alert target:rate(csm_fanotify_events_dropped_total[5m]) > 0paired with a short for-clause.csm_fanotify_placement_unverified_total(counter): files and directories moved, linked or created into watched trees that did not open as the file the placement monitor reported, so realtime did not check them there. A file replaced again right after it was placed is counted here. Sustained growth means placements are being raced; the deep scan covers those files.csm_fanotify_placement_skipped_total(counter): placed files not analyzed again because the same version was analyzed recently under the same conditions, such as an atomic save renamed to its final name.csm_fanotify_kernel_queue_overflow_total(counter): kernel-side fanotify queue overflows (FAN_Q_OVERFLOW). Unlike analyzer-queue drops, the kernel gives no file descriptor for lost events, so only directories with concurrent analyzer drops can be reconciled; the rest are covered by the next scheduled deep scan. Any nonzero rate during a write storm means realtime coverage had a gap.csm_spool_fanotify_queue_overflow_total(counter): kernel queue overflows on the mail spool watcher. In permission mode an overflowed event means the message was delivered without being scanned; a Warning finding is raised when this happens.csm_fanotify_reconcile_latency_seconds(histogram): how long the post-overflow reconcile pass takes to walk drop-affected directories and rescan recent files. Buckets: 0.01 s .. 60 s. Watch p95: reconcile stealing tens of seconds means bulk events are piling up faster than the walker can keep up.csm_checks_domlog_discovery_dropped_total{reason}(counter): per-vhost access-log paths the WP brute-force domlog discovery helper dropped before scanning. Labels:reasonisevalsymlinks_error(broken symlink, attacker-removed log file) orstat_error(file vanished between glob and stat, permission regression on the log directory). Steady growth means a real chunk of vhosts is being silently skipped each cycle. Stale-mtime drops are intentional filtering and are NOT counted here.csm_realtime_content_scan_truncated_total{check}(counter): cumulative real-time content checks where the underlying file was larger than the main read window, so the full-rule pass saw only the leading window. The read cap protects RE2 cost on huge files; sustained growth on a label means full-rule coverage is capped on large files. Labels currently emitted:phpcontent_inline(known webshell filename),phpcontent_uploads(PHP in uploads),php_check(generic PHP content scan),crontab(per-user /var/spool/cron write),htaccess(per-vhost .htaccess write),user_ini(per-vhost .user.ini write),html_phishing(HTML phishing heuristic), andcgi_backdoor(CGI backdoor heuristic). Compare against finding history forwebshell_content_realtime(or the matching check name for non-PHP labels) to judge whether a raised cap would surface real findings.
Mail scanning and parse workers
csm_spool_hold_expired_total{stage}(counter): mail opens answered underemail_av.fail_modebecause the hold budget ran out.stagerecords where the message was when its hold expired:queued(no scan slot yet; these count toward the hold bypass),waiting(behind another message from the same sender) orscanning(a slow scan; the scan continues and can still quarantine). Sustainedqueuedgrowth means scan capacity is short.csm_spool_source_waits_total{outcome}(counter): mail opens whose sender was already using its share of the scan slots.outcomeiswaited(the message waited its turn without taking a scan slot) oroverflow(too many messages from that sender were already waiting, so it was scanned at once without waiting its turn). Overflow scans can use every scan slot.overflowgrowth raises anemail_av_source_backlogfinding naming the sender, at most once a minute.csm_parse_worker_jobs_total{pool,outcome}(counter): parse worker jobs.poolismail(email attachments) orzip(tenant zips checked for phishing kits).outcomeisok,timeout,oom,crash,protocol(reply rejected),spawn(the unit did not start),refused(the worker is failing repeatedly) orskipped(the message killed workers before, so it was not unpacked again). Anything butokmeans a message or zip was not fully checked. Canceled requests, stopped workers and local descriptor exhaustion do not increment this counter.csm_parse_worker_starts_total{pool}(counter): worker units started. A worker serves many jobs; a start rate close to the job rate means workers keep dying or being retired.csm_parse_worker_memory_bytes{pool}(histogram): worker memory and swap in use after a successful job. Buckets: 32 MiB .. 1 GiB. Compare the high buckets withemail_av.worker_memory_limit. Unavailable memory readings are omitted; earlier readings are not reused.
Periodic check runner
-
csm_checks_crontab_base64_truncated_total(counter): crontab base64 candidates that exceeded the per-blob decode cap before decoded-content pattern matching ran. Sustained growth means encoded cron content is larger than the scanner currently inspects; review affected crontabs and tune the scanner before redeploying. -
csm_check_duration_seconds{name,tier}(histogram): wall-clock time each check takes to complete. Labelnameis one of the periodic check-runner names (fake_kernel_threads,webshells, …); labeltieriscritical,deep, orall. Buckets: 0.01 s .. 900 s. Most checks keep the 300 s timeout ceiling; heavy filesystem checks can run up to 900 s. Useful aggregations:# p95 of the slowest check in the critical tier: histogram_quantile(0.95, sum by (le, name) ( rate(csm_check_duration_seconds_bucket{tier="critical"}[10m]) ) ) # total time each cycle spends in deep-tier checks: sum by (tier) (rate(csm_check_duration_seconds_sum{tier="deep"}[1h]))
Threat intelligence
Registered when reputation.upstream.enabled: true.
csm_threatintel_cache_hits_total(counter): upstream threat-intel lookups served from CSM’s local per-IP cache.csm_threatintel_cache_misses_total(counter): upstream threat-intel lookups not served from the local cache. A miss may still fail open without an HTTP request when the breaker is open.csm_threatintel_backend_failures_total(counter): upstream backend failures from network errors, non-200 responses, malformed JSON, response IP mismatches, or out-of-range scores.csm_threatintel_breaker_open(gauge): 1 while the upstream circuit breaker is refusing calls, 0 when closed or allowing its single cooldown probe.
AI-crawler IP ranges
csm_botranges_refresh_total{result}(counter): AI-crawler IP-range refresh attempts, labelledsuccess(at least one vendor feed updated) orfailure. Driven by the daemon’s auto-updater and bycsm update-bot-ranges.csm_botranges_prefixes{bot}(gauge): current number of published IP prefixes per crawler identity in the active overlay.csm_botranges_last_success_timestamp_seconds(gauge): Unix time of the last successful refresh.
Firewall
csm_blocked_ips_total(gauge): number of IPs currently on the firewall block list. Excludes expired temp bans – the store’sLoadFirewallStatefilters those before the gauge reads.csm_firewall_rules_total(gauge): total firewall rules across all four categories (blocked IPs, allowed IPs, blocked subnets, port-specific allows). Excludes expired temp blocks and allow-list rows. Sudden drops are worth investigating; expected drops happen when temporary block or allow deadlines pass.
Config reloads
csm_config_reloads_total{result}(counter): SIGHUP reload attempts, by outcome. Labels:resultis one of:success– safe fields swapped in place, integrity hash re-recorded, live config updated.restart_required– one or more fields that need a full restart changed; live settings unchanged.error– YAML parse, validation, re-hash, or subsystem apply failure; live settings unchanged. If signing succeeded before an apply failure, the live integrity hashes follow the signed config.noop– file edit produced no semantic change (identical values, whitespace edit, etc.). Alert target:rate(csm_config_reloads_total{result="error"}[5m]) > 0paired with a short for-clause.
Auto-response
csm_auto_response_actions_total{action}(counter): every auto-response action fired, by class. Labels:actioniskill,quarantine, orblock. Incremented once per finding the correspondingAuto*helper produces, so a batch blocking four IPs in one cycle adds 4 toaction=block. Useful for detecting response storms. Verified admission blocks also increment this counter, from every source admission runs (scan, challenge, central and incident), once per engine result, without waiting for alert delivery:rate(csm_auto_response_actions_total[5m]).csm_admission_handoff_seconds(histogram): how long a response funnel waits to hand an automatic response to the admission ledger, which applies, previews or forecasts it. Preparing evidence for a possible later response adds no sample; selected refusals and responses each add one. Buckets: 0.001 s .. 1 s. The share of handoffs at or under 10 ms iscsm_admission_handoff_seconds_bucket{le="0.01"} / csm_admission_handoff_seconds_count; it should stay at or above 0.99.
Challenge
csm_challenge_routed_total{check}(counter): IPs routed to the proof-of-work challenge, labelled by the sourcecheckthat flagged them (e.g.http_scanner_profile,wp_login_bruteforce,ip_reputation). Graph per-detector challenge volume withsum by (check) (rate(csm_challenge_routed_total[5m])).csm_challenge_escalated_total{outcome}(counter): challenge entries that timed out and were escalated to a hard block, by firewalloutcome:live(a new block landed),noop(the IP was already blocked), plusdry_run,allowed, andallowlisted.outcome=liveis how many challenges actually became blocks, including those applied by admission after the timeout handoff.csm_challenge_passed_total{method}(counter): challenged addresses that got through, bymethod:pow(solved the puzzle),captcha,verify_cookie(passed earlier and came back),admin_cookie, andcrawler(verified search crawler).
Firewall blocks
csm_firewall_block_outcome_total{outcome,source}(counter): every IP block attempt, labelled by firewalloutcome(live,dry_run,allowed,allowlisted,noop,protectedfor refused infrastructure IPs,errorfor engine failures or an unavailable engine) and bysource(scan,challenge,incident,central_intel,cli,web_ui, orunknownfor an invalid internal source). All combinations are exported at zero from startup, and unexpected outcome or source values collapse into the fixederrororunknownbuckets instead of creating new labels. Admission execution reports into these same counters. Failed or unknown batch outcomes count as errors; forecasts and queued deferrals add no attempts. Retrying an outcome write or audit delivery adds no count. This is the aliveness signal for auto-response: alert whensum(rate(csm_firewall_block_outcome_total{outcome="live",source!~"cli|web_ui"}[1h])) == 0while attack findings keep coming, and watchoutcome="error"for a dying engine.
Retention (when retention.enabled: true)
csm_retention_sweeps_total(counter): number of retention sweep cycles completed since daemon start. A flatline after a restart means the sweep goroutine is not scheduling; a healthy daemon increments this on everysweep_intervaltick.csm_retention_deleted_total(counter): cumulative entries deleted across thehistory,attacks:events, andreputationbuckets. Spikes on the first sweep after enabling retention (initial backlog), then settles to the steady-state churn. Useful for estimating when the file might benefit from a restart or acsm store compactmaintenance window.
PHP-relay (email abuse, cPanel only)
All series are prefixed csm_php_relay_. Registered when email_protection.php_relay.enabled: true and the host is cPanel; otherwise zero across the board. See Real-time detection.
csm_php_relay_findings_total{path}(counter): findings emitted per detection path. Labels:pathis one ofheader,volume,volume_account,fanout(and laterbaseline,reputationfor Stages 2-3). Userate(...)to spot detection storms; a sudden rate jump onheadertypically means a contact-form vulnerability is being exploited, onvolume_accounttypically means an account password was leaked.csm_php_relay_actions_total{action,result}(counter): auto-freeze invocations attempted. Labels:actionis currentlyfreeze;resultisokorfail. Pair withcsm_php_relay_findings_totalto confirm freeze keeps up with detection.csm_php_relay_action_gone_total(counter): messages already absent from the spool by the timeexim -Mfran. Normal queue churn; not a failure. Sustained growth means the spool is moving fast and the freezer is racing the queue runner.csm_php_relay_path_skipped_total{path,reason}(counter): path evaluation that bailed before producing a finding. Labels:pathmatches the finding labels above;reasonenumerates the gate that fired (e.g. ignore-list match, missing scriptKey).csm_php_relay_spool_scan_fallbacks_total{reason}(counter): AutoFreeze fell back to a full spool walk to find msgIDs. Labels:reasoniscapped(the in-memoryactiveMsgsper script hit its cap, so a fresh disk walk was needed) orreputation(a late reputation finding arrived for a script with no liveactiveMsgs). Sustained growth oncappedmeans a single script is firing faster than the in-memory window keeps state for; consider raisingheader_score_volume_minor adding an ignore.csm_php_relay_active_msgs_capped_total(counter): per-scriptactiveMsgsset hit its cap and dropped the oldest entry. Counts the eviction event itself; the next freeze for that script will land incsm_php_relay_spool_scan_fallbacks_total{reason="capped"}.csm_php_relay_windows_active{kind}(gauge): retained per-script / per-IP / per-account window state. Labels:kindisscript,ip, oraccount. Sized by Flow E sweep cadence (5 min for windows, 24 h retention for accounts); flat values across hours are normal.csm_php_relay_msgid_index_size{layer}(gauge): msgID dedup index size by storage layer. Labels:layerismemory(in-process map) orbbolt(persisted batch writer). Memory ceiling is 200k entries; bbolt grows freely until the 25 h Flow E sweep prunes it.csm_php_relay_msgindex_persist_dropped_total(counter): bbolt persist queue overflow drops (the 4096-deep buffered channel was full when the watcher tried to enqueue). Should be zero in steady state; a non-zero value means the bbolt writer is blocked on disk and the in-memory dedup is the only thing protecting against double-fire on a queue-runner re-write.csm_php_relay_msgindex_persist_errors_total(counter): bbolt commit failures from the async batch writer. Each bump also emits a Criticalemail_php_relay_msgindex_persist_failedfinding. Disk-full or permissions issue on/var/lib/csm/state/csm.db.csm_php_relay_inotify_overflows_total(counter): kernelIN_Q_OVERFLOWevents on the spool watcher. Each one triggers a bounded recovery scan (default cap 1000 files); if the cap fires, also emitsemail_php_relay_overflow_scan_truncatedCritical. Sustained growth means the spool is churning faster than inotify can keep up – usually a backup restore or a real attack.csm_php_relay_spool_read_errors_total(counter):emailspool.ParseHeaderserrors on-Hfiles the watcher tried to consume. Usually transient (file disappeared between inotify event and open) and self-correcting; sustained growth points at a permissions or filesystem problem.csm_php_relay_userdata_errors_total(counter):cpanelUserDomainsresolver errors reading/var/cpanel/userdata/. Used by the Path 1Frommismatch check; errors here mean Path 1 is potentially undercounting until the read recovers.
Anti-spam learning
All series are prefixed csm_antispam_learning_. Pass totals and state gauges update after each completed Spam folder pass while learning runs. Contributions are counted when committed, including background retries. No series carries an account, mailbox or UID label. See Anti-spam.
csm_antispam_learning_reports_admitted_total(counter): Spam folder reports the learner accepted.csm_antispam_learning_contributions_committed_total(counter): cluster contributions committed from those reports.csm_antispam_learning_skips_total{reason}(counter): entries and reports skipped.reasonis a fixed code such astagged,malformed,opted_out,oversizeoraccount_capacity. On Postfix hosts it also counts Dovecot mailboxes not watched, such asdriverorno_junk, andcancel_unconfirmedwhen a pass could not confirm that a domain or mailbox was removed and so dropped no reports.csm_antispam_learning_pending_reports(gauge): reports waiting after the last pass. When discovery was incomplete, including unreadable folders or discovery limits, this is a lower bound.csm_antispam_learning_backlog_age_seconds(gauge): age of the oldest waiting report at the last pass.csm_antispam_learning_last_pass_timestamp_seconds(gauge): when the last pass completed.csm_antispam_learning_live_clusters,csm_antispam_learning_zero_target_records,csm_antispam_learning_body_bytes(gauges): learned clusters, zero-target guards and stored anchor message bytes.csm_antispam_learning_uncertain_intents(gauge): anchor writes whose outcome is still being reconciled.csm_antispam_learning_oldest_unchecked_timestamp_seconds,csm_antispam_learning_audit_completed_timestamp_seconds(gauges): anchor audit progress.csm_antispam_learning_reconstructed_clusters(gauge): missing live anchors newly scheduled for reconstruction since the learner started. Retries, ordinary value corrections and removal of retired anchors do not increase it.
If the learning ledger cannot be read after a pass, the engine-state gauges
(live_clusters, zero_target_records, body_bytes, uncertain_intents,
oldest_unchecked_timestamp_seconds, audit_completed_timestamp_seconds
and reconstructed_clusters) report NaN until a successful read.
Signature retroactive rescans
csm_signature_rescans_total(counter): rescans queued by a change to a tracked signature file’s content that completed. A rescan completes when the rolling deep YARA scan has scanned every file since the change. Re-installing identical rules, as a package upgrade or a repeated download does, does not count. Multiple updates before a rescan completes coalesce into one.
The watcher compares content hashes when a file’s mtime or size changes; symbolic links use the target file’s metadata. Failed reads and files replaced during hashing are retried without discarding the last verified state. First observations and file removals do not queue a rescan.
The rolling deep YARA scan reads files in path order and resumes each deep run where the previous one stopped, so one pass spans many deep runs. A rules update does not restart the pass: restarting on every update would never reach the files that sort last when updates come faster than a pass. The scan records where it stood when it took the update, and the rescan completes once the scan has wrapped and come back to that point. Every file is then scanned with the new rules within one pass. An update during a rescan starts a new one from where the scan then stands.
If the set of scan roots changes, or a root cannot be listed, the rescan starts a new lap from where the scan stands; the scan itself keeps going and still covers every root it can read. On a host whose accounts change faster than one pass, the rescan can stay pending; scanning is unaffected.
The rolling scan uses the YARA rules. YAML rule updates reach existing files only when those files change, as before.
A queued rescan is stored with the rule state that caused it, and the scan
stores its progress with its cursor, so daemon restarts keep both. The
saved queue is restored even if the state store becomes available after
the first watcher tick. Invalid queue records log a warning and are
repaired by queuing a fresh rescan; the scan neither tracks nor clears a
queue it cannot read. A deep run cancelled by shutdown or by the check
timeout saves no progress. Coverage gaps, reported by yara_scan_incomplete, do
not keep the queue pending.
Setting detection.rescan_on_signature_update to false stops the
watcher and pauses the queue: the scan keeps running but neither starts
nor completes a rescan. Re-enabling it resumes the saved work.
csm check-deep advances the same rolling scan, so it can complete a
queued rescan and increment this counter. Its findings are returned and
kept in scan history but not alerted or acted on, and the live scan does
not revisit those files until its next pass. Use csm run-deep when
findings should alert.
State from older builds contains only mtimes. The first successful read adds a hash without queuing a rescan if the recorded mtime still matches. A moved mtime without a recorded hash conservatively queues one rescan.
Counter reset semantics
Prometheus counters in CSM live in process memory. They reset to zero
whenever the daemon restarts (config change, binary upgrade, crash
recovery). This is the standard behaviour for every
Prometheus-instrumented daemon; Prometheus’s scrape pipeline detects
counter resets on its own and rate(), increase(), and
rate_over_time() all handle them correctly.
Operators should not alert on “counter decreased across a scrape” as
a failure condition. Alert on rate() or increase() of a counter
over a window long enough to absorb expected restarts.
Persisting counters across restarts would require writing to bbolt on every increment, which would not pay for itself. If a specific metric needs restart-stable behaviour later, a gauge-over-the-bbolt-counter pattern can be added for that one case without affecting the rest.
Caveats
- Scrape the web UI’s HTTPS port, not a separate listener.
curl -k/insecure_skip_verifyis appropriate only when the cert is self-signed and the network path is trusted. Pin a CA for anything else.- Prometheus label cardinality: per-account and per-IP labels are deliberately not exposed. Shared-hosting deployments with 1000+ cPanel users would otherwise overwhelm a Prometheus server.
- Metric vectors cap label-value combinations at 1000 children per
metric, including the overflow bucket. Once a vector reaches that
cap, new combinations are aggregated under
_overflow_.
Not instrumented (yet)
- Per-account labels on any metric. Deliberately off: shared-hosting deployments with 1000+ cPanel users would blow out Prometheus cardinality.
- Fanotify inline auto-response actions (the quarantine-while-
seeing-the-write path in
fanotify.go). The periodiccsm_auto_response_actions_totaldoes not count those; a follow- up may split the metric or add asourcelabel. - bbolt per-bucket size breakdown,
csm_store_used_bytes, andcsm_store_last_compact_ts. Deferred to the online-compaction follow-up of the retention work (seeROADMAP.md).
Audit Log (SIEM)
Audit Log
CSM ships source observations and notification findings to one or more SIEM-friendly sinks, deduplicated by observation identity within each batch and against each destination’s recent successful deliveries. Audit records include sources suppressed by notification filtering or rate limits, so SIEM correlation can still identify the original observation.
Two sink types ship today, both opt-in via csm.yaml. They can be
enabled together or independently.
Schema
Every event, regardless of transport, has the same shape:
{
"v": 1,
"ts": "2026-04-28T10:32:14.512938Z",
"finding_id": "8e3f1c204c1d8b95",
"severity": "CRITICAL",
"check": "webshell_realtime",
"message": "PHP execution primitive in uploads/",
"details": "...",
"file_path": "/home/customer/public_html/uploads/x.php",
"hostname": "host.example.com"
}
The v field is the schema version. CSM bumps it on incompatible
changes and will not bump it for additive fields, so SIEM parsers
can pin on v: 1 and ignore unknown keys.
An optional context key carries background beside the details, such as
ModSecurity being switched off for the account a malware finding concerns.
It is not part of finding_id or receipt matching, so a change in it alone
never produces a new record.
Recognized credentials in context are redacted before alert and audit output,
using the same policy as messages and details.
finding_id is a stable 16-hex-char hash of the canonical fields
(timestamp, check, severity, message, file path). Two emits of the
same finding produce the same ID, so downstream dedup works across
re-runs.
Firewall actions caused by a finding carry this same identity in the action log, including failed and refused attempts. The identity is captured before reason text is shortened and survives queued retries, challenge timeouts, central-intelligence decisions, and permanent-block promotion. Subnet and incident escalation link the latest known contributing finding. Older stored evidence without an identity and manual or maintenance operations remain unlinked; CSM does not reconstruct an identity from display text. Database session blocks link the original database finding, not a synthetic IP candidate.
The daemon audits source observations even when they repeat an earlier finding
or are filtered from operator notifications. Distinct observations keep their
own identities; recent replays of the same observation are suppressed per sink.
Receipts are kept in bounded memory and survive temporary sink failures. A
destination that missed a record can receive its replay without duplicating a
healthy destination’s record. Replays refresh their receipt’s position, keeping
recently replayed observations ahead of inactive receipts.
Receipt matching includes original details, scanner identity and source
attribution; distinct findings can share a legacy finding_id and still need
separate records.
Process enrichment alone does not create a new observation. Restarting or
reconfiguring sinks clears receipts; observations no longer in the receipt cache
can be emitted again. The legacy finding_id remains an action correlation key;
collectors must not treat it alone as a unique-record key.
Notification suppression and downstream finding observers keep their existing
behavior. Audit delivery still depends on the configured sink and scan findings
reaching the dispatcher.
The ts field records when CSM raised the finding, including process
monitoring, automatic actions, scanner health, and mail relay storage errors.
It is independent of when a sink delivers the event. A producer that builds
a finding without a time gets the moment the daemon received it, so ts is
never the zero time.
Alert dispatch fills missing times on its own copy of the findings. Reusing an unstamped input for a later occurrence gets a fresh time and finding id; replaying a finding with an existing time preserves its time and id.
Process context
Exec and outbound-connection findings on BPF-backed hosts carry an
optional process object with PID, PPID, UID, user, cPanel account
(when known), comm, exe, sanitized cmdline, and a parent chain. The
field is omitted when no context is available, so existing parsers
that ignore unknown keys see no schema change.
{
"severity": "HIGH",
"check": "outbound_connection",
"message": "Suspicious outbound connection",
"process": {
"pid": 4242,
"ppid": 4200,
"uid": 1001,
"user": "alice",
"account": "alice",
"comm": "ncat",
"exe": "/usr/bin/ncat",
"cmdline": ["ncat", "203.0.113.10", "587"],
"parent": {
"pid": 4200,
"ppid": 4100,
"uid": 1001,
"comm": "sh"
}
},
"timestamp": "2026-05-07T12:34:56Z"
}
The parent chain may be truncated at depth 5 and may stop early if an intermediate parent has been evicted from the cache.
File sink (JSONL)
alerts:
audit_log:
file:
enabled: true
path: /var/log/csm/audit.jsonl # default
The default path is created with mode 0640 and the parent dir
with 0750. The packaged logrotate fragment uses copytruncate
mode so the daemon’s open file descriptor stays valid across
rotation – no SIGHUP needed. It rotates daily and keeps 14 compressed
rotations. The 100 MB threshold permits early rotation when the host runs
logrotate more often than daily; it does not cap growth between runs. The sink
never drops records to stay under that size, because a detection flood would
otherwise blind the audit trail until the next rotation. On busy hosts, run
logrotate more often than daily so the size trigger can take effect.
Installation and upgrades refresh the fragment, including upgrades through
csm rehash.
If you move the audit log off the default path, add your own logrotate stanza for it: the packaged fragment names the default path only.
Tail it for an interactive view:
tail -F /var/log/csm/audit.jsonl | jq -c
Or hand it to a log shipper like Vector, Filebeat, or Fluentbit.
Syslog sink (RFC 5424)
alerts:
audit_log:
syslog:
enabled: true
network: udp # udp | tcp | unix | unixgram | tls
address: 127.0.0.1:514 # host:port for udp/tcp/tls, path for unix*
facility: local0 # default
tls_ca: "" # optional PEM file for tls transport
Wire-line is RFC 5424 with the JSON event embedded as the MSG body, so receivers that already understand the JSONL schema parse it the same way regardless of transport. UDP and unix-datagram emit one datagram per message; TCP, TLS, and unix-stream use LF framing.
Severity mapping onto the standard syslog level set:
| CSM severity | Syslog level | Numeric |
|---|---|---|
| CRITICAL | crit | 2 |
| HIGH | err | 3 |
| WARNING | warning | 4 |
Automated tests cover RFC 5424 output and UDP, TCP, TLS, Unix datagram, and Unix stream framing. Validate the chosen receiver configuration in a staging environment before production rollout.
Delivery failures and recovery
Each enabled destination retries independently after an open or write failure. The delay starts at one second and doubles up to one minute. A later dispatch retries a missing destination once its delay has elapsed; a quiet daemon waits until another finding arrives. Healthy destinations stay open during retries. Reloading audit configuration waits for in-flight writes before replacing sinks.
Failures and recovery are logged to journald. Monitor
csm_audit_sink_degraded{sink="jsonl"} and
csm_audit_sink_degraded{sink="syslog"} on /metrics: one means a configured
destination failed, zero means it is healthy or disabled. These values are
initialized when the audit pipeline first runs. Syslog health reflects local
connection/write results, not an acknowledgement from the receiving SIEM.
csm_audit_events_dropped_total{sink} counts events whose destination was
unavailable or whose write failed. Failed deliveries are not replayed
automatically; use the backfill command below to recover stored findings.
Backoff resets after a successful delivery, and configuration changes take
effect on the next dispatch.
Backfill
When you first turn on the audit log, the SIEM has no history. Use
csm export --since <when> to dump prior findings in the same JSONL
schema:
csm export --since 24h > recent.jsonl
csm export --since 2026-04-01T00:00:00Z > q2.jsonl
<when> is either an RFC 3339 timestamp or a duration relative to
now (24h, 7d). The output is one JSON event per line on stdout,
identical in shape to what the live sinks emit, so you can pipe it
straight into the same ingest pipeline.
Requires a running daemon.
What gets logged
Source observations and notification findings reaching the audit dispatcher, deduplicated by observation identity within each batch, before:
- the per-account rate limiter (so audit signal is not lost when email and webhook are throttled);
- the “blocked IP suppression” filter (so SIEM correlation sees events that operators were spared);
- the per-sink disabled-checks list (audit log is not subject to
email’s
disabled_checks).
This means audit-log volume is generally higher than the email or webhook stream. Plan SIEM retention accordingly.
What does not get logged
Before a record is written, its message and details replace recognized
password fields, API tokens, command-line secrets and cPanel session
identifiers with [REDACTED]. Session redaction covers cPanel, WHM,
Webmail, the shared server daemon, DAV and security purge log lines. It
keeps the account name beside a session identifier, leaves unrelated
lines of a multiline finding alone, and leaves already redacted text
unchanged when it runs again. Repeated and quoted values are all
covered, including the displayed form of NUL-separated arguments.
The same redaction runs on email digests, on new finding history in both the bbolt and the legacy JSONL backend, and therefore on the history the web UI serves and exports as CSV. Attack event messages are redacted before truncation, so a truncated line cannot hide a credential behind a cut service tag; attack events store no finding details. Account and IP attribution is read from the original finding, and finding IDs are computed from it too, so audit records still correlate with remediation records. Other structured fields are copied unchanged.
Two limits are worth knowing. Records written by earlier versions are not rewritten, so an existing log keeps whatever it already holds. The active-finding snapshot and the pending queues are process-local state that is read back and compared by the daemon itself, so they are not redacted.
The audit log is not a replacement for csm.history (the bbolt
history bucket). Only findings that pass through the audit dispatcher
are emitted. Internal state changes – daemon startup, reload events,
config changes – live in journald via csm.service and are not
mirrored here.
Action log
The audit log records what CSM found. The action log records what CSM did: one record per action, one file, one schema, whether the action came from the daemon, an operator command or the web UI.
/var/log/csm/actions.jsonl
csm actions # recent actions, one line each
csm actions --since 24h # RFC 3339 timestamp or a duration
csm actions --op respond.block_ip
csm actions --json # the raw records
csm action show <action_id> # one admission response and its attempts
Schema
{
"v": 1,
"ts": "2026-09-09T10:32:14.512938Z",
"hostname": "host.example.com",
"op": "respond.quarantine_file",
"actor": "daemon",
"finding_id": "8e3f1c204c1d8b95",
"target": "/home/alice/public_html/uploads/x.php",
"account": "alice",
"reason": "webshell_realtime",
"before": {"exists": true, "sha256": "5f2b...", "size": 1841, "mode": "-rw-r--r--", "uid": 1001, "gid": 1001},
"after": {"exists": false},
"result": "applied",
"recovery_path": "/opt/csm/quarantine/2026-09-09T10-32-14-x.php"
}
| Field | Meaning |
|---|---|
op | The privileged operation, using the IDs from the capability matrix. The matrix says what an operation may do; this says what it did. |
action | The specific change within an operation, such as block, unblock, apply or rollback. |
actor | daemon, cli or webui. actor_detail carries the operator’s source address or the command name. |
finding_id | The finding that caused the action, using the same ID the audit log emits, so the two streams join. |
command | The exact argv when CSM ran a program. Absent when the action used system calls only. |
before / after | The target file’s digest, size, mode and owner around the change. "exists": false after a quarantine is the record that the file is gone. |
result | applied, dry_run, failed or refused. A dry-run record says what CSM would have done. |
undo | A command that reverses the action, when an exact inverse is available. |
recovery_path | Retained quarantine content or a pre-clean backup, with a .meta sidecar. Use the quarantine recovery workflow; csm restore accepts state archives, not these files. |
Attempted operations include failures and safety refusals. An already blocked address does not produce another applied record. A successful file replacement or quarantine with a later warning stays visible with its recovery path.
A refused action is recorded too. “The safety rules stopped this” and “CSM never looked” are different operational states, and only one of them is a reason to change the configuration.
v is the schema version. It is bumped only on an incompatible change, so a
parser can pin on v: 1 and ignore unknown keys.
Long reasons and error messages are truncated with an explicit marker so one action cannot make the history unreadable. Targets and command arguments are preserved.
A file larger than 64 MiB is recorded without a digest rather than stalling the action while it is hashed. Quarantine digests come from the captured copy, after the move. Cleaning digests describe the exact bytes read and installed through the pinned file handles. Directories and symlinks have no digest.
What is covered
The capability matrix has an “Action record” column.
These operations write here today: firewall blocks and unblocks (automatic and
operator), whole-ruleset changes, file quarantine, surgical file cleaning,
process termination, mailbox suspension, the account-wide outgoing mail
hold, and an operator’s release of either (operate.release_mailbox, one
record per part undone), and lifting Stalwart’s ban of an address unblocked
in CSM (operate.clear_stalwart_ban, one record per ban). A hold record
carries the quoted whmapi1 release_outgoing_email command that lifts it. Everything else appears in the
daemon log only. The column is the authoritative list. Tests exercise the
operation paths, including failures and concurrent blocks.
The firewall keeps its own audit file because the web UI and the API read it. Its entries also appear here; dry-run decisions, failed attempts and whole ruleset apply/rollback records are recorded directly on this stream. Startup rollback records are written before the daemon requests a restart.
Admission records describe each step of an automatic response the admission
ledger admitted: op is respond.block_ip, action the kind of response
(block_ip, block_subnet, block_service, promote or challenge),
target its canonical target, reason the lane and the absolute expiry, and
result the step (reserved, executing, then applied, narrowed,
failed or unknown). action_id and action_version name the attempt and
the step. A record is written durably before the ledger lets it go, so after a
crash it can appear twice; drop a copy whose action_id, action_version and
ts match another. An admitted IP block takes these steps while the ledger
executes, with auto_response.dry_run false, the firewall on and outside
observe mode. Otherwise the ledger previews each response it is handed, so
an admitted response is reserved and then observe and is never
executed; in dry run with the firewall on outside observe mode it forecasts
block responses instead.
Their terminal outcomes are counted in the ledger without a new per-attempt
action-log row; bounded forecast examples and summaries are separate.
csm action show <id> asks the daemon for what the ledger holds on one
response, by the action_id of an admission record or by its candidate ID.
It prints JSON: the candidate’s kind, target, check, state, reason, outcome and
expiry, each attempt with its lane, reservation, finish and outcome, and for a
response refused because a block already enforced it, that block’s action and
end (link). A response whose history has retired, or that was never
admitted, is reported as not recorded. A retained retired attempt still has
its unknown outcome and retired: true; retirement is not proof of success.
Retention removes the candidate and all its attempts together. If a retained
candidate is missing an earlier attempt, the command reports ledger damage;
it does not claim the missing part aged out or return a truncated history.
The command uses candidate_id for the candidate and attempts[].action_id
for each attempt, matching the action log’s action_id. It never uses the
log’s action operation field for an identifier. The other top-level fields
are kind, target, optional check and finding_id, state, optional
reason, disposition, expires_at and link, and attempts (always an
array, including when empty). Each attempt has seq, state, optional
disposition and lane, reserved, optional finished, expires_at, and
retired only when true. Times use RFC 3339. link.effect identifies the
existing block; an omitted link.expires_at means permanent. Omitted
candidate expiry means no expiry was assigned; it is not a permanent block.
This command requires a running daemon and never changes admission or
firewall state. Each read is audited with op: respond.block_ip, action: show, actor: cli, actor_detail: csm action show, and the requested ID as
target. result: read means the lookup succeeded, never that a block was
applied; failed lookups use result: failed and error. These read records
have no lifecycle action_id or transition version. They use the existing
bounded, best-effort action log outside the admission outbox, so a sink
failure does not disable the diagnostic read. The finding-stream exporter
accepts these records as reads, salts their targets as opaque references and
drops error text; they cannot claim a block was applied.
It offers no resolve,
retry or undo operation; durable firewall actions remain under
csm firewall actions. The admission.action_show.v1 capability advertises
this local read command; it does not indicate that automatic execution is on.
Once an hour the ledger owner appends respond.block_ip summary records that
count its decisions on the responses handed to it during the hour ending at
ts. A summary sets count, how many responses it covers, and names no
target: action is the kind of response, reason the check that asked for
it, actor_detail the entry it came through (scan, or a derived entry such
as challenge_timeout or central), result the decision (queued,
coalesced, observe or refused) and error the refusal reason. An
attribution refusal also names its cause: no_observation (no reader
stamped the finding), kind_not_allowed (no parser vouched for the address in
a way the check accepts), target_mismatch (the response names an address its
evidence does not attribute), unreachable (the address is a client behind a
trusted proxy, which sends this host no packets of its own), proxy_peer (the
address is a known CDN or website-firewall edge, which carries many visitors)
or rewritten_log (the web server replaced the logged client address, or CSM
could not read its configuration, so the connecting peer is unknown). Refusals
at submission and queue placement keep the same cause as refusals at evidence
minting. A queued response that a later check refuses (its revalidation, an
inventory refresh, a configuration opt-out, a lost queue position after
reassessment, or the recheck of a pick before an attempt or a preview) counts
as refused too, with its reason and cause, in the hour the ledger ended it.
A displaced queued response also counts once. These rows retain that hour even
after a newer hour was reported. Hours discarded when a logging outage
exhausts summary retention stay discarded; later refusals for them never
return as partial summaries. An owner that runs automatic blocks and forecasts
them in dry-run mode also counts dry_run decisions, each with a prediction
of what live admission would have done: capacity_admit, existing_actual,
existing_shadow, ceiling, set_full, pending_recovery,
forecast_limited, not_modelled, capacity_evict, capacity_rotate or
equal_tier_wait. A prediction is not a refusal reason. A clean stop writes
the open hour early, so summaries for the same hour add up; a crash loses the
open hour, so the counts are lower bounds. Each summary also sets epoch, the
daemon run that counted it: a number that rises at every start of the owner,
so summaries of two runs never share one.
Refused previews and submissions displaced from the memory queue count as
refused with their reason, including every report merged into a displaced
submission. While the owner executes, a block it runs shows only in its
attempt rows above: the summaries count a live block pick only when the
ledger refuses it at its recheck or reservation. A batch the firewall
engine refuses while applying it, and a pick the panel verdict callback
allows, add no refused row; the first shows in its attempt’s outcome, the
second in the log and in the verdict counters of status. Summaries continue while the ledger or its clock is unavailable,
and stopping flushes completed decisions even if the final drain fails.
Their hour never precedes the latest recorded admission hour, including
clock readings from an incomplete startup.
actor uses the caller’s attribution when available and otherwise identifies
the process performing the action. Some web UI requests therefore record as
daemon; the web UI’s own action log retains the operator’s source address.
Rotation
The file rotates to actions.jsonl.1 at 10 MB. csm actions reads both.
Readers and writers coordinate rotation through actions.jsonl.lock.
Readers pin both files before releasing the lock, so rotation cannot hide
records and a slow reader does not hold up a writer.
CSM handles this rotation itself; the installed logrotate configuration leaves
the live file alone and archives actions.jsonl.1, which the sink never writes
to again, for 90 days. Older history is compressed there and is not read by
csm actions; ship the stream to a SIEM if you need it queryable.
Recording is best effort. Sink errors or panics do not change an action’s outcome. A write waits at most 250 ms, with at most 64 writes outstanding; a stalled sink or saturation can lose records. Normal CLI writes finish before the command exits. Keep this log on a local filesystem and monitor write-error warnings. The stream is not a transactional guarantee that every host change survives a crash.
PAM hook for CSM
The CSM daemon listens on /var/run/csm/pam.sock for authentication
events emitted by a PAM module (pam_csm.so). The module is shipped
with the package at /usr/lib/csm/pam/pam_csm.so; without it installed
the daemon’s PAM listener stays attached but never hears anything, and
the dashboard reports the watcher as deaf.
Quick install
sudo csm pam install # preview first with --dry-run
sudo csm pam status # confirm
csm pam install stages pam_csm.so into the platform’s security
directory (/lib64/security on RHEL, /lib/x86_64-linux-gnu/security
on Debian) and appends two lines to the standard service files. These
report successful logins:
auth optional pam_csm.so # managed-by-csm
session optional pam_csm.so # managed-by-csm
Targets: /etc/pam.d/sshd, /etc/pam.d/su, /etc/pam.d/sudo,
/etc/pam.d/password-auth (RHEL) or /etc/pam.d/common-auth (Debian).
Files that don’t exist on the host are skipped. Every edit creates a
timestamped .csm-backup-YYYYMMDDTHHMMSSZ next to the original.
Failed logins need one more line. A PAM module cannot see whether the
modules before it accepted the password, so the failure hook goes where
only a failed attempt arrives: directly before the terminal pam_deny.so
of the shared auth stack (password-auth on RHEL, common-auth on
Debian), which the standard remote login services include:
auth optional pam_csm.so authfail # managed-by-csm
A jump that skips pam_deny.so on success, such as Debian’s
[success=1 default=ignore] pam_unix.so, is widened by one so it skips the
hook too; uninstall narrows it back, including jumps to or past the end.
Each attempt that reaches this denial is reported once, whichever service
ran it. Earlier failures that return before it are not reported by this
hook. SSH failures drive the PAM brute-force and credential-stuffing
findings; only a successful SSH login clears those failures. Blocking
follows auto_response.block_ips. Failures from other services raise the
non-blocking pam_auth_failures alert. CSM reads pure-ftpd, Dovecot and
Exim logs separately; other services, such as vsftpd, proftpd or webmin,
get no blocking signal from PAM.
The installer refuses the failure-hook edit when it cannot prove the placement is safe. Any shared stack that refuses it makes the command exit non-zero, even if another stack was hooked. Success hooks may already have been added to service files or the shared stack. Placement is refused when:
- the stack is a symlink, which is how authselect manages it on RHEL 8 and
later. authselect rewrites the file on its next run, so the hook needs a
custom authselect profile; add the line above to that profile’s
password-authbeforepam_deny.so, widening any jumps that skip the denial so they also skip the hook. - there is no single
requiredorrequisitepam_deny.soauth line. - a jump crosses an
include,substackor@includeline, so its target cannot be counted. An authincludeor@includebefore the denial is also refused because its own jumps can escape into this stack. - a
requireddenial is followed by an authresetaction or an expanded auth include that could clear its failure. - a jump cannot be adjusted within PAM’s integer range, or the file uses line continuations or syntax the editor cannot parse.
On Debian and Ubuntu, pam-auth-update --force regenerates
common-auth and drops every CSM line; run csm pam install again after
it. csm pam status shows whether failed logins are reported.
Hosts installed before the failure hook existed report successful logins
only. Run csm pam install again after upgrading; it adds the failure
hook and leaves the existing lines alone.
Logins from loopback addresses are ignored. The module passes on the
remote host sshd reports; with UseDNS yes that is a host name, which
CSM cannot track or block. The daemon logs that once and ignores those
logins, so keep UseDNS no, the OpenSSH default.
Safety rails
- The
optionalcontrol flag is mandatory: a CSM outage must not block authentication.csm pam installwrites nothing else. - Test from a second SSH session before closing the one that
installed. If SSH or sudo breaks, rename the
.csm-backup-file back over the broken target and runcsm pam uninstall --keep-moduleon the live session. - The PAM module never reads or modifies passwords, never decides whether to permit auth, and never writes to disk. It opens a Unix socket, writes one line, closes.
Rebuild from source
The package ships the C source alongside the compiled module so operators can rebuild on hosts with a different libc:
cd /usr/lib/csm/pam
make
sudo install -m 0755 pam_csm.so /lib64/security/pam_csm.so # RHEL
# or
sudo install -m 0755 pam_csm.so /lib/x86_64-linux-gnu/security/pam_csm.so # Debian
Requires gcc and libpam-devel (RHEL) or libpam0g-dev (Debian).
From a source checkout, make -C build/pam check runs the module through
real libpam stacks. It writes test services under /etc/pam.d, so run it
as root in a throwaway container.
Uninstall
sudo csm pam uninstall # removes managed hooks and the module
sudo csm pam uninstall --keep-module # only edits, leave .so in place
Uninstall is idempotent, removes only lines marked # managed-by-csm,
restores any jump count install widened, and creates one fresh
.csm-backup- per file it edits. Files with no managed hooks are left alone.
If an edited file has an uncountable jump, an auth include before the
failure hook, or syntax it cannot parse, uninstall stops with an error
naming the file and leaves it as it is; restore the backup or remove the
marked lines by hand and fix the jumps. Parse errors identify the line
without printing its module arguments. The PAM listener’s dashboard verdict
returns to deaf the next time the dashboard polls.
What the dashboard tells you
| Dashboard label | Meaning |
|---|---|
ok | Attached AND has emitted at least one finding in the last 7 days. |
idle | Attached, upstream alive, no recent findings (healthy quiet). |
deaf | Attached, but no upstream process is feeding the watcher. |
degraded | The watcher failed to attach on startup (config error, missing kernel). |
For the PAM listener specifically, deaf is the verdict whenever the
socket has not received a single connection in the last 24 hours and
the daemon has been up longer than the 15-minute grace window. Hover
the badge in the UI for the install hint.
Pre-start hooks for the CSM daemon
CSM does not implement its own hook protocol because systemd already ships one: drop-in units. This document shows how phpanel-server-agent (or any operator-supplied automation) wires a pre-start script that runs before the daemon binds.
Why drop-ins, not a custom hook directory
Inventing a hook_pre_start: config or a magic /etc/csm/hooks/
directory would duplicate systemd.unit(5) drop-ins, which already
support: per-service ordering, ExecStartPre= (cancel start on failure),
EnvironmentFile= (load secrets), User=/Group= (privilege drop), and
Conditional*= (skip when prerequisites absent). Every distro CSM
supports already runs systemd. Use it.
Recipe for phpanel-server-agent
-
Author your hook script. Convention:
/opt/phpanel/hooks/csm-prestart.sh.#!/bin/bash set -e # Re-resolve the panel hostname and stash the IP for CSM's infra_ips. PANEL_IP=$(getent hosts panel.example.com | awk '{print $1}' | head -1) if [ -n "$PANEL_IP" ]; then cat > /etc/csm/conf.d/10-phpanel-runtime.yaml <<YAML infra_ips: - $PANEL_IP YAML fi -
Install the drop-in. Copy the example from
/usr/lib/csm/profiles/csm-prestart.example.conf(shipped by the CSM package) to/etc/systemd/system/csm.service.d/10-phpanel-prestart.conf, editing theExecStartPre=line to point at your script. -
Reload systemd.
systemctl daemon-reload systemctl restart csm.service -
Verify with
systemctl cat csm.service- your drop-in should appear at the bottom under# /etc/systemd/system/csm.service.d/....
Standard environment available to the hook
CSM_HOSTNAME- set by CSM at runtime, not available inExecStartPre. If you need the hostname pre-start, read/etc/hostnamedirectly.CSM_INFRA_IPS- same caveat. Resolve from your panel agent’s context (it knows the panel address).
Failure semantics
ExecStartPre= blocks startup on non-zero exit. Use ExecStartPre=-/path
(dash prefix) when you want CSM to start even if the hook fails. For
phpanel-server-agent’s reconcile step, the dash prefix is recommended so
a transient panel outage doesn’t keep CSM from running.
Supply-chain Advisory Database
CSM’s supply-chain check (supply_chain_vuln) parses customer
dependency lockfiles – composer.lock and package-lock.json – and
matches the resolved versions against a local advisory database. The
scanner ships in the binary; the advisory data does not. With no
advisory file present the check is a silent no-op.
This mirrors the YARA-forge mirror posture (see ROADMAP item 1): CSM carries the matching machinery, and the data is delivered out of band by an operator or a sync job. Shipping a stale CVE snapshot inside the binary would give false confidence and rot between releases.
Location
<state_path>/advisories/supply-chain.json
state_path is the daemon’s configured state directory. The file is
read on each scan; updating it requires no daemon restart and no
recompile.
Format
OSV-subset JSON. A dependency version is flagged when it falls inside
any range: version >= introduced AND (fixed == "" OR version < fixed).
introduced: "0" means “from the first release”.
{
"advisories": [
{
"ecosystem": "npm",
"package": "lodash",
"ranges": [{ "introduced": "0", "fixed": "4.17.21" }],
"id": "CVE-2021-23337",
"severity": "high",
"summary": "Command injection via template."
},
{
"ecosystem": "composer",
"package": "vendor/package",
"ranges": [{ "introduced": "2.0.0", "fixed": "2.4.1" }],
"id": "CVE-2024-xxxxx",
"severity": "critical",
"summary": "..."
}
]
}
ecosystem:composer(Packagistvendor/name) ornpm(package name as it appears undernode_modules/).severity:critical-> Critical finding,high-> High, anything else (medium/low/empty) -> Warning.- Version comparison is numeric per dotted segment; a leading
vand any pre-release/build suffix (-rc1,+build) are ignored.
Populating it
Generate the file from a public feed and write it to the path above on a schedule. Two common sources:
- OSV (
https://osv.dev) for the npm/Packagist ecosystems. Export the relevant ecosystem ranges into the subset format above. - WPScan / wpvulndb style feeds for WordPress plugin advisories (future ecosystem; not parsed by the current lockfile scanner).
A signed mirror job analogous to the YARA-forge mirror is the intended long-term delivery mechanism; until then an operator cron that fetches and transforms a feed is sufficient.
Scope (current)
- Parsed:
composer.lock,package-lock.json(npm lockfile v1, v2, v3). - Not yet parsed:
requirements.txt/Pipfile.lock(Python),Gemfile.lock(Ruby), WordPress plugin manifests. These are follow-on ecosystems for the same matcher.
Upstream Threat-Intel HTTP Contract (v1)
This document specifies the HTTP API CSM expects when
reputation.upstream.enabled: true. A panel host (or any HTTP server)
can implement it; phpanel’s reference implementation lives in
phpanel-server-agent.
Endpoint
GET <reputation.upstream.url>/lookup?ip=<ip>
reputation.upstream.url must be an https:// URL with a host. Plain
http:// is accepted only for loopback hosts such as 127.0.0.1,
[::1], or localhost. CSM appends /lookup to that base URL.
Authentication
CSM sends Authorization: Bearer <token> when
reputation.upstream.token (or its env var) is set. The server SHOULD
reject requests with a missing/invalid token via HTTP 401.
Before each HTTP request, CSM reads reputation.upstream.token_env from
its process environment. A non-empty value overrides the static token
field. Editing an environment file or exporting a value in another shell
requires a daemon restart to take effect; configuration reload does not
replace the process environment. See credential rotation.
Request
ipquery parameter: IPv4 dotted-quad (e.g.1.2.3.4) or IPv6 in any RFC 5952 form. The server SHOULD normalize before lookup.
Response
200 OK
{
"ip": "1.2.3.4",
"score": 75,
"source": "upstream",
"ttl_sec": 900
}
ip(required): echoes the input.score(required): integer 0..100. Higher means worse.0is “no signal” and the aggregator excludes it from averaging.source(optional): free-form identifier the server may set for audit-log surfacing.ttl_sec(optional): number of seconds CSM should cache this answer. Overridesreputation.upstream.cache_ttl_min. The server SHOULD return the smaller of its own cache lifetime and a sane default.
401 / 403
CSM treats as “no signal” - does not retry, does not cache. Logged as a warning so operators see auth misconfigurations.
4xx / 5xx other
Treated as “no signal.” The aggregator continues with whatever scores the other sources returned.
CSM rejects malformed 200 responses, including an ip field that
does not match the requested IP or a score outside 0..100.
Caching
CSM caches each (ip, score) pair locally for cache_ttl_min minutes
(default 15) or for ttl_sec if the response provides it. A cache hit
does not contact the upstream. Bursts of detection events that touch
the same IP within the TTL window produce a single upstream call.
The local cache is capped at 10,000 IPs. When the cap is reached, CSM prunes expired entries first and then removes the entry with the nearest expiry time until the cache is back under the cap.
After repeated upstream failures, CSM opens a short circuit breaker and temporarily treats the source as “no signal.” When the cooldown expires, only one probe request is allowed through; concurrent callers keep using the fail-open path until that probe succeeds or fails.
Sample server stub (Go)
http.HandleFunc("/api/csm/ti/lookup", func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer "+os.Getenv("PHPANEL_CSM_TOKEN") {
http.Error(w, "unauthorized", 401); return
}
ip := r.URL.Query().Get("ip")
score := lookupCachedScoreFromAbuseIPDBOrRedis(ip) // your impl
json.NewEncoder(w).Encode(map[string]interface{}{
"ip": ip, "score": score, "source": "phpanel-cache", "ttl_sec": 900,
})
})
Verdict Callback HTTP Contract (v1)
When auto_response.verdict_callback.enabled: true, CSM POSTs each
auto-block decision to the configured URL and reads an advisory
response. The endpoint is not a per-tenant nftables enforcement
plane - it lets phpanel observe and label decisions, downgrade them to
audit-only, and attach tenant attribution.
CSM runs its local block validation before calling the endpoint. Malformed IPs, disabled IPv6 targets, infra IPs, and configured block limits are rejected locally and are not sent to the callback.
Callers
Two parts of CSM call the endpoint, told apart by the request source:
auto_response: each automatic block of a single address, after local validation and before theauto_response.dry_rungate.severityis"auto". Anallowverdict skips that block. In dry run with the firewall on outside observe mode, the admission ledger also asks for locally eligible block attempts, so one block can bring an additional request. Its requests carryCSM auto-block: <check>as thereason, run up to four at a time and honourallow_unsignedandrequire_response_signature; anallowends the forecast without an example. Existing effects, protected targets and full capacity need no owner request; retries ask anew and reload discards old policy answers. The v1 request contains no dry-run field, and request totals are not an exact multiple of legacy calls.bpf_enforcement: only whenbpf_enforcement.verdict_callback: trueis also set. CSM asks about the destination of an outbound connection the BPF hook denied, or would have denied in dry-run.reasonisbpf_enforcement:<check>:<port>andseverityis the finding’s severity (WARNING,HIGHorCRITICAL). The answer only annotates findings: the verdict,tenant_idandnoteare added to the finding details. It never changes the kernel decision, which is made before the call. Answers are cached for one minute per destination, reason and severity, and the finding that triggers a lookup is sent without one.
Subnet blocks, promotion of an existing block to permanent, and manual blocks from the CLI, web UI or API do not call the endpoint.
Endpoint
POST <auto_response.verdict_callback.url>
Content-Type: application/json
Accept: application/json
X-CSM-Signature: sha256=<hex> (HMAC-SHA256 over raw body, omitted if no secret configured)
auto_response.verdict_callback.url must be an http:// or https://
URL with a host. CSM POSTs directly to that URL (no path appended).
Authentication
CSM signs the raw request body with HMAC-SHA256 using the configured
secret and sends the digest in X-CSM-Signature: sha256=<hex>. The
secret is resolved once per request/response exchange: if
hmac_secret_env is set and the env var is non-empty, it wins over the
static hmac_secret field. Request and response verification use the same
key for that exchange. The value comes from the running process environment;
editing an environment file requires a daemon restart, even if configuration
is reloaded. See credential rotation.
CSM refuses to enable the callback without a non-empty hmac_secret or
resolved hmac_secret_env value. Operators can set
allow_unsigned: true only for staged rollouts that intentionally send
unsigned requests and accept unsigned responses. Without that explicit
opt-in, CSM rejects an unsigned allow verdict and proceeds with the
default block.
The server MUST verify the signature with constant-time compare and reject unsigned or invalid requests with 401.
Response signing (required by default)
The server MUST sign its response body with the same HMAC-SHA256 scheme
(same secret, same X-CSM-Signature: sha256=<hex> header). CSM verifies
the response signature with constant-time compare before parsing the
verdict; an unsigned or forged response is rejected and CSM falls back
to its default block. Without this check an on-path attacker could
downgrade every “block” decision to “allow”.
If the server cannot sign responses yet (e.g. mid-rollout), operators
can set auto_response.verdict_callback.require_response_signature: false in csm.yaml to temporarily accept unsigned responses. The
default is true and there is no other way to disable the signature
requirement.
When response signing is disabled but a secret is configured, CSM still
checks any nonce or timestamp fields the server echoes. A nonce must
match the request nonce, and a timestamp must stay within the five-minute
skew window. A legacy response that omits both fields keeps working.
Request body
{
"ip": "203.0.113.10",
"reason": "mail_bruteforce",
"severity": "auto",
"source": "auto_response",
"nonce": "9a7e84f0c123ab45e102d4f8b9c1d3e0",
"timestamp": 1748345600
}
ip(required): the address CSM is about to block, or forbpf_enforcementthe connection’s destination.reason(required): short reason string, for exampleCSM auto-block: wp_login_bruteforce.severity(optional):"auto"for auto-response decisions; the finding’s severity forbpf_enforcement.source(optional):"auto_response"or"bpf_enforcement"; see Callers.nonce(always sent): a 128-bit hex random value unique to this call. The panel MUST echo it verbatim when response signing is on. When response signing is disabled, CSM still checks the value if the panel echoes it.timestamp(always sent): unix seconds when CSM produced the request. The panel does not need to validate it but MAY use it to reject stale calls.
Response body (200 OK)
{
"verdict": "block",
"tenant_id": "tenant-42",
"note": "matched dovecot user pattern",
"nonce": "9a7e84f0c123ab45e102d4f8b9c1d3e0",
"timestamp": 1748345600
}
verdict:"block"(default; also returned if field omitted) or"allow"to override. Any other string is rejected by CSM as a protocol error, treated as “no decision” (CSM proceeds with the default block per fail-open semantics).tenant_id(optional): attribution string; CSM logs it alongside the decision.note(optional): free-form note logged for the verdict.nonce(required when response signing is on): MUST equalrequest.nonce. CSM compares with constant-time equality and rejects mismatches as replay attempts. When response signing is disabled, CSM still checks this field if present.timestamp(required when response signing is on): unix seconds when the panel produced the reply. CSM rejects replies whose timestamp drifts more than five minutes from local clock. When response signing is disabled, CSM still checks this field if present.
Failure semantics
- Network error / timeout / non-200 HTTP response: CSM logs a warning and proceeds with the default block. The hook is fail-open at the transport level.
- Local nonce generation failure: CSM does not send the request and proceeds with the default block.
- 200 OK with
verdict: "allow": CSM does not modify nftables; logs the override to stderr. If no HMAC secret is configured, this is accepted only whenallow_unsigned: trueis set. - 200 OK with
verdict: "block"or omitted: standard block path runs (which still honorsauto_response.dry_runif set). - 200 OK with unknown
verdictstring: rejected; treated as fail-open (default block). - 200 OK with
verdict: "allow"and no HMAC secret whileallow_unsignedis false: rejected as an unsigned downgrade; treated as fail-open (default block). - 200 OK with missing or invalid
X-CSM-Signature(andrequire_response_signaturenot turned off): rejected as forged; treated as fail-open (default block). Operators see the rejection in stderr; recurring rejections indicate either a panel-side rollout gap or an active on-path attack. - 200 OK with mismatched
nonceor timestamp drift over five minutes: rejected as replay whenever a secret is configured and the field is present; treated as fail-open (default block). - 200 OK with missing
nonceortimestamp: rejected as replay when response signing is required. Missing replay fields are accepted only whenrequire_response_signature: falseis set for a legacy response shape.
Limits
- CSM caps response body at 64 KB. Larger responses are rejected.
- Default request timeout is 2 seconds (configurable via
timeout_sec, range 1-30).
Sample server stub (Go)
http.HandleFunc("/api/csm/verdict", func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
if !verifySignature(r.Header.Get("X-CSM-Signature"), body, secret) {
http.Error(w, "bad signature", 401); return
}
var req struct {
IP, Reason, Severity, Source string
Nonce string
}
_ = json.Unmarshal(body, &req)
tenant := lookupTenantOwning(req.IP) // your impl
verdict := "block"
if isPanelInfra(req.IP) { // never block our own infra
verdict = "allow"
}
respBody, _ := json.Marshal(map[string]any{
"verdict": verdict, "tenant_id": tenant,
"nonce": req.Nonce, "timestamp": time.Now().Unix(),
})
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(respBody)
w.Header().Set("X-CSM-Signature", "sha256="+hex.EncodeToString(mac.Sum(nil)))
_, _ = w.Write(respBody)
})
When to graduate this to per-tenant enforcement
This v1 hook surfaces tenant attribution but does not synthesize per-tenant nftables rules. If/when phpanel wants real per-tenant enforcement (block IP X from tenant Y’s mailboxes only, allow elsewhere), that’s a separate plan that will need:
- Multi-set nftables rule generation per tenant
- A way to map an IP+tenant pair to an nftables verdict
- Lock-step config sync between CSM and the panel’s tenant table
Do that work then, against a real call site, not on spec.
Fleet Correlation Contract (v1)
CSM is a per-host sensor. It does not correlate findings across hosts itself; cross-host (“fleet”) correlation is owned by phpanel, which already receives every deduplicated finding through the per-finding webhook. This document defines the contract phpanel implements to turn that stream into one fleet incident per attacker instead of N identical per-host incidents.
This is the phpanel-side correlation design: CSM stays a sensor with no new inbound network surface on hosts. A peer-to-peer ingest endpoint inside CSM was considered and rejected for v1 (it would add an inbound auth plane and key distribution to every host).
What CSM already provides
With alerts.webhook.type: phpanel, CSM POSTs one HMAC-signed
message per deduplicated finding that reaches alert.Dispatch (see
the per-finding webhook in internal/alert/webhook.go). This sink runs
before operator alert suppression and the operator alert rate limit, so
phpanel receives the correlation stream even when local notifications
are muted:
POST <alerts.webhook.url>
Content-Type: application/json
X-CSM-Signature: sha256=<hex> (HMAC-SHA256 over the raw body)
X-CSM-Hostname: <hostname>
{
"hostname": "host-7.example",
"timestamp": "2026-05-29T10:00:00Z",
"finding": {
"check": "ssh_bruteforce",
"severity": 2,
"source_ip": "203.0.113.10",
"message": "...",
"timestamp": "2026-05-29T10:00:00Z",
"tenant_id": "...", // when known
"domain": "...", // when known
...
}
}
CSM keeps each finding in a durable queue and delivers them in order. A 2xx answer acknowledges a finding. HTTP 400, 413 or 422 refuses the record itself: CSM moves it to a bounded local archive, counts it as lost and sends the next finding. Use these codes only for a problem with that one record. After a long run of refusals with nothing accepted, CSM stops archiving and retries instead, so a panel that refuses every request does not lose the queue. The hold survives config reloads and daemon restarts and ends after an accepted delivery. Redirects are not followed and do not acknowledge findings. Any other failure is retried with backoff, and later findings wait behind it.
Every field phpanel needs to correlate across hosts is present:
hostname identifies the reporting sensor, and finding.source_ip
identifies the attacker when the detector has a remote source. For
mail_subnet_spray and smtp_subnet_spray, use source_ip when present
(older findings); otherwise use the single subnet in finding.cidrs.
Other checks’ cidrs, including http_asn_crawl, are response targets
and must not become an attacker key. Severity is the CSM numeric enum
(0 warning, 1 high, 2 critical). No extra
CSM API or peer transport is required; fleet correlation is a matter of
phpanel aggregating the stream it already receives.
Correlation rule (phpanel side)
- Verify
X-CSM-Signature(HMAC-SHA256 over the raw body) with the shared secret before trusting any field. Reject unsigned or mismatched messages. - Group incoming findings by the attacker address described above within a sliding window (recommend matching CSM’s 15-minute incident merge window).
- When the same attacker address appears in findings from two or more
distinct
hostnamevalues inside the window, open one fleet incident keyed on that address rather than per-host incidents. - Attach each contributing
(hostname, finding)to the fleet incident’s timeline so an operator sees the spread across the fleet. - Account/mailbox/domain identifiers are host-local; do not use them as the cross-host key (they collide across tenants). Use only the attacker address as the cross-host key.
Severity and de-duplication
- A fleet incident’s severity is the max of its contributing findings.
- Findings carrying no attacker address (host-integrity events, config drift) are not fleet-correlated; they stay per-host.
- The same
(hostname, attacker address, check)arriving repeatedly is the normal heartbeat of an ongoing attack; collapse on the attacker address and count contributions rather than opening a new incident per message.
Out of scope (v1)
- Peer-to-peer ingest between CSM hosts.
- A CSM-side fleet incident store. CSM remains stateless with respect to other hosts; the fleet view lives in phpanel.
- Cross-host auto-block orchestration. phpanel may drive per-host blocks through each host’s existing control surface; CSM does not coordinate blocks across the fleet.
Abuse Report Contract (v1)
When reputation.report.enabled: true, CSM queues a minimized report for
each eligible confirmed-abuse finding, one copy per usable target. This page
defines the request a collector receives and the answers CSM understands. Use it to
build a private collector. The central abuse database implements the same
contract and accepts ed25519 targets only. Node setup is in
Threat Intelligence.
What CSM reports
CSM reports a finding only when all of these hold:
- Its check maps to a class listed in
reputation.report.classes. The mapping is in Threat Intelligence. - It is Critical, or High or above for
bad_asn_egressandspam. - It names a source address that parses as an IP address.
- Its finding timestamp is not zero.
- That address passes CSM’s protection checks. CSM excludes private,
loopback, link-local, multicast, carrier-grade NAT, documentation,
benchmarking and selected special-purpose ranges, literal IPs and
CIDRs in
infra_ipsandfirewall.infra_ips, Cloudflare edges, and built-in or operator-configured crawler IP ranges. This check does not resolve infrastructure hostnames or consult rDNS crawler results. Teredo clients are not excluded solely for using Teredo. For NAT64 and 6to4 sources, the same checks also apply to the embedded IPv4 address. - For
spam, the address does not belong to a shared mail service.
Request
POST <target url>
Content-Type: application/json
X-CSM-Node: <node_id>
X-CSM-Key: <key_id>
X-CSM-Timestamp: <unix seconds, UTC>
X-CSM-Nonce: <32 lowercase hex characters>
X-CSM-Signature: ed25519=<hex> | sha256=<hex>
Authorization: Bearer <token> (hmac targets whose token_env variable is set)
CSM posts to the configured URL as written and appends no path. Remote URLs must use HTTPS; plain HTTP is accepted only for loopback hosts. CSM does not follow redirects: a 3xx answer is a failed delivery and is retried. Each request times out after 15 seconds.
Body
{"ip":"203.0.113.10","class":"bruteforce","count":1,"first_seen":"2023-11-14T22:13:20Z","last_seen":"2023-11-14T22:13:20Z"}
ip: the source address in canonical text form. IPv6 is compressed and lowercase. An IPv4-mapped IPv6 address is sent as dotted IPv4.class:bruteforce,php_relay,credential_stuffing,bad_asn_egressorspam.count: observations in this report. Version 1 always sends 1.first_seen,last_seen: RFC 3339 UTC time of the finding, which may carry fractional seconds. Version 1 sends the same value in both.
The body carries no hostnames, account names, domains, mailboxes, paths or process data. The signature covers the SHA-256 of the body, so verify against the raw bytes received, not a re-encoded copy.
Signature
CSM signs a message built from six fields followed by the timestamp. Each field is written as a 4-byte big-endian length, then its bytes:
| Order | Field | Value |
|---|---|---|
| 1 | Node | X-CSM-Node value |
| 2 | Key | X-CSM-Key value |
| 3 | Method | POST |
| 4 | Path | Decoded path of the configured URL, or / when it has none. The query string is not included. |
| 5 | Body hash | SHA-256 of the raw body, 32 raw bytes |
| 6 | Nonce | X-CSM-Nonce value |
The X-CSM-Timestamp value follows as an 8-byte big-endian signed integer.
The path is the one in CSM’s configuration. A collector behind a proxy that rewrites the path must verify against the original path.
The transport selects the algorithm:
ed25519: an Ed25519 signature (RFC 8032, not prehashed) with the node’s private key.key_envmust hold the 64-byte private key encoded as 128 hex characters, not a 32-byte seed. The signature is sent asX-CSM-Signature: ed25519=<128 hex characters>. The collector verifies it with the public key it registered for that node and key ID.hmac: HMAC-SHA256 keyed with thekey_envvalue exactly as set, not hex-decoded, sent asX-CSM-Signature: sha256=<64 hex characters>.
This differs from the per-finding webhook, whose HMAC covers only the body.
Collector checks
A collector must:
- Find the key for
X-CSM-NodeandX-CSM-Key. - Rebuild the signed message from the raw body and headers, then verify the signature. Compare HMAC values in constant time.
- Reject a timestamp outside a window you choose, for example five minutes either side of your clock.
- Remember each node and nonce pair for that window and reject a repeat.
Answer these failures with 401 or 403, never 422. CSM keeps retrying queued reports so approval or clock repair can restore delivery. Queue overflow can still discard reports during the outage; see Delivery.
Test vector
This is a signature test using a documentation address. CSM’s reporting gate excludes that address from real reports.
Inputs:
X-CSM-Node: node-1X-CSM-Key: key-1- Target URL
https://abuse.example.net/report, so the path is/report - The body shown in Body, without a trailing newline
X-CSM-Nonce: 0123456789abcdef0123456789abcdefX-CSM-Timestamp: 1700000000- HMAC secret
example-secret
Body SHA-256:
56f3e2c77c1456d41f210109cd069f5795c66ea3d825af3ac5f1d0bc1a008d2c
Signed message, hex:
000000066e6f64652d31000000056b65792d3100000004504f5354000000072f7265706f7274
0000002056f3e2c77c1456d41f210109cd069f5795c66ea3d825af3ac5f1d0bc1a008d2c
000000203031323334353637383961626364656630313233343536373839616263646566
000000006553f100
Header:
X-CSM-Signature: sha256=ae5e149cc7789d78c65b30d8d52a372a6d3bb999a84355d88d42f9c1874666e8
An ed25519 target signs the same message bytes.
Responses
CSM ignores the response body.
| Answer | What CSM does |
|---|---|
| 2xx | Delivered. The report leaves the queue. |
| 409 | Treated as delivered. Use it only for a report already stored. |
| 422 | Dropped and counted as lost, never resent. Use it only for a body you will never accept. |
| 429 | Retried after a positive Retry-After, in seconds or as an HTTP date, capped at one hour. A missing, invalid, zero or past value uses one minute. Repeated throttling can increase short waits as described below. |
| Any other status, timeout or network error | Retried after one minute. |
Threat Intelligence describes how waits grow while a collector keeps throttling.
Delivery
- Reports first enter a bounded memory queue before being written to disk. Memory overflow, failed persistence or a crash before persistence can lose reports. Normal shutdown drains accepted memory work to disk.
- Persisted reports survive restarts and are sent to each target in queue order. When a target fails, its later reports wait for the next pass while other targets can continue receiving theirs.
reputation.report.spool_maxis one shared capacity across all targets. Each report uses one entry per target. When full, CSM drops the oldest entries across the whole queue, including entries for other targets.- Target names identify queued destinations. Use unique, stable names: the last usable target with a duplicate name wins. Removing or renaming a target, or a key error that makes it unusable at startup, discards its queued entries on the next delivery pass.
- A retry sends the same body with a new nonce, timestamp and signature, so the same report can arrive more than once, for example when an answer is lost. Deduplicate on node and body hash, and answer 409 for a body already stored.
Signed scored-set downloads
When reputation.central.auth_target names an ed25519 target, CSM signs
each scored-set download with that target’s node key:
GET <set_url>[?since=<version>]
X-CSM-Node: <node_id>
X-CSM-Key: <key_id>
X-CSM-Timestamp: <unix seconds, UTC>
X-CSM-Nonce: <32 lowercase hex characters>
X-CSM-Signature: ed25519=<128 hex characters>
The signed message is built as in Signature, with method
GET, the decoded path of the download URL, and the SHA-256 of an empty
body. The query string is not signed. Each request has a fresh nonce. CSM
sends no body, no bearer token, no credentials from the URL and no other
X-CSM- headers, and it does not follow redirects. A full download carries
no since; a later one carries exactly one.
| Answer | What CSM does |
|---|---|
| 200 | Verifies the signed snapshot or diff, then uses it. |
| 304 | Keeps its current list. |
| 401 | Keeps its last verified list, logs that the node was refused, and tries again at the next refresh. |
| Any other status, timeout or network error | Keeps its last verified list and tries again at the next refresh. |
Sample collector (Go)
This HMAC signature-verification skeleton is not a complete collector.
secretFor returns the shared secret for a node and key ID, or nil when
unknown. Before using it, implement the timestamp and replay checks above,
body validation, duplicate detection and durable storage at the marked
point. Return 2xx only after storing the report. If configured, also verify
the bearer token. The body limit rejects oversized requests rather than
verifying a truncated prefix.
// signedMessage rebuilds the bytes CSM signed for this request.
func signedMessage(r *http.Request, body []byte) ([]byte, error) {
ts, err := strconv.ParseInt(r.Header.Get("X-CSM-Timestamp"), 10, 64)
if err != nil {
return nil, err
}
sum := sha256.Sum256(body)
var msg []byte
for _, f := range [][]byte{
[]byte(r.Header.Get("X-CSM-Node")),
[]byte(r.Header.Get("X-CSM-Key")),
[]byte(r.Method),
[]byte(r.URL.Path), // the path CSM was configured with
sum[:],
[]byte(r.Header.Get("X-CSM-Nonce")),
} {
msg = binary.BigEndian.AppendUint32(msg, uint32(len(f)))
msg = append(msg, f...)
}
return binary.BigEndian.AppendUint64(msg, uint64(ts)), nil
}
func handleReport(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 64<<10))
if err != nil {
http.Error(w, "read failed", http.StatusBadRequest)
return
}
secret := secretFor(r.Header.Get("X-CSM-Node"), r.Header.Get("X-CSM-Key"))
msg, err := signedMessage(r, body)
scheme, sigHex, _ := strings.Cut(r.Header.Get("X-CSM-Signature"), "=")
sig, hexErr := hex.DecodeString(sigHex)
if secret == nil || err != nil || hexErr != nil || scheme != "sha256" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
mac := hmac.New(sha256.New, secret)
mac.Write(msg)
if !hmac.Equal(sig, mac.Sum(nil)) {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
// Check the timestamp window and the nonce here, then decode and
// store the report. Answer 409 for a body this node already sent.
w.WriteHeader(http.StatusNoContent)
}
For an ed25519 target, check for the ed25519 scheme and replace the HMAC
step with ed25519.Verify(publicKey, msg, sig).
Building & Testing
Toolchain and prerequisites
Use the Go version required by go.mod (currently 1.27.4), including its
formatter. A newer local formatter can disagree with the pinned CI linter.
For an installed Go launcher that supports toolchain selection:
export GOTOOLCHAIN=go1.27.4
export PATH="$(go env GOROOT)/bin:$PATH"
go version
Linux tests need PHP CLI and python3-cryptography for the shipped PHP runtime
and release verification regressions. The release-verifier tests skip when the
module is absent and fail when CSM_REQUIRE_PYTHON_VERIFIER=1, which the CI
test jobs set so a missing module cannot pass as a silent skip. The Web UI
JavaScript tests need Node 24 or newer the same way: they skip without it and
fail when CSM_REQUIRE_NODE=1, which the CI test job sets. Builds with yara,journal,bpf also need
CGO, pkg-config, YARA-X 1.23.0 and the systemd
development library. Use the release builder or the documented test images.
CI images ship the module toolchain and run with GOTOOLCHAIN=local, so a
stale image fails instead of downloading another toolchain in every job.
Build
# Standard build (no YARA-X)
go build ./cmd/csm/
# Build with YARA-X support (requires libyara_x_capi)
CGO_LDFLAGS="$(pkg-config --libs --static yara_x_capi)" go build -tags yara ./cmd/csm/
Test
Some tests keep scratch folders under their package directory, because system temporary directories change the behavior they test. Source inventories skip a folder removed during their walk; a missing root or failed read still fails.
Process-context unit tests stub start times for synthetic PIDs so unrelated processes cannot supply cache identity evidence. UID and command mismatches are checked separately. Atomic replacement fixtures require newer birth times only when testing WAF log replacement; plugin-copy tests also work without filesystem birth-time support.
Admission retry tests repeat evidence with its original observation timestamp and check the refusal reason. Changing that timestamp creates conflicting evidence and bypasses episode placement. Forged-account flood tests keep the host and real account backlogged in each evidence class and check repeated alternating turns. A single account pick can pass through sort order alone.
The default CI job runs every package with the race detector, no extra build tags, and a 30-minute per-package timeout. Run its exact test command on Linux:
go test -race -timeout=30m -covermode=atomic -coverprofile=coverage.out -coverpkg=./internal/... ./...
make test and the test step of make ci use -short for local iteration;
they do not reproduce the full CI suite. make test-full runs the default-tag
suite with the CI flags (no -short, fresh results, the 30-minute package
timeout). Neither default command tests the shipped optional backends. The additional required production job runs:
scripts/production-tests.sh portable
That script selects all packages with yara,journal,bpf, -race, -count=1,
-p=2 and -timeout=30m, retains JSON execution evidence, and checks the test
inventory. See production and kernel tests for the
builder, tagged lint/security commands and required kernel execution.
On macOS, use scripts/go-linux.sh for Linux code. Build the repository’s
PHP-enabled test image once with an available container builder:
docker build -f build/Dockerfile.systemd-test -t csm-linux-test .
GO_LINUX_RUNTIME=docker GO_LINUX_IMAGE=csm-linux-test scripts/go-linux.sh \
bash -ec 'apt-get update -qq && apt-get install -y --no-install-recommends python3-cryptography
go test -race -timeout=30m -covermode=atomic -coverprofile=coverage.out -coverpkg=./internal/... ./...'
The wrapper derives the default Go version from go.mod, shares persistent
caches across worktrees, and grants fanotify/nftables capabilities. Its plain
Go image does not include PHP, python3-cryptography or the production CGO
libraries. The command above installs Python’s verifier in the disposable
test container. An image built
with another runtime can be selected through GO_LINUX_RUNTIME and
GO_LINUX_IMAGE; local test execution still goes through the wrapper.
Kernel firewall regressions use isolated network namespaces:
scripts/go-linux.sh go test -tags nftkernel ./internal/firewall -race -count=1
The separate kernel gate requires an isolated Linux runner with the documented boot capabilities. A successful macOS or default-tag suite does not demonstrate BPF LSM attachment.
Antispam learning benchmarks
Run the learning benchmarks on Linux with repeated iterations:
TMPDIR=/var/tmp/csm-gotest go test ./internal/antispam/learn ./internal/antispam/learnwatch -run '^$' -bench . -benchtime=3x -count=3
Each publication iteration starts with one live anchor and adds one new contributor. Each flood iteration starts from an empty baseline before new messages arrive. Database state is reset between iterations; setup and cleanup are excluded from timings. The flood wait metric is the mean across iterations. The benchmarks check that publication commits, discovery visits the corpus, the second account submits before the flood finishes, and audit coverage completes.
Storage metrics count logical keys and values, including contribution charges, account usage and pending-message indexes. They exclude bbolt page overhead and anchor message bodies. These fixed workloads do not establish maximum database sizes or production latency: message names, contributor mixes, filesystem performance and native rspamd calls change the cost. Runtime allowances remain unchanged.
Fuzz
CSM has fuzz targets for parsers that read attacker-controlled input, including Exim mainlog lines, Dovecot maillog lines, Apache Combined Log Format, /proc/net/tcp rows, wp-config.php bodies, /etc/shadow, auditd comm fields, and finding messages coming back from the WebUI.
Fuzz targets live in *fuzz*test.go files across the internal packages and scripts. Fuzz targets do two things:
- Their seed corpus runs as part of the normal test suite.
go test ./...executes every seed, so a known-bad input stays a regression test forever. - The actual fuzzer runs with
-fuzz=FuzzFoo.
Run a target for a fixed time while investigating:
go test ./internal/checks/... -run=^$ -fuzz=^FuzzExtractPHPDefine$ -fuzztime=30s
Run only the seeds:
go test -run=Fuzz ./...
If the fuzzer finds a crasher it writes the failing input to testdata/fuzz/FuzzFoo/<hash>. Commit that file alongside the fix and the input becomes a permanent seed.
Adding a fuzz target:
func FuzzMyParser(f *testing.F) {
// Seeds: real-world valid shape, empty, malformed.
f.Add("valid input")
f.Add("")
f.Add("corrupt/truncated")
f.Fuzz(func(t *testing.T, s string) {
_ = myParser(s) // must not panic on any input
})
}
Keep the target tight: call one function, assert it returns. Output verification belongs in a regular test.
Lint
make lint # must pass before push
make fmt-check # checks tracked Go files with the pinned formatter
make lint uses repo-local cache directories under .cache/ and the timeout
set in .golangci.yml, the same one CI uses. Install the pinned tools with
make tools; the golangci-lint version is GOLANGCI_LINT_VERSION in the
Makefile, and CI runs the same release. It lints the Linux build, so a macOS
host checks the code that ships rather than reporting its linux-only callers
as unused. Production-tag lint needs the Linux CGO libraries described in
production tests.
make sec, make vuln, and make check-fixtures are the local security,
vulnerability and fixture checks. For an aggregate local check use make ci,
then run the full default and production commands above. Passing local checks
does not replace the required kernel and cloud jobs.
The gosec pin includes the upstream module loader fix. Version 2.30.0 can split large packages in deeply nested checkouts and report false type errors. Keep this pin aligned in the Makefile and both build images until a tagged release includes the fix.
Linter config in .golangci.yml: errcheck, govet, staticcheck, unused, ineffassign, gocritic, misspell, bodyclose, nilerr.
CI/CD
GitLab CI (.gitlab-ci.yml) is the internal build pipeline. It runs lint/test/package jobs, publishes internal packages, mirrors to GitHub, and creates the public GitHub release artifacts.
| Stage | What it does |
|---|---|
| .pre | Manual CI tools and builder image builds (amd64 and arm64) block dependent jobs when their definitions change on a branch. |
| preflight | Release preflight rejects version tags without a usable cPanel image, after required images are built. |
| lint | Pinned golangci-lint and formatter, vet, blocking gosec/govulncheck, fixture privacy, and Prometheus config validation. |
| test | Full default-tag race/coverage suite (30-minute package timeout), shipped-tag lint/security/race gate, required kernel/service gate, four pinned clean-application gates, and the attack replay corpus. |
| build | amd64 and arm64 release binaries with YARA-X CGO and the yara journal bpf build tags. arm64 builds run under QEMU in rootless BuildKit. |
| package | RPM + DEB via nFPM |
| integration | Spin up cloudv-1 AlmaLinux and Ubuntu hosts plus the configured clean cPanel image via phctl, install the pipeline-built amd64 packages, run the integration test binary, collect coverage, and confirm every test server was really deleted. main integration is manual and can omit cPanel; version tags require the cPanel image, baseline-to-candidate package upgrade, installer, WHM, mail watcher and forward guard checks. |
| sign | Detached signatures on release artifacts |
| publish | Internal GitLab Generic Package Registry (versioned + latest) |
| repo | Publish RPM/DEB to the public mirrors.pidginhost.com apt/dnf repos |
| pages | Docs + coverage HTML (GitLab Pages preview) |
| cleanup | Remove old package versions |
| release | GitLab release on tags matching v* |
| github | Mirror to GitHub + upload release artifacts (auto on tag push) |
Lint and test jobs use the shared Go cache runner. Compiled and analysis caches rotate with the CI tools and builder image tags because Go does not detect changes to installed C libraries. Module downloads remain shared. The default suite executes on every pipeline even when compiled packages are cached. Release build and package jobs stay on separate runners and do not consume these caches.
Preflight, lint, test, build and package jobs retry up to twice when the runner, registry, scheduler or API fails; a failing script is not retried. A newer push to the same ref cancels the older pipeline’s lint and test jobs. Integration, kernel tests and publishing jobs neither retry nor get cancelled. Integration retries the Ubuntu package index update while the mirror syncs; the image repair and the CSM install run once.
The test job uses gocover-cobertura -strict so unresolved package information
fails coverage conversion. A successful test job must include every covered
package in its report.
When an image definition changes on a branch, run its blocking manual image jobs: ci-tools, or both builder architectures. Preflight waits for those builds, and both release builds wait for preflight while still running alongside lint and tests.
Release builds and preflight run in branch and tag pipelines, including scheduled and manually started runs. They are excluded from merge-request pipelines; merge requests use the checks in the branch pipeline.
Public Releases
To cut a release:
- Move the
[Unreleased]heading inCHANGELOG.mdto the new version (e.g.[2.4.2] - YYYY-MM-DD), commit asrelease: cut X.Y.Z.CHANGELOG.mdholds one block of ten minor versions. The first release of a new block (3.40.0, 3.50.0, 4.0.0) moves the previous block and its link references todocs/changelog/<first>-<last>.mdand adds that file to the archive list at the top ofCHANGELOG.md. - Tag and push:
git tag vX.Y.Z git push origin main vX.Y.Z - Wait. The tag pipeline runs integration, publishes packages to the mirror, creates the GitHub release, and uploads every artifact including the fresh
merged-coverage.out. No manual pipeline clicks needed.
Tag pipelines require INTEGRATION_CPANEL_IMAGE to name a clean cPanel CI
image, or CSM_RELEASE_WITHOUT_CPANEL to state why no image is available.
INTEGRATION_CPANEL_PACKAGE optionally selects its compute package and
defaults to cloudv-2. A release taken without cPanel coverage records
cpanel_coverage: "absent" in dist/cpanel-release.json; see
cPanel release tests.
Tag-specific publish dependencies require preflight, unit tests,
fixtures, corpus, production tags, kernel tests, signed artifacts and
integration. Repository publication and the GitLab release depend on
publish; the GitHub release also directly requires integration and the
test gates. A missing csm-kernel runner leaves publication pending. A
missing cPanel image fails tag preflight.
See cPanel release tests for image acceptance and
the roadmap
for remaining operational readiness work. These configured dependencies are
not evidence of a successful live release run.
The Pages workflow builds the coverage report and badge from merged-coverage.out of the latest GitHub release that carries one (it walks back through releases if the newest is missing the asset), and renders it against that release tag’s sources, so a file removed since the release does not break it. A failed release-list request or response parse stops the workflow. Mirrored pushes to main that change the docs, README, workflow or coverage renderer can run before the new release’s assets exist and show the previous release. The release job requests another Pages run after uploading the assets to refresh the badge; a manual workflow run can retry a missed refresh.
Installs and upgrades on end-user servers come from the GitHub release artifacts or the apt/dnf mirror. The internal GitLab package registry is operational tooling only.
Code Conventions
- Imports: stdlib, blank line, third-party, blank line, internal. Use
goimports -local github.com/pidginhost/csm - Errors: Return up the call stack. Wrap with
fmt.Errorf("context: %w", err) - Store:
store.Global()singleton bbolt DB. Always nil-check. - State:
state.Storehandles finding dedup, alert throttling, baseline tracking, latest findings persistence. Passed to subsystems at init - Web UI: Vanilla JS, no framework, no build step, and syntax up to ES2019 (a test rejects later syntax). Tabler CSS framework. Each script keeps its helpers in a function scope and adds only to the
CSMnamespace; the shared runtime iscsm-core.js,csm-format.js,csm-page.jsandcsm-live.js. UseCSM.get()/CSM.post()/CSM.delete()for API calls. Escape string-built markup withCSM.esc(); prefer DOM APIs for attacker-controlled values. Run the script tests withnode --test ui/. - API responses: answer through
writeJSONError,writeOK,writeItems/writeAll/writeCappedandwriteJSON, so every route follows the contract in API Reference: JSON errors,okon actions, lists underitems, UTC instants, durations in seconds and severity labels. - Logging: New code should use
internal/log(wrapslog/slog). Legacyfmt.Fprintf(os.Stderr, "[%s] ...", ts())call sites remain valid until migrated.
Content-bound dismissal state
state.Store.SetBaseline retains a known content hash when a result for the
same finding has no hash, including duplicate results and baseline resets.
An explicit new hash replaces the earlier one. Findings absent from the new
baseline are still removed.
ReconcileDismissedContent accepts fresh content evidence only. Empty hashes
and ScanCarryForward snapshots neither adopt content nor revoke a dismissal.
Revocation deletes the entry, invalidating pending delivery and resolution
callbacks tied to it. Finding keys and fingerprints stay unchanged. The daemon
calls it before deduplication, FilterNew and delivery in dispatchBatch, and
before filtering and delivery in respondToInitialScan. Every observation in
the batch contributes content evidence even when notification deduplication
keeps only one finding. For an ended dismissal it returns the batch without
that key’s hashless, carried or previously dismissed observations, so
deduplication keeps the new content, and it logs the ended dismissal with the
key quoted so file names cannot inject log lines or terminal controls. The
initial scan retains the full batch in history and audit, including the
observations omitted from notification and response. Pending shutdown findings
retain ScanCarryForward in their private storage format so replay cannot
treat stale content as fresh.
Undo restores the previous content hash only when the dismissal created the baseline or changed its content. Content adopted after a dismissal that left it alone survives the undo.
Response admission primitives
internal/admission is a standard-library-only package that live responses
reach only as a preview. Its persisted enum values are fixed by golden tests.
Assess sets ReassessBy to the first instant the current class or severity
falls, or all roots become stale, without new evidence. Redundant evidence
can preserve the tier after another root expires. Each corroborating pair
lasts until its local root loses freshness or its support expires; the
longest-lived pair determines how long corroboration remains available.
Equal deadlines and input order do not affect the result. Direct compromise
evidence may expire while corroboration continues to preserve the same class.
SlotLimits.Take adds a scope holding only for one-unit block_ip work.
Zero-cost and multi-unit work do not add holdings, even when marked as
eviction-capable. Free slots still decrease by the granted cost.
Effect records use an expiry-ordered key followed by the action ID. Decoding
requires a canonical value that reproduces the key and all recorded fields.
Victim references accept either an admission attempt ID or an import’s
eff_ name followed by 32 lowercase hex digits. Record keys remain attempt
IDs; an import has no record key. Imports have the fixed tier (C2, High)
and occupy general slots. The codecs keep their existing versions: an
older build refuses a ledger containing import claims, so such a ledger
must stay on a compatible build. A ledger that never executed has no such
claims.
The effect records’ partition tally rejects negative counts, underflow and overflow;
failed additions return the original tally. Combining a tally with set
occupancy also returns an error and the original occupancy if either
partition’s total is invalid or overflows. A full-set deferral raises the
capacity notice for any assessed tier in a live ledger. A preview ledger
raises it only for Critical work. A pending block that evicts names its
victim in its effect record and its intent; an admission victim’s record
names the block that claimed it and stops holding a slot once that block’s
application is verified with the victim’s removal proven. A verified
block that never removed its victim releases the claim; an unproven
removal is an unknown outcome, never a verified one. A definite failure
releases the victim claim; unknown outcomes keep it through retirement
until the victim expires. An import, a stamped legacy block the firewall
lists each turn, is a (C2, High) victim with no record of its own: only
the claiming block’s record and intent name it, by its eff_ name, and
no other block claims it while that record does. Opening rejects duplicate
import claims and requires a general slot above the import tier. Execution
uses the current turn’s list; a missing or changed import drops the claim.
Scheduling, reservations, observe previews and wake calculations share
that list and count at least as many general imports as it names, while
preserving a larger occupancy count. An overfull partition funds no
replacement until the complete state fits. Observe previews write no
claim or effect record; wake calculations write nothing. A preview
reports the block whose slot it took, so the owner counts previewed
replacements in status; a readback reports none.
A member’s sweep also ends its import claim; a delayed outcome may still
name the import in its intent without looking for a victim record.
A verified eviction marks its victim’s record gone at the evicting block’s
outcome. A verified block that two consecutive firewall answers no longer
hold is gone from the first of them, whatever removed it: an operator
unblock, undo or flush, or a lost kernel element. A held answer in
between starts the count again; the count lives in memory, so a restart
only delays the mark. Failed answers keep all previous misses and count
nothing new. Bounded batches get two answered checks before advancing,
so a larger configured capacity cannot hide later records. A record that
changed in between is left to what changed it. A gone record holds no
slot, so waiting work can take it, and stays until its expiry sweep.
The first stored gone record moves the ledger to schema 14, which older
builds refuse. Schema 14 requires the live-start marker, effect bucket and
tally even after all records expire. A check that marks nothing leaves
the schema unchanged. Only a live ledger has effect records, so preview
and forecast hosts never write schema 14.
The owner asks at most once a minute, and only while the ledger counts a verified block. The firewall answers with the retirement rule reversed and writes nothing: an element still at the address that no other action’s verified row proves counts as held, even after a forward wall step makes the block’s own row look expired, and so does an address whose family the firewall cannot read. A failed or short answer marks nothing and counts as no miss.
Startup drops the state row of an admission-record victim whose eviction the ledger verified, and of a block the ledger marked gone, so a restart cannot restore an element whose slot the ledger no longer counts. A running owner retires the row of a block it marks gone as it retires an evicted victim’s, so a firewall reload cannot restore it either. A cleanup pass that clears no row is retried twice at once, then after waits that double from two seconds to a minute; a row that leaves ends the wait, and a new eviction goes ahead of the waiting rows and is tried at once. Imports have no durable cleanup record: a restart before retirement restores the import only until its original expiry. Startup readers report an error if the effects bucket disagrees with the schema or an effect is a nested bucket; they do not treat damaged eviction storage as an absent record. Expiry can remove either record first. A late definite outcome still ends the claim using the retained intent. A reservation that expired without running releases its claim. Only records still holding slots contribute to occupancy and expiry wakes. Scheduling retains scope turns while a lane has no usable slot or strictly lower victim. A live ledger on a trusted clock may instead rotate an equal-tier record out of a full partition for fresh work: work whose episode row has had no verified response, never work queued without one. The victim is a verified record of exactly the work’s tier, in the partition of the work’s own lane, whose 30-minute hold passed on both the admission clock and the ceiling’s elapsed time since its verified application. A reboot grants no elapsed time and a degraded reading allows no rotation. Each partition is shared among its active scopes, using the queue’s integer share rule: the scopes whose unexpired, unclaimed records hold its slots, the scopes of queued work that would hold one at the last schedule cycle, and the requester’s. Each live schedule cycle on a trusted reading that reads the victim pool first moves each partition’s remainder cursor to the next active scope and stores it beside the queue cursors, so reservations share the set as their cycle did; a preview ledger never moves it. A zero-share requester waits for a later cursor turn. Work at or over a positive share rotates only its own scope’s records; work below it reclaims only the scope with the greatest excess among those holding a verified record of its tier in that partition at another address, never taking that scope below its share, and waits if that scope has no record whose hold passed. The oldest application, then the lowest action, goes first. Imports never rotate. The claim and its outcome are those of an eviction, opening accepts an equal-tier claim only inside the block’s own lane partition, and a live rotation raises the capacity notice. Notice allocation preserves the new attempt’s recovery hold; insufficient room sends the notice to its fixed overflow record. An older build refuses a ledger holding a rotation claim or a moved set cursor. Saved ceiling turns also keep their earned credits through slot pressure and restart; backoff keeps its existing release behavior. Records and intents without an eviction keep their version 1 encoding.
Policy lookups can overlap across registration, minting and validation, so they must be immutable or safe for concurrent use. Callers must serialize every operation on an inventory generation tracker. Off cPanel, hosting inventory reads account directories without checking mount state: an empty readable root contributes no accounts, while other readable roots still do. An unreadable required root fails the whole snapshot. A changed incarnation token gives an account a new generation, even when the account was replaced between two observations. Every account carries one: cPanel’s recorded creation date, elsewhere the device, inode and birth time of its home directory. An account whose token cannot be read, whose home is not a directory or whose Linux filesystem supplies no birth time fails the snapshot, including a home listed as a symlink. The cPanel creation date must contain only decimal digits and be recorded exactly once. Missing, empty, overlong or overflowing values fail the snapshot. cPanel records zero for its own system entries; recreating one between observations keeps its generation when the token stays the same. An observed absence retires the generation even for a zero-dated entry. Tenant edits do not change the token.
The admission ledger (store.AdmissionLedger) keeps this state durably in
the daemon’s state database, in adm: buckets it creates the first time it
is opened. The daemon’s admission owner (internal/admissionowner) opens it
at startup, settles what a previous process left outstanding after its first
clock reading and ceiling, before ingress or admission, and holds the only handle: one goroutine makes every change,
records clock readings on a timer, applies the ceiling at startup and, after
a reading at the saved limit, on reload or the first tick that sees a changed
configured value, refreshes the inventory from
complete reads, delivers audit rows to the action log and notices through
the queue-health path with their own pacing, and reads status on a timer.
Once the ingress has made new decisions, the owner drains it on a short
timer. Each group is frozen before its fresh clock reading, so concurrent
submissions wait for a reading that includes their observation time. A
clean stop drains all held work and checkpoints final decisions before
closing the generation. A failed drain, including its clock or publication
step, closes admission until a drain succeeds. Its cause survives tick,
inventory and reload recovery, so a lasting failure is one stop with its cause.
A drain whose only failure is a damaged ledger record is not a failed drain:
the arrivals naming that record are discarded and counted lost, the rest
commits, admission stays open, and status and csm doctor keep the damage
cause until the daemon restarts. If a later write, checkpoint or snapshot
fails in that drain, admission closes until a drain succeeds; recovery
clears the drain failure but keeps the damage cause for the discarded work.
Failed ceiling reloads stay pending until a tick applies and revalidates
them; inventory refreshes cannot reopen admission in the meantime.
The daemon wires the owner into the response funnels
(checks.SetResponseAdmission) while it runs: each automatic response the
legacy funnels select is minted as evidence and submitted to the ingress, and
derived responses (challenge timeout, central, incident, mail subnet, ASN
crawl, netblock, permanent block) carry the root evidence that caused them.
Legacy blocking never waits for admission and its outcome never changes a
legacy decision. The owner serves its queue as observe previews: a schedule
reserves and charges each pick and ends it observed in one transaction, so a
crash cannot leave a preview reserved. Each hour it appends counts of its
decisions to the action log. One owner serializes every write, and each call
is one transaction, so a failed call changes nothing. Admission time
comes only from recorded clock readings: a wall clock that steps back never
lowers it, a new boot credits no elapsed time, and a reopened ledger admits
no new work until it records a fresh reading. Evidence is immutable once
published, and later reports of the same observation are kept as bounded
links. A later mint that differs only in its owner, because the inventory
changed between the two mints, is refused as a stale identity. If the owner
and any other field differ, including the finding, the record is invalid.
A candidate takes its entry, check and finding link from its primary
evidence, coalesces repeated requests without refreshing its queue age, and
moves from queued through reserved and executing to one outcome. A proven
failure may requeue after a backoff, with at most three attempts in total;
an unknown outcome never retries. Reserve and Execute report whether the
call granted the step; reading back an attempt that is already reserved or
running grants nothing. A new process settles what the previous one left
outstanding before it reserves anything: a reservation, which recorded no
intent and so dispatched nothing, fails and may retry; a running attempt
ends unknown. Nothing is dispatched again. A handle that has granted a
reservation refuses to settle, and an unknown outcome needs the intent its
attempt recorded until that uncertainty is retired, including when reading
the outcome back. A missing intent or intent bucket returns a corrupt-record
error without changing the ledger.
An empty ledger settles without a clock reading.
The owner’s kill-point tests stop it at each step of a block turn and of
audit delivery, then start a new owner on the same ledger. After the
restart a reservation has failed and retries as its next attempt, a running
attempt is unknown and holds its address until it retires, the startup
reads hold back every row not finished applied, the consumers see only
outcomes finished applied, and no attempt reaches the engine twice.
Original attempts keep their charges, and retries consume new charges
without renewing the original expiry. Synthetic clocks keep the scenarios
independent of runner load.
Candidate records without an existing-effect link keep version 1 and their original encoding. A refused existing-effect candidate can carry the block name and expiry in version 2. Its maximum encoded size includes the link only when present; attempts and audit rows keep their existing formats.
Schema 2 of the ledger adds its queue, and the first open upgrades a schema 1 ledger in the same transaction; legacy work that cannot fit the unassessed general partition refuses the upgrade intact. Ingress, queued and in-flight work share 1000 positions; 200 take only direct compromise or corroborated evidence. Each partition reserves 64 positions for memory transfers, so durable work cannot consume handoff space. In a full partition each verified scope has a fair share: a scope at its share can only replace its own lowest-tier, newest candidate with stronger work, and a scope below its share takes one position from the scope most over its share. A displaced candidate ends as queue overflow. Queued candidates end at their age-out, or at their effect expiry while waiting to retry, and are assessed again when a root’s freshness changes; an inventory refresh ends those whose roots name a retired account. The scheduler serves the reserved lane first, alternating direct and corroborated turns, then the general lane at C3:C2:C1 4:2:1. Inside a class, verified scopes rotate, a scope serves Critical:High:Warning 4:2:1 and a severity serves its oldest candidate; deficit accounting lets a candidate that costs several block units wait its turn without starving. The scheduler’s position persists, and every pick is revalidated before it is returned. Regression tests reopen the ledger during unfinished scope and reserved sub-lane rotations, then check service order across the restart. Overload recovery uses elapsed-time refill and verifies each reservation’s charge, so a fresh cursor or an uncharged pick cannot satisfy the test. Free-slot rotation tests require a granted reservation, its stored effect, and the expected occupancy and charge, including after a limit increase. A missing effect must fail rather than count as proof that no victim was taken. A schedule passes over candidates the caller names as pending, such as one waiting for the panel’s verdict: they are never picked or deferred, their scopes keep their turns, and the next wake waits only for their deadlines. A candidate the caller names as allowed ends without a charge, so it is picked as uncharged work at no ceiling or history cost, even with both spent, and wakes the owner at once. Detectors hand work to a nonblocking ingress that applies the same fair shares against the owner’s latest snapshot of the queue. It judges freshness at that snapshot’s admission time advanced by elapsed monotonic time, so an idle owner does not make new evidence look future-dated. Durable victims remain counted until commit. Reports arriving during a drain remain held for a later report-only commit, and overflow counts persist with their links. Ingress cursor progress and loss counts are checkpointed even when only refusals occurred; publication cannot overwrite later memory decisions. The owner persists held items in groups; an ingress generation that ends without a clean close is recorded as interrupted, since its unpersisted items cannot be counted. A failed snapshot after a committed drain stops new submissions until the owner publishes a snapshot at least as new as that commit, from the same ingress generation. Shared queue damage retains the whole handoff; only arrivals carrying damage are discarded.
Schema 3 of the ledger adds the emergency ceiling; the first open upgrades a schema 1 or 2 ledger in the same transaction, with empty buckets because its recent spend is unknown. The engine sets the hourly ceiling L; a fifth of it, rounded up, is reserved for direct compromise and corroborated work, and the general lane cannot spend it. Each lane refills a token bucket at its hourly rate from elapsed time within a boot, up to ten minutes of that rate with a one-unit floor for a nonzero lane, and the first limit of a new ledger fills both buckets once. A new ledger takes that first limit together with the legacy hourly block count, charged at the end of its hour and subtracted from the fill; a count that cannot be read starts the ledger without credit. The reader rejects null scalar values and uses the last instance of a repeated local hour, even when other hours separate its instances. Retained import reads validate the ceiling totals and the charge links to ordinary attempts. A later limit only clips saved credit; the owner checkpoints elapsed time at the saved rate before changing it. Every reservation, retry included, is charged to the lane its schedule picked, in the reservation’s transaction, after a reserved lane is rechecked against the candidate’s current assessment. A readback grants and charges nothing; its lane must be zero or equal the recorded lane, including when an upgraded attempt has no recorded lane. A charge counts until a full hour of admission time and of elapsed time have both passed, so neither downtime nor a forward clock step releases it early. Schedules serve no more than each lane can charge, and the ledger’s next wake includes when waiting work gains budget. Challenge work spends fair-turn and history budgets without spending ceiling units.
Schema 4 adds storage accounting; the first open upgrades a schema 1, 2 or 3 ledger in the same transaction. Evidence stays stored while a candidate names it as a root. Evidence no candidate names, and candidates that ended before any attempt, each wait in a bounded ring and leave oldest first, so rejected traffic cannot allocate rows without bound. A candidate’s details become history at its first reservation: each reservation charges the growth of their largest possible size to the allowance of its lane, and a fifth of the fixed history budget is reserved for direct compromise and corroborated work. Each allowance admits history at a rate that spreads its size over the seven-day review window and saves at most ten minutes of that rate, so a flood cannot fill a week of history in an hour. Scopes earn history turns in fixed quanta, so large records cannot take more than their share, and a record that has earned its turn holds its lane until the credit covers it. An ended candidate’s details are kept through the review window and, unless the candidate was only previewed, for as long as the firewall row its attempt may have written can last: a failed attempt’s row can outlive a failed restore write, and startup must still know its action while the row could be restored. Endings written before this rule keep their earlier time. They are retired after thirty days, or earlier when an allowance needs the room, oldest first. An unresolved outcome is pinned in a separate recovery reserve until recovery settles it, and new work waits while that reserve is full. Tick refills the allowances and retires history at its target in the clock’s transaction. The next wake runs the scheduler read-only; unassessed or overdue queue entries wake immediately for a sweep. Report links stay readable after a policy change.
Schema 5 adds the outbox; the first open upgrades a schema 1 to 4 ledger
in the same transaction. Every admitted step of an attempt (reservation,
execution, outcome) writes an audit row in its own transaction, and the
reservation holds room for all three in the recovery reserve, which the
outbox shares; ended history is retired only after its rows are
acknowledged. Response gaps are recorded in the transaction that causes
them as coalesced notices, one record per reason, check and action family,
with a bounded count of examples, a fixed overflow record per kind and a
fixed Critical summary that no flood can refuse. An applied block whose
root is a Critical outbound destination (c2_connection) raises a
Critical partial-containment record, reason unsupported_containment
with outcome applied: the input-chain drop stops replies from the
address, but outbound-only traffic to it still leaves. Its response ran,
so it is not counted in the Critical summary or the doctor’s gap row.
Records may use only a
fixed share of the reserve, a key is due for delivery at most once an hour
and a summary once a minute after acknowledged delivery. While the owner
previews or forecasts, ledger notices go to findings history only, as
Warnings starting “Admission preview:”. The independent stopped-admission
notice still reaches alert channels as a Warning saying existing blocking
is unaffected. While it executes,
a gap is an outage of automatic blocking: notices go to the alert channels
with their kind’s severity and no preview prefix, and the stopped-admission
notice is Critical. A ledger notice keeps the mode it was raised in: a
record with an event from while the owner executed, or from a real attempt’s
outcome, is delivered live after a dry run starts, and one raised only in a
dry run stays a preview after execution returns. Other records from
before a restart take the mode the owner starts in. If the startup clock
fails, later notice mode boundaries use the first valid clock reading;
restored notices keep the initial startup mode. Deliveries before that
reading use the saved ledger time to retain the mode of events raised
during delivery. Live and preview groups back off
separately, so failing previews never delay a live notice. The stopped-admission notice
uses the mode at delivery, including a retried one. Stop retries retain their original timestamp, cause and stop identity. Acknowledgements
carry the count and first-event time of the record read, so a repeated
acknowledgement cannot consume a later record under a reused key. An audit
row’s acknowledgement carries the row’s time as well, so one held past the
row’s retirement cannot remove the row a re-minted attempt writes later.
After the action log write and before the acknowledgement, the owner hands
the applied block_ip rows of each batch to Options.Applied; no other
row reaches it. A failed call acknowledges nothing, so its consumers see the
same row again and must be idempotent by action and transition. A row this
process already wrote to the action log is not written again while its
consumer or acknowledgement fails, even if new rows sort ahead of it into
another batch. Each row’s successful acknowledgement clears only that row’s
write receipt; a restart writes pending rows once more. The daemon opens the
threat and attack databases before the owner starts, and the consumer refuses
rows while either is missing. Delivery
errors remain visible in admission.owner.audit_error until a pass succeeds;
the independent audit queue reports a stall if a pending batch cannot advance.
The daemon marks the address blocked in the attack database, as the legacy
path marks a block that applied, and, while the action’s verified firewall
row still stands, records a threat database row that ends at the block’s
absolute expiry. After that write, a daemon goroutine records those
standing blocks where a legacy block is recorded (checks.RecordAdmissionBlocks):
a tracker row that netblock escalation counts, a permanent-block strike and
the Critical auto_block finding that reaches history, alerts and the block
digest. The in-memory backlog uses the existing audit delivery bound; a
full consumer leaves the durable audit delivery pending for retry. A block
already tracked at its expiry is skipped, so a repeated delivery changes
nothing. The tracker batch is saved before strikes and notices: a crash can
lose those best-effort side effects but cannot repeat them on replay.
An unreadable tracker is left untouched; a failed tracker save emits no
strike or notice. The strike promotes nothing, since a permanent
block is not an admission response. That goroutine waits for the tracker
lock a scan holds; the owner never does. Blocks already queued when the
daemon stops are not recorded there; a delivery after the consumer stops is
refused and stays pending in the audit outbox, so the next start records it. One durable transaction per delivery batch precedes threat
memory publication; read or write errors leave delivery pending and threat
memory unchanged. Local evidence survives feed withdrawal;
permanent local evidence and longer temporary rows remain unchanged.
Local threat writers serialize separately from lookups. The consumer reads
the firewall state under the threat writer lock, and unblocks and flush
cleanup remove the stored row under that lock after the firewall row, so an
unblock that lands during delivery removes what the batch wrote. An applied batch
does not hold the lookup mutex during its durable write, so reputation reads
and whitelist reload continue while that batch waits. Store transaction callbacks do not call back
into threat mutations; loaders finish reading the store before publishing an
instance, and reload changes the existing instance’s config whitelist only.
Repeats preserve the stored timestamp, counters and undo entries, including
after restart. Queue
events and attempt outcomes are also counted into five-minute, hourly and
daily buckets. Status reads every section in one read transaction without
the ledger’s lock or a current clock, each section with its own error, and
the pure doctor rules turn it and the ingress’s own health into fixed rows.
Missing buckets fail the sections that read them; an unreadable database
fails every section. Quiet notice indexes are checked with their records.
Health snapshots own copies of the admission view, and a clean ingress
generation clears the interruption marker.
Schema 6 adds offense episodes; the first open upgrades a schema 1 to 5
ledger in the same transaction, without inventing episodes for candidates
queued before. Each ledger draws episode IDs from its own random nonce and
a counter, so IDs never repeat, even in a ledger created again after loss.
Opening proves every episode row against the candidates it names.
History entries written before episode accounting keep their charges and
encoding at upgrade. New entries require the episode allowance; an older
entry is exempt only while it has no row for its episode.
The ledger, not the caller, assigns each arrival its episode and
generation when it persists the arrival; an arrival that names either is
refused. Responses of every kind at one target share its episode. An
observation made within an hour of the episode’s last one, or while its
work is queued or in flight, joins it; a later one opens the next episode,
and one older than the previous episode’s end, or already accepted by
that episode while its work held it open, is refused as stale. A
degraded clock reading never ends an episode, and an episode ends only
once the ceiling’s elapsed time has passed its end as well: a forward wall
step cannot end it early, and a reboot grants it no time. Each row records
that elapsed end, and its quiet hour runs on elapsed time from the reading
that places the observation, so a report processed late after a step
cannot end it early either; a row written before it existed ends on wall time alone
until its next write, including maintenance that leaves its last
observation unchanged. Existing elapsed ends survive maintenance, and
verified closing replaces the quiet end with the block’s remaining
lifetime, read at the block’s execute reading, so a wall step
between execution and its outcome cannot shorten it. Execution readings
are released when an outcome commits, including startup settlement.
A verified episode keeps that boundary: a later verified block of the
same episode moves neither end. A legacy verified row gains its missing elapsed end from
the original expiry when closed again. An answered block frontier without
candidate references retains that verified boundary on both clocks, using
schema 15; earlier readers refuse it. Its verified predecessor’s retirement collects it
after both ends pass. Address and subnet block histories remain charged
while their episode boundary is still active on either clock. Scheduling
and wake times count that retained history as unavailable. Bounded
cleanup continues past retained histories across ticks, so they cannot
hide later expired history. The first row with a positive elapsed end moves
the ledger to schema 13 on every host, which older builds refuse. Schema 13
records nothing of the live start or the effect records, so a schema 13 ledger
proves them from their own markers. Episode placement refuses an
observation dated after the ledger’s current reading, even within
assessment’s clock-skew tolerance. This keeps future times out of observation
frontiers and prevents an early episode boundary. A queued candidate coalesces
later observations before its first attempt. A queued retry remains live
but answers new arrivals without coalescing; one that ended before any
attempt is followed only by a later observation of that kind. Once a
candidate’s effect is verified or previewed, later observations of its kind
in the episode are refused as an existing effect, which raises no notice,
and still extend the episode. After an unknown outcome they are refused the
same way but never extend the episode, which stays open until recovery
retires the outcome. Other kinds may queue at that target, but scheduling
and admission defer them as pending_recovery until the unknown is retired.
An attempt still recorded as running, left by a crash before settlement or
by an outcome that failed to commit, holds its target the same way until
its outcome is recorded.
The hold also covers candidates from earlier episodes. A missing intent
bucket refuses the operation without changing the ledger. Each handle keeps
the held targets as an index it builds at open from the proven intents and
changes only when a transaction that writes or deletes an intent commits, so
scheduling and reservation look a target up instead of reading every intent.
A final failure or a retired unknown outcome lets a later
observation start the next generation. An episode row lives only as long as a
candidate it names: ring eviction and history retirement clear that
candidate reference but retain its generation, and whether its effect
answers its kind, while another candidate holds the row. The last retained candidate takes the
row, and each admitted
candidate’s history charge covers a row. A target without a row opens a
new episode, so a retired candidate’s ID is never minted again. A
verified block ends its episode at the block’s original expiry instead of
an hour after the last observation, even while work of another kind is
queued or in flight; an unknown outcome or a response of another kind does
not end it. The expiry is set on the engine’s lease clock, which after a
wall clock step back leads admission time; the owner passes that lead to
FinishOnLease or, with a victim, FinishEviction, so the episode ends when
the kernel element expires in admission time. Its elapsed end is the time
left until then, so a forward wall step does not end the episode while the
kernel element remains. Verified outcomes use the wall and elapsed values
of the last committed clock reading, retained together by the ledger
handle and restored at open. An unreadable ceiling or a failed later tick
does not prevent recording a result for work that already ran. New work
still requires a successful reading. An isolated member’s isolation
ends the same way.
Retirement and retry staleness stay on the lease frame: late retirement
only holds an address longer, and the engine refuses an expired member.
On a live ledger, a verified block whose record is gone stops answering its kind before the episode’s original verified end: a later observation of that kind, made after the removal and with every root observed after it, queues a charged new generation in the same episode. On a live ledger the block line records the generation of its latest verified block, whose history outlives later generations, so queued, ended and forgotten returns keep that evidence gate against the latest removal; such lines also need schema 15. Older lines recover that generation from an unbroken chain of retained predecessors on their next arrival and persist it with the next episode write. A gap leaves the line answered. Opening a ledger does not rewrite older lines, but rejects a reference to an unverified candidate or a block older than another retained verified block of the same kind and episode. Forgetting a refused return keeps the verified boundary until both ends pass. The episode earns no equal-tier rotation. An observation or root at or before the removal is answered or stale. Past the original end, as when a reboot keeps the episode open on elapsed time, the block keeps answering until the episode ends on both clocks. The owner caps a re-admission’s first attempt at the episode’s original verified end, the time left in admission time added to the lease clock; a retry keeps its first expiry. A re-admission with no time left, or whose episode row has been replaced or lost, is refused as expired, or as invalid in a preview. A forecast caps a block returning to an address whose forecast block the session replaced at that block’s expiry. It applies the cap before the firewall snapshot and again before deciding each pick, since an earlier pick can replace that address in the same turn. A shortened expiry must still leave the firewall’s minimum timeout; otherwise the return ends uncharged without changing forecast occupancy or retaining an example.
A candidate refused as an existing effect can name the block that enforces
it: the admission action that wrote it, or the firewall’s eff_ name of an
operator or earlier block, with its expiry. TerminateExisting ends a
queued pick that way, including a retry whose earlier attempt never ran;
FinishExisting fails the current attempt, keeping its charge, and ends
the pick linked in the same transaction, or leaves it
failed on its last attempt. A historical result cannot end a later retry;
a repeated linked ending must name the same block and expiry instant. A linked record is candidate version 2 and an
admitted candidate’s history pays for its bytes; without room the
transaction changes nothing. A linked record requires at least schema 8,
with the same buckets; this build opens schemas 7 to 14 without rewriting
them, and older builds refuse an unknown schema rather than misread it.
An unchanged schema-7 ledger stays readable by older builds until a write
needs a newer format. A write never lowers the schema.
A panel verdict that allows a queued candidate ends it refused as
verdict_allow, like any other refusal: no attempt, charge or history
allocation, one ended count, and the notice that Critical or C3 work raises
when it is refused. A retry keeps the charge of its failed attempt. The
episode is not answered, so a later observation can queue the next
generation. Such a record requires at least schema 9, with the same
buckets; a later schema is retained and older readers refuse unknown schemas.
Queued work whose response the configuration no longer allows ends refused
as disabled: no attempt, charge or history allocation, one ended count and
no notice, since the operator turned the response off. The episode is not
answered. The first such record moves the ledger to schema 10, with the same
buckets; older builds refuse it. An ingress checkpoint carrying disabled
refusal counts also requires schema 10, even without a stored candidate.
A full outcome bucket, every key at its largest count, is now 32872 bytes
against a bound of 33 KiB.
EndDryRun ends queued work that dry-run mode forecast instead of running:
the candidate moves to dry_run without an attempt, ceiling charge,
history allocation, audit row or notice. The ending counts one dry_run
outcome in the windows and no queue event. A retry keeps its failed
attempts, their charges and the history they paid for, which ends; roots it
gained while it waited are dropped as on other endings. The work must be
ready as a reservation requires: a retry waits out its backoff, an
unresolved attempt still holds the target, and stale work is refused. The
same ending again changes nothing. The episode is not answered. A dry-run
ending requires at least schema 9 and retains any later schema.
SchedulePreview selects queued work for a dry-run forecast. It is not
admission: spent live ceiling and history budgets do not stop it, and it
reserves, charges and defers nothing for capacity. Every candidate, a
challenge included, weighs one fair unit and no history. Until the ledger
starts live a challenge’s comparison preview still spends live history, so
the selection passes one over, with no wait or fair turn, while the live
history budget cannot pay for it, as a schedule does, and serves it on a
lane that can. Challenge picks share that live allowance within a
selection; an earlier pick leaves less for the next. On a started ledger a
challenge preview spends no history and is selected as other work is.
Direct and
corroborated work alternates within a bounded reserved share of each batch,
and the general lane serves the rest by class and severity, where
reserved-eligible work competes too. Revalidation, age-out, retry backoff
and recovery holds gate it as they gate a schedule, a held target is
deferred as pending_recovery, a stale live ceiling or storage_share
wait clears as a schedule clears it, pending work is passed over, and the
live schedule and the selection share one fair position, which advances
once per selection. PreviewWake is the matching
wake: now while selectable work waits, otherwise a backoff end, a queued
deadline or, until the ledger starts live, when the live history budget
can pay for a passed-over challenge, whatever the other live budgets.
CeilingSnapshot reads the committed
ceiling and its retained charges in one read transaction, proven against
each other, for a forecast to start from.
Preview selection and wake calculations check saved ceiling, storage and live candidate history even though live budgets do not limit them. Damaged accounting refuses the operation without advancing the fair position.
RecordDryRunBlocks writes one turn’s forecast examples, at most one batch,
into the dry_run_blocks bucket the legacy writer also uses. A record keeps
the legacy ip, reason (the registered check) and timeout_sec, and adds
candidate_id, session_id, kind and expires_at; its key is a
fixed-width UTC time with nanoseconds, the canonical address and the
candidate, so two decisions for one target never overwrite each other. The
writer validates every record and the encoded key and value bounds before
writing; one bad record refuses the batch. A full bucket makes room by
evicting earlier sessions’ first keys in key order in the same transaction.
It refuses a batch when no room can be made without evicting its active
session, or while the bucket remains above its cap.
MaintainDryRunBlocks runs one bounded step of a pass over the bucket,
trimming an oversized bucket without evicting the active session,
removing records whose key time is strictly older than a cutoff,
and, for a purge, every record of a session other than the one it keeps,
legacy records included. A purge with no active session removes everything;
a purge with an active session preserves that session regardless of time.
It resumes after the last key it visited, including protected keys, writes
nothing when nothing is due, and a failed step changes nothing.
The forecast model keeps charges in admission-time order, including when
imported legacy spend is dated in the future. Earlier simulated charges can
leave their window while that spend remains. Only block_ip consumes
simulated ceiling credit and temporary address space; other kinds return
not_modelled without changing capacity. These primitives do not activate
owner execution or replace the comparison-only challenge path.
Schema 7 adds execution intents; the first open upgrades a schema 1 to 6
ledger in the same transaction. Execution marks a batch of up to 64
reserved attempts as running and records, in the same transaction and
before any host change, what each member may apply: its kind, its target
and its absolute expiry. Executing the same members again reads them back;
a member that runs in another batch refuses the call. A batch mixing
reserved and running members refuses as a transition conflict, regardless
of member order or target overlap. An applied or failed
outcome removes the member’s intent, while an unknown one keeps it, so a
restart knows which effects are unproven. A reservation holds room in the
recovery reserve for its intent, and an unknown outcome pins the intent
with its history. The upgrade records an intent for any attempt that ran
without an outcome or ended unknown and holds each pinned outcome’s intent
in the reserve, and opening proves every intent against its attempt.
Startup reads the unresolved intents directly, without opening the ledger;
a ledger older than intents has none, and one it cannot read is an error.
Ledger buckets without schema metadata are an error, including unfamiliar
buckets.
An unknown outcome also holds the slot of the row that will retire it.
Once its absolute expiry has passed on a current, trusted clock, its
outcome rows are delivered and the engine finds nothing of it remaining,
recovery retires the uncertainty: the outcome stays unknown and counts as
nothing new, nothing is dispatched again, the intent goes, the history and
intent leave the recovery reserve, and an expired_unproven audit row
records the event. Status counts the attempts that run and the unknown ones
not yet retired, with the earliest expiry after which one can be retired.
A missing or inconsistent intent marks only that section unreadable.
Opening checks retired attempts against the stored clock even after their
retirement events have been acknowledged.
Firewall startup restores temporary blocks only for their remaining lifetime.
Timed elements use the same millisecond encoder in legacy and durable state;
rows with less than a millisecond left or an unrepresentable lifetime are
skipped. Blocks, operator allows and subnets use one startup clock reading.
Durable updates compare absolute expiries before encoding timeouts, so
millisecond rounding does not replace unchanged timed blocks.
Kernel timeouts come from a lease clock that never runs backward: the later
of the wall clock and its last reading advanced by the time elapsed since.
Reapplying the rules after the wall clock is set back restores each timed
block for the time it had left, never for the step on top. Batch checks and
sends read the same clock. Fresh legacy blocks record their deadline on
that clock too. Operator undo retains the original deadline and refreshes
its remaining timeout after persistence and before each bounded retry.
Fresh legacy sends also refresh after persistence. Ruleset installation ages
precomputed block timeouts after table discovery and rule construction.
Durable action planning, verification, sends, and whole-set retries use the
same lease clock for timed blocks. Allow and subnet expiry still use wall
time. The first Apply resumes the checkpoint before durable planning, so
its evidence and the installed timeouts share one clock.
After every ruleset it installs, the engine saves that clock with the boot
it was read in and the time since boot (lease.json beside the state file).
A timed block saves it before the legacy or durable state intent too, so a
new deadline created during a forward clock step cannot gain time after
correction and restart. If that save fails, the new block is refused before
state or kernel mutation. A save failure after ruleset installation warns
without undoing the installed rules; the next Apply retries it. Separate clock samples are
ordered so sampling latency can only shorten restoration.
A new engine resumes from it before restoring timed blocks: within the same
boot at the checkpoint plus the time elapsed since it, whatever the wall
clock did meanwhile; on a new boot at the checkpoint itself, crediting no
downtime. Before the first Apply the daemon also moves the clock forward to
the time the admission clock proves has passed: its saved high-water mark
or, within the same boot, its checkpoint plus the time elapsed since. A
valid but degraded admission clock still supplies that bound and logs the
discrepancy; a damaged checkpoint or an unreadable ledger logs the missing
coverage, and without either the engine starts from the wall clock. Forward
steps can shorten restored leases. After a step back the lease clock stays
ahead unless wall time catches up through a later correction; elapsed time
alone does not close the lead. New rows record expiries that far ahead; the
engine logs the first lead beyond the readback slack once per engine. The
admission owner reads the engine’s lease clock once per batch and sets a
first attempt’s expiry on it, so the lead is not lost from the block’s
lifetime and a short block is not refused as already expired; a retry
keeps the first expiry. The owner already defers while the admission clock
is degraded, the only time the lease could trail it.
The engine also provides internal batch entry points for up to 64 timed
full-address blocks. The admission owner runs them only when the daemon gives
it an engine, and production gives none until cutover. Each member is
checked against one state and kernel snapshot over bounded fresh sockets;
unreadable state refuses the whole call. Expired rows remain available for
kernel cleanup in both the intent and restored state. Rejected members
consume no slot and cannot exclude a valid member. An address counts as
already blocked only when a verified row and its kernel element agree: an
untimed element for a permanent row, or one with the time a timed row has
left, accounting for elapsed time across both set reads and all member checks.
Apply results also age existing-block proofs through the batch’s writes.
A timed proof that can already have expired is not coverage, even if its row
has time left within the readback slack. The row must also last as long as
the member, within the readback slack: a shorter block holds the address
until it ends. Any other row or element holding the address refuses the
member as pending recovery. An existing-effect refusal, at check or apply
time, names the block that holds the address: the admission action that
wrote its row, or eff_ and a digest of an operator or earlier row’s
identity, with its expiry. The same row gets the same name on every check
and after a restart. Set readback proves membership, not enforcement, so a
batch, a member check and a state probe each read the input chain’s rule
handles before the set snapshot, up to the last source-address drop
of a filtered blocked set. A missing or altered family guard, source load,
register flow, lookup or drop verdict refuses them. The check decodes the
raw expression list, so an expression unknown to the nftables decoder cannot
disappear from it; other rules contribute their handles without their
expressions being understood. The batch reads the handles again just
before its write and refuses when they changed, as after a ruleset reload
or a rule inserted
ahead of the drops. That refusal writes and sends nothing and holds
execution until a probe finds the drops again, without an apply fault.
The namespace generation is not read: any other nftables user changes it.
The batch rechecks, writes one
intent with expiry instants stored in UTC, refreshes timeouts after the
write, and uses one atomic transaction
on a fresh socket with a two-second deadline. Exact-family readback proves
each applied member’s key and timed lifetime. Capacity is checked only at
apply: a check before reservation never answers set_full. Victim checks require
retained rows at the address to agree on action, expiry and verification.
An unreadable startup ledger disqualifies all victims; members can still use
free slots. The admission snapshot lists at most 256 imports, soonest
expiry first, then name: timed, verified rows a legacy automatic block
stamped, each alone at its kernel address with a proven element and named
by its raw row’s eff_ name. Mapped addresses share the canonical address;
a retained zoned alias also disqualifies an import. A lone zoned row stays
unlisted and unclaimed. The imported occupancy count still includes
unlisted stamped occupants, so a bounded victim list cannot free capacity.
A member whose
victim, the admission block or import the ledger claimed, still has that row, name,
expiry and proven element takes the victim’s slot if the resulting set fits the current limit: the same
transaction deletes the victim’s element, and only a readback of the victim’s set
proves it removed; a member whose removal is unproven is unknown, even
with its own add read back. A victim that no longer qualifies is spared
and its member needs a free slot. A victim with no more than the batch’s
minimum lifetime left at the send is spared and its member fails set_full
alone, since its expiry in the kernel would reject the whole transaction.
A victim removed by other means before the send rejects the whole
transaction with ENOENT, which fails its members like any rejection. The victim’s row stays until the owner retires it
after the ledger records the outcome, so a restart before then restores
the victim. Retirement removes an evicted import’s row by its eff_ name
only while the stamped row still matches it. An operator or unstamped row
is never retired under an import name. Definitive kernel rejection
restores state; uncertain exchanges retain unknown rows. Unknown rows are
marked unverified and excluded from counts and seeding. Expiry retains them
until the owner retires the matching attempt. A failed state
write is returned as a state-integrity error and retains live suppression
until a successful state write; later batches refuse while that error is
held. The admission owner must reconcile action-linked rows before startup
or any downgrade to a build that ignores their verification fields.
The retry marker for a kernel rejection requires a successful state restore.
Marking with no action IDs still writes the safe state to release a state hold.
RemoveBlockActionRows removes, in one state write, the rows the ledger has
settled (failed, retired after expiry, or evicted with proven removal).
An error reports no completed removal, including a durability failure after
rename. Startup calls it after marking every unproven row, with the settled subset
store.ReadSettledBlockActions returns, so a start that stops between the two
leaves the rows held back and the next start removes them: it asks the ledger
about every saved action, including rows an earlier start held back and
other actions sharing their addresses. An expired row no start marked is never
restored and is left to expiry, so it holds back no other block at its
address. The engine selects those rows on its lease clock, retaining that
reading through Apply so a backward wall-clock step cannot restore an
unchecked action. Marking preserves those separate rows until cleanup can
remove only the settled actions. Removing a hold therefore cannot restore
an unchecked action or erase another block.
Until a restart proves the saved actions, the engine removes no block for
a batch and its admission snapshot says so; the owner then plans no
eviction or rotation, a dry run forecasts none, and higher work waits for
a free slot.
The engine’s kill-point tests stop a batch around its intent write, kernel
batch, readback, unknown marking and rejected-state restoration, then start
the ledger and a new engine over the same state, after a process restart or
after a reboot that empties the kernel. Around an eviction they also stop
before the outcome and before the victim’s row is retired: until the ledger
records the eviction a restart restores the victim and holds the block
back. After a committed eviction an admission victim’s row goes and only
the block is restored. An import has no startup cleanup record: after a
committed eviction, a restart before row retirement restores both blocks
until the import’s original expiry, including after a reboot. An unreadable ledger restores
unexpired unmarked rows with degraded coverage; a readable ledger keeps a
committed applied outcome and holds the rest. Attempts without a committed
outcome are unknown. Their rows are held back, a new block of an address a
held row names waits, and once the expiry passes retirement removes the
rows and the ledger retires the attempts. A new boot retains the saved
lease floor. The test kernel reads emitted transactions, including set
families, timeouts and base chains, instead of assuming a successful send
installed the requested effects. It rejects incomplete batches and wrong
set destinations before modeling loss. Mapped IPv6 keys never become IPv4
blocks. In a real kernel, a batch killed after the kernel took it leaves
its element until a new engine’s Apply replaces the table without the
marked row.
An unconfirmed removal is written again on retry even if its rows already
appear absent. Cleanup preserves other actions’ rows, including distinct
actions sharing one address.
RetireBlockActions checks the kernel for up to 64 unknown attempts under one
engine lock: an attempt is clear when its address has no element, or the
element matches a verified row of another action, remaining lifetime included,
and its own row then goes in one write. An element still present keeps it
held. Retirement requires a kernel snapshot of the target’s address family;
a missing or disabled IPv6 set cannot prove absence, so it holds that
attempt while the rest of the batch is answered. Cleanup resumes and
saves the lease checkpoint first; an unreadable clock or checkpoint
authorizes no removal. Address matching for recovery
accepts legacy mapped, uppercase and zoned spellings within the engine’s
namespace; this does not relax the admission parser.
Unverified rows cannot suppress alerts or reputation checks through stale legacy
tracker entries. They do not count as enforced blocks or select threat evidence
for operator flush cleanup. Live deny limits count their kernel elements using
kernel timeouts rather than the unresolved rows’ requested lifetimes, including
replacement accounting. Permanent promotion requires a verified block row.
A row an admission action wrote belongs to the owner: an automatic legacy
block of its address ends as already blocked, even when the row is unproven
or its element is gone, never evicts it at the temporary cap, and permanent
promotion refuses it (ErrAdmissionOwnsBlock). Operator blocks, unblocks,
allows, flushes and undo keep their authority over it. The guard also
protects action rows already present in restored state.
An executed admission block writes no legacy tracker (blocked_ips.json)
entry. Reputation, alert suppression and its injected loader, flush cleanup,
threat lookups and the blocked list read the verified firewall row instead,
so it counts without one.
A read-only admission snapshot answers the same local checks for a bounded batch, each answer naming its member, with the temporary occupancy and limit from the same snapshot. No member takes a slot and no repeat is a duplicate there: the caller checks only the members it is ready to run, in order, for repeats and capacity. The snapshot also counts which of the bounded forecast addresses the sets already hold as a timed or an untimed element. A malformed or repeated forecast address refuses the call before any read, and other platforms report the snapshot unavailable.
With an engine, each owner turn runs its block_ip picks as one batch. The
engine checks every member against one snapshot before anything is reserved. An
address a row or element holds without proof defers a member;
a block the engine proves already holds the address ends it linked to that
block, and an invalid or protected address or a spent lifetime ends it, all
without a charge. If the ledger refuses that link (no history budget for
it), the pick stays queued with the deferral reason and the rest of the
batch runs; the next check tries again. Each member carries the victim its
reservation claimed, as its intent names it, and the ledger records what the
batch did to that victim with the member’s outcome. Whole-batch failures,
short replies and delayed outcome writes keep the victim and lease clock
information. Once a verified eviction is recorded, the owner marks an
admission victim’s row and retires it, as startup
does, rotating failed or held rows behind waiting rows and retrying on
later turns, including while waiting for an absent engine during dry run.
Cleanup follows the retirement policy, respects state-repair probe waits
and preserves independent apply faults. Its errors remain
visible alongside unknown-attempt retirement errors; a restart
removes the row before the firewall restores its state. The turn hands the
ledger the imports the set read listed. Clearing that set also clears its
imports, including a failed read, an idle turn or entry into dry run.
Returning to previews reads a fresh set and import list. The dry-run
forecast reads the list from its own snapshot and may replace a listed
import, which it remembers by name until the import’s expiry; it never
claims one.
An import’s row is retired by its
eff_ name without a mark. An import-only retirement still retries a
shared state repair after its probe wait, without marking any import or
clearing an independent apply fault, including during dry run.
An import still enforced at retirement (a reload or restart restored its
row) is forgotten. An error or short retirement answer
keeps the request for retry; no ledger record names an evicted
import, so a restart before its retirement restores it until its own
expiry. After an unknown
outcome the victim’s row stays until the victim expires. A proven block
found only when the batch applies is checked again when the outcome is written.
With a fresh proof the pick ends refused as an existing effect, linked to
the block, keeping its charge. A member with a victim records its eviction
outcome before attempting that link; a member without one records its
failure and link in one write. Without a fresh proof, or if the link write
fails, the attempt still records its failure. If the outcome cannot be
saved, execution waits for the pending write; a crash before it is saved
leaves an unknown attempt. After a successful write the ordinary bounded
retry or the next check decides. On the
last attempt the pick already ends failed. The rest are reserved,
recorded as running with their intents, applied and finished by their own
effect. Failed outcome writes retry before more work is selected and once at
shutdown, even without a fresh clock reading; the effect is never dispatched
again for that attempt. An outcome the ledger refuses for good (a damaged
record or a conflicting state) is not retried: its attempt stays unresolved
until recovery. Each turn, before scheduling, the owner retires unknown
outcomes whose expiry has passed: it asks the engine, for up to 64 at a time,
whether anything of them remains (RetireBlockActions); running attempts are
never sent. The engine removes each clear attempt’s own row, and the ledger
then retires it (RetireUnproven). An element still present, as after a
forward wall step, keeps that attempt for a minute on its own wait, so a held
batch cannot starve the intents behind it. Each checked batch advances through
action order, wrapping around, even when its retries are due by the next turn.
A degraded clock, a missing engine or an engine that cannot check retires
nothing; the next pass waits one second, doubling to a minute. An attempt
whose retirement the ledger refuses (undelivered outcomes, or a ledger clock
that cannot yet prove the expiry) is asked again after one second, doubling
to a minute. A damaged record or an attempt that ended another way is asked
again a minute later; any other fault, such as a failed write, retries that
attempt after one second, doubling to a minute. A
missing or unreadable attempt uses the same wait without preventing other
attempts from retiring. An unreadable intent list delays the whole pass.
Each fault is reported once, in admission.owner.retirement_error and the
doctor row admission retirement, until no attempt, list read or engine
retirement check is left faulted; no retirement fault fails a turn. An
owner without the firewall engine, or under a refused execution policy,
retires nothing and keeps each unknown’s address held past its expiry:
admission.owner.retirement_held_by names that cause (engine or policy)
while unknown outcomes remain, and the doctor row admission unresolved outcomes names it with its fix instead of a retire time. Policy changes
publish the retirement cause with the execution hold, preserving the
cached ledger sample and its age. Once the cause
is gone, each unknown still retires only after its expiry and the engine’s
proof under a valid clock. A configured engine that has not started names
no cause: retirement retries on its waits and the firewall reports its own
startup error. A stop
between the row removal and the ledger retirement leaves a
row-less unknown that the next pass retires. The turn wakes for the earliest
expiry or monotonic retry. With one noted, a turn lists no intents until it
is due; with none noted, every turn lists them. A turn that fails after its
retirement pass fails with its own step’s error. It never rebases a retry on
admission time or adds time spent inside
an engine call. Expiry conversion uses a monotonic sample taken before reading
the admission clock, so a slow tick or later call cannot postpone another
expiry. Running attempts create no retirement wake; retired attempts leave no
retry entry.
Every other kind is still previewed. A batch the kernel
rejected fails every member, charged; each then retries alone, one per turn,
until its next outcome or its expiry. A state write the engine could not make,
a failed batch check or apply (including per-member transport and readback
errors), or a missing engine holds automatic execution: block picks wait,
deferred as an unavailable engine, while one probe runs per wait, from one
second doubling to a minute. A held state write is probed by writing the safe
state again and checking its snapshot. Without an engine, idle scheduling
pauses and resumes on the first schedule check that finds an engine, without
waiting for the old probe deadline. Other idle holds still probe on their
waits; successful snapshots cannot reset a failing member check’s backoff. A
failed apply keeps its cause and backoff through state repair, ledger refusals
and member refusals until an applied effect proves recovery. Its probe is one
member: every pick is checked and refusals end, but only the first accepted
pick is reserved and applied, and the rest wait untouched for the batch after
it applies. An isolated retry does not narrow these probe checks; it still
runs alone if selected. An apply-confirmed firewall window holds the engine’s member
checks, batches, probes and retirements (ErrConfirmPending) from before its
snapshot until it is confirmed or rolled back, so its rollback cannot erase a
block the owner wrote inside it; a batch in flight finishes first and lands in
the snapshot. A rollback with saved block state restores it and
then gates the rows on the ledger again, as startup does, before the hold
ends: a mark or removal the ledger asked for inside the window is not lost
with the restore. The restored ruleset also gives each timed block the time
it had left when the window opened. When the hold ends, RetimeBlocks
uses one strictly read row image and replaces the blocked sets in one
transaction. Normal rollback restores configuration first; explicit
confirmation accepts the current configuration. If startup with IPv6 disabled
left no blocked-set reference, retiming binds the restored IPv6 set after a
successful transaction. An unreadable image sends nothing, even with a cache
or a successfully preserved corrupt state file.
A retiming failure warns and releases the hold. A restart with a window
pending holds the same checks and batches before the owner sees the engine,
until startup recovery settles the window. A window opened between
member checks and apply also holds execution, but is not an apply fault:
after it ends, a successful snapshot probe clears that hold even on an idle
turn. Any earlier apply fault still needs a successful one-member apply.
The read-only
admission snapshot of a forecast still answers. A kernel rollback alone
does not release the hold when the saved engine configuration is unreadable;
the confirmation remains pending until recovery succeeds or the operator
confirms it. Keep its state snapshot through partial recovery, so another
rollback restores the same state even after intervening legacy writes.
Failed marker removal keeps the hold. Once the marker is gone,
the hold and recovery fault clear even if snapshot cleanup fails; that error
is still reported. Hold changes copy
the cached owner status without rereading the ledger. Status
(admission.owner.execution_held) and the csm doctor row admission execution name the cause. An owner publishes admission.owner.executing
only while it runs block picks: it has Options.Blocks and its policy is
not a dry run. admission.DoctorImpact then closes every
admission row with ExecutionAffected instead of PreviewUnaffected and
turns the outages that stop automatic blocks, held execution included, from
warnings into failures.
A degraded admission clock cannot bound a lease, so
it defers block picks the same way. One batch starts a second at most; with an
engine, a drain that queues new work runs the schedule at once instead of
waiting for its timer. An owner with an engine also selects, checks or
forecasts work once a second at most, including turns that dispatch
nothing: work that falls due sooner, from a drain or a panel answer, stays
due for the next second, while retirement of earlier attempts keeps its own
waits. Before the next live batch may start, a turn selects nothing and
leaves the next selection due. While execution is held, a turn before its
probe is due selects nothing either, so held work keeps its place in the
rotation; probe turns still select. A missing engine has no probe: its
hold still selects once per wait, from one second doubling to a minute,
including during a clock outage, so work that needs no engine is still
previewed. A returning engine selects immediately even after a recent
missing-engine preview; batch starts still keep their spacing, and an
earlier apply fault still requires a one-member probe.
Clock or engine outages still defer waiting
blocks without spending fair turns or running early probes; work already
deferred for capacity keeps its reason. Dry-run
forecasts and previews without an engine keep their own cadence.
The policy installed at the turn boundary decides whether the batch wait
applies; changing between live execution and dry run does not spend a turn
that cannot select work. During a batch wait, opt-outs end silently before
any outage is reported. Outage deferrals revalidate waiting work first, so
expired evidence and changed ownership keep their own terminal reasons.
The first time an owner executes, its live start runs once per ledger, in one transaction, before
anything is drained or served under that policy: every drain and turn reads
the clock first, and the reading runs the live start. It charges the legacy
hour as the greater of its count and the charges the ledger already retains
in that hour, since previews and legacy blocking count the same responses;
earlier preview charges stay as handover charges with their lane, cost and
release times, so a later forecast cannot refund them. Pending legacy
entries stay legacy’s to retry and count nothing: a finding admission takes
never enters that queue. Block work the preview period left queued, deferred
or not, ends as disabled with no new attempt, charge or notice: legacy
blocking answered it. Work still in the ingress drains as live work; the
engine’s one row per address ends any whose block legacy applied as an
existing effect. It ends the episode answers of earlier
observe previews, so the next observation of a previewed address queues a
new generation, or a new episode when no other candidate kept the old one;
challenge previews and verified blocks keep their answers. It clears the
notice records raised while admission previewed, so none goes out as a live
alert or delays a key’s first live notice. The sender discards the preview
delivery retry delay once it sees that committed start; later live delivery
failures keep their own backoff. The start also makes the previews’ history
eligible at once. Until it runs, an executing owner delivers no ledger
notice; with execution enabled the sender reads the mode and the due notices
together, and an acknowledgement taken before the live start never covers a
record raised after it. It raises the schema to 11, which every-host
schemas 13 and later already exceed, and records the live feature bit in the
ledger’s meta bucket; every earlier build refuses that unknown key, so a
binary rolled back across the live start does not open the ledger and keeps
legacy-only blocking. A feature bit this build does not know is refused as a
newer schema. It leaves blocked_ips.json in place. A legacy state file it cannot read
keeps no saved credit, and admission.owner.import.error names the error.
A failed live start fails the turn, reload or drain that ran it, and a
failed tick or drain closes admission. A failed live-start tick reports its
cause through tick_error and the stopped-admission notice until a later
tick succeeds. The import error describes a committed handover: an aborted
transaction does not replace it, and later startup retries do not erase it.
After the live start a preview spends nothing that live work lives on. The
owner previews what it does not run: every kind but block_ip, and every
pick of an owner without an engine. Such a preview ends as a dry run does,
with no attempt, slot, ceiling charge or history, and it leaves its episode
open, so the next observation is previewed again; the comparison still
counts it as observed. A refused preview counts in the hourly comparison
only after its terminal ending commits, in the hour of that ending.
Deferrals and failed endings add no refusal count. A live pick the
engine’s check or its reservation refuses is counted the same way; one an
existing block answers is not a refusal, and a forecast’s picks keep the
rows a dry run already writes.
The schedule serves those kinds as uncharged picks,
so they take no lane’s ceiling budget or history allowance from the blocks
that run. A dry-run selection on a started ledger no longer waits for
history a challenge preview does not spend, and an owner without an
engine selects its previews as a dry run does, outside the lanes, and
wakes for them at once.
An owner with an engine binds to one execution policy: the dry-run mode,
the verdict callback and its timeout, the hourly ceiling and the response
lifetimes, read together from one config publication. Every publication
that changes auto_response or the mode gets a new epoch. The owner does
not start without a valid policy; it names the refusal instead, even with
the callback off, and mode: observe is refused. Each turn and reload
installs a newer epoch before it selects work, and the turn uses that
policy for every pick; a reload returns once the policy is installed, after
any turn already running. An older epoch is ignored. Changed settings or
missing required dependencies under an installed epoch hold execution, as
does an invalid newer policy, with the refusal in execution_held, until a
valid newer epoch arrives; the daemon reports a refusal at reload as a
config_reload_error finding. While a refusal holds execution, a selected
pick of any firewall kind is deferred as engine_unavailable with nothing
previewed, reserved or charged, and the schedule then passes it over, as it
passes over work waiting for the panel, until a valid policy is installed;
a refused policy also suspends engine probes and repairs. The schedule
checks for newer epochs even while idle, so a publication without a reload
resumes held work and any probes suspended by dry-run. A challenge keeps
its comparison preview. A comparison-only owner never reads the policy.
The policy also carries whether the configuration still allows each queued
response: checks.AutomaticResponseAllowed, the switches the legacy funnels
test before they act (auto_response.enabled and block_ips for every
firewall kind, block_cpanel_logins for the checks that need it, and each
derived entry’s own opt-in; challenges follow challenge routing). Under a
valid policy, a pick it no longer allows ends refused as disabled before
any panel ask, engine check, forecast, preview or reservation, live or dry
run. The test reads the configuration live at each turn, not the epoch’s:
challenge, incident, central and per-check settings can change without a
new epoch, and a saved change applies from the next turn. A queued block of
an address the live configuration now names as a trusted proxy, or of a
known CDN or website-firewall edge that one of its roots may have seen
relaying, also ends as disabled: the mint would refuse it now. Roots are
read only for an edge address.
Disabled responses end together before capacity selection, including challenge previews. Spent ceiling or history budgets, retry backoff and another attempt’s recovery hold do not delay their refusal or generate capacity notices. The check visits only the bounded durable queue and leaves allowed work’s scheduling turns and capacity checks intact. Refusing queued work never releases an earlier attempt’s charge or recovery hold. Startup and ceiling reload also refuse disabled work before inventory or queue revalidation can report expired evidence as a withheld response.
Under a dry-run policy the owner forecasts instead of running. It selects
work with SchedulePreview, whatever the live budgets, and ends each pick
as a dry run with a prediction: no attempt, charge, history, audit row or
mutating engine call. A forecast session starts at startup or on entry into
dry-run from the committed ceiling and its retained charges
(ForecastSeed), then lives in memory apart from it: the same committed
clock meters it, a new limit applies without new credit, and live credit is
never copied over it. The seed leaves out the charges of observe previews,
which are comparison work, not responses: their units leave the window and
their credit returns to its lane up to the bucket’s cap (Uncharge);
legacy and genuine charges stay. Ceiling, charges and keyed attempts come
from one read-only transaction. Each retained attempt’s charge must match
its reservation time, lane and response cost; damaged records refuse the
seed without changing committed state. While the owner forecasts, the
ledger also stops counting a preview as the answer of its block line
(PreviewsAnswer), so an address previewed before the session queues its
next generation on a new observation and is forecast; a verified block,
or its remembered answer, still answers until its expiry, and challenge
lines keep following their previews. Leaving dry run turns previews back
on. One
read-only admission snapshot answers a turn’s block_ip picks, with the
real temporary occupancy and the real blocks of forecast addresses. A
refused target ends or waits as a live check has it. A real block that
covers the address predicts existing_actual, unlinked; a forecast block
that lasts as long, within the readback slack a real block is allowed,
predicts existing_shadow and a shorter one pending_recovery, and
neither is renewed. Other picks are checked against
the modelled temporary capacity, the bound on forecast blocks and the
modelled ceiling, and only capacity_admit, capacity_evict and
capacity_rotate add a forecast block and a simulated charge. The model
budgets slots with the ledger’s rule (SetOccupancy.Limits): with no free
slot a pick may take the first strictly lower-tier forecast block or listed
import in eviction order and predicts capacity_evict. Forecast blocks
rank by their pick’s tier and next attempt; one without a tier is never
replaced. Each records its pick’s scope and the admission time and session
elapsed time of its decision. Once the general partition is full, the
session shares each partition as a live ledger does: among the scopes of
its forecast blocks the real sets do not hold, the scopes of the turn’s
block picks and the requester’s, from session cursors that each such turn
moves once before its first pick. A fresh pick may then rotate out a
forecast block of exactly its tier by the same rule as the ledger, once
that block’s hold passed on the admission clock and the session’s metered
elapsed time, and predicts capacity_rotate. Fresh means the ledger’s
episode row says so (AdmissionLedger.Fresh, which needs no live start)
and no forecast block of the address left the model before that block’s
expiry: a replaced or rotated forecast block leaves the model and answers
its address until then (spec 5.2). Active forecast blocks and retained
answers share the existing modelling bound; when it fills, an otherwise
funded pick predicts forecast_limited. Answers expire only at their
recorded boundary, never to free space for new predictions. A pick that finds no room while a
lane’s partition holds a scoped block of exactly its tier at another
address predicts equal_tier_wait; other full sets predict set_full.
Real admission blocks count in occupancy but in no scope. A replaced
import is remembered by name until its expiry and subtracted only while
the engine still lists it.
With the verdict callback on, a pick the model would
judge is asked about first and waits for its answer with no forecast
change; a usable allow ends it as verdict_allow before any modelled
capacity is judged, and a covering forecast block, or a full modelled set
with nothing lower to replace and nothing to rotate, ends it without a call. Other firewall kinds end as not_modelled, with no
call; a challenge keeps its comparison preview. Without an engine or
snapshot, or with a degraded clock, picks wait as engine_unavailable and
the forecast does not change. A decision is installed only once its ending
commits, and only then counted in the hourly comparison summaries as a
dry_run decision with its prediction, a field apart from the refusal
reason. The prediction names live in the admission package, which the
finding stream tool reads too. A forecast neither probes nor repairs the engine: a hold from live
work stays in execution_held until the next live turn. A live policy ends
the session and never replays a dry-run ending; a later observation of the
target is a new generation. A session whose model cannot be metered by a
committed clock reading or a new limit ends too: the forecast error is
model_error until the next session starts, and that session names
model_error as its reset rather than a mode change.
Each committed capacity_admit, capacity_evict or capacity_rotate leaves one example in
dry_run_blocks. A
turn writes its examples in one batch after its endings, also when a later
pick stops the turn; a failed write loses them without undoing the endings
or the forecast, and nothing retries it. An owner that runs blocks keeps the
bucket alone: one bounded maintenance step a second trims it to its cap and
removes examples older than a week. Installing a live policy after a dry run,
or at startup, asks for a purge of every example but the active session’s;
the purge pass starts over whenever the active session changes, so a dry run
that resumes keeps its own examples whatever their times, and the examples
are stale until a purge pass completes. The daemon’s legacy prune and live
purge leave the bucket to such an owner.
Such an owner reports its forecast as owner.dry_run in status: whether it
forecasts, the session and why it started, the model’s coverage, real
against predicted temporary occupancy from the turn’s snapshot, the
forecast’s own ceiling, each prediction’s session and last-hour counts, lost
examples and summaries, failed writes, a pending purge, and a fixed code
for why forecasting last failed. A turn that succeeds clears the code; a new
session clears its time too. Leaving dry run clears the forecast and keeps
the last session’s identity; a dry run whose session has not started shows
none. The loss counters cover the process.
csm doctor renders it in three rows that warn and never fail.
The forecast’s committed predictions are also summarized in history, never
sent to an alert channel: one auto_block Warning, “Dry-run capacity
forecast: N responses passed modelled capacity checks”, with the other
predictions’ counts, its interval, a session count and a bounded list of
distinct examples in committed order, each naming its candidate, finding,
target and check. The first summary is due a minute after the first
prediction it counts, and each later one a minute after the previous
attempt, a failed one included. One summary is pending at a time: the
sender takes it whole, later predictions start the next one, and a failed
delivery goes back in front of them as one summary, retried with the notice
backoff and counted in the notice queue while it waits. A mode change never
flushes or relabels a pending summary; Stop sends one its minute allows and
counts any other as lost. Executable owners supply a cancellation-aware,
history-only summary sink separately from legacy notice delivery. An error
must mean the summary was not committed. Stop cancels an active delivery,
settles its receipt, and bounds the final attempt outside the owner
goroutine. Failed or cancelled summaries remain counted until delivered or
reported lost. The status getter returns a deep copy of its cached snapshot.
With the policy’s verdict callback on, a live owner asks the panel about
each block_ip pick before anything is reserved, as a dry run does before
it forecasts one. One read-only admission
snapshot answers the turn’s local checks without giving any pick a slot:
a refused, protected or already enforced target ends or waits as a batch
check would have it, all without a call. Room is the ledger’s: it
schedules a pick only on a free slot or a victim it can claim.
Any other pick is asked about, as CSM auto-block: <check> for its
canonical address, and stays queued; the schedule passes it over until
the answer arrives. Workers, outstanding jobs and cached answers are bounded;
a full stage leaves picks queued and is never read as an answer. Each answer
is keyed by candidate, attempt and policy epoch and expires from completion,
including time awaiting receipt. Admission uses the configured signature
and replay validation, including explicit signing exceptions, through the
same client as legacy blocking. A valid allow ends the pick as one
verdict_allow refusal with no attempt or charge,
even with the ceiling spent: the schedule picks it as uncharged work.
Block, an empty verdict or a failed exchange continues the local path,
and the reservation consumes the answer. Local checks run again before an
answer is used. A pick that stops being eligible has its call cancelled.
A new epoch, a mode change included, cancels every call, drops every
answer and starts new workers only after the old ones return; Stop
cancels the calls and joins the workers. A cancellation is never an
answer. The owner status carries a verdict object for an owner with an
engine provider: stage gauges, process counters and the last failure as a fixed
class, never response text. Its outstanding jobs are also the
admission.verdicts queue row. Workers count completed calls before delivery;
late or duplicate results cannot count another call or consume a different job.
The admission ledger exposes two row-recovery reads. ReadSettledBlockActions
names failed attempts, retired unknown outcomes and verified blocks later
evicted by a proven removal among the supplied action IDs.
ReadUnprovenBlockActions includes every non-verified attempt and those
evicted blocks. UnresolvedIntents gives the owner the same ordered intents
as the handle-free startup reader. Callers must inspect each attempt’s outcome:
an executing intent is unresolved but cannot be retired as unknown.
Attack event storage
Attack events live in attacks:events; attacks:events:ip stores empty values
under <ip>/<TimeKey> keys. The writer chooses an unused primary key inside
the write transaction because batch counters can repeat for the same timestamp.
Primary rows, index entries and the event count are updated atomically, including
count-cap pruning.
Address queries walk the index newest-first and resolve primary rows until the requested limit is met. Older index entries can still contain a full event copy; the reader falls back to that copy if the primary row is missing, malformed or belongs to another address. Both forms must match the requested address. The UTC time-key migration preserves index values verbatim, so readers must keep handling both forms until older entries age out.
Structured Logging (slog)
Legacy daemon call sites emit log lines via fmt.Fprintf(os.Stderr, "[%s] ...", ts()). The internal/log package provides a drop-in slog wrapper so operators can opt into JSON output for log-shipping pipelines (Loki, ELK, Datadog) without a big bang migration.
Operator controls
Two environment variables, read once at daemon startup:
| Variable | Values | Default | Effect |
|---|---|---|---|
CSM_LOG_FORMAT | text, json | text | Output handler |
CSM_LOG_LEVEL | debug, info, warn, error | info | Minimum log level |
Set via systemd drop-in:
# /etc/systemd/system/csm.service.d/logging.conf
[Service]
Environment="CSM_LOG_FORMAT=json"
Environment="CSM_LOG_LEVEL=info"
Then systemctl daemon-reload && systemctl restart csm.
Writing new logging code
import csmlog "github.com/pidginhost/csm/internal/log"
csmlog.Info("scan complete", "findings", len(f), "duration_ms", d.Milliseconds())
csmlog.Warn("log not found, will retry", "path", path, "retry_in", "60s")
csmlog.Error("alert dispatch failed", "err", err, "channel", "email")
Keys should be snake_case. Values should be machine-parseable (numbers, strings, booleans) – avoid formatted strings when you can pass the raw value. Pass paths and errors as they are: the text handler quotes and escapes any value with whitespace, a quote or a control character, so a value quoted beforehand is escaped twice.
Migrating legacy call sites
Migration is incremental and optional. The legacy format stays valid. Start with the hottest subsystems (alert dispatch, firewall operations, WAF handlers) where structured fields provide the most value, then work outward. Do not batch-convert – each subsystem should get a dedicated commit with before/after log samples in the PR description.
Keep the [TIMESTAMP] prefix of journalctl lines readable by humans: slog’s text handler uses time=... level=... msg=... which is also human-parseable, so journalctl viewers still work.
Worker processes
Code that parses attacker-controlled input and can loop, crash or exhaust
memory runs in a child process under internal/workerproc. The daemon can
kill a child; it cannot stop a stuck goroutine.
The launcher gives the child one Unix socket as both fd 0 and fd 1. A frame
is a 4-byte big-endian length followed by a JSON envelope, and it may carry
descriptors. The length is checked against a cap before anything is
allocated. Passing a descriptor never calls Fd(), so the caller’s pipe keeps
its nonblocking mode and deadlines. Request descriptors are duplicated before
launch, and the caller may close its files once a call returns.
A request that misses its deadline kills and reaps the worker, and that
worker is never used again. Any other round-trip failure does the same, and
its error matches workerproc.ErrWorkerFailed with its text unchanged. Three
failures in a row pause spawning for 60 seconds, then one trial request is
let through. Each supervisor publishes a requests queue-health row.
workerproc.FailureReason classifies counted failures as timeout,
protocol, crash, oom, or spawn, including through wrapped errors.
Malformed frames and rejected descriptors are protocol failures; a broken
socket is a crash unless the process reports an OOM kill. The daemon cannot
read a unit worker’s kill from the unit’s own cgroup: by the time it sees the
worker die, systemd has removed that cgroup and its counters. On cgroup
v2 each supervisor uses its own slice, and the slice’s memory.events,
which counts its children’s kills and outlives them, is compared with its
value at start. Before that snapshot, startup waits for any previous unit’s
cgroup to disappear, bounded by StartTimeout: a reaped systemd-run client
alone does not prove its service has exited. Standalone CLI scans include
their process ID in the prefix to keep their slices distinct.
On v1 that counter is not hierarchical, so an eventfd on
memory.oom_control reports the kill as it happens; OOM and removal
notifications are counted together even when they arrive in separate reads,
and a single notification with the directory gone remains undecided until the
watch closes after the client’s exit. Diagnostic text
and underlying errors remain unchanged. A failed start matches ErrSpawn,
not ErrWorkerFailed.
Config.OnFailure receives each counted failure while the supervisor lock
is held. The callback must return promptly and must not call back into the
supervisor. Stats.Kills counts these failures by reason, including failed
starts, and returns an independent snapshot. Cancellation, stopped or
breaker refusals, and local descriptor exhaustion before the round trip
do not reach the hook or these counters.
Config.OnJob receives one outcome for every request that reached the
breaker: ok with the worker’s memory use after the job (0 when unknown),
a failure reason, or refused when the breaker turned the request away. It
runs under the same lock and rules as OnFailure. Cancellation, a stopped
supervisor and local descriptor exhaustion are not reported. The daemon
feeds it into the csm_parse_worker_* metrics.
Some errors are not worker failures and never count toward the breaker:
- caller cancellation, including during launch; a canceled trial restarts the breaker cooldown;
- the daemon running out of descriptors (EMFILE or ENFILE) while duplicating request descriptors or starting a child. The request counts as lost work, but the healthy worker stays. Outside load can push the daemon to its descriptor limit, and that must not switch a worker off;
- PHP parser panics reported in a valid reply, which count as lost requests but do not replace the worker.
Config.BreakOnSpawnOnly narrows the breaker to failed starts. The mail
parse workers set it: a refused extraction delivers mail unscanned under
fail_mode: open, so crafted mail that kills workers must not open the
breaker for unrelated messages. Each message then costs at most one spawn.
Every successful start resets the failed-start count, even if that worker
then fails on its input or the caller cancels the request.
A known limit: when the daemon is at its descriptor limit, the kernel drops descriptors a reply carries. That looks the same as a worker sending too many, so it counts as a worker failure, classified as a crash rather than a protocol violation.
Config.MaxJobs retires a worker after that many requests, and
Process.Retire lets a launcher retire one after a request, for example over
a memory threshold. Config.IdleTimeout stops a worker left unused; the idle
countdown restarts only after a served request and never stops a worker in
the middle of one. Retirement keeps the reply and is not a failure. Both
settings default to zero; the PHP taint worker leaves them at those defaults.
workerproc.Handshake sends hello and checks the child’s protocol version
before any real request. It honors cancellation and the context deadline,
refuses reply descriptors and clears the socket deadline before returning.
workerproc.Strikes remembers inputs that made a worker fail. Two failures
within 24 hours block an input, for at most 1,024 inputs; NewStrikesWith
sets another limit and window, each failure within the window of the one
before. The phishing check keys it by the zip’s SHA-256 and reports a blocked
zip with the phishing_kit_archive_unscannable warning instead of retrying
it on every scan. The spool watcher keys it by Exim message ID, and counts
kills per message source with a limit of three in ten minutes.
Worker diagnostics must go to stderr; stdout carries the protocol socket. The
PHP taint worker (csm phptaint-worker) runs through UnitLauncher as
csm-phptaint-worker-<generation>, with the parse worker’s lockdown and a
512 MiB memory cap. The YARA workers run through it too; the section below
describes them.
Parse worker units
The archive parse worker (csm parse-worker) runs through UnitLauncher:
each worker is a transient systemd unit started with systemd-run --pipe,
named csm-parse-worker-<slot>-<generation>, with the properties listed in
service confinement. There is no fallback to
running it directly. The daemon installs one lazy zip lister, using slot
zip0, before scans start. It publishes archive_list.requests and is
stopped during daemon shutdown. The worker retires after 1,000 requests or
ten idle minutes. Each archive RPC has a 30-second timeout; unit startup has
a separate ten-second timeout, and reaping can take another ten seconds.
After the hello exchange, before an archive request, the launcher checks that the worker’s main PID is in that unit’s own memory cgroup, reads back the RAM cap, caps swap, and reads current usage. On cgroup v1 it caps memory and swap together, or sets swappiness to zero where there is no swap accounting. On cgroup v2 it disables swap; without swap accounting it requires a host with no active swap. A missing cgroup, a cap that does not read back, unreadable counters, or usage already over the cap fail the start. Process RSS polling cannot replace this: it misses memfd pages and other processes in the unit.
A watchdog reads the cgroup counters every 500 ms and kills a worker over its cap, or whose usage cannot be read. It skips a tick when the daemon itself runs out of descriptors (EMFILE or ENFILE). Memory kills are identified by the watchdog or the unit’s cgroup OOM counter; a missing or unreadable counter alone is not evidence of an OOM kill.
A worker over three quarters of its cap after a request is retired. Units
also retire at three quarters of their lifetime cap. The supervisor checks
retirement before reusing an idle
worker, as well as after a request, so expired workers get no new input.
Kill closes the socket at once and asks systemd
to kill the unit by name; it never signals a PID and never blocks. The daemon
attempts to kill leftover csm-parse-worker-* units at startup, with a
30-second timeout. Cleanup failures are logged; lister initialization still
proceeds. Disabling scheduled phishing checks does not disable cleanup or
archive listings for requested account scans.
The worker gets an open zip descriptor, never a path, and reads it with
positional reads, so the daemon’s file offset stays put. It returns entry
names in a memfd it seals against writes, growth and shrinkage. The daemon
treats every reply byte as hostile: it checks the seals, then the size, every
record and the entry count, before the visitor sees one name. Records are
charged against the request’s directory budget, never a size the worker
reports. A malformed reply kills the worker and matches
workerproc.ErrWorkerFailed. An archive the worker could not read reports
itself incomplete, without counting as lost work. A worker can still leave
names out; validating the reply does not verify the archive. Name
classification stays in internal/checks.
workerproc.Pool holds a fixed number of supervisors, one per slot, and
Call names the slot. The mail extractor (parseworker.Extractor, op
extract_message) sizes its pool from email_av.scan_concurrency and names
units csm-parse-worker-mail<slot>-<generation>. The worker writes each
non-empty attachment to a memfd sealed against writes, growth and shrinkage.
A reply carries at most 200 parts. Names, content types and archive names are
cut to 255 bytes, the partial reason to 512, and From, Subject and each of at
most 100 recipients to 998.
The mail extraction API accepts open spool header and body descriptors.
Spool event descriptors are close-on-exec so worker launch commands cannot
inherit queued messages.
Its reply must be one UTF-8 JSON object with a parts field, which may be
null for an empty result. Each part must have a readable, sealed descriptor
of the claimed size. Individual sizes and their total must fit the request’s
limits without integer overflow. Invalid replies discard all parts and kill
the worker with reason protocol. The spool watcher extracts every message
through it: each scan goroutine owns one worker slot, because parts are
charged to the memory cgroup of the worker that wrote them. A failed
extraction becomes a partial result naming the failure, never an error.
Watcher replacement waits for the previous scan goroutines to finish before
reusing their slots. On shutdown, queued spool events release their
descriptors without starting extractions. The daemon waits for active scans
and watcher restarts to finish before stopping or clearing the parse pools.
If a panic unwinds the daemon before shutdown is signalled, cleanup skips
that wait so the process can exit. Extractor lookup and removal share the
watcher lifecycle lock because a replacement may still be starting.
Unit tests use temporary proc and cgroup trees, never the host’s. The kernel
tests in internal/workerproc/unit_kernel_test.go run real units as root on
the CI kernel runner.
YARA-X Worker Process
By default CSM never compiles or scans YARA rules in the root daemon. Two
kinds of csm yara-worker process run through workerproc.UnitLauncher
with the parse worker’s lockdown: no root, no network, no account files, a
memory cap and a lifetime cap. Wiring lives in
internal/daemon/yara_backend.go; the daemon side is yaraworker.Backend
in internal/yaraworker/backend.go and the worker side yaraworker.Serve
in internal/yaraworker/worker.go.
- The compile unit,
csm-yara-compile-worker-<generation>, runs for one compile: 1 GiB memory cap, files up to 256 MiB (its reply), five-minute unit lifetime. The daemon allows two minutes for the compile request. The daemon reads the rule files (trust check, disabled-rule stripping) and passes their text in one sealed memfd. The worker compiles with includes disabled, so rule text comes only from that memfd, and returnsRules.WriteTooutput in a sealed memfd. The daemon copies it into its own sealed memfd and keeps it. - The scan unit,
csm-yara-worker-<generation>, is a single worker: memory cap 512 MiB plus the larger ofthresholds.full_scan_max_file_mband, with mail scanning on,email_av.max_attachment_size, read from the live config at each launch; no file writes, 24 hour lifetime. Before the supervisor may send it a scan, the launcher loads the kept rules into it. A restart loads them again and never recompiles: 55 ms for the shipped rules, about 3.4 s for Forge full. - Both units run with
MemoryDenyWriteExecute=no(UnitConfig.AllowExecutableMemory): wasmtime maps the compiled rules executable, and with the denial the load fails as “invalid YARA-X compiled rules file”. NoRLIMIT_ASis set; the cgroup cap is the limit. - A reload compiles in a new compile unit. On success the new rules replace
the kept ones, the scan worker on the old rules retires before its next
request, and a background warm-up starts its replacement. On failure the
current rules stay; with none loaded the failure is
CompileError(). Compiles run one at a time; a canceled startup retry stops waiting even when a rule update is already compiling. - Forge downloads are compile-tested in the compile unit before they are
installed. If the merged reload afterwards fails because the compile unit
could not run, the new tier stays on disk, its version is not recorded,
and a Critical
yara_backend_unavailablefinding says so; only rules that do not compile, or a collapsed rule count, roll the tier back. - At startup the daemon attempts to kill leftover units of both prefixes; cleanup failures are logged without preventing startup.
Failures and findings:
- The backend is active from boot, before the first compile. While no rules are loaded because a compile failed or could not run, every scan returns an error, never a clean result.
- The boot compile runs before the daemon reports ready to systemd, so it is
bounded to 45 seconds. A compile that cannot start, crashes, times out or
outlives that bound is retried in the background with backoff; when the
boot attempt and three retries have failed, a Critical
yara_backend_unavailablefinding fires. - Rules that do not compile raise Critical
yara_worker_compile_failedand are not retried. So does a rule set the compile unit cannot fit: more than 1024 rule files (refused before a unit starts), an out-of-memory kill under the 1 GiB cap, or compiled rules over 256 MiB. Each message namessignatures.yara_worker_enabled: false, which compiles in the daemon. - Any counted scan worker failure (failed start, timeout, crash, out of memory,
protocol) marks
watcher: yara_workerfailed and raises Criticalyara_worker_crashed, at most once a minute after the last alert; an attached worker is killed. The next request starts a replacement; a served request marks the watcher attached again. - Only failed starts count toward the worker breaker (
BreakOnSpawnOnly), so content that kills the worker costs one restart per scan and cannot switch YARA off for other files. The backend keepsworkerproc.Strikesby SHA-256 of exactly the bytes sent for scanning. Content with two scan worker failures within a day is refused as a scan error without starting another worker. The check and failure recording run under the supervisor’s request lock (Supervisor.CallGuarded), so queued copies see the failures ahead of them. A failure is recorded against the rules the worker actually loaded, even if a reload lands while it starts. Failed starts, cancellation, breaker refusals and descriptor pressure do not strike content. Hashing runs only while an unexpired strike is recorded or to record a new failure. Records expire a day after the last failure and a successful rule reload clears them. The list holds at most 1024 content hashes, evicting the oldest failure when full.
signatures.yara_worker_enabled: false runs YARA-X inside the root daemon
instead; csm validate warns about it.
Wire contract (workerproc frames on the unit’s socket pair):
| Op | Request descriptors | Reply |
|---|---|---|
hello | 0 | protocol version |
compile | 1: rule texts back to back, named and sized by sources | rule_count and a sealed memfd of compiled rules, or compile_error and no descriptor |
load | 1: compiled rules | rule_count |
ping | 0 | rule_count of the loaded rules |
scan_fd | 1: content, read positionally up to max_bytes | matches, always an array |
Malformed requests get an error frame, which kills the worker. A rule compile error is a result, never a worker failure, and its text is cut to fit the frame. Scans need a regular descriptor and every requested byte; the worker gets its own read-only open of a file. Byte scans travel as sealed memfds, so the frame cap does not limit them. With no rules, or every rule disabled, byte scans answer clean without a worker unless compilation failed, and mail-part scans fail.
Testing: internal/yaraworker launches the test binary as the worker with a
fake engine (helper_test.go), including fault modes for crashes, hung loads
and bad compile replies. Tests tagged yara run the real engine, including
backend recovery from invalid rules with disabled rules still excluded. Unit
lockdown is verified on the CI kernel runner and on scratch hosts.
Building the Documentation
cd docs
mdbook build # generates docs/book/
mdbook serve # local preview at http://localhost:3000
Clean application corpus
The same job measures the shipped rules against long uninterrupted base64, hex and word runs, dense variable calls, and PDF-shaped streams. It discards one warm-up and scores the fastest of two further scans against a 5-second per-file budget. Each attempt has a 10-second engine timeout; a timeout never counts as a completed scan, and a cold timeout alone cannot fail the gate. Both scored attempts timing out fails it. The job shares the heavy-test resource group to avoid competing with other heavy tests in this project.
A rule with no literal atom to match on can still pass match tests while
stalling mail delivery. Investigate slow rules with yr scan --profiling.
Run the gate and its fixture/timing checks locally with YARA-X installed:
CGO_LDFLAGS="$(pkg-config --libs --static yara_x_capi)" go test -count=1 -v -tags yara ./internal/yara -run 'TestShippedRulesScanWithinBudget|TestRuleScanBudget'
Every pipeline runs the required clean-corpus gate in the production YARA-X builder image. Package publication and GitHub releases depend on its success.
The attack replay corpus gate runs the same way and gates the same jobs.
See cPanel release tests for the required image, upgrade baseline, release dependencies and retained evidence.
Production tag selection, execution artifacts, and the required isolated kernel runner are described in Production build and kernel tests.
make check-fixtures is also a blocking GitLab job and publication dependency.
It checks all tracked and unignored files under testdata and fixtures,
including files without extensions. Scanner failures stop the check; reports
identify the file and line without printing the suspected address. The
same run scans every tracked and unignored file for private names when
CSM_PRIVATE_TERMS (or -terms) names a file holding one case-insensitive
regular expression per line. That file stays outside the repository: the
GitLab job receives it as a file-type CI variable and fails without it
(-require-terms), while a local run without one prints a skip notice. Reports
name the file and line, never the matched text. Filenames are checked too;
a matching filename is reported with its path redacted. The
fixture sanitisation rules
describe the additional manual privacy review.
Attack replay payloads are decoded in memory before checking. Every data
file in a sample directory is decoded regardless of its suffix; .b64
files are decoded elsewhere in the attack corpus test data too. Manifests,
corpus-root notes and saved Go fuzz seeds are checked as plain text.
Invalid base64 fails the check, and diagnostics never print decoded content.
Admission preview ledger
Admission previews reserve, charge history and capacity, and end observed in one transaction. They write both audit steps, never execute a response, never count as applied, and do not end the evidence episode or raise a notice. Repeated evidence is answered until its quiet window closes.
Imported legacy usage retains its accounting-window charge after an attempt is retired. Retirement of old attempt history cannot create new admission capacity before that window closes.
The scheduler records only ceiling, history-budget and recovery-reserve deferrals, once per changed reason. Fair turns, member bounds and retry backoff produce no deferral; a cleared budget reason produces no new gap notice. Challenges spend fair turns and history without spending block credit, and bounded durable holds preserve the earned turn of charged work while challenges proceed.
The owner schedules queued preview work on its timer and uses the next ledger wake even when no new arrival appears. Reloads recompute that wake under the current ceiling. A first attempt lasts the legacy-selected lifetime, or the configured response’s, never longer than auto_response.block_expiry (spec 5.6): an executable owner reads it from the policy of the epoch, an owner without an engine from the current configuration, and a policy without a positive one is refused. Retries keep that absolute expiry. The owner observes each pick transactionally and continues delivery retries independently of scheduling. A refused preview ends only its own queued work, so an invalid expiry cannot starve other picks; a storage failure retries the turn, twice at once and then after waits that double up to a minute, as a held probe is, and shows as admission.owner.turn_error and the doctor row admission turns rather than a stopped owner.
Held responses coalesce by evidence, kind and effective entry, whether the root entry is named explicitly or omitted. Derived entries validate their producer binding before fresh placement, coalescing and answered results, including direct queue requests. Retained queued work also revalidates its entry binding before scheduling or coalescing. The first selected temporary lifetime is durable and cannot be extended by a later report of the same candidate; absent metadata retains the configuration-selected lifetime.
Root preparation and selected submission use the cached admission view and do not wait for ledger writes. Missing observation is an attribution refusal; malformed input remains invalid. Unminted Critical responses refused while the ingress is stopped count in its stopped status. Minting accepts canonical IPv6 evidence, while the selected kind checks containment support: HTTP challenges do not depend on the firewall family setting.
Per-address funnels hand every selected block or challenge to admission before
the existing responder runs. A process-local finding flag skips only the
second evaluation of the same finding, including its queued block retry;
unrelated pending work and fresh observations still run. Admission uses the
canonical address selected by the legacy extractor, including its message
fallback, without changing the finding or its audit identity. Distinct
observations still submit even when their address is already challenged.
While admission executes blocks, a block it accepts, or refuses because the
client is behind a trusted proxy or is a CDN edge, skips the legacy block:
no engine call, tracker row, queued retry or hourly charge. For a block it
accepts, the scan also drops every other legacy block of that address in the
same pass: its queued retry, findings admission could not take and mail
spray findings. Every other mint or synchronous ingress refusal keeps the
legacy block. Mail spray blocks, and derived blocks asked to last longer
than auto_response.block_expiry, stay legacy: admission would cut them to
it. The funnel reads that lease as the scan loop does, with a 24-hour
fallback for an empty, unparsable or nonpositive value; the daemon passes a
nonpositive value to the policy unchanged, the policy refuses it, and the
owner then executes nothing, so legacy keeps every block. Acceptance is not application: a later hold, dry-run change or queue
displacement never returns that request to legacy. A derived block admission
takes ends before the engine as an expected refusal: timeout escalation and
central intel log nothing, and the incident correlator counts it as
requested. Previews and challenges never change a responder decision.
Subnet funnels submit every selected response before the existing responder evaluates its available execution budget. Constituent evidence follows the selected mail response; paths without eligible observations retain their designed refusal rather than minting a replacement root.
A challenge entry retains its routed root until timeout escalation hands the selected response to admission. Escalation never creates new root evidence, and permanent selections take the designed rootless policy refusal.
Central selections use the locally corroborating finding as their root and carry its original attribution if preparation refuses. They do not adopt a remote report as local evidence, and preserve the selected temporary lifetime.
Incident event preparation retains the attesting finding’s check and severity, its evidence or its preparation error until a block is actually selected. Roots use the canonical address the legacy block selects while retaining the original finding. Only that selection counts an admission decision; generic and spray paths preserve the original check and cause. A derived response never mints a second root.
Hourly summaries count committed arrivals, memory-coalesced reports, mint refusals, displaced submissions and both observed and terminally refused picks. The displacement callback runs under the ingress mutex on the submitting goroutine and uses the funnel’s clock; drain callbacks use the owner’s fresh clock. Summary time never regresses: event hours are clamped to the latest admission-clock or funnel hour after a backward wall step, including clock readings from an incomplete startup. Immutable write acknowledgments cannot consume newer rows. Summaries continue during ledger or clock outages, and shutdown flushes completed decisions even when a final drain fails. Failed writes retain a bounded chronological set of completed hours, so missing recordings remain explicit lower bounds.
Ingress placement on an ended or in-flight candidate is a counted invalid refusal. An episode that cannot be encoded is ledger damage. Shutdown closes ingress before returning new blocks to legacy, even while an owner turn is still running. Previously accepted work is drained for restart; new legacy-owned blocks cannot enter that queue. Incident background response loops stop before the owner. An owner that executes blocks ends, as disabled, the incident and spray blocks an earlier process queued: the correlator reads its selectors at start, so that work may no longer be selected.
Selected work also clears a stale budget reason without another count or notice. The scheduler preserves deferrals owned by other components. Wake times consider both the next charged turn and the history needed by challenges, while independent history holds retain their earned credit through a ceiling wait.
A batch readback requires every member to be running. Direct queue requests can reserve a retired action ID again. If another member of that batch still runs, execution refuses the mixed batch without writes.
Clean application corpus
The required test:clean-corpus job downloads the public vendor ZIP archives in
scripts/clean-corpus/manifest.json. The manifest pins the version, official
HTTPS source, SHA-256, license file, and exact regular-file count. Archives are
cached under their checksum, so editing metadata or adding a source downloads
only new pins. Each archive is verified before extraction into a new private
directory. Corrupt cache entries are replaced with a verified download;
unverified downloads never enter the reusable cache. Failed downloads, missing
licenses, empty or incomplete inventories, unsafe archive entries, and read
failures stop the job. No downloaded PHP or JavaScript is executed.
CSM_CORPUS_ARCHIVE_CACHE selects the cache directory. The tool reserves its
.csm-corpus-v1 subdirectory for checksum-named archives, temporary downloads,
and a persistent lock. Do not put other files in that subdirectory. Cleanup
only touches generated archive and temporary names there, never ZIPs elsewhere
in the configured directory or symlink targets. A symlinked managed directory
is refused. Concurrent preparations sharing a cache fail with a busy error;
retry after the active run finishes.
Valid flat cache entries from older versions are verified and copied into the managed cache without downloading. Their originals are left untouched because names alone cannot establish ownership; operators may remove those old inputs once no older tool uses them. Managed archives no longer pinned and interrupted downloads are pruned before fetching, even if preparation later fails. GitLab saves verified downloads and cleanup on failed jobs too, so retries retain progress without accumulating retired pins.
Every source names the CMS it belongs to in a cms field, and the manifest
lists each supported CMS that has no pinned source yet under pending, with
the roadmap item that blocks it. The manifest is version 2; the validator
refuses older versions, a source without a CMS, a supported CMS that is
neither sourced nor pending, and one that is both, so a database scanner for
a new CMS cannot ship without a corpus decision. A pending entry records
absent evidence only: no scanner, admission rule, hit budget or result filter
reads it, and files from that CMS are treated exactly as before on a server.
When a source for a pending CMS lands, its pending entry is removed in the
same change. The manifest, not this page, is the record of which CMS has a
false-positive gate.
Ordinary unit tests validate the manifest offline, including with
go test -trimpath. Invalid metadata is rejected before cache, extraction or
report files are created or changed. Command tests check the full archived manifest
and every inventory record; these checks do not replace the detector gate.
The initial corpus contains 10,178 files: WordPress 6.8.2 (GPL-2.0-or-later), WooCommerce 9.9.5 (GPL-3.0-or-later), and Elementor 3.29.2 (GPL-3.0-only). The archives retain their license notices, including notices for bundled components. These are fixed test versions, not installation recommendations. Source URLs and license file locations are in the manifest.
Run the same gate locally with the Go toolchain from go.mod and the matching
YARA-X C library installed:
bash scripts/clean-corpus-test.sh
The script compares the installed C library version with both the Go binding
and production builder pins. It runs YARA, YAML signatures, PHP taint, and
JavaScript taint gates. Ordinary unit tests may omit external corpus inputs;
the required job sets CSM_CORPUS_REQUIRED=1, so missing input is fatal.
Package publication and GitHub releases explicitly depend on the job.
corpus-results/ is retained for one year, including on failures:
manifest.jsonidentifies the authenticated archives and licenses.inventory.jsonlists every extracted file, its size and SHA-256.engines.txtrecords the commit, toolchain, YARA-X and parser versions.- Engine JSON reports include counts, per-rule hits including zeroes,
thresholds, and taint status counts.
tests.jsonlrecords execution.
YARA and both taint engines require zero findings. YAML retains the existing
reviewed false-positive budgets in internal/signatures/corpus_fp_gate_test.go;
new rules default to zero. The pinned corpus currently produces 10 YAML hits
across five rules, all within those existing budgets. This is a regression
gate, not a claim that every engine has zero false positives. A second YAML
check fails if any regex would be skipped on a corpus file it matches.
The signature engines apply the production default 16 MiB bound and archive admission policy; 10,157 inputs reach each engine. PHP taint reads 10,171 inputs within its source limit, including 163 that complete analysis. JavaScript admission receives all 10,178 files and completes analysis on 289; seven are oversized and twelve have parse errors. These JavaScript gaps have explicit ceilings in the required gate; canceled, resource-limited and panic outcomes have a zero ceiling. Neither oversized nor unparsed inputs count as proof of successful analysis. The required taint gates each need at least 100 completed analyses as well as their corpus inventory checks.
To update the corpus, download a versioned archive from its official source, review its license and file inventory, and commit the new manifest pin with the measurements. Investigate new matches and narrow detectors using content semantics. Do not add scan-path exclusions or raise budgets to pass a run. The gate also compiles deliberately overbroad YARA and YAML rules and verifies that their clean-file matches are rejected.
Attack replay corpus
The required test:attack-corpus job plants known attack samples and
near-miss controls in a temporary cPanel account and runs them through the
three production paths that read files: realtime analysis, the scheduled deep
tier and the operator full account scan. Each path applies its own production
responders. The job fails when a path stops catching a sample, catches it at a
lower severity, flags or changes a control, remediates differently, or starts
catching a recorded gap. Package publication and GitHub releases depend on it.
Unit tests prove that a detector works on the input it was written for. The corpus proves that the whole path still works: admission, path scoping, severity, quarantine eligibility and the responders. A change that breaks a detector end to end fails here even when every unit test passes.
The corpus measures file content only. Log and event replay and expected incidents are later stages.
The three paths
Every sample runs alone, with its own account tree, cPanel domain map, state
store, quarantine directory and monitor. The platform is pinned to cPanel, the
layout the samples use. The config is the shipped configs/csm.yaml template
with auto-response and file quarantine turned on and observe mode off, and
nothing else changed. Both signature engines load the shipped rules from
configs/. A partial rule load, an empty rule set or a scan error fails the
gate.
- Realtime. The driver builds the file monitor without fanotify, opens
each planted file and derives its event path the way the event loop does.
The file then goes through the production admission filter and, when
admitted, through
analyzeFile. A file the filter drops counts as not detected. The driver drains the dropper tracker and the staged package queue at once instead of waiting for their timers. Remediation is the inline quarantine thatanalyzeFileperforms. - Deep tier. The driver runs the file checks of the deep tier through the
production runner,
runParallelWithContext, so the runner’s responders act on the findings as they do in production. Every deep check is classified as a file check or a host check (network, database, mail, packages). Host checks do not run. A test fails when a new deep check has no classification, so it cannot stay out of the gate unnoticed. Before planting, the driver runs the tier once on the empty account tree with the same state store. Findings that name no planted file and also appeared in that empty run are dropped as host noise. No check or path is allowlisted. - Full scan. The driver runs the account scan with the options that
csm scan <account> --full --quarantineuses, then the scan job’s quarantine loop. That loop ignores the auto-response settings, as it does in production.
The tier and the full scan run PHP taint analysis in-process. Production runs it in a separate worker, which has its own tests.
Offline
The run never reaches the network. Requests through the default HTTP
transport are refused. On the two scan paths, database queries get an empty,
successful answer for a synthetic site, and host commands fail as a missing
program would. The realtime WordPress checksum fetcher is stopped. Refused
requests, database queries and wp runs are counted per sample and appear in
the report as network_attempts. A sample whose verdict depends on such a
lookup says so in its description.
Sample format
Samples live in internal/attackcorpus/testdata/corpus/<category>/<name>/.
Each directory holds a sample.json and the base64 content files it names.
Content is base64 at rest and decoded only in memory or under a test’s
temporary directory, because endpoint antivirus deletes decoded web shells
from a checkout.
sample.json fields:
name: unique across both halves of the corpus, public and private.category: one ofwebshell,dropper,malicious_plugin,credential_stealer,injected_js,phishing_kit,spam_script,persistence,binary,archive.kind:maliciousorcontrol.source:synthetic,fixtureorreal.description: one line on what the sample is. It also names any policy, shortfall or offline lookup that explains its expectations.provenance: where the content came from. For a fixture, the test it was taken from and how the planted shape differs; for a synthetic sample,written for the corpus.files: each withpath, account-relative (public_html/...) or@crontabfor the account’s crontab; exactly one ofb64(a file beside the manifest, ending in.b64) orselftest(a sample from the self-test bundle); and an optionalmode. JSON has no octal, so0755is written493.expect: one entry for each ofrealtime,tierandfullscan:findings: each withcheck,file(a planted path) andmin_severity(warning,highorcritical). Every listed finding must arrive at that severity or above. Extra findings are reported, not failed.gap: why this path misses the sample. Exclusive withfindings.remediation: per planted file,quarantined,cleanedoruntouched. A file left out is expected untouched.cleanedneeds the SHA-256 of the cleaned bytes incleaned_sha256. The deep tier and the full scan clean a WordPress core, plugin or theme file in place, instead of moving it, when a Critical quarantine-eligible finding names it. A plugin or theme file with a known web shell name is quarantined instead, and so is a file that cleaning would leave empty with no page output, or that its core or plugin release does not ship. Whole-file malware expectsquarantined; an injected site file expectscleaned, oruntouchedwhen the cleaner refuses the removal.
A missing file counts as quarantined only when a quarantine entry names its
path, or a parent directory’s, and the retained copy matches the original
bytes. A damaged entry is an error. A file that disappears with no entry is
reported as removed, which no manifest may expect. A file whose bytes
changed is cleaned.
All files of a sample are planted before the first one is replayed. Manifest order is not write order, so a sample whose verdict depends on the order of writes is judged on the final tree.
Every self-test bundle sample is referenced exactly once, so the bundle and the corpus cannot drift apart.
Adding a sample
- Write or decode content only in a temporary directory that your
antivirus excludes, never in the checkout. Commit only the
.b64file. - Add files by explicit path. Never run
git add -Aorgit add .: an antivirus that deletes a decoded file leaves a deletion for the next commit to pick up. - Public samples follow the public repository rules: documentation IP
addresses only (
192.0.2.0/24,198.51.100.0/24,203.0.113.0/24,2001:db8::/32),example.com,example.netorexample.orghosts, and no real names, accounts, domains, mailboxes or credentials. This applies to names, descriptions and provenance as well as content. make check-fixturesdecodes every corpus data file and scans it for addresses and private names. It decodes one layer only. A payload nested inside the sample, such as base64 inside PHP, a compressed blob, an archive member or aString.fromCharCodelist, needs a manual check of the same rules before commit.- Write the expectations from what the sample is (see below), then run the
gate.
TestPublicCorpusLoadsin the default test job loads the public corpus and checks category coverage, so a malformed manifest fails there first.
Expectation rules
Judge each expectation from what the sample does. A run may draft the findings, but every expected finding and severity must match what the attack deserves, not what the detector happens to report today, except a recorded severity floor (below).
- Gaps. A malicious sample that a path misses records a gap with the
specific reason: which check misses which shape. A gap is a measurement, not
an exemption. When a detector starts catching a recorded gap, the gate fails
with
gap closed:until the manifest replaces the gap with the finding. Only a finding on a planted file, or one that names no file but quotes an admitted planted file in its text, closes a gap; a finding on a directory or other path is reported as an extra finding. Files rejected by admission cannot close a gap. Closing a gap is a corpus edit in the same change as the detector fix. - Never weaken an expectation. Do not delete a sample, lower a
min_severity, turn a finding into a gap, or change an expected remediation to make the gate pass. A regression is fixed in the detector or the responder. Never allowlist a path or name to make a sample pass. - Explicit policy. Where production deliberately leaves a detected file in
place, because the check is not on an explicit quarantine list or the
responder is off in the shipped config, the path expects
untouchedand the description names the policy. When the policy changes, the expectation becomesquarantinedin the same change. - Severity floors. When a path detects a sample below the severity the
attack deserves,
min_severityrecords the observed level and the description names the shortfall. The gate then stops the severity from dropping further, but does not raise it. - Remediation shortfalls. When a detector heuristic, not an explicit policy, leaves a file in place that should move, the expectation records the observed remediation and the description names the shortfall.
Controls
Every category needs at least one malicious sample and one control, or the run
fails. A control is the legitimate file a detector for that category is most
likely to confuse: a cPanel MultiPHP handler in .htaccess, an inline base64
image, a vendored phar, a capability-checked upload handler. A control expects
no findings and untouched on every path, and cannot carry a gap.
Any finding on a control’s file fails the gate, including a file the realtime admission filter dropped. A finding that names no planted file but quotes a control’s path is a false positive. Any other unattributed finding on a control is an error.
Failures
Each failure in the report and the test log starts with its kind:
| Prefix | Meaning |
|---|---|
missed: | An expected finding did not arrive. |
severity: | An expected finding arrived below its min_severity. |
false positive: | A control drew a finding. |
remediation: | A planted file ended quarantined, cleaned, untouched or removed other than expected. |
gap closed: | A path caught a sample recorded as a gap on it. |
error: | The driver did not complete for this sample, or a control drew a finding that names no planted file. |
A run also fails before its first sample when a corpus cannot load, a required private half is not set, a private sample reuses a public sample’s name, a category lacks a malicious sample or a control, a self-test sample is not referenced exactly once, or an engine did not load. A test filter that excludes a sample, a driver that stops early, a check that panics, times out or reports partial coverage, and a failed quarantine all fail the sample.
Driver errors appear in the test log by type only, unless
CSM_ATTACK_CORPUS_SHOW_ERRORS=1 is set (see below). Reports carry check
names, severities, planted paths and verdicts, never sample content or finding
text, because either can quote decoded bytes.
Measurement limits
What the corpus does not measure yet:
- No driver replays program execution, so YARA scans of programs started from
a document root (
docroot_binary_exec) are not measured. - No driver replays the critical tier, so the crontab check is not measured on the scheduled path.
- The daemon’s batch responders (permission fix, database malware response,
companion sweep queueing, WP-Cron fix) are not driven. Realtime remediation
is the inline quarantine only, and the tier stops at the runner’s
responders. In production, realtime findings also reach the batch process
kill and file quarantine. Verdicts agree today because the only realtime
check on the batch quarantine list,
signature_match_realtime, passes the same high-confidence gate there as inline. A realtime check added to that list would change production remediation without the gate seeing it. supply_chaincannot fire: a fresh install has no advisory file.group_writable_phpcannot fire: planted files belong to the test user’s group, not a web server group.cleanedis not exercised: the.htaccesscleaner is off by default, the full scan skips.htaccessand crontab findings, the crontab fix is operator-only, and no sample draws a Critical quarantine-eligible finding on a file both scan paths clean in place (seeremediationabove).- The throttled performance file checks run once per sample, in the second tier run only. A tier run that skips any check fails the sample.
- The full scan quarantine step is a copy of the scan job’s loop. A change to that loop must be mirrored in the driver.
- Realtime does not admit
.jsor.pharfiles, so those controls are not exercised on that path.
Private half
Real attack samples, scrubbed of customer data, live in a private GitLab
project in the same format, with <category>/<name>/ at the project root.
The project comes from the CSM_ATTACK_CORPUS_PROJECT CI variable, and it
must allow this project’s CI job token to read it.
This repository pins the private commit in
internal/attackcorpus/private.commit: one line holding the full
40-character lowercase commit hash. Moving the pin is a reviewed public
commit.
In CI the script does the following before any test runs. Any failure stops the job.
- It fails when
CSM_ATTACK_CORPUS_PROJECTis unset or the pin file is missing or malformed. The pin accepts one full lowercase hash, with or without a single final newline, before any clone or checkout runs. - It clones the project with the CI job token into a new temporary directory, open only to its owner and outside the checkout and its artifacts. A credential helper reads the token from the environment, so the token never appears in a command line, a URL or the clone’s git config, and no configured helper can store it. The helper serves only the exact project on the CI server over HTTPS, and Git refuses redirects. A temporary directory inside the checkout or reports is rejected. The temporary clone is removed on success or failure, including a failed clone.
- It checks out the pinned commit and fails unless
HEADis exactly that commit, so a short hash or a ref name fails. - It runs
scripts/fixturecheck -corpus -require-termsover the checkout. Every file is scanned for IPv4 addresses outside the documentation ranges and for the private terms in theCSM_PRIVATE_TERMSfile variable. Sample data is decoded first, one layer deep, as in the public check. - It sets
CSM_ATTACK_CORPUS_PRIVATEto the checkout andCSM_ATTACK_CORPUS_PRIVATE_REQUIRED=1, so the tests fail rather than run without the private half.
engines.txt records the pin as private corpus: <commit>.
Outside CI nothing is cloned and the private half is not required. A local
run uses the public half alone unless CSM_ATTACK_CORPUS_PRIVATE names a
checkout. engines.txt then says private corpus: local, unpinned, and
otherwise private corpus: none.
The coverage rules apply to the public half alone. The private half must load without validation errors and hold at least one sample. Sample names are unique across both halves. Reports and test logs print sample names and planted paths, so private samples follow the public naming rules for both: no real names, accounts, domains or mailboxes.
Running it
Run the gate in the release builder image, the one CSM_BUILDER_IMAGE and
CSM_BUILDER_TAG name in .gitlab-ci.yml:
GO_LINUX_RUNTIME=docker GO_LINUX_IMAGE=<release-builder-image> \
scripts/go-linux.sh bash scripts/attack-corpus-test.sh
The script checks that the YARA-X C library matches the Go module and the
active builder pin exactly, then runs every TestAttackCorpus test in the
harness, daemon and checks packages with -tags yara. It writes to
attack-corpus-results/, which the job keeps for one year, including on failure:
realtime.json,tier.json,fullscan.json: per sample results and a per-category summary (detected, gaps, missed, false positives, closed gaps, remediation mismatches, errors). The script fails if any is empty.engines.txt: Go version, YARA-X version, commit, the private corpus line (see Private half) and run time.tests.jsonl: thego test -jsonevent stream.
The output directory and its report files are open only to their owner, including when the script reuses an existing directory. The script does not change the umask, so under the usual 022 the drivers create files with the modes the daemon gives them. CI artifact access still follows the project’s artifact permissions.
Environment:
CSM_ATTACK_CORPUS_REPORT_DIRrequests a run and names the report directory. Without it the corpus tests skip.CSM_ATTACK_CORPUS_REQUIRED=1turns that skip into a failure. The script sets both.CSM_ATTACK_CORPUS_PRIVATEadds the private half, a corpus checkout in the same format. It must load and hold at least one sample.CSM_ATTACK_CORPUS_PRIVATE_REQUIRED=1fails the run whenCSM_ATTACK_CORPUS_PRIVATEorCSM_ATTACK_CORPUS_REPORT_DIRis not set, instead of skipping. The script sets it in CI.CSM_ATTACK_CORPUS_SHOW_ERRORS=1logs the full reason of a driver error instead of its type. Local runs only: a reason can quote decoded sample bytes, and CI keeps test logs as artifacts. The report script unsets it, including when inherited from the caller.
The other TestAttackCorpus tests check the drivers themselves, on their own
samples, and need no report directory.
Recorded finding streams
Calibrating correlation (the coordinated-attack threshold, corroboration
grading, sequence rules) needs what a real host produced over days: every
finding with its check, severity, timestamp, owner and text, including the
false-positive floods and the long-lived rows that never clear. CSM already
writes that stream: every dispatched finding lands in
/var/log/csm/audit.jsonl (rotated copies are gzip files beside it). A
recorded stream is a copy of those files with every identity removed.
scripts/finding-stream turns the raw files into an anonymized stream:
# On the operator machine, after copying the files read-only from the host:
go run ./scripts/finding-stream anonymize \
--salt-file ~/.local/share/csm/finding-streams/salt \
--out ~/.local/share/csm/finding-streams/host-a.jsonl.gz \
raw/audit.jsonl raw/audit.jsonl-*.gz
What the tool replaces, in every structured field and in free text:
- Host names become
host-<id>, account namesacct-<id>, domainsdom-<id>.example, mailboxesuser-<id>@dom-<id>.example(a mailbox truncated after the@keeps the sameuser-<id>). The<id>is derived from an HMAC of the value under a private salt, so the same salt maps one account to one pseudonym on every host and streams can be joined without knowing who is who. Names are replaced wherever they sit: in/home/<account>/paths,Account:lines, process context, LiteSpeed vhost tokens, and inside longer tokens such asexample.com-ssl_logorweb1.log. A name holding an underscore or starting with a dot, such as a bare mail login, is replaced as a whole wherever it is not glued to a letter or digit, ignoring case. Where identities overlap, the leftmost complete one wins, and a full mailbox or home path keeps its usual mapping. A replacement is never rewritten again. A learned name made only of punctuation is not replaced, so it cannot turn separators into identities; the final check refuses the run if one appears. The host’s short name (its first label, when it carries a digit) maps to the same pseudonym as the full name. Any other token that looks like a domain (two or more labels, alphabetic last label that is not a file extension) is mapped too, even if no field named it; that over-reaches on a few dotted names likeoptions.optionand is accepted. Domain-shaped names inside filenames are replaced even when no structured field named them. Only pseudonyms actually emitted by this run are exempt; a raw name beginning withhost-oracct-is still scrubbed. - IPv4 addresses map into 198.18.0.0/15 and IPv6 addresses into
2001:db8::/32, both reserved and never routed; a raw value always maps to
the same pseudonym. The IPv4 range holds 131,072 addresses, so distinct
addresses can share a pseudonym. A manifest counts distinct addresses and
distinct pseudonyms per family, and a replay cannot tell merged addresses
apart.
Loopback addresses, system users such as
rootornobody, and bare numeric uids are kept: they identify nobody and carry meaning. Equivalent IPv6 spellings share one pseudonym across all streams; IPv4 addresses written in IPv6 form use the IPv4 pseudonym. Addresses inside filenames and before numeric rotation suffixes are also replaced. This can replace address-shaped version numbers; privacy takes priority over preserving ambiguous numeric text. New audit rows can carry acidrslist. Those addresses use the same pseudonyms as addresses in text, keep their prefix lengths, and participate in the output leak check. Older audit rows remain readable without the list. - The details of a credential-leak finding are dropped entirely, and generic
password=,secret:andtoken=material is blanked anywhere, including quoted keys and values containing spaces. - Finding ids become
fid-<id>, a salted id of at least 128 bits that keeps case, so an action row can name its finding without exposing the raw id. - Timestamps, check names, severities, path structure below the account, plugin and file names, and process names are kept unless they contain an identity: they are what calibration reads. Paths and mailboxes in process command lines and parent processes also teach the scrubber which identities to remove elsewhere.
Before writing, the tool scans its own output for every identity it learned from structured fields, paths and mail addresses, for domain-shaped names, for raw ids (including inside longer tokens), and for any mailbox or address outside the reserved ranges. It refuses to write if it finds one. The raw-id scan includes check names and severity labels; these fields otherwise keep their original vocabulary. Structured finding ids must be ids this run emitted. A name glued to underscores or file extensions is still found. The summary prints row counts, the number of distinct checks, replacement counts, the time span and a salt fingerprint, never identities, check names, paths or the salt itself. An error names only the stream, the file’s position on the command line, the line number and a fixed reason.
Every row must parse as exactly one JSON object of the known schema, with a supported version and a timestamp field. Rows that older CSM versions wrote with a zero time are kept, counted in the manifest and left out of time spans. Unknown or repeated fields, nulls, data after the object and values over the size limits refuse the whole run. Malformed Unicode, non-JSON whitespace and timestamps that cannot be written back as JSON also refuse the run before a salt is created. Typed rows accept only the exact pseudonym spellings emitted during that run.
Handling rules:
- Copy the audit files read-only (
taroverssh, orscp) into a local directory with mode 0700, run the tool, then delete the raw copies. Nothing runs on the monitored host. - The salt file is created exclusively on first use with mode 0600. Existing salts must be regular files without group or other access; symlinks are refused. A concurrent run that reads an unfinished salt fails without replacing it; retry after the creator finishes. Keep it private and reuse it for every host whose stream should be joinable with the others; losing it makes new recordings unjoinable with old ones.
- Outputs cannot replace an input, the salt, the input manifest or each
other, whether named by path, through a symlinked directory or as a hard
link, and an existing output must be a regular file. These checks run
before the salt is created, including for missing directories and paths
containing
... Case-only and Unicode-equivalent path variants are refused on every platform, as are paths that would need a file to also serve as a directory. Every output is written to a private temporary file beside its destination, and all are complete before any is renamed into place; if publishing fails partway, the outputs already replaced are restored. New directories have mode 0700 and outputs mode 0600, even when replacing a less restricted file. - Filesystem cleanup failures are reported without claiming success. If rollback fails, remaining recovery copies are retained. If every output was published but backup cleanup fails, the error explicitly says the outputs were published; check the manifest and remove the leftover backups before sharing the directory. Other cleanup errors report unchanged outputs with temporary files left to remove.
- Recorded streams stay outside the repository entirely, in a private
directory such as
~/.local/share/csm/finding-streams/with mode 0700. A pseudonymized stream still describes real incidents on a real host, and this repository is public. Do not keep them in an ignored directory inside the checkout:git clean -fdxremoves ignored files too, and a recording that took a host weeks to accumulate is not reproducible from anywhere else.
Joining actions and firewall entries
The action log (actions.jsonl) and the firewall audit log
(<state>/firewall/audit.jsonl) say what CSM did about the findings. The same
run can anonymize both next to the findings, so the three streams share
pseudonyms. A joined run needs a manifest, and a manifest needs a build of a
known commit without local changes, so build the tool from a clean checkout
first:
go build -o /tmp/finding-stream ./scripts/finding-stream
/tmp/finding-stream anonymize \
--salt-file ~/.local/share/csm/finding-streams/salt \
--out host-a/findings.jsonl.gz \
--actions raw/actions.jsonl --actions-out host-a/actions.jsonl.gz \
--firewall-audit raw/firewall-audit.jsonl --firewall-out host-a/firewall.jsonl.gz \
--manifest host-a/manifest.json \
raw/audit.jsonl raw/audit.jsonl-*.gz
Action and firewall rows are rebuilt from a closed list of fields rather than scrubbed:
- An action row keeps its time, operation and action, actor kind, result, a
reason category, whether an error occurred, whether the file existed before
and after, a block lease, and salted ids for the finding, incident, action
and the action it undoes. Addresses map as in findings, networks keep their
prefix length and endpoints their port and protocol. File paths, process ids
and other targets become
tid-<id>. Command lines, error and reason text, undo commands, recovery paths, digests, sizes, modes and owners are dropped. Account names are always mapped, system users included. - A firewall row keeps its time, action, target, reason category, source and lease. Firewall entries carry no ids, so nothing joins them to actions or findings; the manifest says so rather than guessing from times or addresses.
- Admission rows keep their kind and step, including the retirement of an
unknown outcome (
expired_unproven). An audit row’s target maps like a firewall target, so it joins the legacy row for the same address, and its reason category names the lane. An hourly summary, which setscount, keeps its check, entry, decision, refusal reason and attribution cause, dry-run prediction, count andepochand names no target; a check that matches a learned account, host or domain refuses the run. An attribution cause belongs only on a refusal summary; action inspection rows carrying one refuse the run. Anepochbelongs only on a summary; any other row carrying one refuses the run. - An operation, action, actor, result or source the tool has not been reviewed against refuses the run instead of passing through.
The manifest is written last and is the bundle’s completion marker. It lists every input and output with its stream kind, position, SHA-256 of the exact bytes, record count and time span, the tool’s source revision, the salt fingerprint, join counts, counts of discarded fields, action results by value (admission steps and decisions included, at zero when absent), and coverage for each stream. Check every output’s digest against it; a bundle without a manifest, or with a digest that does not match, is incomplete.
- An action is matched when it names a finding id present in the recording, and nothing more. Actions naming a finding the recording lacks, and actions naming none, are counted apart.
- Repeated finding rows are kept and counted. A durable action row repeated exactly is a retransmission and does not count as another outcome; rows that share an action id and version but differ are reported as conflicting, never resolved by taking the latest.
- A result is what the writer recorded. An applied block or a firewall entry is an observation, not a verified effect or a reviewed correct action.
--input-manifest takes the collector’s inventory of what the host has:
{"v": 1, "streams": [
{"kind": "findings", "availability": "present", "sha256": "<digest of the copied file>", "records": 1200},
{"kind": "actions", "availability": "not_recorded"},
{"kind": "firewall_audit", "availability": "absent"}
]}
Every supplied file must appear as a present entry with its digest and record
count, and every present entry must be supplied. absent means the collector
looked and found none; not_recorded means the host does not keep that
stream. Ledger and review streams can only be stated absent or not recorded.
Without an inventory, a stream that was not supplied is reported as not
supplied.
Comparing the admission preview
While the admission ledger only previews or forecasts, the legacy path
decides every automatic block and the ledger sees the same responses.
Compare a dry-run window: once admission executes, the legacy path leaves
the blocks admission owns to it. compare reads the outputs of one joined
run and sets each legacy automatic block in the preview window against what
the preview did with its finding. Older unstamped finding rows still supply
their identifiers and checks for joins:
/tmp/finding-stream compare --findings host-a/findings.jsonl.gz --actions host-a/actions.jsonl.gz --status host-a/status.json
--status takes the host’s csm status --json output, taken at the end of
the run being judged; compare reads only the ledger’s ingress generation,
which names that run, and the admission owner’s block_entries from it.
Each hourly summary names the daemon run that counted it (epoch), and
compare reports each run apart, since a restart loses the hour in progress.
Summaries recorded before they named their run are reported as one unmarked
run. A run’s window spans its first to its last hourly admission summary;
the hour the first one counts may have begun before the run, so it is not
counted. A block
is matched by one observed attempt for the same finding, response kind and
typed target within an hour; a scan block retried from the pending queue may
land later, until its retry ages out two hours after it was queued. Spray
blocks match through their own entries: the incident spray block of one
address through incident_spray, the mail spray subnet block through
mail_subnet. Matching can reassign compatible decisions when their windows
overlap; each step explains at most one legacy row across all runs. An
unused step from before a restart may still explain a pending retry. The
same legacy row in a shared boundary hour needs only one step, even when
both run reports include it. Observed steps and coalesced decisions are
matched before designed refusals, which explain discrepancies but do not
satisfy coverage. Coalesced decisions and designed refusals each spend one
summary count from their own run for the same entry, kind and check in the
same or an adjacent hour. A block the old process made in the hour of a
restart also falls in the next run’s window; when the earlier run’s own
summaries explain it, the next run reports it as explained by another run,
and it adds no coverage there. When both runs can explain several boundary
blocks, matching assigns distinct rows before spending spare counts on rows
already explained by the other run. Each run still spends only its own
counts and earns coverage only from its own matches. Hourly aggregation
cannot prove the exact event delay. A backward wall step clamps counts to
the admission hour floor; larger clock disagreement stays unexplained and
must be checked through clock status.
Attribution explains only scan-pass provenance gaps for checks served through
the matching entry; policy explains only netblock, permanent-block escalation,
challenge timeout, incident or incident spray paths without a retained root. A rootless
derived path may spend its fixed unknown-check policy count without
a finding link. Other policy and attribution refusals stay unexplained.
Summaries must be stamped at an hour boundary, and only refused decisions
may carry a refusal reason. Counts that overflow refuse the comparison,
and output errors never repeat file paths or host data.
Summary matches are aggregate lower bounds, not proof about one finding. Anything else is listed
as unexplained. The report also totals Invalid and queue overflow refusals,
and names what only csm status and the metrics show: Critical deferrals,
queue evictions and the handoff latency.
Each run ends with its gate; only the run the status names is judged. It
passes once its window spans at least 48 hours, nothing is unexplained, both
totals are zero, and three blocks are matched for every entry the status
lists and for every scan check that blocked, even if a later configuration
change removes scan blocking from the status. An entry short of three, or
listed but never blocking, holds the gate until the run spans seven days;
the gate then passes and names it unproven. An unmarked run, or a comparison
without --status, never passes. The report ends with the verdict on the
status’s run: an earlier run that passed does not pass it.
The operator must also verify the named status and metric criteria. Shape
validation cannot authenticate a prior run: compare accepts outputs only
from the joined run that already passed its identity verifier. The report
describes one host’s responses; keep it private. Summary retention discards
whole oldest hours after failed writes exceed its bound. Missing summaries
may mean idle hours, failed writes or a crash; they cannot certify coverage.
Rootless designed-refusal matches consume aggregate counts and provide no
finding join. The operator must prove collector coverage and an undamaged
ledger through status and doctor before accepting the comparison as passed.
What a recording does and does not contain
The audit log is the dispatch record: it holds findings that were alerted, after deduplication. It is not the persisted latest-state set, which is larger because every scan re-emits the findings it still sees and the merge refreshes their timestamps. On one production host a single sweep dispatched 49 critical findings while the active set carried 157 refreshed critical rows.
A replay therefore understates how full the persisted correlation window gets. Compare windows against each other, and treat an absolute rate from a replay as a property of the replay, not of the live host.
Replaying a stream through correlation
scripts/correlation-calibrate replays a recording through the production
cross-account correlation so its thresholds can be re-derived from what hosts
produced rather than from an assumption:
go run ./scripts/correlation-calibrate \
~/.local/share/csm/finding-streams/host-a.jsonl.gz
It reports how often account-and-check pairs repeat, which checks dominate
the input, and what the aggregates did under both derivations:
per dispatch batch, and over the persisted latest-state set. For each it prints
a threshold sweep using the same eligible, windowed accounts as the aggregate.
--window selects the persisted correlation bound: it defaults to
one hour, and --window 0 reproduces the original unbounded correlation.
Longer windows are applied directly, without the production default limiting
them. The source set keeps the store’s retention and size limit. Batch
correlation counts all qualifying rows grouped into that dispatch batch.
Repeated pairs can include distinct findings on the same account; this metric
does not establish how many rows are re-reports of the same finding.
Batch boundaries are inferred from timestamp gaps. Persisted correlation is recomputed at every recorded arrival, including ignored checks that only advance time. Raised duration is measured between those observations, not against an invented scan schedule. Recordings do not contain empty scans, purges or dismissals, so this replay models latest-state accumulation rather than reconstructing every store transition. Rows with no timestamp are counted and skipped because they have no replay position; runtime correlation still counts unstamped stored rows.
The tool needs no host access and reads nothing but the recording. It uses cPanel account roots supplied directly to correlation, without platform discovery on the replay machine.
Replaying scan admission
scripts/response-replay replays a recording through a model of how
automatic scan blocks are admitted today: the hourly block limit, the retry
queue and its age limit, and eviction at the temporary deny limit. It reports
what the model would have done with the recorded findings, not what the host
did, and not whether any block was right.
go build -o /tmp/response-replay ./scripts/response-replay
/tmp/response-replay --findings host-a/findings.jsonl.gz --out replay/host-a.json \
--max-blocks-per-hour 200 --deny-temp-ip-limit 500 --seed 1 --hour-zone UTC \
--manifest host-a/manifest.json
- The live path tries queued candidates in Go map order. The model tries them
in a reproducible random order chosen by
--seed, so compare several seeds rather than reading one run as the order the host used. --hour-zoneis the zone the host’s clock runs in: the hourly limit resets when that zone’s hour changes.Localis refused because it would depend on the machine running the replay.--php-relay-ip-recipient-modemirrorsemail_protection.php_relay.ip_recipient_modeand defaults toobserve. Useenforceto model form-relay client responses. Observed findings never drive a response in either mode. The report records the selected mode.- A recorded block from a challenge timeout, central intel, a credential spray or an incident is applied as recorded: it takes a firewall slot but not the hourly budget. Only a single IP address is accepted as the block target; malformed targets are counted as unclassified rows. A permanent block is counted, not modelled.
- The audit log does not record a finding’s structured source address, so
a reputation finding, which names its address only there and in its
message, has no address in a replay by default and is counted as missing
one.
--reconstruct-reputation-sourcerecovers the address from that message’s fixed form; the report then counts the recovered rows and lists the reconstruction among its assumptions. - A finding keeps the severity it was recorded with. A later change in how a check grades severity is not applied, so an older recording can replay as blockable a finding the current build would only challenge. Every report lists this among its assumptions.
- Batches are inferred from equal timestamps. Recordings hold no empty scans, so queued work drains only when another finding arrives. Rows without a timestamp are counted and left out.
- The report lists the effective policy with every default it applied, the counts, queue delay and eviction residence distributions, blocks per elapsed hour, and the assumptions and mechanisms the model leaves out: infrastructure and allowlist protection, verdict callbacks, subnet blocks, permanent escalation, retries and failures, and manual unblocks. It carries no address, id, name or text from the recording.
- Candidates the queue dropped (aged out or overflowed) are reported apart from work still queued when the recording ended, which the end of the recording cut off rather than the queue.
- The demand is the audit log’s record of dispatched findings, not every finding the admission path was called with.
- The report counts the recorded subnet, ASN-crawl, netblock, permanent and dry-run block rows the model leaves out, and the findings that feed the subnet paths. ASN-crawl subnets spend the same hourly budget the model gives single addresses, so free scan slots in a replay are an upper bound.
- Hourly distributions include empty elapsed hours across the full recorded time span, without allocating a sample for each empty hour. The first stamped row establishes the replay clock, including dates before year one.
- With
--manifest, the report checks that the manifest describes this exact recording and carries its coverage and recorded outcomes beside the replay. Without one, the report says the recording has no statement of what was collected with it. Negative outcome and join counts are refused. - A report is written only to a new path: it never replaces an existing file, and it cannot alias its recording or manifest through symbolic links, parent-directory traversal or hard links. It is staged beside the resolved destination and published as a private file, refusing the run if another writer creates the destination in the meantime. A temporary file cleanup failure is reported with whether the report was published.
- Like a manifest, a report needs a build of a known commit without local changes.
Crawl Detector Calibration
The distributed crawl detector judges each site and URL pattern over a window of minutes: it removes the heaviest clients, then asks whether the remaining traffic is both far above the pattern’s usual rate and spread over enough distinct clients. Every one of those settings, and the memory bounds behind them, must come from what real hosts served, not from guesses. Two tools turn copies of a host’s access logs into that evidence without retaining raw visitor identities. Pseudonymous recordings are still private. This revision validates bundles and certified coverage and replays them through chronological sessions, but bounded experiments with executable acceptance limits are not yet implemented, so it is still for synthetic data only. Do not convert private recordings, delete preserved recordings on the strength of these outputs, or use reports to choose detector settings until those contracts pass their tests and independent review. Real-data collection and conversion then require separate operator approval.
scripts/domlog-stream converts local copies of domlogs into an anonymized
record stream. The following example is for synthetic fixtures only, from a
clean committed checkout; use absolute local log-copy paths in the inventory:
# RECORDING_DIR is a mode-0700 synthetic scratch directory outside the checkout.
# Set WINDOW_MINUTES from the synthetic grid; do not reuse a real recording salt.
go build -o "$RECORDING_DIR/domlog-stream" ./scripts/domlog-stream
"$RECORDING_DIR/domlog-stream" convert \
--salt-file "$RECORDING_DIR/synthetic-salt" \
--inventory "$RECORDING_DIR/inventory.json" --labels "$RECORDING_DIR/labels.json" \
--bot-evidence "$RECORDING_DIR/bots.json" \
--out "$RECORDING_DIR/host-a.records.jsonl.gz" \
--volume-out "$RECORDING_DIR/host-a.volume.jsonl.gz" \
--manifest "$RECORDING_DIR/host-a.manifest.json"
Every line goes through the detector’s own record parser and request identity rules, with the same canonicalization. Deployed log escaping and proxy semantics still need qualification before real recordings can be trusted. A record keeps the logged time and order, status, whether the request is dynamic or an expensive query, the Referer class (none, malformed, cross-site or same-site), a claimed bot identity with its verified-bot proof class, an infrastructure flag and the operator’s label. Sites, accounts, client bindings (the full IPv4 address or the IPv6 /64), episode names and URL patterns become salted pseudonyms that keep equality and the site/pattern hierarchy. Two distinct names that would share a pseudonym stop the conversion instead of being merged. No path, query value, address, user agent or Referer is written. Sites and accounts use the same pseudonyms as recorded finding streams under the same salt, so the two can be joined. A volume stream counts lines and bytes per site and minute, including lines without a usable target or client.
The inventory names the recording period, each site’s verified identity and
its log copies. The period is bounded by whole UTC minutes, from inclusive and
to exclusive, and must have ended. Site names and aliases must be lowercase
DNS names without a terminal dot, at most 253 bytes each, and one host
belongs to one site: an alias listed twice or naming another site is
refused. Use absolute log paths: the tool also accepts a relative path, but
resolves it against the directory it runs in, which ties the inventory to
that directory:
{
"period": {"from": "2026-09-20T00:00:00Z", "to": "2026-09-27T00:00:00Z"},
"sites": [
{"name": "example.com", "account": "acct1",
"aliases": ["example.com", "www.example.com"],
"logs": ["/srv/crawl-recording/raw/example.com",
"/srv/crawl-recording/raw/example.com-ssl_log.gz"]}
],
"trusted_proxies": ["198.51.100.9"],
"infrastructure": ["192.0.2.200"]
}
The inventory, labels and bot evidence files must be regular files, not symlinks, devices or FIFOs, in their exact JSON form: every member spelled exactly and given once, no null values and nothing after the document.
Verified-bot identity comes only from the host’s own verified-bot list. The
optional bot evidence file is an export of that list for the recording
period: the list’s source revision and configuration digest, then published
ranges and reverse-DNS verdicts, each for one identity and valid from from
(inclusive) to to (exclusive):
{
"format_version": 1,
"d2_revision": "<40 or 64 hex digits>",
"config_sha256": "<64 hex digits>",
"proofs": [
{"bot": "googlebot", "kind": "range", "prefix": "203.0.113.0/24",
"from": "2026-09-20T00:00:00Z", "to": "2026-09-27T00:00:00Z"},
{"bot": "googlebot", "kind": "dns", "addr": "192.0.2.60", "verdict": "positive",
"from": "2026-09-20T00:00:00Z", "to": "2026-09-21T00:00:00Z"}
]
}
The proofs list is required; use [] for an explicitly empty export.
The converter looks nothing up. A claimed identity is range when a range of
that same identity held the client at the logged time, else dns or
negative from a verdict for that exact address; a user agent alone, another
bot’s range, an expired proof or contradictory verdicts leave the claim
unverified. The manifest records the evidence digest and its list revision.
Bundle validation refuses a proof without that reference or a client binding.
A bot the host verifies only through operator-configured user-agent
substrings (reputation.verified_bots) has no claimed identity here: its
requests are ordinary traffic in the bundle, and bot-label qualification
must account for them separately.
Aliases decide which Referers count as same-site; a www host counts only
when it is listed. Behind a trusted proxy the client is the rightmost
forwarded address only when the qualified proxy appends the original client
there. A proxied line without a usable forwarded address, or whose rightmost
address is another trusted proxy (an unqualified second hop), counts as
attribution loss; a peer that is not a plain address counts as an invalid
client. Neither gets a client binding. The optional labels file marks time ranges of a
site as attack or overload (each with an episode name) or healthy,
optionally only for a first path segment or parameter-name prefix. Each rule
requires nonzero RFC 3339 from (inclusive) and to (exclusive) timestamps;
matching preserves fractional seconds even though stream timestamps use
whole Unix seconds. Prefixes match canonical query names, with ASCII letters
lowercased and non-ASCII case preserved. The first matching rule wins and
anything else stays unlabeled. Labels are what the calibration measures
detection delay and false positives against.
Handling rules for later qualified real-data use (not authorized by this prototype):
- The operator copies the logs read-only into a private local directory
(mode 0700) after explicit collection approval. Take each copy no earlier
than the end of the recording period plus the lateness bound, so requests
that complete after the last minute are already written; when a copy was
taken earlier, exclude the minutes it cannot vouch for as
partial_minute. Run the tool, review coverage against those copies and independently recorded handler/logging liveness, then let the operator delete the approved copies. Nothing runs on the monitored host. - The salt is created on first use with mode 0600 and must stay private;
reuse the finding-stream salt so the streams join. Keep one identity
registry per salt, in the salt’s directory. The tool derives its filename
as
registry-<salt SHA-256>.json, using the full lowercase hex digest of the salt bytes. Optional--registry FILEmust name that same file; another filename or directory is refused, even with--new-registry. Directory symlink aliases are accepted. It records every site, account and episode pseudonym the salt has issued, as keyed digests, and refuses a later bundle whose different name would take one; it is locked while a conversion runs. - A run that creates the salt starts its registry. For a salt that already
exists but never had a registry, such as the finding-stream salt, pass
--new-registryon the first conversion only; it is refused once a registry exists. Without that flag, a missing registry stops the run instead of starting an empty one that would forget issued pseudonyms. Back up the salt, registry and persistent.lockfile together, and restore them together. Never remove or replace the lock during conversion. If an earlier tool wrote a custom registry filename, stop conversions and move that registry and its lock to the derived names before reusing the salt. Do not start an empty registry to replace existing history. - The tool refuses to run from an unknown or modified build, refuses to replace an existing output, and publishes nothing unless every log was read. Each log copy must be a regular file that stays unchanged while it is read. Reads stop at its initial size; size, modification time and change time must still match afterward, so restoring the modification time cannot hide a rewrite. Snapshot checks support Linux and macOS; other platforms refuse conversion. A second path or hard link to a copy already read, or a copy whose decompressed content repeats another, is refused.
- The manifest, published last, records the digests of every input (as read and decompressed) and output, the tool revision, the period and, per site, the observed extent and the bytes read. Every line read is either a record or exactly one of: oversized, refused by the parser, invalid time, time in the future, time outside the period, or a final line without a newline, which is incomplete and never a request. Records without a usable target or client are counted too. Record bytes are in the volume stream; every other byte is counted as unplaced. For lines without a usable time, the manifest keeps the logged times of the timed lines around them, which bounds where the lost requests belong.
- The manifest also lists every site, account and episode pseudonym of the bundle with the full keyed digest it is a prefix of. Bundles replayed together must agree on every digest, so bundles converted against a forked or restored registry cannot merge two names unnoticed.
- The first and last timed lines describe an observed extent, not proof of complete minutes, and the manifest keeps them apart from coverage. Only a coverage proof certifies minutes. An idle log alone cannot certify zero traffic.
- Streams, labels, inventories and reports stay outside the repository.
scripts/crawl-calibrate replays an experiment through exact models of the
detector. Build it from a clean committed checkout as well:
go build -o "$RECORDING_DIR/crawl-calibrate" ./scripts/crawl-calibrate
"$RECORDING_DIR/crawl-calibrate" --experiment "$RECORDING_DIR/experiment.json" \
--out "$RECORDING_DIR/experiment.report.json"
The experiment file lists the bundles in chronological order, the parameter sets to replay and the truth of every scored episode. Paths are relative to the experiment file’s directory unless absolute. Like the inventory, it must be a regular file in its exact JSON form:
{
"format_version": 1,
"identity_version": 1,
"window": 10,
"runs": [
{"params": {"w": 10, "r": 3, "f": 5, "k": 20, "d": 50, "c": 80,
"baseline": {"alpha": 0.1, "min_obs": 1, "min_age": 10080, "floor_per_min": 1}},
"sketch": {"m": 64, "h": 128, "seed": 7}}
],
"fixtures": [],
"bundles": [
{"manifest": "week1.manifest.json", "records": "week1.records.jsonl.gz",
"volume": "week1.volume.jsonl.gz", "coverage": "week1.coverage.json",
"role": "training",
"states": [{"from": 29840820, "to": 29850899, "state": "normal"}]},
{"manifest": "day8.manifest.json", "records": "day8.records.jsonl.gz",
"volume": "day8.volume.jsonl.gz", "coverage": "day8.coverage.json",
"role": "scoring", "score": [{"from": 29850900, "to": 29852339}],
"states": [{"from": 29850900, "to": 29852339, "state": "normal"},
{"site": "dom-0a1b2c.example", "key": {"level": 2, "key": "k-0123456789abcdef"},
"from": 29851500, "to": 29851560, "state": "protected"}]}
],
"truth": [
{"episode": "e-0123456789abcdef", "label": "attack", "site": "dom-0a1b2c.example",
"keys": [{"level": 1, "key": "k-1111111111111111", "parent": "k-0123456789abcdef"}]}
]
}
Bundles must share one salt and identity contract, follow one another in
time without overlap, and agree on every identity digest; a disagreement
means their identity registries forked or one was restored from an older
copy, and the experiment is refused. Each bundle is read once and refused
if its manifest is not exactly what domlog-stream wrote, a site is not a
unique pseudonym, or any row, digest or total disagrees with the manifest.
Without parameter sets the report holds only volume, silence and shape
diagnostics. State declarations and truth definitions are validated even
without runs. Replaying needs a coverage proof for every bundle, which the
operator builds from evidence independent of the logs: the collection record, handler and logging
liveness, and a measured bound on how long after its logged time a request
can be written. The proof names the manifest by digest and tiles the
recording period for every site with certified spans and reasoned
exclusions (partial_minute, collection_gap, liveness_unknown,
topology_change). It cites evidence only by digest:
{
"format_version": 1,
"manifest_sha256": "<SHA-256 of the manifest file>",
"lateness_seconds": <measured completion-delay bound>,
"evidence": [
{"kind": "collection", "sha256": "<digest>"},
{"kind": "logging_liveness", "sha256": "<digest>"},
{"kind": "lateness", "sha256": "<digest>"},
{"kind": "pre_application", "sha256": "<digest>"}
],
"sites": [
{"site": "dom-0a1b2c.example",
"spans": [{"from": 29840821, "to": 29840838}],
"excluded": [{"from": 29840820, "to": 29840820, "reason": "partial_minute"},
{"from": 29840839, "to": 29840839, "reason": "partial_minute"}],
"rejects": [{"from": 29840827, "to": 29840827, "category": "no_target",
"lines": 1, "evidence": "<digest of the pre_application entry>"}]}
]
}
A replay uses only certified minutes without unknown loss. A minute with a
targetless or unattributed record is removed, and so is every minute a line
without a usable time may belong to: the minutes between the timed lines
around it, widened by the lateness bound. A rejects entry keeps those
minutes only when independent evidence shows that exactly that many
targetless or oversized lines were refused before the application ran; the
HTTP status alone is not such evidence. Removed minutes break windows like
any unknown minute and never train a baseline. These bounds assume each copy
lists requests in the order they completed. The manifest records, per copy,
how far any logged time trails an earlier one; that disorder is a lower bound
on completion delay, so a proof whose lateness bound is smaller is refused.
The declared disorder must also cover inversions between the timed neighbours
of lost lines, including neighbours outside the recording period.
A bundle without a proof contributes diagnostics over each site’s observed
extent only, and the report says its coverage is unqualified.
Each parameter set replays one session through all bundles in order. Every
minute is judged against the history learned before it and only then
learned, so held-out minutes never inform earlier ones; windows run on
across adjacent bundles, and a gap between them restarts every window. A
minute trains a baseline only when its state is declared normal, no
finding is active on the key, and either a complete window judged its traffic
or the minute was observed silent. Idle keys learn those covered normal zero
minutes on reactivation, once only; unknown minutes never become zeros. A
minute with no declared state is unknown and never trains. protected,
degraded and recovery_hold minutes are frozen the same way whether the
protection was applied or only decided. A key’s own declaration wins; otherwise the nearest ancestor’s applies
(an L2 key’s covers its L1 keys, the site key’s covers every key), and then
the site’s. When a finding starts, the key’s hour-of-week profile and its trust
are pinned until a complete window clears it; a coverage gap leaves the
finding active but uncertain, and it never counts as a new finding.
The session snapshot API preserves findings across restores. Changing the identity contract also drops old key declarations and non-finding keys; changing baseline semantics resets history and learning watermarks and re-pins active findings without changing their start minute. Either change, or a changed window, sketch mode, size, seed or hash version, requires a complete new window. Threshold and shuffle changes keep compatible evidence and pinned history. A snapshot made immediately after invalidation can be restored again with the same result.
A High transition is the only finding event. Only transitions in a
scoring bundle’s scoring spans are reported; findings still active from
training appear as active when scoring began. An episode is detected by the
first transition in a scored minute at a key its truth entry names, whose
window holds the episode’s requests; ancestor findings that were already
active are listed, never credited. Every labeled episode in a scoring bundle
needs a truth entry for its site. A window’s majority label is only a
suggested class for review.
The report records its provenance: the experiment and calibrator digests, the salt fingerprint and, per bundle, its role and scoring spans, manifest and proof digests, converter revision, period and bot evidence. Per bundle and site it records certified and covered minutes, excluded minutes by reason and every line count. It also holds host lines and bytes per minute over the observed extent within the recording periods; gaps between bundles contribute no zero-traffic samples. It also holds line counts for nonempty site-minutes, in-file timestamp disorder across all validated records (including excluded minutes) and how many patterns and clients are active per window. Diagnostic windows continue across adjacent bundles and restart at coverage gaps. Key churn counts each site key only once across the experiment and combines samples in the same UTC hour.
For every parameter set it lists each scored transition with its margins, scope, window labels, suggested class and credited episodes; per site and UTC day the scored and judged minutes, requests by label, transitions and credited transitions; each truth episode’s outcome, detection delay, margins and scope match; key and client peaks over scored minutes; and, with a sketch, an independent exact session’s outcomes, lost or extra decisions and lost findings, beside how far the bounds fall below the exact values of the same window. These sketch comparisons use only scored minutes; both sessions still learn from training and other unscored minutes. Fixtures add synthetic attacks from one-, three- and twenty-request clients with optional padding and churn, replayed cold and scored by suggestion.
Episode onset remains the first labeled request across the experiment that
the detector counts, even when its minute is excluded; infrastructure and
static requests inside a labeled range never move it. If a later segment
introduces the first counted request, findings active at onset are captured
again at that request’s minute. An episode with no counted request keeps its
first request as onset and is never scored. An undetected episode
that began outside the scoring spans, or none of whose counted requests was
replayed in a scored minute, is reported as not_scored, not as a qualified
detector miss; one whose later requests were replayed and scored is a miss.
Relevant anomalous windows keep their margins even when an already active
finding prevents a new detection.
Only covered records contribute to windows, baselines and
detections.
The silence diagnostic joins adjacent spans but stops at an unknown gap. A silent run alone does not certify zero traffic.
A replay is a hypothesis about the detector, not a record of what the host did: logs hold completed requests, not offered load, backend harm or queued work, and the parser is only as good as the server’s log escaping. After qualification, chosen values and their evidence go to a private calibration ledger. Episode delays are offline event-time delays: they exclude request completion, log flush, lateness watermark and tick delay, so they are not correct-finding deadlines on their own. The packed footprint estimate is not measured memory or disk usage.
Qualification must use explicit complete-minute coverage, independently verified zero-traffic minutes, separate training and scoring intervals, and held-out healthy, attack and legitimate-overload cohorts. Warm seasonal profiles require history and observations for every enabled seasonal slot. A cold-start replay is a separate experiment. Neither a model delay nor logged timestamp disorder measures request completion or log-flush delay; disorder is only a lower bound on it.
Review every transition per site, key and day. Majority labels are a summary, not the definition of a correct finding or permission to ignore mixed traffic. The tools model High findings only; absence of Critical or actions here is not an end-to-end safety test. Full-bound resource measurements and a private ledger approval are required before production implementation. The production components must repeat these measurements with their own allocations and I/O.
A cleanup failure returns a fixed error. If the filesystem refuses removal, private temporary or output files can remain; retain them for operator-approved cleanup and do not retry at those paths or consume them as accepted outputs.
The identity registry reserves site, account and episode pseudonyms across all bundles under the same salt. Key and binding pseudonyms are 64 bits and are checked within each bundle only; registering every client and pattern would grow the registry with all traffic ever converted. The registry and its lock must be private regular files; inconsistent digest prefixes are refused. Registry replacement is synced to its directory before bundle publication. Keep the registry and its lock file together; never replace or remove the lock while a conversion is running. A failed publication may leave conservative identity reservations. Keep those reservations on retry instead of rolling the registry back.
A site with no independently certified minutes remains in the bundle: use an empty certified-span list and reasoned exclusions covering its whole period. It contributes no replay windows, while other qualified sites remain usable.
cPanel release tests
Version tags require either a cPanel integration image or a recorded reason for
releasing without one. release-preflight runs before builds, and integration
repeats the preflight before allocating servers. The tag-specific publication
dependencies require that integration succeeds; package registry publication,
repository publication and releases cannot use an AlmaLinux/Ubuntu-only result
unless the omission was acknowledged. Main-branch integration remains manual and
may run without cPanel when no image is configured.
Releasing without cPanel coverage
CSM has no licensed cPanel image while no cPanel licence is available for
disposable CI clones. Set the protected variable CSM_RELEASE_WITHOUT_CPANEL
to a sentence stating why, for example no licensed cPanel image available.
A reason must have at least 12 characters after trimming, contain at least two
words, and fit on one line. Blank values and bare flags such as 1 are rejected.
The reason is published in the pipeline log and in
dist/cpanel-release.json as cpanel_coverage: "absent".
Do not read such a release as cPanel-tested. WHM plugin installation, mail integration, cPanel platform paths, service confinement under a real cPanel layout, and upgrade behaviour are unvalidated in that pipeline, and the findings that depend on them (F19, F22, F25) stay open. Clear the variable as soon as an image exists.
The CSM release maintainer owns the CI variables and image refresh schedule.
The PidginHost cloud image administrator owns capture and publication of the
private reusable image. The release maintainer must verify a clone before
setting INTEGRATION_CPANEL_IMAGE to its immutable ID or versioned slug.
Do not set it to a base AlmaLinux image: the package test requires a working
cPanel installation and fails if it is absent.
Image preparation
- Allocate a dedicated AlmaLinux 9 x86_64 build VM with the CI SSH key.
The default test plan is
cloudv-2(8 GB RAM, 80 GB disk); override withINTEGRATION_CPANEL_PACKAGEif needed. Follow the supported OS, hostname, storage and licensing requirements in the cPanel installation guide. - Install full cPanel/WHM using the official installer. Complete initial setup with EA4 Apache, Exim, Dovecot, rsyslog and a valid license arrangement for disposable clone IPs. DNSOnly is insufficient. Do not install CSM or create tenant accounts on the image.
- Verify
systemctl is-active exim dovecot,exim -bV,doveconf -n, and the WHM AppConfig tools./var/log/maillogmust exist. Ensure cloud-init preserves a valid cPanel hostname and injects the dedicated CI SSH key into the existingphuseraccount with passwordless sudo. - Have the cloud image administrator capture the VM using the provider’s image preparation process. Remove build credentials and tenant data; regenerate host keys and machine identity on clones. Record the OS, cPanel version, image build date and responsible maintainer in the image metadata. Never publish an image containing a customer backup.
- Publish the private image under an immutable ID or versioned slug and
verify that the CI account sees it in
phctl compute image list. The current CLI can create VM snapshots but cannot promote them to reusable OS images; that step requires cloud image administration access. - Set the protected GitLab project variable
INTEGRATION_CPANEL_IMAGE. Run the manual integration job once these checks are on main before relying on the image for a tag release.
Refresh the image for cPanel/OS security updates and review it at least monthly. License activation, SSH injection, empty CSM state and mail service readiness must work on a new clone, not only on the image-build VM.
What the release job checks
The cPanel VM starts without CSM. The job downloads the previously released
RPM pinned in build/integration-upgrade.json and verifies its SHA-256. It
installs that version, writes operator configuration and a state sentinel,
starts its daemon, then upgrades to the RPM built by the current pipeline.
It checks that configuration, drop-ins and state survive and that the installed
binary has the exact SHA-256 of the current build artifact.
The candidate’s package-mode installer is exercised on that cPanel host, and the service is restarted using its installed unit. Dedicated tests require:
- cPanel and Exim platform detection with existing platform-derived paths.
- Working Exim and Dovecot services and configuration commands.
- Executable WHM CGI, registration reported by the read-only WHM application-list API, and the expected redirect.
- A running notify service with strict filesystem protection, healthy state, and an attached mail-log watcher.
The ordinary real-system integration suite then runs on that same host and on the AlmaLinux and Ubuntu hosts. No test sends customer mail; alert delivery and automatic response are disabled in the integration drop-in. The cPanel package test only runs with an explicit disposable-server marker and refuses an image with CSM or its state already present.
Update the baseline pin deliberately after reviewing the released RPM and its
published SHA-256. Never resolve a moving latest package during this test.
Evidence and cleanup
Integration retains cPanel package output, service/journal diagnostics,
baseline and candidate status snapshots, and cpanel-release.json for one
year. The JSON identifies the candidate commit, binary/package hashes,
image, plan and upgrade baseline. Its package_checks field covers the
package phase; the integration job must also pass the later real-system tests
and verified server cleanup before publication is allowed.
Server IDs are recorded immediately after creation. Verified cleanup runs
before successful completion, with an after_script retry for errors or
cancellation. A failed package check retains diagnostics and still reaches
that cleanup. Image preparation and a successful live cPanel job are required
operational steps; compiling the integration binary locally does not replace
them.
The candidate daemon also enables and removes the forward guard through a test drop-in and service restart. This runs the actual cPanel rebuild through the packaged service sandbox and checks that removal preserves operator configuration.
Production build and kernel tests
The default suite tests the portable stubs. test:production also runs the full
repository with yara,journal,bpf, the tags used by shipped Linux binaries.
It uses the release builder’s YARA-X C library and verifies that its version
matches both the Go module and the builder recipe. PHP 8.2 is installed for
runtime tests. The same job runs the pinned linter and security analyzer with
all shipped tags; scanning APIs retain documented exceptions for intentional
reads of caller-selected local files. Rule trust checks cover every accepted
filename extension, regardless of letter case. Run the same gate in a suitable Linux image with:
scripts/production-tests.sh portable
Each gate writes the selected test inventory, Go JSON test events, engine
versions, and commit ID under production-results/. The inventory comes from
go list and the selected Go test files across all packages. A newly added
package or tagged test is included automatically. Verification rejects missing
tests, package failures, and skipped required regressions. Other environment
skips remain visible in the JSON events. These are not kernel coverage.
Kernel runner
test:kernel is a required job on a dedicated Linux shell runner tagged
csm-kernel. Provision an ephemeral VM for each job, locked to this project,
with local Docker, cgroup v2, BTF, BPF LSM enabled in the boot-time LSM list,
BPF ring buffers, fanotify, and nftables. Give it access to pull the pinned
builder image. It must have no unrelated workloads or host credentials.
The ordinary Kubernetes runner does not satisfy this contract automatically.
Provision it from the cloud catalogue’s alma9 image (AlmaLinux 9, x86_64),
the closest available match to the EL production family; alma10, ubuntu26
and debian13 also satisfy the contract. Enable BPF LSM before registering the
runner, since it is not in the default boot-time LSM list:
grubby --update-kernel=ALL --args="lsm=capability,yama,selinux,bpf"
reboot
cat /sys/kernel/security/lsm # must list bpf
stat -fc %T /sys/fs/cgroup # must print cgroup2fs
Register with gitlab-runner register --executor shell --tag-list csm-kernel,
lock it to this project, and disable it for other projects.
This runner does not reproduce the production kernel. Supported hosts run
CloudLinux 8 and EL8 on 4.18; no catalogue image offers that kernel. Treat a
passing test:kernel as evidence for a 5.14-or-newer kernel only. Capabilities
that differ across those kernels – pidfd_open is present on the runner and
absent on 4.18 – are probed at runtime instead and reported by csm doctor
and the health status.
The job builds build/Dockerfile.production-test on the release builder and
uses scripts/go-linux.sh to boot systemd inside a disposable container.
GO_LINUX_PRIVILEGED=1 is an explicit Docker-only test option; the wrapper’s
normal capability set remains the default. Use the same setup locally only on
a dedicated Linux test VM:
docker build --build-arg BUILDER_IMAGE=<release-builder-image> \
-f build/Dockerfile.production-test -t csm-production-test .
GO_LINUX_RUNTIME=docker GO_LINUX_PRIVILEGED=1 \
GO_LINUX_IMAGE=csm-production-test scripts/go-linux.sh \
bash scripts/systemd-account-roots-test.sh production
Systemd runs the packaged service sandbox regression and a second test service.
The second service executes the shipped tags plus nftkernel,kernelintegration.
Its inventory selects every test added by those tags across all packages, plus
explicitly required attachment tests. New kernel test names need no prefix.
Required checks include real nftables transactions in isolated network
namespaces, kernel-produced BPF ring events, AF_ALG BPF attachment and shutdown,
and journal delivery from two actual services followed by reader cancellation.
The journal test covers both empty and existing history, excludes unrelated
services, and rejects replayed records.
Both the CLI and service test binary use the shipped tags.
The kernel test service has an explicit Go workspace and toolchain selection;
EL8 system services may start without a home directory. Once tests stop and
their results and journals are saved, the collector exits the disposable
systemd manager directly with the test result, avoiding EL8 exit-target loops.
The first service verifies remediation and restore under the packaged unit,
custom account-root grants, doctor checks, and process-handle signaling.
It does not exercise the daemon’s complete watcher startup; the cPanel package
gate and cloud integration cover that separately.
Missing capabilities fail the kernel gate. A skipped AF_ALG attachment test is not accepted. Both production and kernel jobs retain artifacts for one year and are explicit dependencies of publication, including tag-specific dependencies. A missing kernel runner leaves the job pending and publication blocked.
The separate clean application gate provides pinned vendor corpus measurements. cPanel release tests provide package, upgrade, and primary-platform validation. None substitutes for another.
The journal reader starts after existing matching records and also follows services with no prior records. Package integrity rechecks preserve a reported mismatch for the original finding even if that file is now absent or no longer executable; current file mode is not evidence that the modification was repaired.
Required Queue inventory
Both runner modes validate scripts/queue-inventory.json before test selection
and execution. Its reviewed owners cover channel handoffs, durable work,
coalesced pending work and kernel buffers. Each owner records its health rows,
bounds and required publication and lifecycle regressions. The runner writes
the union of those regressions and its existing requirements to
production-results/<mode>/queue-required.json, then requires actual passing
events for every selected requirement.
The portable baseline in scripts/production-required.json also requires the
shared accounting, health reporting, doctor and gate regressions. Keep recovery
and concurrency regressions with their owner when they test an owner-specific
boundary; keep shared contracts in the baseline. A passing full suite alone
does not protect an omitted requirement against a later skip.
PHP relay publication requirements start the Linux wiring with temporary filesystem paths and a real state store. They check live registration, persistence, shutdown evidence and failed watcher attachment. They must also reject removal of either production registration; manually registering a provider in a test only proves that provider’s behavior.
The allocation scanner is available for ownership reviews:
go run ./scripts/queuegate -scan > queue-allocations.json
It scans repository Go source across all build constraints, excluding test files,
testdata, hidden directories and vendored dependencies. A directory removed
during the scan, such as a test’s scratch folder, is skipped; a missing scan root,
unreadable directory or failed source read still fails the gate. It records raw
channel allocations and imported queuehealth.NewChannel calls, including renamed and
dot imports. Named channel aliases resolve across every package variant in a
repository import directory; an alias that is a channel in only some variants fails
explicitly. Imported named types use the Go toolchain’s source importer.
Unresolved types, reflective allocation and indirect references to the accounted constructor fail explicitly.
Supporting a new constructor or generic constraint requires scanner tests and
an ownership review first.
Allocation identities use the file, enclosing function, assignment target,
constructor kind and ordinal. The descriptor includes the allocation expression,
expanded repository constants and assignments or local var initializers for
the capacity and its local inputs in the enclosing function. Expression grouping
and build-variant values are retained, including array lengths, literal indices and slice bounds.
Implicit constant declarations, iota, closures and named composite literals
in capacity expressions need explicit scanner support and are rejected, including
when nested inside field selections. Import aliases are resolved as package
names, including aliases that match predeclared identifiers. Runtime
collection lengths stay symbolic; only explicit construction and slice bounds
enter the local capacity-input graph. The scanner does not infer arbitrary
function bodies or type layouts and is not whole-program data-flow analysis.
A version 1 manifest contains allocations and owners. Each allocation copies
its scanner descriptor and adds a reviewed class, rationale, and queue
owner where applicable. Classes are work, lifecycle, maintenance, and
constructor. A semaphore or completion signal can order data stored elsewhere;
trace that data before deciding whether it belongs to a work owner. Every work
allocation must reference an owner. Changes, omissions and stale descriptors
fail validation.
PHP Shield eval-site admission and result channels belong to the
php_shield.eval_sites work owner: the ownership proof determines the grade
of a finding held by the event reader. Both modes require publication,
timeout, result retention, recovery and fail-closed grading regressions.
The advisory health row retains a timed-out filesystem walk and a completed
result until the caller and walk both settle. A timeout or refused proof
counts as lost grading work while the finding stays High.
Each owner records health rows, reviewed bounds, and separate publication
and lifecycle evidence lists. An evidence entry names a package, a top-level
Go test and its required portable or kernel mode. Owners without a channel
allocation need source anchors: a path, symbol and canonical shape. Supported
anchors include struct fields, types, variable or constant declarations, and
function signatures. Missing or changed anchors fail validation.
For a reviewed manifest, generate requirements for the existing test verifier:
go run ./scripts/queuegate -manifest scripts/queue-inventory.json \
-mode portable -base-required scripts/production-required.json \
-required-out queue-required.json
Use the resulting file as scripts/testgate -required with the selected test
inventory and actual Go JSON test events. It combines existing required tests
with the owner’s evidence for that mode. A missing, skipped or failed required
test is not accepted. A passing scanner or a test name in JSON alone does not
prove health publication or lifecycle coverage; the named tests need substantive
assertions and execution evidence. Runner regression tests exercise both modes
with real test events, including changed capacities, unclassified allocations,
missing tests and skipped required evidence.
The scanner cannot discover arbitrary queues held in maps, heaps, durable storage, kernel buffers or dependencies. Those need explicit owner entries and source anchors, plus tests that drive real admission, progress, loss and cleanup. Adding a new non-channel owner remains a code-review responsibility. No syntax inventory substitutes for reviewing the behavior of its required tests.
Release Signing
CSM has two separate signing paths:
- Package repository signing for the normal APT/DNF install path.
- Detached Ed25519 artifact signatures for raw binaries, tarballs, and package files downloaded outside the package manager.
Do not reuse keys between these paths. The package repositories use GPG because APT and DNF verify repository metadata that way. Detached release signatures use Ed25519 because the standalone install and deploy scripts verify raw artifact bytes with OpenSSL.
Status
| Surface | Key type | CI variable | Notes |
|---|---|---|---|
| APT repository metadata | GPG | CSM_GPG_SIGNING_KEY | Published by repo:publish; operators install with signed-by=/etc/apt/keyrings/csm.gpg. |
| RPM packages and repository metadata | GPG | CSM_GPG_SIGNING_KEY | Published by repo:publish; operators use gpgcheck=1 and repo_gpgcheck=1. |
Raw binaries, tarballs, .deb, .rpm siblings | Ed25519 | CSM_SIGNING_KEY | Detached .sig files for direct downloads and standalone scripts. |
| YARA Forge rule ZIPs | Ed25519 | CSM_SIGNING_KEY | Signed by the yara-forge-mirror job; clients verify via signatures.signing_key. |
The preferred operator path is the signed APT/DNF repository documented in Installation. Standalone scripts also verify detached signatures: scripts/install.sh embeds the Ed25519 public key in EMBEDDED_SIGNING_KEY. Override it at runtime with CSM_SIGNING_KEY_PEM. Current releases fail before execution if the key, signature, or compatible verifier is unavailable. Setting CSM_REQUIRE_SIGNATURES=0 does not bypass verification for current releases.
Public Key
The same Ed25519 key signs release artifacts and YARA Forge rule ZIPs.
Hex form, for signatures.signing_key in CSM config:
2d1472b2a1d9728c2717b75111487145a7863f7ce731c1b44181f7a68bb908f7
PEM form, for standalone script verification (EMBEDDED_SIGNING_KEY / CSM_SIGNING_KEY_PEM):
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEALRRysqHZcownF7dREUhxRaeGP3znMcG0QYH3pou5CPc=
-----END PUBLIC KEY-----
Package Repository Signing
repo:publish runs on version tag pipelines and rebuilds the public package repositories from the current tag plus the retained historical releases.
Required protected CI variables:
| Variable | Type | Purpose |
|---|---|---|
CSM_GPG_SIGNING_KEY | File | GPG private key used to sign APT metadata, RPM packages, and RPM repo metadata. |
CSM_MIRROR_SSH_KEY | File | SSH key used to publish the mirror output. |
CSM_MIRROR_KNOWN_HOSTS | Variable | SSH host keys for the mirror host. |
The job exports the public key as csm-signing.gpg and publishes it at the mirror root so install docs can reference:
https://mirrors.pidginhost.com/csm/csm-signing.gpg
APT verifies signed repository metadata through the signed-by= keyring. DNF verifies both RPM package signatures and repository metadata via gpgcheck=1 and repo_gpgcheck=1.
Detached Artifact Signatures
sign:artifacts signs binaries and packages with the Ed25519 private key in CSM_SIGNING_KEY when that variable is present. The publish and GitHub release jobs create and sign csm-assets.tar.gz after assembling it. Each signed file gets a .sig sibling uploaded with the artifact.
Examples:
csm-X.Y.Z-linux-amd64
csm-X.Y.Z-linux-amd64.sha256
csm-X.Y.Z-linux-amd64.sig
csm-assets.tar.gz
csm-assets.tar.gz.sha256
csm-assets.tar.gz.sig
csm_X.Y.Z_amd64.deb
csm_X.Y.Z_amd64.deb.sig
csm-X.Y.Z-1.x86_64.rpm
csm-X.Y.Z-1.x86_64.rpm.sig
The signature covers the raw artifact bytes with no hashing wrapper. Verification uses:
openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
-sigfile csm-X.Y.Z-linux-amd64.sig -in csm-X.Y.Z-linux-amd64
Detached Signature Setup
On a trusted workstation:
openssl genpkey -algorithm ed25519 -out csm-signing.key
openssl pkey -in csm-signing.key -pubout -out csm-signing.pub
Store the private key in GitLab as a protected CSM_SIGNING_KEY variable. Keep the private key in an offline password manager and a second secure backup location. Do not commit it.
For standalone script verification, either:
- Embed the public key PEM in
EMBEDDED_SIGNING_KEYinscripts/install.sh,scripts/deploy.sh, andscripts/deploy-gitlab.sh. - Or pass the public key at runtime with
CSM_SIGNING_KEY_PEM.
Signature verification is mandatory for current standalone releases. To also reject missing signatures on historical pre-signing releases:
curl -fsSLo /tmp/csm-install.sh https://raw.githubusercontent.com/pidginhost/csm/main/scripts/install.sh
sudo env CSM_REQUIRE_SIGNATURES=1 bash /tmp/csm-install.sh
Detached verification requires OpenSSL 3.0 or newer because the Ed25519 command uses pkeyutl -rawin. The scripts stop on older OpenSSL, including the default on EL8/CloudLinux 8, Ubuntu 20.04 and Debian 11. Use the signed APT/DNF repository on these supported platforms; its GPG verification does not depend on this OpenSSL command. The scripts never download and execute a verifier to bypass this requirement.
If a .sig file exists but verification fails, the installer aborts regardless of CSM_REQUIRE_SIGNATURES. A missing .sig (HTTP 404) is tolerated only with CSM_REQUIRE_SIGNATURES=0 and an explicitly selected complete release version older than v2.2.0, with no suffix or leading zeroes; for any later release the scripts abort even without CSM_REQUIRE_SIGNATURES, because every such release ships a signature and its absence means the download is incomplete or tampered. An unknown version or latest cannot qualify for the exception. This historical exception is checksum-only installation and is reported as unverified, not signature-verified.
The same policy applies to the csm-assets.tar.gz.sha256 checksum: releases published before checksums existed produce a warning and skip the checksum step, while CSM_REQUIRE_SIGNATURES=1 makes the missing checksum fatal. A checksum that exists but does not match always aborts.
The check action only compares checksums and reports update availability; it
does not download or execute a release binary. Installation and upgrade perform
the signature checks above.
Key Rotation
Package repository GPG key rotation:
- Generate a new GPG signing key.
- Replace
CSM_GPG_SIGNING_KEYin protected CI variables. - Publish a tag pipeline so
repo:publishexports the new public key to the mirror. - Update install docs or automation if the key URL changes.
Detached Ed25519 key rotation:
- Generate a new Ed25519 key pair.
- Replace
CSM_SIGNING_KEYin protected CI variables. - Update the embedded public key in standalone scripts, or rotate the
CSM_SIGNING_KEY_PEMvalue used by automation. - Tag a new release.
Old detached signatures remain verifiable only with the old public key. Archive old public keys alongside release metadata so historical releases can still be checked.
Manual Detached Verification
VERSION=X.Y.Z
curl -LO "https://github.com/pidginhost/csm/releases/download/v${VERSION}/csm-${VERSION}-linux-amd64"
curl -LO "https://github.com/pidginhost/csm/releases/download/v${VERSION}/csm-${VERSION}-linux-amd64.sig"
openssl pkeyutl -verify -pubin -inkey csm-signing.pub -rawin \
-sigfile "csm-${VERSION}-linux-amd64.sig" -in "csm-${VERSION}-linux-amd64"
openssl pkeyutl -rawin needs OpenSSL 3.0 or newer. EL8 and CloudLinux 8 ship
OpenSSL 1.1.1, whose command line cannot verify Ed25519 at all. Use an already
installed CSM build there, which verifies with Go’s implementation and needs no
OpenSSL:
csm verify-release csm-signing.pub \
"csm-${VERSION}-linux-amd64.sig" "csm-${VERSION}-linux-amd64"
It exits zero only for an artifact signed by the given key. The installer and
deploy scripts try OpenSSL 3.0+, then an installed csm verify-release, then
python3-cryptography, and refuse the artifact when none is available.
The Go and Python verifiers reject special files and limit input sizes: keys to 64 KiB, signatures to exactly 64 bytes, and nonempty artifacts to 512 MiB. Python verification ignores the working directory, user site packages, and Python environment settings when importing its dependencies. Install python3-cryptography as a system package.
If verification fails, treat the artifact as untrusted. Do not install it.
Architecture direction
Why the host agent stays autonomous, keeps one database owner and grows by slices instead of a rewrite.
Keep the host agent autonomous and self-contained, with bbolt owned by one process. Privilege separation must preserve that ownership: the helper and online CLI clients use scoped requests for reads and writes. They must not open the live database independently, even read-only, because bbolt’s writable owner holds an exclusive file lock. Offline maintenance needs exclusive ownership, with the daemon and helper quiesced before handoff.
A local database replacement needs measured contention, query complexity or recovery requirements that the current design cannot meet. Both bbolt and SQLite WAL serialize writers; a switch alone does not remove that constraint. See the bbolt transaction documentation, SQLite WAL concurrency and bbolt read-only mode for the separate-process lock behaviour.
Use the existing store, action log, privileged-operation inventory, scan jobs and incident correlator as the starting points. Add domain interfaces as each slice needs them; a package renaming campaign or a generic bucket/key API is not a prerequisite. Keep operator configuration in YAML and migrate remaining authoritative operational state by domain, with a rollback contract.
The architectural priorities are privilege isolation, durable host actions and browser credential isolation. They complement the harm-based ordering of the roadmap and never defer a protection failure or a precision and response defect. Browser sessions and their HTTP/domain boundary are implemented. Delivery order for the rest:
- A firewall block/unblock slice of the durable action lifecycle, including storage measurements and an explicit state-owner contract. Complete the firewall state migration with recovery proof, then use that action contract for the first privileged-helper verbs.
- Expand helper coverage and action recovery by response family; extend the existing job model as each long operation moves behind a service. Drop main-process privileges only when the required reads, descriptors and mutations have verified replacements.
- Improve correlation through the shared replay harness, then add outbound fleet ingest. Panel availability must never gate local protection.
No mandatory local broker, database server or orchestration platform is added. The fleet service owns its database choice separately; the agent does not select or build a central database stack.
Auto-response safety model
How automatic actions are tiered, budgeted and paused so that one bad detector cannot damage a protected server.
What exists
auto_response.dry_run defaults to on. The scan block path uses the
configurable max_blocks_per_hour budget and service restarts use
max_restarts_per_hour. The virtual-patch mode has a safe default. The
verdict callback lets a panel downgrade a block. Process signalling goes
through pidfd. Quarantine and virtual patching resolve paths with openat2
and RESOLVE_BENEATH. The incident correlator has safety caps and a dry-run
mode. Firewall changes record a rollback point. mode: observe refuses to run
any of it. See auto-response and
observe mode.
Automatic file responses share one admission model: realtime and scheduled quarantine, PHP cleaning and access-file cleaning draw on persistent host and account budgets over a rolling hour. Reservations survive reloads, restarts and interrupted actions. Repeated failures pause these responses, and unavailable safety state refuses changes. Pause warnings are deduplicated while the original detections stay visible. Account identity comes from account-home paths; unknown paths share a budget. Failed cleaning leaves the source and recovery evidence for review. Whole-directory and special-file quarantine require manual review. Actions revalidate the target after budget persistence, and the cleaners receive the same file identity captured before admission. The file-response failure pause is host-wide across automatic quarantine and cleaning paths.
Automatic per-address blocks use the firewall engine’s infrastructure,
local-address, operator-allow and verified-crawler guards. Block candidates
are admitted through reserved lanes and fair scope shares in
internal/admission, so independently supported findings keep bounded
capacity when one scope floods.
The firewall’s admission snapshot also estimates how long imported and unclaimed temporary occupants have left. Kernel estimates age through later set reads and member checks using elapsed time. Subnet estimates follow the wall clock. A missing kernel expiry leaves the drain estimate unknown. These estimates are observation only; no host enforces admission from them.
Risk tiers
Every operation in internal/privops carries a risk tier, and a test fails
when one is missing or disagrees with the reviewed inventory:
| Tier | Meaning | Examples |
|---|---|---|
| 0 | alert only | most findings |
| 1 | recommendation or dry-run record | dry_run blocks, virtual-patch preview |
| 2 | low-risk reversible | challenge, rate limit, mail hold |
| 3 | quarantine or block | file quarantine, nftables block, virtual patch |
| 4 | destructive or process-affecting | process kill, service restart, config rewrite |
Each tier gets a confidence floor, a per-action circuit breaker that follows the rules below, mandatory identity revalidation immediately before tiers 3 and 4 (inode and device for files, pidfd for processes, rule handle for firewall entries), and enough recorded metadata to reverse the action. The durable action lifecycle carries that metadata; there is no parallel policy switch or second set of limits.
Adversarial workloads
Safeguards are designed for attacker-triggered load as well as operational faults. Limits bound both resource use and harm to hosted sites while keeping detection and response health visible. Finite action, queue and storage budgets sit alongside evidence-based failure breakers. Budgets are calibrated against representative response demand, including bursts, and measured host capacity; behaviour under overload is validated and withheld responses are visible.
- A breaker pauses automatic action only. Detection, findings and alerts continue at their own severity.
- Budgets and pauses are scoped by verified account and action ownership, with shared bounds for common resources and unknown ownership. A source address or check name alone does not establish an independent actor. Admission is deterministic and fair, with bounded reserved capacity for independently supported findings.
- Only the affected action scopes pause unless a shared mechanism is unsafe. Verified shared storage or execution failures can require a wider pause at once; failures in several accounts alone do not prove that condition. Unavailable intent or budget persistence still refuses new mutations.
- Target refusals are recorded separately from execution and storage failures. Admitted attempts stay charged, and uncertain outcomes are reconciled before a retry. Every pause has bounded recovery and revalidation; reloads, restarts and policy changes do not silently reset it.
- Alternative containment must be supported for the affected protocol, honor existing mode and action opt-ins, and pass admission and identity checks. Resource pressure alone never authorizes broader targets or stronger actions. If no safe action is available, the finding is retained with the deferred or refused outcome and the required operator review.
- A pause that leaves a Critical finding without its automatic action raises
a deduplicated Critical alert and is reported by status and
csm doctor. Every deferred, refused or dropped candidate is accounted for without an unbounded alert queue. A dispatched request is not proof of containment.
Firewall feedback
For firewall blocks, correctness feedback supplements resource limits:
- Hard address exclusions and automatic allow protections stay at the authoritative action boundary, including range overlap checks and the existing explicit operator-command semantics. Source attribution is verified against the connecting peer or an explicitly trusted proxy chain. Any operator-session safeguard needs authenticated provenance, expiry and service scope.
- A detector-specific breaker needs reviewed correctness evidence linked to the original finding and action. Authentication or an operator unblock alone is not proof of a false positive. Independent attack evidence and other healthy response paths are preserved; corrective feedback does not demote findings.
- Temporary-block expiry and renewal are specified together with the escalation lifecycle. Escalation evidence is validated with corroboration grading; response records and repeated reports are not new corroboration. Operator exclusions and undo are preserved.
- Firewall sets and admission stay bounded. Range or ASN-based containment requires separate corroboration, protected-range checks and an explicit policy limiting collateral impact; it is not a substitute for capacity.
One admission authority
The same admission authority covers automatic entry points, retries and recovery. Under privilege separation the executor verifies policy and admission independently of caller-supplied findings or state, while a single owner holds the live database.
Calibration
Budgets are calibrated with recorded finding streams joined to action outcomes and reviewed operator decisions. Raw recordings and operational tuning stay private; only sanitized fixtures and aggregate validation results are published.
Durable action lifecycle
Why every host mutation records its intent before acting, and how recovery and undo are meant to work.
internal/actionlog writes one JSONL record per action to
/var/log/csm/actions.jsonl, keyed by the operation IDs from
internal/privops. A record carries the actor, the finding_id the SIEM
audit log also emits so the two streams join, the exact argv when CSM ran a
program, the target file’s digest before and after, the result including
refusals, and the command that reverses the change. csm actions reads it.
See action log. The wiring pattern is settled: record at
the operation chokepoint, not at the entry point, so a CLI-driven and an
automatic call produce the same record.
A JSONL outcome is evidence, not durable intent. It cannot alone distinguish a refused request from a mutation applied just before a crash, and some effects, including process termination, cannot be undone.
Decision
Every host mutation goes through a shared lifecycle at its operation
boundary. Execution stays in the responsible domain; the lifecycle owns
identity, admission, persistence, recovery and audit linkage. It reuses
mode: observe, auto_response.dry_run, the existing action-specific
switches and the file-response budget and breaker settings. There is no
second policy switch and no parallel set of response limits.
The first slice is firewall block and unblock. Before mutation it persists a stable action ID, operation, actor, target identity, finding and incident links when present, intended effect and recovery metadata. States distinguish planned, executing, applied and verified from refused, failed, partial, unknown and rolled back. After a crash, incomplete records are reconciled against actual host state before any retry; an uncertain outcome stays visible until it can be proved. A stable ID supports deduplication, not a promise of exactly-once host effects.
A firewall action can carry a batch: a header and a manifest of
single-target members, each with its own charge and the rows it removes to
make room. Each member ID belongs to exactly one batch. Admitting a known ID
with any change to the members, their order or their effects is refused, and
a refused batch charges nothing. A single-target block that removes a row to
make room records that row as its victim: the only row naming the address
before the action, with none naming it after. Selection requires a verified,
unexpired row in its canonical spelling and uses complete committed state.
Safety and capacity use the active view of the same snapshot; the saved
result preserves unrelated rows. In-memory unverified or failed marks also
exclude victims, including for operator blocks, without changing the saved
rows. Its audit writes an evict_temp record beside
the block’s record in the same durable write. A batch only adds its members’
rows, each starting when the batch is planned: nothing evicts for a batch, so admission refuses one whose result drops a row that
has not expired or changes any other entry. Kept rows retain their order.
In batch planning, unverified rows remain until their owner settles the
outcome, even after expiry, and a batch cannot add a block for an address they hold; an
unverified address stays pending even when another spelling of the same
address has a verified block. Actions written before batches read as
one-member batches and keep their stored format. A record written by a newer
build is refused as unsupported, never read or overwritten. A firewall-only
import moves the durable journal and its snapshot as one unit, and refuses
while either database has an unresolved action or undelivered audit. A
refusal, including one for nested firewall buckets on either side, leaves the
target database unchanged.
The block batch planner checks each automatic temporary block on its own
against one committed snapshot, leaves out refused members and runs the rest
as one batch with one kernel transaction; a lone valid member runs as an
ordinary single action, and a stored batch ID is observed, never planned
again. A member too short to outlive its own verification is refused, since
its expiry would leave the batch unknown. A batch has one outcome. A crash or lost write at any step, from
admission to audit acknowledgement, leaves recovery to prove an outcome or
keep it unknown; the kernel is written at most once and each member is
charged once. Storage and audit failures reach the caller even when recovery
can later prove the outcome. A kernel holding only some members proves
neither outcome, so the batch stays unknown until the kernel proves one,
an operator resolves it, or it retires. Audit delivery writes one record per
member, with the member’s action ID and the batch’s outcome and version, in
one durable write. An unknown block action retires as expired_unproven
once all member targets are timed and expired, every row its before and
after states disagree on is a verified timed block that has
expired by the lease clock, its owner attests a trusted clock, its unknown
outcome has reached the audit trail,
and the kernel holds exactly the rows both states agree on. Recovery then
commits those rows as the next revision in one transaction and frees the
pending interlock. The outcome stays unknown, its charges stay spent, its
audit records say expired_unproven, and nothing is applied again. A lease
clock running ahead of the kernel finds an element the kernel still holds
and retires nothing. A permanent, unverified or non-block effect, or any
change outside the blocked rows, waits for an operator. A retired action
cannot be resolved or undone. A missing or degraded clock attestation
leaves the action pending; a failed attestation also reports its error.
Saved blocked-set evidence must be present, and
a changed set topology prevents retirement. IPv6 proof uses saved evidence
even if IPv6 is now disabled. Kernel timeouts are checked against the clock
sample taken before each read, and a slow read refuses retirement. The
unknown outcome must carry its recorded audit acknowledgement; an empty
outbox alone does not prove delivery.
Batch evidence selects each member’s current row by ID, target and planning time; retained rows from an earlier use of that ID are not current evidence. An unresolved batch with an expired member remains unknown: absence does not prove that member was never applied. This also holds during immediate verification after apply. A durable terminal outcome is preserved on replay after expiry. A rejected kernel batch is not retried inside the lifecycle. Its owner must first prove rejection and use a new charged action for any retry. Audit is delivered at least once; consumers deduplicate each member or victim event by action ID, transition version, action and target, so a block and its eviction remain distinct.
Batch replay returns the stored manifest, not a fresh decision on candidates left out of the original action. Nil refusals on replay do not mean every supplied candidate ran. The stored requests must appear as an ordered subsequence; extra candidates, including previously refused duplicate IDs, never execute. The original refused list cannot be reconstructed. A singleton uses its member ID as the durable action ID; its unused header ID is not reserved. Retrying that exact automatic temporary block returns the original action before checking current policy or expiry, and cannot add other members. Two stored singletons in one call are a conflict. Dry run refuses only new work: a retry still returns the stored action. A caller needing a durable header alias for singletons must wait for a stored-format extension.
Replaying a settled batch leaves its charges, committed revision and kernel unchanged. An acknowledged audit is not delivered again. Whole-action undo requires the complete saved after-state and kernel proof. Member undo removes only that member’s unchanged row and preserves other members and later live changes. Another live or unverified block on its address refuses member undo. Member undo checks alias expiry on the engine’s lease clock, so a backward wall-clock step cannot make an expired alias prevent undo. Every undo omits expired blocks, allows and subnets, including expired unverified blocks. Restored rows keep their original absolute expiry.
Batch planning dates member rows and kernel evidence by one lease-clock instant, even when the wall clock moves backward during a lease. Operator IP and port allows come from the same committed planning snapshot; allow expiry uses one wall-clock reading. A changed committed revision refuses admission for both batches and lone members.
If intent or budget persistence fails, new automated mutations are refused and the finding is retained. If outcome persistence fails after mutation, the pending intent is preserved, action health reports degraded, and reconciliation runs before another attempt. A bbolt transaction cannot atomically commit a filesystem or kernel change, so host I/O stays outside the database transaction and target identity is revalidated immediately before execution, including on recovery and undo.
JSONL and csm actions remain the operator-facing audit interfaces, linked
by action_id. Durable state and audit delivery need a retry and
reconciliation contract so an applied action cannot silently lose its audit
outcome. csm action show <id> and typed undo dispatch follow; a stored
command string is never executed as the authority to reverse a change. Undo
verifies the current target, records its own linked action, and refuses
changed or irreversible targets. Retention keeps unresolved intent and
required recovery evidence. Backup restore must not resurrect pending actions,
jobs or sessions as executable or authenticated live state; what is disarmed
and what requires review is defined per record type.
The open work is tracked in the roadmap.
Privilege separation
Why a minimal privileged helper will own the operations that need root, and what it must enforce on its own.
Today the whole daemon, including detection, correlation, parsers, threat
intelligence, web UI and API, runs as root inside the systemd confinement
described in service confinement:
ProtectSystem=strict, syscall filtering, a ReadWritePaths allow-list, and
systemd-run for the panel scripts that need the whole filesystem. One helper
pattern exists: the mail forward guard executes a fixed helper subcommand of
the same binary. A parser bug anywhere is therefore a root compromise of the
host.
Decision
A minimal privileged helper owns the operations that require root: nftables
and firewall changes, process termination, quarantine and other moves across
account boundaries, fanotify and BPF initialisation with the resulting
descriptors passed back over the socket, and privileged filesystem and
configuration writes. Everything else runs with reduced privileges. Reduced,
not none: reading every account’s files still needs CAP_DAC_READ_SEARCH,
so the unprivileged side keeps a small capability set and loses the ability
to write, signal and reconfigure. The helper exposes a narrow Unix-socket RPC
authenticated by peer credentials, with fixed verbs and arguments validated
against the same path, user, process and firewall scopes the actions enforce
today.
The inventory stage is complete. internal/privops lists every operation
that needs privilege beyond reading CSM’s own files, with what it writes, the
config key that stops it, its risk tier and what an operator loses by
withholding the privilege. csm privileges prints it, the
capability matrix ships it, and two gates in
internal/ci compare it against the packaged systemd unit in both
directions, so a writable-path grant that no operation claims fails the
build. That table is the action set the helper has to cover.
mode: observe is the interim posture for operators who will not grant a
root daemon the ability to act: detection and alerting run, automatic
remediation and integration deployment do not. It reduces what the root
process does, not what it could do, so it is a stopgap rather than a
substitute.
Staging
Each stage ships on its own: the helper for firewall, then signals and quarantine, then privileged configuration and filesystem writes; then descriptor passing for fanotify and BPF; then dropping capabilities in the main process. Each family first needs its safety classification and durable action contract, not completion of every other family’s migration.
What the helper must enforce
Peer credentials authenticate the caller, not the requested operation. The helper enforces target and account scope, identity, permitted verbs and safety policy even if the main process is compromised. It exposes no arbitrary shell, command execution or unrestricted file-write RPC. Request sizes and execution time are bounded; unknown verbs, malformed requests, stale identities and unauthorized peers are tested.
How the helper verifies persisted intent and safety admission is decided before the first helper verbs ship. A record in a store writable by the main process is caller-controlled too; reading it back through RPC does not make it trusted approval. One process owns the store, and helper-enforced policy, budgets and replay protection survive a compromised caller and a helper restart. If the main process remains the owner, its records are evidence only: it must not be able to reset or forge the helper’s safety admission. Forged intent, replay and attempted budget reset are tested alongside valid requests. Socket ownership, protocol compatibility and unavailable-helper behaviour are part of the first slice. Findings are retained when a mutation cannot safely proceed.
The open work is tracked in the roadmap.
Firewall state migration
Why the firewall engine’s authoritative state moves from state.json into
bbolt, and the rules the cutover has to keep.
The lossless firewall state storage contract is implemented and tested
independently of runtime callers. It preserves complete ordered state with
revision checks and atomic replacement. Firewall buckets and store methods
exist, and pending configuration rollback already uses bbolt. The engine
still reads and writes its authoritative runtime state in state.json.
This belongs with response correctness. JSON writes already use atomic replacement, and the block path persists intended state before touching the kernel. Those guarantees stay; changing the storage format alone cannot make the database and nftables one transaction.
Engine interface
A domain-owned firewall state interface is injected into the engine. It reuses the existing blocked, allowed, subnet and per-port buckets without exposing bbolt transactions to firewall callers. The original store schema and methods were not a lossless engine backend: subnet rows lacked expiry and used a different creation-time field, and loaders hid read and decode failures. The schema and error contract are extended before cutover; original times and explicit provenance are preserved instead of recreated through add methods. A failed or corrupt read must not become a successful empty or partial ruleset. Each logical state change commits together, and the hot-path cache updates only from committed state. Kernel application and recovery follow the durable action lifecycle.
Cutover
Migration is a one-shot step through the owning daemon, or under an exclusive offline maintenance lock. It validates the entire source, imports transactionally, records a schema and cutover marker, and retains the original JSON for rollback. A crash at any cutover step is recoverable. After cutover only bbolt is authoritative; rollback preserves post-cutover changes instead of silently restoring the now-stale JSON. Desired configuration stays in YAML. Existing exports disarm pending configuration rollback; that property is kept and the treatment of new action intent is defined with it. Migration must not reset existing response budgets or failure pauses when those move to bbolt.
The open work is tracked in the roadmap.
Email Forwarder Protection and Visibility – Design and Roadmap
Status: shipped. This page is the design the forward guard, the held-copy quarantine and its retention prune were built from; the configuration keys below are documented in configuration. Date: 2026-06-07. Owner: CSM mail subsystem.
Problem
cPanel/exim servers run many account forwarders that blindly relay inbound
mail – including spam and null-sender bounces – to external free providers
(Yahoo, Gmail, Outlook). Forwarding spam to those providers generates
complaints and tanks the sending IP reputation. The provider then defers all
mail from the IP (observed: Yahoo 421 4.7.0 [TSS04] ... unexpected volume or user complaints, thousands of deferrals/day), the queue fills with retrying
forwards and backscatter, and legitimate mail is delayed.
Operators today have no view of which forwarders exist, where they point, how much each contributes to the queue, or which outbound IPs are being throttled by which providers – and no control to stop relaying spam externally.
Goal / non-goals
Goal:
- Full operator visibility: every forwarder, its destination, owner, queue impact, and the outbound-IP reputation picture.
- Opt-in protection that holds spam/backscatter before it is relayed to an external provider, protecting outbound IP reputation, with no risk of losing legitimate mail.
Non-goals:
- Not a general inbound spam filter (SpamAssassin owns local-delivery scoring).
- Not an MTA replacement. CSM observes, supplies data, manages quarantine + UI.
- Not auto-deleting or auto-disabling forwarders without operator action.
Approved decisions (from brainstorming)
- Enforcement action: hold the external-forward copy in a CSM-owned quarantine, always keep/deliver the local copy. Held mail is recoverable (operator release), so false positives mean review, not loss.
- What to hold (layered signals, any match -> hold), each individually
toggleable:
- SpamAssassin spam-flagged,
- ClamAV / YARA-X malware hit,
- null-sender
<>bounce (backscatter), - sender IP present in CSM attack DB / reputation,
- SPF + DKIM + DMARC all fail.
- Rollout: single global toggle, default OFF; dry-run mode (log/account only,
hold nothing); reuse existing
known_forwardersas a skip-list for trusted relays. - Visibility surfaces (all included): forwarder table, outbound-IP reputation panel, per-provider deferral rollup, queue composition breakdown, per-forwarder spam/volume trend.
- Build phased: visibility first (read-only, low risk), enforcement second (opt-in), breadth (postfix/non-cPanel) third.
Core architectural decision: CSM is NOT in the live mail path
The MTA performs the hold natively. CSM only (a) generates MTA config + a reputation lookup file, (b) watches the quarantine, (c) renders visibility and handles release/delete.
Rationale (production safety): if CSM is down or crashes, mail must still flow.
CSM must never be a delivery dependency. The MTA evaluates the layered signals
itself – exim has native SPF/DKIM/DMARC/spam variables, a null-sender test,
and an lsearch/cdb lookup against a CSM-maintained bad-IP file. CSM
generates the rule; the MTA enforces it.
Failure posture: any error in the MTA rule path must defer (exim retries),
never silently drop. CSM-generated config is idempotent and reversible
(Remove() restores normal forwarding) and survives cPanel exim rebuilds by
using the cPanel-preserved advanced-configuration include sections.
Components (platform-abstracted, OSS-minded)
New package tree internal/mailfwd/:
inventory– enumerate forwarders cross-platform. cPanel:/etc/valiases/*(reuse existingParseValiasEntries/IsExternalDestfrominternal/checks/forwarder.goandinternal/daemon/forwarder_parse.go). Non-cPanel:/etc/aliases,~/.forward, postfixvirtual. Source selected viainternal/platform. Returns a normalizedForwarderlist.intel– parseexim_mainlogfor: per-destination-provider deferrals with reason codes (TSS04 etc.); per outbound-IP reputation (who is deferring it); queue composition (bounce vs real, per-recipient, oldest, frozen). ReuseeximQueueSize/eximQueueDetailsininternal/webui/email_api.go.policy– the single source of truth for the layered verdict. One functionVerdict(msg MessageMeta, cfg ForwardGuardConfig) (hold bool, reasons []string)used by BOTH the dry-run accounting and the rendered MTA rule (the MTA rule is generated from the same policy definition so they cannot drift).adapter– interfaceForwardGuard { Apply(Policy) error; Remove() error; Status() (GuardStatus, error) }. exim adapter first (renders router + transport + bad-IP lookup, reloads exim, writes to cPanel-preserved include). postfix adapter later. Idempotent, reversible.quarantine– CSM-owned maildir for held forwards; release (re-inject via the platform sendmail) and delete; per-forwarder held counters. Mirror the existinginternal/emailavquarantine + webui pattern (apiEmailQuarantineList/apiEmailQuarantineAction).
WebUI: extend the existing /email page (handleEmail, internal/webui/ email_api.go, email_groups.go). New read endpoints under requireRead;
mutating actions under requireAuth + requireCSRF (existing pattern).
Config
Extends EmailProtection in internal/config/config.go. Remember the
three-places rule: DefaultConfig + installer template + packaging
csm.yaml.default (CI drift test enforces). Use presence checks
(yamlPathExists) where a literal 0/false must be distinguishable from unset.
email_protection:
forward_guard:
enabled: false # master switch, default off
dry_run: true # log/account only, hold nothing
hold_signals:
bounce_backscatter: true
spam_flagged: true
malware: true
bad_sender_ip: true
auth_fail: true
quarantine_retention_days: 14
On cPanel hosts in enforce mode, CSM applies these settings at startup and on config reload. Failed changes are retried on the next 15-minute lookup refresh, using the latest settings. A change that waits for another CSM change to the Exim configuration, such as an antispam one, is logged as deferred, not as a failure, and retried the same way. CSM waits up to one second for the lock before it records a change and before a worker starts one, so short holds do not defer a change. A reload never waits for a forward-guard change: the change runs in the background, and reloads that arrive while one runs are applied right after it, with the latest settings. Shutdown lets the running change finish and discards queued reloads; startup applies the saved settings. Disabled and dry-run settings start no removal worker when no guard is installed. While another CSM change to the Exim configuration runs, even one started before a daemon restart, the removal is deferred the same way. CSM records each change before its worker starts, and a worker whose change is no longer the latest, such as one started before a restart, changes nothing. Each submission has a distinct identity, even when its settings and sender list match an earlier submission. A delayed worker cannot replace a sender list refreshed since the newer submission. If its request record is missing or unreadable, the worker leaves the installed guard untouched. Request records are readable by root only; the Exim configuration and the bad-IP lookup stay readable by the mail transport user. Observe mode and hosts without cPanel leave the installed guard untouched; in observe mode CSM only cancels pending changes. Cancellation does not wait for another Exim transaction to finish. A worker that has already checked its request under the Exim lock finishes its transaction; later workers cannot apply canceled requests.
WebUI surfaces (email page)
- Forwarder table: source -> destination(s); provider badge (yahoo/gmail/outlook/external/local); owner (cPanel user); keep-local vs forward-only; queued/stuck count; held-or-would-hold count; status (e.g. “Yahoo throttling”); row actions: disable forwarder, add to skip-list, view held.
- Outbound-IP reputation panel: per sending IP, provider deferrals + parsed reason codes + throttle state.
- Per-provider deferral rollup: provider -> queued + throttle + % of queue.
- Queue composition: bounce vs real, top stuck recipients, oldest/frozen, one-click flush backscatter.
- Held forwards: quarantine list with release/delete (phase 2).
Safety / OSS
- Default off; dry-run first; skip-list for trusted relays.
- MTA-native enforcement -> fail-open (CSM down never blocks mail).
- MTA rule errors
defer, never drop. - Cross-platform via adapters; exim/cPanel first; capability flag
mail.forward_guard.v1ininternal/health/capabilities.go. - All mutating actions audit-logged; fully reversible.
- No private hostnames/IPs in committed code/tests/docs (use RFC 5737/3849).
Testing
policy.Verdict: table-driven over each signal, combinations, skip-list, dry-run on/off. Real assertions.- inventory parsers: fixtures per platform + fuzz target (attacker-controlled valias/aliases/.forward content).
- adapter: render-to-golden exim config; idempotent apply/remove; reload failure -> no-op rollback (no half-applied state).
- intel parsers: real-format exim deferral lines (TSS04, Gmail, Spamhaus) + fuzz.
- quarantine: release re-injects correctly; retention prune; per-forwarder counts.
- No live-mail tests on production. Cross-compile + vet locally; verify on phctl scratch hosts (fresh Ubuntu + AlmaLinux).
Roadmap (each item = its own spec-slice / plan / TDD / codex review / MR)
Phase 1 – Visibility (read-only, ship first):
mailfwd/inventory: cross-platform forwarder enumeration + unit/fuzz tests.- Email-page forwarder table + API (source/dest/provider/owner/keep-local).
mailfwd/intel: outbound-IP reputation + per-provider deferral rollup (exim_mainlog parse) + the two panels.- Queue composition panel + one-click backscatter flush action.
- Dry-run
policyengine: per-forwarder “would-hold” accounting surfaced in the table (no enforcement yet).
Phase 2 – Enforcement (opt-in):
6. Config schema + settings-page toggles + validation/defaults (three-places).
7. policy -> exim adapter: router + transport + bad-IP lookup; idempotent
apply/remove; cPanel-rebuild survival.
8. mailfwd/quarantine for held forwards + release/delete + per-forwarder
held counts.
9. Live enforcement wiring (dry-run -> enforce), audit log, findings,
capability flag.
Phase 3 – Breadth: 10. postfix adapter (non-cPanel hosts). 11. Reputation aids: provider feedback-loop hints, SRS guidance, delisting checklist surfaced to the operator.
Existing code to reuse (do not reinvent)
internal/checks/forwarder.go:ParseValiasEntries,IsExternalDest,IsPipeForwarder,parseLocalDomainsContent.internal/daemon/forwarder_parse.go:parseValiasFileForFindings,checks.IsKnownForwarder; liveForwarderWatcher.internal/webui/email_api.go:apiEmailStats,eximQueueSize,eximQueueDetails,topMailSenders, quarantine list/action handlers.internal/webui/email_groups.go: email-page grouping + sort.internal/config/config.go:EmailProtectionstruct,KnownForwarders.internal/emailav: quarantine model + spool watcher (ClamAV/YARA).internal/platform: OS/panel/path/MTA abstraction.internal/health/capabilities.go: capability flags.
Conventions (from AGENTS.md)
Production-grade, no shortcuts. TDD for security-relevant logic (test first).
go test ./... -race, golangci-lint, go vet, govulncheck clean before
push. gosec annotations need a real reason. CHANGELOG entry (1-2 sentences,
plain English, no internals) in the same commit as behaviour changes. One
conventional-commit subject per commit, no Co-Authored-By. Branch off main;
never push/MR without explicit instruction. codex review at each item, then
self-review (read each changed file end to end). Platform paths only via
internal/platform. Never allowlist scan paths.
PHP form-relay state
The form-relay store keeps the per-client recipient history behind php_relay
Path 6, described in PHP-Relay.
The ip_recipient_spread and ip_recipient_window_hours settings supply the
recipient limit and observation window; the store adds no settings.
Counts saturate at the recipient limit. At capacity the store retains the most recent distinct outside recipients, including one entry per unparsed message. New activity replaces older evidence so a busy client does not lose its recent history when its first recipients expire. Lowering the limit retains the newest evidence within the new bound. Raising it cannot recover previously discarded recipients.
The window ends at the latest time observed by recording, cooldown checks, or sweeping. Delayed messages may contribute only while they remain in that window; replaying expired mail cannot recreate expired history or cooldowns. The time boundary survives removal of idle clients. Expired recipients are removed before capacity eviction, so stale counts cannot displace active clients. Expiry alone does not report capacity loss.
Each account retains its most recently observed staff clients. A delayed staff record cannot replace a newer one merely because it arrived later. Staff history is stored apart from recipient history and expires on its existing schedule.
All state transitions and reads hold the store mutex. The evaluator runs only for a message the store counted. It reads the client’s count, script shares and recipient domains and claims the client cooldown in one call under that lock, so expiry cannot remove evidence between counting it and building the finding. Before counting, that call drops recipients the client reached through an account where it is now staff, or through a script that is now ignored, so a current exemption also covers mail recorded before it applied. Script shares are grouped by the script’s normalized name, the one findings print. Returned script shares and recipient domains are copies.
When that call shows the client would fire, the evaluator first reads the sites’ access logs for staff evidence, outside the lock and without claiming. Each distinct log is read once per crossing, even when several scripts share it. Only successful GET requests to recognized staff pages qualify; public or subscriber pages and ambiguous request paths do not. The latest qualifying request time across those logs sets the staff expiry, so rereading an old log cannot renew an exemption. Invalid, future and expired timestamps do not qualify. It marks any account with evidence as staff, then repeats the call, which claims the cooldown only if the remaining accounts still cross the threshold.
The store also records when each normalized script name was last reported, so a script finding repeats neither for a later client nor after the routine sweep of per-script state within the window.
Client findings carry the triggering Exim message ID as their observation in the spool stream. Admission can trace direct responses and later challenge timeouts to that same mail. Script findings remain advisory and carry no address evidence.