Metrics (Prometheus)
CSM exposes a /metrics endpoint on its HTTPS web UI port
(default 9443). The endpoint serves the Prometheus text exposition
format (Content-Type: text/plain; version=0.0.4) and is safe to
scrape every 15 seconds.
“Available metrics” below is the shipped set. New call sites are
instrumented in ongoing releases; check CHANGELOG.md under
## [Unreleased] for the latest additions.
Enabling
Metrics are on whenever webui.enabled: true is set in csm.yaml.
The endpoint has its own auth knob:
webui:
enabled: true
auth_token: "<UI login token>"
auth_token_issued_at: "<UTC RFC3339 time when the UI token was generated>"
metrics_token: "<long random string for Prometheus scraper>"
metrics_token_issued_at: "<UTC RFC3339 time when the scrape token was generated>"
metrics_token_expires_at: "<optional UTC RFC3339 expiry; leave empty for none>"
metrics_token is optional. When set, a Bearer header containing
this exact value grants access to /metrics. The UI auth_token or a valid
UI session cookie is also accepted so the dashboard can self-scrape,
but keeping the two tokens separate is recommended: rotating
auth_token does not then break Prometheus scraping, and giving
your monitoring stack the scrape token does not also give it UI
access.
Each credential expires at its expires_at setting and never expires
without one. Rotate the metrics token and its dates together, update the
scraper’s secret, then send SIGHUP to CSM. A malformed or future issue time,
or a malformed or past expiry, refuses the metrics credential. See
credential rotation for recovery instructions.
Prometheus scrape config
scrape_configs:
- job_name: csm
scheme: https
tls_config:
# CSM serves a self-signed cert by default; either skip
# verification here or pin the CA you chose.
insecure_skip_verify: true
authorization:
type: Bearer
credentials: "<metrics_token from csm.yaml>"
static_configs:
- targets:
- csm-host-1.example.internal:9443
- csm-host-2.example.internal:9443
A complete, validated version of this snippet (with global: block)
ships as docs/src/examples/prometheus-scrape.yml. The CI pipeline
runs promtool check config against that file in the promtool-check
job; if the example ever stops validating, the pipeline fails.
Quick check
curl -sk -H "Authorization: Bearer $METRICS_TOKEN" \
https://localhost:9443/metrics | head
Available metrics
Build / process
csm_build_info{version}(gauge, always 1): build metadata. Scrape once to discover the running version. Join on it in queries viagroup_left(version).
Go runtime
Always exposed. Use these to watch daemon memory over time and tell live
growth from GC headroom (rising heap_alloc = real growth; large
heap_idle = retained-but-freed memory).
go_memstats_heap_alloc_bytes(gauge): heap bytes allocated and still in use (live).go_memstats_heap_inuse_bytes/go_memstats_heap_idle_bytes/go_memstats_heap_released_bytes(gauges): in-use spans, idle spans, and bytes returned to the OS.go_memstats_heap_sys_bytes/go_memstats_sys_bytes(gauges): heap and total bytes obtained from the OS.go_memstats_next_gc_bytes(gauge): heap-size target for the next GC.go_memstats_gc_cpu_fraction(gauge): fraction of CPU time spent in GC since start.go_goroutines(gauge): live goroutine count.
For a deeper leak hunt, enable the loopback pprof endpoint (debug.pprof_listen)
and run go tool pprof http://127.0.0.1:<port>/debug/pprof/heap over an SSH tunnel.
The same endpoint serves profile (CPU, 30s by default), goroutine, mutex
and block. Contention sampling for the last two starts with the listener and
stops when the last listener exits, including on a listener failure. Failed or
refused binds do not enable sampling.
While enabled, mutex profiling samples one in 100 contention events. Block
profiling samples waits of at least 10 microseconds and a proportion of shorter
waits. Timing and collecting these samples adds CPU overhead even when nobody
fetches a profile, so use this endpoint for diagnostics and leave
debug.pprof_listen empty when it is not needed. Profiles accumulate for the
process lifetime; disabling sampling does not erase previously recorded data.
For a CPU investigation, collect the CPU profile first and the goroutine dump alongside it:
curl -o cpu.pb.gz 'http://127.0.0.1:<port>/debug/pprof/profile?seconds=60'
curl -o goroutine.txt 'http://127.0.0.1:<port>/debug/pprof/goroutine?debug=2'
YARA-X worker (default-on; off only if signatures.yara_worker_enabled: false)
csm_yara_worker_restarts_total(counter): cumulative number of counted YARA scan worker failures (failed start, timeout, crash, out of memory or protocol), including launches that never create a worker. Rule reloads and lifetime retirement do not count. Alert on sustained growth: a steady climb means scans keep failing and are reported as not scanned.
Findings
csm_findings_total{severity}(counter): every finding CSM records is counted here. Severities areCRITICAL,HIGH, andWARNING(matching thealert.Severityenum). Userate(...)for arrival velocity; watch for sudden CRITICAL spikes.
Alert delivery
-
csm_alert_dispatch_failures_total(counter): alert channel sends that failed after CSM detected findings. Counts email, webhook, and phpanel delivery failures. Sustained growth means findings are not reaching operators; check SMTP, webhook reachability, and credentials. -
csm_audit_sink_degraded{sink}(gauge): one when an enabled JSONL or syslog destination failed to open or write, zero when healthy or disabled. Values appear after the audit pipeline first runs and remain degraded during retry backoff. Syslog reports local write results, not receiver acknowledgement. -
csm_audit_events_dropped_total{sink}(counter): events whose destination was unavailable or whose write failed. Usecsm export --since <when>for backfill; failed live deliveries are not replayed automatically.
State
csm_store_size_bytes(gauge): on-disk size of the bbolt state database (/var/lib/csm/state/csm.dbby default). Retention sweeps bound logical growth. Startup compaction reclaims freelisted pages automatically when the file is large and mostly slack;csm store compactdoes the same immediately with the daemon stopped. The retention sweep logs a restart hint only under that same condition. It includes deleted pages still held by active readers, which become reclaimable when the daemon stops. Later writes can change whether compaction is due at startup. History and attack-event writes pack pages densely when their timestamps arrive mostly in order. A write uses balanced page splits when events older than the newest stored one are most of its records or at least half its inserted data. Space savings depend on record sizes and arrival order. Archive imports keep the snapshot’s page layout; compaction repacks an existing file.
Fanotify realtime monitor
csm_fanotify_queue_depth(gauge): current number of queued events waiting for the analyzer pool. The queue capacity is 4000; sustained values near that cap mean drops are imminent. Alert target:max_over_time(csm_fanotify_queue_depth[5m]) > 3500.csm_fanotify_events_total(counter): delivered fanotify file events whose admission decision has completed. Filesystem marks receive writes anywhere on their superblocks, including other bind mounts. Kernel queue overflow notifications are counted separately.csm_fanotify_events_admitted_total(counter): events queued for analysis, including dropper tracking. Subtract this counter andcsm_fanotify_events_dropped_totalfromcsm_fanotify_events_totalto count events rejected before queue admission. This includes path-resolution failures as well as filter rejections. Use rates over the same interval to compare these independently scraped counters.csm_fanotify_events_dropped_total(counter): cumulative events dropped because the analyzer queue was full. The reconcile pass still rescans drop-affected directories 60 s later, so dropped events do not disappear from detection – they arrive delayed. Alert target:rate(csm_fanotify_events_dropped_total[5m]) > 0paired with a short for-clause.csm_fanotify_placement_unverified_total(counter): files and directories moved, linked or created into watched trees that did not open as the file the placement monitor reported, so realtime did not check them there. A file replaced again right after it was placed is counted here. Sustained growth means placements are being raced; the deep scan covers those files.csm_fanotify_placement_skipped_total(counter): placed files not analyzed again because the same version was analyzed recently under the same conditions, such as an atomic save renamed to its final name.csm_fanotify_kernel_queue_overflow_total(counter): kernel-side fanotify queue overflows (FAN_Q_OVERFLOW). Unlike analyzer-queue drops, the kernel gives no file descriptor for lost events, so only directories with concurrent analyzer drops can be reconciled; the rest are covered by the next scheduled deep scan. Any nonzero rate during a write storm means realtime coverage had a gap.csm_spool_fanotify_queue_overflow_total(counter): kernel queue overflows on the mail spool watcher. In permission mode an overflowed event means the message was delivered without being scanned; a Warning finding is raised when this happens.csm_fanotify_reconcile_latency_seconds(histogram): how long the post-overflow reconcile pass takes to walk drop-affected directories and rescan recent files. Buckets: 0.01 s .. 60 s. Watch p95: reconcile stealing tens of seconds means bulk events are piling up faster than the walker can keep up.csm_checks_domlog_discovery_dropped_total{reason}(counter): per-vhost access-log paths the WP brute-force domlog discovery helper dropped before scanning. Labels:reasonisevalsymlinks_error(broken symlink, attacker-removed log file) orstat_error(file vanished between glob and stat, permission regression on the log directory). Steady growth means a real chunk of vhosts is being silently skipped each cycle. Stale-mtime drops are intentional filtering and are NOT counted here.csm_realtime_content_scan_truncated_total{check}(counter): cumulative real-time content checks where the underlying file was larger than the main read window, so the full-rule pass saw only the leading window. The read cap protects RE2 cost on huge files; sustained growth on a label means full-rule coverage is capped on large files. Labels currently emitted:phpcontent_inline(known webshell filename),phpcontent_uploads(PHP in uploads),php_check(generic PHP content scan),crontab(per-user /var/spool/cron write),htaccess(per-vhost .htaccess write),user_ini(per-vhost .user.ini write),html_phishing(HTML phishing heuristic), andcgi_backdoor(CGI backdoor heuristic). Compare against finding history forwebshell_content_realtime(or the matching check name for non-PHP labels) to judge whether a raised cap would surface real findings.
Mail scanning and parse workers
csm_spool_hold_expired_total{stage}(counter): mail opens answered underemail_av.fail_modebecause the hold budget ran out.stagerecords where the message was when its hold expired:queued(no scan slot yet; these count toward the hold bypass),waiting(behind another message from the same sender) orscanning(a slow scan; the scan continues and can still quarantine). Sustainedqueuedgrowth means scan capacity is short.csm_spool_source_waits_total{outcome}(counter): mail opens whose sender was already using its share of the scan slots.outcomeiswaited(the message waited its turn without taking a scan slot) oroverflow(too many messages from that sender were already waiting, so it was scanned at once without waiting its turn). Overflow scans can use every scan slot.overflowgrowth raises anemail_av_source_backlogfinding naming the sender, at most once a minute.csm_parse_worker_jobs_total{pool,outcome}(counter): parse worker jobs.poolismail(email attachments) orzip(tenant zips checked for phishing kits).outcomeisok,timeout,oom,crash,protocol(reply rejected),spawn(the unit did not start),refused(the worker is failing repeatedly) orskipped(the message killed workers before, so it was not unpacked again). Anything butokmeans a message or zip was not fully checked. Canceled requests, stopped workers and local descriptor exhaustion do not increment this counter.csm_parse_worker_starts_total{pool}(counter): worker units started. A worker serves many jobs; a start rate close to the job rate means workers keep dying or being retired.csm_parse_worker_memory_bytes{pool}(histogram): worker memory and swap in use after a successful job. Buckets: 32 MiB .. 1 GiB. Compare the high buckets withemail_av.worker_memory_limit. Unavailable memory readings are omitted; earlier readings are not reused.
Periodic check runner
-
csm_checks_crontab_base64_truncated_total(counter): crontab base64 candidates that exceeded the per-blob decode cap before decoded-content pattern matching ran. Sustained growth means encoded cron content is larger than the scanner currently inspects; review affected crontabs and tune the scanner before redeploying. -
csm_check_duration_seconds{name,tier}(histogram): wall-clock time each check takes to complete. Labelnameis one of the periodic check-runner names (fake_kernel_threads,webshells, …); labeltieriscritical,deep, orall. Buckets: 0.01 s .. 900 s. Most checks keep the 300 s timeout ceiling; heavy filesystem checks can run up to 900 s. Useful aggregations:# p95 of the slowest check in the critical tier: histogram_quantile(0.95, sum by (le, name) ( rate(csm_check_duration_seconds_bucket{tier="critical"}[10m]) ) ) # total time each cycle spends in deep-tier checks: sum by (tier) (rate(csm_check_duration_seconds_sum{tier="deep"}[1h]))
Threat intelligence
Registered when reputation.upstream.enabled: true.
csm_threatintel_cache_hits_total(counter): upstream threat-intel lookups served from CSM’s local per-IP cache.csm_threatintel_cache_misses_total(counter): upstream threat-intel lookups not served from the local cache. A miss may still fail open without an HTTP request when the breaker is open.csm_threatintel_backend_failures_total(counter): upstream backend failures from network errors, non-200 responses, malformed JSON, response IP mismatches, or out-of-range scores.csm_threatintel_breaker_open(gauge): 1 while the upstream circuit breaker is refusing calls, 0 when closed or allowing its single cooldown probe.
AI-crawler IP ranges
csm_botranges_refresh_total{result}(counter): AI-crawler IP-range refresh attempts, labelledsuccess(at least one vendor feed updated) orfailure. Driven by the daemon’s auto-updater and bycsm update-bot-ranges.csm_botranges_prefixes{bot}(gauge): current number of published IP prefixes per crawler identity in the active overlay.csm_botranges_last_success_timestamp_seconds(gauge): Unix time of the last successful refresh.
Firewall
csm_blocked_ips_total(gauge): number of IPs currently on the firewall block list. Excludes expired temp bans – the store’sLoadFirewallStatefilters those before the gauge reads.csm_firewall_rules_total(gauge): total firewall rules across all four categories (blocked IPs, allowed IPs, blocked subnets, port-specific allows). Excludes expired temp blocks and allow-list rows. Sudden drops are worth investigating; expected drops happen when temporary block or allow deadlines pass.
Config reloads
csm_config_reloads_total{result}(counter): SIGHUP reload attempts, by outcome. Labels:resultis one of:success– safe fields swapped in place, integrity hash re-recorded, live config updated.restart_required– one or more fields that need a full restart changed; live settings unchanged.error– YAML parse, validation, re-hash, or subsystem apply failure; live settings unchanged. If signing succeeded before an apply failure, the live integrity hashes follow the signed config.noop– file edit produced no semantic change (identical values, whitespace edit, etc.). Alert target:rate(csm_config_reloads_total{result="error"}[5m]) > 0paired with a short for-clause.
Auto-response
csm_auto_response_actions_total{action}(counter): every auto-response action fired, by class. Labels:actioniskill,quarantine, orblock. Incremented once per finding the correspondingAuto*helper produces, so a batch blocking four IPs in one cycle adds 4 toaction=block. Useful for detecting response storms. Verified admission blocks also increment this counter, from every source admission runs (scan, challenge, central and incident), once per engine result, without waiting for alert delivery:rate(csm_auto_response_actions_total[5m]).csm_admission_handoff_seconds(histogram): how long a response funnel waits to hand an automatic response to the admission ledger, which applies, previews or forecasts it. Preparing evidence for a possible later response adds no sample; selected refusals and responses each add one. Buckets: 0.001 s .. 1 s. The share of handoffs at or under 10 ms iscsm_admission_handoff_seconds_bucket{le="0.01"} / csm_admission_handoff_seconds_count; it should stay at or above 0.99.
Challenge
csm_challenge_routed_total{check}(counter): IPs routed to the proof-of-work challenge, labelled by the sourcecheckthat flagged them (e.g.http_scanner_profile,wp_login_bruteforce,ip_reputation). Graph per-detector challenge volume withsum by (check) (rate(csm_challenge_routed_total[5m])).csm_challenge_escalated_total{outcome}(counter): challenge entries that timed out and were escalated to a hard block, by firewalloutcome:live(a new block landed),noop(the IP was already blocked), plusdry_run,allowed, andallowlisted.outcome=liveis how many challenges actually became blocks, including those applied by admission after the timeout handoff.csm_challenge_passed_total{method}(counter): challenged addresses that got through, bymethod:pow(solved the puzzle),captcha,verify_cookie(passed earlier and came back),admin_cookie, andcrawler(verified search crawler).
Firewall blocks
csm_firewall_block_outcome_total{outcome,source}(counter): every IP block attempt, labelled by firewalloutcome(live,dry_run,allowed,allowlisted,noop,protectedfor refused infrastructure IPs,errorfor engine failures or an unavailable engine) and bysource(scan,challenge,incident,central_intel,cli,web_ui, orunknownfor an invalid internal source). All combinations are exported at zero from startup, and unexpected outcome or source values collapse into the fixederrororunknownbuckets instead of creating new labels. Admission execution reports into these same counters. Failed or unknown batch outcomes count as errors; forecasts and queued deferrals add no attempts. Retrying an outcome write or audit delivery adds no count. This is the aliveness signal for auto-response: alert whensum(rate(csm_firewall_block_outcome_total{outcome="live",source!~"cli|web_ui"}[1h])) == 0while attack findings keep coming, and watchoutcome="error"for a dying engine.
Retention (when retention.enabled: true)
csm_retention_sweeps_total(counter): number of retention sweep cycles completed since daemon start. A flatline after a restart means the sweep goroutine is not scheduling; a healthy daemon increments this on everysweep_intervaltick.csm_retention_deleted_total(counter): cumulative entries deleted across thehistory,attacks:events, andreputationbuckets. Spikes on the first sweep after enabling retention (initial backlog), then settles to the steady-state churn. Useful for estimating when the file might benefit from a restart or acsm store compactmaintenance window.
PHP-relay (email abuse, cPanel only)
All series are prefixed csm_php_relay_. Registered when email_protection.php_relay.enabled: true and the host is cPanel; otherwise zero across the board. See Real-time detection.
csm_php_relay_findings_total{path}(counter): findings emitted per detection path. Labels:pathis one ofheader,volume,volume_account,fanout(and laterbaseline,reputationfor Stages 2-3). Userate(...)to spot detection storms; a sudden rate jump onheadertypically means a contact-form vulnerability is being exploited, onvolume_accounttypically means an account password was leaked.csm_php_relay_actions_total{action,result}(counter): auto-freeze invocations attempted. Labels:actionis currentlyfreeze;resultisokorfail. Pair withcsm_php_relay_findings_totalto confirm freeze keeps up with detection.csm_php_relay_action_gone_total(counter): messages already absent from the spool by the timeexim -Mfran. Normal queue churn; not a failure. Sustained growth means the spool is moving fast and the freezer is racing the queue runner.csm_php_relay_path_skipped_total{path,reason}(counter): path evaluation that bailed before producing a finding. Labels:pathmatches the finding labels above;reasonenumerates the gate that fired (e.g. ignore-list match, missing scriptKey).csm_php_relay_spool_scan_fallbacks_total{reason}(counter): AutoFreeze fell back to a full spool walk to find msgIDs. Labels:reasoniscapped(the in-memoryactiveMsgsper script hit its cap, so a fresh disk walk was needed) orreputation(a late reputation finding arrived for a script with no liveactiveMsgs). Sustained growth oncappedmeans a single script is firing faster than the in-memory window keeps state for; consider raisingheader_score_volume_minor adding an ignore.csm_php_relay_active_msgs_capped_total(counter): per-scriptactiveMsgsset hit its cap and dropped the oldest entry. Counts the eviction event itself; the next freeze for that script will land incsm_php_relay_spool_scan_fallbacks_total{reason="capped"}.csm_php_relay_windows_active{kind}(gauge): retained per-script / per-IP / per-account window state. Labels:kindisscript,ip, oraccount. Sized by Flow E sweep cadence (5 min for windows, 24 h retention for accounts); flat values across hours are normal.csm_php_relay_msgid_index_size{layer}(gauge): msgID dedup index size by storage layer. Labels:layerismemory(in-process map) orbbolt(persisted batch writer). Memory ceiling is 200k entries; bbolt grows freely until the 25 h Flow E sweep prunes it.csm_php_relay_msgindex_persist_dropped_total(counter): bbolt persist queue overflow drops (the 4096-deep buffered channel was full when the watcher tried to enqueue). Should be zero in steady state; a non-zero value means the bbolt writer is blocked on disk and the in-memory dedup is the only thing protecting against double-fire on a queue-runner re-write.csm_php_relay_msgindex_persist_errors_total(counter): bbolt commit failures from the async batch writer. Each bump also emits a Criticalemail_php_relay_msgindex_persist_failedfinding. Disk-full or permissions issue on/var/lib/csm/state/csm.db.csm_php_relay_inotify_overflows_total(counter): kernelIN_Q_OVERFLOWevents on the spool watcher. Each one triggers a bounded recovery scan (default cap 1000 files); if the cap fires, also emitsemail_php_relay_overflow_scan_truncatedCritical. Sustained growth means the spool is churning faster than inotify can keep up – usually a backup restore or a real attack.csm_php_relay_spool_read_errors_total(counter):emailspool.ParseHeaderserrors on-Hfiles the watcher tried to consume. Usually transient (file disappeared between inotify event and open) and self-correcting; sustained growth points at a permissions or filesystem problem.csm_php_relay_userdata_errors_total(counter):cpanelUserDomainsresolver errors reading/var/cpanel/userdata/. Used by the Path 1Frommismatch check; errors here mean Path 1 is potentially undercounting until the read recovers.
Anti-spam learning
All series are prefixed csm_antispam_learning_. Pass totals and state gauges update after each completed Spam folder pass while learning runs. Contributions are counted when committed, including background retries. No series carries an account, mailbox or UID label. See Anti-spam.
csm_antispam_learning_reports_admitted_total(counter): Spam folder reports the learner accepted.csm_antispam_learning_contributions_committed_total(counter): cluster contributions committed from those reports.csm_antispam_learning_skips_total{reason}(counter): entries and reports skipped.reasonis a fixed code such astagged,malformed,opted_out,oversizeoraccount_capacity. On Postfix hosts it also counts Dovecot mailboxes not watched, such asdriverorno_junk, andcancel_unconfirmedwhen a pass could not confirm that a domain or mailbox was removed and so dropped no reports.csm_antispam_learning_pending_reports(gauge): reports waiting after the last pass. When discovery was incomplete, including unreadable folders or discovery limits, this is a lower bound.csm_antispam_learning_backlog_age_seconds(gauge): age of the oldest waiting report at the last pass.csm_antispam_learning_last_pass_timestamp_seconds(gauge): when the last pass completed.csm_antispam_learning_live_clusters,csm_antispam_learning_zero_target_records,csm_antispam_learning_body_bytes(gauges): learned clusters, zero-target guards and stored anchor message bytes.csm_antispam_learning_uncertain_intents(gauge): anchor writes whose outcome is still being reconciled.csm_antispam_learning_oldest_unchecked_timestamp_seconds,csm_antispam_learning_audit_completed_timestamp_seconds(gauges): anchor audit progress.csm_antispam_learning_reconstructed_clusters(gauge): missing live anchors newly scheduled for reconstruction since the learner started. Retries, ordinary value corrections and removal of retired anchors do not increase it.
If the learning ledger cannot be read after a pass, the engine-state gauges
(live_clusters, zero_target_records, body_bytes, uncertain_intents,
oldest_unchecked_timestamp_seconds, audit_completed_timestamp_seconds
and reconstructed_clusters) report NaN until a successful read.
Signature retroactive rescans
csm_signature_rescans_total(counter): rescans queued by a change to a tracked signature file’s content that completed. A rescan completes when the rolling deep YARA scan has scanned every file since the change. Re-installing identical rules, as a package upgrade or a repeated download does, does not count. Multiple updates before a rescan completes coalesce into one.
The watcher compares content hashes when a file’s mtime or size changes; symbolic links use the target file’s metadata. Failed reads and files replaced during hashing are retried without discarding the last verified state. First observations and file removals do not queue a rescan.
The rolling deep YARA scan reads files in path order and resumes each deep run where the previous one stopped, so one pass spans many deep runs. A rules update does not restart the pass: restarting on every update would never reach the files that sort last when updates come faster than a pass. The scan records where it stood when it took the update, and the rescan completes once the scan has wrapped and come back to that point. Every file is then scanned with the new rules within one pass. An update during a rescan starts a new one from where the scan then stands.
If the set of scan roots changes, or a root cannot be listed, the rescan starts a new lap from where the scan stands; the scan itself keeps going and still covers every root it can read. On a host whose accounts change faster than one pass, the rescan can stay pending; scanning is unaffected.
The rolling scan uses the YARA rules. YAML rule updates reach existing files only when those files change, as before.
A queued rescan is stored with the rule state that caused it, and the scan
stores its progress with its cursor, so daemon restarts keep both. The
saved queue is restored even if the state store becomes available after
the first watcher tick. Invalid queue records log a warning and are
repaired by queuing a fresh rescan; the scan neither tracks nor clears a
queue it cannot read. A deep run cancelled by shutdown or by the check
timeout saves no progress. Coverage gaps, reported by yara_scan_incomplete, do
not keep the queue pending.
Setting detection.rescan_on_signature_update to false stops the
watcher and pauses the queue: the scan keeps running but neither starts
nor completes a rescan. Re-enabling it resumes the saved work.
csm check-deep advances the same rolling scan, so it can complete a
queued rescan and increment this counter. Its findings are returned and
kept in scan history but not alerted or acted on, and the live scan does
not revisit those files until its next pass. Use csm run-deep when
findings should alert.
State from older builds contains only mtimes. The first successful read adds a hash without queuing a rescan if the recorded mtime still matches. A moved mtime without a recorded hash conservatively queues one rescan.
Counter reset semantics
Prometheus counters in CSM live in process memory. They reset to zero
whenever the daemon restarts (config change, binary upgrade, crash
recovery). This is the standard behaviour for every
Prometheus-instrumented daemon; Prometheus’s scrape pipeline detects
counter resets on its own and rate(), increase(), and
rate_over_time() all handle them correctly.
Operators should not alert on “counter decreased across a scrape” as
a failure condition. Alert on rate() or increase() of a counter
over a window long enough to absorb expected restarts.
Persisting counters across restarts would require writing to bbolt on every increment, which would not pay for itself. If a specific metric needs restart-stable behaviour later, a gauge-over-the-bbolt-counter pattern can be added for that one case without affecting the rest.
Caveats
- Scrape the web UI’s HTTPS port, not a separate listener.
curl -k/insecure_skip_verifyis appropriate only when the cert is self-signed and the network path is trusted. Pin a CA for anything else.- Prometheus label cardinality: per-account and per-IP labels are deliberately not exposed. Shared-hosting deployments with 1000+ cPanel users would otherwise overwhelm a Prometheus server.
- Metric vectors cap label-value combinations at 1000 children per
metric, including the overflow bucket. Once a vector reaches that
cap, new combinations are aggregated under
_overflow_.
Not instrumented (yet)
- Per-account labels on any metric. Deliberately off: shared-hosting deployments with 1000+ cPanel users would blow out Prometheus cardinality.
- Fanotify inline auto-response actions (the quarantine-while-
seeing-the-write path in
fanotify.go). The periodiccsm_auto_response_actions_totaldoes not count those; a follow- up may split the metric or add asourcelabel. - bbolt per-bucket size breakdown,
csm_store_used_bytes, andcsm_store_last_compact_ts. Deferred to the online-compaction follow-up of the retention work (seeROADMAP.md).