Troubleshooting

Use this page when the sniffer starts but does not produce the files or behavior you expect.

No connections*.ndjson File Appears

Check these first:

1. Confirm sniffer.mode: offline
2. Confirm you did not use --dry-run
3. Confirm the output path is writable
4. Confirm packets matched the configured BPF filter
5. Check the sniffer log file

With rotation enabled, output files are named connections.0.ndjson, connections.1.ndjson, and so on. With rotation disabled, the output file is connections.ndjson.

Example:

tail -100 /var/log/pqc-sniffer/pqc-sniffer.log

Permission Denied on Live Capture

Live capture typically requires root privileges or equivalent packet-capture capabilities.

sudo pqc-sniffer --config /etc/pqc-sniffer/config.yaml --foreground

Running without sudo is expected to fail. The binary intentionally carries no file capability, because setcap cap_net_raw+eip would grant packet capture to every local account able to execute it. Use sudo for manual runs.

If the service hits this error, the ambient capability is not reaching it. Check the unit rather than the binary:

getcap /usr/bin/pqc-sniffer                          # expected: no output
systemctl show pqc-sniffer -p AmbientCapabilities    # expected: cap_net_raw

If AmbientCapabilities is empty, the unit has been edited or overridden; restore AmbientCapabilities=CAP_NET_RAW and run sudo systemctl daemon-reload.

Binary Reports an Embedded Library Search Path

This check should produce no output:

readelf -dW /usr/bin/pqc-sniffer | grep -E 'RPATH|RUNPATH'

If it prints a line such as:

0x000000000000001d (RUNPATH)  Library runpath: [/opt/conan2/p/b/libpq.../p/lib:...]

then the binary carries a list of directories that the dynamic loader searches before the system library paths when resolving libpcap, libstdc++ and its other dependencies. Those are build-machine directories. They do not exist on your host, so whichever account is able to create one first decides which library the sniffer loads — and the service holds CAP_NET_RAW.

A path that ends in :, or contains ::, is worse: an empty element means the loader also searches the current working directory.

What to do:

1. Do not start the service.
2. Obtain a package built from a current release. Releases are gated on this check, so a current package cannot carry one; a binary that does was built from an older tree or copied out of a build directory by hand.

3. If you must use the artifact you already have, you can strip the path in place and then re-run the check:

   sudo patchelf --remove-rpath /usr/bin/pqc-sniffer
   readelf -dW /usr/bin/pqc-sniffer | grep -E 'RPATH|RUNPATH'
   

   Treat that as a stopgap. The binary came from a build that should not have produced one, so the rest of that build is worth questioning too.

Re-run the check after every install and upgrade, and keep /opt, /etc/pqc-sniffer and /var/lib/pqc-sniffer owned by root and not writable by unprivileged users.

Pre-Capture Dies Immediately With "Permission Denied"

Symptom, in the application log, on a path root can obviously write:

tcpdump exited immediately after launch (exit code 1 - tcpdump said: dropped privs to
tcpdump tcpdump: /var/lib/pqc-sniffer/pcap/precap.pcap0: Permission denied)

This affected a sniffer started by hand as root (sudo pqc-sniffer), never one started through the systemd unit. tcpdump is built to drop its privileges to its own account -- on RHEL, the tcpdump user -- whenever it is started as root, and that account does not own the 0700 capture directory. Under the unit the sniffer already runs as pqc-sniffer, so tcpdump is not root, drops nothing, and the problem never arises.

Fixed in 2.0.0-11: a root run now pins tcpdump to root so it can write, and logs a warning saying so. If you see this on an older build, run through the service instead:

sudo systemctl start pqc-sniffer

PCAP File Does Not Process

Make sure:

• the path exists
• the file is readable
• the configured capture.interface points to a .pcap, .pcapng, or file:///... input

If the file opens and is then refused with "Unsupported capture link type", its link layer is one the sniffer does not decode. See Capture link types for the supported set and how to convert a capture into one of them. Ethernet, the Linux any cooked formats, raw IP from tunnel interfaces and the loopback types are all read directly.

Service Reports "Activating" and Then Fails to Start

The unit is Type=notify: the sniffer sends READY=1 to systemd once the capture source is open and the worker threads are running, and systemd holds the unit in activating (start) until then. If startup never reaches that point, systemd kills it at TimeoutStartSec (120s) and Restart=on-failure retries (a timeout kill is a failure, so the restart policy applies).

That is deliberate. Before TQR-1393 the unit was Type=simple, so systemd called the service started the moment exec returned — a startup that blocked (a FIFO given as the offline input, an unreachable database) left the unit reported active (running) indefinitely while doing nothing, and the restart policy never fired because the process never exited.

So a unit stuck in activating is now reporting a real problem rather than hiding one. journalctl -u pqc-sniffer -n 50 shows how far startup got; the last log line before the stall names the step that did not complete.

systemctl status also shows the sniffer's own status line once ready, e.g. "Capture running on eth0".

Running the binary by hand is unaffected: with no NOTIFY_SOCKET in the environment the notifications are silent no-ops.

Service Restarted by the Watchdog

pqc-sniffer.service: Watchdog timeout (limit 1min)!
pqc-sniffer.service: Killing process 1797 (pqc-sniffer) with signal SIGABRT.
pqc-sniffer.service: Failed with result 'watchdog'.
pqc-sniffer.service: Scheduled restart job, restart counter is at 1.

The sniffer reports to systemd once every 30 seconds for as long as its main loop keeps working. It stops reporting when that loop has made no progress for a full minute, and systemd then restarts the service. Failed with result 'watchdog' identifies this specific cause, as distinct from a crash or a configuration error.

This is the sniffer asking to be restarted, not systemd losing patience. The service was still running and still reported active (running); it had simply stopped capturing. Detection takes up to about two minutes in total.

A slow dependency does not trigger it. The sniffer allows a full minute of no progress before it stops reporting, which is far longer than any wait it performs normally. In offline mode — the only mode in this release — nothing the main loop does waits on the network at all.

Where a database is in use, most failures cost nothing. One that is slow, stopped, or whose host has gone away keeps the loop cycling in about a second, measured over a four-minute outage without a single restart. The exception is a server that holds the connection open but stops answering on it — a frozen VM, a stuck backend. That does stall the loop, and costs exactly one restart: the restarted sniffer cannot re-establish the connection, so it falls back to file logging and runs normally from there. A single watchdog restart that coincides with database trouble and then holds steady is expected, not a fault to chase.

What to look at. A crash report is written to /var/log/pqc-sniffer/pqc-sniffer-crash.log, with a stack trace recording where the process was stuck when it was killed. That is the point of using SIGABRT rather than simply restarting: without it the same hang would have to be rediscovered the next time. The shipped binary is stripped and loads at a different address on every run, so resolve each frame's offset — the pqc-sniffer+0x... value, not the raw address beside it — with the .debug artifact archived for that exact build:

addr2line -e pqc-sniffer-<version>.<arch>.debug -f -C <offset>

It prints the function and the source file and line. A raw address resolves to ?? ??:0. Frames marked (shared library) are in system libraries and are not resolved with this file.

Send that file with the log if you raise a support case; it names the call the sniffer was stuck in.

If it repeats. Five restarts within five minutes exhausts the unit's start limit, after which systemd stops retrying and leaves the unit failed:

systemctl reset-failed pqc-sniffer && sudo systemctl start pqc-sniffer

A single restart that then holds steady needs no action beyond keeping the crash report. Repeated restarts are a genuine fault — collect the crash reports and the journal for the whole window rather than restarting in a loop.

systemctl stop Takes 30 Seconds and Then Kills the Service

Stopping PQC Network Sniffer...
pqc-sniffer.service: State 'stop-sigterm' timed out. Killing.
pqc-sniffer.service: Killing process 2043 (pqc-sniffer) with signal SIGKILL.

The journal's last line from the sniffer itself is one of these, and Shutdown complete never appears:

Stopping packet capture...
Stopping raw socket capture on interface: lo

Affects releases before 2.0.1, and only these two settings:

• capture.interface: "any"
• capture.enable_raw_sockets: true

A named interface (eth0, lo) with raw sockets left off is unaffected — that path stops in well under a second.

Both settings start an extra capture thread that waited on its socket with no timeout, and neither wakeup mechanism the shutdown relied on actually worked against that wait. The service therefore sat in stop-sigterm until TimeoutStopSec=30s expired, and systemd sent SIGKILL.

What it costs. SIGKILL cannot be caught, so everything the graceful path does after stopping capture is skipped — including the final flush and the end-of-capture summary. Records already written to disk are intact; anything still buffered at the moment of the stop is not. Every systemctl restart pays this, which makes routine config reloads the most likely place to lose records.

Resolution. Upgrade to 2.0.1 or later, where both capture threads poll with a bounded wait and observe the stop request within capture.timeout_ms.

On an affected build, capture from a named interface rather than "any" and leave enable_raw_sockets at its default of false. If "any" is genuinely needed, stop the service during a quiet window and treat the 30-second kill as expected rather than as a second fault.

Connections Established but Never Recorded

If sessions complete normally yet produce no NDJSON or database records — and the heartbeat reports no queue drops and no kernel drops — the connection is completing into a state the writer will not accept.

Before TQR-1392 this had a specific cause worth knowing about, because the symptom named nothing. A duplicated FIN advanced the TCP state machine as though it were the peer's FIN, sending the connection through CLOSING to TIME_WAIT without ever marking it writeable. Every terminal notification the writer acts on is gated on that flag, so a complete SSH login was recorded as nothing at all, with no error anywhere. A genuine simultaneous close — both sides sending FIN at once, with no duplication involved — hit the same path.

Both are fixed. If you see this symptom on a current build:

• Check whether the capture is duplicated. A sniffer sharing a hypervisor host with a monitored guest sees every packet twice. Set capture.duplicate_window_ms to 2–5 and look for the Duplicate packets suppressed: line — a figure near 50% confirms it (see Duplicated packet delivery).
• Confirm the handshake was captured. A flow adopted mid-stream has no handshake_analyzed, which the non-TLS output gate also requires.
• For TLS, both a ClientHello and a ServerHello are required. A capture that missed one direction produces a connection that is tracked but not written.

"Not post-quantum" or "rather than TLS 1.3" warnings on outbound connections

The sniffer reports what its own outbound TLS connections negotiate — telemetry to Quantum Command and the PostgreSQL connection — and warns when that is weaker than intended (TQR-1390). These warnings are about the sniffer's management traffic, not about the traffic it is analysing.

A healthy connection logs one INFO line and no warning:

Quantum Command registration TLS: TLSv1.3 / X25519MLKEM768 (hybrid post-quantum)

negotiated TLSv1.2 rather than TLS 1.3 — the endpoint does not support TLS 1.3. Hybrid ML-KEM exists only in TLS 1.3, so this connection's key exchange is classical and recorded traffic would be readable by a future quantum attacker. The sniffer sets a TLS 1.2 floor and does not require 1.3, so the connection still succeeds. Fix this at the endpoint or the TLS-terminating proxy in front of it.

negotiated TLS 1.3 with key exchange 'x25519', which is not post-quantum — the protocol is right but no ML-KEM group was agreed. On a packaged build this means the peer did not offer a hybrid group. Ask whoever runs the endpoint whether it supports X25519MLKEM768.

The other cause — a build that linked an OpenSSL older than 3.5, which has no ML-KEM at all — can no longer reach a released package. Since 2.0.1 the build refuses to configure against an OpenSSL below 3.5 or a libpq below 15.18 instead of quietly falling back to the distribution's (TQR-1422), and every build configuration resolves its dependencies from one Conan recipe, conanfile.py, whose options cannot change a version (TQR-1411). If you built from source yourself, those two checks are still the first things to look at.

Expected when running a validated FIPS module

A CMVP-validated OpenSSL FIPS provider may contain no ML-KEM, in which case classical key exchange is the correct and expected result rather than a fault. If you have deliberately deployed such a module, this warning is noise — see security.

"Configuration ... could NOT be written / persisted" after a database sync

Configuration from database (version 4 -> 5) is active in this process but could NOT be
written to '/etc/pqc-sniffer/config.yaml': ... It will be lost on restart. This is
expected under ProtectSystem=strict, which keeps /etc read-only by design.

This is normal on a packaged deployment and does not indicate a failed sync. The unit sets ProtectSystem=strict, which makes /etc read-only to the service, so the write-back is refused by design (TQR-1391).

What actually happened:

• The configuration was fetched and merged, and the new values are active in the running process. The sync worked.
• Only the write to disk failed, so the change does not survive a restart.

Before TQR-1391 this path logged "Configuration synchronized" regardless of whether the write succeeded, which told operators a change was durable when it was not. The warning exists to correct that, not to report a new fault.

If you want database configuration to persist across restarts, the value has to reach the config file by some route the sandbox permits — deploy it with the package, or relax ProtectSystem deliberately and accept the consequence. Do not treat the warning as something to silence.

Service Exits Immediately — Exit Code Reference

systemctl status pqc-sniffer reports the exit code as status=N. The sniffer uses these:

Code	Name	Meaning	Restarted by systemd?
0	Success	Clean exit. Offline PCAP analysis reaching end of file exits 0.	No — Restart=on-failure
1	ConfigError	Malformed config, failed validation, or security.fips_mode: required when FIPS could not be activated.	No — RestartPreventExitStatus=1, because retrying a bad config every 10s achieves nothing
2	DatabaseError	Database unreachable or rejected the connection. Not reachable in this release — connected (PostgreSQL) mode is unavailable, so nothing connects to a database. The code is reserved.	Yes — treated as transient
3	NetworkInterfaceError	Capture interface missing or could not be opened.	Yes
4	PermissionDenied	Insufficient privileges, usually a missing CAP_NET_RAW.	Yes
5	SignalReceived	Terminated by SIGTERM/SIGINT — which is what systemctl stop and systemctl restart send.	No — SuccessExitStatus=5 declares it a clean exit
6	ReplayIncomplete	An offline replay read its file but could not represent it faithfully — the connection table filled, or output could not be written. The NDJSON exists and is short.	No — re-running unchanged would fail identically
10	InternalError	Unhandled exception.	Yes

Two of these are commonly misread:

• Exit 0 with the service showing inactive (dead) on an offline run is correct. The file was processed and the sniffer finished. It is not restarted because it did not fail.
• Exit 1 never auto-restarts. If the service is not running and journalctl shows a configuration error, fix the config and systemctl start explicitly — waiting for a restart that will not come is the usual confusion here.
• status=5/NOTINSTALLED after a normal stop or restart is not a fault. NOTINSTALLED is systemd's generic label for exit code 5, not a statement about the package. Exit 5 is a clean shutdown on SIGTERM. On builds before this one the unit did not declare that, so a routine systemctl restart also logged Failed with result 'exit-code' and left systemctl is-failed reporting failed. If you are seeing that, the service is fine — upgrade to pick up the fix.

Service Killed by SIGALRM

systemctl status reporting code=killed, signal=ALRM does not mean the sniffer timed something out. It means the crash handler hit its own deadline.

The handler runs under a five-second alarm, so that a crash can never leave the service wedged. Before it was added the service stayed in the handler for the full TimeoutStartSec (120s), and systemctl restart blocked behind it for the same period.

The handler no longer calls into the system unwinder, which was what used to hang; it reads the stack directly. The remaining reason the deadline can fire is a write to the crash log that does not complete, which points at the log directory rather than at the sniffer -- a network or otherwise stalled filesystem under /var/log/pqc-sniffer. Check that directory before investigating the crash itself.

SIGALRM is then the reported cause of death, and the signal that actually crashed the process is in the crash log:

sudo head -10 /var/log/pqc-sniffer/pqc-sniffer-crash.log

=== PQC-SNIFFER FATAL SIGNAL ===
signal=11 (SIGSEGV (Segmentation fault))
code=1
fault_address=0x7fff12345678
instruction_pointer=0x556194101498
load_base=0x556193ea9000
instruction_offset=0x258498
pid=1204547
epoch_seconds=1762277415

That block is written before anything that can block, so it is present even when the stack trace below it is missing. Send it with the bug report: instruction_offset is the value that locates the fault in the build's .debug artifact. It reads outside-pqc-sniffer when the crash was inside a system library.

A short stack trace is not a truncated one

A trace in the crash log may be shorter than one from a debugger, and may stop inside the sniffer rather than at main. That is expected: the handler follows the stack itself rather than asking the system unwinder, which is what makes it unable to hang, and the trail ends where it reaches a bundled third-party library. Send the report as it is.

Where no trace could be produced at all, the log says so in place of the frames rather than leaving the section empty:

  [no frames: stack not walkable]
  [stack walk skipped: this thread is not registered with the crash handler]

Both are worth reporting. The first usually means the stack itself was destroyed, which is part of the diagnosis.

Service Will Not Start: "the 'b64:' credential format has been removed"

Applies wherever the database password was stored with a b64: prefix -- including offline deployments. The database section of the configuration is parsed whatever the operating mode, so the value stops startup even on a sniffer that never connects to PostgreSQL.

b64: was base64 encoding, never encryption. Anyone able to read config.yaml -- or a backup of it -- could recover the password with one shell command, so the format has been removed. The previous release still read it and logged a warning on every use; that release was the migration window.

The service now refuses to load such a configuration:

the 'b64:' credential format has been removed. Convert the configuration with
'pqc-sniffer --config /etc/pqc-sniffer/config.yaml --migrate-credentials', then
treat that password as disclosed and change it at the source

It fails rather than carrying on, because the alternative is worse: treating the value as plaintext would send the literal string b64:cGFzc3dvcmQ= to PostgreSQL and produce an authentication failure with nothing in the logs to explain it.

Your configuration file is not modified by the failed load, so the stored value is still there to convert. One command does it:

sudo pqc-sniffer --config /etc/pqc-sniffer/config.yaml --migrate-credentials
sudo systemctl restart pqc-sniffer

--config is required, and older builds did not say so

--setup is the only mode that runs without --config. Builds before 2.0.0-13 printed this command without it — both on the failed start and during dnf upgrade — and run that way it stops with ERROR: Configuration file is required. Use -c/--config or -s/--setup and exits 1. Nothing is converted and nothing is damaged; just name the file as shown above and run it again.

That converts every b64: credential in the named file to the encrypted aes: form, in place. Your comments and formatting are preserved -- only the credential value changes -- and the file is replaced atomically, so an interrupted run cannot leave a half-written configuration. Point --config at a file elsewhere if yours is not in the default location.

The file's owner is kept. Its permissions are kept too, with one exception: a config owned by the pqc-sniffer account that is readable by its group or by everyone is tightened to 0600 on the way past, and the command says so. The service reads the file as its owner, so nothing loses access, and a file holding a database password has no business being readable by anything else. A packaged install is already 0600 pqc-sniffer:pqc-sniffer -- the RPM sets that on every install and upgrade -- so this normally has nothing to do.

A loose config owned by somebody else is left exactly as found, with a warning. On an installation predating 2.0.0-9 the file is 0640 root:pqc-sniffer and the service reads it through the group; dropping that permission would leave a config the service cannot open. Settle the ownership and the mode together if you see that warning:

sudo chown pqc-sniffer:pqc-sniffer /etc/pqc-sniffer/config.yaml
sudo chmod 0600 /etc/pqc-sniffer/config.yaml

It is safe to run when there is nothing to do, and safe to run twice: it reports No legacy 'b64:' credentials ... and exits 0. Upgrades print the same instruction, so you may have seen it during dnf upgrade.

If it refuses -- because a value does not decode as base64, or no encryption key is available -- it changes nothing and says why. Convert by hand in that case:

# Recover the plaintext (everything after "b64:")
echo 'cGFzc3dvcmQ=' | base64 -d

# Seal it, and paste the printed aes: string into database.password
sudo /usr/share/pqc-sniffer/encrypt_password.py

encrypt_password.py needs python3-cryptography; the package recommends it, so an ordinary install already has it (dnf install python3-cryptography if not). --migrate-credentials is built into the binary and needs no Python.

Rotate the password as well

A credential that was stored as b64: should be treated as disclosed. Change it in PostgreSQL, then encrypt the new one -- re-encrypting the old value protects it from here on but does nothing about wherever the config file has already been.

Config Validation Fails

Run:

sudo pqc-sniffer --config /etc/pqc-sniffer/config.yaml --validate-config

Common causes:

• a required section left out -- e.g. "offline_output is a required section, but the configuration does not define it". sniffer, capture, offline_output and logging must all be defined; see Required Sections. A section written with nothing under it counts as left out. New in 2.1.0: a configuration with no offline_output section used to load and write to the default path.
• non-absolute offline_output.output_file
• offline_output.output_file contains ..
• output or log path outside the permitted service directories — e.g. "output path is outside the permitted service directories". Use a path under /var/log/pqc-sniffer or /var/lib/pqc-sniffer, or set PQC_SNIFFER_ALLOWED_OUTPUT_DIRS for a custom location (see Output path security).
• offline PCAP input outside the permitted capture directories — e.g. "Refusing to open PCAP file '/tmp/x.pcap': PCAP input path is outside the permitted capture directories: /tmp". Offline inputs default to /var/lib/pqc-sniffer and /etc/pqc-sniffer; approve another location with PQC_SNIFFER_ALLOWED_PCAP_DIRS (see Offline input path security).
• offline PCAP input that is not a regular file — e.g. "PCAP input is not a regular file (FIFOs, devices, sockets and directories are refused)". Point capture.interface at an actual savefile. Before TQR-1394 a FIFO here hung startup silently instead of reporting anything.
• offline PCAP input that is a symlink, world-writable, or owned by another user — the file is refused rather than read. Copy the capture into an approved directory owned by the service account instead of linking to it.
• offline PCAP input over the size cap — e.g. "PCAP input is 9663676416 bytes, over the configured limit of 4294967296 bytes". Raise capture.max_offline_file_size_mb, or set it to 0 to disable the cap.
• output or log path is a symbolic link, or its parent directory is world-writable or owned by an unexpected user

"output directory has an unexpected owner" under sudo

On builds before 2.0.0-12 this appeared when the command above was run as root against the sniffer's own directory:

Configuration validation failed:
offline_output.output_file: output directory has an unexpected owner: /var/log/pqc-sniffer

The service started on the same file, which is what made it confusing. The sniffer only trusted a directory owned by root or by the account running it. Under the unit that account is pqc-sniffer, which owns /var/log/pqc-sniffer, so the check passed; under sudo the account is root, the directory still belongs to pqc-sniffer, and the same directory was refused. It affected every sudo pqc-sniffer command, not just --validate-config.

Fixed in 2.0.0-12: the service account is trusted alongside root and the current user. A directory owned by anyone else is still refused. On an older build, run the check as the service account instead:

sudo -u pqc-sniffer pqc-sniffer --config /etc/pqc-sniffer/config.yaml --validate-config

The same release also stops a root run from stranding the service. Files created by sudo pqc-sniffer are handed to the account that owns the directory they are written into, so the service can still open its own log and output afterwards. Before that, one troubleshooting run as root left root-owned files behind and every subsequent start reported Permission denied on them until someone corrected the ownership by hand:

sudo chown -R pqc-sniffer:pqc-sniffer /var/log/pqc-sniffer /var/lib/pqc-sniffer

- invalid buffer or rotation values - missing required sections

Logs Look Fine but Output Is Empty

This usually means one of two things:

• no traffic matched the BPF filter
• the capture source did not contain the handshake or protocol events you expected

It can also be expected behavior for unknown plain traffic. Offline mode suppresses generic records when the application protocol remains unknown, including plain DNS/UDP that is not classified as a supported crypto protocol. Narrow the BPF filter to TLS, QUIC, SSH, or known HTTP traffic when validating crypto telemetry.

A Protocol Is Missing From the Report

The sniffer reports on what it was given. It does not report on what it never received, and it cannot tell the difference between a protocol that is absent from the network and one the capture filter discarded. Both look like a clean result.

If you expected findings for a protocol and got none — no IKEv2, no QUIC, no RADIUS, no WireGuard — check the capture scope before concluding the network is clear.

1. Read the effective filter from the log. It is written once at capture start:

[INFO] Effective BPF filter: tcp or udp or proto 47 or ip6 proto 0 or ip6 proto 43 or ip6 proto 44 or ip6 proto 51 or ip6 proto 60 or ip6 proto 135

That is the default, and it admits every protocol the sniffer can parse. If the line shows something narrower, that filter — not the network — decided what could be found.

2. Check what the filter excludes.

Filter	Silently excludes
tcp port 443 or tcp port 22	QUIC/HTTP-3, DTLS, IKEv2, NTP/NTS, WireGuard, RADIUS, SMTPS, TACACS+, DNS-over-TLS, GRE/ERSPAN, every alternate TLS/SSH port
tcp	every UDP protocol above, and GRE/ERSPAN
ip proto 47	GRE carried over IPv6 — ip proto is IPv4-only; use proto 47

3. Confirm the traffic is actually on the wire, independently of the sniffer:

sudo tcpdump -i <interface> -c 5 'udp port 500'     # IKEv2, for example

If tcpdump sees it and the sniffer reports nothing, the filter is the difference.

Deployments provisioned before 2.0.0-6

Installs configured through the database loader or the setup wizard before this release received tcp port 443 or tcp port 22 as a default without it being written to config.yaml. Those deployments captured HTTPS and SSH only. Any report they produced should be treated as covering those two protocols rather than as a full assessment — re-run after upgrading. See Choosing a capture filter.

An Offline Replay Exited 6

Exit 6 (ReplayIncomplete) means the run read the whole capture file and wrote NDJSON, but that NDJSON does not represent the capture. It is deliberately separate from both 0 and 10: nothing malfunctioned, and the output file looks perfectly normal, so a CI gate that only checked for a crash or an empty file would pass it.

The log names which of the two causes applied.

Connection table full

Connection table full during offline replay: 41093 flow(s) refused at a ceiling of 100000.
Those flows are absent from the NDJSON, so this run is NOT comparable with a live capture of
the same traffic.

That is the fast mode message, and that run stopped where it is. A timestamp-paced replay reports the same condition differently, because it keeps going:

Connection table full during paced replay: 41093 flow(s) refused so far at a ceiling of
100000. The replay CONTINUES to the end of the file and will admit flows again as entries
expire, which is what a live capture of this traffic would have done.

Either way the run ends by naming the total:

Connection table refused 41093 new flow(s) during this replay, at a ceiling of 100000. The
NDJSON is short by whatever those flows would have produced.

In fast mode this is almost always a consequence of the replay rather than of an undersized table, and the distinction decides which fix is right.

Connection entries expire on the wall clock. Fast replay reads the file as quickly as the workers accept packets, so a capture spanning an hour can be replayed in seconds — during which nothing ages out. Every flow in the file is therefore open simultaneously, which is not a state the traffic was ever in. A live capture of the same traffic would have retired old flows continuously and might never have exceeded a few thousand concurrent.

Two fixes, in order of preference:

1. Pace the replay so expiry sees the intervals that actually occurred:

capture:
  offline_replay_mode: timestamp-paced

This is the correct fix when the goal is to compare offline output against live output. The run then takes as long as the capture did.

Pacing also changes what a full table does. A paced replay does not stop at the first refusal — it reads to the end of the file, refusing new flows while the table is full and admitting them again as entries expire, exactly as a live capture of the same traffic would have. It still exits 6, because the output is still short of the file.

If a paced run still reports refusals, then pacing has done its job and the replay is no longer the explanation: the traffic genuinely needs more than connection_tracker.max_connections allows, and a live capture of it would have refused flows as well. That is a real result about this configuration, and the fix below is the one that applies.

1. Raise the ceiling past the number of flows in the whole file:

connection_tracker:
  max_connections: 500000

Correct when the goal is bulk analysis and you simply need every flow tracked. Budget roughly the total flow count of the file, not its concurrent peak — fast replay makes those the same number.

If a paced replay still reports a full table, the ceiling really is too low: the traffic genuinely held that many flows open at once, and a live capture on this configuration would have refused them too. That is a finding about the deployment, not about the replay.

Output could not be written

Offline output could not accept a connection record: the writer has failed I/O.

The writer hit a real I/O error — a full disk, a revoked permission, a removed mount. An offline run waits indefinitely for writer space rather than discarding records, so this is never mere slowness; a writer that is merely behind is waited out. Check free space on the offline_output.output_file filesystem and the service user's access to it, then re-run.

Offline Runs Reporting Huge Queue-Drop Counts

On builds before this one, an offline replay could finish reporting millions of queue drops while having lost nothing at all.

Offline replay has always waited for queue space rather than discarding packets, but it waited by retrying the same call a live capture uses — and that call counts a full queue as a dropped packet. A packet waiting on a busy shard was therefore counted once per retry, every 100 microseconds, for as long as it waited. The packet was delivered; the counter said otherwise.

The counting now belongs to the caller, so a replay that loses nothing reports nothing. If you see a non-zero Packets dropped (internal queue) on an offline run of a current build, it is real — but note that it should not be possible, and is worth raising.

The related counter is new and distinct:

Packets abandoned (replay stopped early): 0

That one counts packets still in hand when the run was stopped — a cancelled replay, not a pipeline that fell behind. A non-zero value means the run did not reach the end of its file, so the NDJSON is short of whatever those packets would have produced.

Fewer Records Than Connections

If a client completed more handshakes than the sniffer wrote records, the difference is reported rather than left unexplained. Every tracked connection ends in exactly one of four places, and the shutdown statistics account for all of them:

Connections tracked: 4973
Connections dropped before output: 37 (incomplete TLS handshake: 37, protocol never identified: 0, other: 0)
Connections still open at shutdown (never written): 0
Records written to output: 4936
Records discarded by output queue: 0

Dropped before output. By default the sniffer writes a TLS connection only when it parsed both the ClientHello and the ServerHello, and then when the connection ends, so a connection whose handshake was partly missed is discarded rather than reported with unknown crypto. That is deliberate — it keeps the output trustworthy — but it means packet loss during a handshake removes the whole connection. Set capture.capture_incomplete_handshakes: true to record them anyway when you are diagnosing exactly this. See Emission Rules for the full set of rules, including what changed in 2.1.0.

Protocol never identified. Traffic the sniffer has no parser for — ESP, plain DNS, proprietary protocols — is counted here and not written. Before 2.0.1, four protocols it does parse were counted here as well, and so produced no records: DTLS, IKEv2 on UDP 4500 (NAT traversal), plaintext MQTT, and QUIC flows that were already running when capture started. If an older build's counter is large on a network with those protocols, that is why.

Still open at shutdown. Connections that never reached a close, reset, or timeout. A flow whose closing packets were missed stays open indefinitely, so a non-zero count here on a short run is normal; a large one on a long run suggests the capture is missing the end of flows.

Discarded by the output queue. A different loss, one stage further down: the connection was analysed, a record was produced, and the queue between the sniffer and the writer thread was full when it arrived. These are connections the sniffer understood completely and then threw away.

Any non-zero discard count is missing evidence. The other three outcomes each describe a connection the sniffer could not fully account for; this one describes a connection it could, and lost anyway. Nothing downstream can reconstruct it. See offline_output.max_queue_size for what to do about it — briefly, raise the queue if the loss is bursty, and reduce what is written if it is steady.

A fifth thing to check when handshakes come up incomplete. Dropped before output says a handshake was never completed; it does not say why. One cause the sniffer can now name is its own reassembly buffer filling:

TCP fragments dropped (reassembly buffer full): 0

Each drop is a fragment refused because that half-stream already held 1000, which leaves a gap the handshake cannot be parsed across. Until this counter existed the loss was silent and looked identical to a handshake genuinely cut short on the wire — and the two call for opposite responses. A non-zero count points at the capture path (loss or heavy reordering), not at a limit to raise: fragments are released as soon as both handshake messages parse, so a healthy flow holds single digits. Compare it against the kernel drop counters below.

DTLS handshake fragments are counted separately, because they are a different mechanism with a different cause and the two are not comparable:

DTLS handshake fragments refused (reassembly limit): 0

A DTLS handshake message too large for one datagram is fragmented by DTLS itself, and those fragments are held per connection under fixed limits — see DTLS handshake fragments. A non-zero count means a connection offered more fragments, more concurrent messages, or a larger message than those limits allow, so its handshake cannot be assembled and the connection is recorded without crypto detail.

Before 2.0.1 this state was shared between connections

The DTLS fragment buffer used to live in a parser that every flow shared, keyed on the DTLS message sequence number alone. That number is 0 for the first ClientHello of every handshake, so two peers fragmenting a ClientHello at the same moment collided: where their declared message lengths matched — which they do whenever both run the same TLS stack — a hello could be assembled out of fragments from both, and the record described a handshake that never happened. Nothing bounded how long fragments were held, how many, or how large, and a completed message was never released. If you have DTLS records from an earlier release whose crypto detail looks implausible for the peer involved, this is a candidate cause.

It is also reported between heartbeats rather than only at shutdown, so a long-running sensor surfaces it while there is still something to do about it:

Heartbeat output loss - Records discarded by the offline output queue: 1284

Compare all of these against the packet counters in the same block — if drops are zero, the loss is not happening in capture and the handshakes genuinely were incomplete on the wire.

"Async NDJSON batch write took ...ms" Warnings

Async NDJSON batch write took 143.20ms for 512 records (file size: 18MB, warning threshold 100ms)

The writer thread timed one batch write to the NDJSON file and it exceeded offline_output.slow_write_warning_ms. The warning is informational: writes are not skipped, delayed, or retried because of it, and no records are lost by it.

Whether it means anything depends on the storage. The default threshold of 100 ms is a real anomaly on an NVMe volume and unremarkable on a spinning disk, a contended SAN LUN, or an NFS mount — which is why it is configurable rather than fixed. During load and stress testing on slower storage it can repeat often enough to bury the lines you were actually reading.

• If the storage genuinely is slow and this is expected, raise offline_output.slow_write_warning_ms to match what it does, or set it to 0 to turn the warning off. See Slow-write warnings.
• If the storage should be fast, check whether the slow writes are also costing you records: a non-zero Records discarded by output queue in the same run means the writer fell far enough behind that the queue overflowed. See Fewer Records Than Connections.

Silencing the warning does not change the write-time statistics — every batch is timed and accumulated regardless of the threshold — so the average write time in the final statistics stays available as a check.

Queue Drops and Packet Rate

Queue drops counts packets the capture thread could not hand to the processing thread because the queue between them was full. Two different situations produce it, and they call for different responses.

Sustained drops mean the processing thread cannot keep up with the offered rate. A larger queue does not help: it fills more slowly and then drops just the same. What helps is giving it less to do — narrow capture.bpf_filter to the traffic you actually need, and make sure capture.immediate_mode is off, since per-packet delivery costs a large fraction of capture throughput.

Bursty drops, where the counter jumps and then sits still, mean a spike outran the queue's depth. capture.queue_size is the right lever there — it applies per shard, so raising it raises every worker's headroom.

Adaptive thread scaling

When capture.processing_threads is auto (the default), the sniffer adjusts the number of processing workers while it runs, once per second, from the drop counters themselves.

The opt-out is per stage. Writing an explicit number for one setting does not switch off the other: capture.fanout_threads pinned to a value the NIC and driver were tuned around says nothing about whether the parsing side should follow the traffic. Each stage logs its own state at startup:

Adaptive scaling active for processing: up to 4 worker(s) within 8 effective CPUs
(2 reserved, at most 2 worker(s) per capture thread)
Adaptive scaling disabled for capture: capture.fanout_threads is set explicitly to 2,
which is honoured as written

A pinned stage is still watched, and reports what it would have done without doing it — rate-limited to once per cooldown, so a sustained condition does not flood the log:

capture.processing_threads=1 is set explicitly; 2 worker(s) would be used otherwise
(shard queue drops: 438602/s)

That line is the one to look for when a deployment with hand-tuned thread counts starts losing packets: it names the setting, the value, and the evidence, without overriding a number somebody chose deliberately.

Each signal is answered differently, because each one means a different stage is behind:

What rose	What it means	What happens
Kernel drops	Capture could not drain the kernel ring buffer	A capture thread is added to the live fanout group, within the CPU budget. See the note below for when it cannot be.
Shard queue drops	Capture kept up; parsing and tracking did not	Workers are added, one or two at a time
Output queue drops	Parsing kept up; output did not	Nothing is scaled. Adding parsers would pile work onto a stage already waiting on the one behind it
Shard queue drops, but workers are at the pairing limit	Parsing is behind and cannot be given more workers without starving capture	Nothing is scaled. Reported with the setting that lifts the limit, once per cooldown
Nothing, for a long time	Excess capacity	One worker is given back

Scaling events are logged as they happen:

Scaling processing workers: 2 -> 4 (shard queue drops: 4218/s)

Three deliberate limits:

• Nothing reacts to a single bad second. Three consecutive lossy samples are needed before scaling up, there is a cooldown afterwards so the effect of a change can be observed before another is made, and scaling down needs five minutes of quiet. Drops are bursty; a controller without hysteresis would add and remove threads continuously.

• The CPU budget is never exceeded, and neither is the pairing. capture threads + workers + reserve stays within the CPUs the process may actually use — affinity and cgroup quota, not the host core count. Separately, and more importantly, the worker ceiling is two workers per capture thread.

  That second limit is not a tidy-up; without it this feature makes things worse. The controller's response to shard queue drops is to add a worker, and at 2,000,000 packets/sec offered to a single capture thread, 2 workers captured 99.6% with 112,119 queue drops while 3 workers captured 55.4% with 26,781,341 kernel drops. The queue drops at 2 workers are a real signal on an otherwise healthy capture, and answering them by adding a third worker would have taken that capture down by nearly half. Past the pairing, extra workers stop adding capacity and start taking the capture thread's — the giveaway in the measurements is that user CPU falls while system CPU rises six-fold.

  So a capture that is dropping at its pairing limit is telling you it needs another capture thread, not another worker, and that is a restart (see the note below). - A single hot flow is reported, not answered with workers. A flow belongs to exactly one shard by definition, so no number of workers can spread it:

Hot flow detected (shard queue drops: 2000/s, but only one shard is deep (4096 queued)
while the rest are idle. A single flow belongs to one shard and cannot be spread; more
workers will not help.)

Capture threads grow, but never shrink

Since 2.0.1 a sustained kernel-drop condition adds a member to the live PACKET_FANOUT group, within the CPU budget:

Scaling capture fanout: 1 -> 2 (kernel drops: 710/s)

New members join the same group, so the kernel starts spreading across them immediately, and no flow changes shard as a result — the dispatcher, not the kernel's hash, decides which shard owns a flow.

Shrinking waits for a restart. Removing a member means closing its socket, and closing an AF_PACKET socket discards whatever is still in its ring buffer — packets the kernel has already accepted and counted, lost silently. That is the same asymmetry the worker pool has: quick to add capacity, slow to give it back.

Growing can still be refused — a savefile input has no group, the kernel may decline another member, or the group may be at FANOUT_THREADS_MAX. It says so rather than doing nothing:

Capture is losing packets in the kernel (kernel drops: 900/s) and the fanout group
could not be grown past 2 thread(s); set capture.fanout_threads=3 and restart

When there is no spare CPU at all, the controller's preferred answer is a trade — give up one processing worker, spend the CPU it frees on capture. Only half of that can be done while running, and applying that half alone would leave less parsing capacity and no more capture capacity, so neither half is applied and both are reported:

Capture is losing packets in the kernel and there is no spare CPU (kernel drops: 900/s,
no spare CPU: trading one processing worker for a capture thread). Neither half of this
can be done while running: set capture.fanout_threads=2 and capture.processing_threads=1
together and restart.

fanout_threads: 1 now behaves like every other value

Before 2.0.1 a single capture thread ran on a plain AF_PACKET socket with no fanout group, and therefore without PACKET_FANOUT_FLAG_DEFRAG. That flag asks the kernel to reassemble IP fragments before delivery, so the sniffer saw different packets depending on a thread count: fanout_threads: 1 delivered raw fragments, 2 delivered reassembled datagrams. A fragmented flow could be analysed on one deployment and missed on another, with no setting to explain the difference.

A one-member group is now always created, so what a packet looks like is a property of the interface and the filter rather than of the thread count. Deployments running fanout_threads: 1 on fragmented traffic will see fragments reassembled where they previously did not, which generally means more flows analysed, not fewer.

Datagram lengths and fragments

Datagram lengths are now read from the headers, not from the captured frame

Before 2.0.1 the parsers treated the captured frame length as the authority on how much data a datagram carried, and never read the IPv4 Total Length, the IPv6 Payload Length or the UDP length. Two effects were visible in the output:

• Padding was reported as payload. A frame below the 60-byte Ethernet minimum is padded by the sending NIC. A one-byte UDP datagram therefore presented 18 bytes of payload to the protocol detectors, 17 of them padding.
• Fragments after the first produced flows that never existed. Such a fragment carries no transport header, but the eight bytes following the IP header were read as one, which invented a source and destination port out of payload bytes.

Both now follow the headers. Expect two changes on upgrade: payload_length on records for small datagrams is smaller, and a non-initial fragment is refused and counted under Malformed packets instead of becoming a record. In both cases the new figure is the correct one. A first fragment does carry a transport header and is still analysed.

IPv6 extension headers are now walked as well, so a packet carrying a Hop-by-Hop, Routing or Destination Options header reaches its transport header rather than being discarded as unidentified. Those packets were previously counted under Malformed packets too, so that counter may fall on IPv6 traffic while the number of records rises.

Live capture sees little of the fragment change, because the kernel reassembles fragments before delivery (above). Offline PCAP replay delivers whatever the file holds, so that is where it shows.

One case moved the other way. A UDP datagram declaring more than was captured used to be refused outright, so a capture.snaplen low enough to truncate removed every QUIC, DTLS and WireGuard record while TCP flows, which declare no length, still produced short ones. A truncated datagram is now kept with its payload bounded by the capture, so Malformed packets falls and records appear where there were none. A UDP length that does not fit a capture that was not truncated still means the header is wrong, and is still refused. See Lowering snaplen.

Malformed-packet warnings are summarised, not repeated

A frame that is invalid or shorter than its link layer allows is refused before anything identifies it. Each one is counted under Malformed packets, and a warning naming the condition is written at most once a minute:

Frame refused before parsing: valid=true, length=4, and this capture's link layer needs at
least 14 bytes. Further refusals are summarised at most once a minute; the running total is
reported as "Malformed packets".

If refusals continue, the next line a minute later says how many it stands for:

Frame refused before parsing: valid=true, length=4, and this capture's link layer needs at
least 14 bytes. A further 418,233 were refused since the previous line of this kind.

Nothing is lost by the rationing. The condition is counted every time, the heartbeat reports the running total, and the end-of-run statistics report both that total and how many warnings were withheld:

Malformed packets: 418234
Malformed packet warnings suppressed by rate limiting: 418233

So a single warning next to a malformed count in the hundreds of thousands is expected, and the second line is what connects them. A non-zero suppressed count is not itself a problem — it means the condition recurred, which the first line already said.

Before 2.0.1 this was a warning per frame

The line was emitted for every refused frame, and prefixed [PARSER_DEBUG] — debug text at warning severity. A flood of short frames therefore became a log-volume failure in its own right: the sniffer survived the traffic the parser rejected safely, and then fell over on its own logging. Worse, through the database-backed LOG_WARN each line took a process-wide lock, so under a flood every worker thread serialised on it. If you have logs from an earlier release that are mostly this one line, that is what you are looking at, and the Malformed packets total in the same log is the number that matters.

What a disabled log level costs

Nothing worth measuring, as of 2.0.1 — which matters when deciding whether to leave logging.level at info for performance reasons. You do not need to. See logging.

Before 2.0.1 a log line was partly paid for even when the level discarded it: the arguments were evaluated before the level was consulted, so a helper call or a string built for a debug line ran and was thrown away, and every call copied a reference-counted pointer shared by all worker threads. The level is tested first now, so a suppressed line is a single atomic read.

Raising the level to debug on a busy sensor still costs what writing those records costs — that has not changed, and logging.file.rate_limit_per_sec is the control for it.

Flow shards

Drops with a shallow average but a deep Deepest shard queue mean the load is uneven rather than excessive. Either a few heavy flows dominate — which per-flow sharding cannot split — or the traffic mix is concentrating on one shard. Check Shards ready alongside it: if that is near zero while one queue is deep, it is a hot flow and no number of workers will change it.

Packets are routed to one of 64 fixed virtual shards by a hash of their flow, so both directions of a connection always reach the same shard. A shard owns its connection tracker and duplicate filter for the life of the process; a worker borrows a shard, drains a bounded batch of it, and hands it back. That is why the worker count can change while the sniffer runs without moving a single flow — the shard count is what routing hashes against, and it never changes.

Three consequences are worth knowing when reading drop counters:

• A single connection cannot be spread. It belongs to one shard by definition. Sharding raises the aggregate packet rate and connections per second, not the ceiling for one very busy connection.
• Adding workers does not relieve capture. Capture is a separate thread that reads from the kernel and routes. If it is the thing falling behind, workers sit idle waiting for packets the kernel already discarded. See the next section for how to tell the two apart.
• Workers and Shards are different numbers. Shards is always 64. Workers is how many threads are currently draining them, and it moves on its own when capture.processing_threads is auto.

"Connection table full" reports a process-wide limit

Connection table full: refusing new flow. The process-wide ceiling of 100000 flows
(connection_tracker.max_connections) was reached; 100000 are tracked right now, 1557 of
them on this shard. 1 flow(s) refused process-wide so far. Raise the ceiling or shorten
idle timeouts -- sustained refusal means flows are arriving faster than they expire.

Every figure is labelled with the scale it belongs to, and only one of them is a limit. connection_tracker.max_connections is a single budget shared by all 64 shards, so the ceiling is the number you configured — not that number divided by the shard count.

The shard figure is local context, and it is roughly a sixty-fourth of the total when traffic spreads evenly. It is worth reading for one thing: a shard holding far more than a sixty-fourth means the load is concentrated, which is hash skew or a few flows dominating — the same condition Flow shards describes for queue depth. It is not the limit, and raising anything on the strength of it will not help.

The warning is emitted once for the first process-wide refusal and then every ten thousandth, by whichever shard happens to observe that refusal. One saturation event therefore produces one line, not one per shard.

Kernel Drops: Capture Cannot Keep Up

Queue drops and Dropped describe failures at two different stages, and confusing them sends you after the wrong setting.

Counter	Where the packet was lost	What helps
Queue drops	Inside the sniffer, between capture and the workers. Capture read the packet; no worker had room for it.	Narrow capture.bpf_filter, raise capture.queue_size for bursts, check capture.immediate_mode is off
Dropped (main heartbeat line)	Inside the kernel, before the sniffer ever saw it. The capture thread did not read from the ring buffer fast enough and the kernel overwrote packets.	capture.fanout_threads

Dropped growing with Queue drops at zero is the signature of a capture bottleneck. The workers are keeping up with everything they are given; the problem is how much is reaching them. Raising processing_threads will not change it — the packets are discarded before any worker is involved.

First check capture.processing_threads. Workers waiting for packets compete for CPU with the single capture thread that feeds them, and enough of them will starve it: four workers were measured losing 31,880,873 packets to kernel drops at a load two workers captured completely. If lowering it clears the drops, that was the cause.

If the worker count is already low, raise capture.fanout_threads. It opens several capture sockets in a Linux PACKET_FANOUT group so the kernel spreads arriving packets across several reader threads, and it also removes the starvation above without giving up the workers.

Confirm the group actually formed — the kernel can refuse it, and the sniffer then carries on with one thread rather than failing to start:

Capture fanout: 2 thread(s) sharing group 27183

If Dropped keeps climbing with several capture threads, the link genuinely carries more than the host can inspect. Narrow capture.bpf_filter so less of it has to be.

Reading the Heartbeat CPU Figure

Every heartbeat reports what the sniffer process is consuming:

Heartbeat - Uptime: 300s, Packets: 18422119, Connections: 48213, Writes: 48213,
Dropped: 0, Memory: 412.18 MB, CPU: 107.42% (54% of 2-CPU quota), Buffered: 0,
Parse Errors: 0, Malformed: 0

CPU is percent of one core, not percent of the host. 100% is one core fully consumed and 200% is two full cores, so a sniffer with two capture threads and four workers on an eight-core host can legitimately read several hundred percent. That is the same unit top, ps and pidstat report for a process, so the figures are directly comparable. It is deliberately not capped at 100: capping would hide the one thing the number exists to show.

The clause in brackets appears only when a cgroup CPU quota applies, and it is the share of that allowance rather than of the host's cores. It is the figure to read when the question is "does this sensor need more CPU" — 54% of a 2-CPU quota means half the allowance is still there, whatever the host happens to have. A container, a systemd unit with CPUQuota=, or a Kubernetes CPU limit each produce one; a plain host process does not, and the line then reads CPU: 107.42%, exactly as it always has. See What counts as a CPU for how the quota is found.

The quota is read once, at startup. A quota changed under a running process — systemctl set-property, docker update — is not reflected until the sniffer restarts.

The figure describes the most recent two-second window, not the whole heartbeat interval. It answers "what is this process doing now": it is sampled on its own fixed interval and cached, so the heartbeat prints a spot reading taken near it rather than an average of the minute before. A load that spikes and subsides between heartbeats will not appear in it. The drop counters are where that shows up, because they accumulate.

Before 2.0.1 this number was wrong, not merely coarse

Until 2.0.1 the CPU percentage was computed inside the memory monitor, and several monitoring paths each advanced the window it was measured over. Two of them — the memory pressure check and the heartbeat itself — ran on the same tick of the main loop, so every heartbeat differenced a few milliseconds of a counter that only moved in 10 ms steps. A sniffer steadily consuming one core reported CPU: 188.68%. Readings from earlier builds are not comparable with these and should not be quoted; take them from pidstat, top or /proc/<pid>/stat instead.

Reading the Capture-Quality Counters

Every heartbeat reports whether the capture is seeing complete flows, and how many flows are being held in memory:

Heartbeat capture quality - Flows tracked: 48213, Queue drops: 0, Interface drops: 0,
Mid-flow adoptions: 12, Role corrections: 3, TLS direction mismatches: 0,
Connections dropped before output: 0, New flows refused (table full): 0,
New flows refused (memory limit): 0, Memory RSS: 412 MB,
Memory admission: open (not enforced), Capture threads: 2, Workers: 4,
Shards: 64, Shards ready: 3, Deepest shard queue: 18, Busiest shard: 0.9% full,
Largest shard table: 12104, Unrouted packets: 0,
Kernel drops/s: 0.0, Queue drops/s: 0.0, Output drops/s: 0.0

Most of these are cumulative totals, so what matters is how fast they grow between heartbeats. The three /s figures do that arithmetic for you — they are the change since the previous heartbeat divided by the measured interval, so they answer "is this happening now" rather than "did this ever happen". They read 0.0 on the first heartbeat, because there is no previous sample to difference against.

Counter	Meaning
Flows tracked	Connections currently held in memory, whether or not they are ever written. This is the number to watch for connection-table growth. Distinct from Connections on the main heartbeat line, which counts records written.
Queue drops	The capture thread outran packet processing. This is the sniffer's own queue, not the kernel's. Check whether capture.immediate_mode is on before anything else; then narrow the BPF filter so less traffic is parsed. Raising capture.queue_size absorbs bursts but does not raise sustained throughput.
Interface drops	The interface's own discard counter (ps_ifdrop), which many Linux drivers never populate — it is frequently zero even when packets are being lost. The counter that matters for a capture falling behind is Dropped on the main heartbeat line, which is the kernel's ring-buffer overflow count.
Mid-flow adoptions	Flows first seen without a TCP handshake, because capture started late or their opening packets were lost. They are still tracked and analysed, but their client/server roles begin as a guess.
Role corrections	Flows whose client/server assignment was corrected once better evidence arrived — a SYN, or a ServerHello that could only have come from the server.
TLS direction mismatches	Connections where a ServerHello arrived on the client-to-server path, meaning the roles were reversed. Counted once per connection.
New flows refused (table full)	The connection table hit connection_tracker.max_connections, so new flows were not tracked. Sustained refusals mean flows are arriving faster than they expire — shorten the timeouts or raise the ceiling.
New flows refused (memory limit)	Process RSS reached resource_limits.max_memory_mb while enforce_connection_memory_limit was on, so new flows were not tracked. Existing flows continued to be processed. Admission resumes on its own once RSS falls to resume_memory_mb; a count that keeps climbing means the limit is below what the sniffer needs for this traffic. Zero unless enforcement is enabled. On 2.0.0-5 and 2.0.0-6 this always read zero — each worker counted its refusals correctly, but the figure was not summed into the process-wide total (TQR-1421). On those versions read the Memory admission closed at ... refused N new flow(s) warnings in the log instead.
Memory RSS	Total resident memory for the sniffer process — every allocation, not just tracked flows. This is the figure the memory limit is compared against.
Memory admission	Whether new flows are currently being accepted on memory grounds: open, closed, or open (not enforced) when enforce_connection_memory_limit is off. A gate that reads closed for long stretches means the threshold is too close to the sniffer's working set.
Connections dropped before output	See the section above.
Capture threads	How many threads are reading from the kernel — capture.fanout_threads, resolved. Fixed for the life of the process. It is on this line because it is the number Workers has to be read against: the measured safe pairing is at most two workers per capture thread.
Workers	How many threads are parsing and tracking right now. Since 2.0.1 this can change while the sniffer runs — see Adaptive thread scaling — so it will not always equal capture.processing_threads.
Shards	Virtual flow shards — always 64. A flow always hashes to the same shard, so shard state never moves when the worker count changes.
Shards ready	Shards holding packets that no worker has picked up yet. Read together with the next row: many shards ready means too few workers; one deep queue with nothing ready means a single hot flow, which more workers cannot help.
Deepest shard queue	Packets held by the fullest shard, not the total. Compare against Queue drops: drops with a deep queue mean genuine overload; drops with a shallow one mean traffic is landing unevenly across shards.
Busiest shard	Deepest shard queue as a percentage of one shard's queue capacity, which is the bound that decides when drops start. A depth on its own cannot say whether a shard is idling or about to overflow. At 100% that shard is dropping.
Largest worker table	Flows held by the fullest worker's connection table. The same distinction as above, for memory rather than queue depth.
Kernel drops/s	Packets the kernel discarded before the sniffer saw them, per second. The most severe of the three: nothing downstream can recover them. Answered by a capture thread, not by workers.
Queue drops/s	Packets dropped from the sniffer's own shard queues, per second. Capture kept up and parsing did not.
Output drops/s	Records discarded by the output queue, per second. Parsing kept up and output did not — adding workers makes this worse, not better.
Unrouted packets	Packets with no usable flow key — malformed, truncated, or a protocol without ports. They are spread across workers rather than piled on one, but connection state for them may be split. A large number means routing is degraded for that traffic.

Adoptions, corrections and mismatches all mean the same underlying thing: the start of some flows was not captured. They should track the drop counters. A rising mismatch count with zero drops anywhere is worth investigating — it means flows are being seen out of order without any packet loss to explain it.

Checking the Memory-Refusal Count Against Itself

Flows refused on memory grounds are counted twice, independently: once by the tracker that turned the flow away, and once by the admission gate itself. The closing summary reports both, so the figure can be checked rather than taken on trust:

New flows refused (memory limit): 565808
Memory refusals counted by admission gate: 565808

The gate's figure is sampled first, and both counters only ever increase, so the tracker sum that follows can never legitimately be the smaller of the two. If it is, the sniffer logs an error naming both:

[error] Internal accounting inconsistency: the per-tracker sum of memory refusals (0)
is below the admission gate's own count (565808), which was sampled earlier. Both
counters only increase and measure the same refusals, so the sum cannot legitimately be
the smaller of the two: a statistic is being lost between the workers and this report.
Treat the gate's figure as the reliable one. Flows refused on memory grounds produce no
output record, so this run's report covers less traffic than it states.

The check is deliberately one-sided. On a live capture the two cannot be read at the same instant, so during a heartbeat the tracker sum will sometimes sit slightly ahead of the gate's — flows refused between the two reads. That is normal and is not reported. Only a shortfall is a fault.

This is not an operational condition — no capture setting causes it and none will clear it. It means the reporting path has lost a worker's contribution, so treat the run's coverage as unknown and raise it: take the larger of the two numbers as the count of flows that were never recorded, and keep the log, which is the only place the figure survives. The check runs on every heartbeat as well, so a long capture reports the fault when it starts rather than at shutdown.

Memory Growth Under Connection Churn

If Flows tracked climbs steadily and memory grows with it, the sniffer is retaining flows that have finished. The usual cause is short-lived UDP: a client that opens a socket, sends one datagram and closes it produces a new source port — and therefore a new tracked flow — every time.

Check the heartbeat for the shape of the problem:

• Flows tracked rising without falling → flows are outliving their usefulness. Lower connection_tracker.udp_unanswered_timeout_seconds, which governs UDP flows that never received a reply.
• New flows refused (table full) above zero → the table has reached connection_tracker.max_connections. New flows are being dropped rather than tracked. Shorten the timeouts first; raising the ceiling alone trades memory for the same problem.
• New flows refused (memory limit) above zero → process RSS reached resource_limits.max_memory_mb. This is the memory gate doing its job, and it resumes by itself once RSS falls to resume_memory_mb. Treat a steadily rising count the same way as the one above — shorten the timeouts before raising the threshold. See Limiting the connection table by memory. On 2.0.0-5 and 2.0.0-6 this counter always read zero, so it cannot be used to rule the problem out on those versions — see the note in the table above.
• Queue drops rising alongside → packets are being lost between capture and processing. Narrow the BPF filter so the sniffer is not parsing traffic you do not need, and confirm capture.immediate_mode is off. Raising capture.queue_size only buys a deeper buffer.

A warning is logged when roles cannot be resolved at all:

3 connection(s) were still open at shutdown with UNRESOLVED client/server roles: no TCP
handshake was captured and no ServerHello identified the direction...

Those flows are not written, because with the direction unknown their endpoints cannot be trusted — the source and destination could be the wrong way round. Capturing from before the flows start resolves it.