Collecting Sniffer Output with BigFix (Agentless / No Elastic Path)

Some network segments cannot reach the Elastic (or Splunk) ingest tier — a co-located forwarder has no route out of the enclave. On those segments, run the sniffer in sealed-file mode and let BigFix ferry the completed files off the host over its native relay chain, exactly as it already ships scan-result files. This removes the per-segment forwarder-to-Elastic path; only the BigFix relay root needs one path to the ingest tier.

This is the collection recipe. It pairs with the offline_output.rotation_mode: sealed configuration documented in configuration.md.

Why sealed mode is required

The default circular rotation reuses fixed filenames (connections.N.ndjson) and truncates them on wrap. A polling collector can lose the unread tail of a file that gets overwritten between polls, and reused filenames cause duplicate re-ingestion. Sealed mode writes to a single active file and, on rollover, atomically renames it to a unique, immutable file that is never reopened or reused — safe to grab and delete.

Sniffer configuration

offline_output:
  output_file: "/var/log/pqc-sniffer/connections.ndjson"
  rotation_enabled: true
  rotation_mode: "sealed"
  max_size_mb: 100          # size trigger per sealed file
  max_files: 10             # retention cap if the collector stalls (oldest pruned, warned)
  seal_interval_seconds: 300  # also seal every 5 min so quiet/lab segments hand off promptly
  seal_gzip: true           # gzip each sealed file (~10:1) to cut relay/transfer load

Resulting files in /var/log/pqc-sniffer/:

File	Meaning	Collect?
connections.ndjson.active	The live file being written	No
connections-YYYYMMDD-HHMMSS-<seq>.ndjson	Sealed (uncompressed)	Yes
connections-YYYYMMDD-HHMMSS-<seq>.ndjson.gz	Sealed (gzipped, when seal_gzip: true)	Yes
.connections-*.raw, .connections-*.gz.part	Dot-prefixed compression intermediates	No

Switching a host that already ran in circular mode

Nothing to do by hand. A host that previously wrote connections.N.ndjson (or connections.ndjson, with rotation off) still holds connection records under names the collection glob below does not match. On the first start in sealed mode the sniffer seals those files into the connections-*.ndjson* naming — gzipped when seal_gzip: true — so the next collection cycle ships them with everything else. Their sealed names carry the time each file was last written, not the restart time.

/var/log/pqc-sniffer/pqc-sniffer.log records each file as it is sealed, followed by one Sealed N NDJSON file(s) written under circular rotation warning. If some could not be sealed, a warning says so; those files keep their old names and are retried on the next start.

The sealed leftovers count towards max_files. Keep max_files at least as high as it was in circular mode for the first start, or the oldest of them are pruned before a collection cycle can take them.

The collection loop (BigFix task/property, run on a schedule)

1. Match the sealed glob connections-*.ndjson* in the output directory. This covers both .ndjson and .ndjson.gz, and never matches the .active file or the dot-prefixed intermediates. Do not collect connections.ndjson.active — it is still being written.
2. Ship each matched file upstream over the BigFix relay chain to the collection point at (or near) the relay root.
3. Delete each file only after its upload is confirmed. Because sealed filenames are unique and never reused, a failed upload is safe to retry, and re-shipping a file that was already delivered is harmless to dedup on event.id (see below).

Pseudo-flow:

for f in connections-*.ndjson*:        # excludes .active and dot-intermediates
    if upload(f) == success:
        rm f
    # else: leave it; the retention cap bounds the backlog, retry next cycle

Cadence — poll faster than the seal rate

The exposure window is how fast sealed files accumulate to the retention cap (max_files). Size the BigFix poll interval below the time it takes to fill and prune max_files × max_size_mb. On a busy segment that can be minutes:

Segment load (completed conns/s)	Data rate (~3.5 KB/record)	Time to fill max_files × max_size_mb (10 × 100 MB)
~10/s (small)	~35 KB/s	~8 hours
~100/s (moderate)	~350 KB/s	~50 minutes
~1,000/s (busy)	~3.5 MB/s	~5 minutes
~5,000/s (max, single sniffer)	~17.5 MB/s	~1.5 minutes

If the collector falls behind the cap, the sniffer prunes the oldest sealed files first (whole files, never a partial record) and logs a warning — so a rising prune-warning count is the signal that the poll interval or link is too slow for that segment. seal_gzip: true cuts the transferred volume ~10:1; seal_interval_seconds ensures even idle segments hand off on a predictable cadence.

Last mile — one ingest path

At the BigFix collection point, land the files and ingest them into Elastic once, centrally (Elastic Agent filestream, Logstash, or the Elastic ingest pipeline). Gunzip .ndjson.gz on the way in if your ingest does not do it natively. This is the whole win: one managed path from the relay root to Elastic instead of a path per segment.

Idempotency and dedup

Sealed filenames are unique and never reused, so at-least-once delivery (a retry after an unconfirmed upload) is safe. If the same file is delivered twice, deduplicate downstream on event.id — the connection key — as already documented for the Sniffer dashboards. Do not rely on filename or offset tracking for dedup.

Latency caveat

BigFix collection is store-and-forward on a polling cadence (minutes or more). It suits the posture / inventory / compliance views (30-day windows). It is not near-real-time, so the Active Threat "live downgrade feed" (60-second refresh) is not served over a BigFix-ferried segment — that dashboard's expectation assumes streaming ingest. State this explicitly for any segment collected this way.