Splunk Universal Forwarder Integration¶

Manually configure a Splunk Universal Forwarder (UF) to monitor and forward the sniffer's offline NDJSON output to a Splunk indexer or heavy forwarder.

Note

This page covers manual .conf file setup only — it does not cover the Splunk Deployment Server / deployment apps workflow.

Info

A Universal Forwarder does not do search-time parsing — for most field extraction it forwards events to an indexer or heavy forwarder. JSON structured-data extraction is the exception: with INDEXED_EXTRACTIONS = JSON, fields are extracted at input time on whichever host reads the file, and with a UF in the path that host is the UF. So the structured-data props.conf (with INDEXED_EXTRACTIONS, line breaking, and timestamp fields) belongs on the UF, while the indexer gets a small props.conf that turns off search-time re-extraction. Both tiers get a props.conf for pqc:connections, with different contents — see Step 4. inputs.conf and outputs.conf remain UF-only.

Two Hosts, Two Roles¶

Every step below is explicitly labeled UF host or indexer/heavy forwarder host. Confusing the two is the single most common way this integration fails to send data — configuration that only exists on the wrong tier loads silently and produces no error, just no data.

Host	Role	Description	Default install path	Config files that belong here
Universal Forwarder	Collector / forwarder	Runs on the sniffer host and reads the NDJSON files, applies indexed JSON extraction, and forwards events onward.	`/opt/splunkforwarder`	`inputs.conf`, `outputs.conf`, `props.conf` (structured JSON extraction — Step 4.b)
Indexer / heavy forwarder	Parser / storage tier	Receives forwarded events, stores the data, and holds the index-side parsing and index definitions.	`/opt/splunk`	`props.conf` (disable search-time re-extraction — Step 4.a), `indexes.conf`

Prerequisites¶

Splunk Universal Forwarder installed on the host running the sniffer. Confirm the install path and that the service is running before continuing:

sudo /opt/splunkforwarder/bin/splunk status

Set SPLUNK_HOME for your shell session so the commands on this page can be copy-pasted as-is. The commands below assume this variable is set — if it is empty, paths silently resolve relative to / (for example $SPLUNK_HOME/etc/apps/... becomes /etc/apps/...), files get created outside the Splunk installation, and Splunk never loads them.

export SPLUNK_HOME=/opt/splunkforwarder
echo "$SPLUNK_HOME"

Set this in every new shell

export SPLUNK_HOME=... only applies to the current shell session. If you open a new terminal, SSH in again, or switch users (including via sudo -i), re-export it — or use the full literal path (/opt/splunkforwarder/...) instead of $SPLUNK_HOME throughout if you'd rather not rely on the variable at all.

Network reachability from the UF to the indexer's receiving port. Reachability alone is not sufficient — the indexer must also be actively listening on that port, which is configured separately in Step 3. A firewall that allows port 9997 will still refuse the connection if Splunk itself has not been configured to receive on it.
The sniffer already validated and producing output. Check for the file with a glob, not an exact filename — depending on whether rotation is enabled, the active file may be connections.ndjson or an indexed file such as connections.0.ndjson:

ls -lh /var/log/pqc-sniffer/connections*.ndjson
tail -f /var/log/pqc-sniffer/connections*.ndjson

If no file matches, resolve that with Troubleshooting before continuing — do not assume output is missing just because connections.ndjson specifically doesn't exist.

Step 1: Create a Dedicated App (UF Host)¶

Use a custom app directory instead of editing etc/system/local/ directly, so the configuration is easy to manage and redeploy.

sudo mkdir -p "$SPLUNK_HOME/etc/apps/tychon_pqc_sniffer/local"

Set ownership to match the account the Universal Forwarder runs as (commonly splunkfwd; confirm with ps -ef | grep splunkd if unsure):

sudo chown -R splunkfwd:splunkfwd "$SPLUNK_HOME/etc/apps/tychon_pqc_sniffer"

Step 2: Configure the Monitor Input (UF Host)¶

Write inputs.conf directly with a heredoc rather than opening an editor and typing the path in by hand — this avoids the file landing in the wrong directory or the path getting entered as a shell command by mistake:

sudo tee "$SPLUNK_HOME/etc/apps/tychon_pqc_sniffer/local/inputs.conf" > /dev/null <<'EOF'
[monitor:///var/log/pqc-sniffer/connections*.ndjson]
disabled = false
index = pqc_network
sourcetype = pqc:connections
crcSalt = <SOURCE>
EOF

The glob connections*.ndjson matches both the unrotated connections.ndjson file and rotated connections.0.ndjson, connections.1.ndjson, etc. See Rotation and Retention.
index = pqc_network assumes a dedicated index has been created on the indexer tier (Step 5). If you use a different index name, see Keeping the Index Name Consistent below.
crcSalt = <SOURCE> keys Splunk's read-tracking off the full file path rather than the first bytes of file content, which matters because rotated files are reused by name.

Confirm the UF's service account can actually read the file. A correctly loaded monitor stanza still sends no data if the account running the forwarder lacks read access to the directory or file:

sudo -u splunkfwd head -n 1 /var/log/pqc-sniffer/connections*.ndjson

The connections*.ndjson glob covers every case: the unrotated connections.ndjson, and any rotated files such as connections.0.ndjson or connections.6.ndjson — including instances where the file has already rotated. When more than one file matches, head prints each with a ==> filename <== header, so you can confirm the account can read all of them.

If this fails with Permission denied, fix directory/file permissions before moving on — a monitor stanza pointed at a file the UF can't read fails silently from the operator's point of view.

Confirm Splunk actually loaded this configuration before moving to Step 3. This is the single highest-value check on this page — it directly tells you whether the file landed in the right place and was picked up, instead of waiting to find out later that no data arrived:

"$SPLUNK_HOME/bin/splunk" btool inputs list --debug | grep -A8 -B2 '/var/log/pqc-sniffer'

You should see your four settings (crcSalt, disabled, index, sourcetype) attributed to the file you just created, e.g. /opt/splunkforwarder/etc/apps/tychon_pqc_sniffer/local/inputs.conf. If btool shows nothing for this path, or attributes the values to a different file, the stanza was not loaded — re-check $SPLUNK_HOME and the directory it actually wrote to.

Step 3: Configure Forwarding to the Indexer (UF Host)¶

Enable the receiving port first (indexer/heavy forwarder host)¶

The UF cannot successfully forward until the indexer is actively listening on the receiving port — reachability (firewall) is necessary but not sufficient. On the indexer, in Splunk Web:

Settings → Forwarding and receiving → Configure receiving → New Receiving Port → 9997

Then confirm the indexer is actually listening:

ss -lntp | grep ':9997'

Don't confuse Splunk Web (8000) with the receiving port (9997)

If you only know the Splunk Web URL (for example http://<indexer>:8000), do not use port 8000 in outputs.conf below. Splunk Enterprise commonly uses three separate ports: 8000 for Splunk Web, 8089 for the management/REST API, and 9997 for forwarder-to-indexer data. outputs.conf always targets 9997 (or whatever port you configured above) — never 8000.

From the UF host, confirm the port is reachable before configuring outputs.conf:

nc -zv <indexer-address> 9997

Configure outputs.conf (UF host)¶

sudo tee "$SPLUNK_HOME/etc/apps/tychon_pqc_sniffer/local/outputs.conf" > /dev/null <<'EOF'
[tcpout]
defaultGroup = pqc_indexers

[tcpout:pqc_indexers]
server = <indexer-address>:9997
useACK = true
EOF

Replace <indexer-address> with your actual indexer or heavy forwarder address — double-check it carefully. A single mistyped octet here (for example the wrong third octet on a /24) produces a No route to host error in splunkd.log that looks like a network problem but is actually a configuration typo.

Tip

To encrypt forwarder-to-indexer traffic, add TLS settings under [tcpout:pqc_indexers] (clientCert, sslPassword, sslVerifyServerCert, etc.) per your organization's certificate setup.

Verify forwarding immediately, before moving on¶

Don't wait until the end-to-end search in Step 7 to discover a forwarding problem. After restarting the UF (Step 6), check the active destination directly:

"$SPLUNK_HOME/bin/splunk" list forward-server

Expected output:

Active forwards:
    <indexer-address>:9997
Configured but inactive forwards:
    None

If the address under Active forwards doesn't match, or nothing is listed as active, also check what was actually loaded:

"$SPLUNK_HOME/bin/splunk" btool outputs list --debug

Step 4: Configure Parsing¶

JSON parsing needs a props.conf for the pqc:connections sourcetype on both tiers, but with different contents. This is because INDEXED_EXTRACTIONS = JSON performs its field extraction at input time, on whichever host reads the file — and with a Universal Forwarder in the path, that host is the UF, not the indexer. The indexer then only needs to be told not to re-extract the same fields again at search time.

Do both parts.

Step 4.a: Disable Re-Extraction on the Search Tier (Search Head or Indexer)¶

KV_MODE and AUTO_KV_JSON are search-time settings, so this stanza belongs on whichever tier runs searches. In a single-instance deployment that is the indexer; in a distributed deployment it is the search head(s), not the indexers. Create it on a default Splunk Enterprise install (/opt/splunk):

sudo mkdir -p /opt/splunk/etc/apps/tychon_pqc_sniffer/local
sudo tee /opt/splunk/etc/apps/tychon_pqc_sniffer/local/props.conf > /dev/null <<'EOF'
[pqc:connections]
KV_MODE = none
AUTO_KV_JSON = false
EOF

Set ownership on the app directory to match the account Splunk Enterprise runs as (commonly splunk; confirm with ps -ef | grep splunkd on that host). The user and group must match that account, or Splunk may not load the app:

sudo chown -R splunk:splunk /opt/splunk/etc/apps/tychon_pqc_sniffer

The UF already produces indexed fields (Step 4.b), so the search tier must not also auto-extract JSON at search time — that would duplicate and conflict with the indexed fields.
KV_MODE = none and AUTO_KV_JSON = false turn off search-time key/value and JSON auto-extraction for this sourcetype.
Other search-time knowledge for this sourcetype belongs in this same stanza and on this same tier. In particular, if you add the CIM field aliases from Align with the Splunk CIM Network Traffic Model, put them here (on the search head in a distributed deployment), not on the UF or indexers.

Then export the app globally, or the suppression above is silently ignored. Search-time settings only apply to searches run in the same app they live in, unless the app is exported to system (global). Dashboards and ad-hoc searches usually run in a different app (a dedicated dashboards app, or Search & Reporting) than tychon_pqc_sniffer, so without a global export Splunk falls back to the default AUTO_KV_JSON = true at search time and the field comes back duplicated (true true). Add an app metadata file:

sudo mkdir -p /opt/splunk/etc/apps/tychon_pqc_sniffer/metadata
sudo tee /opt/splunk/etc/apps/tychon_pqc_sniffer/metadata/local.meta > /dev/null <<'EOF'
[]
export = system
EOF
sudo chown -R splunk:splunk /opt/splunk/etc/apps/tychon_pqc_sniffer

Note

Use metadata/default.meta if the app is packaged or delivered by a Deployment Server, so the export ships with the app; metadata/local.meta is for a hand-deployed, instance-local app. Only the search-time settings need this — the index-time INDEXED_EXTRACTIONS in Step 4.b applies at ingest regardless of app export.

Restart Splunk Enterprise (see Step 6), then confirm the search tier sees the stanza as global with the REST API:

curl -sk -u admin \
  "https://localhost:8089/servicesNS/nobody/search/configs/conf-props/pqc%3Aconnections?output_mode=json"

The response should report sharing = global, AUTO_KV_JSON = 0, and KV_MODE = none. If sharing is app or user, the export did not take effect and the duplicate field will persist.

No other search-time extraction for this sourcetype

The search tier must not carry any additional REPORT-*, EXTRACT-*, FIELDALIAS-* (beyond the intended CIM aliases), or TRANSFORMS-* rules targeting pqc:connections. The fields are already created at index time (Step 4.b), so any extra search-time rule re-derives them and reintroduces the duplicate. Confirm none are present: /opt/splunk/bin/splunk btool props list pqc:connections --debug.

Step 4.b: Configure Structured Extraction on the Universal Forwarder¶

On the UF (/opt/splunkforwarder, i.e. $SPLUNK_HOME as set in Prerequisites):

sudo tee "$SPLUNK_HOME/etc/apps/tychon_pqc_sniffer/local/props.conf" > /dev/null <<'EOF'
[pqc:connections]
category = Network & Security
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
AUTO_KV_JSON = false
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIMESTAMP_FIELDS = @timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TZ = UTC
MAX_TIMESTAMP_LOOKAHEAD = 32
TRUNCATE = 100000
EOF

INDEXED_EXTRACTIONS = JSON extracts every field from each JSON line at index time — performed here on the UF — including nested fields like source.ip, tls.cipher, and pqc.quantum_safety.
TIMESTAMP_FIELDS = @timestamp tells Splunk to use the record's own event time instead of file modification time.
TZ = UTC is required. The sniffer writes @timestamp in UTC with a trailing Z (for example 2026-08-05T20:49:03.299Z), but the Z in TIME_FORMAT is a literal character, not a timezone directive — it does not tell Splunk the value is UTC. Without TZ = UTC, Splunk interprets the clock time in the forwarder's local timezone. On a host behind UTC (for example US Eastern/Central) this pushes _time into the future, so latest=now searches and dashboards return nothing even though the events indexed correctly. See Troubleshooting for how to confirm the fix and re-ingest already-mis-timed events.
SHOULD_LINEMERGE = false and LINE_BREAKER keep each NDJSON line as one event, since the sniffer writes one complete JSON object per line with no multi-line records.

Verify TIME_FORMAT against your own NDJSON output before relying on it

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ assumes every @timestamp value has exactly three fractional-second digits and a literal trailing Z (UTC), matching the example in the NDJSON Schema Reference. If your deployment emits a different fractional-second precision or a numeric UTC offset instead of Z, adjust TIME_FORMAT (and MAX_TIMESTAMP_LOOKAHEAD if the string length changes) to match a real sample line from your own connections*.ndjson before treating timestamp extraction as validated.

Restart each tier you edited — see Step 6.

Step 5: Create the Target Index (Indexing Tier)¶

Note

indexes.conf belongs on the tier that actually stores the data — normally the indexer(s). A heavy forwarder needs an index definition only if it indexes locally; a heavy forwarder that parses and forwards all data to indexers does not store events and does not need indexes.conf. Create the index where the data lands.

You can create the index either through Splunk Web or by editing indexes.conf directly. These are two different surfaces, not two views of the same fields — the Splunk Web index creation screen does not expose every indexes.conf setting, most notably frozenTimePeriodInSecs. Pick one method; don't try to reconcile the two field-for-field.

Method A: Splunk Web¶

Settings → Indexes → New Index, and set the fields the current UI exposes:

Index name
Index data type
Home path
Cold path
Thawed path
Maximum index size
Bucket size
Frozen archive path (if you want frozen data archived rather than deleted)

There is no separate UI field for frozenTimePeriodInSecs (retention time before freezing) — if you need to set that explicitly, use Method B instead.

Method B: indexes.conf¶

On the indexer (/opt/splunk), or a heavy forwarder only if it indexes locally:

sudo tee /opt/splunk/etc/apps/tychon_pqc_sniffer/local/indexes.conf > /dev/null <<'EOF'
[pqc_network]
homePath = $SPLUNK_DB/pqc_network/db
coldPath = $SPLUNK_DB/pqc_network/colddb
thawedPath = $SPLUNK_DB/pqc_network/thaweddb
maxTotalDataSizeMB = 512000
frozenTimePeriodInSecs = 7776000
EOF

The quoted heredoc (<<'EOF') writes $SPLUNK_DB to the file literally — Splunk expands it internally, so do not substitute a real path here. Retention is independent of the sniffer's own local circular rotation — frozenTimePeriodInSecs controls how long Splunk itself retains the ingested data.

Keeping the Index Name Consistent¶

pqc_network is only an example — use whatever index name fits your environment. If you change it, that name must match everywhere, not just in indexes.conf:

index = in inputs.conf (Step 2)
The index definition itself (Method A or B above)
Every verification search (Step 7)
All three storage paths, if using Method B — a common mistake is renaming the index stanza but leaving the paths pointing at the old name:

$SPLUNK_DB/<index-name>/db
$SPLUNK_DB/<index-name>/colddb
$SPLUNK_DB/<index-name>/thaweddb

For example, an index named tychon-sniffer should use $SPLUNK_DB/tychon-sniffer/db, not $SPLUNK_DB/pqc_network/db — a mismatch here creates an index whose logical name and on-disk location disagree, which still works but makes storage layout confusing to audit later.

Step 6: Restart the Affected Tiers¶

Restart whichever tier(s) you actually changed — not necessarily both:

Changed the search-tier props.conf (Step 4.a — search head, or the indexer in a single-instance deployment) or indexes.conf on the indexer (Step 5)? Restart that Splunk Enterprise host:

sudo /opt/splunk/bin/splunk restart

Changed inputs.conf or outputs.conf (Steps 2–3) or the UF's props.conf (Step 4.b)? Restart the Universal Forwarder:

sudo /opt/splunkforwarder/bin/splunk restart

If the UF is managed by systemd instead, use the service unit directly:

sudo systemctl restart SplunkForwarder.service

Warning

Do not run sudo $SPLUNK_HOME/bin/splunk restart here without first confirming which value $SPLUNK_HOME currently holds in this shell — if it's unset, this silently expands to sudo /bin/splunk restart and fails with command not found. The literal paths above avoid that failure mode entirely.

Step 7: Verify¶

On the indexer, search — set the time range picker to All time for this first check. Timestamp extraction issues (see the TIME_FORMAT warning in Step 4) or clock differences between hosts can place newly-arrived events outside a default Last 24 hours window, which looks identical to "no data arrived" even though it did:

index=pqc_network sourcetype="pqc:connections" | head 5

Once you've confirmed events exist somewhere, narrow the time range back down to confirm they're arriving with current timestamps as expected.

Confirm structured fields are extracted:

index=pqc_network sourcetype="pqc:connections"
| table _time, source.ip, destination.ip, tls.cipher, pqc.quantum_safety

Confirm each field is scalar, not duplicated. This is the check that catches double extraction — a field can be present and still be wrong if it was extracted twice (once at index time, once at search time). Every value_count should be 1:

index=pqc_network sourcetype="pqc:connections"
| where isnotnull('pqc.pqc_downgrade_detected')
| eval value_count=mvcount('pqc.pqc_downgrade_detected')
| stats count by value_count

Any value_count=2 means the search tier is re-extracting an already-indexed field — fix it with the Step 4.a suppression stanza and its global export, then re-check (the fix applies to historical events too). See the duplicate-field troubleshooting entry for the full diagnosis.

Confirm boolean predicates behave. Dashboards commonly test boolean fields with tostring(...), which only works when the field is scalar. This end-to-end check returns a real count once extraction is correct, and zero (or an error) while the field is still duplicated:

index=pqc_network sourcetype="pqc:connections"
| eval downgrade_flag=lower(tostring('pqc.pqc_downgrade_detected'))
| where downgrade_flag IN ("true","1")
| stats count as downgrade_events

Confirm the search-tier app is applied globally. If the dashboards run in a different app than tychon_pqc_sniffer, the Step 4.a suppression only takes effect when the app is exported to system:

curl -sk -u admin \
  "https://localhost:8089/servicesNS/nobody/search/configs/conf-props/pqc%3Aconnections?output_mode=json"

Expect sharing = global, AUTO_KV_JSON = 0, and KV_MODE = none. If sharing is app, apply the global export in Step 4.a.

Optional: Align with the Splunk CIM Network Traffic Model¶

If this data feeds Splunk Enterprise Security or other CIM-aware apps, add field aliases so the sniffer's ECS-style field names map onto CIM's expected names. Field aliases are applied at search time, so add these to the search-tier props.conf — the search head(s) in a distributed deployment, or the indexer in a single-instance one — in the same Step 4.a stanza, not on the UF or the indexers:

[pqc:connections]
FIELDALIAS-cim_network = source.ip AS src destination.ip AS dest source.port AS src_port destination.port AS dest_port network.transport AS transport network.bytes AS bytes network.protocol AS app

Troubleshooting¶

No events reach the index

Confirm the forwarder can reach the indexer and is actively forwarding: "$SPLUNK_HOME/bin/splunk" list forward-server.
Confirm the monitor stanza was actually loaded, and from the file you expect: "$SPLUNK_HOME/bin/splunk" btool inputs list --debug | grep -A8 -B2 '/var/log/pqc-sniffer'.
Confirm the monitor stanza path matches the sniffer's actual offline_output.output_file from YAML Reference, and that it uses a glob (connections*.ndjson) rather than an exact filename that may not exist if rotation is enabled.
Confirm the UF's service account can read the file(s): sudo -u splunkfwd head -n 1 /var/log/pqc-sniffer/connections*.ndjson. The connections*.ndjson glob is rotation-safe — it covers the unrotated connections.ndjson and any rotated connections.N.ndjson. This single check distinguishes four otherwise-identical-looking failure modes: a missing file, a wrong path, a permissions problem, and an empty file.
Check $SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder for input and connection errors — but see the note below about stale log entries.

Events arrive but fields aren't extracted (everything lands in _raw)

Confirm the props.conf with INDEXED_EXTRACTIONS = JSON exists on the UF (Step 4.b) — this is where JSON structured extraction happens when a UF reads the file. Verify what the UF actually loaded: "$SPLUNK_HOME/bin/splunk" btool props list pqc:connections --debug.
Confirm the UF was restarted after that file was created (Step 6) — INDEXED_EXTRACTIONS only takes effect for data indexed after the restart, not retroactively.
On the indexer, confirm its props.conf (Step 4.a) has KV_MODE = none and does not re-extract: /opt/splunk/bin/splunk btool props list pqc:connections --debug.

A field appears duplicated as a multivalue (for example pqc.pqc_downgrade_detected reads true true)

Symptom: a JSON field is extracted twice, so mvcount(...) = 2 with two identical values, and dashboard logic using tostring(...) fails even though valid events exist. The _raw line contains the value only once.

Cause: the field is created once at index time (Step 4.b INDEXED_EXTRACTIONS) and then extracted again at search time by Splunk's default AUTO_KV_JSON = true.
First check the Step 4.a suppression stanza (KV_MODE = none, AUTO_KV_JSON = false) actually exists on the search tier.
If it exists but the duplicate persists, the app is almost certainly not exported globally. Confirm with the REST check in Step 4.a: if sharing is app (not global), the dashboards run in a different app than tychon_pqc_sniffer and never apply the suppression. Add the metadata/local.meta (export = system) file and restart.
This is a search-time duplicate, so the fix is retroactive: mvcount should drop to 1 for both new and historical events, with no reindex needed. Verify with ... | eval c=mvcount('pqc.pqc_downgrade_detected') | stats count by c.

Events index but dashboards are empty (time is in the future)

Symptom: a search over All time (or latest=+6h) returns the events, but latest=now returns zero, and Dashboard Studio panels show nothing. _time is set several hours ahead of the current server time.

Cause: the timestamp is being interpreted in the wrong timezone. The sniffer writes @timestamp in UTC with a trailing Z, but the Z in TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ is a literal character, not a timezone directive. Without TZ = UTC in the sourcetype stanza, Splunk assumes the forwarder's local timezone; on a host behind UTC this places _time in the future, outside a normal -30d@d,now dashboard window.
Fix: add TZ = UTC to the [pqc:connections] stanza in the UF's props.conf (Step 4.b — timestamp assignment happens where INDEXED_EXTRACTIONS runs), then restart the UF.
Confirm the fix on newly indexed events: index=tychon-sniffer sourcetype="pqc:connections" | eval delta=_time-now() | stats max(delta) should be at or below zero, not several hours positive.
Already-indexed events keep their wrong _time — Splunk cannot rewrite it. Those events remain future-dated until you remove and re-ingest them (for example, ... | delete over the affected range, or clean the index on a test system), so only data indexed after the fix will appear in normal time ranges.

Forwarder connects but a specific host sends no data, or logs show No route to host / Connection refused

No route to host usually means the address in outputs.conf is wrong — re-check every octet against the actual indexer address, not just that it "looks right."
Connection refused usually means the indexer has a firewall rule allowing the port but is not actually listening on it. Confirm on the indexer with ss -lntp | grep ':9997', and configure receiving under Settings → Forwarding and receiving → Configure receiving if nothing is listening.

splunkd.log shows old connection errors even though data is flowing now

Log files retain historical entries. A Connection to host failed line from before you fixed the address or enabled the receiving port does not mean the problem is still occurring. Confirm current state with "$SPLUNK_HOME/bin/splunk" list forward-server and recent log output (tail -100 "$SPLUNK_HOME/var/log/splunk/splunkd.log") together, rather than treating any error anywhere in the log as current.

Gaps or duplicate events after rotation

Circular rotation truncates and reuses indexed filenames rather than renaming them away — see Rotation and Retention. If gaps are unacceptable, run with rotation_enabled: false on hosts feeding Splunk, and set flush_per_connection: true in offline_output so records are written promptly rather than sitting in a write buffer.