We're trying to do:
UF (Win Event Logs) --> HF (v7.2.5 on Linux) --> Indexers (Linux) -AND- external Syslog destination.
This works, but only sporadically at an acceptable rate. Most of the time I start up the HF, it routes the data properly to Indexers and Syslog destination, but it's extremely slow (only one event per second or so) and then the queues start to get blocked (the indexer queue first). Data still reaches each destination but very slowly.
Every now and then, I restart the HF and immediately works and it's blazing fast and continues to work - unless I restart the HF again (with NO changes) and it bogs down again.
When it's working, events make it from the Windows UF to the indexers and Syslog within one second. When it's not working, it just gets increasingly further behind, but is STILL routing the data to each destination very slowly.
If I remove the syslog pointer in props.conf so it goes ONLY to the indexers, it works just fine every time.
Here is the config:
props.conf:
[host::winhost01]
TRANSFORMS-ntdc = indexers, extSyslog
transforms.conf:
[indexers]
REGEX = (.)
DEST_KEY = _TCP_ROUTING
FORMAT = primary_indexers
[extSyslog]
REGEX = (.)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = destSyslog
outputs.conf:
[syslog:destSyslog]
server = 10.11.190.163:514
type = udp
I have a Splunk support case open and sent multiple diags but we're all stumped as to what's going on here. We've checked the bandwidth to the syslog destination, tried a couple different internal ones, changed settings here and there, and I've reinstalled Splunk on the HF. This has plagued me for 3-4 weeks now.
Any ideas would be appreciated.
↧