Hi there,
I have 3 kinds of devices:
- device1 (IP: 192.168.10.12, 192.168.10.13, 192.168.10.27, 192.168.10.28)
- device2 (IP: 192.168.20.12, 192.168.20.13, 192.168.20.27, 192.168.20.28)
- device3 (IP: 192.168.30.12, 192.168.30.13, 192.168.30.27, 192.168.30.28)
All of them send their log data via syslog to a Splunk Heavy Forwarder (HF) that acts as a syslog collector for all the devices that cannot run a Universal Forwarder.
HF processes the data (sets the sourcetype and index) and forwards it to Splunk Indexers. HF processes and forwards only the data received from device1, device2 or device3 IPs. If anyone else sends anything to HF syslog, that data is dropped (license limits, as well as I would like to have a control over what is being sent to indexers).
Currently I'm filtering stuff based on the device IP, but please do not focus on that. The REGEX filtering could (and most likely will) be done based on something else.
I have the following configuration:
**inputs.conf**
[udp://514]
index = nullIndex
connection_host = ip
disabled = 0
**props.conf**
[source::udp:514]
TRANSFORMS-010-device1 = device1_sourcetype, device1_index
TRANSFORMS-020-device2 = device2_sourcetype, device2_index
TRANSFORMS-030-device3 = device3_sourcetype, device3_index
TRANSFORMS-999-drop_everything = drop_null_index
**transforms.conf**
[device1_index]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.10\.(1[23]|2[78])$
DEST_KEY = _MetaData:Index
FORMAT = device1_i
[device1_sourcetype]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.10\.(1[23]|2[78])$
DEST_KEY = MetaData:Sourcetype
FORMAT = device1_st
[device2_index]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.20\.(1[23]|2[78])$
DEST_KEY = _MetaData:Index
FORMAT = device2_i
[device2_sourcetype]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.20\.(1[23]|2[78])$
DEST_KEY = MetaData:Sourcetype
FORMAT = device2_st
[device3_index]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.30\.(1[23]|2[78])$
DEST_KEY = _MetaData:Index
FORMAT = device3_i
[device3_sourcetype]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.30\.(1[23]|2[78])$
DEST_KEY = MetaData:Sourcetype
FORMAT = device3_st
[drop_null_index]
REGEX = ^nullIndex$
SOURCE_KEY = _MetaData:Index
DEST_KEY = queue
FORMAT = nullQueue
**outputs.conf**
[tcpout]
defaultGroup = splunk_indexers
disabled = 0
[tcpout:splunk_indexers]
server = splunk-indexers.local:9997
maxQueueSize = 500MB
useACK = true
disabled = 0
Pretty much all input data is marked as *nullIndex* right away in **inputs**, and then in **props** and **transforms** the desired devices are re-marked with their respective sourcetypes and indexes, and everything else is dropped.
I am not sure if this is the BEST configuration (I've spent some time trying to get my head around this), but it certainly WORKS for me quite well.
However, **the problem is** - device1 creates about 50 GB log data / day. That means 50 GB of data goes through seven REGEX filters, even though only the first two apply to device1 data. From the performance perspective, I would like to avoid the other five REGEX filters.
So here's **my question**: is there any way to tell Splunk, right after finishing `TRANSFORMS-010-device1` line in props.conf, to pretty much be done with other transformations and send the device1 data immediately to Splunk Indexers?
Something similar like "Discard" tilde in RSYSLOG configuration:
:fromhost-ip, isequal, "192.168.10.12" @@(o)syslog.local:6514
& ~
:fromhost-ip, isequal, "192.168.10.13" @@(o)syslog.local:6514& ~
:fromhost-ip, isequal, "192.168.10.27" @@(o)syslog.local:6514& ~
:fromhost-ip, isequal, "192.168.10.28" @@(o)syslog.local:6514& ~
however not to discard the data, but remove it from any further processing other than just sending it to Splunk Indexers.
So my props.conf would look something like this (?) :
[source::udp:514]
TRANSFORMS-010-device1 = device1_sourcetype, device1_index, device1_fast_exit
TRANSFORMS-020-device2 = device2_sourcetype, device2_index, device2_fast_exit
TRANSFORMS-030-device3 = device3_sourcetype, device3_index, device3_fast_exit
TRANSFORMS-999-drop_everything = drop_null_index
where **device1_fast_exit** would be (pseudo-code) "*send the device1 log data immediately to Splunk Indexers and do not process that data with TRANSFORMS-020-device2 and TRANSFORMS-030-device3 lines.*"
↧