I have a scenario wherein each heavy forwarder has syslog listeners running. I need an alert or something in the dashboard to show that a particular heavy forwarder has the following listener down.
I did the following and was able to list the Splunk heavy forwarder, listener and its associated PID:
index=operatingsys host=hf1 OR host=hf2 or host=hf3 source="/var/run/syslog/*" | rex .......... | table host listenername PID
o/p was as follows
host listenername PID
hf1 ciscolistener 123
hf1 winlistener 567
hf2 ciscolistener 345
hf2 winlistener 789
hf3 ciscolistener 654
hf3 winlistener 523
hf3 whitecoat 231
Now , I can share an input CSV list as below which is static
host listenername
hf1 ciscolistener
hf1 winlistener
hf1 whitecoat
hf2 ciscolistener
hf2 winlistener
hf2 whitecoat
hf3 ciscolistener
hf3 whitecoat
hf3 winlistener
As you see in my search output, hf2 and hf3 have whitecoat missing (meaning whitecoat listener status is down, and technically, it will not have PID assigned as it is down). I need to show that the listeners on the heavies are down in a dashboard/report, and for an alert to be generated whenever any listener is down on one of our heavy forwarders.
Open to discussion on using any different approach, if possible to wrap this up
↧