For analyzing the issue, I went for splunkd.log file in my forwarder:
It reads as the following (a certain part):
12-01-2016 03:49:59.902 -0500 INFO TailReader - File descriptor cache is full (100), trimming...
12-01-2016 03:50:11.768 -0500 INFO TailReader - File descriptor cache is full (100), trimming...
12-01-2016 03:50:22.282 -0500 INFO TailReader - File descriptor cache is full (100), trimming...
12-01-2016 03:50:32.875 -0500 INFO TailReader - File descriptor cache is full (100), trimming...
12-01-2016 03:50:37.084 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xx.x.x..x.x_xp490-0.x.x.com_x-0_AF0B42B5-4755-44C4-B2FvfbF-8F04543E3B7A
12-01-2016 03:50:37.090 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_151.162.96.34_8089_xp490-0.na.ko.com_x-0_AF0B42scB5-4755-44C4-B2FF-8F04543E3B7A
12-01-2016 03:50:46.228 -0500 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
12-01-2016 03:50:54.255 -0500 INFO TcpOutputProc - Connected to idx=1xxxxx:9yyy using ACK.
....//(3 similar lines bt diffrent indexers)
12-01-2016 03:51:37.100 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xx_xp490-0.xx.xx.com_xp490-0_AF0B42B5-4755-44C4-B2FF-8F04543E3B7A
12-01-2016 03:51:37.106 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_1xxx_xp490-0.xx.xx.com_xp490-0_AF0B42B5-4755-44C4-B2FF-8F04543E3B7A
12-01-2016 03:51:53.962 -0500 WARN TcpOutputFd - Connect to xxxx.xx.yyyy failed. Connection refused
12-01-2016 03:51:53.962 -0500 ERROR TcpOutputFd - Connection to host=**xxxx.xx.yyyy** failed//-------->this failed connection is to a heavy forwarder
12-01-2016 03:52:00.782 -0500 WARN TcpOutputProc - Forwarding to indexer group dev_cluster blocked for 100 seconds.
I need to know what is the reason behind:
12-01-2016 03:51:53.962 -0500 ERROR TcpOutputFd - Connection to host=**xxxx.xx.yyyy** failed//-------->this failed connection is to a heavy forwarder
Please advise
↧
Why is one Splunk forwarder not reporting in, but the splunkd service is running fine in the server?
↧