I have a heavy forwarder (Splunk Enterprise 7.0) that needs to parse a very nasty log file. I am interested in only a few of the events, so the forwarder needs to discard most of them. Most of the content are one-line messages about Comfort Noise.
At some time, a large multi-line event may appear that signals the creation of an RTSP stream. This is what I am interested in.
This is how it looks like in the log file:
09.11.2017 11:49:32,350; INFO; T0329; [sometext]; : [01 RTPReceiverTETRA::Receive(CN)]
09.11.2017 11:49:32,350; INFO; T0329; [sometext]; : Receive Comfort Noise -> Provide Silence (1600 Samples, 43246 Sequence Nr)
09.11.2017 11:49:32,350; INFO; T0042; [sometext]; : [01 RTPReceiverTETRA::Receive(CN)]
09.11.2017 11:49:32,350; INFO; T0042; [sometext]; : Receive Comfort Noise -> Provide Silence (1600 Samples, 42925 Sequence Nr)
09.11.2017 11:49:32,350; INFO; T0431; [sometext]; : [01 RTPReceiverTETRA::Receive(CN)]
09.11.2017 11:49:32,350; INFO; T0431; [sometext]; : Receive Comfort Noise -> Provide Silence (1600 Samples, 43399 Sequence Nr)
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : RTSPRcv <192.168.25.181|37093>
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : ANNOUNCE rtsp://192.168.25.138:554/iprecorder/ RTSP/1.0
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : CSeq: 3912
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : User-Agent: RecService
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : Timestamp: 260983967488 0
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : Content-Type: application/sdp
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : Content-Length: 291
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; :
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : v=0
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : o=unnamed 0 0 IN IP4 192.168.25.138
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : s=conversation
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : c=IN IP4 0.0.0.0
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : t=0 0
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : u=sip:9109000@192.168.25.180:5060
18.10.2017 08:08:56,032; INFO; T0030; [sometext]; : m=audio 0 RTP/AVP 99 123
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : a=rtpmap:99 x-ptt-TETRA/8000
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : a=rtpmap:123 x-ptt-R2S/8000
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : a=sendonly
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : a=type:Radio-Rxonly
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : a=R2S-KeepAlivePeriod:200
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : a=R2S-KeepAliveMultiplier:10
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; :
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : [01 RTSPServer::OnRTSPRequestAnnounce()]
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : [02 RTSPController::TraceRTSPMessage()]
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : RTSPTxm <192.168.25.181|37093>
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : RTSP/1.0 200 OK
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : CSeq: 3912
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : Timestamp: 260983967488 0.001
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : Server: RECORDER.RTSP/3.30.012 libTPP/0.07.000
18.10.2017 08:08:56,033; INFO; T0030; [sometext]; : Date: Wed, 18 Oct 2017 08:08:56 GMT
On the Forwarder, I have transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[set-rtsp-events]
REGEX = RTSPRcv|RTSPTxm|RTPChannel|rtsp:\/\/
DEST_KEY = queue
FORMAT = indexQueue
And the props.conf:
[rtsp]
TZ=UTC
SHOULD_LINEMERGRE=true
BREAK_ONLY_BEFORE_DATE=false
LINE_BREAKER=(; : {4}([\r\n]+)|(([\r\n])+\d+.\d+.\d+ \d+:\d+:\d+,\d+)
MUST_NOT_BREAK_AFTER=\]; : (?RTSPRcv|RTSPTxm) <(?\d+\.\d+\.\d+\.\d+)\|(?\d+)>
MUST_BREAK_AFTER=\]; : {4}([\r\n]+)
TRANSFORMS-parsing=setnull,set-rtsp-events
Here I need some help.
The setnull parsing should take care to discard everything. Only the things discovered by the regexes will have the indexQueue.
As long as I don't use the BREAK-rules, it works fine, but I don't get the multi line events (only the first line of each).
When I try to fetch the multi-line event, it captures far too much, mostly several thousands of lines, because the end-criteria don't match.
Unfortunately, every line of the multi-line event begins with a timestamp, so I can't use the BREAK_ONLY_BEFORE_DATE.
↧