I need help installing a py script to call the fireeye HX API and GET all HX json data (more data than collected from the FireEye App and Add-on for Splunk Enterprise) into Splunk.
I got an awesome python script written by Ruairi O'Mahony for HX. When I use it in pycharm terminal (free edition), it returns the data I want. So now I want to test it in my standalone Dev instance of Splunk. However ultimately the script will reside on a HF (Heavy Forwarder) in a distributed production environment.
I am new to scripting in Splunk so I am not sure what the best way to go about this.
The script has options as to what data you wish to collect.
I run the command in the pycharm terminal with an option like this...
`C:\Users\myname\PycharmProject\HX_API> C:\Python27\python.exe C:/Users/myname/PycharmProject/HX_API/RO_hx_api_script.py 1`
as you can see there is a number one at the end which correlates to one of the following options
To use this script append one of the following options to the end of the command line:
0 - All options - Run all options
1 - HX version info - v2/version
2 - List all alerts - v2/alerts?limit=200000
3 - List source alerts from another appliance for all hosts - v2/source_alerts?limit=200000
4 - List of hosts - v2/hosts?limit=200000
5 - List of acqusitions - v2/acqs/files?limit=200000
6 - List of bulk acqusitions - v2/acqs/bulk?limit=200000
7 - List of Triages - v2/acqs/triages?limit=200000
8 - List all searches - v2/searches?limit=200000
9 - List all scripts - v2/scripts?limit=200000
10 - Download all scripts - v2/scripts.zip
11 - Indicator caregories - v2/indicator_categories?limit=200000
12 - Custom indicators - v2/indicator_categories/custom?limit=200000
13 - FireEye indicators - v2/indicator_categories/fireeye?limit=200000
14 - FireEye-CMS indicators - v2/indicator_categories/fireeye_cms?limit=200000
15 - Imported indicators - v2/indicator_categories/imported?limit=200000
16 - DTI intel from Mandiant - v2/indicator_categories/mandiant?limit=200000
17 - Containment states - v2/containment_states?limit=200000
Here is a copy of the full script (permitted to share by the author)...
#!/usr/bin/python
# Rev. 20170313.1
# Copyright 2016-2017, Ruairi O'Mahony
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#### import libraries
import urllib2
import ssl
import json
import csv
import base64
import getpass
import sys
import time
import os
#### define variables for loggin
appliance = "your ip adress goes here"
username = "your HX api analyst username goes here"
password = "your HX api analyst pswd goes here"
#### set-up options list for use in menu
options = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17"]
name = ["All options", "HX version info", "List all alerts", "List source alerts from another appliance for all hosts",
"List of hosts", "List of acqusitions", "List of bulk acqusitions", "List of Triages", "List all searches",
"List all scripts", "Download all scripts", "Indicator caregories", "Custom indicators", "FireEye indicators",
"FireEye-CMS indicators", "Imported indicators", "DTI intel from Mandiant", "Containment states"]
uri = ["Run all options", "v2/version", "v2/alerts?limit=200000", "v2/source_alerts?limit=200000",
"v2/hosts?limit=200000", "v2/acqs/files?limit=200000", "v2/acqs/bulk?limit=200000",
"v2/acqs/triages?limit=200000", "v2/searches?limit=200000", "v2/scripts?limit=200000", "v2/scripts.zip",
"v2/indicator_categories?limit=200000", "v2/indicator_categories/custom?limit=200000",
"v2/indicator_categories/fireeye?limit=200000", "v2/indicator_categories/fireeye_cms?limit=200000",
"v2/indicator_categories/imported?limit=200000", "v2/indicator_categories/mandiant?limit=200000",
"v2/containment_states?limit=200000"]
#### Functions used
# http handler
def build_request(url, tests):
handler = urllib2.HTTPHandler()
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
data = tests
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
request = urllib2.Request(url, data=data)
return request, ctx
# get token from HX
def authorization(appliance, encoded_authorization_string):
request = build_request('https://' + appliance + ':3000/hx/api/v1/token', None)
get_request = request[0]
ctx = request[1]
get_request.add_header('Accept', 'application/json')
get_request.add_header('Authorization', 'Basic ' + encoded_authorization_string)
get_request.get_method = lambda: 'GET'
try:
response = urllib2.urlopen(get_request, context=ctx)
except urllib2.HTTPError as e:
print e.read()
except urllib2.URLError as e:
print 'Failed to connect to HX API server.'
print 'Reason: ', e.reason
else:
return response.headers['X-FeApi-Token']
# get json
def get_json(token, appliance, uri, string):
request = build_request('https://' + appliance + ':3000/hx/api/' + uri + '', None)
put_request = request[0]
ctx = request[1]
put_request.add_header('X-FeApi-Token', token)
put_request.add_header('Accept', 'application/json,application/octet-stream')
put_request.add_header('Content-Type', 'application/json')
put_request.get_method = lambda: 'GET'
try:
response = urllib2.urlopen(put_request, context=ctx)
except urllib2.HTTPError as e:
print e.read()
except urllib2.URLError as e:
print 'Failed to connect to HX API server.'
print 'Reason: ', e.reason
else:
print '############### Result for: ' + string + ' (https://' + appliance + ':3000/hx/api/' + uri + ')' + ' ###############\n'
result = response.read()
parsed = json.loads(result)
result = 0
result = json.dumps(parsed, indent=2, sort_keys=True)
print result
print '\n\t\t\t############### Result end ###############\n'
return result
# function to download all the scripts
def script_get(token, appliance, uri, string):
request = urllib2.Request('https://' + appliance + ':3000/hx/api/v2/scripts.zip', None)
context = ssl._create_unverified_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
request.add_header('X-FeApi-Token', token)
request.add_header('Accept', 'application/octet-stream')
request.get_method = lambda: 'GET'
try:
response = urllib2.urlopen(request, context=context)
except urllib2.HTTPError as e:
print e.read()
except urllib2.URLError as e:
print 'Failed to connect to HX API server.'
print 'Reason: ', e.reason
else:
scripts = response.read()
op = open(filepath, 'w')
op.write(scripts)
op.close()
op1 = open(filepath1, 'w')
info = "Script was run on: " + now + "\nScript was ran by: " + user
op1.write(info)
op1.close()
print '\n############### Result for: ' + string + ' (https://' + appliance + ':3000/hx/api/' + uri + ')' + ' ###############\n'
print 'The scripts have been downloaded to \n' + filepath + '\nand stored in xml format \n' # note that file opens easier with an app like unrar
print '\t\t\t############### Result end ###############\n'
# print the options for the script
def print_useage():
print "To use this script append one of the following options to the end of the command line: \n"
for x, y, z in map(None, options, name, uri):
print str(x) + " - ", str(y) + " - ", str(z)
print "\n"
print "Option 10 will be stored in the following location: " + filepath + "\n"
#### main execution
# read login details, will check that default details have been removed, if not it will use them
if appliance == "":
appliance = raw_input("HX IP address: ")
if username == "":
print "**User needs to be api_analyst role**"
username = raw_input("Username: ")
if password == "":
password = getpass.getpass() # the get pass function keeps user input off the screen
# create the auth string using base 64 encoding
decoded_authorization_string = username + ":" + password
encoded_authorization_string = base64.b64encode(decoded_authorization_string)
put_request = 0
filepath = os.path.join(os.path.expanduser('~'), 'HX_Scripts.zip') # create new folder to store results
filepath1 = os.path.join(os.path.expanduser('~'), 'HX_Timestamp.txt')
token = authorization(appliance, encoded_authorization_string)
now = time.strftime("%c") # get current time
user = getpass.getuser() # currently logged in user account
print "\n"
if len(sys.argv) > 1: # check if arguments passed into script
n = sys.argv[1]
m = len(options)
if n in options: # check the argument passed is in the options range
if n == "0": # if argument == 0 run all the options
for o in range(0, m):
if o != 10:
get_json(token, appliance, uri[o], name[o])
elif o == 10:
script_get(token, appliance, uri[o], name[o])
o += 1
elif n == "10":
for o in range(0, m):
if int(n) == o:
script_get(token, appliance, uri[o], name[o])
o += 1
elif n != "0": # if argument != 0 run the required option
for o in range(0, m):
if int(n) == o:
get_json(token, appliance, uri[o], name[o])
o += 1
else:
print("No option selected")
else:
print_useage()
I am not sure of best way to run this or where to place the script. Any advice or guidance or reference is much appreciated.
I have been using this reference http://docs.splunk.com/Documentation/Splunk/6.6.2/Data/Getdatafromscriptedinputs
Any other suggestions as to how to run this script with the different option choices?
I am getting the script to run but still need guidance on where to append the option number (that I choose) in the conf files. Unless my only option is to hard code the script....
Thank you
↧