Splunk

 What is Splunk?...

Splunk is ‘Google’ for our machine-generated data. It’s a software/engine that can be used for searching, visualizing, monitoring, reporting, etc. of our enterprise data. Splunk takes valuable machine data and turns it into powerful operational intelligence by providing real-time insights into our data through charts, alerts, reports, etc.

How to extract IP addresses from logs?...

rex field=_raw  "(?<ip_address>\d+\.\d+\.\d+\.\d+)"

How to troubleshoot splunk performance?...

• Check splunkd.log for errors
• Check server performance issues, i.e., CPU, memory usage, disk I/O, etc.
• Install the SOS (Splunk on Splunk) app and check for warnings and errors in its dashboard
• Check the number of saved searches currently running and their consumption of system resources
• Install and enable Firebug, a Firefox extension. Log into Splunk (using Firefox) and open Firebug’s panels. Then, switch to the ‘Net’ panel (we will have to enable it). The Net panel will show us the HTTP requests and responses, along with the time spent in each. This will give us a lot of information quickly such as which requests are hanging Splunk, which requests are blameless, etc.

Command for restarting the Splunk Daemon?...

splunk start splunkd