___ # Tags #splunk #splunk-queries #udemy #training #documentation #certifications # Links to other notes [[Splunk - Using Eval]] # Resources #### Quizlet Flash Cards https://quizlet.com/360840399/splunk-core-certified-power-user-flash-cards/ #### Splunk Certification Competencies https://www.splunk.com/pdfs/training/Splunk-Test-Blueprint-Power-User-v.1.1.pdf #### Splunk Certification Handbook https://www.splunk.com/pdfs/training/splunk-certification-candidate-handbook.pdf # Tips and Tricks - You can add comments to your searches like you are commenting code ```text index IN (so-nsm) ```this is a comment``` `src_ip_is_local` ```this searches for only internal ips``` `dest_ip_is_not_local` ```this searches for only external ips``` ``` - Ctrl + Shift + E will expand the current search you are working on - ![[Pasted image 20220401182742.png]] # Three Major Components to Splunk - Forwarder - Universal forwarder, heavy forwarder, intermediate forwarder - Indexer - Processes the raw data - Search Head - Craft SPL and search queries here # Types of Deployments - Standalone - No forwarders - Searching, indexing, parsing, inputs - All-in-one - Basic - Forwarders deployed - Multi-instance - Function separation - Searching, indexing, inputs are all handled separately - Clustering - Increases search capacity - Enable collaboration across environments - Need a minimum of 3 search heads to be considered a cluster - Also increases replication - Think - Search Head - Indexer - Forwarder # Data Pipeline - Input - Forwarders have data, data = streams - Parsing - Processing of data, data = events - License Usage - The utilization of how much data in a day you can ingest - Indexing - Data is written to disk, data = compressed # Input Phase - Input types - Files and directories, network traffic, log files, HEC (http event collector) - Source - /Path/to/the/data, method used for collecting - Host - Who sent the data - Sourcetype - Format of the data # Data Preview, Collecting Logs, and Data Inputs - Uploading a file to preview the data set before it gets written to disk - Settings, Data inputs, Choose a category, Add data, Upload, Select the file, Data preview - Collecting logs being sent over the network to a port - Settings, Data inputs, Choose a category, Add data, Monitor, Choose a source - The data preview pane can be found in Settings > Add Data - This will allow you to preview and select difference sourcetypes for parsing and view the impact it'll have on the upload - Monitor local files on your Splunk install with Settings > Add Data > Monitor # App vs Addon - Not very interchangeable terms - App, typically has a GUI to it for manipulating data - TA, technology addon - Parsing, regex, map data from appliance to CIM, non-GUI component - Adds functionality to the data, adds functionality in the background - Apps and Addons can be installed on all components in Splunk, SH, I, or FWDR # Search Operators - `AND`, searching for a value AND another value - `OR`, searching for a value OR another value - `=`, searching for a value equaling your value - `! =` , searching for value that is NOT the value - >, searching for a value greater than another value - <, searching for a value is less than another value - NOT - Note about `NOT` and `! =`, you will always have more results returned with NOT than =! as your search operator, NOT will return ANYTHING that does not equal the value you are indicating - `*`, searching for a partial value and wildcarding the rest to throw a larger net into your search # Knowledge Objects - KOs are a broad term, encompassing anything from an alert, fields, field extractions, tags, data models, macros, etc - Knowledge manager, someone who owns a knowledge object, as an example it could be someone who owns a dashboard - Naming convention, GROUPNAME_KO-TYPE_DESCRIPTION - SOC_ALERT_LOGINFAILURES # Permissions - Globally/All Apps - Accessible to everyone across the entire Splunk instance - Within an app/This App Only - Only accessible within the app it was created - Private - Only accessible by the user who created it # What are Fields - Fields are key-value pairs - Searchable by the name of the field - it is case sensitive - Created either by Splunk or recognized by an add-on - Splunk defaults to implicit `and` when searching fields unless you use a boolean operator to say otherwise - You can create custom fields as well - `fieldname!=value` vs `NOT fieldname=value` will bring back different results - `fieldname!=value` will search for everything not containing the value specified - `NOT fieldname=value` will search for anything that isn't `fieldname=value` and where `fieldname` doesn't exist - `fieldname!=value` is a subset of the search `NOT fieldname=value` - In the fields sidebar, the `a` symbol means the fields value is alphanumeric and the `#` symbol means the field value is numeric # SPL, Search Processing Language - SPL uses color-coding for different modifiers - Orange, `OR, NOT, AND, as, by` - Command modifiers - Blue, `stats, table, rename, dedup, sort, timechart` - Commands - Green, `limits, span` - Arguments - Purple, `tostring, sum, values, min, max, avg` - Functions #### Example Usage - ![[Pasted image 20220401182906.png]] # Transforming Commands - `top limit=20 [fieldname]` - Will show you the top 20 results of the field - You don't need to specificy a limit on your results - `rare limit=1 [fieldname]` - The rarest results of a field - You get more results if the field you are manipulating is a numerical value - `stats sum(bytes)` - This will calculate the total amount for the field bytes - `stats dc(fieldname)` - `dc` is distinct count and will give you the count of how many individual results are in the field specified - `stats list(fieldname)` - Instead of counting like `dc` it will list each individual value for the specific field - `stats values(fieldname)` - Will list all of the values for the specified field - `fillnull value= "Some Random String"` - Will fill empty values with whatever you put in-between the quotes #### Example Usage - This example shows login src_ip of the user and the user they are logging in as and how many times they tried ```txt index IN (so-hids) | stats values(data.dstuser) as "Linux Login User", count(data.dstuser) as "Login Attempts" by data.srcip | fillnull value="No Data Available" ``` # Transactions Commands - The difference between `transaction` and `stats` is in the cost of the search on the search head based on the amount of data you are parsing - `transaction` is good for correlating entire event chains over a small window of time, better for hunting - `stats` is good for sorting and grouping large volumes of data impersonally, better for speed - `maxspan=timeframe` - The length of time events can span in a transaction - `maxpause=timeframe` - The total pause between events grouped in the transaction - `startswith=fieldname` - Will group together specified fields in that transaction - `endswith=fieldname` - Marks the end of the transaction if matched # Eval, Where, and Search Commands - ![[Pasted image 20240504162805.png]] - You want to use the `where` command to compare two fields or match on a condition #### Comparison Between the Where and Search Commands - ![[Pasted image 20240504163146.png]] #### Example Usage - This takes http status codes and generates new fields based on the code specified in the `eval` command ```txt index IN (so-nsm) earliest IN (-1m) status_code IN (*) | eval status_codes = case((status_code = 200), "Successful response", (status_code = 204), "No content", (status_code = 404), "Error, not found") ``` #### Example Results - ![[Pasted image 20220401182928.png]] #### Example Usage - This command uses the `_internal` index and creates a field called `epoch_time`, then creates a field called `human-readable-time` using the `strftime` which is `string format time` and then converting it from `epoch_time` using the quotes to our desired format of `%m/%d/%y | %H:%M` ```txt index IN (_internal) | eval epoch_time = strptime(_time, "%s") | eval human-readable-time = strftime(epoch_time, "Calendar - %m/%d/%y | Clock - %H:%M") | table _time, epoch_time, human-readable-time ``` #### Example Results - ![[Pasted image 20220401183038.png]] #### Example Usage - This will run `stats` by `counting` all of the connections that get `eval'd` with `status_code == 200` and create a field called `Number of successful connections` ```txt index IN (so-nsm) earliest IN (-1m) status_code IN (*) | stats count(eval(status_code == 200)) as "Number of successful connections" ``` #### Example Results - ![[Pasted image 20220401183058.png]] # Rex and Erex - Regex - Unstructured data - Delimeters - Structured data - Commands - Work with rex and erex in SPL - Rex - Can create your own regex - Erex - SPL help with generating regex #### Example Usage for Regex - `(?<field_name>"the-regex-we-make-for-the-new-field")` - This is the default template for how to use `rex` to extract a new field - ![[Pasted image 20220401213809.png]] #### Example Usage for Erex - ![[Pasted image 20220402170708.png]] # To open the field extractor #### Three Ways to get to the Field Extractor - Settings > Fields > Field extractions > Open field extractor - After running a search you can scroll to the bottom of the Fields sidebar and there should be an option for Extract New Fields - You can also get to it in the Event Action drop down menu when clicking into the raw logs # Lookups - A file - Static data that is not in an index - Example - a CSV of all employees - A tool - Adds additional fields to search for - Fields are added to the fields bar menu - ![[Pasted image 20240504203010.png]] - You can get to the Lookup tables by going to Settings > Lookups > Lookup Table Files > Create New Lookup #### How to Use - Data enrichment - Add information and store in a table/file format to search - Commands - `Lookup`, - `inputlookup`, looks at data that exists - `outputlookup`, writes data to a table - `output` - `outputnew` - Create or upload - Select a file to upload or make one to reference - Can be configured to run automatically - Settings, Lookups, Lookup table files - Click New Lookup Table File #### Example Usage - To invoke and review the results in a lookup table use the following command ```txt | inputlookup [name-of-the-file.csv] ``` - You can also narrow down the results within the lookup table using commands like `where` ```txt | inputlookup [name-of-the-file.csv] where (fieldname=field) ``` # Visualizing your Data, Part One - Types - Tables - Charts - Maps - Commands - `timechart` - Display statistical trends over time, single or multi-series - `span=1d` - `chart` - Line, area, bar, bubble, pie, scatter, etc - Can stack - To remove empty values, `Useother=f`, or `Usenull=f` - Will not give you data-over-time - `stats` - Stacking - On, events are vertically stacked top to bottom - Off, counts are horizontally stacked - Overlay - Add two line charts over each other - Trellis - Display multiple charts at once - Multi-series - On, y-axis split on each value - Off, all fields share y-axis - Options for the panels - In the UI of Splunk - When you run a search that does not contain any visuals being generated by default the sub-menus will give you pivots - ![[Pasted image 20220403132549.png]] - ![[Pasted image 20240504205204.png]] #### Example Usage - Creating a visual based on PfSense firewall `vendor_action` counts ```txt index IN (firewall) | stats count by vendor_action ``` - Here is an example of available visualizations based around this information - ![[Pasted image 20220403140346.png]] # Visualizing your Data, Part Two - `iplocation` - Adds localization to your visualizations - `geostats` - Calc functions for displaying cluster mapping - `latfield`, `longfield`, `globallimit`, `locallimit` #### Example Usage - This search is searching the `so-nsm` index for the Bro/Zeek connection logs using the macro `src_ip_is_not_local` to weed out all of my intranet and then `dest_ip_is_wan_ip` to see only the traffic which is hitting my public IP address - Then generate the `lat` and `lon` fields `geostats` is going to use with the `iplocation` command ```txt index IN (so-nsm) sourcetype IN (bro:conn:json) `src_ip_is_not_local` `dest_ip_is_wan_ip` | iplocation src_ip | geostats latfield=lat longfield=lon globallimit=10 count by src_ip ``` - ![[Pasted image 20220403153355.png]] #### Example Usage - This search uses the `firewall` index to check for `vendor_action` where the `src_ip_is_not_local` and the `dest_ip_is_wan_ip`, then the `iplocation` does a geographic lookup and presents those fields to `geostats` and then starts geo-plotting based on the `blocks` against the wan IP being greater than 1000 ```txt index IN (firewall) `src_ip_is_not_local` `dest_ip_is_wan_ip` | iplocation src_ip | geostats count by vendor_action | where block>1000 ``` - ![[Pasted image 20220403160002.png]] - `addtotals` - Add multiple values together on a chart and compute total sums - `Fieldname`, `label`, `labelfield` - `trendline` - Overlay on a chart to show the moving avg - `sma`, simple moving avg - `ema`, exponential moving avg - `wma`, weighted moving avg # Reports and Drilldowns - Reports - A saved search - Live results - Shareable knowledge object - Drilldown Functionality - Actions - Link to search, dashboard, or report - $ tokens $ - Key role in passing variables from panel to panel - Export - PDT, print, or include in a report - Home dashboard - To set in the UI - Settings, Dashboards, Edit, Set as Home Dashboard #### Example Usage - When creating a dashboard that requires user input, generate your search and then save it to a dashboard - Then modify the dashboard to allow a user input and a submit button - Then move the token into the actual search to accept a user input ```txt index IN (_internal) earliest IN (-1d) log_level IN (*) | eval Date=strftime(_time, "%m/%d/%Y") | where isnotnull(reason) | table log_level, reason, Date | dedup reason | sort Date ``` - ![[Pasted image 20220403164556.png]] - So we have the search saved to a dashboard - Let's allow user input - Add input - Add text - Add submit - ![[Pasted image 20220403164722.png]] - To edit user input click the pencil icon - ![[Pasted image 20220403164826.png]] - To edit the field of the search to take user input you can have it be whatever you want it named, I would stick with field names - `log_level` being an actual field name let's modify that - `token` is also `log_level` in this example - ![[Pasted image 20220403164957.png]] - Now edit the search to have the token/user input you just created to actual affect the underlying search - ![[Pasted image 20220403165118.png]] - Make sure whatever the token you created looks like `$token name #### Example Usage - Adding a Time Picker - ![[Pasted image 20220403171525.png]] - ![[Pasted image 20220403171553.png]] #### Usage Example - Allowing Clickable Fields - Add a token name called `userclick` to pass a token to a new panel to show the events of what you just clicked instead of pivoting to a new search - ![[Pasted image 20220403173551.png]] # Alert Creation - Saved Searches - Run on a schedule an in real-time - Content Matches - Fire when a condition is met - Create trigger actions - Log, Send email, webhook, custom action - Create trigger conditions - Per-result - Number of results - Number of sources - Custom - Throttle # Tags and Events - Tag - Quick reminders - Aid for reading data - Case sensitive - Event types - Highlighter - Add colors, mark events with specific criterion - Like a report - Save searches as specific event types, sort into categories - Example, `status=404` can be saved as `Not Found` - More specific - Set strings, field values, and tags # Creating Tags and Eventtypes - In the UI for tagging your data - Settings, Tags, List by tag name, Sort by (me) as the owner - When you expand our your search results below you have the option to perform an action of tagging your data - ![[Pasted image 20220404190239.png]] - In this example I tagged `data.win.system.eventID=4624` as `login` - You can now search Wazuh logs by ```txt index IN (so-hids) tag IN (login) ``` - Now you don't need to remember that field name and eventcode every single time you want these search results back - In this example I am creating an eventtype and tag for `packetloss` which will search Zeek logs for any loss greater than 5% and highlight it in red - ![[Pasted image 20220404191047.png]] - After creating the eventtype and tag we can run the following search ```txt index IN (so-nsm) tag IN (packetloss) ``` - Here are the results - ![[Pasted image 20220404191307.png]] # Macros - Shortcuts - Repeatable - Expandable - Macro names will have backticks in order to execute them - In order to create a macro in the UI - Settings, Advanced search, Search macros or Add new - # Workflow Actions - Assess actions - Always depends on use-cases - Three available workflow actions - Create workflow action - Push, pull, or search data - Configure workflow action - Configure previously defined actions using a 3rd party source - Validation - Check to see if data is being pushed or pulled successfully - Two actions - GET, create HTML links to interact with sites such as querying WHOIS or doing a Google Search - POST, generate a request to a specified URI such as creating entries in forums or management systems - Search - Launching secondary searches to query data using field values - In the UI - Settings, Fields, Workflow actions - ![[Pasted image 20240504223028.png]] #### Example Usage - Performing a IP Lookup on External Source IPs - First we have to generate a search to find relevant IP data we want to lookup, I've done this by utilizing the following search - I run Suricata on the side on a network SPAN port and have routed my WAN/LAN traffic through the switch so I get the entire connection, not just the internal traffic ```txt index IN (so-ids) `src_ip_is_not_local` `dest_ip_is_wan_ip` | iplocation src_ip | table src_ip, src_port, City, Region, Country, dest_ip, dest_port, alert.signature, alert.signature_id ``` - The result of this search looks like this - ![[Pasted image 20220404200416.png]] - Let's use the IP `209.141.32.162` and perform a WHOIS query - Using the [domaintools service and whois lookup](https://whois.domaintools.com/209.141.32.162) - In order to add this workflow action to the `event actions` menu in the events you have to build it - Below I have named the action `Domaintools WHOIS` and assigned it to only apply to the `src_ip` field - ![[Pasted image 20220404200700.png]] - Now when you have any event with the field `src_ip` there will be a pivot option in the `event actions` menu of the event - ![[Pasted image 20220404203400.png]] # Data Normalization and Troubleshooting - Field aliases - ![[Pasted image 20220404203546.png]] - In the UI - Settings, Fields, Field aliases - Calculated fields - In the UI - Settings, Fields, Calculated fields - Buckets - Hot - Only writeable bucket - Data is searchable - Warm - Older bucket - Data is still searchable - Cold - Older data - Still searchable - Job inspector - Tool - Allows you to troubleshoot your search efficiency and reasons for failing - Informative - Info about how the search completed and time it took to run - Tips - Auto-suggested searches based on your knowledge objects (KO) # Datamodels - Hierarchical - Parent and child relationship, root dataset - Dataset searching - Select specific datamodel and dataset you want to search - Normalization tool - CIM compliance, mapping data to a common model - Large data searches - Search large datasets with tstats and accelerated datamodels - ![[Pasted image 20240505213743.png]] #### Usage Example of Datamodel Searching - ![[Pasted image 20220404213535.png]] - In order to get datamodels you will have to download the [CIM app from the Splunk app store](https://splunkbase.splunk.com/app/1621/) # CIM, Common Information Model - A model - Common standard for how all data should be mapped - An application - CIM builder add-on and CIM add-on are free - Data normalizer - All fields having the same name in the end across all datasets allowing all apps to coexist together - Splunk premium apps - Splunk ES relies heavily on data being CIM compliant - Health check tool - Perform more efficient searches using data models instead of raw events - Ease of use - Audit - Check to see if all data is CIM compliant