Using Elastic Curator To Clean Up ELK

I recently setup ELK in order to begin collecting logs from Sysmon for security monitoring in my lab. The problem I could foresee running into was the issue of disk space. Unfortunately when my ELK server runs out of space, it runs out of space. I needed a way to clean up the logs when the server began to reach a threshold. This led me to Elastic Curator. Curator allows us to manage our indices which includes deleting indices over a given number of days ago. However, this is just a one-off command we can run so I wanted to add some logic to the process.

Disclaimer: This is strictly for one node setups, if you have a large setup with multiple clusters you will want a different solution. (You can use this method still but you might get some weird issues)

Install Curator:

sudo pip install elasticsearch-curator

Create Curator Directory:

sudo mkdir /etc/curator

Create config file:

sudo nano /etc/curator/config.yml

Paste in:

Create delete config action file:

sudo nano /etc/curator/delete-after.yml

Paste in:

Unless you followed this guide you will most likely have to change some of the details in this action config file. Namely the filters –> value field. You will need to put the name of your index here.

Create Script file:

sudo nano /etc/curator/cleanup.sh

Paste in:

Warning: The above script will check that the disk space is less than 80 percent. It will check the free space of the root or whatever drive is mounted on “/”. If the free space is less than 20 percent, it will begin to delete the oldest indices, once it deletes the oldest one, it will check again and delete the next oldest until the disk space usage is below 80 percent. It will stop deleting regardless of disk space if there is less than 2 days of indices left. Since I couldn’t find an easy way to simple delete the oldest indices, I start at 90 days ago and move forward until it starts to find indices, if you have more than that, please adjust the script (the days variable at line 8). You may also want more than 2 days as a safety net.

Add script to cron:

sudo su
(crontab -l 2>/dev/null; echo "5 0 * * * /etc/curator/cleanup.sh") | crontab -

This cronjob will run the script 5 minutes after midnight forever.

Curator should be added to the HELK soon. If you aren’t aware of the HELK and want to get into Threat Hunting (Or just want a super quick way to spin up and ELK stack) you should definitely look into the HELK. The incredible HELK (Hunting, Elasticsearch, Logstash, Kibana) makes it is easy as running a script to setup a reliable ELK stack tailored for threat hunting.

Check out these labs: https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat_98.html for an in-depth guide on how to set this stuff up manually as well as build the lab around it.

If you run into any issues, feel free to reach out to me on Twitter or by email!

The advice and scripts contained and referenced in this point are provided with no warranty. As always, never blindly trust scripts off the internet let alone throw them into a cron job running as root. 

 

Using ElastAlert to Help Automate Threat Hunting

I first want to say thanks to CyberWarDog for his fantastic lab walk through for setting up a Threat Hunting Lab. It is hands down the best guide I have read to getting started with Threat Hunting. I followed his guide and got my lab completely setup. I then decided that Elastalert would be pretty nice for getting some of the highly likely IOC’s sent off to a security team for further analysis. This post will guide you through setting up Elastalert to get notifications when certain actions are logged.

This guide assumes you have gone through all parts of CyberWarDogs tutorials: https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat.html

Not required, but it also assumes that you have set up Enhanced Powershell Logging so that we can begin to capture useful PowerShell data. https://cyberwardog.blogspot.com/2017/06/enabling-enhanced-ps-logging-shipping.html

Also not required but useful for this guide: A Slack Channel.

Cool, ready to go?

  • SSH or Console into your Ubuntu server running your ELK stack.
  • Download Elastalert from Yelp’s GitHub.
git clone https://github.com/yelp/elastalert
  • Copy Elastalert to ‘/etc/’
sudo cp -r elastalert /etc/
  • Change directory into your new Elastalert directory.
cd /etc/elastalert
  • If not already installed, install pip.
sudo apt install python-pip
  • Install Elastalert
pip install elastalert
  • Install ElasticSearch-py
  • pip install "elasticsearch>=5.0.0"
  • Install dependencies:
pip install -r requirements.txt
  • Lets make a directory for our Elastalert templates:
sudo mkdir templates
  • Change directory into our new templates directory
cd templates
  • Create a new template for monitoring commands executed:
sudo nano cmd_template.yaml

Paste:

es_host: localhost
es_port: 9200
name: "PLACEHOLDER"
index: winlogbeat-*
filter:
- terms:
    event_data.CommandLine: ["PLACEHOLDER"]
type: any
alert:
- slack
slack_webhook_url: "SLACK_WEB_HOOK"

es_host: This is the host your ELK stack is running on.

es_port: This is the port Elastic Search is listening on.

index: This is the index you setup with CyberWarDog’s blog.

filter: This is tell Elastalert to filter its search, in this case, we are filtering with ‘terms’ and we are looking for ‘event_data.CommandLine’ that equals whatever we put in place for PLACEHOLDER.

type: This means that Elastalert should alert on an matches that our Filter hits. We could also specify this Type to alert on new values identified, a spike in certain logs, a lack of logs and a bunch of other cool things.

alert: This tells elastalert how to alert you! There are a bunch of ways to get these alerts and I chose Slack for its simplicity to set up and use. For more options you can visit: http://elastalert.readthedocs.io/en/latest/ruletypes.html#alerts

  • Create a new template for monitoring powershell commands executed:
sudo nano powershell_template.yaml

Paste:

es_host: localhost
es_port: 9200
name: "PLACEHOLDER"
index: winlogbeat-*
filter:
- terms:
    powershell.scriptblock.text: ["PLACEHOLDER"]
type: any
alert:
- slack
slack_webhook_url: "SLACK_WEB_HOOK"
  • Create your main config.yaml file.
cd ..
sudo nano config.yaml

Paste:

rules_folder: alert_rules
run_every:
    seconds: 30
buffer_time:
    seconds: 60
es_host: localhost
es_port: 9200
alert_time_limit:
    days: 1
writeback_index: elastalert_status
alert_text: "Username: {0} \nHost: {1} \nTime: {2} \nLog:{3}"
alert_text_type: alert_text_only
alert_text_args: ["user.name","host", "@timestamp","log_name"]

To change the body of the alert, you can modify the last three lines, you can add or remove attributes to include in your report. https://elastalert.readthedocs.io/en/latest/ruletypes.html#alert-content

  • Create our Rules directory:
sudo mkdir alert_rules
cd alert_rules
  • Copy our templates here:
sudo cp ../templates/* .
  • Make copies of our templates.
cp cmd_template.yaml cmd_whoami.yaml
cp powershell_template.yaml powershell_invoke_webrequest.yaml
  • Modify cmd_whoami.yaml to alert when whoami is executed.
sudo nano cmd_whoami.yaml
  • Replace the PLACEHOLDER text in both locations with ‘whoami’, you can also copy this file many times over to alert on multiple commands ran.
es_host: localhost
es_port: 9200
name: "whoami"
index: winlogbeat-*
filter:
- terms:
 event_data.CommandLine: ["whoami"]
type: any
alert:
- slack
slack_webhook_url: "SLACK_WEB_HOOK"
sudo nano powershell_invoke_webrequest.yaml
es_host: localhost
es_port: 9200
name: "invoke-webrequest"
index: winlogbeat-*
filter:
- terms:
    powershell.scriptblock.text: ["webrequest"]
type: any
alert:
- slack
slack_webhook_url: "SLACK_WEB_HOOK"

Only query lowercase terms.

  • Remove the two template files we copied over:
sudo rm *template.yaml
  • Run elastalert-create-index and follow the prompts
elastalert-create-index

Remember: You host is localhost and your port is 9200, if you followed CyberWarDog’s guide, you also do not have authentication set up for ElasticSearch (You used nginx instead) so leave username and password empty. You also don’t have SSL or TLS setup.

  • Change directory back to /etc/elastalert
cd /etc/elastalert
  • Run elastalert –verbose
elastalert --verbose
  • Go to your Windows machine running winlogbeat and open up your command prompt.
  • Enter whoami and monitor your slack.
whoami
  • Profit

Commands you may want to monitor for:

Whoami

Netstat

Wmic

Powershell Functions you may want to monitor on:

Invoke-WebRequest

Invoke-Obfuscation

Downloadstring

Invoke-ShellCommand

If you are going to take this Threat Hunting thing seriously, you will most likely want to add alerts for Spikes, Frequency, Cardinality and a billion other types of things that are good ideas to check for with any Production system.

For comments, questions, concerns you can reach me at Twitter or via Email

[UPDATE: Several issues fixed 12/26]

 

Honey Accounts

I recently saw a tweet mentioning the use of an AD account with the password in the description attribute and logon hours set to none. I can’t find that tweet anymore so I apologize for the lack of attribution.

The idea is that when someone does breach your network perimeter, some of the first steps in performing recon is collecting information from Active Directory. In this recon, they stumble on a DA account called ‘helpdeskDA’. They even discover a password in the description! Well this looks like an easy win and a critical finding. In order to figure out how to leverage this new found user, the attacker attempts to RDP or use psexec to move to a higher value target. In doing so, AD checks the credentials and returns to the attacker that his newfound account is not allowed to login during this time. Meanwhile, this login attempt has triggered an alert and is being investigated.

I personally love security tools that aren’t just blinky lights and can run natively on devices we already possess. I took a stab at setting this up in my test environment

Setting up the AD Account:

  1. Create your new account in AD Users and Groups and add to the Domain Administrators group.
  2. Set the user description to something believable or just set the password in there and let the attacker make their own assumptions.

  1. Go to the Account tab and select Logon Hours…
  2. Set Logon Denied to 24×7

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\AD\login.png

Group Policy:

There are a couple Group Policy options that need to be enabled in order for this to work. Depending on your organization, these are already configured.

  1. Edit the Default Domain Policy.

  1. Navigate to: Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> Audit Policies -> Account Logon -> Audit Kerberos Authentication Services and set to audit Failure events.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\GP\gp_config1.png

Don’t forget to update group policy on the client by running: gpupdate /force

Event Viewer:

Here is an example of what the event now looks like in Event Viewer.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\EventViewer\eventLog.png

And here is the XML view with more details, this is important for writing our event trigger in Task Scheduler. Note the event ID. A normal failed login is event ID 4625. However, since this won’t be a traditional failed login due to the login hours, it falls under 4768 which indicates that the Kerberos authentication ticket was requested and failed due to various reasons, in this case the logon hours.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\EventViewer\eventDetails.png

Task Scheduler:

Configuring the task scheduler took a bit of time. This is due to the way the event was logged. Since it is not the traditional 4625 failed login event, the event trigger based on User did not work. I had to write a custom XML trigger rule to catch the username in this context.

  1. Open Task Scheduler and create a new task. Name it whatever you like and make sure to select ‘Run whether user is logged on or not’.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\TaskSched\step1_CreateTask.png

  1. Move to the trigger tab and edit trigger.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\TaskSched\step2_NewTrigger-Custom.png

  1. Select New Event Filter and select the XML tab. Check Edit query manually. Modify the following query to fit your user and domain. Since we don’t know what username format the attacker will use, we need some OR statements in there to cover our bases. Typically, we could use the SID but in Event 4678, it is not logged. You can also add granularity and only log on certain events but in this case, I thought it best to alert on any account activity.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\TaskSched\step3_EventCustomFilter.png

  1. You should be able to save that and move to the Action tab. You can do whatever sort of event here that you like. I chose a powershell script that alerts various people and provides details about the event.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\TaskSched\step4_actionEmail.png

  1. Select Start the task only if computer is on AC power.

Q:\Restricted\SECURITY\Application Security Analyst\HoneyAccounts\TaskSched\step5_conditionsTask.png

Powershell Script:

function sendMail{

Write-Host "Sending Email"

#SMTP server name

$smtpServer = "smtpserver"

#Creating a Mail object

$msg = new-object Net.Mail.MailMessage

#Creating SMTP server object

$smtp = new-object Net.Mail.SmtpClient($smtpServer)

$file = "C:\users\Administrator\Desktop\alert.txt"

$att = new-object Net.Mail.Attachment($file)

#Email structure

$msg.From = "[email protected]"

$msg.ReplyTo = "[email protected]"

$msg.To.Add("[email protected]")

$msg.To.Add("[email protected]")

$msg.subject = "ALERT:Honey Account"

$msg.body = "Honey Account Activity Detected"

$msg.Attachments.Add($att)

#Sending email

$smtp.Send($msg)

 


}

function collectDetails{

 $attachment= Get-WinEvent -LogName "Security" -FilterXPath "*[EventData[(Data='[email protected]' or Data='ws_admin' or Data='SECURITY-DC\ws_admin')]]" | Format-List -Property * | Out-File C:\users\Administrator\Desktop\alert.txt

}

#Calling function

collectDetails

sendMail