I recently setup ELK in order to begin collecting logs from Sysmon for security monitoring in my lab. The problem I could foresee running into was the issue of disk space. Unfortunately when my ELK server runs out of space, it runs out of space. I needed a way to clean up the logs when the server began to reach a threshold. This led me to Elastic Curator. Curator allows us to manage our indices which includes deleting indices over a given number of days ago. However, this is just a one-off command we can run so I wanted to add some logic to the process.
Disclaimer: This is strictly for one node setups, if you have a large setup with multiple clusters you will want a different solution. (You can use this method still but you might get some weird issues)
sudo pip install elasticsearch-curator
Create Curator Directory:
sudo mkdir /etc/curator
Create config file:
sudo nano /etc/curator/config.yml
Create delete config action file:
sudo nano /etc/curator/delete-after.yml
Unless you followed this guide you will most likely have to change some of the details in this action config file. Namely the filters –> value field. You will need to put the name of your index here.
Create Script file:
sudo nano /etc/curator/cleanup.sh
Warning: The above script will check that the disk space is less than 80 percent. It will check the free space of the root or whatever drive is mounted on “/”. If the free space is less than 20 percent, it will begin to delete the oldest indices, once it deletes the oldest one, it will check again and delete the next oldest until the disk space usage is below 80 percent. It will stop deleting regardless of disk space if there is less than 2 days of indices left. Since I couldn’t find an easy way to simple delete the oldest indices, I start at 90 days ago and move forward until it starts to find indices, if you have more than that, please adjust the script (the days variable at line 8). You may also want more than 2 days as a safety net.
Add script to cron:
(crontab -l 2>/dev/null; echo "5 0 * * * /etc/curator/cleanup.sh") | crontab -
This cronjob will run the script 5 minutes after midnight forever.
Curator should be added to the HELK soon. If you aren’t aware of the HELK and want to get into Threat Hunting (Or just want a super quick way to spin up and ELK stack) you should definitely look into the HELK. The incredible HELK (Hunting, Elasticsearch, Logstash, Kibana) makes it is easy as running a script to setup a reliable ELK stack tailored for threat hunting.
Check out these labs: https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat_98.html for an in-depth guide on how to set this stuff up manually as well as build the lab around it.
The advice and scripts contained and referenced in this point are provided with no warranty. As always, never blindly trust scripts off the internet let alone throw them into a cron job running as root.