Disk monitoring with Splunk Enterprise

Disk monitoring with Splunk Enterprise can tell you how many dodgy hard drive you have running on your network so you can be proactive. This is al thanks to Luke Murphey and its Disk Monitoring app for Splunk Enterprise. In this article we'll explore how to do it

First of all , go ahead and download the app for your Splunk Enterprise from this link:

https://splunkbase.splunk.com/app/3279/

Installing and configuring the app on your Splunk Enterprise Server

Installation for Windows

Once you have installed the app on your Splunk Enterprise server, you need to install the Splunk Universal Forwarder on your windows client machines

1.Check to accept the license

2. Leave the "Deployment Server" blank, we'll do it later with a command, but enter the "Receiving Indexer" with the IP of your Splunk Enterprise

3.Once you finish installing the Universal Forwarder, copy the Disk Monitoring app (the "smartmon" folder) from the location SPLUNK_HOME$\etc\app on the Splunk Enterprise to the Windows client. Use "netstat" to verify that you are connected to the Splunk server by port 9997

4. Open the Command Prompt with Administrative privileages, and navigate to the "smartmon" folder > bin and run the 2 x windows scripts that are there, this will generate some data that will be passed to the Splunk Disk monitoring app. Then issue these two commands to ensure that splunk starts at boot up

C:\Program Files\SplunkUniversalForwarder\bin\splunk enable boot-start

C:\Program Files\SplunkUniversalForwarder\bin\splunk restart

 If the cmd files don't work, copy the "systemct.exe" to C:\windows and modify the script files to launch the utility from the C:\windows path

Once you copy the systemctl.exe  to the C:\Windows, edit the .cmd files located in C:\Program Files\SplunkUniversalForwarder\etc\apps\smartmon\bin and configure them as follows:

smartmon_results.cmd

@echo off

:: Determine if this is a 32-bit or 64-bit OS
Set ARCH=win64
IF %PROCESSOR_ARCHITECTURE% == x86 (
  IF NOT DEFINED PROCESSOR_ARCHITEW6432 Set ARCH=win32
  )

:: Create a reference to the smartctl command
SET SMARTCTL=C:\Windows\smartctl.exe

:: Get the list of drives
SET COMMAND_TO_RUN="%SMARTCTL%" --scan

:: Run smartctl on each disk
for /f "usebackq tokens=*" %%a in (`%COMMAND_TO_RUN%`) do (
    for /f "tokens=1 delims= " %%b in ("%%a") do (
        "%SMARTCTL%" -a %%b
    )
)

smartmon_short_test.cmd

@echo off

:: Determine if this is a 32-bit or 64-bit OS
Set ARCH=win64
IF %PROCESSOR_ARCHITECTURE% == x86 (
  IF NOT DEFINED PROCESSOR_ARCHITEW6432 Set ARCH=win32
  )

:: Create a reference to the smartctl command
SET SMARTCTL=C:\Windows\smartctl.exe

:: Get the list of drives
SET COMMAND_TO_RUN="%SMARTCTL%" --scan

:: Run smartctl on each disk
SET TEST="short"
for /f "usebackq tokens=*" %%a in (`%COMMAND_TO_RUN%`) do (
    for /f "tokens=1 delims= " %%b in ("%%a") do (
        "%SMARTCTL%" -t %TEST% %%b
    )
)

Installation on iMacs

This is the fun part! For the MAC OSX to talk to Splunk, be very much aware that the port 9997 (which is the beloved port for Splunk) is in use on the macOSX world for some kind of chat program called "palace". You can verify that by open the "Network Utility" on a Mac OSX and select to scan for the ports of your Splunk Enterprise server

That means that we have to use another port for communicating to Splunk Enterprise. So go ahead and visit this website that list all ports assigned on the Internet and use a completely free one of your choice

https://www.adminsub.net/tcp-udp-port-finder/

For my example, I'm using port 10587, which as you can see if not being assigned to anything:

You need to add that port as well (of course) on the Splunk Enterprise. To do that visit Settings >Forwarding and Receiving > Receive Data and add port 10587 so the clients can send data to that port

We need to create a Server Class as well for this app, so in your Splunk Enterprise go to Settings > Forwarder Management > Server Class and create a new one called "DiskMon"

Now copy the "smartmon" folder to the location /opt/splunk/etc/deployment-apps in the Splunk Enterprise and yes... restart Splunk for this last change to be effective

Choose any iMac that you like, and copy to the /Applications/SplunkForwarder/etc/deployment-apps the same folder. On that give iMac, create a file called deploymentclient.conf and add this info to it:

[target-broker:deploymentServer]
targetUri = 192.168.0.44:8089

You need to do the above only once, and just on one iMac, this is so that the .sh scripts are detected and manageable by Splunk Enterprise. Once you finished go to Data Inputs > Scripts > and clone any of the .sh scripts, pointing them to the right location in the /Applications/SplunkForwarder folder in the iMacs. Finally, visit Data Inputs > Remote Scripts and enable the two newly detected .sh scripts in your Splunk Enterprise:

Moving ahead, and once you got the port configured, let's start by downloading the Splunk Universal Forwarder for macOS and install it ;visit this link and logon to your Splunk account to proceed: https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/macos  Remember to do this job only when logging on as admin

After the installation, start the service

If you're prompted to install additional code, go ahead and install it

If installing for the first time, you should get a "successfully" notification

Then click on the "Splunk" icon on your desktop, and logon using the admin details you use to logon to your Splunk Enterprise server

And yes to the next window, start Splunk:

You are done (for now) with Splunk

Install Smartmontools in macOS

Visit this link and install the smartmontools package: https://sourceforge.net/projects/smartmontools/files/

 You should received a green tick after the install, well done! :)

The installation of smartmoontools places the "smatctl" program and associated in the folder /usr/local/sbin, which your iMac won't find on its path unless you add it to your environment. I find much easier to just visit that folder (Open Finder > Go > Go to Folder...) and copy the files to the location /usr/local/bin, which will be found by your iMac and therefore by Splunk. To facilitate all this copy process, open the terminal and run this command:

sudo cp /usr/local/sbin/* /usr/local/bin

Just like on Windows, copy the "smartmoon" folder into /Applications/SplunkForwarder/etc/apps

 Once you copy, open the Terminal and run this command:

sudo chmod uo+x /Applications/SplunkForwarder/etc/apps/smartmon/bin/*.sh

Configure the Splunk Forwarder

Using the Terminal, visit the location /Applications/SplunkForwarder/etc/system/local and run the following:

sudo vi inputs.conf

Populate the inputs file with this, change the IP address of your Splunk Enterprise host accordingly, and remember to change the NAME to match the name of the client!

[default]
host = NAME

[script://$SPLUNK_HOME/etc/apps/smartmon/bin/smartmon_results.sh]
disabled = false 
host = NAME
index = main
interval = 300 
source = smartmon
sourcetype = smartmon

[script://$SPLUNK_HOME/etc/apps/smartmon/bin/smartmon_short_test.sh]
disabled = false 
host = NAME
index = main
interval = 300      
source = smartmon
sourcetype = smartmon

Now either create or edit the outputs.conf file to look like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.0.40:10587

[tcpout-server://192.168.0.40:10587]

Now open another terminal and execute:

sudo /Applications/SplunkForwarder/bin/splunk enable boot-start

sudo /Applications/SplunkForwarder/bin/splunk restart

Job done, you should be able to see the hard disk reporting to the Splunk app, good luck with that!

Troubleshooting commands

netstat -ap tcp  //**verify that ports 10587 is open to the splunk enterprise server

./splunk list monitor //**the scripts should be listed there, if it doesn't use:

sudo ./splunk add monitor /Applications/SplunkForwarder/etc/apps/smartmon/bin/smartmon_results.sh

sudo ./splunk add monitor /Applications/SplunkForwarder/etc/apps/smartmon/bin/smartmon_short_test.sh

./splunk list forward-server //**it should be active

GIVE A GOOD REBOOT TO THE iMAC TO ENSURE THAT THE SPLUNK SERVICE STARTS AUTOMATICALLY

If you have enjoyed reading this article about Disk monitoring with Splunk Enterprise, you will also enjoy reading this other one:

• Install Splunk Enterprise in Linux https://www.nazaudy.com/install-splunk-enterprise-in-linux

 London, 16 August 2019