Custom Checks

Send any data as check results.

Preface

Custom checks allow you to gather data from any application, device or system and use this data as check results. Based on your custom checks you can set up rules for alerting.

Add a custom check

Using the web interface, select a host and click on the plus sign of "Custom Check".

Enter a name of the check. The name must not contain white space or special characters. This name is the common prefix of all check results you'll send later. Let's say you want to monitor the success of your backups. So backup is a perfect name for your custom check.

Optionally enter a frequency - minutes or hours - you expect the custom check to send new data. If you enter an expected update interval alerting is automatically enabled if your custom check stops sending data within the expected period. It's highly recommended to use the option. Otherwise your check script may fail completely and you wont notice it. If you perform backups once a day an expected update interval of 25 hours is perfect. Your check execution and our check intervals cannot be synchronised to a second. So put some tolerance.

Custom checks must be created on a host level. Creating global custom checks for a group of hosts or all hosts is not yet possible.

Add a custom check

Sending data using csender

Starting with cagent version 1.3.0 a handy command line utility called csender has been introduced. With csender you can send your custom check data with ease. Csender is included in all versions of cagent (Windows and Linux). Using the installer packages (DEB, RPM, MSI) the csender or csender.exe binary is in your path. You can execute it without specifying the full path.

Examples

Create a custom check called backup and copy the token to your clipboard.

A custom check called "backup"

Now execute csender to submit any data

csender -t GXXXXOBN \
-n backup \
-s 1 \
-a "This text triggers an alert. Optional" \
-w "This text triggers a warning. Optional" \
any_number=1 \
any_float=0.1245 \
any_string="Put your check result here"

This example shows a single command spread over multiple lines. On Linux the backslash \ is used at the end of the line. On the Powershell use a backtick ` On the cmd.exe prompt use the ^ sign.

Specifying -t <TOKEN> and -n <NAME OF CHECK> is mandatory. The rest is optional. -a is for sending an optional alert message. It triggers an immediate alert and will be sent to all recipients. If you omit the alert message all previous alerts will be recovered. -w is for sending an optional warning message. It triggers an immediate warning and will be sent to all recipients. If you omit the warning message all previous alerts will be recovered. -s is for specifying the success of your custom check as number, where 1 is considered success and 0 as failure. Sending 0 triggers an alert that is sent to all recipients.

You can append any key-value-pairs separated by an equal sign.

The order of command line arguments is important. Always put parameters likes -a -n -w -s first and the list of metrics last.

Sending data using a script

To send results you need to know the name of your custom check, and the unique token of it. You can use any http client to send json-encoded check results. You can send up to 12 key-value-pairs for each custom check.

The data must be sent using HTTP POST to the URL https://hub.cloudradar.io/cct/.

All keys must be prefixed with the name of your custom check, for example backup.success. If the key prefix doesn't belong to the custom check identified by the token, the request is rejected.

Sending data for multiple custom checks within a single request is not allowed, as you can use only one token and only one key prefix in a single request.

Your are free to use any keys as long as they don't contain whitespace or special characters. The key length is limited to 100 bytes and the value is limited to 500 bytes. If you exceed this limit with one of the key-value-pairs the entire request is rejected and your check results are not stored.

Using Curl

A simple example.

curl -X POST \
https://hub.cloudradar.io/cct/ \
-H 'X-CustomCheck-Token: uGXgXr2Zy' \
-d '{
"backup.success": 1,
"backup.data_sent_B": 256791,
"backup.num_folders_in_backup": 345,
"backup.target" : "backup.example.com",
"backup.transport": "rsync",
"backup.duration_s": 2569
}'

Injecting variables using a bash script requires proper usage of quotation marks. The above example doesn't allow variables because the json payload is inside singe quotation marks and bash treats it literally. Use double quotation marks and escape them with a backslash.

#!/bin/bash
#
# How to inject variables into a curl request
#
ANY_NUMBER=$RANDOM
ANY_FLOAT=${RANDOM}.${RANDOM}
curl -X POST \
https://hub.cloudradar.io/cct/ \
-H 'X-CustomCheck-Token: uGXgXr2Zy' \
-d "{
\"asterisk.success\":1,
\"asterisk.any_string\":\"Put your check result here as text\",
\"asterisk.any_number\": ${ANY_NUMBER},
\"asterisk.calls_per_day\": ${ANY_NUMBER},
\"asterisk.average_calls_per_hours\": ${ANY_FLOAT},
\"asterisk.alert\": \"This text triggers an alert. Optional.\",
\"asterisk.warning\": \"This text triggers a warning. Optional\"
}"

If numbers or floats should be treated as such, do not put them inside quotation marks. Otherwise the values are converted to strings.

A python example

custom-check.py
#!/usr/bin/env python
import requests
import json
import sys
token = '1234abc'
url = "https://hub.cloudradar.io/cct/"
payload = {
"backup.success": 1,
"backup.data_sent_B": 256791,
"backup.num_folders_in_backup": 345,
"backup.target" : "backup.example.com",
"backup.transport": "rsync",
"backup.duration_s": 2569
}
headers = { 'X-CustomCheck-Token': token }
response = requests.request("POST", url, data=json.dumps(payload), headers=headers)
if response.status_code != 204:
sys.stderr.write("Sending data failed. " + response.text + "\n" )
sys.exit(response.status_code)
print("All data sent.")
Review the data sent by a custom check

Trigger events (alerts or warnings) from custom checks

Using csender to trigger events

Triggering events (alerts or warnings) is easy and it doesn't require a complex rule set up.

With csender you can use the -a or -w option to trigger an alert or warning. For example csender -u -t <TOKEN> -n filecounter -s 1 -a "Limit reached" counter=11.

Alert event triggered by a custom check

To recover an event (alert or warning) just send fresh data without the -a or -w option or send 0 or null as alert or warning. For example csender -t <TOKEN> -n filecounter -s 1 -a 0 counter=11.

Using a script or curl to trigger events

If you want to trigger an event just include json keys ending in .alert or .warning for example

curl -X POST \
https://hub.cloudradar.io/cct/ \
-H 'X-CustomCheck-Token: <TOKEN>' \
-d '{
"filecounter.success": 1,
"filecounter.counter": 256791,
"filecounter.alert": "This is an alert"
}'

To recover an event just send fresh data without the .alert or .warning key or use null as value, for example

curl -X POST \
https://hub.cloudradar.info/cct/ \
-H 'X-CustomCheck-Token: 245JkJmM0F' \
-d '{
"filecounter.success": 1,
"filecounter.counter": 256791,
"filecounter.alert": null
}'
Example of an alert email triggered by a custom check.

Fine tune alerts or warning for custom checks

Some pre-defined rules are responsible for triggering event. You will find the in the Rules section. You are free to disable or edit them.

Change the default rules of custom checks

Examples

Python: How to monitor the size of some files

filemonitoring.py
#!/usr/bin/env python
import requests
import json
import sys
import os
#
# Example how to monitor the size of some files
# and how to trigger an alert or warning
#
# List of files you want to supervise
files = ['/etc/hosts','/etc/passwd','/etc/issue','/tmp/foo']
# Threshold in bytes. If exceeded, an alert is triggered.
threshold = 10000
# Your token of the custom check
token = 'xxxxxxx'
# Name of the custom check. Needs to be created on my.cloduradar.io first!
check = 'files'
# URL of cloudradars data hub. No need to change.
url = "https://hub.cloudradar.io/cct/"
payload = dict()
alerts = []
warnings = []
for file in files:
try:
# Read the file size in bytes
file_size = os.stat(file).st_size
if file_size > threshold:
alerts.append("%s exceedes size of %s" % (file,threshold))
except OSError:
# Set file size to -1 if the file does not exist
file_size = -1
warnings.append("%s file not found" % file)
payload[check+'.'+file] = int(file_size)
if len(alerts) > 0:
# Include alerts that will trigger a message on cloudradar
payload[check+'.alert'] = ",".join(alerts)
if len(warnings) > 0:
# Include warnings that will trigger a message on cloudradar
payload[check+'.warning'] = ",".join(warnings)
headers = { 'X-CustomCheck-Token': token }
response = requests.request("POST", url, data=json.dumps(payload), headers=headers)
if response.status_code != 204:
sys.stderr.write("Sending data failed. " + response.text + "\n" )
sys.exit(response.status_code)
print("All data sent.")

Windows: Monitor a spooldir for old files

The example uses a free third party tool called Swiss File Knife, a very handy command to search, filter and list files. Download sfk.exe and place it in the path.

In this example we use C:\spooldir as the directory to be monitored and we use C:\tmp as a folder to store temporary data. Create both directory before running the check.

The basic job of finding file older than a given time is done by sfk.exe with

sfk times -nofolow -pure -before 2m C:\spooldir *.*
Finding file older than 2 minutes with sfk

As we are only interested in the total number of files instead of their name we use windows build-in command find to count the result.

sfk times -nofolow -pure -before 2m C:\spooldir *.*|find /C "."
Counting lines with find

Test the procedure by using -before 999d to list files older than 999 days. It should print 0.

Next step is to have the result as a variable for further usage in a script. The windows command line cannot directly populate a variable with the output of a command. It must be written to a temporary file and read from it.

sfk times -nofolow -pure -before 2m C:\spooldir *.* | find /C "." > C:\tmp\num_files.txt
set /p NUM_FILES=<C:\tmp\num_files.txt

Now you can use the variable %NUM_FILES% inside your console or inside a script. Next csender comes into play to submit the collected data to CloudRadar. Create a custom check called oldfiles and copy the generated token to your clipboard.

Now you are able to submit the number of files older than two minutes using csender.exe.

csender -t <TOKEN> -n oldfiles files=%NUM_FILES%
reviewing the data on my.cloudradar.io

Run a scheduled task every 5 minutes

Save all the steps in a script called old_files.bat and execute it to be sure fresh data arrives at cloudradar.

C:\tmp\old_files.bat
echo @off
REM Count the number of files older than 2 minutes in the sppol dir
REM and write the results to a temporary file
sfk times -nofolow -pure -before 2m C:\spooldir *.* | find /C "." > C:\tmp\num_files.txt
REM Load the resulst from the temporary file into a environment variable
set /p NUM_FILES=<C:\tmp\num_files.txt
REM Transmit the results to cloudradar using csender
REM Requires the previous setup of a custom check
csender -t <TOKEN> -n oldfiles files=%NUM_FILES%

Don't forget to insert your personal token that has been created for that specific hosts. Don't use the same token on different hosts.

Script with all three steps

Create a new scheduled task executes the script every 5 minutes. Make sure it doesn't require a user to be logged on. The script should run indefinitely and directly after the schedules task has been created.

Create a schedules task
let it run every 5 minutes

Create a rule for alerting

The last and final step is creating a rules that fires and alert if there are some files (>0) older than 2 minutes in the spooldir. Use the wand icon at the right side.

Creating an alert rule