Check_MK Werk 8275: Alert Handlers - execute automatic actions upon state changes of hosts and services

ID: 8275 Title: Alert Handlers - execute automatic actions upon state changes of hosts and services Component: cmc Level: 3 Class: New feature Version: 1.2.7i2 Check_MK now supports automatic actions (scripts) to be executed upon the state change of a host or service. This is similar to Nagios "Event Handlers" but has a much more flexible configuration and other advantages. At the beginning there is a state change of a host or service. It does not matter whether this change is "soft" - because the maximum number of check attempts has not been reached. It simply matters that the state has changed from one of OK/WARN/CRIT/UNKNOWN to another. Whenever this happens a new global rule chain of <i>Alert Handler Rules</i> is being processed. Each rule that matches calls an external script of your choice. Most times people want to restart services, trigger garbage collections of Java machines or do similar stuff. Please note that some folks insist that monitoring should not try to repair things or by any other means actively <i>change</i> things. Whether you share this opinion or not is your own decision. Alert handlers do not limit you in what you exactly do with them. But you have been warned. When you compare alert handlers with the Rule Based Notifications (RBN) then here are some important differences: LI:Notifications are being suppressed during downtimes, outside of the notification period, when the host is down and other situations. Alerts are never suppressed. LI:Notifications cannot be triggered at soft state changes. LI:Alert handlers only work with the Check_MK Micro Core. If you need Nagios or Icinga please use the Event Handlers of those cores. LI:Alert handling rules do not allow cancelling. All matching rules are being executed. LI:As long as no alert rule is defined the alert handling mechanism is deactivated in the core. Note: this is just the first implementation of the Alert Handlers. Next steps will the introduction of error tracking, notifications tied to alert actions, even more flexible conditions, a system for secure remote execution and much more. Stay tuned! H2:Setting up Alert handlers For setting up alert handlers you first need to create a script that should be called. This can be written in any programming language - most people will use a simple BASH script. It must be executable and be installed in <tt>~/local/share/check_mk/alert_handlers</tt> and made executable. The script is provided with all information about the alert with environment variable that being with <tt>ALERT_</tt> - very similar to a notification script. A good start for testing is the following script: F+:/local/share/check_mk/alert_handlers/foo #!/bin/bash env | grep ALERT_ | sort > /tmp/alert.out F-: This will dump all the variable of the alert into the file <tt>/tmp/alert.out</tt>. When specifying this script in the alert handler rule - simply write <tt>foo</tt> here. Useful for debugging is to set <i>Alert handler log level</i> to <i>Full dump of all variables</i> and <i>Logging of the alert processing</i> to <i>on</i> in the global settings. You will find information it <tt>~/var/log/cmc.log</tt> and <tt>~/var/log/alerts.log</tt>.