Icinga 2 Integration Guide

This guide describes how to integrate your Icinga 2 installation with PagerDuty using PagerDuty Agent.

Please note that you must be logged in as root to complete the installation. You might need to slightly alter these instructions depending on your exact Linux distribution and your Icinga 2 configuration.

If you are having trouble completing the installation, please contact us.

Looking for Icinga 1? Go here

In PagerDuty

1. Go to the Services menu and select Service Directory.
2. On the Service Directory page:
   • If you are creating a new service for your integration, click +New Service and follow the steps outlined, selecting this integration in step 4.
   • If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click Add a new integration.
3. Under Select the integration(s) you use to send alerts to this service search and select this integration.
4. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.
5. Find the integration in the list, copy the Integration Key and keep it in a safe place for later use.

On Your Icinga 2 Server

1. 1. Install the PagerDuty Agent. The agent receives events from Nagios Core and sends them to PagerDuty using a queue, provides logging that helps troubleshoot any problems, and automatically retries sending events if there is any connection failure (i.e. if your Nagios Core server temporarily loses connectivity). Note: If you haven’t already, we highly recommend switching to Python 3 to use the PagerDuty Agent. The Agent does not run on CentOS 5 or lower, as it requires a newer version of Python than the version included with CentOS 5. Please use the Perl-based integration for Icinga2 on older operating systems.
   2. Download the configuration template, pagerduty-icinga2.conf, from GitHub:

      wget https://raw.githubusercontent.com/PagerDuty/pdagent-integrations/master/conf.d/pagerduty-icinga2.conf

   3. Open the file with your text editor (nano, vim, emacs, etc.) and enter the integration key corresponding to your Icinga 2 service into the pager field. The integration key is a 32 character string that can be found on the integration’s detail page.
   4. Move the Icinga 2 configuration file into place:

      mv pagerduty-icinga2.conf /etc/icinga2/conf.d/

Alternatively, if you’re using Icinga’s objects.d configuration:

mv pagerduty-icinga2.conf /etc/icinga2/objects.d/

1. Add the custom attribute enable_pagerduty to your configuration’s host and service configuration objects.
2. vars.enable_pagerduty = true # Add this line to hosts/services that will send events to PagerDutyNote, the easiest way to do this is to add it to templates that are used by all of your configuration objects. For instance, on Debian-based systems, the default configuration has host objects that use the generic-host template and service objects that use the generic-service template. The templates are defined in /etc/icinga2/conf.d/templates.conf and can be modified as follows:

3. template Host "generic-host" {
    max_check_attempts = 5
    check_interval = 1m
    retry_interval = 30s
   
    check_command = "hostalive"
   
    vars.enable_pagerduty = true # Add this line
   }

   template Service "generic-service" {
    max_check_attempts = 3
    check_interval = 1m
    retry_interval = 30s
   
    vars.enable_pagerduty = true # Add this line
   }

4. If you added the custom attribute to a template in step 5, ensure that the template is used by your host and service objects. For instance, if you added it to the generic-host and generic-service templates, your objects should have one of the following lines:

     import "generic-host" # All host objects should have this line in them

     import "generic-service" # All service objects should have this line in them

5. Restart Icinga 2:

   /etc/init.d/icinga2 restart

FAQ

What if a Icinga event happens while my network is down?

If a PagerDuty server can’t be reached for any reason, events will be stored to an on-disk queue. The installed cron job will attempt to re-send the events at one minute interval.

Since Icinga needs my external Internet connection to send failure reports to PagerDuty, how will I receive notification if our site loses external connectivity?

You should configure an external ping check service like Pingdom to monitor your site’s external connectivity. Of course, you can use PagerDuty to forward alerts from these services.

It doesn’t seem to be working. What’s going on?

Check the Icinga log file for messages containing “pagerduty”. On most systems:

grep -i pagerduty /var/log/icinga2/icinga2.log

Another log to check is the syslog for messages containing “Icinga event”:

grep "Icinga event" /var/log/syslog

Please contact us if you’re unable to sort out the difficulty.

What sort of Icinga messages does PagerDuty understand?

PagerDuty can process PROBLEM, ACKNOWLEDGEMENT, and RECOVERY messages. All other messages, including FLAPPINGSTART and FLAPPINGSTOP, are ignored. If you’d like PagerDuty to process additional Icinga messages, please let us know!