Using PagerDuty for Aquaponics: Upward Farms Implementation Guide
Upward Farms’ mission is to radically elevate the world’s fresh food supply by setting new standards for safety, quality, and cost. The company operates complete ecosystems of plants and fish in vertical farms, supplying these organic and local products to grocers. Their first products, washed and ready-to-eat packaged salads, are sold in New York City by Whole Foods Market.
The team at Upward Farms employs a highly innovative solution to grow produce and fish together in what’s called an aquaponic system, where fish are sustainably farmed and their manure is used to fertilize plants, all within a controlled environment. They use the PagerDuty Events API to connect a variety of equipment with the team to keep them aware of the health of their system. The team watches everything from the dissolved oxygen levels in the fish tanks to the air temperature in the produce-growing areas. It’s a delicate system that requires vigilant observation and balance. For instance, if the dissolved oxygen levels in the fish tanks fall too low, the fish could die within the hour.
This implementation guide will highlight some of the solutions that the Upward Farms team created to keep tabs on their aquaponic system. These unique use cases can be defined in three pipelines as described below. Each of these pipelines consumes alerts from different types of sensors and systems and then utilizes PagerDuty’s Events API to enqueue events. These pipelines include:
- Sensor readings → InfluxDb → Kapacitor → Custom Kapacitor alarm scripts → PagerDuty
- Email-based alarm sources → Mailgun (parsing email into JSON) → AWS Lambda → A custom Python script → PagerDuty
- Sensor hardware logging → MQTT → Fluentd + PagerDuty output plugin + custom configuration → PagerDuty
Pipeline 1
The first pipeline includes gathering environmental metrics in InfluxDB, which is an open-source time series database optimized for fast, high-availability storage of time series data such as sensor readings. The types of measurements that they gather include ambient air quality and chemistry, such as air temperature, relative humidity, and carbon dioxide; water chemistry, such as pH, dissolved oxygen, water temperature, salinity, and nitrates (NO3-); and equipment that monitor things like electrical currents and pressure in plumbing.
Some raw measurements are combined or processed further to create the metrics that actually trigger alarms. For instance, relative humidity is not an especially useful measurement. However, combined with air temperature and some math, it becomes Vapor Pressure Deficit, a useful measurement in botany applications. Similarly, the electrical current measurements are not compared against absolute thresholds. Instead, for example, the team at Upward Farms processes the patterns of equipment switching on and off (low current draw to high) to determine cycle times.
Various environmental measurements have different meanings in different contexts and require distinct alarms. The growing space air temperature is warm while product cold storage is fully refrigerated. Salmon tank water temperature is quite chilly compared to bass tank water temperature. Each of these relies on different monitoring for alarm conditions.
Data is transformed and inspected using Kapacitor, a data processing engine for InfluxDB, acting on the data in real time using its programming language TICKscript. Upward Farms wrote custom TICKscripts to monitor their environmental data streams and push alarm data into PagerDuty.
Something to note here is that, as of the time of this writing, InfluxData (the company behind InfluxDB and Kapacitor) is transitioning to their 2.0 product. The concepts are the same, but the tool stack is different. TICKScripts are subsumed by a much more capable functional data handling programming language named Flux.
Pipeline 2
The second scenario that Upward Farms addressed transformed email-based alarms from equipment into actionable incidents. This is a common alarm handling flow for PagerDuty users. In the case of Upward Farms, their equipment produced alarm emails all with the unhelpful subject line of “Alarm Message.” They utilized email parsing service Mailgun to extract and manipulate information from the body of the alarm emails.
Upward Farms routes their alarm emails to Mailgun where the message is parsed into a JSON payload and then forwarded to an AWS Lambda function. That function is a Python script that matches the Mailgun JSON payload to the appropriate PagerDuty service, creates an alarm message with the payload details, and triggers a usable incident using the PagerDuty Events API.
Pipeline 3
Upward Farms’ custom embedded sensor hardware gathers internal logs in addition to sensor readings. Both sets of data are passed over MQ Telemetry Transport, or MQTT, to Upward Farms’ system. Sensor hardware logging statements are processed with Fluentd as an MQTT subscriber. Fluentd is an open-source data collecting tool for unifying the logging layer of an application. With Fluentd, they parsed the JSON payloads of the log messages and used a Fluentd plugin to trigger incidents with the PagerDuty API upon Error or Exception log messages.
While a PagerDuty integration for Fluentd already existed, at the time it was too simplistic for Upward Farms’ needs. So the team contributed additions to that plugin so that it was fully compatible with PagerDuty’s Common Event Format.
These are just a few examples of what Upward Farms have done with a little bit of code and a lot of imagination. To get more information about building integrations and applications on the PagerDuty Developer Platform, check out the PagerDuty Developer site or post questions over in the PagerDuty Community forum.