Bulletproof, a major provider of cloud management and consultation services in Australia, is a company whose name and motto of “Mission Critical Cloud” proclaim a commitment to reliability. Delivering on this commitment to Bulletproof’s more than 700 customers requires careful management of the infrastructure and services that power the company’s operations. Some years ago, after struggling to handle incidents using ad-hoc processes and legacy technologies, Bulletproof adopted PagerDuty, which has enabled revolutionary improvement in their ability to maintain SLA response times and avoid critical service disruptions. This has driven increased customer satisfaction by enforcing compliance with entitlements.  For example where a Customers service is entitled to be patched on a regular basis but their last patched date is out of bounds, their automation utilizes PagerDuty to assign an Engineer to remedy.

Challenges: Inefficient Alerting and Lack of Automation

Bulletproof’s engineers manage cloud infrastructure that is spread across multiple continents. It includes both the company’s internal infrastructure and the cloud environments of customers that they support on a long-term basis. Those customers are spread across a diverse set of industries, including government, finance, manufacturing and retail. Maintaining uninterrupted service for such a large spread of critical applications across multiple cloud environments is no small task.

Initially, Bulletproof’s support team relied on a primitive notification system utilizing physical pagers to deliver alerts to its engineers when incidents occurred. The alerts were delivered by a third-party company that forwarded notifications to the pagers, which engineers had to literally carry and check regularly at all time. Reliance on a third-party messaging company added cost, latency and inefficiency to the alert system, which offered no support for integrations with other tools like chat platforms, or automation of tedious tasks such as on-call scheduling. These processes were instead performed manually, often on an ad-hoc basis.

Improving Engineers’ Happiness and Response Time with PagerDuty

Five years ago, Bulletproof decommissioned its physical pager alerting system in favor of PagerDuty and saw immediate results.

One significant benefit was PagerDuty’s scheduling automation features, which provide “the clarity to know whether someone is on task and on schedule,” said Greg Cockburn – Chief Cloud Officer at Bulletproof. This scheduling visibility enabled a better life-work balance for Bulletproof’s staff because it made it easy to identify on-call engineers during an incident and avoid disrupting off-duty employees. “A better on-call system has been a significant improvement for our team’s happiness, and PagerDuty has been a big part of that story,” Greg said. He noted as well that PagerDuty helps to automate incident escalation and handoffs between engineers, further improving the efficiency of workflows and reducing the amount of manual coordination required from engineers.

“We’ve been using PagerDuty for five years. We have had tens of thousands of incidents. We’ve never encountered a problem with the PagerDuty service. It has always just worked!”

– Greg Cockburn, Chief Cloud Officer, Bulletproof

With the help of PagerDuty, Bulletproof also significantly improved its response time when handling incidents — an essential benefit for a company whose ability to maintain service agreements with customers is a crucial part of its value. “One of our key SLA targets to our customers is human response time. PagerDuty is the foundation for us to provide that,” said Greg.

Most important of all, PagerDuty has delivered a bullet-proof incident management solution for Bulletproof’s engineers. As Greg explained, “we’ve been using PagerDuty for five years. We have had tens of thousands of incidents. We’ve never encountered a problem with the PagerDuty service. It has always just worked.’”

Bulletproof already leverages a range of PagerDuty features, from automated alert grouping to SLA reporting, to the ChatOps integration with HipChat. Going forward, however, with offerings like DevOps Support the company plans to take even greater advantage of the functionality that PagerDuty offers. “PagerDuty is a next-generation platform, with constant evolving features that we didn’t even know existed,” Greg says.