Learn how Squadcast helped Hippo Video automate their on-call and incident response process

We were a small, but growing DevOps team which was resolving incidents manually. As we scaled, monitoring different microservices, assigning incidents to team members and tracking the resolution status became challenging. Thanks to Squadcast, we established a formal incident response process with escalation policies, routing & tagging rules, postmortems, and an incident dashboard. Also the flexible pricing plans made Squadcast the most suitable platform for our needs.

J. Prasanna Kumar

Cloud Architect, Hippo Video

About

Hippo Video

Industry:

Technology

Location:

India

Hippo Video is a video engagement platform for B2B sales teams. In today’s text-heavy world, personalised videos can bridge the gap with customers and create the engagement desired. As a fast-growing organisation, Hippo Video was looking for an infrastructure reliability partner that could work in tandem with their monitoring tech stack to alert on any incident that causes service outage. Uptime is a major concern for Hippo Video since their customers are reliant on videos hosted on the cloud, for engaging with their prospects and leads.

Initially when they were a small team, they were able to manage alerts manually. But as their team size grew, manually managing alerts was not feasible. The need for automation also became apparent as their systems became more complex with many microservices.

Ever since they adopted Squadcast, their on-call team’s job has been less chaotic. They now have escalation policies in place which route alerts to respective team members based on the severity and priority of incidents, a formal process to conduct effective postmortems, etc. In the next revamp of their reliability practices, Hippo Video has plans to lay the foundation of SRE best practices by creating a StatusPage for each microservice.

Challenges and Solutions

Challenges

Solutions

Lack of formal Incident Response: Prior to using Squadcast, alerts were delivered directly to their Slack channel and via emails which meant incident resolution was a manual process. This lead to a cluttered inbox and broken incident response structure.

Established Incident Response process: Hippo Video established a formal incident response process by implementing Squadcast and routing incoming alerts (from monitoring services) straight to the concerned on-call team/ engineers. This was a much needed move as the team was increasing in size and responsibility.

Critical alerts were being missed: As their team size grew and the process became more complex over a period of time, their team started missing critical alerts for configuration changes and from microservices. There was neither a record of incident history nor of resolution status.

Centralized Incident management Dashboard: Squadcast served as a centralized dashboard for managing critical alerts and reviewing incident status. This helped them refine the alerting and ticketing process for incidents.

Overpriced feature-set in other platforms: All the other Incident management platforms that Hippo Video was evaluating supported either Webhooks or just Slack / Email integration. They had to pay more for additional integrations and also had a cap on the number of alerts sent.

Cost effective pricing: With Squadcast’s flexible pricing plans, there was neither a limit on the number of alerts sent nor on the alert integration endpoints. They could monitor and set up alerts for as many services as they wanted.

Non-existence of escalation policies and postmortems: Hippo Video was looking for a platform that could give more granular control for escalating critical alerts to the right people at the right time.

Better Escalation Policies: Squadcast’s tagging and routing rules helped Hippo Video define flexible escalation policies. Based on the nature and severity of alerts, they could route specific alerts to the respective team. Also, features like auto incident timelines after major outages helped them prepare better postmortems/ incident reports.

Key Benefits

Escalations & On-call routing

Squadcast’s tagging rules helped Hippo Video define the severity of incidents and their corresponding escalation policies. This helps them route incident notifications to the relevant on-call team.

Centralized Dashboard to track Metrics

Squadcast’s feature rich dashboard proved to be a single source of truth for analyzing past incidents and for tracking resolution status. Also, they are now able to effectively measure MTTA and MTTR.

Better Postmortems

By using Squadcast’s auto-incident timeline feature, Hippo Video is able to record critical information and event timelines of major outages. This is helping them learn from past incidents and prepare for better incident response in the future.

Best features, cost-effective pricing

Squadcast’s Pro plan eliminated unnecessary costs as the ‘pay per-user’ model meant they were only paying for what they were using. Other incident management platforms seemed more expensive even for a smaller feature-set.

Key Benefits

Escalations & On-call routing

Squadcast’s tagging rules helped Hippo Video define the severity of incidents and their corresponding escalation policies. This helps them route incident notifications to the relevant on-call team.