Elevating Reliability Engineering: How Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management

Squadcast is a critical tool for our support model. We primarily use Squadcast for On-Call alerting but already see additional capabilities within our environment. We have multiple platforms that integrate into Squadcast and this consolidation allows our teams to focus on delivering effective, efficient, and responsive solutions to our clients. Squadcast has allowed our business to evolve and focus on operational excellence.

Kegan Adams

Chief Operating Officer

93%

Improvement in MTTA times by removing human intervention

93%

Improvement in MTTA times by removing human intervention

93%

Improvement in MTTA times by removing human intervention

About

Charter

Industry:

IT Services

Location:

British Columbia, Canada

Charter, an award-winning IT solution and managed services provider, was founded in 1997 in Victoria, BC, Canada. We offer a comprehensive portfolio of innovative IT solutions, managed services, project services, and consulting services. Our mission is to align people, processes, and technologies to build better organizations, enhance communication, boost operational performance, and modernize businesses. Our team of experts leverages a business architecture methodology and a human-centered design approach to drive successful digital transformations for our clients, unlocking new opportunities, generating value, and promoting growth. We provide knowledge and support that extends beyond our clients' businesses, empowering them to focus on their core operations.

Forward, Together with Charter, achieving your possible.

Task at Hand ‍

In the context of the evolving economy, Charter doesn't just collaborate with clients; they also play a pivotal role in shaping and bolstering their efforts toward success. Acknowledging the significant business impacts of downtime, Charter deployed Squadcast to support Reliability requirements for their clients' operations. While each client has unique operational aspects, the common factor among them is that operational failures can result in downtime and significant financial, reputational, and regulatory repercussions that can strain relationships and partnerships. This case study explores how Charter, in collaboration with Squadcast, successfully integrated Incident Management processes into its Reliability Engineering efforts for their Managed Services clients.

Challenges and Solutions

Challenges

Solutions

Limited Integration Capabilities: As Charter's business thrived, the company faced an increasing need for a more cohesive operational system. The existing tooling lacked seamless integration capabilities, which led to inefficient Incident Management processes.

Platform Extensibility With Squadcast: With Squadcast’s expansive native integrations, Charter is able to route alerts from Monitoring tools of their choice. Furthermore, Squadcast's Webhooks provide a seamless way to integrate and connect multiple applications, leading to marked improvements in website behavior.

Lack of Real-Time Metrics: For Charter, the need to respond to incidents promptly and effectively was crucial for client satisfaction. However, measuring the effectiveness of Incident Management activities was not only labor-intensive but also lacked real-time insights. This deficiency led to an unclear understanding of the impact of Incident Management efforts.

Real-Time Visibility into the Impact of Incident Management Via Analytics: Squadcast's Organizational Analytics offers real-time metrics that are crucial for performance evaluation, including Incident Volumes, Mean Time to Acknowledge (MTTA), Mean Time to Resolve (MTTR), Assignment Data, and trends. As a result, Charter's teams can now immediately see the impact of their Incident Management efforts, enabling more efficient and effective responses.

Manual Tracking of Incident Lifecycle: In their role as a Managed Services provider, Charter must maintain complete visibility throughout the Incident Lifecycle, from trigger to resolution. Prior to implementing Squadcast, Charter found themselves entangled in manual processes for gathering data on Incidents. This cumbersome approach made it challenging to offer clients transparent and timely insights into Incident Management efforts.

Automated Incident Lifecycle Tracking: Squadcast automatically tracks the complete lifecycle of an incident, starting from the moment it's triggered to its ultimate resolution. Every action is monitored and recorded, making it easier than ever for Charter to share comprehensive incident data with their clients. This is particularly crucial for SEV0 and SEV1, where gaining insights into resolution trends is essential for effective Incident Management.

Manual On-Call Scheduling: Resource scheduling for On-Call rotations at Charter was manually executed, making even small changes a logistical challenge. Simple yet critical tasks, such as transferring an On-Call rotation or reallocating resources, became cumbersome and relied on specialized knowledge or time-consuming knowledge transfer.

Streamlined On-Call Scheduling: Squadcast’s Schedules provide a single, unified platform for planning and coordination of On-Call resources across different shifts; any information is readily accessible and available. This eliminates the need for complicated hand-offs or knowledge transfers, making the entire scheduling process more efficient and less error-prone.

Inefficient Manual Alert Management: Prior to implementing Squadcast, Charter grappled with an inefficient and time-consuming alert management system. Alerts were manually dispatched via email by a third-party provider, who would subsequently contact the On-Call team to check if the alerts were being addressed. In high-frequency alert scenarios, such as a network link being down, the team would become overwhelmed with constant phone calls. This barrage of communication also extended to follow-up calls, disrupting the team's focus on resolving the incident at hand.

Structured Alert Routing and Acknowledgment: Post implementation of Squadcast , alerts generated through email or monitoring tools are directly integrated into the Squadcast platform. These alerts are then systematically routed to On-Call engineers through multi-channel alerting, including phone calls, SMS, and email. Once an incident is acknowledged within the platform, no further calls are required from third-party services to confirm that the issue is being addressed. This allows engineers to concentrate on resolving the incident, streamlining the entire process.

Operational Overload Due to Excessive Alert Volume: Before Squadcast was implemented, Charter faced significant operational challenges due to the massive volume of alerts generated. Each alert created a new ticket, leading to an overload that not only distracted the team but also resulted in inefficient resource allocation.

Enhanced Operation Efficiency Through Alert Consolidation: Squadcast's Deduplication Rules have transformed how Charter manages its alerts. Instead of generating independent alerts for every repeated or related incident, Squadcast consolidates these under a parent-child relationship. This results in a single, manageable Incident that can have multiple alerts associated with it. For example, in the case of an extended device outage where alerts are received at defined intervals, the team can now focus solely on resolving the issue rather than being bogged down with multiple alerts. This streamlined approach significantly reduces distractions and enhances operational efficiency.

Limited Incident Response Capabilities: Before adopting Squadcast, Charter faced limitations managing incidents via mobile devices. This rigidity prevented On-Call engineers from efficiently executing incident actions when away from their workstations.

Enhanced Flexibility With Mobile App: With the implementation of Squadcast Mobile App, Charter’s On-Call engineers can now seamlessly perform Incident actions, increasing efficiency and responsiveness. Additionally, Charter is looking to further evolve its incident operations by levergaging Squadcast’s ChatOps integrations for better Incident collaboration.

Alert Fatigue, Slow Response Times: Before Squadcast was brought into the picture, Charter lacked an effective mechanism to distinguish between actionable and non-actionable alerts. This led to "alert fatigue," which diverted engineers' focus from critical alerts and consequently prolonging overall incident response times.

Context Rich, Actionable Alerts: Upon implementing Squadcast, Charter has benefited from a more intelligent alerting routing system. Now, incidents come enriched with context, allowing for immediate assessment of their priority and required action. Tagging rules are employed to assign priorities to each incident, enabling On-Call engineers to focus on the most critical issues without the distraction of non-actionable alerts.