When SRE becomes firefighting: A reality check on reliability

SRE teams are stuck in firefighting due to rising complexity and toil. Digital SRE uses AI and automation to reduce operational burden and shift focus back to proactive reliability engineering.
5 min read
Poonam Sharma
Poonam Sharma
Senior Product Manager, Hybrid Cloud Business Unit, HCLTech
5 min read
When SRE becomes firefighting: A reality check on reliability

Modern distributed systems generate massive volumes of operational signals—metrics, logs, traces and events. Yet despite increased observability, reliability challenges continue to grow. According to the 2024 Observability Forecast by New Relic, organizations experience a median of 77 hours of downtime annually, with high-impact outages costing up to USD 1.9 million per hour.

At the same time, operations teams are spending a significant portion of their effort responding to disruptions rather than engineering long-term reliability. The same study found that engineering teams spend nearly 30% of their time addressing incidents and operational disruptions.

The persistent operational burden on SRE teams

Even within Site Reliability Engineering (SRE) practices—designed specifically to reduce operational burden—manual work remains a growing challenge. Recent SRE industry reports indicate that as much as 30% of SRE effort is still spent on operational toil: repetitive work that adds little long-term engineering value.

SRE was originally designed to improve reliability and resilience through engineering—not to constantly manage operational firefighting. However, as systems have become more complex, SRE teams often spend a large portion of their time managing alerts, responding to incidents and handling repetitive operational tasks.

This raises a fundamental question:

If SRE was meant to engineer reliability, how do we help SREs return to that original purpose?

One emerging answer is the application of AI within SRE practices—often referred to as Digital SRE.

What is SRE?

As modern applications evolved—from monoliths to distributed, cloud native architectures—the complexity of running reliable systems increased exponentially. Traditional operations models, built around manual monitoring and reactive firefighting, could not scale.

emerged to address this challenge by applying software engineering principles to operations. The goal was clear:

Build systems that are reliable, scalable and resilient—by design, not by heroics.

At its core, SRE focuses on:

  • Reliability and availability of services
  • Defining and tracking SLIs, SLOs and error budgets
  • Incident management and postmortems
  • Automation and standardization of operational work
  • Continuous improvement of system resilience

In theory, SREs are engineers first—individuals who write code, design systems and improve reliability through engineering rigor rather than repetitive manual effort.

The reality check: Where SRE time actually goes

One of the defined responsibilities within SRE practices is toil reduction. Toil refers to repetitive, manual operational work that:

  • Scales linearly with system growth
  • Adds little long-term value
  • Distracts engineers from strategic reliability engineering

Ironically, this is where many SRE teams find themselves stuck.

Instead of spending time on system design, resilience patterns, or innovation, SREs often spend a disproportionate amount of their time:

  • Tuning alerts
  • Investigating recurring incidents
  • Manually correlating logs, metrics and events
  • Executing repetitive remediation steps
  • Fighting operational noise rather than engineering it away

The intent of SRE was to eliminate toil. The reality is that SREs often end up fighting it every day.

Where the cracks start to show at scale

This creates a fundamental paradox:

The very engineers meant to engineer reliability are consumed by the operational burden of keeping systems afloat.

As environments grow larger and more dynamic, this problem compounds. Infrastructure capacity is a clear example. While SREs may not formally own capacity planning, they are often the ones dealing with the consequences when capacity decisions fall short.

Under-provisioned systems manifest as latency spikes, increased error rates and outages—directly impacting service reliability and SLOs. Over-provisioning, on the other hand, reflects inefficiencies that may not surface immediately but signal deeper gaps in operational intelligence.

In practice, SREs are frequently pulled into reactive firefighting—manually scaling systems, diagnosing capacity-related incidents and mitigating performance degradation—despite not being responsible for the original capacity assumptions. This persistent operational drag further shifts SRE effort away from engineering-driven reliability improvements toward short-term remediation.

Digital SRE as the enabler, not the replacement

The recurring pattern across these challenges is not a lack of skilled SREs—it is the growing gap between system complexity and human capacity to manage it at scale. As environments become more distributed, dynamic, and data-rich, expecting SRE teams to manually interpret signals, correlate events and respond in real time becomes increasingly unsustainable.

This is where Digital SRE begins to take shape.

Digital SRE is not a new role and not a replacement for human SREs. Instead, it represents an evolution of SRE practices—where data, automation and AI-driven intelligence are embedded into operational workflows to assist decision-making, reduce manual effort and minimize reactive firefighting.

By automating repetitive operational tasks, correlating signals across observability data and providing predictive insights into reliability risks, Digital SRE helps SRE teams reclaim time for what they were originally meant to do: engineer resilience into systems.

From reactive operations to proactive reliability engineering

Rather than responding to incidents after reliability has already been compromised, SREs can shift toward proactive and preventive reliability engineering—supported by intelligent systems that continuously learn from operational data.

The outcome is not ‘hands-off operations’, but better-focused human expertise, applied where it matters most.

Digital SRE is best understood not as a destination, but as a direction—one that acknowledges the limits of manual operations and embraces intelligent assistance as a force multiplier for reliability. It does not eliminate the need for experienced SREs; it enables them to operate at the scale and speed modern systems demand.

The path forward

Digital SRE may still be evolving, but the shift is real. At HCLTech, this evolution is already taking shape, as SRE teams are increasingly augmented with intelligent automation, data-driven insights, AI-assisted operations and agentic Ops capabilities.

These advancements are reducing toil and shifting focus back to engineering-led reliability—enabling the transition from traditional SRE practices toward a more digital, scalable and future-ready reliability model.

Share On
DFS Digital Foundation Blogs When SRE becomes firefighting: A reality check on reliability