Reliability and observability

Operational toil

Also known as Toil in SRE

By WeavePublished 2 min read

Definition

Operational toil is recurring work needed to keep a service running that tends to be manual, tactical, automatable, and without lasting improvement. Its volume often grows with the service unless the underlying need is reduced.

Repetition is a useful clue

An engineer who manually restarts the same stalled job every morning is maintaining the current state. Fixing the failure mode or building a reliable recovery mechanism creates a lasting change.

The distinction is about the nature of the work, not whether the engineer enjoys it. A difficult cleanup can produce durable value. A simple command can still be toil when someone must keep executing it as demand grows.

Evaluate an automation candidate

For an illustrative service, account provisioning requires an engineer to copy several values into an administrative tool. The task happens repeatedly and follows clear rules.

Before automating it, check the exceptions, access requirements, and failure handling. A script that works only for the common case may shift the work into manual recovery rather than remove it. Sometimes simplifying the underlying process is more effective than reproducing every step in code.

Make hidden work visible

Operational interruptions can explain why a team's planned delivery slows despite unchanged staffing. Track the recurring work and its frequency, then ask which tasks are consuming attention without improving future operations.

Do not label every meeting, investigation, or support interaction as toil. Some require judgment or create lasting understanding. A useful inventory identifies specific repetitive tasks and the cost of addressing them, so the team can choose improvements rather than treating all non-feature work as waste.

How this relates to Weave

Weave's output and work-pattern views can inform a discussion about where engineering effort goes. Toil often includes operational activity outside pull requests, so combine code-related evidence with the team's operational records. Otherwise, the work keeping a service running may be absent from the picture.

Explore Engineering intelligence

Sources and further reading

  1. Eliminating Toil, Google SRE