A flaky test can be a useful signal, but once it starts failing intermittently in CI it can become a governance problem. Teams often respond by quarantining it, which can reduce alert fatigue and keep pipelines moving. That helps, until quarantine becomes a dumping ground for real failures, and the organization quietly accepts degraded confidence as normal.

A good flaky test quarantine policy is not a workaround for weak automation, it is a control system. It defines when a test is allowed to stop blocking, who owns the triage, what evidence is needed to quarantine or restore it, and how long the team can tolerate instability before it becomes a product risk. The goal is not to make CI green at any cost. The goal is to preserve release confidence while keeping false noise from overwhelming engineers.

Quarantine is a temporary risk-management decision, not a permanent status for inconvenient tests.

What flaky test quarantine is, and what it is not

In software testing, flakiness usually means a test alternates between passing and failing without a corresponding change in the code under test. The root cause might be timing, environmental dependence, test data collisions, clock sensitivity, order dependence, network instability, or genuine product nondeterminism. The broader testing context matters here, because not every intermittent failure has the same meaning. The same failing test may indicate a brittle assertion, an unstable dependency, or an actual defect in the system.

A quarantine policy should separate three cases:

  1. A test is flaky but still valuable, so it should be tracked, isolated, and repaired.
  2. A test is failing because the product is broken, so quarantining it would hide risk.
  3. A test is no longer relevant, so the right action is deletion or redesign, not quarantine.

This distinction is important because quarantine can create a false sense of control. If teams only classify failures by whether they block the pipeline, they lose the ability to distinguish signal from noise. That is a failure mode in many CI environments, especially when the team uses green build status as a proxy for quality.

For background on the underlying discipline, see software testing, test automation, and continuous integration.

Why teams quarantine tests in the first place

Most teams do it for the same practical reasons:

  • A flaky test produces repeated false alarms.
  • Engineers start ignoring failed pipelines.
  • Release managers lose confidence in the build signal.
  • The time cost of manual reruns becomes larger than the remaining value of the test.

Those are real pressures. A failing pipeline can block merges, delay releases, and consume a lot of unplanned triage time. But reducing the friction of false failures is only half the problem. If the team cannot tell the difference between a flaky test and a regression, then quarantining blindly reduces noise while increasing blind spots.

A good policy starts with a simple question: what does this test protect, and how much risk are we willing to defer? If the answer is “this catches an important user journey,” the bar for quarantine should be much higher than for a low-value edge case. If the answer is “this test verifies a third-party integration that is already outside our control,” the policy might permit temporary quarantine with a strict revalidation rule.

Define the quarantine policy around risk, not convenience

A quarantine policy should be written as a decision framework, not a slogan. The core fields are straightforward:

1. Eligibility criteria

Specify when a test is allowed to enter quarantine. Examples:

  • It fails intermittently across at least two independent runs.
  • The failure reproduces in a clean environment or in a controlled rerun, but not consistently.
  • The failure is attributable to known timing, dependency, or infrastructure variance.
  • The test owner can explain why the failure is not currently believed to be a product regression.

Avoid allowing quarantine for “anything annoying.” If the criterion is too broad, the team will move uncertain failures out of the way instead of investigating them.

2. Required evidence

Require enough information to make quarantine a disciplined choice:

  • Links to CI runs showing pass/fail variation.
  • Error messages and stack traces.
  • Environment details, browser or runtime versions, and recent dependency changes.
  • Notes on reproduction attempts.
  • The suspected failure mode, if known.

This evidence matters because a flaky symptom is not a root cause. A policy should enforce the difference between an observation and a diagnosis.

3. Approval authority

Decide who can quarantine a test. Common patterns include:

  • Test owner approves quarantine for low-risk tests.
  • QA lead or release manager approves quarantine for high-signal tests.
  • Platform or CI engineering must approve anything involving shared infrastructure.

The key is to avoid invisible quarantine. If any engineer can suppress a test without review, the pipeline becomes less trustworthy over time.

4. Expiration date

Every quarantine entry should expire automatically. That expiration date forces re-evaluation and prevents permanent drift. A sensible default is a short time window measured in days or one or two sprint cycles, depending on team cadence. The exact number matters less than the existence of a deadline.

5. Exit criteria

Define how a test leaves quarantine:

  • Fixed and verified across a minimum number of clean runs.
  • Rewritten to remove the unstable dependency.
  • Retired because coverage is duplicated or the product path is obsolete.

A quarantine that has no exit criteria is not a policy, it is storage.

Separate ownership from blame

One of the biggest governance failures is unclear ownership. A flaky test often sits at the intersection of product code, test code, test data, infrastructure, and deployment topology. If ownership is not explicit, triage becomes a game of passing tickets between teams.

A workable model is:

  • Test owner: responsible for the assertion, the scenario, and the corrective action.
  • Service or feature owner: responsible when the failure indicates a product change or a broken contract.
  • Platform owner: responsible when the cause is shared infrastructure, browser farm stability, container images, timeouts, DNS, or CI agents.
  • Release manager or QA lead: responsible for deciding whether the system can ship with a quarantined test, based on the risk profile.

This is less about bureaucratic hierarchy and more about making triage actionable. If one person owns the next step, the test does not remain in limbo. The policy should also say how ownership is assigned for cross-cutting systems, because many flakes happen precisely where systems intersect.

If nobody is accountable for deciding what happens next, quarantine will accumulate faster than repair work.

Choose which failures can block release, and which cannot

Not all flaky tests should have the same operational treatment. A practical policy often divides tests into categories:

Blocking tests

These are the tests whose failures should stop a release unless there is explicit override. They usually cover:

  • Critical user paths
  • Security-sensitive flows
  • Payment, authentication, data integrity, or compliance-related behavior
  • Areas with high business impact and low observability in production

If a blocking test flakes, quarantine should be rare and carefully reviewed. In some organizations, the safer approach is to keep it blocking and prioritize fix work immediately.

Non-blocking but visible tests

These tests still matter, but they should not stop the pipeline on every intermittent failure. They can be marked as quarantined, skipped, or informational, while remaining visible in dashboards and release reviews.

Diagnostic tests

These are tests that help with observability, regression hunting, or exploratory checks. They can be valuable even if they are not suitable as release gates.

This separation is useful because the right response to flakiness depends on the test’s purpose. A test that is not meant to gate release should not be treated as though it were.

Make quarantine a process, not a label

The test quarantine process should be explicit and repeatable. A practical flow looks like this:

  1. A failing test is detected in CI.
  2. The failure is reproduced or at least observed across multiple runs.
  3. The owner classifies the likely cause and decides whether quarantine is justified.
  4. If quarantined, the test is tagged, excluded from blocking status, and linked to a ticket.
  5. The ticket carries an expiry date and a required follow-up action.
  6. The test is re-run regularly, ideally under known-good conditions, to check whether the issue still exists.
  7. Once fixed, the test must demonstrate stable passes before being restored to blocking status.

This flow works better than simply adding the test to a skip list. The policy should make quarantine visible in the same places engineers already look, such as CI results, release dashboards, and defect trackers.

A practical implementation often includes a metadata field or tag such as quarantined, blocked-by-flake, or needs-triage, plus a separate list of owners and expiration dates. If the team uses test automation frameworks or a test case management system, the quarantine state should be stored in a durable place rather than in a local spreadsheet.

Use CI signals carefully, not mechanically

CI is useful because it turns intermittent failure into a measurable pattern, but CI noise reduction can become dangerous if the team over-optimizes for build pass rate.

A green build after quarantine does not necessarily mean the system is healthier. It may only mean that one class of failures is no longer visible in the main gate. That is why the quarantine policy should track both the mainline status and the quarantined pool.

Recommended reporting dimensions include:

  • Number of quarantined tests
  • Age of each quarantine entry
  • Time since last successful execution
  • Ownership status
  • Frequency of flake recurrence
  • Whether the failure clusters around a specific service, browser, test data set, or branch pattern

The point is not to create vanity metrics. The point is to ask whether instability is shrinking, moving, or hiding behind a cleaner dashboard.

Decide how long to tolerate instability

There is no universal quarantine duration, because the right tolerance depends on risk, release cadence, and the ability to detect problems elsewhere. Still, the policy should answer two questions:

  1. How long can a test remain quarantined before it must be re-reviewed?
  2. How many quarantined tests can exist before the pipeline signal is considered degraded?

The second question is especially important. A team with a dozen quarantined tests covering critical workflows has a different risk posture than a team with a few quarantined smoke checks. If the quarantine list keeps growing, the policy has failed even if releases stay unblocked.

A common failure mode is turning quarantine into a parking lot for unresolved work. The limit should be low enough to force prioritization. If there is no pressure to reduce the list, it will expand until the release process no longer reflects reality.

Build the policy around root-cause patterns

It helps to classify flakes by failure mode, because the remediation differs.

Timing and synchronization

Typical causes include sleeping instead of waiting, asserting too early, animations, debounced events, or data propagation delays. These often point to brittle automation rather than product problems. The remedy is usually better waits, stronger selectors, or a more stable test design.

Shared data and order dependence

Tests that depend on mutable shared data, reused accounts, or uncontrolled ordering can fail intermittently. The fix is usually isolation, namespaced test data, or environment reset discipline.

Environment and infrastructure instability

Browser crashes, container resource limits, network blips, DNS issues, and third-party service outages can all present as flakiness. In those cases, the quarantine decision may be justified, but only alongside infrastructure remediation.

Product nondeterminism

Sometimes the product truly behaves inconsistently, perhaps due to race conditions, eventual consistency, or asynchronous backend processing. Quarantining such tests without fixing the underlying behavior can conceal a real defect.

This classification helps the team avoid a lazy assumption that every flaky test is a test problem. It might be a product problem, and if so, quarantining it simply delays the corrective work.

Add guardrails so quarantine does not become a trash bin

A quarantine policy needs explicit anti-abuse rules.

Guardrail 1, no indefinite quarantine

Every entry expires and requires renewal with evidence.

Guardrail 2, no silent skipping

Quarantined tests should still appear in reporting and ownership lists.

Guardrail 3, no quarantine for known regressions without a release decision

If the failure is a real product regression, the release manager should make a risk decision, not hide the issue behind a skip.

Guardrail 4, no quarantine without a ticket

If there is no follow-up work item, there is no durable ownership.

Guardrail 5, no quarantine without tracking recurrence

If the test starts failing more often, the policy should escalate it, not normalize it.

A policy with guardrails is easier to defend in executive review because it shows that the organization is managing risk rather than suppressing information.

A short GitHub Actions example for visible quarantine tracking

One simple pattern is to separate blocking and quarantined test jobs, while preserving visibility in the pipeline.

name: test

on: [push, pull_request]

jobs: blocking-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npm test – –grep “@blocking”

quarantined-tests: runs-on: ubuntu-latest continue-on-error: true steps: - uses: actions/checkout@v4 - run: npm ci - run: npm test – –grep “@quarantined”

This is not a complete policy by itself. It only demonstrates a useful principle, quarantined tests should remain visible as a separate signal. If the quarantined job fails every day, the team should see that pattern and act on it.

How to restore a quarantined test safely

Restoration deserves as much discipline as quarantine. A test should not return to blocking status the first time it passes after a fix, especially if the original cause was nondeterministic.

A safer approach is:

  • Require repeated passes in the same environment where the flake appeared.
  • Confirm that the fix addresses the suspected failure mode, not just the symptom.
  • Check for collateral damage, for example, a timeout increase that masks a slower regression.
  • Re-enable the test in the blocking set only after the owner signs off.

This matters because a flaky test that has not been really fixed can return later and erode trust in the process.

What to report to engineering leadership

Engineering managers and directors do not need a wall of pipeline detail, but they do need a clear view of quality risk. Reporting should answer questions such as:

  • How many tests are quarantined now?
  • Are the same tests failing repeatedly?
  • Which teams or services generate the most quarantined tests?
  • Are any quarantined tests covering critical customer journeys?
  • Is the quarantine backlog shrinking or growing?

A useful report focuses on trend and ownership, not just raw counts. A stable number of quarantined tests may still be unhealthy if the same critical ones remain unresolved for months.

Leadership should also ask whether the quarantine rate signals deeper design problems. If a specific component repeatedly causes flakes, the team may need better contract tests, more deterministic test data, or a test pyramid adjustment rather than more quarantine capacity.

A pragmatic policy template

If you need a starting point, use this structure:

  • Purpose: Reduce false CI noise without hiding important regressions.
  • Eligible tests: Intermittently failing tests with evidence of nondeterminism.
  • Exclusions: Known product regressions, deprecated tests, and tests without ownership.
  • Approval: Test owner plus QA or release lead for critical paths.
  • Expiration: Mandatory review after a fixed period.
  • Tracking: Ticket, owner, last failure date, suspected cause, next action.
  • Exit criteria: Stable passes after fix, or deletion if the test is obsolete.
  • Reporting: Quarantine count, age, recurrence, and risk coverage.

Keep the policy short enough that teams can actually use it, but specific enough that it constrains behavior.

Final judgment: quarantine should reduce noise, not judgment

A flaky test quarantine policy is valuable when it helps teams preserve signal. It is harmful when it becomes a substitute for analysis. The best policies are deliberately skeptical, they ask what the test protects, what evidence supports quarantine, who will own the follow-up, and how the team will know when the instability has become unacceptable.

If your organization can answer those questions clearly, quarantine can be a disciplined part of QA governance. If it cannot, then quarantine is probably masking more risk than it removes.

The practical aim is simple, keep CI usable, keep real regressions visible, and make every temporary exception easy to find, easy to challenge, and hard to forget.