Coverage vs Incident Correlation

Overview

Not all coverage metrics are equally meaningful. A repository with 85% coverage and zero incidents is a different situation from a repository with 85% coverage and a new incident every two weeks. The coverage number alone does not tell you whether your tests are protecting you in practice.

Koalr's Coverage vs Incident Correlation feature computes the statistical relationship between coverage changes and incident frequency for each repository in your organization. A strong positive correlation — coverage drops, incidents rise — is an actionable signal: the tests in that repository are materially reducing defects, and investing in coverage there will directly reduce production failures.

How correlation is computed

Koalr uses the Pearson correlation coefficient (r) to measure the linear relationship between two weekly time series per repository:

• Coverage % per week — the end-of-week coverage reading from Codecov or SonarCloud
• Incident count per week — the number of incidents opened that week for the service(s) mapped to this repository

The correlation is computed over a rolling 90-day window (approximately 13 weekly data points). The window rolls forward weekly as new data arrives.

r = Pearson(coverage_series, incident_series)
  where both series have weekly resolution over the past 90 days

Pearson r ranges from -1 to +1:

r value	Interpretation
+0.5 to +1.0	Strong positive correlation — coverage declines predict more incidents
+0.2 to +0.49	Weak positive correlation — some relationship, not conclusive
-0.2 to +0.19	No meaningful correlation — coverage changes do not predict incidents for this repository
-0.2 to -1.0	Negative correlation — coverage and incidents move together inversely (uncommon; may indicate confounding factors)

The sign convention used in the Koalr UI is inverted for readability: a positive displayed score means "coverage drops are associated with more incidents" — which matches the intuitive meaning of "bad." Internally, the raw Pearson r is computed on (coverage, incident_count) and then the display sign is flipped so that a green/positive result means the tests are working as expected.

Interpreting the correlation score

Strong positive correlation (r ≥ 0.5)

Coverage in this repository is a leading indicator of incidents. When coverage drops — even modestly — incident frequency tends to rise in the weeks that follow. This is the clearest possible evidence that test investment pays off in this codebase.

Recommended action: Treat this repository's coverage as a hard gate. Consider setting a coverage threshold alert at or above the current level, and blocking deploys when patch coverage drops below 60%.

Weak positive correlation (0.2 ≤ r < 0.5)

There is a directional relationship between coverage and incidents, but it is not strong enough to act on directly. The tests provide some protection, but other factors (infrastructure, external dependencies, deployment risk) are also significant contributors to incidents.

Recommended action: Continue maintaining coverage and monitor whether the correlation strengthens over time as more data is collected.

No correlation (|r| < 0.2)

Coverage changes in this repository do not appear to predict incidents. This could mean:

• The incidents affecting this service originate in a dependency, an external API, or infrastructure rather than in the application code
• Coverage is already high enough that marginal changes do not affect defect rates
• The test suite is large but tests high-confidence paths, leaving the risky paths untested regardless of coverage percentage

Recommended action: Investigate incident root causes for this repository to understand what is actually driving failures. Coverage investment alone may not be the right lever here.

Negative correlation

Coverage and incidents move in the same direction — both rising or both falling together. This is unusual and typically indicates a confounding variable: for example, a period of rapid feature development that simultaneously increases coverage (more code = more test files added) and incidents (more code = more bugs).

Recommended action: Investigate the time periods where the correlation inverts. This pattern often reveals a specific release cycle or team change worth understanding.

The scatter plot panel

The Incident Correlation tab at /coverage shows a scatter plot for each repository with at least 13 weeks of data. Each data point represents one week:

• X axis: Coverage % for that week
• Y axis: Incident count for that week
• Trend line: Ordinary least-squares regression line fitted to the scatter

The slope of the trend line and the tightness of the cluster around it provide a visual complement to the r value. A steep negative slope (lower coverage → more incidents) with points clustered close to the line is a strong correlation; a shallow slope with widely scattered points is weak or absent.

Per-repository breakdown

The full breakdown table lists every repository with sufficient data, sorted by absolute correlation strength:

Column	Description
Repository	Repository name with link to GitHub
Correlation (r)	Pearson coefficient, displayed with direction label
Coverage (current)	Latest coverage reading
Coverage trend	Direction over the past 30 days
Incidents (90d)	Total incidents mapped to this repo's services in the past 90 days
Data points	Number of weekly observations used to compute r

Repositories with fewer than 8 weekly data points display "Insufficient data" rather than a coefficient. The minimum is necessary to prevent spurious correlations from small samples.

Requirements

• Coverage integration: Codecov or SonarCloud must be connected and actively reporting coverage for the repositories you want to analyze.
• Incident integration: PagerDuty, OpsGenie, or incident.io must be connected and have at least 8 weeks of incident history.
• Service catalog mapping: Repositories must be mapped to services in Settings → Services so Koalr can associate incidents with the correct repositories.

Limitations

Correlation does not imply causation. A strong positive correlation tells you that coverage declines co-occur with incident increases for this repository. It does not prove that the lack of tests caused those incidents — other factors (deployment timing, infrastructure changes, dependency updates) may be responsible.

Minimum data requirement. The 90-day rolling window requires approximately 13 weekly data points. For repositories or services added recently, the correlation tab will show "Insufficient data" until enough history accumulates.

Weekly resolution smooths intra-week patterns. A coverage drop on a Tuesday and an incident on a Friday both fall in the same weekly bucket. High-frequency deployment teams may find that the weekly granularity misses same-day relationships. Future versions will offer a configurable resolution.

Small incident counts reduce significance. If a repository has had only 2 or 3 incidents in the past 90 days, even a mathematically high r value should be interpreted cautiously. The correlation table shows raw incident counts so you can assess this directly.

Public API

Coverage correlation data is available via the Koalr Public API:

GET /api/v1/coverage/correlation?days=90
Authorization: Bearer <api_key>

Query parameters:

Parameter	Default	Description
days	90	Rolling window in days (30–365)
repo	—	Filter to a single repository slug
min_data_points	8	Minimum weekly observations required to include a repository

Example response:

{
  "window_days": 90,
  "computed_at": "2026-03-23T08:00:00Z",
  "repositories": [
    {
      "repository": "my-org/payments-service",
      "correlation_r": 0.74,
      "correlation_label": "strong_positive",
      "coverage_current": 71.2,
      "coverage_trend": "declining",
      "incidents_in_window": 12,
      "data_points": 13
    }
  ]
}

Required API key scope: read:coverage. See API Keys for how to generate a scoped key.