CI Analytics

Overview

Flaky CI jobs are one of the most persistent sources of developer friction in fast-moving engineering organizations. A job that fails one-in-five runs forces engineers to re-trigger pipelines, context-switch back to merged work, and second-guess whether a real regression is hiding behind noise. At scale, a handful of flaky jobs can consume hours of engineering time per week.

Koalr's CI Analytics feature connects to GitHub Actions via webhook events and builds a continuous reliability record for every workflow and job in your organization. Rather than hunting through GitHub's own UI workflow-by-workflow, you get a single cross-repository view of which jobs are flaky, which workflows are slowing down, and where to focus stabilization effort.

What Koalr tracks

Metric	Description
Pass rate	Percentage of runs that completed with success status, per workflow and per job, over a rolling window
Flakiness rate	Ratio of flap events to total runs — a flap is a failure → success or success → failure transition on the same job within a short window
Duration P50	Median job or workflow run duration
Duration P75	75th-percentile duration — more sensitive to tail latency than P50
Flap count	Absolute number of flap events in the selected period
Run count	Total number of executions, used to assess statistical significance of rates

CI data is ingested from GitHub's workflow_run and workflow_job webhook events. No polling is used — all data arrives in real time as jobs complete.

Dashboard sections

Navigate to Delivery → CI Analytics to view the dashboard.

Org summary

The top row shows organization-wide aggregates for the selected date range:

• Total workflow runs across all repositories
• Overall pass rate (all workflows combined)
• Number of jobs classified as flaky (flakiness rate > 10%)
• Median workflow duration trend (current period vs previous period)

Per-workflow table

A sortable table listing every workflow observed in the selected period. Columns:

Column	Description
Workflow	Workflow name and repository
Runs	Total run count
Pass rate	Success / total runs
P50 duration	Median run time
P75 duration	75th-percentile run time
Trend	Pass rate direction vs prior period (up/down/stable)

Click any workflow row to expand and see the per-job breakdown for that workflow.

Per-job flakiness table

Below the workflow table, the Flaky Jobs panel lists all jobs where the flakiness rate exceeds 10% and at least 10 runs have been recorded (the minimum for a statistically meaningful rate). Jobs are sorted by flakiness rate descending — the most disruptive jobs appear first.

Column	Description
Job name	Step name within the workflow
Workflow	Parent workflow
Repository	Source repository
Flakiness rate	Flap events ÷ total runs
Flaps (7d)	Raw flap count over the last 7 days
Pass rate	Overall pass rate for this job

A job appearing in this table is an explicit signal that it warrants investigation: the failure pattern is inconsistent, meaning the job is not reliably detecting real regressions and is generating noise instead.

Duration trend chart

A line chart showing P50 and P75 workflow duration over time for the selected workflow. Use this to detect gradual slowdowns — a workflow that takes 30% longer than it did 30 days ago is worth investigating even if pass rate is unchanged. Common causes: dependency install time growth, test suite expansion without parallelization, or environment cold-start regressions.

How flakiness is calculated

A flap is recorded when the same job name in the same workflow completes with a different outcome than its immediately preceding run on the same branch. Specifically:

flap = (run[n].conclusion != run[n-1].conclusion)
     where both runs are on the same workflow + job + repository

The flakiness rate for a job is:

flakiness_rate = flap_count / total_runs

A job is classified as flaky when:

• flakiness_rate > 0.10 (more than 10% of runs are flap events), and
• total_runs >= 10 (enough data for the rate to be meaningful)

The 10% threshold is the default. It can be adjusted per organization in Settings → CI Analytics → Flakiness threshold.

This definition deliberately excludes jobs that consistently fail — those are broken, not flaky. Flakiness specifically captures jobs whose pass/fail outcome cannot be predicted from the code under test.

How to use CI Analytics

Identifying your highest-priority flaky jobs:

1. Open Delivery → CI Analytics.
2. Set the date range to the last 30 days for a broad view.
3. Sort the Flaky Jobs table by Flaps (7d) descending — this prioritizes jobs causing the most friction right now.
4. Look for jobs with both a high flakiness rate and a high run count. A 25% flakiness rate on a job that runs 200 times per week is far more costly than the same rate on a job running 10 times.

Tracking stabilization progress:

After your team fixes a flaky job, check back in 7–14 days. A successful fix will show the flakiness rate dropping toward 0 and the job disappearing from the flaky jobs table. The pass rate trend column in the workflow table also surfaces recent improvements.

Detecting duration regressions:

Use the duration trend chart to catch workflow slowdowns before they become critical. A workflow that used to complete at P50 = 4 minutes but is now trending toward 8 minutes represents a compounding productivity cost — especially on a team that deploys frequently.

Requirements

• GitHub integration connected — CI Analytics requires the GitHub integration with webhook delivery enabled. See GitHub Integration.
• The GitHub App must have permission to receive workflow_run and workflow_job webhook events. These are enabled by default when connecting GitHub via Koalr's standard OAuth flow.
• Data is available from the date the GitHub integration was connected. Historical backfill is not available for CI events (GitHub does not expose historical workflow event data via webhook replay).

Relationship to other metrics

CI Analytics connects directly to several other Koalr metrics:

• DORA Metrics — Deployment Frequency: Flaky CI directly suppresses deployment frequency by blocking merges and triggering manual re-runs. Reducing flakiness is one of the fastest ways to improve this DORA metric.
• Flow Metrics — Cycle Time: CI wait time (including flaky re-runs) contributes to the "review and verify" stage of cycle time. The duration trend chart surfaces this contribution directly.
• Deploy Risk: A failed CI check on a PR contributes to the Review Coverage signal in the deploy risk score. Jobs with known high flakiness introduce noise into this signal — the flakiness rate table helps you identify which CI failures are meaningful vs. incidental.