Deploy Risk Setup Guide

Koalr scores every pull request from 0 to 100 based on 32 research-validated signals before it merges. This guide walks you through the full setup -- from connecting GitHub to receiving Slack alerts when a high-risk PR is opened.

Time to complete: 20--30 minutes.

Plan requirement: Deploy Risk requires the Business plan or higher. The Accuracy and Signal Weight Tuning pages require the Enterprise plan.

---

Prerequisites

• A Koalr account with Admin or Owner role.
• GitHub connected to Koalr. If you have not done this yet, follow Step 1 in the GitHub + CODEOWNERS Setup Guide.
• At least one repository with open or recently merged pull requests.

---

Step 1: Open the Deploy Risk dashboard

Navigate to Deploy Risk in the left sidebar (under the Delivery section). If your organization is on the Business plan or has an active trial, you will see the risk dashboard. Otherwise, you will see an upgrade prompt.

On first load with no data, the dashboard will be empty. Risk scores are computed as PRs are opened and updated -- so if you have recently connected GitHub, scores will begin appearing within minutes as webhook events arrive.

---

Step 2: Understand your first risk scores

Once scores appear, the dashboard shows several sections:

Organization risk summary

Four KPI tiles at the top:

• Total PRs Scored -- how many PRs Koalr has analyzed in the selected period.
• High-Risk PRs -- count of PRs scoring 70 or above.
• Avg Risk Score -- the mean score across all PRs.
• Risk Trend -- week-over-week direction (improving or worsening).

Risk distribution chart

A bar chart showing how PRs break down across risk levels (Low 0--39, Medium 40--69, High 70--100). A healthy distribution has the majority of PRs in the Low bucket.

PR risk table

Every scored PR with its risk score badge, top contributing signals, author, repository, and a link to the GitHub PR. Click any row to expand the full signal breakdown.

Signal breakdown

Each PR's score is a weighted sum of factors across five categories. The expanded view shows each factor's individual score, weight, and contribution. For a deep dive into every signal, see How Risk is Scored.

---

Step 3: Enable the GitHub Check Run

The GitHub Check Run posts the risk score directly in the GitHub PR interface -- engineers see the score without leaving GitHub.

If you installed the Koalr GitHub App

The check run is automatic. Every PR opened or updated on a connected repository will show a check named Koalr Deploy Risk in the Checks tab. No additional setup is needed.

If you use a Personal Access Token

You can use the Koalr Deploy Risk GitHub Action instead:

1. Create an API key at Settings then API Keys.
2. Add it as a repository secret named KOALR_API_KEY in your GitHub repo settings.
3. Add the workflow file .github/workflows/koalr-deploy-risk.yml (see the GitHub Action docs for the full YAML).

Block high-risk merges with branch protection

To prevent Critical-risk PRs from being merged:

1. In your GitHub repository, go to Settings then Branches.
2. Edit the branch protection rule for your default branch.
3. Under Require status checks to pass before merging, add Koalr Deploy Risk.
4. Save.

The check run passes for Low and Medium risk, shows a neutral/warning status for High risk, and fails for Critical risk (score 85+) and DDL hard gates.

For full check run details, see GitHub Check Run.

---

Step 4: Configure Slack alerts for high-risk PRs

Receive a Slack notification whenever a PR is scored High or Critical:

1. Ensure Slack is connected (see Slack Setup).
2. Navigate to Automations in the left sidebar.
3. Create a new automation rule:
   • Trigger: High-risk PR detected
   • Action: Post to Slack channel
   • Channel: Select your engineering or release channel
4. Save.

The Slack message includes the PR title, author, risk score, top 3 contributing signals, and a link to the full breakdown on the Koalr dashboard.

---

Step 5: Review risk by environment

The Deploy Risk dashboard includes an Environment Breakdown section that shows risk scores segmented by deployment target (production, staging, development). This helps you understand whether risk is concentrated in production deploys or spread across environments.

Use the period selector (7d, 14d, 30d) at the top of the dashboard to adjust the time window.

---

Step 6: Tune signal weights (Enterprise)

After your organization has accumulated enough deployment outcomes (at least 30 logged as success or incident), you can tune how much each signal contributes to the overall score.

1. Navigate to Deploy Risk then click Accuracy in the breadcrumb navigation.
2. The Signal Weight Tuning section at the bottom of the page shows each signal with a slider.
3. Adjust weights to reflect your codebase patterns. For example, if DDL migrations are routine and well-tested in your environment, you might lower the Schema Migration weight.
4. Click Apply weights to save.

Koalr also runs weekly auto-tuning using logistic regression on your deployment outcomes. Manual adjustments override auto-tuning for the signals you have changed.

---

Step 7: Track accuracy over time (Enterprise)

The Accuracy page shows how well Koalr's risk predictions correlate with actual incidents:

• Prediction Accuracy Score -- a composite score out of 100.
• Incident Correlation Rate -- what percentage of high-risk PRs actually resulted in incidents.
• Accuracy Trend -- a week-over-week chart showing the model improving as it learns from your data.
• Risk Distribution vs Incident Rate -- a chart showing incident rates for Low, Medium, and High risk buckets. In a well-calibrated model, High-risk PRs should have a significantly higher incident rate than Low-risk ones.
• High-Risk PRs Merged Anyway -- a table of PRs that scored 70+ but were merged, showing whether they led to incidents.

This data helps you validate that the risk model is accurate for your specific codebase and identify where it needs calibration.

---

Verification checklist

• Deploy Risk dashboard loads and shows scored PRs.
• A new or updated PR on a connected repo triggers the Koalr Deploy Risk check run in GitHub.
• Branch protection is configured to require the check (if you want to block Critical merges).
• Slack channel receives a notification for a High or Critical risk PR.
• Accuracy page shows data (requires 30+ deployment outcomes over time).

---

What to do next

• Connect CODEOWNERS -- CODEOWNERS compliance is a core deploy risk signal. See GitHub + CODEOWNERS Setup.
• Connect a coverage provider -- test coverage delta on changed files is another high-value signal. See Coverage Integration Guide.
• Connect an incident provider -- PagerDuty or OpsGenie data enables Change Failure Rate and incident correlation for accuracy tracking. See DORA Metrics Setup.
• Set up teams -- deploy risk can be filtered by team to compare risk profiles across your organization. See Team Setup Best Practices.