incident.io Notification Channel Setup for Alerts

Prerequisites

Before configuring incident.io as a notification channel in SigNoz, ensure that you have:

• An incident.io account with alerts enabled (On-call → Alert routing is visible).
• An HTTP alert source in incident.io. SigNoz delivers alerts to its Alert Events endpoint; you'll create one in the next step.

Create an HTTP alert source in incident.io

1. In incident.io, navigate to On-call → Alert routing → Sources and create a new HTTP alert source. Give it a name that identifies SigNoz, e.g. signoz.
2. On the source's setup page, copy two values:
   • The alert events URL, of the form https://api.incident.io/v2/alert_events/http/<source_config_id>.
   • The secret token.

Creating a new Notification channel

To create a new incident.io notification channel in SigNoz:

• Navigate to Settings > Notification Channels.
• Click on New Channel and enter a name.
• Select incident.io as the channel type and fill in:

• Click Test to send a test alert, then Save.

Editing a Notification channel

To edit an existing incident.io channel, navigate to Settings > Notification Channels, click Edit on the channel, adjust any field except the name and type, and save.

Send metadata to incident.io

Every alert event carries a flat metadata map that incident.io can use for filtering, routing, and display:

• Sent automatically: all the alert's labels: severity, alertname, and any labels you set on the alert rule (e.g. team, service, environment).
• Additional metadata (channel setting): static key-value pairs added on top. On a key clash, the channel's value overrides the alert's label. Values may use Go templates, e.g. {{ .CommonLabels.severity }}; a value that fails to render is sent as the raw string, so delivery is never blocked by a template typo.

To make metadata usable inside incident.io (visible on alerts, filterable in routes), map it to alert attributes on your source; see incident.io's Attributes and Priorities documentation. Send a test alert first so incident.io can suggest attribute mappings from real payloads.

Receive alerts in incident.io

• An alert firing creates an incident.io alert. The description renders the markdown template, including deep links back to SigNoz: View in SigNoz, and View related logs / View related traces when the alert carries those links. The alert's Links panel also carries a link to the SigNoz rule.

• When several alerts fire together, they are delivered as a single incident.io alert for the group.
• Repeat notifications while the alert is firing are ignored by incident.io, so the alert keeps the content of its first event, with no duplicates created. For current values, follow the SigNoz links on the alert.
• A resolve closes the incident.io alert (Send resolved alerts on). If the same alert fires again later, a fresh incident.io alert is created.

• If someone resolves the alert manually in incident.io while the SigNoz alert is still firing, the next repeat notification from SigNoz re-creates it.

Limits handled automatically

• incident.io caps event payloads at 512 KB; SigNoz truncates oversized descriptions before sending.
• incident.io rate-limits each alert source to 120 events/minute; rate-limited requests (429) are retried automatically.

Alert routes and incidents

Alerts don't create incidents by themselves. In incident.io, alert routes decide which alerts become incidents and how they group. Points worth knowing:

• Connect your SigNoz source to an alert route to auto-create incidents; see incident.io's Creating incidents automatically via alerts.
• With default grouping, alerts on a route within a fixed 30-minute window group into one incident; incidents start in Triage.
• A SigNoz resolve closes the incident.io alert, not the incident; incidents are closed by responders.

Customizing the message

The Title is the alert's title (plain text). The Description is standard markdown, rendered natively by incident.io. Both accept Go templates.

incident.io keeps an alert's content from its first event for as long as it fires, so avoid putting fast-changing values in the title. Put stable identity there and let the SigNoz links carry current state.

Turn off resolved alerts

The Send resolved alerts toggle is on by default. To stop SigNoz from resolving incident.io alerts:

• Navigate to Settings > Notification Channels.
• Click Edit on the channel.
• Toggle off Send resolved alerts and click Save.

With the toggle off, incident.io alerts are created when SigNoz alerts fire but stay open until resolved in incident.io.

Configure programmatically

You can also configure the incident.io notification channel using the SigNoz API.

API Endpoint: POST /api/v1/channels

Send a POST request to /api/v1/channels with the following curl command. Replace <your-api-key> with an API key from Settings → Service Accounts (see Service Accounts to create one).

curl '<your-signoz-url>/api/v1/channels' \
  -H 'SIGNOZ-API-KEY: <your-api-key>' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "name": "Prod Critical Alerts - incident.io",
    "incidentio_configs": [
        {
            "send_resolved": true,
            "url": "https://api.incident.io/v2/alert_events/http/<source_config_id>",
            "token": "<your-source-token>",
            "title": "[{{ .Status | toUpper }}{{ if eq .Status \"firing\" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}",
            "metadata": {
                "environment": "prod"
            }
        }
    ]
}'

Request Body Attributes:

Troubleshooting

If you encounter issues:

• 401 → the token is wrong. Re-copy it from the alert source's setup page (with or without the Bearer prefix; both work).
• 404 → the URL is wrong, usually a bad or missing <source_config_id>. Re-copy the full alert events URL.
• Test succeeded but no alert appears → check the source's Requests tab in incident.io. Each event shows an outcome: Alert ingested, No update to existing alert (a repeat of an already-firing alert), or No alert to resolve (a resolve for an alert incident.io doesn't have; harmless).
• Metadata not visible on alerts → metadata must be mapped to attributes on the incident.io source; see Send metadata to incident.io.
• No incident created → alerts only create incidents through an alert route; see Alert routes and incidents.

Next steps

• Group and route alerts into incidents with alert routes.
• Map SigNoz labels to attributes and priorities for routing and escalation.