Commit 866a32a9 authored by John Skarbek's avatar John Skarbek Committed by Craig Barrett

Adds information to help track down high rails error rates

parent d2271c56
......@@ -12,7 +12,7 @@ groups:
severity: critical
annotations:
description: Rails is returning 5xx errors at a high rate for {{ $labels.type }} . Traffic is impacted and users are likely seeing 500 errors.
runbook: troubleshooting/gitlab-com-is-down.md
runbook: troubleshooting/high-error-rate.md
title: High Rails Error Rate on Front End
- alert: HighWebErrorRate
......
......@@ -6,9 +6,14 @@
## Symptoms
* Message in prometheus-alerts _Increased Error Rate Across Fleet_
- Message in prometheus-alerts _Increased Error Rate Across Fleet_
- PagerDuty Alert: _HighRailsErrorRate_
## Troubleshoot
### Kibana
- [All 5xx statuses in rails](https://log.gitlab.net/goto/c0d8ed2d964e4a792838e77a4ac1f942)
- [All 5xx statuses by controller](https://log.gitlab.net/goto/19bccd903f408085535df92734176cec)
- [Check for abuse from a specific IP](https://log.gitlab.net/goto/d4c6a0d68a565a0ac70b3840306f8eca)
- Check the [triage overview](https://dashboards.gitlab.net/dashboard/db/triage-overview) dashboard for 5xx errors by backend.
- Check [Sentry](https://sentry.gitlab.net/gitlab/gitlabcom/) for new 500 errors.
- If the problem persists send a channel wide notification in `#development`.
- Check [Sentry](https://sentry.gitlab.net/gitlab/gitlabcom/) for new 500 errors or an uptick.
- If the problem persists send a channel wide notification in [#development](https://gitlab.slack.com/archives/development).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment