Commit b2288b4c authored by Nick Thomas's avatar Nick Thomas
Browse files

Merge branch 'add-insns-for-silencing-alerts' into 'master'

Add instructions for silencing alerts

Closes #710

See merge request gitlab-com/migration!157
parents 38e55e86 fe34a041
...@@ -143,12 +143,18 @@ an hour before the scheduled maintenance window. ...@@ -143,12 +143,18 @@ an hour before the scheduled maintenance window.
1. [ ] ☎ {+ Comms-Handler +}: Post to #announcements on Slack: 1. [ ] ☎ {+ Comms-Handler +}: Post to #announcements on Slack:
* Staging: `We're rehearsing the failover of GitLab.com in *1 hour* by migrating staging.gitlab.com to GCP. Come watch us at ZOOM_LINK! Notes in GOOGLE_DOC_LINK!` * Staging: `We're rehearsing the failover of GitLab.com in *1 hour* by migrating staging.gitlab.com to GCP. Come watch us at ZOOM_LINK! Notes in GOOGLE_DOC_LINK!`
* Production: `GitLab.com is being migrated to GCP in *1 hour*. There is a 2-hour downtime window. We'll be live on YouTube. Notes in GOOGLE_DOC_LINK!` * Production: `GitLab.com is being migrated to GCP in *1 hour*. There is a 2-hour downtime window. We'll be live on YouTube. Notes in GOOGLE_DOC_LINK!`
1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: Create a maintenance window in PagerDuty for [GitLab Production service](https://gitlab.pagerduty.com/services/PATDFCE) for 2 hours starting in an hour from now.
1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: [Create an alert silence](https://alerts.gitlab.com/#/silences/new) for 2 hours starting in an hour from now with the following matcher(s):
- `environment`: `prd`
1. [ ] 🔪 {+ Chef-Runner +}: Stop any new GitLab CI jobs from being executed 1. [ ] 🔪 {+ Chef-Runner +}: Stop any new GitLab CI jobs from being executed
* Block `POST /api/v4/jobs/request` * Block `POST /api/v4/jobs/request`
* Staging * Staging
* https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2094 * https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2094
* `knife ssh -p 2222 roles:staging-base-lb 'sudo chef-client'` * `knife ssh -p 2222 roles:staging-base-lb 'sudo chef-client'`
* Production * Production
* [Create an alert silence](https://alerts.gitlab.com/#/silences/new) for 3 hours (starting now) with the following matchers:
- `environment`: `prd`
- `alertname`: `High4xxApiRateLimit|High4xxRateLimit`, check "Regex"
* https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2243 * https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2243
* `knife ssh -p 2222 roles:gitlab-base-lb 'sudo chef-client'` * `knife ssh -p 2222 roles:gitlab-base-lb 'sudo chef-client'`
- [ ] ☎ {+ Comms-Handler +}: Create a broadcast message - [ ] ☎ {+ Comms-Handler +}: Create a broadcast message
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment