Merge branch 'add-insns-for-silencing-alerts' into 'master'

Add instructions for silencing alerts

Closes #710

See merge request gitlab-com/migration!157

.gitlab/issue_templates/failover.md

@@ -143,12 +143,18 @@ an hour before the scheduled maintenance window.
 1. [ ] ☎ {+ Comms-Handler +}: Post to #announcements on Slack:
     * Staging: `We're rehearsing the failover of GitLab.com in *1 hour* by migrating staging.gitlab.com to GCP. Come watch us at ZOOM_LINK! Notes in GOOGLE_DOC_LINK!`
     * Production: `GitLab.com is being migrated to GCP in *1 hour*. There is a 2-hour downtime window.  We'll be live on YouTube. Notes in GOOGLE_DOC_LINK!`
+1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: Create a maintenance window in PagerDuty for [GitLab Production service](https://gitlab.pagerduty.com/services/PATDFCE) for 2 hours starting in an hour from now.
+1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: [Create an alert silence](https://alerts.gitlab.com/#/silences/new) for 2 hours starting in an hour from now with the following matcher(s):
+    - `environment`: `prd`
 1. [ ] 🔪 {+ Chef-Runner +}: Stop any new GitLab CI jobs from being executed
     * Block `POST /api/v4/jobs/request`
     * Staging
         * https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2094
         * `knife ssh -p 2222 roles:staging-base-lb 'sudo chef-client'`
     * Production
+        * [Create an alert silence](https://alerts.gitlab.com/#/silences/new) for 3 hours (starting now) with the following matchers:
+            - `environment`: `prd`
+            - `alertname`: `High4xxApiRateLimit|High4xxRateLimit`, check "Regex"
         * https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2243
         * `knife ssh -p 2222 roles:gitlab-base-lb 'sudo chef-client'`
 - [ ] ☎ {+ Comms-Handler +}: Create a broadcast message