Commit dd391943 authored by Brett Walker's avatar Brett Walker

Merge branch '756-update-failover-templates-with-proper-script-references' into 'master'

Resolve "Update failover templates with proper script references"

Closes #756

See merge request gitlab-com/migration!188
parents e89afcfe 3d0b2ea3
Pipeline #88698 passed with stage
in 23 seconds
...@@ -124,9 +124,9 @@ first. ...@@ -124,9 +124,9 @@ first.
1. [ ] 🐺 {+ Coordinator +}: Perform (or coordinate) Preflight Checklist 1. [ ] 🐺 {+ Coordinator +}: Perform (or coordinate) Preflight Checklist
1. [ ] **PRODUCTION ONLY** ☎ {+ Comms-Handler +}: Tweet from `@gitlab`. 1. [ ] **PRODUCTION ONLY** ☎ {+ Comms-Handler +}: Tweet from `@gitlab`.
- Tweet content from `/opt/gitlab-migration/bin/scripts/02_failover/030_t-1d/010_gitlab_twitter_announcement.sh` - Tweet content from `/opt/gitlab-migration/migration/bin/scripts/02_failover/030_t-1d/010_gitlab_twitter_announcement.sh`
1. [ ] **PRODUCTION ONLY** ☎ {+ Comms-Handler +}: Retweet `@gitlab` tweet from `@gitlabstatus` with further details 1. [ ] **PRODUCTION ONLY** ☎ {+ Comms-Handler +}: Retweet `@gitlab` tweet from `@gitlabstatus` with further details
- Tweet content from `/opt/gitlab-migration/bin/scripts/02_failover/030_t-1d/020_gitlabstatus_twitter_announcement.sh` - Tweet content from `/opt/gitlab-migration/migration/bin/scripts/02_failover/030_t-1d/020_gitlabstatus_twitter_announcement.sh`
# T minus 3 hours (__FAILOVER_DATE__) [📁](bin/scripts/02_failover/040_t-3h) # T minus 3 hours (__FAILOVER_DATE__) [📁](bin/scripts/02_failover/040_t-3h)
...@@ -149,7 +149,7 @@ an hour before the scheduled maintenance window. ...@@ -149,7 +149,7 @@ an hour before the scheduled maintenance window.
1. [ ] **PRODUCTION ONLY** ☎ {+ Comms-Handler +}: Tweet from `@gitlabstatus` 1. [ ] **PRODUCTION ONLY** ☎ {+ Comms-Handler +}: Tweet from `@gitlabstatus`
- `As part of upcoming GitLab.com maintenance work, CI runners will not be accepting new jobs until __MAINTENANCE_END_TIME__ UTC. GitLab.com will undergo maintenance in 1 hour. Working doc: __GOOGLE_DOC_URL__` - `As part of upcoming GitLab.com maintenance work, CI runners will not be accepting new jobs until __MAINTENANCE_END_TIME__ UTC. GitLab.com will undergo maintenance in 1 hour. Working doc: __GOOGLE_DOC_URL__`
1. [ ] ☎ {+ Comms-Handler +}: Post to #announcements on Slack: 1. [ ] ☎ {+ Comms-Handler +}: Post to #announcements on Slack:
- `/opt/gitlab-migration/bin/scripts/02_failover/050_t-1h/020_slack_announcement.sh` - `/opt/gitlab-migration/migration/bin/scripts/02_failover/050_t-1h/020_slack_announcement.sh`
1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: Create a maintenance window in PagerDuty for [GitLab Production service](https://gitlab.pagerduty.com/services/PATDFCE) for 2 hours starting in an hour from now. 1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: Create a maintenance window in PagerDuty for [GitLab Production service](https://gitlab.pagerduty.com/services/PATDFCE) for 2 hours starting in an hour from now.
1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: [Create an alert silence](https://alerts.gitlab.com/#/silences/new) for 2 hours starting in an hour from now with the following matcher(s): 1. [ ] **PRODUCTION ONLY** ☁ {+ Cloud-conductor +}: [Create an alert silence](https://alerts.gitlab.com/#/silences/new) for 2 hours starting in an hour from now with the following matcher(s):
- `environment`: `prd` - `environment`: `prd`
...@@ -168,7 +168,7 @@ an hour before the scheduled maintenance window. ...@@ -168,7 +168,7 @@ an hour before the scheduled maintenance window.
- [ ] ☎ {+ Comms-Handler +}: Create a broadcast message - [ ] ☎ {+ Comms-Handler +}: Create a broadcast message
* Staging: https://staging.gitlab.com/admin/broadcast_messages * Staging: https://staging.gitlab.com/admin/broadcast_messages
* Production: https://gitlab.com/admin/broadcast_messages * Production: https://gitlab.com/admin/broadcast_messages
* Text: `gitlab.com is [moving to a new home](https://about.gitlab.com/2018/04/05/gke-gitlab-integration/)! Hold on to your hats, we’re going dark for approximately 2 hours from XX:XX on 2018-XX-YY UTC` * Text: `gitlab.com is [moving to a new home](https://about.gitlab.com/2018/04/05/gke-gitlab-integration/)! Hold on to your hats, we’re going dark for approximately 2 hours from __MAINTENANCE_START_TIME__ on __FAILOVER_DATE__ UTC`
* Start date: now * Start date: now
* End date: now + 3 hours * End date: now + 3 hours
1. [ ] ☁ {+ Cloud-conductor +}: Initial snapshot of database disks in case of failback in Azure and GCP 1. [ ] ☁ {+ Cloud-conductor +}: Initial snapshot of database disks in case of failback in Azure and GCP
...@@ -280,11 +280,11 @@ Running CI jobs will no longer be able to push updates. Jobs that complete now m ...@@ -280,11 +280,11 @@ Running CI jobs will no longer be able to push updates. Jobs that complete now m
#### Phase 2: Commence Shutdown in Azure [📁](bin/scripts/02_failover/060_go/p02) #### Phase 2: Commence Shutdown in Azure [📁](bin/scripts/02_failover/060_go/p02)
1. [ ] 🔪 {+ Chef-Runner +}: Stop mailroom on all the nodes 1. [ ] 🔪 {+ Chef-Runner +}: Stop mailroom on all the nodes
* `/opt/gitlab-migration/bin/scripts/02_failover/060_go/p02/010-stop-mailroom.sh` * `/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p02/010-stop-mailroom.sh`
1. [ ] 🔪 {+ Chef-Runner +} **PRODUCTION ONLY**: Stop `sidekiq-pullmirror` in Azure 1. [ ] 🔪 {+ Chef-Runner +} **PRODUCTION ONLY**: Stop `sidekiq-pullmirror` in Azure
* `/opt/gitlab-migration/bin/scripts/02_failover/060_go/p02/020-stop-sidekiq-pullmirror.sh` * `/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p02/020-stop-sidekiq-pullmirror.sh`
1. [ ] 🐺 {+ Coordinator +}: Sidekiq monitor: start purge of non-mandatory jobs, disable Sidekiq crons and allow Sidekiq to wind-down: 1. [ ] 🐺 {+ Coordinator +}: Sidekiq monitor: start purge of non-mandatory jobs, disable Sidekiq crons and allow Sidekiq to wind-down:
* In a separate terminal on the deploy host: `/opt/gitlab-migration/bin/scripts/02_failover/060_go/p02/030-await-sidekiq-drain.sh` * In a separate terminal on the deploy host: `/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p02/030-await-sidekiq-drain.sh`
* The `geo_sidekiq_cron_config` job or an RSS kill may re-enable the crons, which is why we run it in a loop * The `geo_sidekiq_cron_config` job or an RSS kill may re-enable the crons, which is why we run it in a loop
* The loop may be stopped once sidekiq is shut down * The loop may be stopped once sidekiq is shut down
1. [ ] 🐺 {+ Coordinator +}: Wait for repository verification on the **primary** to complete 1. [ ] 🐺 {+ Coordinator +}: Wait for repository verification on the **primary** to complete
......
...@@ -7,7 +7,7 @@ ISSUE_TEMPLATES_DIR=${ROOT_DIR}/.gitlab/issue_templates ...@@ -7,7 +7,7 @@ ISSUE_TEMPLATES_DIR=${ROOT_DIR}/.gitlab/issue_templates
function find_script_ref() { function find_script_ref() {
# shellcheck disable=SC2016 # shellcheck disable=SC2016
grep -Eho "\`/opt/gitlab-migration/bin.*?\`" .gitlab/issue_templates/*.md|sed -E 's#`/opt/gitlab-migration/|`##g'|sort -u grep -Eho "\`/opt/gitlab-migration/migration/bin.*?\`" .gitlab/issue_templates/*.md|sed -E 's#`/opt/gitlab-migration/migration/|`##g'|sort -u
} }
find_script_ref | while IFS='' read -r file; do find_script_ref | while IFS='' read -r file; do
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment