Add script to fill event log gaps manually

.gitlab/issue_templates/failover.md

@@ -337,6 +337,8 @@ state of the secondary to converge.
     * You can use `sudo gitlab-rake geo:status` instead if the UI is non-compliant
     * If failures appear, see Rails console commands to resync repos/wikis: https://gitlab.com/snippets/1713152
     * On staging, this may not complete
+    * Fill event log gaps manually
+      * `/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p03/010-fill-event-log-gaps.sh`
 1. [ ] 🐺 {+ Coordinator +}: Wait for all repositories and wikis to become verified
     * Staging: https://dashboards.gitlab.net/d/YoKVGxSmk/gcp-failover-gcp?orgId=1&var-environment=gstg
     * Production: https://dashboards.gitlab.net/d/YoKVGxSmk/gcp-failover-gcp?orgId=1&var-environment=gprd
@@ -349,7 +351,7 @@ state of the secondary to converge.
     * In GCP: `Geo::EventLogState.last_processed.id`
     * The two numbers should be the same
 1. [ ]  🐺 {+ Coordinator +}: Ensure the prospective failover target in GCP is up to date
-    * `/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p03/check-wal-secondary-sync.sh`
+    * `/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p03/020-check-wal-secondary-sync.sh`
     * Assuming the clocks are in sync, this value should be close to 0
     * If this is a large number, GCP may not have some data that is in Azure
 1. [ ] 🐺 {+ Coordinator +}: Now disable all sidekiq-cron jobs on the secondary

bin/scripts/02_failover/060_go/p03/010-fill-event-log-gaps.rb

+#!/usr/bin/gitlab-rails runner
+
+GEO_EVENT_LOG_GAPS = 'geo:event_log:gaps'.freeze
+
+def with_redis
+  ::Gitlab::Redis::SharedState.with { |redis| yield redis }
+end
+
+# Get gaps tracked in redis
+gap_ids = with_redis { |redis| redis.zrangebyscore(GEO_EVENT_LOG_GAPS, '-inf', '+inf', with_scores: true) }; nil
+gap_ids.each { |gap| puts "Found event log gap with id #{gap.first} @ #{Time.now.to_i - gap.second} seconds ago" }; gap_ids.count
+
+# Fetch the events from database, if they exist
+gap_events = Geo::EventLog.where(id: gap_ids.map(&:first)); gap_events.count
+
+# Use the daemon to process the gap events
+daemon = Gitlab::Geo::LogCursor::Daemon.new; nil
+gap_events.each { |gap_event| daemon.send(:handle_single_event, gap_event.first) }; nil
+
+# Delete the gaps from redis
+with_redis { |redis| redis.zrem(GEO_EVENT_LOG_GAPS, gap_events.map(&:id)) }
+
+puts "Filled #{gap_events.count} gaps out of #{gaps_ids.count}"

bin/scripts/02_failover/060_go/p03/010-fill-event-log-gaps.sh

+#!/usr/bin/env bash
+
+set -euo pipefail
+IFS=$'\n\t'
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+UNSYMLINKED_SCRIPT_DIR="$(greadlink -f "${SCRIPT_DIR}" || readlink "${SCRIPT_DIR}" || echo "${SCRIPT_DIR}")"
+# shellcheck disable=SC1091,SC1090
+source "${UNSYMLINKED_SCRIPT_DIR}/../../../../workflow-script-commons.sh"
+
+# --------------------------------------------------------------
+
+if [[ ${FAILOVER_ENVIRONMENT} == "stg" ]]; then
+  DEPLOY_HOST=deploy.stg.gitlab.com
+elif [[ ${FAILOVER_ENVIRONMENT} == "prd" ]]; then
+  DEPLOY_HOST=deploy.gitlab.com
+fi
+
+# remote_gitlab_runner will log the command internally
+remote_gitlab_runner "${DEPLOY_HOST}" "${SCRIPT_DIR}/010-fill-event-log-gaps.rb"