Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
What's new
4
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
gitlab-com
migration
Commits
bdbe3dd6
Commit
bdbe3dd6
authored
Aug 11, 2018
by
Toon Claes
🎨
Browse files
Add script to fill event log gaps manually
parent
26856140
Pipeline
#89119
passed with stage
in 1 minute and 4 seconds
Changes
4
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
.gitlab/issue_templates/failover.md
View file @
bdbe3dd6
...
...
@@ -337,6 +337,8 @@ state of the secondary to converge.
*
You can use
`sudo gitlab-rake geo:status`
instead if the UI is non-compliant
*
If failures appear, see Rails console commands to resync repos/wikis: https://gitlab.com/snippets/1713152
*
On staging, this may not complete
*
Fill event log gaps manually
*
`/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p03/010-fill-event-log-gaps.sh`
1.
[ ] 🐺 {+ Coordinator +}: Wait for all repositories and wikis to become verified
*
Staging: https://dashboards.gitlab.net/d/YoKVGxSmk/gcp-failover-gcp?orgId=1&var-environment=gstg
*
Production: https://dashboards.gitlab.net/d/YoKVGxSmk/gcp-failover-gcp?orgId=1&var-environment=gprd
...
...
@@ -349,7 +351,7 @@ state of the secondary to converge.
*
In GCP:
`Geo::EventLogState.last_processed.id`
*
The two numbers should be the same
1.
[ ] 🐺 {+ Coordinator +}: Ensure the prospective failover target in GCP is up to date
*
`/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p03/check-wal-secondary-sync.sh`
*
`/opt/gitlab-migration/migration/bin/scripts/02_failover/060_go/p03/
020-
check-wal-secondary-sync.sh`
*
Assuming the clocks are in sync, this value should be close to 0
*
If this is a large number, GCP may not have some data that is in Azure
1.
[ ] 🐺 {+ Coordinator +}: Now disable all sidekiq-cron jobs on the secondary
...
...
bin/scripts/02_failover/060_go/p03/010-fill-event-log-gaps.rb
0 → 100644
View file @
bdbe3dd6
#!/usr/bin/gitlab-rails runner
GEO_EVENT_LOG_GAPS
=
'geo:event_log:gaps'
.
freeze
def
with_redis
::
Gitlab
::
Redis
::
SharedState
.
with
{
|
redis
|
yield
redis
}
end
# Get gaps tracked in redis
gap_ids
=
with_redis
{
|
redis
|
redis
.
zrangebyscore
(
GEO_EVENT_LOG_GAPS
,
'-inf'
,
'+inf'
,
with_scores:
true
)
};
nil
gap_ids
.
each
{
|
gap
|
puts
"Found event log gap with id
#{
gap
.
first
}
@
#{
Time
.
now
.
to_i
-
gap
.
second
}
seconds ago"
};
gap_ids
.
count
# Fetch the events from database, if they exist
gap_events
=
Geo
::
EventLog
.
where
(
id:
gap_ids
.
map
(
&
:first
));
gap_events
.
count
# Use the daemon to process the gap events
daemon
=
Gitlab
::
Geo
::
LogCursor
::
Daemon
.
new
;
nil
gap_events
.
each
{
|
gap_event
|
daemon
.
send
(
:handle_single_event
,
gap_event
.
first
)
};
nil
# Delete the gaps from redis
with_redis
{
|
redis
|
redis
.
zrem
(
GEO_EVENT_LOG_GAPS
,
gap_events
.
map
(
&
:id
))
}
puts
"Filled
#{
gap_events
.
count
}
gaps out of
#{
gaps_ids
.
count
}
"
bin/scripts/02_failover/060_go/p03/010-fill-event-log-gaps.sh
0 → 100755
View file @
bdbe3dd6
#!/usr/bin/env bash
set
-euo
pipefail
IFS
=
$'
\n\t
'
SCRIPT_DIR
=
"
$(
cd
"
$(
dirname
"
${
BASH_SOURCE
[0]
}
"
)
"
&&
pwd
)
"
UNSYMLINKED_SCRIPT_DIR
=
"
$(
greadlink
-f
"
${
SCRIPT_DIR
}
"
||
readlink
"
${
SCRIPT_DIR
}
"
||
echo
"
${
SCRIPT_DIR
}
"
)
"
# shellcheck disable=SC1091,SC1090
source
"
${
UNSYMLINKED_SCRIPT_DIR
}
/../../../../workflow-script-commons.sh"
# --------------------------------------------------------------
if
[[
${
FAILOVER_ENVIRONMENT
}
==
"stg"
]]
;
then
DEPLOY_HOST
=
deploy.stg.gitlab.com
elif
[[
${
FAILOVER_ENVIRONMENT
}
==
"prd"
]]
;
then
DEPLOY_HOST
=
deploy.gitlab.com
fi
# remote_gitlab_runner will log the command internally
remote_gitlab_runner
"
${
DEPLOY_HOST
}
"
"
${
SCRIPT_DIR
}
/010-fill-event-log-gaps.rb"
bin/scripts/02_failover/060_go/p03/check-wal-secondary-sync.sh
→
bin/scripts/02_failover/060_go/p03/
020-
check-wal-secondary-sync.sh
View file @
bdbe3dd6
File moved
Toon Claes
🎨
@toon
mentioned in commit
80b923d7
·
Aug 11, 2018
mentioned in commit
80b923d7
mentioned in commit 80b923d727e8b0bb7f92d3c1b94f5ce72b991c97
Toggle commit list
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment