Unverified Commit 408bee66 authored by Craig Miskell's avatar Craig Miskell
Browse files

Adds probe_future_sets to report on delays in scheduled and retry sets

A long description is available at https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1171 but in short, this now reports how far "behind" Sidekiq is in processing the Scheduled or Retry sets; jobs in those sets are expected to normally be processed within a short time (seconds, not minutes) of their nominal scheduled time, but in bad times we can sometimes get behind, and being able to observe the magnitude of the backlog is desirable.

Adds both a delay time metric (how old is the next job that will be processed) and a backlog count (how many jobs are pending)
parent 20d8cc06
PATH
remote: .
specs:
gitlab-exporter (11.0.1)
gitlab-exporter (11.1.0)
connection_pool (= 2.2.5)
pg (= 1.2.3)
puma (= 5.3.2)
......
require "sidekiq/api"
require "sidekiq/scheduled"
require "digest"
module GitLab
......@@ -76,6 +77,28 @@ module GitLab
self
end
def probe_future_sets
now = Time.now.to_f
with_sidekiq do
Sidekiq.redis do |conn|
Sidekiq::Scheduled::SETS.each do |set|
# Default to 0; if all jobs are due in the future, there is no "negative" delay.
delay = 0
entry = conn.zrangebyscore(set, "-inf", now.to_s, limit: [0, 1], withscores: true).first
# entry[0] is the job as JSON, entry[1] is the score (timestamp) when it should execute
delay = now - entry[1] if entry
@metrics.add("sidekiq_#{set}_set_processing_delay_seconds", delay)
# zcount is O(log(N)) (prob. binary search), so is still quick even with large sets
@metrics.add("sidekiq_#{set}_set_backlog_count",
conn.zcount(set, "-inf", now.to_s))
end
end
end
end
# Count worker classes present in Sidekiq queues. This only looks at the
# first PROBE_JOBS_LIMIT jobs in each queue. This means that we run a
# single LRANGE command for each queue, which does not block other
......
module GitLab
module Exporter
VERSION = "11.0.1".freeze
VERSION = "11.1.0".freeze
end
end
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment