Commit 60d6c720 authored by Ahmad Sherif's avatar Ahmad Sherif
Browse files

Merge branch 'future-sets-metrics2' into 'master'

Adds probe_future_sets to report on delays in scheduled and retry sets

See merge request gitlab-org/gitlab-exporter!147
parents 20d8cc06 ff25289a
Pipeline #204690 passed with stage
in 1 minute and 39 seconds
PATH
remote: .
specs:
gitlab-exporter (11.0.1)
gitlab-exporter (11.1.0)
connection_pool (= 2.2.5)
pg (= 1.2.3)
puma (= 5.3.2)
......
......@@ -34,7 +34,7 @@ metrics.
`git_pull_time_milliseconds`, `git_push_time_milliseconds`
* git processes stats (see Process below)
1. [Sidekiq](lib/gitlab_exporter/sidekiq.rb)
* Stats
* Stats (probe_stats)
* `sidekiq_jobs_processed_total`
* `sidekiq_jobs_failed_total`
* `sidekiq_jobs_enqueued_size`
......@@ -44,14 +44,21 @@ metrics.
* `sidekiq_default_queue_latency_seconds`
* `sidekiq_processes_size`
* `sidekiq_workers_size`
* Queues
* Queues (probe_queues)
* `sidekiq_queue_size`
* `sidekiq_queue_paused`
* `sidekiq_queue_latency_seconds`
* Jobs
* Jobs (probe_jobs_limit)
* `sidekiq_enqueued_jobs`
* Workers (probe_workers)
* `sidekiq_running_jobs`
* Retries (probe_retries)
* `sidekiq_to_be_retried_jobs`
* Future Sets (probe_future_sets)
* `sidekiq_schedule_set_processing_delay_seconds`
* `sidekiq_schedule_set_backlog_count`
* `sidekiq_retry_set_processing_delay_seconds`
* `sidekiq_retry_set_backlog_count`
### Setup with GitLab Development Kit
......
require "sidekiq/api"
require "sidekiq/scheduled"
require "digest"
module GitLab
......@@ -76,6 +77,27 @@ module GitLab
self
end
def probe_future_sets
now = Time.now.to_f
with_sidekiq do
Sidekiq.redis do |conn|
Sidekiq::Scheduled::SETS.each do |set|
# Default to 0; if all jobs are due in the future, there is no "negative" delay.
delay = 0
_job, timestamp = conn.zrangebyscore(set, "-inf", now.to_s, limit: [0, 1], withscores: true).first
delay = now - timestamp if timestamp
@metrics.add("sidekiq_#{set}_set_processing_delay_seconds", delay)
# zcount is O(log(N)) (prob. binary search), so is still quick even with large sets
@metrics.add("sidekiq_#{set}_set_backlog_count",
conn.zcount(set, "-inf", now.to_s))
end
end
end
end
# Count worker classes present in Sidekiq queues. This only looks at the
# first PROBE_JOBS_LIMIT jobs in each queue. This means that we run a
# single LRANGE command for each queue, which does not block other
......
module GitLab
module Exporter
VERSION = "11.0.1".freeze
VERSION = "11.1.0".freeze
end
end
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment