Merge branch 'future-sets-metrics2' into 'master'

Adds probe_future_sets to report on delays in scheduled and retry sets

See merge request gitlab-org/gitlab-exporter!147

Gemfile.lock

 PATH
   remote: .
   specs:
-    gitlab-exporter (11.0.1)
+    gitlab-exporter (11.1.0)
       connection_pool (= 2.2.5)
       pg (= 1.2.3)
       puma (= 5.3.2)

README.md

@@ -34,7 +34,7 @@ metrics.
       `git_pull_time_milliseconds`, `git_push_time_milliseconds`
     * git processes stats (see Process below)
 1. [Sidekiq](lib/gitlab_exporter/sidekiq.rb)
-	* Stats
+	* Stats (probe_stats)
 	  * `sidekiq_jobs_processed_total`
 	  * `sidekiq_jobs_failed_total`
 	  * `sidekiq_jobs_enqueued_size`
@@ -44,14 +44,21 @@ metrics.
 	  * `sidekiq_default_queue_latency_seconds`
 	  * `sidekiq_processes_size`
 	  * `sidekiq_workers_size`
-	* Queues
+	* Queues (probe_queues)
 	  * `sidekiq_queue_size`
 	  * `sidekiq_queue_paused`
 	  * `sidekiq_queue_latency_seconds`
-	* Jobs
+	* Jobs (probe_jobs_limit)
 	  * `sidekiq_enqueued_jobs`
+  * Workers (probe_workers)
 	  * `sidekiq_running_jobs`
+  * Retries (probe_retries)
 	  * `sidekiq_to_be_retried_jobs`
+  * Future Sets (probe_future_sets)
+    * `sidekiq_schedule_set_processing_delay_seconds`
+    * `sidekiq_schedule_set_backlog_count`
+    * `sidekiq_retry_set_processing_delay_seconds`
+    * `sidekiq_retry_set_backlog_count`
 
 ### Setup with GitLab Development Kit
 

lib/gitlab_exporter/sidekiq.rb

 require "sidekiq/api"
+require "sidekiq/scheduled"
 require "digest"
 
 module GitLab
@@ -76,6 +77,27 @@ module GitLab
         self
       end
 
+      def probe_future_sets
+        now = Time.now.to_f
+        with_sidekiq do
+          Sidekiq.redis do |conn|
+            Sidekiq::Scheduled::SETS.each do |set|
+              # Default to 0; if all jobs are due in the future, there is no "negative" delay.
+              delay = 0
+
+              _job, timestamp = conn.zrangebyscore(set, "-inf", now.to_s, limit: [0, 1], withscores: true).first
+              delay = now - timestamp if timestamp
+
+              @metrics.add("sidekiq_#{set}_set_processing_delay_seconds", delay)
+
+              # zcount is O(log(N)) (prob. binary search), so is still quick even with large sets
+              @metrics.add("sidekiq_#{set}_set_backlog_count",
+                           conn.zcount(set, "-inf", now.to_s))
+            end
+          end
+        end
+      end
+
       # Count worker classes present in Sidekiq queues. This only looks at the
       # first PROBE_JOBS_LIMIT jobs in each queue. This means that we run a
       # single LRANGE command for each queue, which does not block other

lib/gitlab_exporter/version.rb

 module GitLab
   module Exporter
-    VERSION = "11.0.1".freeze
+    VERSION = "11.1.0".freeze
   end
 end