Remove Sidekiq `probe_jobs` probe

See merge request gitlab-org/gitlab-exporter!145

This was generally fast but had two unfortunate effects that made it
impractical in production:

1. The script got slower the larger the queue, as it scanned all jobs in
   queue.
2. Because it was a Lua script, it blocked Redis execution entirely
   while running.

The combination of these two can lead to a catastrophic case. When a
queue is very long, that's precisely the time when we want jobs to be
processed quickly. But we'd see that jobs were blocked by the metrics
script, which is the opposite of what we need at that time.

Add probe_jobs_limit probe for Sidekiq

See merge request gitlab-org/gitlab-exporter!143

With large job sizes, we can spend a lot of time in both Ruby and Redis
getting 10,000 jobs. 1,000 gives us an order of magnitude of headroom.

This does the same as #probe_jobs, but only looks at the first
PROBE_JOBS_LIMIT jobs in each queue. This means that we run a single
LRANGE command for each queue, which does not block other commands. For
queues over PROBE_JOBS_LIMIT in size, this means that we will not have
completely accurate statistics, but the probe performance will also not
degrade as the queue gets larger.

DO NOT USE this and probe_jobs together, as they export the same
metric (sidekiq_enqueued_jobs).

Add explicit dast stage

See merge request gitlab-org/gitlab-exporter!144

Remove Unicorn references

Closes #81

See merge request gitlab-org/gitlab-exporter!142

Pin gem dependencies

Closes #82

See merge request gitlab-org/gitlab-exporter!141

So that we can more selectively update them
in the future.

Bump Puma to 5.3.2

See merge request gitlab-org/gitlab-exporter!140

Add missing bundler require on start-up

See merge request gitlab-org/gitlab-exporter!138

This doesn't appear to be required when running standalone,
but was reported causing problems when running on Puma in
Omnibus:
https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6090

Remove ci_repeated_commands_builds metric

Closes #77

See merge request gitlab-org/gitlab-exporter!136

The column ci_builds.commands has been removed from Rails and has
been deprecated for long before that, thus this metric is no longer functioning.

Related to https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54564

Fixes https://gitlab.com/gitlab-org/gitlab-exporter/-/issues/77

Remove info about process probe

See merge request gitlab-org/gitlab-exporter!135

Add ruby GC stats probe

See merge request gitlab-org/gitlab-exporter!130

Notify code producing `process_*` metrics is obsolete

See merge request gitlab-org/gitlab-exporter!126

Aleksei Lipniagov authored Jan 26, 2021 and Ahmad Sherif committed Jan 26, 2021

We remove `process_*` metrics probing from Omnibus, so this code should
be no longer needed.

Optimise `Sidekiq probe`

See merge request gitlab-org/gitlab-exporter!133

Make Puma an explicit dependency

See merge request gitlab-org/gitlab-exporter!134

Matthias Käppler authored Jan 25, 2021 and Ahmad Sherif committed Jan 25, 2021

This also bumps Puma from v4.x (which was bundled by Sinatra) to 5.1.1.

Optimise `DB probe`

See merge request gitlab-org/gitlab-exporter!132

This makes:
- To re-use Redis connection instead of openning
  it each time a probe is requested and leaking
  the connection pool

This makes to scrape current Ruby process
of `GC.stat` to be able to monitor the memory
pressure impact.