Commit c071e828 authored by Adrien Kohlbecker's avatar Adrien Kohlbecker Committed by Stan Hu

Rename gitlab-monitor to gitlab-exporter

parent 41a32d9e
.bundle .bundle
config/gitlab-monitor*.yml config/gitlab-exporter*.yml
PATH PATH
remote: . remote: .
specs: specs:
gitlab-monitor (4.2.0) gitlab-exporter (4.2.0)
connection_pool (~> 2.2.1) connection_pool (~> 2.2.1)
pg (~> 1.1) pg (~> 1.1)
quantile (~> 0.2.0) quantile (~> 0.2.0)
...@@ -66,7 +66,7 @@ PLATFORMS ...@@ -66,7 +66,7 @@ PLATFORMS
ruby ruby
DEPENDENCIES DEPENDENCIES
gitlab-monitor! gitlab-exporter!
rspec (~> 3.5) rspec (~> 3.5)
rspec-expectations (~> 3.7.0) rspec-expectations (~> 3.7.0)
rubocop (~> 0.42) rubocop (~> 0.42)
......
## Introduction ## Introduction
gitlab-monitor is a [Prometheus Web exporter] that does the following: gitlab-exporter is a [Prometheus Web exporter] that does the following:
1. Collects GitLab production metrics via custom probes defined in a [YAML 1. Collects GitLab production metrics via custom probes defined in a [YAML
configuration file](config/gitlab-monitor.yml.example). configuration file](config/gitlab-exporter.yml.example).
2. Custom probes gather measurements in the form of key/value pairs. 2. Custom probes gather measurements in the form of key/value pairs.
3. For each probe, gitlab-monitor creates an HTTP endpoint `/<probe_name>` 3. For each probe, gitlab-exporter creates an HTTP endpoint `/<probe_name>`
(by default on port 9168) that delivers these metrics to a Prometheus scraper. (by default on port 9168) that delivers these metrics to a Prometheus scraper.
A central Prometheus process is configured to poll exporters at a specified A central Prometheus process is configured to poll exporters at a specified
...@@ -17,23 +17,23 @@ Below is a list of probes added by this exporter, and their corresponding ...@@ -17,23 +17,23 @@ Below is a list of probes added by this exporter, and their corresponding
metrics. metrics.
1. Database 1. Database
* [Per-table tuple stats](lib/gitlab_monitor/database/tuple_stats.rb) -- * [Per-table tuple stats](lib/gitlab_exporter/database/tuple_stats.rb) --
`gitlab_database_stat_table_*` `gitlab_database_stat_table_*`
* [Row count queries](lib/gitlab_monitor/database/row_count.rb) -- * [Row count queries](lib/gitlab_exporter/database/row_count.rb) --
`gitlab_database_rows` `gitlab_database_rows`
* [CI builds](lib/gitlab_monitor/database/ci_builds.rb) -- * [CI builds](lib/gitlab_exporter/database/ci_builds.rb) --
`ci_pending_builds`, `ci_created_builds`, `ci_stale_builds`, `ci_pending_builds`, `ci_created_builds`, `ci_stale_builds`,
`ci_running_builds` `ci_running_builds`
* [Bloat](lib/gitlab_monitor/database/bloat.rb) -- * [Bloat](lib/gitlab_exporter/database/bloat.rb) --
`gitlab_database_bloat_$type_$key` with type `btree` (index bloat) or `table` `gitlab_database_bloat_$type_$key` with type `btree` (index bloat) or `table`
(table bloat) and keys `bloat_ratio bloat_size extra_size real_size` (see below) (table bloat) and keys `bloat_ratio bloat_size extra_size real_size` (see below)
* [Remote mirrors](lib/gitlab_monitor/database/remote_mirrors.rb) -- * [Remote mirrors](lib/gitlab_exporter/database/remote_mirrors.rb) --
`project_remote_mirror_last_successful_update_time_seconds`, `project_remote_mirror_last_update_time_seconds` `project_remote_mirror_last_successful_update_time_seconds`, `project_remote_mirror_last_update_time_seconds`
1. Git 1. Git
* [git pull/push timings](lib/gitlab_monitor/git.rb) -- * [git pull/push timings](lib/gitlab_exporter/git.rb) --
`git_pull_time_milliseconds`, `git_push_time_milliseconds` `git_pull_time_milliseconds`, `git_push_time_milliseconds`
* git processes stats (see Process below) * git processes stats (see Process below)
1. [Process](lib/gitlab_monitor/process.rb) 1. [Process](lib/gitlab_exporter/process.rb)
* CPU time -- `process_cpu_seconds_total` * CPU time -- `process_cpu_seconds_total`
* Start time -- `process_start_time_seconds` * Start time -- `process_start_time_seconds`
* Count -- `process_count` * Count -- `process_count`
...@@ -50,7 +50,7 @@ metrics. ...@@ -50,7 +50,7 @@ metrics.
* `process_smaps_private_dirty_bytes` * `process_smaps_private_dirty_bytes`
* `process_smaps_swap_bytes` * `process_smaps_swap_bytes`
* `process_smaps_pss_bytes` * `process_smaps_pss_bytes`
1. [Sidekiq](lib/gitlab_monitor/sidekiq.rb) 1. [Sidekiq](lib/gitlab_exporter/sidekiq.rb)
* Stats * Stats
* `sidekiq_jobs_processed_total` * `sidekiq_jobs_processed_total`
* `sidekiq_jobs_failed_total` * `sidekiq_jobs_failed_total`
...@@ -72,19 +72,19 @@ metrics. ...@@ -72,19 +72,19 @@ metrics.
### Setup with GitLab Development Kit ### Setup with GitLab Development Kit
gitlab-monitor can be setup with the [GitLab Development Kit] for development. gitlab-exporter can be setup with the [GitLab Development Kit] for development.
When using the gitlab-monitor CLI, you'll need to set the `--db-conn` flag to When using the gitlab-exporter CLI, you'll need to set the `--db-conn` flag to
connect to the PostgreSQL instance in your GDK folder. For example: connect to the PostgreSQL instance in your GDK folder. For example:
``` ```
bin/gitlab-mon row-counts --db-conn="dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql" bin/gitlab-exporter row-counts --db-conn="dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql"
``` ```
### Running gitlab-monitor as a Web exporter ### Running gitlab-exporter as a Web exporter
When serving the pages on `localhost`, you'll need to edit the YAML When serving the pages on `localhost`, you'll need to edit the YAML
configuration file. An example can be found under configuration file. An example can be found under
[`config/gitlab-monitor.yml.example`](config/gitlab-monitor.yml.example). For [`config/gitlab-exporter.yml.example`](config/gitlab-exporter.yml.example). For
each probe that has to connect to the database, set the `connection_string` to each probe that has to connect to the database, set the `connection_string` to
`dbname=gitlabhq_development `dbname=gitlabhq_development
host=/Users/<user>/gitlab-development-kit/postgresql` host=/Users/<user>/gitlab-development-kit/postgresql`
...@@ -92,7 +92,7 @@ host=/Users/<user>/gitlab-development-kit/postgresql` ...@@ -92,7 +92,7 @@ host=/Users/<user>/gitlab-development-kit/postgresql`
Once you have this configured, you can then run: Once you have this configured, you can then run:
``` ```
bin/gitlab-mon web -c config/gitlab-monitor.yml bin/gitlab-exporter web -c config/gitlab-exporter.yml
``` ```
Once running, you can point your browser or curl to the following URLs: Once running, you can point your browser or curl to the following URLs:
...@@ -120,7 +120,7 @@ the error. ...@@ -120,7 +120,7 @@ the error.
## Contributing ## Contributing
gitlab-monitor is an open source project and we are very happy to accept community contributions. Please refer to [CONTRIBUTING.md](/CONTRIBUTING.md) for details. gitlab-exporter is an open source project and we are very happy to accept community contributions. Please refer to [CONTRIBUTING.md](/CONTRIBUTING.md) for details.
[Prometheus Web exporter]: https://prometheus.io/docs/instrumenting/exporters/ [Prometheus Web exporter]: https://prometheus.io/docs/instrumenting/exporters/
[GitLab Development Kit]: https://gitlab.com/gitlab-org/gitlab-development-kit [GitLab Development Kit]: https://gitlab.com/gitlab-org/gitlab-development-kit
...@@ -3,13 +3,13 @@ ...@@ -3,13 +3,13 @@
$LOAD_PATH.unshift File.expand_path("../../lib", __FILE__) $LOAD_PATH.unshift File.expand_path("../../lib", __FILE__)
require "optparse" require "optparse"
require "gitlab_monitor" require "gitlab_exporter"
def main def main
clazz = GitLab::Monitor::CLI.for(ARGV.shift) clazz = GitLab::Exporter::CLI.for(ARGV.shift)
runner = clazz.new(ARGV) runner = clazz.new(ARGV)
runner.run runner.run
rescue GitLab::Monitor::CLI::InvalidCLICommand => e rescue GitLab::Exporter::CLI::InvalidCLICommand => e
puts e puts e
exit 1 exit 1
end end
......
lib = File.expand_path("../lib", __FILE__) lib = File.expand_path("../lib", __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require "gitlab_monitor/version" require "gitlab_exporter/version"
Gem::Specification.new do |s| Gem::Specification.new do |s|
s.name = "gitlab-monitor" s.name = "gitlab-exporter"
s.version = GitLab::Monitor::VERSION s.version = GitLab::Exporter::VERSION
s.date = "2016-07-27" s.date = "2016-07-27"
s.summary = "GitLab monitoring tools" s.summary = "GitLab metrics exporter"
s.description = "GitLab monitoring tools to use with prometheus" s.description = "GitLab metrics exporter to use with prometheus"
s.authors = ["Pablo Carranza"] s.authors = ["Pablo Carranza"]
s.email = "pablo@gitlab.com" s.email = "pablo@gitlab.com"
s.files = `git ls-files -z`.split("\x0") s.files = `git ls-files -z`.split("\x0")
s.executables = ["gitlab-mon"] s.executables = ["gitlab-exporter"]
s.test_files = s.files.grep(%r{^(spec)/}) s.test_files = s.files.grep(%r{^(spec)/})
s.require_paths = ["lib"] s.require_paths = ["lib"]
......
module GitLab
# GitLab Monitoring
module Exporter
autoload :CLI, "gitlab_exporter/cli"
autoload :TimeTracker, "gitlab_exporter/util"
autoload :Utils, "gitlab_exporter/util"
autoload :PrometheusMetrics, "gitlab_exporter/prometheus"
autoload :Utils, "gitlab_exporter/util"
autoload :Git, "gitlab_exporter/git"
autoload :GitProber, "gitlab_exporter/git"
autoload :GitProcessProber, "gitlab_exporter/git"
autoload :Database, "gitlab_exporter/database"
autoload :ProcessProber, "gitlab_exporter/process"
autoload :WebExporter, "gitlab_exporter/web_exporter"
autoload :Prober, "gitlab_exporter/prober"
autoload :SidekiqProber, "gitlab_exporter/sidekiq"
end
end
require "yaml" require "yaml"
module GitLab module GitLab
module Monitor module Exporter
# Stores runner classes in a single place # Stores runner classes in a single place
# #
# The entry point is the module method "for" which takes the name of a runner. # The entry point is the module method "for" which takes the name of a runner.
# In case the runner is invalid it will return a NullRunner which fails with an # In case the runner is invalid it will return a NullRunner which fails with an
# InvalidCLICommand error, which contains the general application usage instructions. # InvalidCLICommand error, which contains the general application usage instructions.
module CLI module CLI
EXECUTABLE_NAME = "gitlab-mon".freeze EXECUTABLE_NAME = "gitlab-exporter".freeze
def self.for(name) def self.for(name)
commands.fetch(name, NullRunner) commands.fetch(name, NullRunner)
...@@ -24,7 +24,7 @@ module GitLab ...@@ -24,7 +24,7 @@ module GitLab
def run def run
fail InvalidCLICommand.new("Usage: #{EXECUTABLE_NAME} <command> [options] [arguments...]\n\n"\ fail InvalidCLICommand.new("Usage: #{EXECUTABLE_NAME} <command> [options] [arguments...]\n\n"\
"Available commands are: #{GitLab::Monitor::CLI.commands.keys.join(', ')}") "Available commands are: #{GitLab::Exporter::CLI.commands.keys.join(', ')}")
end end
end end
...@@ -50,10 +50,10 @@ module GitLab ...@@ -50,10 +50,10 @@ module GitLab
def run def run
validate! validate!
::GitLab::Monitor::GitProber.new(labels: labels, source: source) ::GitLab::Exporter::GitProber.new(labels: labels, source: source)
.probe_pull .probe_pull
.probe_push .probe_push
.write_to(@target) .write_to(@target)
end end
def options(args) def options(args)
...@@ -106,9 +106,9 @@ module GitLab ...@@ -106,9 +106,9 @@ module GitLab
def run def run
validate! validate!
::GitLab::Monitor::Database::TuplesProber.new(connection_string: @db_connection_string) ::GitLab::Exporter::Database::TuplesProber.new(connection_string: @db_connection_string)
.probe_db .probe_db
.write_to(@target) .write_to(@target)
end end
private private
...@@ -148,9 +148,9 @@ module GitLab ...@@ -148,9 +148,9 @@ module GitLab
def run def run
validate! validate!
::GitLab::Monitor::Database::RowCountProber.new(connection_string: @db_connection_string) ::GitLab::Exporter::Database::RowCountProber.new(connection_string: @db_connection_string)
.probe_db .probe_db
.write_to(@target) .write_to(@target)
end end
private private
...@@ -235,11 +235,11 @@ module GitLab ...@@ -235,11 +235,11 @@ module GitLab
end end
def run def run
::GitLab::Monitor::ProcessProber.new(pid_or_pattern: @pid || @pattern, name: @name, quantiles: @quantiles) ::GitLab::Exporter::ProcessProber.new(pid_or_pattern: @pid || @pattern, name: @name, quantiles: @quantiles)
.probe_stat .probe_stat
.probe_count .probe_count
.probe_smaps .probe_smaps
.write_to(@target) .write_to(@target)
end end
end end
...@@ -273,13 +273,13 @@ module GitLab ...@@ -273,13 +273,13 @@ module GitLab
def run def run
validate! validate!
::GitLab::Monitor::SidekiqProber.new(redis_url: @redis_url) ::GitLab::Exporter::SidekiqProber.new(redis_url: @redis_url)
.probe_stats .probe_stats
.probe_queues .probe_queues
.probe_jobs .probe_jobs
.probe_workers .probe_workers
.probe_retries .probe_retries
.write_to(@target) .write_to(@target)
end end
private private
...@@ -317,9 +317,9 @@ module GitLab ...@@ -317,9 +317,9 @@ module GitLab
end end
def run def run
::GitLab::Monitor::GitProcessProber.new(quantiles: @quantiles) ::GitLab::Exporter::GitProcessProber.new(quantiles: @quantiles)
.probe_git .probe_git
.write_to(@target) .write_to(@target)
end end
end end
......
module GitLab
module Exporter
# Database-related classes
module Database
autoload :Base, "gitlab_exporter/database/base"
autoload :CiBuildsProber, "gitlab_exporter/database/ci_builds"
autoload :TuplesProber, "gitlab_exporter/database/tuple_stats"
autoload :RowCountProber, "gitlab_exporter/database/row_count"
autoload :BloatProber, "gitlab_exporter/database/bloat"
autoload :RemoteMirrorsProber, "gitlab_exporter/database/remote_mirrors"
end
end
end
...@@ -2,7 +2,7 @@ require "pg" ...@@ -2,7 +2,7 @@ require "pg"
require "connection_pool" require "connection_pool"
module GitLab module GitLab
module Monitor module Exporter
module Database module Database
# An abstract class for interacting with DB # An abstract class for interacting with DB
# #
......
module GitLab module GitLab
module Monitor module Exporter
module Database module Database
# Helper to collect bloat metrics. # Helper to collect bloat metrics.
class BloatCollector < Base class BloatCollector < Base
......
module GitLab module GitLab
module Monitor module Exporter
module Database module Database
# A helper class to collect CI builds metrics. # A helper class to collect CI builds metrics.
class CiBuildsCollector < Base # rubocop:disable Metrics/ClassLength class CiBuildsCollector < Base # rubocop:disable Metrics/ClassLength
......
module GitLab module GitLab
module Monitor module Exporter
module Database module Database
# A helper class to collect remote mirrors metrics. # A helper class to collect remote mirrors metrics.
class RemoteMirrorsCollector < Base class RemoteMirrorsCollector < Base
......
require "set" require "set"
module GitLab module GitLab
module Monitor module Exporter
module Database module Database
# A helper class that executes the query its given and returns an int of # A helper class that executes the query its given and returns an int of
# the row count # the row count
......
module GitLab module GitLab
module Monitor module Exporter
module Database module Database
# A helper class to collect tuple stats from the database # A helper class to collect tuple stats from the database
# #
......
require "open3" require "open3"
module GitLab module GitLab
module Monitor module Exporter
# Git monitoring helping class # Git monitoring helping class
# #
# Takes a repository path for construction and provides 2 main methods: # Takes a repository path for construction and provides 2 main methods:
......
...@@ -26,7 +26,7 @@ require_relative "memstats/mapping" ...@@ -26,7 +26,7 @@ require_relative "memstats/mapping"
# size 149,281,668 kB # size 149,281,668 kB
# swap 1,719,792 kB # swap 1,719,792 kB
module GitLab module GitLab
module Monitor module Exporter
module MemStats module MemStats
# Aggregates all metrics for a single PID in /proc/<pid>/smaps # Aggregates all metrics for a single PID in /proc/<pid>/smaps
class Aggregator class Aggregator
......
# frozen_string_literal: true # frozen_string_literal: true
module GitLab module GitLab
module Monitor module Exporter
module MemStats module MemStats
# Parses one entry in /proc/[pid]/smaps. For example: # Parses one entry in /proc/[pid]/smaps. For example:
# #
......
module GitLab module GitLab
module Monitor module Exporter
# A class to combine multiple probers into one # A class to combine multiple probers into one
class Prober class Prober
def initialize(prober_opts, metrics: PrometheusMetrics.new) def initialize(prober_opts, metrics: PrometheusMetrics.new)
...@@ -30,7 +30,7 @@ module GitLab ...@@ -30,7 +30,7 @@ module GitLab
@prober_opts.each do |probe_name, params| @prober_opts.each do |probe_name, params|
prober_class_name = params[:class_name] || Utils.camel_case_string("#{probe_name}_prober") prober_class_name = params[:class_name] || Utils.camel_case_string("#{probe_name}_prober")
klass = prober_class_name.split("::").reduce(GitLab::Monitor) { |a, e| a.const_get(e) } klass = prober_class_name.split("::").reduce(GitLab::Exporter) { |a, e| a.const_get(e) }
params[:class] = klass params[:class] = klass
end end
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
require_relative "memstats" require_relative "memstats"
module GitLab module GitLab
module Monitor module Exporter
# A helper class to extract memory info from /proc/<pid>/status # A helper class to extract memory info from /proc/<pid>/status
# #
...@@ -95,14 +95,14 @@ module GitLab ...@@ -95,14 +95,14 @@ module GitLab
def probe_smaps def probe_smaps
@pids.each do |pid| @pids.each do |pid|
stats = ::GitLab::Monitor::MemStats::Aggregator.new(pid) stats = ::GitLab::Exporter::MemStats::Aggregator.new(pid)
next unless stats.valid? next unless stats.valid?
labels = { name: @name.downcase } labels = { name: @name.downcase }
labels[:pid] = pid unless @use_quantiles labels[:pid] = pid unless @use_quantiles
::GitLab::Monitor::MemStats::Mapping::FIELDS.each do |field| ::GitLab::Exporter::MemStats::Mapping::FIELDS.each do |field|
value = stats.totals[field] value = stats.totals[field]
if value >= 0 if value >= 0
......
require "quantile" require "quantile"
module GitLab module GitLab
module Monitor module Exporter
# Prometheus metrics container # Prometheus metrics container
# #
# Provides a simple API to `add` metrics and then turn them `to_s` which will just # Provides a simple API to `add` metrics and then turn them `to_s` which will just
......
...@@ -2,7 +2,7 @@ require "sidekiq/api" ...@@ -2,7 +2,7 @@ require "sidekiq/api"
require "digest" require "digest"
module GitLab module GitLab
module Monitor module Exporter
# A prober for Sidekiq queues # A prober for Sidekiq queues
# #
# It takes the Redis URL Sidekiq is connected to # It takes the Redis URL Sidekiq is connected to
......
# frozen_string_literal: true # frozen_string_literal: true
module GitLab module GitLab
module Monitor module Exporter