Commit c071e828 authored by Adrien Kohlbecker's avatar Adrien Kohlbecker Committed by Stan Hu

Rename gitlab-monitor to gitlab-exporter

parent 41a32d9e
.bundle
config/gitlab-monitor*.yml
config/gitlab-exporter*.yml
PATH
remote: .
specs:
gitlab-monitor (4.2.0)
gitlab-exporter (4.2.0)
connection_pool (~> 2.2.1)
pg (~> 1.1)
quantile (~> 0.2.0)
......@@ -66,7 +66,7 @@ PLATFORMS
ruby
DEPENDENCIES
gitlab-monitor!
gitlab-exporter!
rspec (~> 3.5)
rspec-expectations (~> 3.7.0)
rubocop (~> 0.42)
......
## Introduction
gitlab-monitor is a [Prometheus Web exporter] that does the following:
gitlab-exporter is a [Prometheus Web exporter] that does the following:
1. Collects GitLab production metrics via custom probes defined in a [YAML
configuration file](config/gitlab-monitor.yml.example).
configuration file](config/gitlab-exporter.yml.example).
2. Custom probes gather measurements in the form of key/value pairs.
3. For each probe, gitlab-monitor creates an HTTP endpoint `/<probe_name>`
3. For each probe, gitlab-exporter creates an HTTP endpoint `/<probe_name>`
(by default on port 9168) that delivers these metrics to a Prometheus scraper.
A central Prometheus process is configured to poll exporters at a specified
......@@ -17,23 +17,23 @@ Below is a list of probes added by this exporter, and their corresponding
metrics.
1. Database
* [Per-table tuple stats](lib/gitlab_monitor/database/tuple_stats.rb) --
* [Per-table tuple stats](lib/gitlab_exporter/database/tuple_stats.rb) --
`gitlab_database_stat_table_*`
* [Row count queries](lib/gitlab_monitor/database/row_count.rb) --
* [Row count queries](lib/gitlab_exporter/database/row_count.rb) --
`gitlab_database_rows`
* [CI builds](lib/gitlab_monitor/database/ci_builds.rb) --
* [CI builds](lib/gitlab_exporter/database/ci_builds.rb) --
`ci_pending_builds`, `ci_created_builds`, `ci_stale_builds`,
`ci_running_builds`
* [Bloat](lib/gitlab_monitor/database/bloat.rb) --
* [Bloat](lib/gitlab_exporter/database/bloat.rb) --
`gitlab_database_bloat_$type_$key` with type `btree` (index bloat) or `table`
(table bloat) and keys `bloat_ratio bloat_size extra_size real_size` (see below)
* [Remote mirrors](lib/gitlab_monitor/database/remote_mirrors.rb) --
* [Remote mirrors](lib/gitlab_exporter/database/remote_mirrors.rb) --
`project_remote_mirror_last_successful_update_time_seconds`, `project_remote_mirror_last_update_time_seconds`
1. Git
* [git pull/push timings](lib/gitlab_monitor/git.rb) --
* [git pull/push timings](lib/gitlab_exporter/git.rb) --
`git_pull_time_milliseconds`, `git_push_time_milliseconds`
* git processes stats (see Process below)
1. [Process](lib/gitlab_monitor/process.rb)
1. [Process](lib/gitlab_exporter/process.rb)
* CPU time -- `process_cpu_seconds_total`
* Start time -- `process_start_time_seconds`
* Count -- `process_count`
......@@ -50,7 +50,7 @@ metrics.
* `process_smaps_private_dirty_bytes`
* `process_smaps_swap_bytes`
* `process_smaps_pss_bytes`
1. [Sidekiq](lib/gitlab_monitor/sidekiq.rb)
1. [Sidekiq](lib/gitlab_exporter/sidekiq.rb)
* Stats
* `sidekiq_jobs_processed_total`
* `sidekiq_jobs_failed_total`
......@@ -72,19 +72,19 @@ metrics.
### Setup with GitLab Development Kit
gitlab-monitor can be setup with the [GitLab Development Kit] for development.
When using the gitlab-monitor CLI, you'll need to set the `--db-conn` flag to
gitlab-exporter can be setup with the [GitLab Development Kit] for development.
When using the gitlab-exporter CLI, you'll need to set the `--db-conn` flag to
connect to the PostgreSQL instance in your GDK folder. For example:
```
bin/gitlab-mon row-counts --db-conn="dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql"
bin/gitlab-exporter row-counts --db-conn="dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql"
```
### Running gitlab-monitor as a Web exporter
### Running gitlab-exporter as a Web exporter
When serving the pages on `localhost`, you'll need to edit the YAML
configuration file. An example can be found under
[`config/gitlab-monitor.yml.example`](config/gitlab-monitor.yml.example). For
[`config/gitlab-exporter.yml.example`](config/gitlab-exporter.yml.example). For
each probe that has to connect to the database, set the `connection_string` to
`dbname=gitlabhq_development
host=/Users/<user>/gitlab-development-kit/postgresql`
......@@ -92,7 +92,7 @@ host=/Users/<user>/gitlab-development-kit/postgresql`
Once you have this configured, you can then run:
```
bin/gitlab-mon web -c config/gitlab-monitor.yml
bin/gitlab-exporter web -c config/gitlab-exporter.yml
```
Once running, you can point your browser or curl to the following URLs:
......@@ -120,7 +120,7 @@ the error.
## Contributing
gitlab-monitor is an open source project and we are very happy to accept community contributions. Please refer to [CONTRIBUTING.md](/CONTRIBUTING.md) for details.
gitlab-exporter is an open source project and we are very happy to accept community contributions. Please refer to [CONTRIBUTING.md](/CONTRIBUTING.md) for details.
[Prometheus Web exporter]: https://prometheus.io/docs/instrumenting/exporters/
[GitLab Development Kit]: https://gitlab.com/gitlab-org/gitlab-development-kit
......@@ -3,13 +3,13 @@
$LOAD_PATH.unshift File.expand_path("../../lib", __FILE__)
require "optparse"
require "gitlab_monitor"
require "gitlab_exporter"
def main
clazz = GitLab::Monitor::CLI.for(ARGV.shift)
clazz = GitLab::Exporter::CLI.for(ARGV.shift)
runner = clazz.new(ARGV)
runner.run
rescue GitLab::Monitor::CLI::InvalidCLICommand => e
rescue GitLab::Exporter::CLI::InvalidCLICommand => e
puts e
exit 1
end
......
lib = File.expand_path("../lib", __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require "gitlab_monitor/version"
require "gitlab_exporter/version"
Gem::Specification.new do |s|
s.name = "gitlab-monitor"
s.version = GitLab::Monitor::VERSION
s.name = "gitlab-exporter"
s.version = GitLab::Exporter::VERSION
s.date = "2016-07-27"
s.summary = "GitLab monitoring tools"
s.description = "GitLab monitoring tools to use with prometheus"
s.summary = "GitLab metrics exporter"
s.description = "GitLab metrics exporter to use with prometheus"
s.authors = ["Pablo Carranza"]
s.email = "pablo@gitlab.com"
s.files = `git ls-files -z`.split("\x0")
s.executables = ["gitlab-mon"]
s.executables = ["gitlab-exporter"]
s.test_files = s.files.grep(%r{^(spec)/})
s.require_paths = ["lib"]
......
module GitLab
# GitLab Monitoring
module Exporter
autoload :CLI, "gitlab_exporter/cli"
autoload :TimeTracker, "gitlab_exporter/util"
autoload :Utils, "gitlab_exporter/util"
autoload :PrometheusMetrics, "gitlab_exporter/prometheus"
autoload :Utils, "gitlab_exporter/util"
autoload :Git, "gitlab_exporter/git"
autoload :GitProber, "gitlab_exporter/git"
autoload :GitProcessProber, "gitlab_exporter/git"
autoload :Database, "gitlab_exporter/database"
autoload :ProcessProber, "gitlab_exporter/process"
autoload :WebExporter, "gitlab_exporter/web_exporter"
autoload :Prober, "gitlab_exporter/prober"
autoload :SidekiqProber, "gitlab_exporter/sidekiq"
end
end
require "yaml"
module GitLab
module Monitor
module Exporter
# Stores runner classes in a single place
#
# The entry point is the module method "for" which takes the name of a runner.
# In case the runner is invalid it will return a NullRunner which fails with an
# InvalidCLICommand error, which contains the general application usage instructions.
module CLI
EXECUTABLE_NAME = "gitlab-mon".freeze
EXECUTABLE_NAME = "gitlab-exporter".freeze
def self.for(name)
commands.fetch(name, NullRunner)
......@@ -24,7 +24,7 @@ module GitLab
def run
fail InvalidCLICommand.new("Usage: #{EXECUTABLE_NAME} <command> [options] [arguments...]\n\n"\
"Available commands are: #{GitLab::Monitor::CLI.commands.keys.join(', ')}")
"Available commands are: #{GitLab::Exporter::CLI.commands.keys.join(', ')}")
end
end
......@@ -50,7 +50,7 @@ module GitLab
def run
validate!
::GitLab::Monitor::GitProber.new(labels: labels, source: source)
::GitLab::Exporter::GitProber.new(labels: labels, source: source)
.probe_pull
.probe_push
.write_to(@target)
......@@ -106,7 +106,7 @@ module GitLab
def run
validate!
::GitLab::Monitor::Database::TuplesProber.new(connection_string: @db_connection_string)
::GitLab::Exporter::Database::TuplesProber.new(connection_string: @db_connection_string)
.probe_db
.write_to(@target)
end
......@@ -148,7 +148,7 @@ module GitLab
def run
validate!
::GitLab::Monitor::Database::RowCountProber.new(connection_string: @db_connection_string)
::GitLab::Exporter::Database::RowCountProber.new(connection_string: @db_connection_string)
.probe_db
.write_to(@target)
end
......@@ -235,7 +235,7 @@ module GitLab
end
def run
::GitLab::Monitor::ProcessProber.new(pid_or_pattern: @pid || @pattern, name: @name, quantiles: @quantiles)
::GitLab::Exporter::ProcessProber.new(pid_or_pattern: @pid || @pattern, name: @name, quantiles: @quantiles)
.probe_stat
.probe_count
.probe_smaps
......@@ -273,7 +273,7 @@ module GitLab
def run
validate!
::GitLab::Monitor::SidekiqProber.new(redis_url: @redis_url)
::GitLab::Exporter::SidekiqProber.new(redis_url: @redis_url)
.probe_stats
.probe_queues
.probe_jobs
......@@ -317,7 +317,7 @@ module GitLab
end
def run
::GitLab::Monitor::GitProcessProber.new(quantiles: @quantiles)
::GitLab::Exporter::GitProcessProber.new(quantiles: @quantiles)
.probe_git
.write_to(@target)
end
......
module GitLab
module Exporter
# Database-related classes
module Database
autoload :Base, "gitlab_exporter/database/base"
autoload :CiBuildsProber, "gitlab_exporter/database/ci_builds"
autoload :TuplesProber, "gitlab_exporter/database/tuple_stats"
autoload :RowCountProber, "gitlab_exporter/database/row_count"
autoload :BloatProber, "gitlab_exporter/database/bloat"
autoload :RemoteMirrorsProber, "gitlab_exporter/database/remote_mirrors"
end
end
end
......@@ -2,7 +2,7 @@ require "pg"
require "connection_pool"
module GitLab
module Monitor
module Exporter
module Database
# An abstract class for interacting with DB
#
......
module GitLab
module Monitor
module Exporter
module Database
# Helper to collect bloat metrics.
class BloatCollector < Base
......
module GitLab
module Monitor
module Exporter
module Database
# A helper class to collect CI builds metrics.
class CiBuildsCollector < Base # rubocop:disable Metrics/ClassLength
......
module GitLab
module Monitor
module Exporter
module Database
# A helper class to collect remote mirrors metrics.
class RemoteMirrorsCollector < Base
......
require "set"
module GitLab
module Monitor
module Exporter
module Database
# A helper class that executes the query its given and returns an int of
# the row count
......
module GitLab
module Monitor
module Exporter
module Database
# A helper class to collect tuple stats from the database
#
......
require "open3"
module GitLab
module Monitor
module Exporter
# Git monitoring helping class
#
# Takes a repository path for construction and provides 2 main methods:
......
......@@ -26,7 +26,7 @@ require_relative "memstats/mapping"
# size 149,281,668 kB
# swap 1,719,792 kB
module GitLab
module Monitor
module Exporter
module MemStats
# Aggregates all metrics for a single PID in /proc/<pid>/smaps
class Aggregator
......
# frozen_string_literal: true
module GitLab
module Monitor
module Exporter
module MemStats
# Parses one entry in /proc/[pid]/smaps. For example:
#
......
module GitLab
module Monitor
module Exporter
# A class to combine multiple probers into one
class Prober
def initialize(prober_opts, metrics: PrometheusMetrics.new)
......@@ -30,7 +30,7 @@ module GitLab
@prober_opts.each do |probe_name, params|
prober_class_name = params[:class_name] || Utils.camel_case_string("#{probe_name}_prober")
klass = prober_class_name.split("::").reduce(GitLab::Monitor) { |a, e| a.const_get(e) }
klass = prober_class_name.split("::").reduce(GitLab::Exporter) { |a, e| a.const_get(e) }
params[:class] = klass
end
......
......@@ -3,7 +3,7 @@
require_relative "memstats"
module GitLab
module Monitor
module Exporter
# A helper class to extract memory info from /proc/<pid>/status
#
......@@ -95,14 +95,14 @@ module GitLab
def probe_smaps
@pids.each do |pid|
stats = ::GitLab::Monitor::MemStats::Aggregator.new(pid)
stats = ::GitLab::Exporter::MemStats::Aggregator.new(pid)
next unless stats.valid?
labels = { name: @name.downcase }
labels[:pid] = pid unless @use_quantiles
::GitLab::Monitor::MemStats::Mapping::FIELDS.each do |field|
::GitLab::Exporter::MemStats::Mapping::FIELDS.each do |field|
value = stats.totals[field]
if value >= 0
......
require "quantile"
module GitLab
module Monitor
module Exporter
# Prometheus metrics container
#
# Provides a simple API to `add` metrics and then turn them `to_s` which will just
......
......@@ -2,7 +2,7 @@ require "sidekiq/api"
require "digest"
module GitLab
module Monitor
module Exporter
# A prober for Sidekiq queues
#
# It takes the Redis URL Sidekiq is connected to
......
# frozen_string_literal: true
module GitLab
module Monitor
module Exporter
# Simple time wrapper that provides a to_i and wraps the execution result
TrackedResult = Struct.new(:result, :time) do
def to_i
......
module GitLab
module Monitor
module Exporter
VERSION = "4.2.0".freeze
end
end
......@@ -2,7 +2,7 @@ require "sinatra/base"
require "English"
module GitLab
module Monitor
module Exporter
# Metrics web exporter
class WebExporter < Sinatra::Base
# A middleware to kill the process if we exceeded a certain threshold
......
module GitLab
# GitLab Monitoring
module Monitor
autoload :CLI, "gitlab_monitor/cli"
autoload :TimeTracker, "gitlab_monitor/util"
autoload :Utils, "gitlab_monitor/util"
autoload :PrometheusMetrics, "gitlab_monitor/prometheus"
autoload :Utils, "gitlab_monitor/util"
autoload :Git, "gitlab_monitor/git"
autoload :GitProber, "gitlab_monitor/git"
autoload :GitProcessProber, "gitlab_monitor/git"
autoload :Database, "gitlab_monitor/database"
autoload :ProcessProber, "gitlab_monitor/process"
autoload :WebExporter, "gitlab_monitor/web_exporter"
autoload :Prober, "gitlab_monitor/prober"
autoload :SidekiqProber, "gitlab_monitor/sidekiq"
end
end
module GitLab
module Monitor
# Database-related classes
module Database
autoload :Base, "gitlab_monitor/database/base"
autoload :CiBuildsProber, "gitlab_monitor/database/ci_builds"
autoload :TuplesProber, "gitlab_monitor/database/tuple_stats"
autoload :RowCountProber, "gitlab_monitor/database/row_count"
autoload :BloatProber, "gitlab_monitor/database/bloat"
autoload :RemoteMirrorsProber, "gitlab_monitor/database/remote_mirrors"
end
end
end
require "spec_helper"
require "gitlab_monitor/cli"
require "gitlab_exporter/cli"
context "With valid pair of repositories" do
let(:repos) { GitRepoBuilder.new }
......@@ -8,21 +8,21 @@ context "With valid pair of repositories" do
repos.cleanup
end
describe GitLab::Monitor::CLI do
describe GitLab::Exporter::CLI do
it "returns the rigth parser" do
expect(GitLab::Monitor::CLI.for("git")).to be(GitLab::Monitor::CLI::GIT)
expect(GitLab::Exporter::CLI.for("git")).to be(GitLab::Exporter::CLI::GIT)
end
it "returns a null parser if it is not found" do
expect(GitLab::Monitor::CLI.for("invalid")).to be(GitLab::Monitor::CLI::NullRunner)
expect(GitLab::Exporter::CLI.for("invalid")).to be(GitLab::Exporter::CLI::NullRunner)
end
end
describe GitLab::Monitor::CLI::GIT do
describe GitLab::Exporter::CLI::GIT do
let(:output) { StringIO.new }
it "works end to end" do
args = CLIArgs.new([repos.cloned_repo, output])
ssh = GitLab::Monitor::CLI::GIT.new(args)
ssh = GitLab::Exporter::CLI::GIT.new(args)
ssh.run
output.rewind
expect(output.read).to match(/git_push_time_milliseconds \d+ \d+/)
......
require "spec_helper"
require "gitlab_monitor/database/bloat"
require "gitlab_exporter/database/bloat"
describe GitLab::Monitor::Database::BloatCollector do
describe GitLab::Exporter::Database::BloatCollector do
let(:connection_pool) { double("connection pool") }
let(:connection) { double("connection") }
......@@ -25,7 +25,7 @@ describe GitLab::Monitor::Database::BloatCollector do
end
end
describe GitLab::Monitor::Database::BloatProber do
describe GitLab::Exporter::Database::BloatProber do
let(:opts) { { bloat_types: %i(btree table) } }
let(:metrics) { double("PrometheusMetrics", add: nil) }
let(:collector) { double("BloatCollector", run: data) }
......
require "spec_helper"
require "gitlab_monitor/database/ci_builds"
require "gitlab_exporter/database/ci_builds"
# rubocop:disable Metrics/LineLength
describe GitLab::Monitor::Database do
describe GitLab::Exporter::Database do
let(:set_random_page_cost_query) { "SET random_page_cost" }
let(:builds_query_ee) { "SELECT BUILDS EE" }
let(:builds_query_ce) { "SELECT BUILDS CE" }
......@@ -80,18 +80,18 @@ describe GitLab::Monitor::Database do
end
before do
stub_const("GitLab::Monitor::Database::CiBuildsCollector::SET_RANDOM_PAGE_COST", set_random_page_cost_query)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::BUILDS_QUERY_EE", builds_query_ee)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::BUILDS_QUERY_CE", builds_query_ce)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::STALE_BUILDS_QUERY", stale_builds_query)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::PER_RUNNER_QUERY_EE", per_runner_query_ee)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::PER_RUNNER_QUERY_CE", per_runner_query_ce)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::MIRROR_COLUMN_QUERY", mirror_column_query)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::REPEATED_COMMANDS_QUERY_EE", repeated_commands_query_ee)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::REPEATED_COMMANDS_QUERY_CE", repeated_commands_query_ce)
stub_const("GitLab::Monitor::Database::CiBuildsCollector::UNARCHIVED_TRACES_QUERY", unarchived_traces_query)
allow_any_instance_of(GitLab::Monitor::Database::CiBuildsCollector).to receive(:connection_pool).and_return(connection_pool)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::SET_RANDOM_PAGE_COST", set_random_page_cost_query)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::BUILDS_QUERY_EE", builds_query_ee)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::BUILDS_QUERY_CE", builds_query_ce)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::STALE_BUILDS_QUERY", stale_builds_query)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::PER_RUNNER_QUERY_EE", per_runner_query_ee)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::PER_RUNNER_QUERY_CE", per_runner_query_ce)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::MIRROR_COLUMN_QUERY", mirror_column_query)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::REPEATED_COMMANDS_QUERY_EE", repeated_commands_query_ee)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::REPEATED_COMMANDS_QUERY_CE", repeated_commands_query_ce)
stub_const("GitLab::Exporter::Database::CiBuildsCollector::UNARCHIVED_TRACES_QUERY", unarchived_traces_query)
allow_any_instance_of(GitLab::Exporter::Database::CiBuildsCollector).to receive(:connection_pool).and_return(connection_pool)
allow(connection_pool).to receive(:with).and_yield(connection)
allow(connection).to receive(:transaction).and_yield(connection)
......@@ -153,7 +153,7 @@ describe GitLab::Monitor::Database do
allow(connection).to receive(:exec).with(unarchived_traces_query_with_time).and_return([{ "count" => 10 }])
end
describe GitLab::Monitor::Database::CiBuildsCollector do
describe GitLab::Exporter::Database::CiBuildsCollector do
let(:collector) do
described_class.new(connection_string: "host=localhost",
allowed_repeated_commands_count: allowed_repeated_commands_count,
......@@ -269,7 +269,7 @@ describe GitLab::Monitor::Database do
end
end
describe GitLab::Monitor::Database::CiBuildsProber do
describe GitLab::Exporter::Database::CiBuildsProber do
let(:writer) { StringIO.new }
let(:prober) do
opts = { connection_string: "host=localhost",
......@@ -277,11 +277,11 @@ describe GitLab::Monitor::Database do
created_builds_counting_disabled: created_builds_counting_disabled,
unarchived_traces_offset_minutes: unarchived_traces_offset_minutes }
described_class.new(opts,
metrics: GitLab::Monitor::PrometheusMetrics.new(include_timestamp: false))
metrics: GitLab::Exporter::PrometheusMetrics.new(include_timestamp: false))
end
before do
allow_any_instance_of(GitLab::Monitor::Database::CiBuildsCollector).to receive(:connected?).and_return(true)
allow_any_instance_of(GitLab::Exporter::Database::CiBuildsCollector).to receive(:connected?).and_return(true)
end
shared_examples "metrics server" do
......
require "spec_helper"
require "gitlab_monitor/database/row_count"
require "gitlab_exporter/database/row_count"
describe GitLab::Monitor::Database::RowCountCollector do
describe GitLab::Exporter::Database::RowCountCollector do
let(:query) {
{ project_1: { select: :projects, where: "id=1" },
project_2: { select: :projects, where: "id=2" } }
......@@ -10,7 +10,7 @@ describe GitLab::Monitor::Database::RowCountCollector do
describe "#run" do
before do
stub_const("GitLab::Monitor::Database::RowCountCollector::QUERIES", query)
stub_const("GitLab::Exporter::Database::RowCountCollector::QUERIES", query)
allow(collector).to receive(:count_from_query_hash).with(query[:project_1]).and_return(3)
allow(collector).to receive(:count_from_query_hash).with(query[:project_2]).and_return(6)
......
require "spec_helper"
require "gitlab_monitor"
require "gitlab_exporter"
# rubocop:disable Metrics/LineLength
describe GitLab::Monitor::GitProcessProber do
describe GitLab::Exporter::GitProcessProber do
describe</