GitLab

alert when we see too many sidekiq exceptions for a controller. Addresses gitlab-com/infrastructure#4651

Currently we are steady at a bit above this threshold because of the
maxmemory setting. In order to silence the alert we will raise this
threshold to 90%. Note we are adjusting the maxmemory settings but even
after setting it we don't believe the memory will be freed.

Signed-off-by: John T Skarbek <jtslear@gmail.com>

Moving the redis swich master alert to production for https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4755

* Any links pointing to checkmk are also horridly out of date
* The two runbooks related to ci runners appears to require an overhaul
as the speak about old infrastructure that has since been moved
* I'm not removing those as they might be relevant to help future
generates, but minimally checkmk is no longer monitoring this particular
setup
Signed-off-by: John T Skarbek <jtslear@gmail.com>

* Current our postgres service alerting is a bit brittle
* This alert will utilize the `pg_up` exporter metric and alert on it
* Addresses: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5016
* Also bumps up the priority of this alert since this would normally be
a pretty bad thing that requires investigation
Signed-off-by: John T Skarbek <jtslear@gmail.com>

* 'master' of gitlab.com:gitlab-com/runbooks:
  Service -> Team mappings
  Add a minimum number of invocations, change channel

Added alert for redis servers switching master - https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4755#note_100125334

Closes https://gitlab.com/gitlab-com/production/issues/458.