Adjust the minimum invocations per second parameter to reduce alert noise

elastic-watcher/watches/gitaly_abuse_1.json

@@ -70,7 +70,7 @@
       "params" : {
         "time_period_seconds" : 600,
         "wall_time_ms_per_second_threshold": 4000,
-        "minimum_invocation_rate_per_second": 2,
+        "minimum_invocation_rate_per_second": 10,
         "invocation_rate_per_second_threshold": 100
       }
     }
@@ -82,7 +82,7 @@
       "params" : {
         "time_period_seconds" : 600,
         "wall_time_ms_per_second_threshold": 4000,
-        "minimum_invocation_rate_per_second": 2,
+        "minimum_invocation_rate_per_second": 10,
         "invocation_rate_per_second_threshold": 100
       }
     }

troubleshooting/gitaly-unusual-activity.md

@@ -10,9 +10,11 @@
 
 ## 1. Review the suspicious activity
 
+- **Check out the abuse dashboard**: https://dashboards.gitlab.net/d/9T-wXWbik/abuse-dashboard?orgId=1&panelId=2&from=now-1h&to=now
+- **Review the abuse reporting data in Kibana**: https://log.gitlab.net/goto/6636a49add992f6326862df0afc6ae54
+- **Review the abuse dashboard**:  https://log.gitlab.net/app/kibana#/dashboard/AWSIfVZhTIzC7JP6Xxn1
 - Keep in mind that this is an open-ended alert, so it alerts to suspicious activity, rather than pin-pointing an issue.
 - Use this as an informational alert, combine it with other signals
-- Also review the abuse dashboard:  https://log.gitlab.net/app/kibana#/dashboard/AWSIfVZhTIzC7JP6Xxn1
 
 ## 2. Evaluate impact