@zensaiyuki Rob Ewashuck, Google SRE, has an excellent set of guidelines on alerting.

https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/mobilebasic?pli=1#heading=h.fs3knmjt7fjy

I've discussed previously:

https://old.reddit.com/r/dredmorbius/comments/2j9xri/alerting_response_google_site_reliability/