Skip to content

Latest commit

 

History

History
78 lines (67 loc) · 4.72 KB

on-call.md

File metadata and controls

78 lines (67 loc) · 4.72 KB

On Call

Onboarding

Find Out When You’re On-Call

To find out when you’re on call follow the below checklist:

  1. Add the SET calendar to your Google Calendar
    1. Log in to OpsGenie via OneLogin
    2. Open each of the following schedules that are relevant to you:
    3. For each click the litte "Open Calendar" calendar button that appears when you hover over the calendar. This will open a webcal:// link, copy this link
    4. In Google calendar (or your prefered calendaring app) Go to “Add Calendar”, select “From URL” and paste the webcal URL. After it's added, rename it to something appropriate.
  2. Find yourself on the calendar
    1. At the top right of Google Calendar click the search icon
    2. Type in the below based on your rotation for SET: “\ Engineering_schedule”
    3. The results should show you the dates you’re on call

Before Your First Day On-Call

There’s a few things you should set up before your first day on call. Work through the following checklist a couple days before you start on-call:

  1. Log in to OpsGenie
    1. Go to https://mattermost.onelogin.com/portal and log in
    2. Search for and click the OpsGenie icon
  2. Install the OpsGenie app on your phone
    1. Search “OpsGenie” in the Apple App Store or Google Play Store
    2. Open the app and type in your email, making sure to select US as the data center region
    3. For organization, enter “mattermost”
    4. This should redirect you to OneLogin to complete the log in
  3. Configure your OpsGenie notification rules in your desktop browser
    1. Go to https://mattermost.app.opsgenie.com/settings/user/notification
    2. Under “Contact methods” add your phone number for SMS and/or Voice
    3. Under “Notification rules” you should configure the rules for:
      1. New Alert
      2. Schedule Start
      3. Schedule End
    4. How you want to configure your notifications is up to you but make sure it’s configured in a way that will get your attention
      1. An example configuration is:
        1. Notify mobile immediately
        2. Notify via SMS after 5 minutes
        3. Notify via phone call after 10 minutes
    5. Test your notification rules by creating a test alert
      1. Go to https://mattermost.app.opsgenie.com/alert/list
      2. Click “Create alert” in the top right
      3. Name the alert “Test Alert”
      4. Add yourself under “Responders”
      5. Click “Create”
      6. Confirm that your notification rules are followed
      7. Acknowledge the alert
      8. Close the alert
  4. Join ~Incidents Status channels on community.mattermost.com
  5. Read Incident Response to familiarize yourself with the response process
  6. Create a test incident to familiarize yourself with the Incident Response plugin
    1. On community.mattermost.com, click the “!” icon in the channel header
    2. Click the “+” to start a new incident
    3. Choose playbook
      1. For SET, use “Sustained Engineering”
      2. For Cloud, use “Cloud Incident Response Playbook”
    4. End the incident

Your First Day On-Call

On your first day on call, do the following:

  1. Check if you’re on-call as primary or back up
    1. Log in to OpsGenie
    2. Go to https://mattermost.app.opsgenie.com/schedule/whoIsOnCall and click on the schedule for the rotation you’re on
    3. You’re on-call as primary if you’re listed under one of the rotations that has “Primary” in the name
  2. Start your shift
    1. OpsGenie will notify you when your shift starts or check the schedule
  3. Watch for alerts from OpsGenie and follow Incident Response to respond to any alerts
  4. End your shift
    1. OpsGenie will notify you when your shift ends or check the schedule