Tutorial 02 / 15 minutes

Let one process fail safely.

Most of us were taught to catch every error and keep the program limping along. That’s how you end up with a service that’s technically up and quietly wrong. In the last tutorial the counter (a process: one small worker with its own data and its own inbox) crashed every seventh message and carried on like nothing happened. Here we slow that down. You’ll see who notices a crash, how you choose what happens next, and how to watch it happen. Fifteen minutes, and “let it crash” will stop sounding reckless.

What’s a supervisor?

Back to our made-up shop. A clerk walks out mid-shift. The manager notices, decides the counter still needs someone, and puts a new clerk there with a clean ledger. The other clerks never stopped working.

In Goer the manager is the supervisor. It’s part of the runtime, and it watches the processes it started. When one stops, the supervisor is told why and follows a restart policy. It never reaches into the process’s data.

1. Fail on purpose

A process fails by calling panic. That’s on purpose. If a process finds out its own data can’t be trusted, the safest thing it can do is stop. Carrying on with a bad ledger is worse. The counter does this for Boom; a real process would do it for a request that should be impossible. You can run this one:

type Add struct{ Amount int }

func Counter(s State, m Msg) State {
    switch x := m.(type) {
    case Add:
        if x.Amount < 0 {
            panic("negative amount")
        }
        s.Count = s.Count + x.Amount
    case Boom:
        panic("counter asked to crash")
    }
    return s
}

2. Pick the restart rule

The rule is the last argument to Spawn. It answers one question: when this process stops, should it come back?

c := sys.Spawn("counter", goer.Init(init), goer.Behavior(behaviour), goer.RestartOnCrash)

_ = c.Tell(&msg.Spec{Type: "Add", Fields: []msg.Field{msg.Int("Amount", 1)}})
_ = c.Tell(&msg.Spec{Type: "Add", Fields: []msg.Field{msg.Int("Amount", -1)}}) // crashes

A supervisor also knows when to quit. If a process crashes five times in ten seconds, restarting it a sixth time probably won’t help. So the supervisor stops, and writes down that it did. The dashboard shows this as gave up.

3. Watch the restart

Run the counter with the dashboard switched on and open the address it prints:

goer run -dash ./examples/counter

The Supervision tab lists counter with its rule and how many times it’s been restarted. The Crashes tab shows each panic: which process, why, what message it was handling, and the file and line. You can get the same things from Go with sys.Restarts() and sys.Snapshot(100).Crashes.

4. Links and monitors

A supervisor is a parent watching a child. Sometimes two processes that aren’t parent and child need to know about each other. There are two ways.

A link ties two processes together: Link(pid) means if either one crashes, the other goes too. Use it when one is useless without the other.

A monitor just watches. Inside a process, Monitor(pid) asks for one Down{Pid, Reason} message when the other process stops, and you handle it in your switch like any other message. From Go, sys.MonitorPid(pid) gives you a channel that does the same job.

A monitor is also how you find out someone’s gone. If you send to a process that has stopped, the message is quietly dropped. You don’t get an error. So if it matters, monitor it.

Our advice: start with supervisors. Add links and monitors only when you really need them.

5. Supervising across machines

The same idea works when the process lives on another machine. SuperviseOn(parent, node, regName, policy) starts the registered template regName on the connected machine node, and puts it under parent’s care on this machine. (Leave parent empty for the top-level supervisor.) You call it from Go.

The restart rule and the restart count stay here. The other machine just runs the process and doesn’t supervise it, so it can never get restarted twice.

// "inventory" must be registered on orders-b; register it there in its Main.
_, err := sys.SuperviseOn("", "orders-b", "inventory", runtime.RestartOnCrash)

What comes back is an ordinary “it stopped”. If it stopped politely, the reason is normal and nothing gets restarted. If it crashed, or the template wasn’t there (noproc), or the connection dropped, that’s a failure and the rule applies just like it would locally.

The dropped connection is the one to remember. Losing the other machine counts as the process failing and uses up a restart. That way a dead machine can’t leave a process missing with nobody the wiser.