Architecture / distribution

Put a process on another node.

One machine was fine until it wasn’t. Maybe you need more room, or you don’t want a single box to be the thing that takes you down. Either way, the scary part of going multi-machine is everything that fails without telling you. Goer’s take is to keep it plain: you name the machines (Goer calls each running program a node), you connect them, and when one goes away you’re told straight away. This page covers how to connect nodes, start and stop processes on them, supervise them from afar, and what a node built from WebAssembly can do.

A node is one running Goer system. Up front, here’s what Goer doesn’t do: it doesn’t find other nodes for you, it doesn’t reconnect on its own, and nodes don’t share a list of names.

What crosses the network

Only messages cross. A process’s data never does. The two nodes speak a small set of things to each other: a hello that says who they are, “deliver this message”, “start this process”, “stop that one”, link and monitor signals, a heartbeat, and a few questions like “what are you running?”.

Connecting: Listen, Connect, Attach

Every node has a name. One listens. The other connects. Call Listen before Connect. There’s a third option, Attach, which joins over any two-way stream of bytes, and works just like a network connection once it’s joined.

func Main(sys *goer.System) error {
    name := os.Getenv("GOER_NODE")
    bind := os.Getenv("GOER_BIND")
    if _, err := sys.Listen(name, bind); err != nil { return err }

    if peer := os.Getenv("GOER_PEER"); peer != "" {
        if _, err := sys.Connect(peer); err != nil { return err }
    }
    return nil
}

GOER_NODE=orders-a GOER_BIND=127.0.0.1:9101 goer run ./orders
GOER_NODE=orders-b GOER_BIND=127.0.0.1:9102 GOER_PEER=127.0.0.1:9101 goer run ./orders

Once connected, the other node shows up in Snapshot().Peers with the time you last heard from it.

Start a process over there

In imgo, Spawn(Remote("orders-b", Worker)) starts Worker on the node called orders-b. You don’t have to install anything there first. The compiled code for Worker, and only what Worker needs, is sent along with the request, and the other node runs it.

The node name can be any string expression. If there’s no such node, you find out when the spawn runs. For a native Go actor, which can’t be shipped, use System.SpawnOn(nodeName, regName) to start one the other node has already registered.

Stop one over there

System.StopRef(ref) asks the other node to stop a process politely, the same way close(pid) does locally. The request waits in the mailbox, the process finishes what it’s doing, and the other node tells you it’s done. A supervisor treats that as a normal stop, so nothing gets restarted.

Supervise from here

You can run a process on one node and keep its supervisor on another. That’s a Go call:

ref, err := a.SuperviseOn("", "orders-b", "flaky", runtime.RestartOnCrash)

SuperviseOn(parent, node, regName, policy) starts the registered template on node and puts it under parent on this node. (An empty parent means the top-level supervisor.) The restart rule and the restart count stay here. The other node just runs the process and doesn’t supervise it, so it can’t be restarted twice.

That last one deserves a second look. Losing the other node counts as the process failing, and uses up one of its restarts. We did that on purpose, so a dead machine can never leave a process missing with nobody the wiser. Run out of restarts and the supervisor gives up, the same as it would locally.

When a machine dies

System.MonitorPid(pid) gives you a channel that receives one Down when the process stops, with the reason. It only watches. System.LinkPids(a, b) ties two processes together so that one crashing takes the other with it. Both work across nodes.

When a node drops off, every monitor on one of its processes fires with "node unreachable", and every local process linked to one of them is stopped with "linked node unreachable". So what your program believes about who’s alive is never stale for long.

The WebAssembly node

Here’s a slightly odd thing you can do: build all of Goer as WebAssembly (GOOS=wasip1) and have it join a cluster as a full member. WebAssembly has no sockets, so the node can’t listen for connections. It talks over its own stdin and stdout, and whatever program started it passes the bytes along.

cmd/wasmnode is a working example. Build it, run it under Node.js (see its run.mjs), and it calls Attach on its own stdin and stdout.

sys := runtime.NewSystem()
sys.Register("wasm-tally", tallyInit, tally)
if err := sys.Attach("wasm", runtime.NewStdioStream(newBlockingReader(os.Stdin), os.Stdout)); err != nil {
    os.Exit(1)
}

GOOS=wasip1 GOARCH=wasm go build -o wasmnode.wasm ./cmd/wasmnode
node run.mjs        # child stdin/stdout become the cluster link

What it can do: everything a regular node can. It receives messages, answers requests (Ask), and answers questions about a process (QueryField). It starts registered templates. It runs process code shipped to it. It can be monitored, stopped politely, and it shuts down cleanly when its stdin closes. There’s a test in the repo that builds the node, launches it, connects it to a regular one, and checks all of that.