goer Download ↓
An experiment in software that keeps going

What if Go had a process model?

You write Go. You’ve also watched one bad request take a whole service down with it. Goer borrows Erlang’s best trick for that: split the work into small pieces that can’t touch each other’s stuff, and have a supervisor, whose only job is to watch them, restart the piece that breaks. You keep writing ordinary Go, and use this where it helps.

01 private state02 let it crash03 change it live
supervisorwatching the system
↗orders12 messages
✦paymentshealthy
↻devicesrestarting safely
+·+

This is just a drawing until you press the button. Then it’s the real thing, running in your browser. It’s about 4 MB, and we only load it if you ask.

The idea

Things break. What matters is whether they take everything else down with them.

Goer gives each piece of work its own little world. When one of those worlds falls over, the rest don’t even notice, and the broken one gets a fresh start.

Go+Er

The name

Go, with some Erlang joy.

The name is just Go + Erlang with the join showing. Erlang is the language phone companies built so their switches would stay up for years. Its good ideas are small private processes, something watching over them, and messages instead of shared memory. We wanted those in a place that feels like Go.

You still write Go, and Go stays in charge of your program. Goer is us asking a question out loud: which of Erlang’s ideas are worth having if you already think in Go?

Why Goer

The language you know. A different way to fail.

You keep Go’s standard library and everything that makes it practical. What changes is what happens when something goes wrong.

01

Small, private processes

A process is one small worker with an inbox, called its mailbox. It keeps its own notes, reads one message at a time, and nobody else can scribble in those notes. That’s why you can restart it without untangling a web of locks first.

How the model works →
02

Let it crash

When a worker hits something it can’t handle, it stops. A supervisor sees that and starts a fresh one. No try-everything error handling.

03

Change it while it runs

You can swap a running process’s code without stopping the system. If its saved data needs a new shape, you write one small function that converts the old shape to the new one.

04

Lots of them, cheaply

A process that’s waiting costs almost nothing: a handful of workers share the running, and each process gets its own small box of memory. There’s also a slimmer kind of process for crowds so big that one goroutine each would hurt.

1M processes, measured

The whole idea in one function

State in. Message through. State out.

That’s it. A process is a function that takes what it knows and one message, and returns what it knows now. This is the counter from our first tutorial: it adds one per tick, and it crashes on purpose when you ask it to. Press run and it’ll compile right here in the page.

See the first tutorial →
counter_actor.go
type State struct{ Count int }
type Tick struct{}
type Boom struct{}

func Counter(s State, m Msg) State {
    switch m.(type) {
    case Tick:
        s.Count = s.Count + 1
    case Boom:
        panic("counter asked to crash")
    }
    return s
}

Let it crash

Seven ticks in, it crashes. Watch what doesn’t happen.

Same counter, told to blow up on every seventh message. The program doesn’t restart. Nothing else stops. And nobody wrote any error handling for it. The supervisor notices, starts a fresh counter, and the ticks just carry on. What you see below is a recording of a real run.

goer run ./examples/counter
  1. goer run ./examples/counter
  2. tick 1 — supervised restarts: 0
  3. tick 2 — supervised restarts: 0
  4. tick 3 — supervised restarts: 0
  5. tick 4 — supervised restarts: 0
  6. tick 5 — supervised restarts: 0
  7. tick 6 — supervised restarts: 0
  8. [supervisor] "counter" crashed: imgo panic: counter asked to crash
  9. [supervisor] restarting "counter" (restart #1)
  10. tick 8 — supervised restarts: 1
  11. tick 9 — supervised restarts: 1
  12. tick 10 — supervised restarts: 1
  13. tick 11 — supervised restarts: 1
  14. ^C
  15. goer: shutting down
A recording of a real run, 6 seconds long. We’ve trimmed the long pauses.

How big can it go?

We measured it, so we don’t have to promise.

There are two kinds of process: the regular kind, and a slim kind that only understands a few fixed messages. They share one scheduler (the part that decides who runs next) and one supervisor. Both are measured by a test program in the repo, cmd/megapark, which won’t even start above 100,000 processes unless you pass -n and mean it.

1M processesThe slim kind reaches a million on one machine, and Go’s garbage collector barely notices them. The regular kind costs more per process (about 827 bytes) but can take any message you like.cmd/megapark
3.4 msThat’s the garbage-collector pause with a million slim processes running (4.1 ms at worst), with 404 MiB of heap, while delivering 2.98 million messages a second. Killing 200,000 of them takes 0.36 seconds.median, measured back to back
298 ns · 0 allocsWhat it costs to send one message the fast way (NewSpec/TellSpec), with no memory allocated. The plain Tell is 329 ns. No lock is taken to deliver it.the message-sending benchmark
~4.4×Code that runs a lot gets turned into machine code. One benchmark (VMCall/Wide) drops from 623.8 to 141.2 ns, and number-crunching loops run 4 to 8 times faster. Message-passing only gains about 15%, because there the mailbox is the slow part.from our benchmark notes

A fair warning about these numbers: they came from a laptop that was busy with other things, so they wobble from run to run. That’s why we compare runs made back to back and report the middle one. On a quiet run the regular kind of process moved 2.23 million messages a second with a 190 ms forced-GC pause (252 ms at worst); after we cut down on garbage it was 2.24 million with 109 ms (144 ms at worst). So the speed didn’t change, and the pause dropped by 40–50%.

Where it runs

Machine code where we have it. The interpreter everywhere else.

Goer runs your process code in an interpreter, which works everywhere. On some platforms it also compiles the busy parts to machine code as the program runs. That’s the JIT (just-in-time) label below. Where there’s no JIT you still get the full thing, just interpreted.

Linux / amd64The main one. Busy code becomes amd64 machine code.JIT
Linux / arm64Same compiler, arm64 output. The whole test set passes under an emulator.JIT
Windows / amd64The compiler builds for Windows, but we haven’t run it there ourselves. Treat it as unproven.JIT
Wasm / wasip1The whole runtime runs as WebAssembly, the portable format browsers run, and can connect to other Goer programs. Interpreted only.Interpreter

Everything else — macOS, the BSDs, and other CPU types — builds and runs with the interpreter. You don’t pay anything for the compiler not being there.

More than one machine

One runtime, many nodes.

A node is one running Goer program, and you can connect several. A process’s address works the same whether the process is next door or on another machine. And when a machine drops off, you’re told. Nothing quietly pretends it’s still there.

Go programs talking to each other

Ordinary Go programs, connected over the network.

  • Start a process over there. The compiled code travels with the request (SpawnOn).
  • Supervise it from here. A crash, a missing process recipe, or a dead connection all count as the child failing.
  • Stop it politely, the same way you’d stop a local one.
  • Find out fast. Connected machines check in with each other constantly (a heartbeat), so a dead one is noticed quickly, and everything watching it gets a Down message.

A node made of WebAssembly

You can build Goer as WebAssembly (the wasip1 target) and connect it to other nodes over any plain stream of bytes. Today that’s stdin and stdout.

  • It does everything a normal node does: it takes messages, runs code sent to it, can be watched, and stops cleanly.
  • It’s single-threaded and interpreted, and it has no sockets, so the host program passes the bytes along for it (Attach).
  • If it dies, that looks exactly like a dropped connection. There’s no auto-reconnect.

Who’s right when two copies disagree?

Pick what your data needs.

Most data in Goer has exactly one owner, so there’s nothing to disagree about. For the small amount of data that has to live on several machines at once and still agree later, there’s an optional library.

The default: one owner

One process owns each piece of data and handles messages in order.

  • Nobody else can change it, so there are no races to chase.
  • A crash stays inside its owner, and a supervisor deals with it.
  • If a machine goes quiet, everything watching it is told.

The option: goer/crdt

Data that several machines can each change, and that ends up the same everywhere.

  • Counters, sets, maps, a last-write-wins value, and a “who’s online” list.
  • Merging is tested to give the same answer whatever order updates arrive in.
  • You use it from regular Go code; it travels over the connection Goer already has.

FAQ

Questions people keep asking us.

Short answers. The docs have the long ones.

What’s a port?

It’s how a process reaches the outside world. A port is regular Go code that looks like another process: you send it a request, it does the file or network work, and it sends back an answer. The process never opens a file itself. That also makes testing easy, because you can swap in a fake port that just records what it was asked.

How is Goer different from Erlang?

The ideas are the same family: small private processes, supervisors that restart them, changing code live, and several machines working together. What’s underneath is different. Goer runs inside a normal Go program. You write the process code in a cut-down version of Go called imgo (files marked //go:build goer), and Goer runs it in its own small virtual machine inside your program.

When should I write a process in regular Go instead?

When what it holds is ordinary Go stuff, like a map of connections or something that needs interfaces. Register it and start it with go Native("name"). Use imgo, our cut-down Go, when you want the stronger promises: nothing can reach in, the code can be changed live, and the only way out is through a port (Go code the process sends requests to). Both kinds run side by side and can message each other.

How does changing code live actually work?

A process switches to the new code between two messages, never in the middle of one. Messages already waiting stay in line. If the data’s shape changed, your conversion function runs once at that moment. HotSwap changes the running process; Migrate also changes what a later restart will use.

What gets compiled to machine code?

Any function in a process’s code that’s been called 1,000 times. (That code is written in imgo, our cut-down Go.) It helps most with loops and number work. If a function can’t be compiled it just keeps running in the interpreter, and the two are tested against each other to give identical results.

Why is the WebAssembly node interpreted only?

Because the compiler produces amd64 or arm64 machine code, and WebAssembly is neither. WebAssembly sets that limit. The whole runtime still runs there, and it’s a full member of the cluster.

What does “you’re told when a machine dies” mean in practice?

Every process watching something on that machine gets a Down message. A supervisor treats it like a crash and uses up one restart. So a dead machine can’t leave a process missing without anyone knowing.

Do I need to learn Erlang first?

No. The tutorials explain each idea in plain words and Go code. Nothing assumes you’ve seen Erlang.

Where does a process keep its data?

In its own block of memory inside your Go program. When the process ends, the whole block is handed back. That memory goes away when the program stops, so saving things to disk is up to you. If you need data to survive across machines, that’s what goer/crdt, our library for shared data, is for.

Read the docs

Learn it one layer at a time.

Get something running first. Then find out why it’s built this way, keep the reference handy while you build, and learn to watch a live system.