Operate / observability API

See pressure before it becomes a failure.

Goer has a dashboard, a built-in web page for watching a running system, and it’s great when you’re looking at it. Most outages start when nobody is. So everything the dashboard shows is also available from code: for a health check, a test, an alert, or whatever tooling your team already has. This page lists what you can ask for and what you get back. It tells you how much work is waiting, how long things take, what broke, and who’s connected. It never hands over your actual data.

Words this page uses

process: One small worker inside your program. It keeps its own private data, reads one message at a time from its own inbox, and can be restarted without disturbing anything else. Some people call it an actor.
mailbox: A process’s inbox: the line of messages waiting for it.
supervisor: The part of Goer that watches processes. When one crashes, the supervisor is told and decides whether to start a fresh one.
node: One running Goer program. Each node has a name.
imgo: The cut-down version of Go that processes are written in. An imgo file is a Go file with //go:build goer as its first line; Goer compiles it and the normal Go compiler skips it.
native actor: A process written in regular Go. It gets the same mailbox and supervisor as any other process, and holds ordinary Go data.
slim process: A stripped-down kind of process that only understands a few fixed messages. It exists for huge crowds; the benchmark uses it. Also called a compact actor.
port: Ordinary Go code that a process can message to reach the outside world: files, networks, clocks, other services. A process can’t touch those directly.
link: A tie between two processes: if either one crashes, the other is stopped too.
monitor: A way to watch another process. You get one message when it stops, and nothing else changes.
dashboard: Goer’s built-in web page for watching a running system: which processes are alive, what’s waiting in their mailboxes, what crashed.
garbage collector: The part of Go that finds memory nobody is using any more and frees it. While it works, the program can pause briefly. Often shortened to GC.

Every word, in one place →

One snapshot has everything

System.Snapshot(limit) gives you a picture of this node right now. The limit caps the list of processes, so a huge system doesn’t hand you a million rows. The dashboard is built entirely from this one value.

inspection.go

snapshot := sys.Snapshot(100)

fmt.Println(snapshot.Live, snapshot.Restarts, snapshot.GaveUp, len(snapshot.Crashes))
for _, actor := range snapshot.Actors {
    fmt.Println(actor.Name, actor.Kind, actor.MailboxDepth, actor.Reductions, actor.Paused)
}

Field	What it tells you
`Live` · `GenericLive` · `CompactLive`	How many processes are alive: in total, the regular kind (imgo and native), and the slim kind.
`QueuedActors` · `SchedulerWorkers`	How many processes are waiting for a turn, and how many workers there are to give them one.
`Restarts` · `GaveUp`	Restarts so far, and how many processes a supervisor has given up on.
`HeapInuse` · `NumGC` · `LastPauseNS` · `Pool`	Go’s memory and garbage-collector numbers, plus how much process memory is set aside and in use.
`Actors`	A capped list of processes. For each: id, name, kind, mailbox size and what’s in it by type, messages handled, timing, paused or stopped, and where it is in the code.
`Events`	The last 128 events.
`Supervisors` · `Peers` · `Crashes` · `Tasks` · `Timers`	Who supervises what, which nodes are connected, the last 32 crash reports, and the slow jobs and timers in flight.

A couple of those need explaining. A process’s Reductions is how many messages it has handled. LastMessageNS is how long the latest one took, and MessageTimeNS is the running total. Treat them as rough signals of where time goes.

Look at a mailbox without opening the mail

System.MailboxSnapshot(pid) tells you how many messages are waiting, how long the oldest has been there, and how many there are of each type. imgo messages go by their struct name (PlaceOrder). Go messages go by their Go type. Goer’s own requests show up as runtime: stop, runtime: code swap, and so on.

You get the names. You never get what’s inside.

mailbox.go

info, err := sys.MailboxSnapshot(pid)
if err != nil { return err }
fmt.Println(info.MailboxDepth, info.OldestMessageAt)
for _, kind := range info.MailboxTypes { fmt.Println(kind.Type, kind.Count) }

Crash reports a person can read

A CrashReport says which process crashed, why, how busy it was, and where: the function, the message type it was handling, and the file, line, and column. Goer keeps the latest 32. You’ll find them in Snapshot().Crashes and at the /crashes endpoint.

What’s left out, on purpose?The process’s data and the message’s contents. A crash report should help you find the problem without becoming a second copy of your customers’ information.

Tracing

Tracing is off until you turn it on. SetTrace records an event for every message one process handles. SetTraceOptions lets you pick receives and sends separately. SetSystemTrace covers things no single process owns, like port calls. And goer run -trace turns the lot on.

trace.go

sys.SetTraceOptions(pid, goer.TraceOptions{Messages: true, Sends: true})
sys.SetSystemTrace(goer.TraceOptions{Ports: true, Links: true, Monitors: true, Yields: true})

Trace events go into the same history as everything else. Some kinds are always recorded: started, stopped, crashed, step. The rest only when you ask: message, send, port, link, monitor, yield. A send event tells you a message went into a mailbox. It doesn’t tell you what the message said.

Supervision

Each supervised process in the snapshot has a Parent, a Policy, and a Strategy. The strategy decides whether a crash restarts only that process or the ones started after it too. You set it like this:

supervision.go

err := sys.SetSupervisorStrategy("orders", runtime.SupervisorRestForOne)

Distribution

System.RemoteSnapshots(limit) asks every connected node for its own summary. The dashboard does the same thing through /cluster-snapshot. Only the summary crosses the network.

Web endpoints

sys.DebugHandler() serves the dashboard and the JSON below. The paths are relative to wherever you mount it.

Endpoint	What it’s for
`GET /`	The dashboard page.
`GET /snapshot?limit=100`	The snapshot for this node. Leave out the limit and you get every process.
`GET /mailbox?id=…&kind=actor`	What’s waiting in one mailbox, by type. `kind` is `actor` or `compact`.
`GET /native?id=…`	The facts a native actor has chosen to show.
`GET /crashes`	Recent crash reports.
`GET /cluster-snapshot?limit=25`	A summary from each connected node.
`POST /control`	Do one thing to one process. The body looks like `{"action":"pause","id":6,"kind":"actor"}`. The actions are `pause`, `resume`, `step`, `step_instruction`, `kill` (you can add a `reason`), `trace` (with `enabled`), and `stop` (which takes a `name` where the others take an id).

terminal

curl -H 'Content-Type: application/json' -d '{"action":"pause","id":6,"kind":"actor"}' http://127.0.0.1:8787/control

Lock it down.These endpoints can pause and kill processes. With no token set, they only answer connections from the same machine. sys.SetDashboardToken(token) (or GOER_DEBUG_TOKEN when you use the goer command) lets in anyone holding that token and nobody else. /control also refuses requests that come from another website.

The dashboardThe same thing, in a browser →Native inspectorsShow a few safe facts →ReferenceEvery call, spelled out →