heartbeat-failure-detector

A named TCP message service in Go and an all-pairs heartbeat failure detector built on top of it.

Overview

Processes address each other by logical name rather than by socket address. A small directory resolves names to host:port, the message service moves length-prefixed protobuf messages between endpoints over TCP, and the heartbeat command uses that service to detect which of its peers have stopped responding. The service is safe for concurrent use, delivers inbound messages on a channel, and shuts down cleanly without leaking goroutines or sockets.

How it works

Directory. directory.Lookup maps a name to a listening address and directory.Register reserves a name. The table is owned by one goroutine that serves requests over a channel, so there is no shared-map locking anywhere in the package. The bundled table has five sample entries on localhost.

Wire format. Every message is a two-byte big-endian length followed by a protobuf-encoded api.Message carrying sender, recipient and an opaque data payload. The prefix caps a message at 64 KiB; Send reports a larger message as *api.MessageTooLong before touching the network. Several messages may be written back to back on one connection.

Receiving. NewMessageService(name) resolves its own address and binds a listener. An accept loop hands each connection to its own goroutine, which reads frames with io.ReadFull, decodes them, and pushes them onto a buffered receive channel exposed by Receiver(). Frames that exceed the maximum length or that arrive on a half-written connection close that connection; undecodable bodies are skipped.

Sending. Send resolves the recipient, dials a fresh connection, writes the frame in full, and closes the connection.

Shutdown. Close runs once. It closes the listener, closes every live connection, waits on a sync.WaitGroup for the accept loop and all connection handlers to exit, and only then closes the receive channel so readers see a clean end of stream.

Failure detection. heartbeat <id> <neighbor> [neighbor ...] starts a message service under its own name, waits detector.StartDelay so peers have time to come up, then sends a heartbeat to every neighbor each detector.BeatInterval. A goroutine drains the receive channel and stamps the last time each neighbor was heard from. A checker fires on the same interval and prints <neighbor> failed exactly once for any neighbor silent for longer than detector.TimeoutDuration. Because every process heartbeats every other, the detector is symmetric: all surviving processes notice a failure at about the same time.

Layout

Path	Contents
api/	MessageService interface, size limit, and the protobuf message definition
directory/	Name-to-address directory served by a single owner goroutine
msgservice/	The TCP message service implementation and its tests
detector/	Timing parameters for the heartbeat detector
cmd/heartbeat/	The all-pairs heartbeat failure detector command

Build and run

go build -o bin/heartbeat ./cmd/heartbeat

Run three detectors in three terminals using names from the sample directory:

./bin/heartbeat lamport lynch mills
./bin/heartbeat lynch   lamport mills
./bin/heartbeat mills   lamport lynch

After the start delay they exchange heartbeats silently. Stop one of them with Ctrl-C; within the timeout the other two print, for example:

mills failed

Testing

go test ./...

The tests cover send and receive between two endpoints, several frames written on one connection, oversized messages, the receive channel closing on Close, fifty concurrent sends, a hand-assembled raw frame decoded off the socket, and the directory's registration rules. They bind real localhost ports using the sample directory names, so run them on a machine where those ports are free.

To regenerate the protobuf code after editing api/message.proto:

make proto

Limitations

• The directory is a static in-memory table. Adding a process means adding an entry.
• Send opens a new TCP connection per message and does not retry or back off.
• Each detector keeps its own view; nothing reconciles what different processes have observed.
• A neighbor that is declared failed is never marked as recovered, even if its heartbeats resume.

Originally built for a university distributed-systems course; the wire protocol and interfaces were specified by the course, the implementation is my own.