witch

A production-grade, multi-database High Availability daemon.

---

What is Witch?

Witch is a single binary that manages database clusters — handling leader election, automatic failover, controlled switchover, replication monitoring, and cluster configuration. It embeds a complete Raft consensus algorithm, replacing all need for an external Distributed Coordination Service (DCS) like etcd, Consul, or ZooKeeper.

Key Features

• Zero Split-Brain — Leader election backed by embedded Raft consensus; no node can become primary without quorum approval.
• Single Binary — Deploy by copying one file. No external dependencies.
• Multi-Database — Protocol-agnostic adapter interface. PostgreSQL is complete; MySQL is scaffolded.
• Production-Hardened — Full Raft implementation, synchronous replication enforcement, hardware/software watchdog, STONITH fencing, snapshot support, failsafe mode.
• No Database Extension Required — Manages databases using standard tools and SQL. No CREATE EXTENSION needed.
• Operator-Friendly — Node-based model, REST health endpoints, event hooks.

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        witch daemon                         │
│                                                              │
│  ┌──────────┐   ┌──────────┐   ┌──────────────────────────┐ │
│  │  Raft   │   │ Cluster │   │    REST API                │ │
│  │  Core   │──▶│  FSM    │──▶│  /health /primary         │ │
│  │         │   │         │   │  /replica /cluster        │ │
│  └──────────┘   └──────────┘   └──────────────────────────┘ │
│       │               │                                    │
│       │         ┌─────▼──────┐   ┌────────────────────┐   │
│       │         │  Protocol  │   │     Watchdog         │   │
│       │         │  Adapter   │   │  (STONITH hook)     │   │
│       │         └─────┬──────┘   └────────────────────┘   │
│       │               │                                    │
│  ┌────▼───────────────▼──────┐   ┌────────────────────┐     │
│  │         Config            │   │    Event Bus          │     │
│  │  (TOML + Raft-replicated)│   │  (hooks/scripts)     │     │
│  └───────────────────────────┘   └────────────────────┘     │
└─────────────────────────────────────────────────────────────┘
          │                    │
 shared control port      db process
 (REST + peer + Raft)   (postgres, etc.)

---

How It Works

Each node runs the witch daemon alongside its database instance. All nodes participate in a Raft cluster that:

1. Elects a Primary — Through Raft consensus when the current primary fails
2. Monitors Replication — Tracks WAL position, lag, and slot status
3. Triggers Failover — Automatically promotes the healthiest standby
4. Handles Switchover — Gracefully hands over primary to a designated node
5. Manages Configuration — Dynamic config changes replicated via Raft log

---

Status

Module	Status
Raft core	✅ Complete
Raft persistent log + hard state	✅ Complete
Cluster FSM (promotion gates)	✅ Complete
Dynamic config via Raft log	✅ Complete
Failsafe mode	✅ Complete
Synchronous replication management	✅ Complete
Switchover / graceful demotion	✅ Complete
Auto-rejoin (pg_rewind / basebackup)	✅ Complete
Cascading replication	✅ Complete
REST API	✅ Complete
Peer status protocol	✅ Complete
Hardware watchdog	✅ Complete
Software watchdog	✅ Complete
STONITH via external command	✅ Complete
PostgreSQL adapter	✅ Complete
MySQL adapter	🔲 Scaffold only

---

Configuration

Witch reads witch.toml from the current directory.

[cluster]
name        = "my-cluster"
node_id     = 1
node_name   = "node-1"
bind_port   = 7000

[database]
host     = "localhost"
port     = 5432
user     = "witch"
password = "secret"

[postgres]
data_dir              = "/var/lib/postgresql/data"
pg_ctl_path           = "/usr/bin/pg_ctl"
replication_user     = "replicator"
replication_password = "repl-secret"

[[peers]]
node_id  = 2
host     = "10.0.0.2"
port     = 7000
db_port  = 5432

[[peers]]
node_id  = 3
host     = "10.0.0.3"
port     = 7000
db_port  = 5432

[raft]
election_timeout_ms = 1000

[failover]
health_check_interval_ms = 1000
maximum_lag_on_failover  = 1048576
failsafe_mode            = true

[tags]
nofailover       = false
failover_priority = 100

[watchdog]
enabled = false

[hooks]
on_start         = ""
on_stop          = ""
on_role_change  = ""
on_failover     = ""
on_switchover   = ""

Dynamic Configuration

The following settings can be changed at runtime via POST /cluster/config and are replicated to all nodes via the Raft log (no restart required):

Key	Default	Description
failover_paused	false	Pause automatic failover
synchronous_mode	false	Require synchronous replication for promotion
synchronous_mode_strict	false	Block writes if no sync standby
max_replication_lag	1048576	Max lag before excluding standby
failsafe_mode	false	Keep primary alive during network partition
failsafe_ttl_ms	30000	Failsafe lease duration
check_timeline	true	Exclude candidates with stale timeline

---

Building

Requires Zig 0.16-dev.

zig build                        # build
zig build run                   # run with witch.toml in cwd
zig build test                  # run all tests
zig build check               # type-check without linking

---

Deployment

See Deployment Guide for:

• Docker sidecar pattern (Witch + PostgreSQL in same container)
• Kubernetes StatefulSet deployment
• Bare metal with systemd
• TLS/mTLS configuration
• Production checklist

---

REST API

Method	Path	Description
GET	/health	Returns 200 if node is healthy
GET	/primary	Returns 200 only on the primary
GET	/replica	Returns 200 only on standbys
GET	/node	Full node state (JSON)
GET	/cluster	Cluster membership and Raft state
POST	/switchover	Request controlled primary handoff
POST	/failover/pause	Pause automatic failover
POST	/failover/resume	Resume automatic failover
POST	/cluster/config	Apply dynamic config
POST	/failsafe	Standby heartbeat during failsafe

---

Cluster Model

Node Roles

Role	Description
primary	Accepts writes. Elected by Raft consensus.
standby	Replicates from primary. Read-only. Eligible for promotion.
standby_sync	Synchronous standby. Zero data loss on failover.
witness	Participates in Raft voting only. No database instance.

Node Tags

Per-node labels that affect HA decisions:

Tag	Effect
nofailover	Node never enters promotion race
noloadbalance	Node excluded from read routing
nosync	Node never placed in synchronous standby list
failover_priority	Higher value wins at equal WAL position
replicatefrom	Replicate from a specific node (cascading)

---

Event Hooks

Witch fires hooks at key events for integration:

• on_start — Daemon started
• on_stop — Daemon stopped
• on_role_change — Node role changed
• on_failover — Automatic failover completed
• on_switchover — Planned switchover completed
• on_node_join — New node joined cluster
• on_node_remove — Node removed from cluster
• on_node_rejoin — Node rejoined after failover

---

Documentation

• Architecture
• Raft Specification
• HA Specification
• PostgreSQL Specification
• Deployment Guide