# MeterCall Gossip Protocol (v1) Status: draft, shipping in node v0.1.0. Transport: pure HTTP (WebSocket deferred). Zero external deps. Node 18+ stdlib only. The gossip protocol is how MeterCall nodes find each other, measure each other, and route work to each other *without a central coordinator*. `metercall.ai` is a passive leaderboard — if it goes dark, the mesh keeps serving traffic. --- ## 1. Identity Every node generates an ed25519 keypair on first boot (stored at `$HOME/.metercall/gossip-key.json`, `0600`). The node's identity is deterministic: node_id = "0x" || sha256(pubkey_spki_base64).hex[0:32] // 128-bit / 16 bytes The `pubkey_spki_base64` string is included in every gossip envelope. Receivers recompute `sha256(pub)[:16]` and reject any envelope where `from != node_id(pub)`. --- ## 2. Peer Discovery Flow 1. **Boot.** Node reads `MC_BOOTSTRAP_PEERS` (comma-separated URLs) or falls back to `node/scripts/bootstrap-peers.json`. Default 6 seeds: `seed-iad-01`, `seed-sjc-01`, `seed-ams-01`, `seed-nrt-01`, `seed-sin-01`, `seed-gru-01`. 2. **Initial pull.** Node HTTP-GETs `/peers` on each seed concurrently. Each response contains the seed's self-descriptor + up to 100 known peers. Merge into local peer table. 3. **First gossip round.** Immediately after bootstrap, the node picks 3 random peers and does a push-pull. 4. **Steady-state gossip.** Every 60 s, pick `FANOUT=3` random peers and `POST /peers/gossip` a signed envelope containing our 40 freshest peers. Peers reply with their own peer list (pull side). Both sides merge. 5. **Health pings.** Every 5 min, GET `/health` on up to 50 peers. Latency is EWMA-smoothed into the peer record. Failures decay reputation. 6. **Pruning.** Entries with `last_seen > 30 min ago` or `reputation <= 0.05` are dropped. Table is capped at 500 peers; eviction prefers high-latency, low-reputation, stale peers. --- ## 3. Gossip Message Format All gossip messages are JSON, ≤ **4,096 bytes** including signature. The envelope: { "v": 1, "kind": "gossip" | "register" | "ping", "from": "0x…", // node_id of sender "pub": "", // recomputed server-side "ts": 1713312000000, // ms since epoch "body": { … }, // kind-specific payload "sig": "" // over JSON.stringify of all above minus sig } For `kind: "gossip"`, `body.peers` is an array of descriptors (node_id, url, pub, lat, lon, region, latency_ms, last_seen). On the wire we cap to ~40 peers so envelopes stay inside the 4 KB budget; if serialization would exceed 4 KB we recursively halve the peer list until it fits. Receivers **always** verify before merging: 1. Total bytes ≤ 4,096 (else reject, bump `gossip_rejected_size`). 2. `sha256(pub)[:16] == from` (else drop — wrong identity). 3. ed25519 `sig` valid (else drop, bump `gossip_rejected_sig`, tombstone source for 1 round). --- ## 4. Peer Table & Reputation Each peer carries a `reputation` ∈ [0, 2]. Start at 1.0. - **+2 %** on successful gossip or health ping. - **×0.7** on request failure (timeout, ECONNREFUSED). - **×0.9** on non-2xx. - **×0.5** on a bad signature or peer masquerade attempt. - At ≤ 0.05 the peer is evicted. Scoring for **eviction** when full: `latency_ms / reputation + staleness_s`. Higher = worse. Scoring for **closest-N lookup**: `latency_ms + 0.1 × haversine_km`, divided by reputation, ×1.5 if stale. --- ## 5. DHT-lite (XOR-distance Content Lookup) For content-addressed lookups (e.g. "who has this cached response?") we overlay a kademlia-style XOR routing layer using the existing `node_id` space (128 bits). key = sha256(content_key).hex[0:32] dist(a, b) = (a XOR b) // lex-compared as bytes closest_n(key, n) = sort peer_table by dist(peer.node_id, key) asc, take n Exposed via `Gossip#closestByXor(key, n)`. Use cases: cache replication, bridge-attestation quorum selection, future k-v store. --- ## 6. Partition Recovery The coordinator tracks an `edges` set (pairs of node_ids seen gossiping in the last hour) and a `peers` set. `GET /api/node/mesh-stats` returns `partitions` = connected components via union-find. In a healthy mesh this should be **1**. If > 1 we have a split — probable causes: a firewall change at a major VPS provider, or a bad bootstrap list after a rename. Recovery is automatic: any node in partition A that learns one peer in partition B (via a heartbeat response, a human-shared URL, or a refreshed bootstrap file) will bridge the two in one gossip round. --- ## 7. Load Shedding When a node's rolling 1-second RPS exceeds `MAX_RPS × 0.90` it forwards incoming L4 requests to the closest peer from its gossip table instead of serving locally. This turns the mesh into a self-balancing load shedder at the edges, without ever consulting the coordinator. Receipts are still signed by whichever node actually served the request. --- ## 8. Security Model - **Identity is cryptographic.** No trust in IP/DNS. Anyone can run a seed, but they can't spoof another node's `node_id` without its private key. - **Sybil pressure comes from stake.** Gossip is cheap; serving requests for reward requires PCP stake. An attacker can flood the peer table with bogus entries but gains nothing unless they also stake. - **Bad-behavior decay.** Signature failures, health-check failures, and non-2xx responses all decay reputation. A peer that misbehaves repeatedly drops out of peer-table rotation within ~3–5 rounds. - **Size DoS.** Any envelope > 4 KB is dropped at the HTTP boundary before JSON parsing. - **Replay.** `ts` is carried but not strictly validated yet; v2 will add `ts` within `±5 min` of receiver clock. --- ## 9. Endpoints | Method | Path | Purpose | | ------ | ------------------------------ | ----------------------------------------- | | GET | `/peers` | Return self + up to 100 known peers. | | POST | `/peers/gossip` | Accept signed envelope, merge, respond. | | GET | `/peers/closest?lat=&lon=&n=` | Closest N peers by latency + haversine. | | GET | `/health` | Liveness + current RPS, used for pings. | Coordinator-side (`metercall.ai`): | Method | Path | Purpose | | ------ | ------------------------------- | --------------------------------------------------------------- | | POST | `/api/node/gossip-register` | Opt-in canonical leaderboard entry (node_id must match pubkey). | | GET | `/api/node/mesh-stats` | peers_total, fresh_5m, active_edges_1h, partitions, msg rate. | | GET | `/api/node/topology` | Nodes + edges for the force-directed viz (`/mesh.html`). | --- ## 10. Convergence With `FANOUT=3` and `GOSSIP_INTERVAL=60 s`, push-pull gossip reaches full network knowledge in `O(log₃ N)` rounds. For `N=500` peers that's ~6 rounds ≈ **6 minutes from a single bootstrap seed**. The first-round immediate push after bootstrap shaves one interval — so realistic full convergence from a cold start is **5–6 minutes**. --- ## 11. Roadmap (v2) - WebSocket transport for nodes behind NAT (node initiates outbound, relays inbound). - Strict `ts` skew check, replay cache. - Gossip of per-chain capability vectors so `closestN` can filter by chain. - Rendezvous hashing for content placement.