Architecture

A fully decentralized open source hosting network (DePIN) where hosts provide compute/storage resources via Firecracker microVMs, earn on-chain token rewards, and participate in a peer-to-peer network with no central authority. Built on Cosmos SDK for sovereign chain governance, IPFS for decentralized storage, and libp2p for P2P host discovery. An AI layer polishes open source apps into deployable packages. Users browse a catalog and deploy services on the decentralized network.

Core Principles

Decentralization first: No single point of failure or control. The network rewards participants, not a provider.
On-chain trust: Host registry, rewards, staking, and governance live on a sovereign Cosmos chain.
IPFS-native storage: All images and persistent data stored on IPFS. Hosts earn rewards for pinning content.
P2P control plane: Hosts discover each other via CometBFT’s P2P gossip layer. No central orchestrator.

1. System Overview

                              OPENINFRA DECENTRALIZED ARCHITECTURE

    +------------------+
    |   User Portal    |        +--------------------------------------------+
    |   (React SPA)    |        |         COSMOS CHAIN (CometBFT)            |
    |                  |  REST  |                                            |
    |  - Browse catalog+------->+  On-chain state:                           |
    |  - Deploy apps   |  (via  |  - Host registry (x/host-registry)         |
    |  - Manage wallet |  any   |  - Token balances (x/bank)                 |
    |                  |  node) |  - Reward distribution (x/rewards)          |
    +------------------+        |  - Staking / slashing (x/staking)          |
                                |  - Service catalog metadata (x/catalog)    |
                                |  - Governance / DAO (x/governance)         |
                                |                                            |
                                |  P2P Gossip Layer (libp2p / CometBFT):     |
                                |  - Host discovery                          |
                                |  - Heartbeat propagation                   |
                                |  - Workload assignment consensus           |
                                +-----+------------------+------------------+
                                      |                  |
                        P2P (libp2p)  |                  |  P2P (libp2p)
                        +-------------+------+   +-------+-------------+
                        |                    |   |                     |
               +--------v--------+   +-------v--v---------+   +-------v---------+
               |   HOST NODE     |   |     HOST NODE       |   |   HOST NODE     |
               | (Agent+Validator)|   | (Agent+Validator)   |   | (Agent only)    |
               |                 |   |                     |   |                 |
               | +-------------+ |   | +------+ +-------+ |   | +-------------+ |
               | | Firecracker | |   | | FC   | | FC    | |   | | IPFS Node   | |
               | | microVM #1  | |   | | VM#1 | | VM#2  | |   | | (pinning)   | |
               | | [Nextcloud] | |   | |[Gitea]| |[PgSQL]| |   | +-------------+ |
               | +-------------+ |   | +------+ +-------+ |   | +-------------+ |
               | +-------------+ |   |                     |   | | Storage     | |
               | | IPFS Node   | |   | +-------------+    |   | | Volumes     | |
               | | (pinning)   | |   | | IPFS Node   |    |   | +-------------+ |
               | +-------------+ |   | | (pinning)   |    |   |                 |
               |                 |   | +-------------+    |   | Metrics + HB    |
               | Chain Validator |   |                     |   +-----------------+
               | Metrics + HB   |   | Chain Validator     |        Host C
               +-----------------+   | Metrics + HB       |      (storage)
                    Host A           +---------------------+
                  (compute+val)           Host B
                                     (compute+val)

2. Technology Decisions

Decision	Choice	Rationale
Backend language	Go	High performance, excellent for infrastructure, single binary deploys
VM isolation	Firecracker microVMs	VM-level security for untrusted multi-tenant workloads, 125ms boot, ~5MB overhead
Blockchain	Deferred	Start with centralized credits ledger, add blockchain when platform matures
Payments	Multi-chain crypto (EVM + Solana)	Decentralized, no payment processor fees. HD wallet derivation per user, on-chain verification via public RPCs
Price oracle	CoinGecko free API	Zero cost, 5-min cached prices for ETH/SOL/USDC to USD
Database	PostgreSQL 16 (self-hosted)	Free, robust, handles all data models
Reverse proxy	Caddy 2	Automatic HTTPS, dynamic config API, single binary
Image storage	MinIO (self-hosted) or Backblaze B2	S3-compatible, cheap ($5/TB/mo on B2)
VPN overlay	WireGuard	Kernel-level, minimal overhead, solves NAT traversal
Agent-server comms	gRPC with mTLS	Efficient bidirectional streaming, strong auth
User API	REST (chi router)	Simple, well-understood, OpenAPI-compatible
SQL	sqlc	Type-safe, no ORM overhead, compile-time checks

3. Host Agent

The Host Agent is a single Go binary that hosts download and run on any Linux machine with KVM support.

Responsibilities

Registration: Register with the control plane, reporting hardware capabilities (CPU, RAM, disk, bandwidth, GPU, KVM support)
Heartbeat: Send periodic heartbeats (every 30s) over gRPC streaming with resource utilization, running VM count, network stats
Workload Management: Receive assignments via gRPC, pull rootfs images, launch/stop/restart Firecracker microVMs
Metrics Collection: Per-VM metrics (CPU %, memory %, disk I/O, network I/O) via Firecracker’s metrics FIFO and /proc parsing
Networking: Set up tap devices per VM, configure iptables, establish WireGuard tunnel to ingress router
Self-Update: Pull new agent versions from control plane, rolling self-update
Storage Mode: Optionally expose local disk as network-attached storage volumes

Design Decisions

Single static binary: Cross-compiled for linux/amd64 and linux/arm64. No runtime dependencies beyond KVM
Firecracker management: firecracker-go-sdk for VM lifecycle. Each VM gets a dedicated tap device, rate-limited vCPU, and cgroup
Local state: SQLite (~/.openinfra/state.db) for assigned workloads, cached images, metrics history. Survives restarts
Graceful degradation: If control plane is unreachable, keep running VMs and buffer metrics locally

Configuration

control_plane: "grpc.openinfra.io:443"
host_id: "<auto-generated-uuid>"
capabilities:
  compute: true
  storage: true
  database: false
max_vcpus: 8
max_memory_mb: 16384
max_disk_gb: 200
wireguard_port: 51820

4. Control Plane

Components

Component	Role	Technology
API Server	REST for users, gRPC for agents	Go, chi router, google.golang.org/grpc
Scheduler	Assigns workloads to hosts based on capacity, latency, reliability	Go, internal package
Health Monitor	Detects offline hosts, triggers rescheduling	Go, cron-based
Reward Engine	Calculates and distributes credits every epoch (1 hour)	Go, cron-based
Ingress Router	Routes user traffic to correct host/VM	Caddy with dynamic config API
Image Registry	Stores Firecracker rootfs images	S3-compatible storage

Deployment (MVP - Cost Minimized)

Single VPS (Hetzner CPX31: 4 vCPU, 8GB RAM, 160GB): ~EUR 15/mo
  - API Server (port 8080 REST, port 9090 gRPC)
  - PostgreSQL 16 (local)
  - Caddy (port 443, reverse proxy + auto TLS)
  - MinIO (S3-compatible image storage, local disk)

Total MVP infrastructure: ~$19/mo

Scaling Path

Phase 2: Separate Caddy onto edge nodes
Phase 3: PostgreSQL replicated or managed
Phase 4: Multiple API server instances behind load balancer

Scheduler Algorithm

score(host) = 0.4 * available_resources_match
            + 0.3 * reliability_score
            + 0.2 * geographic_proximity
            - 0.1 * current_load_ratio

Hosts below reliability threshold (< 0.7) are excluded. Bin-packing to maximize utilization.

Health Monitoring

3 missed heartbeats (90s): host marked degraded
5 minutes no heartbeat: host marked offline, workloads enter rescheduling queue
Scheduler picks new hosts within 60s
Stateless workloads restart immediately; stateful workloads require data migration
Repeated offline events decrease host reliability score

5. Reward System (Centralized Ledger)

Credits are the internal unit of account. 1 credit = $0.001 USD equivalent.

Earning Credits

Rewards calculated every epoch (1 hour):

epoch_reward(host) = base_rate * uptime_factor * resource_factor * quality_factor

  base_rate       = 100 credits/hour (adjustable)
  uptime_factor   = minutes_online / 60  (0.0 to 1.0)
  resource_factor = (allocated_vcpus * 10 + allocated_ram_gb * 5 + allocated_disk_gb * 1) / normalization
  quality_factor  = successful_health_checks / total_health_checks  (0.0 to 1.0)

Fee Splitting

User pays: 100 credits/hour for a 2-vCPU, 4GB VM
  Host operator:     80 credits (80%)
  Platform treasury: 20 credits (20%)

Verification of Host Contributions

Heartbeat verification: Regular heartbeats with signed resource metrics
Challenge probes: Periodic HTTP/TCP probes to running workloads at random intervals
Cross-validation: Lightweight monitoring sidecars for critical workloads
Anomaly detection: Statistical analysis of reported metrics vs. probe results

Ledger (Double-Entry Bookkeeping)

account: HOST_123   +80 credits  (workload fee)
account: TREASURY   +20 credits  (platform fee)
account: USER_456   -100 credits (workload payment)

6. Service Catalog

Packaging Model

Each service is a Firecracker-ready package:

Root filesystem image (ext4): Minimal Linux rootfs (Alpine/Debian-slim) with app pre-installed
Kernel image: Shared minimal Linux kernel (5.10 LTS, Firecracker-compatible)
Service manifest (service.yaml):

name: nextcloud
version: "28.0.1"
description: "Self-hosted cloud storage and collaboration"
category: "storage"
license: "AGPL-3.0"

resources:
  min_vcpus: 1
  min_memory_mb: 512
  min_disk_gb: 10
  recommended_vcpus: 2
  recommended_memory_mb: 2048

rootfs:
  image: "registry.openinfra.io/services/nextcloud:28.0.1"
  size_mb: 850
  checksum_sha256: "abc123..."

kernel:
  image: "registry.openinfra.io/kernels/vmlinux-5.10.217"

networking:
  ports:
    - container_port: 80
      protocol: tcp
      public: true

health_check:
  type: http
  path: /status.php
  interval_seconds: 30
  timeout_seconds: 5

environment:
  - name: ADMIN_USER
    required: true
  - name: ADMIN_PASSWORD
    required: true
    secret: true

volumes:
  - mount_path: /var/www/html/data
    size_gb: 10
    persistent: true

tags: ["cloud", "storage", "collaboration"]

AI Polishing Layer (Phase 5)

Automated pipeline that discovers popular self-hostable open-source apps on GitHub, uses Claude API to generate Firecracker-ready service packages, and provides admin review before publishing.

Pipeline stages: discovered → analyzing → building → pending_review → published/rejected

Components:

internal/pipeline/discovery.go — Scans GitHub Search API for repos by topic (selfhosted, docker) with configurable minimum stars
internal/pipeline/analyzer.go — Uses Claude Sonnet to generate service manifest YAML, Dockerfile, and description from repo README/Dockerfile
internal/pipeline/builder.go — Builds Docker image, exports to ext4 rootfs, uploads to registry
internal/pipeline/pipeline.go — Orchestrator running 3 ticker goroutines (discovery 24h, analysis 30m, builds 15m)

Budget controls: Configurable daily caps (default 5 analyses, 3 builds), tracked in ai_usage_daily table. Pipeline stages can be toggled independently via admin API/UI.

Admin UI: React pages at /admin/pipeline for repo list, detail/review, and settings configuration.

7. Networking

Traffic Flow

User (browser) --HTTPS--> Caddy Ingress (control plane)
  --WireGuard tunnel--> Host Agent --tap device--> Firecracker microVM

Implementation

Domain routing: Each workload gets {workload-id}.openinfra.io. Caddy registers routes dynamically via admin API
WireGuard overlay: Each host establishes a tunnel to the control plane. Solves NAT traversal for residential hosts. Each host gets a /30 subnet (e.g., 10.100.x.x/30)
Inter-VM networking: Workloads on different hosts communicate through the WireGuard mesh via overlay IPs
DNS: Wildcard *.openinfra.io -> control plane IP. Caddy handles TLS with Let’s Encrypt
Bandwidth metering: Agent counts bytes in/out per tap device, reports in heartbeats

8. Data Model

Entities

+----------+       +------------+       +----------+
|  User    |1----*>|  Workload  |*<---1 | Service  |
+----------+       +------------+       +----------+
| id (uuid)|       | id (uuid)  |       | id (uuid)|
| email    |       | user_id    |       | name     |
| password |       | service_id |       | version  |
| credits  |       | host_id    |       | manifest |
| role     |       | status     |       | rootfs   |
| created  |       | config     |       | category |
| api_key  |       | vcpus      |       | created  |
+----------+       | memory_mb  |       +----------+
     |             | disk_gb    |
     |             | subdomain  |
     |             | created    |
     |             +------------+
     |                   |
     v                   v
+----------+       +----------+
|Transaction|      |   Host   |
+----------+       +----------+
| id (uuid)|       | id (uuid)|
| from_acct|       | owner_id |
| to_acct  |       | hostname |
| amount   |       | ip_addr  |
| type     |       | wg_pubkey|
| ref_id   |       | status   |
| created  |       | region   |
|          |       | vcpus    |
+----------+       | memory_mb|
                   | disk_gb  |
                   | reliability|
                   | last_hb  |
                   | created  |
                   +----------+
                        |
                   +----------+
                   |HostMetric|
                   +----------+
                   | id       |
                   | host_id  |
                   | cpu_pct  |
                   | mem_pct  |
                   | disk_io  |
                   | net_in   |
                   | net_out  |
                   | timestamp|
                   +----------+

Key Statuses

Host: pending_verification, online, degraded, offline, banned
Workload: pending, scheduling, starting, running, stopping, stopped, failed, rescheduling
Transaction type: deployment_fee, host_reward, epoch_reward, topup, withdrawal, refund

PostgreSQL Notes

BIGINT for credits (stored as millicredits to avoid floating point)
Partition host_metrics and transactions by month
FOR UPDATE SKIP LOCKED for scheduler queue to allow concurrent schedulers later

9. API Design

REST API (User-facing, port 8080)

Auth: JWT Bearer tokens (Ed25519, 15min access / 7day refresh)
Rate: 100 req/min per user, 1000 req/min per IP

POST   /api/v1/auth/register
POST   /api/v1/auth/login
POST   /api/v1/auth/refresh

GET    /api/v1/user/profile
PUT    /api/v1/user/profile
GET    /api/v1/user/transactions

GET    /api/v1/catalog/services
GET    /api/v1/catalog/services/{id}
GET    /api/v1/catalog/categories

POST   /api/v1/workloads
GET    /api/v1/workloads
GET    /api/v1/workloads/{id}
PUT    /api/v1/workloads/{id}
DELETE /api/v1/workloads/{id}
POST   /api/v1/workloads/{id}/restart

POST   /api/v1/credits/topup
GET    /api/v1/credits/balance
GET    /api/v1/credits/usage

# Admin (role: admin)
GET    /api/v1/admin/hosts
PUT    /api/v1/admin/hosts/{id}/status
GET    /api/v1/admin/stats
POST   /api/v1/admin/catalog/services
PUT    /api/v1/admin/catalog/services/{id}
GET    /api/v1/admin/rewards/config
PUT    /api/v1/admin/rewards/config

gRPC API (Host Agent, port 9090, mTLS)

service HostService {
  rpc Register(RegisterRequest) returns (RegisterResponse);
  rpc Heartbeat(stream HeartbeatMessage) returns (stream ControlMessage);
  rpc GetImage(ImageRequest) returns (stream ImageChunk);
  rpc AssignWorkload(WorkloadAssignment) returns (WorkloadStatus);
  rpc StopWorkload(WorkloadStopRequest) returns (WorkloadStatus);
  rpc ReportMetrics(stream MetricsReport) returns (Ack);
  rpc VerifyChallenge(ChallengeRequest) returns (ChallengeResponse);
}

10. Security

Host Verification

Registration challenge: Prove KVM access by running a micro benchmark VM
mTLS: Client certificates signed by platform CA, rotated every 30 days
Host attestation: Periodic challenges to prove workload is running (e.g., hash of file X inside VM Y)
Anti-fraud: Impossible metrics flagged for review. 24-hour dispute window on payouts

Workload Isolation

Firecracker microVMs: Dedicated kernel, separate network namespace, rate-limited resources
Resource caps: Hard vCPU/memory limits via Firecracker, disk I/O via cgroups
Network isolation: Dedicated tap device per VM, iptables rules prevent cross-VM access
No host access: VMs cannot reach host filesystem, network, or other VMs

API Security

JWT: Short-lived access tokens (15 min), longer refresh (7 days), Ed25519 signed
Rate limiting: Token bucket per-user and per-IP
Input validation: Struct tags with go-playground/validator
CORS: Strict origin allowlist
API keys: For programmatic access, bcrypt-hashed
Audit log: Append-only table for admin actions

Secrets Management

Passwords/API keys: bcrypt (cost 12)
Workload env secrets: AES-256-GCM at rest, master key in environment variable
No secrets in code, config, or logs

11. Project Structure

openinfra/
├── ARCHITECTURE.md
├── CLAUDE.md
├── Makefile
├── go.mod
├── go.sum
├── Dockerfile                   # Multi-stage build (Go API + React SPA)
├── docker-compose.yml           # Local dev (Postgres, MinIO, all 6 migrations)
├── deployments/
│   ├── docker-compose.prod.yml
│   ├── caddy/Caddyfile
│   └── systemd/
│       ├── openinfra-api.service
│       └── openinfra-agent.service
├── cmd/
│   ├── api-server/main.go       # Control plane entry point
│   ├── agent/main.go            # Host agent binary
│   ├── scheduler/main.go
│   ├── reward-engine/main.go
│   └── cli/main.go              # Admin CLI
├── proto/openinfra/v1/
│   ├── host.proto
│   ├── workload.proto
│   └── metrics.proto
├── internal/
│   ├── api/
│   │   ├── middleware/ (auth, ratelimit, cors, audit, prometheus, security)
│   │   ├── handlers/ (auth, catalog, workload, credits, admin, user, dashboard, governance)
│   │   ├── docs/ (embedded Swagger UI + OpenAPI spec)
│   │   ├── docs.go
│   │   ├── spa.go              # SPA file server (serves React build with client-side routing fallback)
│   │   └── router.go
│   ├── grpc/
│   │   ├── server.go
│   │   ├── host_service.go
│   │   └── interceptors/ (auth, logging, metrics)
│   ├── agent/ (agent, firecracker, metrics, networking, storage, updater)
│   ├── scheduler/ (scheduler, queue, reschedule)
│   ├── rewards/ (engine, epoch calculation)
│   ├── payments/ (deposits, evm, solana, price oracle)
│   ├── catalog/ (service, manifest, registry)
│   ├── ledger/ (ledger, transaction, billing)
│   ├── health/ (prober, challenge probes)
│   ├── auth/ (jwt, password, mtls)
│   ├── certs/ (rotator, ca, certs)
│   ├── governance/ (types, service, executor)
│   ├── pipeline/ (discovery, analyzer, builder, orchestrator)
│   ├── metrics/metrics.go
│   ├── config/config.go
│   └── db/
│       ├── postgres.go
│       ├── migrations/
│       └── queries/
├── pkg/
│   ├── models/ (host, workload, service, user, transaction)
│   ├── crypto/keys.go
│   └── version/version.go
├── scripts/
│   ├── build-rootfs.sh
│   ├── gen-proto.sh
│   └── setup-dev.sh
├── api/openapi.yaml              # OpenAPI 3.0 spec (source of truth)
├── test/
│   ├── e2e/
│   └── load/ (k6 scripts)
└── web/                         # React SPA (Phase 4+5) — 70 tests (Vitest + Testing Library)

12. Tech Stack

Backend (Go)

Purpose	Library
HTTP Router	`github.com/go-chi/chi/v5`
gRPC	`google.golang.org/grpc` + `protoc-gen-go`
Postgres driver	`github.com/jackc/pgx/v5`
SQL generation	`github.com/sqlc-dev/sqlc`
Migrations	`github.com/golang-migrate/migrate/v4`
JWT	`github.com/golang-jwt/jwt/v5`
Validation	`github.com/go-playground/validator/v10`
Config	`github.com/caarlos0/env/v10`
Logging	`log/slog` (stdlib)
Firecracker	`github.com/firecracker-microvm/firecracker-go-sdk`
WireGuard	`golang.zx2c4.com/wireguard/wgctrl`
SQLite (agent)	`github.com/mattn/go-sqlite3`
Rate limiting	`golang.org/x/time/rate`
Metrics	`github.com/prometheus/client_golang`
Testing	`testing` + `github.com/stretchr/testify`
Web testing	Vitest + React Testing Library (70 tests)

Infrastructure

Purpose	Tool
Database	PostgreSQL 16 (self-hosted)
Reverse proxy	Caddy 2
Image storage	MinIO / Backblaze B2
VPN overlay	WireGuard
CI/CD	GitHub Actions
Monitoring	Prometheus + Grafana

13. MVP Phases

Phase 1: Foundation (Weeks 1-4)

Host agent can register, launch a Firecracker VM, and report metrics.

Phase 2: Workload Scheduling + Networking (Weeks 5-8)

Users can deploy a service from the catalog and access it via a public URL.

Scheduler wired into workload creation (credit check + host selection + control messages)
Workload lifecycle (assign, start, stop, reschedule on host failure)
Service catalog YAML manifest parser + loader
Image registry integration (IPFS + file:// with LRU disk cache)
WireGuard mesh networking (Curve25519 keypair, deterministic IP assignment)
Caddy dynamic reverse proxy (admin API route management)
Hourly billing loop with auto-stop on insufficient credits
Rescheduling on host failure (max 3 attempts, host exclusion)
Database migration 002 (reschedule_count, last_billed_at, wireguard_endpoint)

Phase 3: Rewards + Credits (Weeks 9-12)

Hosts earn credits, users spend credits.

Phase 4: Production Hardening (Weeks 13-16)

Ready for public launch.

Phase 5: Scale + Ecosystem (Post-Launch)

Status Tracking

Completed

Item	Date
Architecture document	2026-03-06
Technology decisions (Go, Firecracker, Postgres, gRPC)	2026-03-06
Project scaffold (go.mod, Makefile, directory structure)	2026-03-06
PostgreSQL schema + migrations (6 tables)	2026-03-06
Docker Compose (postgres + api-server + ipfs)	2026-03-06
sqlc query layer (type-safe DB access, 6 query files)	2026-03-07
Auth package (bcrypt + JWT access/refresh tokens)	2026-03-07
Auth handlers (register, login, refresh) + JWT middleware	2026-03-07
All REST API endpoints implemented (auth, catalog, user, workloads, credits, admin)	2026-03-07
gRPC host service (register, heartbeat, metrics, challenge)	2026-03-07
Proto code generation (protoc + protoc-gen-go)	2026-03-07
Host agent: gRPC client (register, heartbeat stream, control messages)	2026-03-07
Scheduler (score-based host selection with resource filtering)	2026-03-07
Ledger (credit deduction, hourly billing, insufficient balance handling)	2026-03-07
Stale host detection (90s timeout background goroutine)	2026-03-07
Unit tests (auth, middleware, ledger, scheduler, config)	2026-03-07
E2E tests (health, full auth flow, catalog, credits)	2026-03-07
Real system metrics collection (/proc parsing: CPU, memory, disk, network)	2026-03-07
CI pipeline (GitHub Actions: lint, test with PostgreSQL, build, e2e)	2026-03-07
mTLS for gRPC (ECDSA P256 certs, TLS 1.3, dev cert generator)	2026-03-07
Pluggable VM backend (noop + Firecracker, socket-based API)	2026-03-07
Scheduler wired into workload creation (credit check + host assignment + control messages)	2026-03-07
Hourly billing loop with auto-stop on insufficient credits	2026-03-07
Service manifest parser (YAML) + catalog loader	2026-03-07
Rescheduling on host failure (max 3 attempts, host exclusion)	2026-03-07
WireGuard mesh networking (keypair gen, deterministic IP, peer management)	2026-03-07
Caddy dynamic reverse proxy (admin API route management)	2026-03-07
Image registry (IPFS + file:// with LRU disk cache)	2026-03-07
Database migration 002 (reschedule_count, last_billed_at, wireguard_endpoint)	2026-03-07
Reward engine (hourly epoch, uptime/resource/quality factors, idempotent)	2026-03-08
Challenge prober (HTTP health checks every 5 min, quality_factor tracking)	2026-03-08
Crypto deposit service (EVM + Solana, HD wallet derivation, deposit polling)	2026-03-08
Price oracle (CoinGecko, 5 min cache, USDC/ETH/SOL to USD conversion)	2026-03-08
User dashboard API (summary, spending, balance, workload costs)	2026-03-08
Host dashboard API (rewards history, probe results, resource usage)	2026-03-08
Database migration 003 (reward_epochs, host_rewards, challenge_probes, deposits)	2026-03-08
Dashboard + deposit REST routes wired into router	2026-03-08
Background goroutines: reward engine (1h), prober (5m), deposit poller (30s)	2026-03-08
Unit tests (rewards, prober, deposits, price oracle)	2026-03-08

| Rate limiting (per-IP + per-user, golang.org/x/time/rate) | 2026-03-08 | | CORS middleware (configurable origins) | 2026-03-08 | | Audit logging (async buffered channel, admin endpoint) | 2026-03-08 | | Security hardening (HSTS, CSP, body size limit, JWT validation) | 2026-03-08 | | Prometheus metrics (HTTP, gRPC, business, background jobs) | 2026-03-08 | | Prometheus middleware (chi + gRPC interceptors) | 2026-03-08 | | mTLS certificate rotation (file watcher, dynamic callbacks) | 2026-03-08 | | Internal CA for agent cert signing (ECDSA P256) | 2026-03-08 | | OpenAPI 3.0 spec + embedded Swagger UI | 2026-03-08 | | React SPA (login, dashboard, catalog, workloads, credits, deposits) | 2026-03-08 | | Production deployment config (Docker Compose, Caddy, systemd, backups) | 2026-03-08 | | CI/CD updates (migrations 002-004, web build, GHCR release) | 2026-03-08 | | Load testing scripts (k6: auth, workloads, catalog) | 2026-03-08 | | Database migration 004 (audit_log table) | 2026-03-08 | | AI pipeline: GitHub discovery (search API, topic scanning) | 2026-03-08 | | AI pipeline: Claude analyzer (manifest + Dockerfile generation) | 2026-03-08 | | AI pipeline: Rootfs builder (Docker build → ext4 conversion) | 2026-03-08 | | AI pipeline: Orchestrator (3 ticker goroutines, budget controls) | 2026-03-08 | | AI pipeline: Admin API (repos CRUD, review, config, stats) | 2026-03-08 | | AI pipeline: Prometheus metrics (discoveries, analyses, builds, reviews) | 2026-03-08 | | AI pipeline: React admin UI (dashboard, detail/review, config) | 2026-03-08 | | Database migration 005 (discovered_repos, repo_analyses, rootfs_builds, pipeline_reviews, pipeline_config, ai_usage_daily) | 2026-03-08 | | Web test suite (70 tests, Vitest + React Testing Library) | 2026-03-09 | | Docker multi-stage build includes React SPA | 2026-03-09 | | SPA file server in API router (client-side routing fallback) | 2026-03-09 | | docker-compose.yml updated with all 6 migrations | 2026-03-09 | | E2E test suite expanded (10 test functions, all endpoints) | 2026-03-09 | | CI workflow: web-test job (npm test + tsc) | 2026-03-09 | | Grafana dashboard provisioned (16 panels, full observability) | 2026-03-09 | | Production docker-compose updated (6 migrations, pipeline env) | 2026-03-09 | | Deploy script updated (iterates all migrations) | 2026-03-09 | | .env.example updated (pipeline config) | 2026-03-09 | | DAO governance: migration 006 (governance_config, stakes, proposals, votes) | 2026-03-09 | | DAO governance: internal/governance/ package (stake, propose, vote, tally, execute, unbonding) | 2026-03-09 | | DAO governance: REST API handlers + routes (proposals, votes, staking, admin config) | 2026-03-09 | | DAO governance: executor background goroutine (5 min tick: tally, execute, unbonding) | 2026-03-09 | | DAO governance: Prometheus metrics (proposals, votes, tallies, executions, staked credits) | 2026-03-09 | | DAO governance: React UI (proposal list, detail/vote, create, staking) | 2026-03-09 | | DAO governance: 16 Go unit tests + 19 web tests | 2026-03-09 | | Coledex pilot Phase 1: internal/secrets module (age envelope encryption, host key gen/load/persist with strict 0600 perms, multi-recipient encrypt, roundtrip + tamper + perms tests, 81.2% coverage) | 2026-05-13 |

Coledex Batch-Job Pilot (v2 plan)

Plan: invite-only providers + container runtime + internal credit ledger + Coledex as first batch-workload customer. Tokenomics, Firecracker, IPFS, public registration, and fiat payouts all explicitly deferred.

Done

Item	Date
Phase 1 — `internal/secrets`: age envelope encryption (host key gen/load/persist 0600, multi-recipient encrypt, tamper-rejection tests, 81.2% coverage)	2026-05-13
Phase 2a — `internal/agent/container`: Runtime interface, Docker HTTP-over-unix-socket impl (zero new heavy deps; no docker/docker SDK), Noop impl, Config validation, label-based reaping, tmpfs mounts for secrets. 91.8% coverage.	2026-05-13
Phase 2b — proto: `WorkloadAssignment.runtime / image / image_digest / env_encrypted / command`; `RegisterRequest.age_recipient / pricing_credits_per_cpu_hour / pricing_credits_per_gb_hour / geo / bandwidth_mbps / invite_token`. Migration 007 adds same columns to `workloads` and `hosts` with safe defaults + CHECK constraint on runtime ∈ {firecracker, container}. Fixed `scripts/gen-proto.sh` (was silently producing nothing without `--proto_path=.`).	2026-05-13
Phase 2c — `internal/invites`: single-use, IP-bindable, time-limited host registration tokens. 32-byte entropy, sha256-hashed at rest (plaintext never persisted), atomic single-use commit, default 24h TTL + 14d cap. MemoryRepo + concurrent-consume test proving only one of N goroutines wins. Migration 008 adds `host_invites` table. 92.0% coverage. Admin handler + gRPC Register integration deferred to Phase 2c-wiring.	2026-05-13
Phase 3 — `internal/settlement`: two-sided ledger. Entry kinds `workload_debit` / `provider_credit` / `platform_fee` / `mint`. Default platform fee 10%, capped at 50%. Atomic trio with per-batch idempotency (replay returns existing entries; partial replay rejected). Chain-agnostic accounts (`internal:user:<uuid>`, `internal:host:<uuid>`, future `solana:<pubkey>` / `evm:<0x...>`). `VerifyConservation` invariant proves debits == credits + fees, tested across 500 random settlements + concurrent settlements. Migration 009 adds `amount_raw NUMERIC(20,0)`, `idempotency_key`, `settled_onchain_at`, `chain_tx_hash` to existing `transactions` table for future onchain batch settlement. 93.8% coverage.	2026-05-13
Phase 3.5 — `internal/onchain/solana`: Solana SPL-token settler (gagliardetto/solana-go), batches up to 10 transfers per tx, signs with treasury keypair, broadcasts + confirms. `internal/settlement/cron.go`: `OnchainCron.RunOnce` scans `provider_credit` rows in `solana:*` namespace and submits via injected `Submitter`. Devnet end-to-end test confirmed (TX 56CVQaAGmnyDwbcAsaXN7iqa6Dkf2vQ7GWxTYkssfKDftvKu2ES158rrEgRptqjfwtt5qW13vDn4Y31JWcnPcf7R) — 1 raw unit transferred Go → devnet → recipient balance moved. NoopSettler for unit tests. Solana CLI installed at `~/.local/share/solana`. Devnet treasury (`GzuUbku2wtBHwpvhqDdowyRKCoBiyknNG4SQkJV2TC1T`) and OINFRA-test mint (`9Jkq8WdgUUp2AR4FeXwE6q4DRddoHGfcKREMeCE6wphT`, 6 decimals, 1M supply) live. Coverage: solana 89.9%, settlement 93.6%. Mainnet deployment + Coledex/provider wallet UX deferred to Phase 7.	2026-05-14
Block A (2c-wiring) — invite-gated registration: `internal/db/queries/host_invites.sql` (Insert/Get/MarkConsumed/List/Delete/Count), `internal/invites/sql_repo.go` + `ConsumeWithTx` helper, admin REST handlers (`POST/GET/DELETE /api/v1/admin/host-invites`) under `RequireAdmin` middleware, `internal/grpc/server.go` Register now opens a tx, calls `Service.Validate` → `CreateHost` → `ConsumeWithTx` atomically; gRPC peer IP feeds the bound_ip check. `mapInviteErr` maps sentinel errors → `codes.PermissionDenied`. sqlc regenerated for migrations 007/008/009 columns (Transaction, Host, Workload structs). Pure-helper tests + handler integration via MemoryRepo: invites pkg 61.8% (SQL paths exercised by E2E), grpc pkg 8.4% (mapInviteErr + peer extraction covered). Plaintext invite tokens never persisted nor surfaced after creation.	2026-05-15
Block B (Phase 3 wiring) — settlement engine into the system: proto `WorkloadInfo` gains `exit_code` / `exited_at_unix` / `final_status` / `cpu_seconds_used` / `memory_mb_seconds_used`, `RegisterRequest.solana_payout_pubkey`. Migration 010 adds `workloads.settled_at` + `exit_code` (partial index `idx_workloads_unsettled`), `hosts.solana_payout_pubkey`, `services.kind` + `config` (CHECK + idx) for Block D prep. New `internal/pricing` package — pure integer micro-credit math, 100% coverage, defaults `RATE_PER_CPU_MICRO=1_000_000` (1 credit/cpu-sec), `RATE_PER_MB_MICRO=1_000` (0.001 credit/MB-sec). `internal/settlement/sql_repo.go` implements `Repository` against sqlc; rejects partial replays via `ErrPartialReplay`; reused-in-tx variant `AppendAtomicInTx`. New sqlc queries: `AppendLedgerEntry` (ON CONFLICT idempotent), `GetLedgerEntryByIdempotencyKey`, `LedgerBalance`, `LedgerSumByKind`, `ListUnsettledOnchain`, `MarkLedgerEntrySettledOnchain`, `LockUnsettledWorkload` (SELECT FOR UPDATE on `settled_at IS NULL`), `MarkWorkloadSettled`. `internal/api/handlers/settlement.go`: `POST /api/v1/admin/credits/mint` (admin-only, idempotency-keyed); `GET /api/v1/host/{host_id}/earnings` (owner-or-admin ACL). gRPC `Heartbeat` now observes `WorkloadInfo.final_status` and runs the per-workload settlement tx: row-lock → build trio → `AppendAtomicInTx` → `MarkWorkloadSettled`, all inside one pgx.Tx. Retransmits hit the row-lock guard and short-circuit safely. Provider account namespaced to `solana:<pubkey>` when `hosts.solana_payout_pubkey` is set (ready for Block C onchain executor). Env knobs `RATE_PER_CPU_MICRO` / `RATE_PER_MB_MICRO` / `PLATFORM_FEE_BPS` wired in `cmd/api-server/main.go`. Tests: pricing 100%, settlement 71.7%, handlers + grpc green; SQL paths covered by E2E. `go vet` + full suite clean.	2026-05-15
Block C (Phase 3.5 wiring) — onchain bridge: `internal/onchain/solana/treasury.go` bootstraps the treasury keypair age-encrypted at rest. First call generates a fresh wallet (printed to stderr as `TREASURY_PUBKEY=…` for operator funding), subsequent calls decrypt with the host key. Supports both JSON 64-byte arrays (solana-keygen format) and raw binary 64-byte secret keys. Corrupted/wrong-key load fails loudly — never silently regenerates. `internal/onchain/solana/submitter.go` promotes the `submitterAdapter` from `cmd/devnet-demo` to a public `Submitter` type satisfying `settlement.Submitter`; `cmd/devnet-demo/main.go` refactored to use it (5-line change). `internal/settlement/sql_onchain_repo.go` implements `OnchainRepository` via sqlc queries already added in Block B (`ListUnsettledOnchain`, `MarkLedgerEntrySettledOnchain`). Single-instance executor in v1; horizontally-scaling deployment will upgrade to `FOR UPDATE SKIP LOCKED` (note in source). `internal/settlement/executor.go` drives `OnchainCron.RunOnce` on a 5-minute ticker with a start-tick so backlog drains immediately after deploys; panics in cron are recovered + logged so one bad row can’t crash the goroutine. `MetricsSink` interface decouples it from prometheus (tests inject `recordingSink`; main.go injects `prometheusSink` adapter). Three new metrics: `openinfra_settlement_onchain_submitted_total`, `_failed_total`, `_last_tick_unix`. `cmd/api-server/main.go::startOnchainExecutor` wires the entire chain (LoadHostKey → LoadOrGenerateTreasury → SolanaSettler → Submitter → OnchainCron → Executor) gated by `SOLANA_MINT` env var; defaults to devnet RPC. New config: `OPENINFRA_DATA_DIR`, `OPENINFRA_HOST_KEY`, `SOLANA_MINT`. Tests: solana 82.0%, settlement 69.6% (executor + sql_onchain_repo helpers covered; SQL roundtrips via E2E).	2026-05-15
Block H (Phase 5) — agent register fix + simulated mp3 provider: critical Block-A integration gap fixed — agent’s `register()` now reads `INVITE_TOKEN` + `SOLANA_PAYOUT_PUBKEY` from env and sends both in `RegisterRequest`. `CreateHost` SQL extended with `solana_payout_pubkey` column-binding (sqlc regen); server passes `req.SolanaPayoutPubkey` (trimmed) into `CreateHostParams`. Without this fix, the invite-gated control plane rejected every Register call with `PermissionDenied: invite_token required` — Block A’s gate was wired server-side but never client-side. `images/openinfra-agent/Dockerfile` packages the agent as a multi-stage Alpine image (~25 MB) used by the `agent-mp3-sim` compose service. Full suite green; `go vet` clean.	2026-05-16
Block J (Phase 5) — platform gaps surfaced by the second end-to-end pilot run (the one that proved the workload completes but bills 0): (1) Usage precision: `WorkloadInfo.cpu_seconds_used` / `memory_mb_seconds_used` proto fields renamed to `cpu_ms_used` / `memory_mb_ms_used` (millisecond precision); the agent’s cgroup CPU counter is now reported as `ns / 1e6` instead of `ns / 1e9` (which truncated every sub-second workload to 0). `usage_sampler.go` integrates memory in MB·milliseconds; sample interval dropped from 10 s → 1 s so short-lived containers get at least one in-flight reading before Docker GC’s the cgroup. New `pricing.ComputeMs(vcpus, memoryMB, cpuMs, memMBMs, ratePerCPUMicro, ratePerMBMicro)` does the same per-unit-per-second rate math at ms input scale (`/ msPerSecond` factor) and carries the same half-up rounding; the legacy `Compute` (seconds) is kept for the per-VM hourly billing path. `grpc.Heartbeat` settlement path reads the new fields and calls `ComputeMs`. Tests: `TestUsageSampler_SubSecondCPU` (320 ms → 320 ms recorded, not 0); `TestComputeMsParityWithSeconds` (60_000 ms input matches `Compute(_, _, 60, …)` at whole-second boundary); `TestComputeMsSubSecondCPUBillsNonZero` (500 ms × default rate rounds to 1 credit, not 0). (2) SPL ATA bootstrap: `solana.SolanaSettler.Settle` previously failed with `InvalidAccountData` on the first transfer to any provider (their wallet had no Associated Token Account for the OINFRA-test mint yet). Now prepends one `associated-token-account.NewCreateIdempotentInstruction(payer=treasury, wallet=recipient, mint=…)` per unique recipient in the batch, deduplicated by `solana.PublicKey` map so a multi-row batch to one recipient pays rent once. Idempotent variant is a no-op when the ATA already exists, so no `getAccountInfo` pre-check round-trip is needed. Treasury pays the ~0.002 SOL rent (funded by the existing devnet airdrop loop in `control-plane-up.sh`). Tests: `TestSettleHappyPath` updated to assert 4 instructions (2 ATA-create + 2 transfer), `TestSettleMixedZeroAndPositive` updated to 2 instructions, new `TestSettleDeduplicatesRecipients` (3 transfers to same recipient = 1 ATA-create + 3 transfers). End-to-end verified: workload `ef9be71e-97a7-49ed-97a9-d650761745f2` (988 ms CPU, 28 MB·ms memory) settled `amount=1` credit on the internal ledger AND landed onchain at tx `2FHVJ8keeJfcnpNCQTwc45mxcpqYMWmMRW994MzUXd7x4zi4gJxcYUQN3BYok9uRqLobSQ5fvy4hUvXk2EcdCwg6` — recipient `FXLcpzsfbkHV7Qw9TsfiYrNPUhCbkawPcqPrXxY829Wc` now holds 1 raw unit (0.000001 OINFRA-test). Full suite + `go vet` clean. 3 h soak (concluded early at 2026-05-17T00:14:47Z, ~36 onchain executor ticks): `openinfra_settlement_onchain_failed_total` flat at 0, `submitted_total` flat at 1, tick lag 196 s (cap 360), api-server 0 restarts / 0 ERROR or WARN log lines, Kuma agent `active`, ledger trio + onchain SPL balance intact. Soft signal: `agent-mp3-sim` flapping every ~60 s (175 → 236 restart count over the second hour) — the known agent-host-id-persistence gap (Pending), billing path unaffected because Kuma is the real provider and registers with its own persisted (in-DB) host row.	2026-05-16
Block I (Phase 5) — platform gaps surfaced by the first end-to-end pilot run: 6 inline workarounds in `deploy/local-pilot/run-pilot.sh` replaced with first-class platform support. (1) Migration 011 seeds the system user (`id=00000000-…-000000000000`, `role='system'`) that `grpc.Register` uses as the placeholder owner for invite-onboarded hosts; `auth.Login` rejects `role='system'` as defense in depth. (2) New `cmd/cli` (`openinfra-cli`) with `promote <email>` and `demote <email>` subcommands — operator-scoped first-admin bootstrap that connects directly to the control-plane Postgres (`DATABASE_URL`); deliberately not an API endpoint to avoid drive-by privilege escalation if JWT signing ever leaks. Refuses to clobber `role='system'`. (3) New `GET /api/v1/admin/users` with optional `?email=` exact-match filter; never returns `password_hash` or `api_key_hash` (explicit allowlist via `userToMap`). (4) `POST /api/v1/admin/catalog/services` now accepts `kind` (‘service'	'batch_job’) + `config` (JSONB); `?upsert=true` query flag switches to idempotent `UpsertService` semantics so operator scripts get a single round-trip regardless of whether the row exists. `PUT /api/v1/admin/catalog/services/{id}` preserves existing `kind`/`config` on partial updates. Pipeline + governance executor `CreateService` callers updated for the new sqlc signature. (5) `GET /api/v1/admin/hosts` JSON now includes `solana_payout_pubkey` so operator tooling can match a host by its payout wallet without direct DB access. (6) `container.DockerRuntime.Create` now auto-pulls the image on `/containers/create` 404 (“No such image”): mirrors `docker run` behavior, streams the JSON event stream from `/images/create`, surfaces engine `{"error":...}` events as failures. New `splitImageRef` helper handles `repo:tag`, `host:port/repo:tag`, and `repo@sha256:...` digest refs. 4 new tests cover pull-on-missing, pull-failure, non-image-missing pass-through, and split-ref parsing. `run-pilot.sh` rewritten — `/admin/users?email=` lookup, `POST /admin/catalog/services?upsert=true` jq-built body, `/admin/hosts?limit=100` client-side payout filter; no direct `docker exec postgres psql` calls left, no `ssh kuma docker pull` step. docker-compose mounts migration 011 alongside 001–010. Tests: container 91% (was 92% — denominator grew from new tests, all green); full suite + `go vet` clean.
Block G (Phase 5) — local-pilot deployment assets: `deploy/local-pilot/docker-compose.yml` boots the full control-plane stack (Postgres + migrations 001–010, api-server, OCI registry, Minio + bucket-init, customer-pg with synthetic 50 MB seed, Prometheus, Grafana with anonymous Viewer + provisioned pilot dashboard). `control-plane-up.sh` is the one-command bring-up — builds the api-server image, brings up compose, waits for `/healthz`, greps the api-server logs for `TREASURY_PUBKEY=…`, idempotently airdrops 1 SOL on devnet only if the treasury balance is under 0.1 SOL (don’t burn the faucet on re-runs). `provider-install.sh <ssh-host> <cp-ip> [arch]` cross-compiles the agent, scp’s it, drops a systemd unit, configures Docker insecure-registry trust for the LAN registry, and leaves the agent stopped pending `/etc/openinfra/agent.env`. `run-pilot.sh` walks the 8-step lifecycle (admin login + customer + 100k credits + 2 invites + register kuma + mp3 + build + push image + create batch_job pinned to kuma + poll for completion + verify SPL transfer + verify backup in Minio); bails out with a clear error at any divergence. `docs/LOCAL-PILOT.md` is the operator runbook (preflight, bring-up, verification commands, common troubleshooting, 7-day soak loop, teardown). `docs/HETZNER-MIGRATION.md` is the leanest paid path — CX22 (~€4.50/mo) + Cloudflare DNS + LE TLS via Caddy + Tailscale free-tier overlay (CP↔providers) + Backblaze B2 (S3 swap of Minio); rollback procedure preserves treasury via age-encrypted keypair move. Pilot stays on the 3-machine LAN topology (mp1 control plane, kuma + mp3 providers); migration triggered only after a clean 7-day soak with zero `openinfra_settlement_onchain_failed_total`.	2026-05-16
Block F (Phase 7) — tokenomics spec + validator: `docs/TOKENOMICS.md` ratifies the economic model — units (credit vs OINFRA, 1:1 peg for v1), authorities (treasury hot wallet, admin, governance, operator), supply (fixed 1M on devnet; mainnet open questions enumerated), governable parameter bounds (per-key Min/Max + per-proposal MaxStepDelta with rationale), treasury policy (fees → `internal:treasury`, spend only via `treasury_spend` proposal), emergency policy (no v1 mint override — fork-only). New `internal/tokenomics` package: `Bound{Min,Max,MaxStepDelta}` table with all 9 `AllowedParamKeys`, `ValidateBounds(key, proposed)` (runs at proposal-submit) and `ValidateTransition(key, current, proposed)` (runs at execute; subsumes bounds + rate-of-change cap). 100% test coverage; pinned-bounds test surfaces silent table edits in review. Wired into `governance.ValidateContent` (param_change proposals are now rejected at submit if value parses non-numeric or out of bounds) and `executor.executeParamChange` (transition check against current value before any DB write). New `currentParamValue(cfg, key)` helper fails loudly for tokenomics-validated keys that don’t have a `governance_config` column yet (`base_reward_rate`, `fee_split_host_pct`) so future schema migrations are forced explicit rather than rate-limiting against zero. Open Phase-7 questions (inflation curve, mainnet supply, vesting, treasury split, multisig threshold, credit/OINFRA unpeg) enumerated in TOKENOMICS.md §8. Full suite green; `go vet` clean.	2026-05-15
Block E (Phase 5) — agent usage measurement: `container.Stats` type + `Runtime.Stats(ctx, id)` method added to the runtime interface. `DockerRuntime.Stats` hits `/containers/{id}/stats?stream=false&one-shot=true` and parses `cpu_stats.cpu_usage.total_usage` (cumulative nanoseconds across all cores) and `memory_stats.usage` (bytes RSS). `NoopRuntime.Stats` returns zero by default, primable via `SetStats` for deterministic tests. New `internal/agent/usage_sampler.go`: per-workload sampler runs a goroutine that polls Stats every 10 s, integrates memory-MB-seconds via Riemann sum over wall-clock deltas, and tracks the monotonic CPU counter (never lets it roll backwards on engine restart). `agent.monitorCompletion` now launches the sampler in parallel with `Wait`, takes a final 5 s-bounded Stats reading after exit to capture terminal CPU, and feeds `cpuSec` + `memMBSec` into `recordCompletion`. Heartbeat-reported `WorkloadInfo.CpuSecondsUsed` / `MemoryMbSecondsUsed` are now non-zero — settlement bills real usage instead of `pricing.Compute(_, _, 0, _, _) = 0`. Stats failures (transient daemon hiccups) log at debug and don’t terminate the sampler loop. Tests: container 92.2% (DockerStats parse + engine-error + bad-JSON + 404 mapping; NoopStats default-zero + primed + ReadAt populated), agent sampler tests cover the Riemann sum, monotonic CPU floor, first-sample-no-area invariant, context-cancel exit, and error-resilience. Full suite green; `go vet` clean.	2026-05-15
Block P (Phase 5) — soak harness operability fixes surfaced mid-run: (1) Double-logged checkpoint output. `checkpoint.sh` already self-tees its formatted block to `checkpoint.log` (so the same script works interactively or under cron); the installed crontab line ALSO redirected stdout with `>> checkpoint.log 2>&1`, so every line landed twice — 7 header records on disk, only 4 unique fires. Cron line rewritten to `>/dev/null 2>>cron-errors.log`: stdout dropped (the script already wrote it), stderr pinned to a separate file so a future `set -u` blow-up isn’t silently mailed to a dead alias. (2) Recurring batch_job trigger. The soak originally relied on a one-shot workload at deploy, after which the freshness gauge climbed monotonically — Block M’s `BackupOneMissed` (26 h) and `BackupStale` (50 h) alerts would fire as “false positives” without a real recurring workload to oscillate against. New `deploy/local-pilot/soak/fire-workload.sh` is the missing CP-side scheduler stand-in: admin-login → POST `/api/v1/workloads` fire-and-forget against the pinned `(service_id=c0577b96…coledex-postgres-backup, host_id=2ae205af…mp3-sim)` pair, with the soak’s age recipient (private key in gitignored `age-key.txt`, mode 0600 — decrypt isn’t part of the soak, we just need well-formed encrypted blobs Block M can observe). Cron at `23 6 * * ` (06:23 local — well clear of the `:17` checkpoint slots). End-to-end validated: workload `e54f07fb-…` succeeded in ~20 s, Block M observed age drop from 63 136 s → 15 s* within one scrape cycle. (3) LAN-IP gotcha in workload payload. First attempt at the daily trigger used `127.0.0.1:5434` for the `coledex_pg_url` — works for the script’s API call (host-local) but the workload container spawned via the agent’s docker socket dials its own loopback, not the host’s. Container hung 4 min on `connection refused`. Fixed: hardcoded `LAN_IP=192.168.0.155` for both source DB and Minio endpoint with a comment explaining why auto-detect via `ip -4 route` is fragile here (dual-NIC `enp1s0`+`wlo1`); operator-local soak harness, not deployed code. Also added the file path to `.gitignore`. Surfaced (not closed) follow-ups for Phase 6: a real CP-side cron/scheduler primitive so customers don’t have to bring their own (the daily fire-cron is the simplest possible substitute, not a product); orphan-host filter in Block M’s SQL so `host_last_seen < now() - 7d` series stop emitting `age_seconds` and stop triggering `BackupStale` (two `ZimaOS` series from previous mp3-sim incarnations are visible during the current soak — known and tolerated). Tests still green; no Go changes.	2026-05-19
Block O (Phase 5) — agent heartbeat auto-reconnect + DATA_DIR config alignment: surfaced by the post-Block-K end-to-end deploy. (1) Reconnect: the agent’s previous Run loop opened ONE bidi `Heartbeat` stream at startup and reused it forever; `stream.Send` errors were logged at ERROR and swallowed. When the api-server restarted (compose recreate, OOM, deploy), every subsequent heartbeat logged `failed to send heartbeat error=EOF` indefinitely without reconnecting — the host fell offline 90 s later (stale-host sweeper) and never recovered. New `openHeartbeatStream` opens a fresh stream + spawns the `handleControlMessages` reader goroutine; new `reconnectHeartbeatStream` closes the dead one and rebuilds with exponential backoff (1 s → 2 s → 4 s → 8 s → 16 s → 30 s cap). Run loop now: on `sendHeartbeat` error → log WARN, call `reconnectHeartbeatStream`, swap the stream variable, continue ticking. `sendHeartbeat` refactored to return the `stream.Send` error wrapped with `fmt.Errorf("heartbeat send: %w", err)` so the caller can detect and act on it. Per-stream-lifetime goroutine spawn means workload assignments resume seamlessly after a reconnect (the old design would have lost them silently — the goroutine that exited at EOF was never replaced). Tests: `TestSendHeartbeat_ReturnsStreamSendError` (errors.Is wrap chain + Send call count), `TestSendHeartbeat_NilOnSuccess`. (2) DATA_DIR config alignment: api-server config has always read `OPENINFRA_DATA_DIR`; agent config historically read `DATA_DIR`. The local-pilot compose for `agent-mp3-sim` set ONLY `OPENINFRA_DATA_DIR`, so the agent silently fell back to `defaultDataDir() = $HOME/.openinfra = /root/.openinfra` inside the container — ephemeral. Block K’s `host_id` file was being written there and lost on every restart, masking the very fix Block K was supposed to enable. Now `LoadAgentConfig` reads both names (OPENINFRA_DATA_DIR wins on conflict, so the api-server-style env is the canonical one going forward); compose updated to set both with an explicit comment. (3) Operational verification under live local-pilot: minted fresh invite, recreated mp3-sim, observed `data_dir=/var/lib/openinfra`, `registered host_id=2ae205af-…`, restarted container, observed `resuming with persisted host_id (skipping Register)` — Block K’s promise delivered end-to-end. Triggered a Coledex backup workload (`78867820-…`) pinned to the new host; succeeded in 8 s, `cpu_ms_used=199`, `memory_mb_ms_used=2837612`. Block M emitted a fresh gauge for the (host, service) pair (age=16.7 s → 256 s by next probe), `scrape_errors_total=0` over 11 ticks. Next onchain executor tick bumped `submitted_total: 1 → 2`, `failed_total` still 0. 7-day soak restarted at 2026-05-19T00:21:28Z (replaces Block J’s 24h baseline which was masked by the agent-mp3-sim flap). `deploy/local-pilot/soak/checkpoint.sh` is a new cron-friendly wrapper that runs `check.sh` and appends timestamped output + container status + backup gauges to `checkpoint.log` for offline audit. Crontab installed (`17 /8 * *`) on the CP host for durable monitoring across Claude sessions. Known orphan series: 2 stale `openinfra_backup_age_seconds` from previous mp3-sim host_ids (50+ h age) will trigger Block M alerts during soak — left visible deliberately as real “missed daily backup” signal, deleting orphan hosts is a follow-up. Full suite + `go vet` clean.	2026-05-19
Block N (Phase 5) — Hetzner CX22 cutover automation: new `deploy/hetzner/` directory turns the previously-manual `docs/HETZNER-MIGRATION.md` runbook into a two-command operator workflow gated by a soak-validation preflight. `preflight.sh` is read-only against the existing local CP — queries the local Prometheus for zero `openinfra_settlement_onchain_failed_total` over the soak window (default 168 h = 7 d), ≥1 successful submission (proves the path is actually exercised, not just absent), executor liveness (`last_tick_unix` ≤ 360 s old), Block-M backup watcher health (≥30 attempts/h, zero scrape errors, every active backup age ≤ 26 h), api-server 5xx rate zero, ≥1 provider host `online`. Exits 0 iff every check passes; designed to be run repeatedly without side effects. `cutover.sh` orchestrates the 7 manual migration steps with three operator-facing safety features: (1) idempotent step markers — each step touches `.step-N.done` on the CX22, re-running resumes from the first incomplete step (a docker pull half-failing mid-migration just needs another invocation, not unwinding state); (2) `--dry-run` prints every `ssh`/`rsync`/`docker` command without executing — gives the operator a full diff of intended actions before any change; (3) `--from-step=N` force-resumes from a specific step, used for partial re-runs after a manual fix-up. The seven steps map 1:1 to the runbook sections: provision (apt + docker + tailscale + ufw), ship (rsync `deploy/hetzner/` + `internal/db/migrations/` + reuses `local-pilot/grafana-provisioning/` + `prometheus.yml`/`alerts.yml` so we don’t fork), api-server image (build locally → `docker save` → ship over ssh → `docker load` — avoids a third-registry hop during cutover), boot (compose up + wait-for-healthz with 60 s budget), DNS (prints exact Cloudflare A records; pauses for operator confirmation), provider re-register (prints the `home-lab/install-agent.sh` invocation with the new hostname + invite-mint curl). New `deploy/hetzner/docker-compose.yml` diverges from `local-pilot/`’s on three axes (no customer-pg / no Minio / no agent-mp3-sim — those live with the customer or provider; Caddy fronts api/registry/grafana with LE TLS; new `pg-backup` sidecar dumps the CP postgres to B2 every 24 h via the same `pg_dump → age → rclone rcat` pipeline as the customer-facing Coledex backup so bugs surface in our own backups first); critical secrets (`POSTGRES_PASSWORD`, `JWT_SECRET`, `GRAFANA_ADMIN_PASS`, `B2_ACCOUNT_ID`, `B2_APPLICATION_KEY`, `DOMAIN`, `ACME_EMAIL`) are required and refuse-to-boot — no insecure defaults inherited from the LAN compose. `Caddyfile` template with HSTS + h1/h2/h3 only + per-host timeout bumps (settlement heartbeats can run > 60 s; registry pushes can be > 10 GB). `pg-backup-entrypoint.sh` is the control-plane self-backup shell loop: POSIX-sh, validates env at start, uses a temp file between pg_dump/age/rclone stages so any failure is pinpointable (busybox sh pipefail can’t tell you which pipe stage broke), restarts via container restart policy on iteration failure. `.env.example` documents every variable with the policy decision behind it (e.g. “leave `SOLANA_MINT` empty during burn-in to disable onchain submission entirely”). `docs/HETZNER-MIGRATION.md` updated with an “Automated path (Block N — preferred)” section pointing at the new scripts while preserving the original manual runbook as authoritative for hand-driven migrations. All scripts bash/sh-syntax-clean; no Go changes.	2026-05-16
Block M (Phase 5) — backup observability: new `internal/backups` package with `Watcher` that ticks once per minute over a sqlc-generated view (`ListBackupJobFreshness`) of the latest successful run per (host_id, service_id) for every `kind='batch_job'` service. Two CTEs in one query: `last_success` (DISTINCT ON over `exit_code = 0` rows) and `last_attempt` (DISTINCT ON over any settled row); rows with no ever-success are omitted by design so the alert rule fires on “metric missing” rather than reporting an absurdly large age. `Watcher` exposes a narrow `Repository` interface + `MetricsSink` interface (same pattern as `settlement.Executor`); test suite uses an in-memory stub repo + recording sink, 6 tests covering happy-path freshness export, repo error → counter increment + propagated error, empty repo non-error, nil-dependency input validation, ticker-runs-until-cancel (≥2 calls in 55 ms with 20 ms interval), and negative-age clamping (CP↔DB clock skew). Five new Prom metrics in `internal/metrics`: `openinfra_backup_age_seconds`, `openinfra_backup_last_attempt_exit_code` (0=success, -1=killed, -2=unknown, other=process), `openinfra_backup_last_attempt_unix` (all 3 keyed by `host_id`/`host_hostname`/`service_id`/`service_name`), plus `_scrape_attempts_total` and `_scrape_errors_total` to distinguish “watcher stuck” from “all backups fresh” from “DB errors”. `cmd/api-server/main.go` hoists the previously-block-scoped `queries := sqlc.New(pool)` to the top scope so both the watcher and the existing manifest loader share one instance; new `backupMetricsAdapter` mirrors `prometheusSink`’s pattern (the `metrics` package stays free of upward deps). New `deploy/local-pilot/alerts.yml` with 4 rules: `BackupStale` (age > 50 h, the “2 consecutive missed runs” alert from the pending item, warning severity, 30-min for-clause to ignore single delayed runs); `BackupOneMissed` (age > 26 h, info severity, used as early warning while the team learns operational tempo); `BackupWatcherErroring` (rate of scrape errors > 0 for 15 min — distinct page because the operator action is “fix the CP DB” not “chase a provider”); `BackupWatcherStuck` (zero scrape attempts in 10 min — wedged goroutine that’s neither erroring nor succeeding). `deploy/local-pilot/prometheus.yml` gains `rule_files: - /etc/prometheus/alerts.yml`; `docker-compose.yml` mounts the new alerts.yml read-only into the prometheus container. Cardinality (hosts × batch_job services) is fine for pilot ≤10 and early prod ≤500; documented in the metric Help text. Full suite + `go vet` clean.	2026-05-16
Phase 3.6 — multi-chain settlement + Algorand: generalized host payout from Solana-only to chain-agnostic `payout_address` + `address_kind` (migration 016 preserves existing `solana_payout_pubkey` data). `internal/settlement` gains `NamespaceAlgorand = "algorand"` + `AlgorandAccount()` alongside existing `SolanaAccount()`. New `internal/onchain/algorand/` package mirrors the Solana seam: `AlgorandSettler` builds ASA transfers via `go-algorand-sdk/v2`, signs with ed25519 treasury derived from age-encrypted 25-word mnemonic, submits through algod REST API; `NoopSettler` for unit tests; `Submitter` adapter satisfies the existing `settlement.Submitter` interface. Proto field `RegisterRequest.solana_payout_pubkey` replaced with `payout_address` + `address_kind` (field 26 reused for wire compat); agent falls back to `SOLANA_PAYOUT_PUBKEY` for backward compat. `gRPC settleCompletedWorkload` switches on `host.AddressKind` to namespace the provider account (`algorand:` / `solana:`). `cmd/api-server/main.go` spawns independent chain executors (Solana + Algorand), each scanning its own namespace on a 5-minute tick, gated by `SOLANA_MINT` / `ALGORAND_ASA_ID`. New config: `ALGORAND_RPC_URL`, `ALGORAND_RPC_TOKEN`, `ALGORAND_ASA_ID`. Provider installer/compose/docs updated for `PAYOUT_ADDRESS` + `ADDRESS_KIND`. 85 test packages green, zero regressions. Chose Algorand as the long-term chain: post-quantum Falcon signatures live on mainnet since Nov 2025, $0.00015 avg fee (cheapest), cited by Google Quantum AI as reference PQC blockchain — vs Solana (Falcon roadmap published but not live, $0.0038) and BNB Chain (ML-DSA-44 research report only, $0.09). Solana kept fully operational as parallel chain. Testnet E2E verified 2026-06-27: ASA 765165188 (OINFRA, 6 decimals, 1M supply) on Algorand testnet; treasury funded + ASA created; test provider `7B6XH4S...` funded (0.3 ALGO), opted-in (tx `ZNBXUSC...` round 64758284); `provider_credit` of 100k raw units (0.1 OINFRA) inserted into transactions table → Algorand executor picked it up on first startup-tick (`scanned=1 submitted=1 failed=0`) → ASA transfer tx `2RI4UJVR...` confirmed on-chain round 64758405 (0.001 ALGO fee) → provider balance 0.1 OINFRA, treasury 999,999.9 OINFRA. Full flow: DB insert → ListUnsettled(`algorand:%`) → Submit → `MakeAssetTransferTxn` → `SendRawTransaction` → `WaitForConfirmation` → `MarkSettled` — verified. Host→settlement loop verified 2026-06-27: Host `e2e-algorand-provider` registered in DB with `address_kind=algorand`, `payout_address=7B6XH4S...` (row `0f454a33`). Second `provider_credit` (50k raw = 0.05 OINFRA) settled on startup tick → tx `KKKN67EWB...` confirmed round 64772944 → provider balance 0.15 OINFRA (cumulative). gRPC routing confirmed: `server.go:652-658` → `host.AddressKind=="algorand"` → `AlgorandAccount()` → namespace `algorand:<addr>` → executor `ListUnsettled` matches. Both settlements (tx `2RI4UJVR...` + `KKKN67EWB...`) independently confirmed on-chain. Full agent→gRPC→settlement loop verified 2026-06-29: Agent binary built locally, ran with mTLS certs (signed by CP CA) connecting to `cp.seppelabs.com:9090` with `ADDRESS_KIND=algorand`, `PAYOUT_ADDRESS=7B6XH4S...`. Single-use invite token (SHA-256 hashed) consumed atomically on Register. Agent registered: `host_id=cb36cecd...`, `onchain_payout=true`. Host row confirmed `address_kind=algorand`, `status=online`. Third provider_credit (100k raw, idempotency key `e2e-agent-grpc-001`) settled → tx `SU37E6N...` confirmed round 64852976 (axfer, ASA 765165188, 100k aamt, 0.001 ALGO fee) → provider balance 0.25 OINFRA cumulative. All layers verified: agent env vars (`agent.go:333-334`) → RegisterRequest proto (`agent.go:368-369`) → gRPC mTLS → CP host insert → settleCompletedWorkload routing (`server.go:652-658`) → `algorand:` namespace → `ListUnsettled(LIKE 'algorand:%')` → ASA transfer → on-chain balance. 11 test packages green.	2026-06-29
Block L (Phase 5) — home-lab onboarding assets: new `deploy/home-lab/` directory targets the provider-runs-it-themselves topology (compare `deploy/local-pilot/provider-install.sh`, which is operator-over-SSH against a trusted LAN host). `setup-luks2.sh` is an idempotent LUKS2 data-partition creator: argon2id KDF with 1 GiB memory + 4 iterations + parallel=4, ext4 inside the container labelled `openinfra-data`, mounted at `/var/lib/openinfra` with `0700` perms, `/etc/crypttab` + `/etc/fstab` entries written with `nofail` so a missing/failed disk doesn’t lock the host out at boot. Refuses to clobber a non-LUKS device with existing data unless `--force`. Re-running on an already-LUKS2 device reuses the container and only refreshes crypttab/fstab. Boot-time passphrase prompt is the default; TPM-bound unlock (`systemd-cryptenroll --tpm2-device=auto`) is documented as a per-host policy decision. `install-agent.sh` is the provider-runs-locally installer (cf. operator-runs-over-SSH `local-pilot/provider-install.sh`): downloads the agent binary from `https://<cp>/dist/openinfra-agent-linux-<arch>` (overridable via `OPENINFRA_AGENT_URL`), generates a WireGuard keypair under `/etc/wireguard/`, writes `openinfra.conf` with split-tunnel `AllowedIPs=10.77.0.0/24` + `PersistentKeepalive=25` (consumer-NAT-friendly), drops a hardened systemd unit (`ProtectSystem=strict`, `ReadWritePaths=$DATA_DIR`, `NoNewPrivileges=yes`, `PrivateTmp=yes`), and writes `/etc/openinfra/agent.env` (mode 0600) with `CONTROL_PLANE` (10.77.0.1:9092 default, the CP’s WG-internal address), `INVITE_TOKEN`, `SOLANA_PAYOUT_PUBKEY`. Interactive prompts with env-var override path for CI / scripted deploys. Prints the provider’s WG pubkey at the end — the operator must add it as a `[Peer]` on the CP’s wg0 manually for v1 (auto-bootstrap path is a proto-level follow-up: agent sends pubkey via Register, CP returns assigned IP + its own pubkey; out of scope here because it requires `RegisterRequest.wg_pubkey` plumbing on top of the existing `internal/agent/wireguard/wg.go`). `docs/HOME-LAB-ONBOARDING.md` is the operator+provider runbook: topology diagram, invite-minting cookbook, verification commands tied to the Block K log line (`resuming with persisted host_id`), recovery matrix (WG handshake failure / invite-already-used / LUKS prompt loop / payout-pubkey mismatch). Scripts bash-syntax-clean; idempotency verified by inspection (re-runs reuse keys, regenerate config). No platform code touched.	2026-05-16
Block K (Phase 5) — agent host_id persistence: new `internal/agent/identity.go` with pure helpers `loadHostID` / `persistHostID` / `clearHostID` over `<DataDir>/host_id`. Persist path is atomic (`os.CreateTemp` → `Chmod 0600` → `Sync` → `Close` → `Rename`); load path enforces 0600 perms (fail-loud) and canonical lowercase RFC 4122 UUID format (regex-pinned to surface corruption rather than mismatch silently with the control plane row). New `Agent.loadOrRegister` (called from `Run` in place of unconditional `register`) reads the persisted id first and skips Register entirely when present — Heartbeat alone flips the host row back to `online`, which is the whole point of the fix (Block-A invite-gate rejects re-Register as “invite already used”, so every `systemctl restart openinfra-agent` used to burn a fresh invite). Operator escape hatch `OPENINFRA_FORCE_REREGISTER=1` clears the file before deciding so a wiped DB row can re-onboard without manual intervention. Corrupt-file / permission-broken load falls through to Register (logged WARN) rather than failing startup. Persist failure after a successful Register is a WARN (in-memory id still drives this run; next reboot would Register again — operator must fix the data dir). Tests: 9 identity-helper unit tests (round-trip, missing-file empty, 0644 rejection, malformed-on-disk + malformed-on-write rejection, trailing-newline trim, atomic-replace with no leaked tmp, clear-noop on missing, mkdir-on-demand). 6 `loadOrRegister` integration tests via stub `pb.HostServiceClient` (skip-when-persisted, register-and-persist-when-missing, FORCE_REREGISTER clear+re-register, corrupt-file-fallthrough, error-propagation-without-persist, invite/payout env wiring). Full suite + `go vet` clean.	2026-05-16
Block D (Phase 4) — Coledex pilot: end-to-end batch_job lifecycle. Schema (migration 010 from Block B) `services.kind` + `services.config` JSONB now used: `internal/catalog/manifest.go` adds `Kind` + `Config` fields with validation (`batch_job` requires `config.image`); `internal/catalog/loader.go` JSON-marshals config into the `UpsertService` call. Catalog SQL: `ListServicesByKind`, `UpsertService` extended with `kind` + `config`. `internal/scheduler/scheduler.go::PickPinnedHost` adds pinned-host scheduling for batch_jobs (validates active + sufficient free capacity). `internal/api/handlers/workloads.go` branches on `service.Kind`: batch_jobs skip the hourly-billing prepay (settle on completion via Heartbeat), require `host_id` pin, parse `batchJobConfig`, resolve `{{secret:foo}}` placeholders against `req.Secrets` (fail-closed on missing), and build a `WorkloadAssignment` with `runtime=container` + `image` from config. `internal/agent/agent.go` gets a container runtime (`CONTAINER_RUNTIME=noop` for tests, Docker socket otherwise), branches `handleAssignWorkload` on `assign.Runtime == "container"`, spawns `monitorCompletion(workloadID, containerID)` goroutine that calls `runtime.Wait()` (30-min hard cap), and records the result in a TTL’d `completions` map drained into each heartbeat as `WorkloadInfo.final_status` + `exit_code`. New `images/coledex-pg-backup/`: multi-stage Alpine Dockerfile (<30MB target) with `pg_dump → age → rclone rcat` pipeline; `set -euo pipefail` + `${PIPESTATUS[@]}` check guarantees exit 0 only on full pipeline success. `manifests/coledex-postgres-backup.yaml`: catalog seed, vcpus=1 / memory_mb=512 / disk_gb=2, daily cron at 03:00 UTC. `docs/COLEDEX-OPENINFRA.md`: customer-facing integration contract — roles, lifecycle, recovery procedures, endpoints, dashboards. `Makefile` `pilot-image` / `pilot-image-push` targets. Tests: catalog validation (5 batch_job cases), workloads handler (parseBatchJobConfig + resolveEnvTemplate full coverage of substitution + missing-secret + unterminated-placeholder), agent (record/drain/TTL + splitCommand). Full suite green; `go vet` clean.	2026-05-15
Block U (Phase 7) — long-running container service path + Coledex card-scanner dogfood. Background: the Coledex 2 GB Lightsail box couldn’t host PaddleOCR’s ~1 GB working set, so card-scan was never shipped to prod despite full pipeline code existing in `tcg-proj-amplify/card-scanner-service/`. openinfra was the natural fix (deploy scanner on Kuma, Coledex calls it as a regular HTTP sidecar), but the agent had two gaps: `handleContainerAssignment` ignored `assign.Ports` (only the VM handler honored them) and a hard-coded 30-minute `hardCap` SIGKILL deadline. Fix: `internal/agent/agent.go` now copies `assign.Ports` into `container.Config` via a new `convertAssignmentPorts` helper, and checks `assign.Environment["OPENINFRA_LONG_RUNNING"] == "1"` to set `hardCap=0` (skip the deadline) for services that should run until manually stopped. `monitorCompletion` was extended to honor `hardCap=0` by switching from `context.WithTimeout` to `context.WithCancel` in that branch — usage sampling and completion recording remain identical, so per-tenant billing is unaffected. `internal/api/handlers/workloads.go` adds `batchJobPortSpec` to the `batchJobConfig` struct so manifests can declare ports inside `services.config.ports`, and `batchJobPortsToProto` converts them into `pb.PortMapping` for the assignment. The kind=service path still requires a Firecracker rootfs (deferred follow-up) so the card-scanner registers as `kind: batch_job` with `runtime: container` + `OPENINFRA_LONG_RUNNING=1`. New `manifests/coledex-card-scanner.yaml` (vcpus=1, memory_mb=768, port 18000→8000 on laptop to dodge tradinglab-web), companion `deploy/local-pilot/register-coledex-scanner.sh` + `fire-scanner.sh` (env-driven postgres container name so it also drives the Kuma stack). New `deploy/kuma/docker-compose.yml`: minimal single-node control-plane + provider for the 12-vCPU/62-GB ZimaOS box (postgres with migrations 001→015 mounted, api-server, agent at 8 vCPU / 8 GB advertised capacity, private registry — no MinIO/Grafana/customer-pg clutter). Validation: scanner image (`coledex-scanner:local-test`, 600 MB no-OCR) built, pushed to laptop’s `localhost:5000`, registered as service, fired as workload, observed running as `openinfra-<workload_id>` container with the requested port binding. End-to-end laptop chain — Coledex backend (compose) → SCANNER_URL → openinfra-managed scanner → coledex_dev DB → response — returns identical top-match (“Toxtricity V 048 S P”, 23.2) to direct-scanner calls. Failure-mode rehearsal: killing the scanner container surfaces a clean 502 to the backend; no hang. Then deployed to Kuma over LAN (image scp+load+push to Kuma’s registry; admin user registered + promoted via SQL; invite minted via admin API): scanner workload on Kuma reachable from laptop at 192.168.0.224:18000, same scan result. All 7 new unit tests pass (`workloads_ports_test.go` + `internal/agent/ports_test.go`), full suite green.	2026-05-24

Done (formerly pending)

Item	Date	Notes
Agent workload NetworkMode (`3c52fcc`)	2026-06-20	`Config.NetworkMode` → `HostConfig.NetworkMode` from `OPENINFRA_NETWORK_MODE` env key; Coledex `sets-health-check` joins tailnet → reaches prod DB
Per-tenant API keys end-to-end	2026-06-19	Scope enforcement, `POST /admin/tenants/{id}/keys`, `provision-tenant.sh`
Lightsail→Kuma routing (dissolved)	2026-06-19	Agent dials OUT over gRPC/mTLS; no inbound ports, no VPN needed
Agent `host_id` persistence (Block K)	2026-05-16	Atomic file persist, survives restarts, `OPENINFRA_FORCE_REREGISTER=1` escape hatch
Home lab onboarding (Block L)	2026-05-16	LUKS2 installer, agent installer script, WG config
Hetzner CX22 cutover automation (Block N)	2026-05-16	Preflight + idempotent cutover scripts; superseded by Lightsail deployment
Backup observability (Block M)	2026-05-16	Freshness gauges, Prometheus alert rules
ChessLabs analyzer-worker auth + rate limit	2026-06-21/22	Bearer token enforced + 120 req/min/IP via custom Caddy image
ChessLabs analyzer Phase 2 (broker+batch)	2026-06-23	Scale-to-zero batch on Kuma, always-on broker on CP box — LIVE
`-race` + coverage gate + `govulncheck` in CI (Track D)	2026-06-21	All three blocking; coverage floor 35%
`docs/SECURITY-MODEL.md` (Track D)	2026-06-21	Full threat model with 7 trust boundaries
`goreleaser` + reproducible builds	2026-06-21	CI release job on tags

Current Live State (2026-06-26)

Topology

AWS Lightsail micro (1 GB / 2 vCPU, $5/mo) — cp.seppelabs.com (52.20.186.176)
│
├─ Caddy (openinfra-caddy:ratelimit-2.11.4) — HTTPS for all 6 domains
│   ├─ cp.seppelabs.com          → api-server:8080 (REST)
│   ├─ analyzer.seppelabs.com    → chesslabs-broker:8787 (rate-limited 120/min/IP)
│   ├─ chess.seppelabs.com       → /srv/chess (PWA static)
│   ├─ openinfra.seppelabs.com   → /srv/openinfra (docs static)
│   ├─ seppelabs.com / www       → /srv/portal (landing static)
│   └─ queue.seppelabs.com       → processing-queue:8090 (SSE)
│
├─ api-server (openinfra-api:prod, 45 MB) — REST :8080 + gRPC mTLS :9090
├─ postgres (postgres:16-alpine) — all state, 15 migrations
├─ pg-backup (alpine:3.20) — daily pg_dump → B2, no local copy
├─ registry (registry:2) — internal OCI, localhost:5000 only
├─ chesslabs-broker — engine-less analyzer, owns cache.db
├─ chesslabs-sync + chesslabs-sync-db — coach data sync
├─ processing-queue (v4) — Postgres-backed async job queue + SSE
└─ analyzer-ingress-bridge (alpine/socat) — legacy, kept as rollback

Kuma (12 vCPU / 62 GB ZimaOS home box) — provider host
│
├─ kuma-openinfra-agent (kuma-agent:latest) — gRPC/mTLS to cp:9090, runs workloads
├─ ollama + localai — LLM serving for coach-batch
├─ coledex-data-worker — Coledex jobs (auto-offloads sets-health-check to OpenInfra)
├─ openinfra-631ec89e — ChessLabs analyzer (legacy long-running, rollback)
├─ openinfra-0fcf77de — Coledex card scanner (orphan, retire candidate)
├─ analyzer-cp-tunnel — autossh reverse tunnel (legacy)
└─ Coledex stack (tailscale, headscale, scanner, tcgdex, etc.) — independent

What’s Running: 10 containers on CP, 19 on Kuma

Two tenants live:

Coledex (tenant #1) — sets-health-check batch job auto-offloaded weekly; joins tailnet via NetworkMode
ChessLabs Coach (tenant #2) — analyzer batch (scale-to-zero Stockfish) + coach batch (scale-to-zero LLM)

Two proven workload patterns:

Scale-to-zero batch (analyzer, coach LLM): broker on CP owns cache → cache miss → fire batch workload on Kuma (full vCPU/GB) → POST result back → exit. Metered per cpu_ms. ~4s end-to-end for analyzer.
Long-running container (card scanner, legacy analyzer): kind=batch_job + OPENINFRA_LONG_RUNNING=1 bypasses 30-min hardCap. No auto-restart, no health checks — a stopgap until kind=service.

Five reusable primitives:

Settlement ledger (double-entry, idempotency-keyed, invariant-proven)
Pricing (integer micro-credit math, ms precision)
Secretrules (manifest-driven, tenant-agnostic secret validation)
Data layer (content-addressed IPFS push→pin→reward pipeline)
Processing Queue (Postgres-backed async job queue with SSE delivery)

Infrastructure cost: ~$5/mo (single Lightsail micro; Backblaze B2 pennies)

Known Risks (surfaced 2026-06-26)

Risk	Severity	Mitigation
Single CP box is SPOF — 10 containers on 1 GB micro; OOM or disk loss takes everything down	HIGH	Cloudflare proxy for statics (decouple); second CP node (medium-term); backups tested (restore-drill.sh passes)
Caddy is coupling point — all 6 domains terminate on one Caddy; Caddy down = all dark	HIGH	Cloudflare free proxy in front of static sites (chess, openinfra, portal)
Single provider node — only Kuma; zero failover capacity for batch workloads	MEDIUM	Second provider node (even a small one) would prove multi-host scheduling + rescheduling
Internal registry localhost-only — remote agents can’t pull `localhost:5000/*` images	MEDIUM	Workaround: push to Docker Hub; proper fix: TLS-expose registry via Caddy
Two credit stores — batch jobs settle via `transactions` ledger, `/credits/balance` reads `users.credits` only → Coledex sees 0 despite 10M balance	LOW	Reconcile or make balance endpoint settlement-aware for batch-only tenants
Running binary on go1.25.0 — 2 reachable stdlib CVEs (GO-2025-4006, GO-2025-4007) until redeploy on go1.25.11	LOW	Redeploy api-server (already fixed in go.mod + CI)
No workload auto-restart — container crash = agent records completion, doesn’t respawn	LOW	First-class `kind=service` with restart policy
mTLS cert expiry 2027-06 — server + client certs have 1-year validity	LOW	Wire CertRotator or re-mint before anniversary
Test coverage ~39% — CI gate at 35%, aspirational target 80%	LOW	Ratchet up over time; core paths (auth, settlement, pricing) well covered
No monitoring on the micro — Prometheus+Grafana committed but not deployed (would swap-thrash the box)	LOW	Deploy when box is sized up; external uptime ping as stopgap
Caddyfile inode gotcha — `sed -i` swaps inode, reload reads stale; must `docker restart` or edit in-place	LOW	Documented; operator awareness

Consolidated Pending Items (2026-06-26)

Tier 1 — Operational Resilience ✅ DONE (2026-06-30)

#	Item	Status	Notes
O1	Redeploy api-server on go1.25.11	✅	govulncheck clean (0 CVEs), pgx bumped to v5.9.2 (GO-2026-5004), image shipped to CP, Kuma auto-reconnected
O2	Decommission legacy analyzer path	✅	analyzer-ingress-bridge (CP), analyzer-cp-tunnel + openinfra-631ec89e (Kuma) removed; Caddyfile updated
O3	External uptime ping	⚠️	Needs you: sign up at healthchecks.io → create check for `https://cp.seppelabs.com/healthz` (5m period, 2m grace)
O4	Wire Alertmanager + notification channel	✅	Prometheus + Alertmanager deployed on CP (66 MB), 9 alert rules active, targets UP. Notification channels are opt-in (Telegram/email) — uncomment in alertmanager.yml
O5	Age-encrypt backups	✅	Keypair generated, `BACKUP_AGE_RECIPIENT` set on CP — next backup encrypted. Private key stored separately
O6	B2 lifecycle rule	⚠️	Needs you: B2 web console → `openinfra` bucket → Lifecycle Rules → prefix `control-plane/` → expire >31 days
O7	CloudFront CDN for static sites	✅	3 CloudFront distributions (portal, chess, docs) with ACM cert + Route 53 ALIAS. Origin: `origin.seppelabs.com` → Lightsail. Backend domains (cp, analyzer, queue, sync) bypass CF
O8	Update seppelabs-portal docs	✅	Algorand tag + Phase 2 tagline deployed to seppelabs.com

Tier 2 — Platform Maturity (medium-term, ~months)

#	Item	Context
P1	First-class `kind: service` + `runtime: container`	Replace `OPENINFRA_LONG_RUNNING` env-var convention with a proper catalog kind. Includes: health checks, restart policy, port exposure, and a managed public ingress route. Today’s `kind=batch_job` piggyback works but has no safety net.
P2	Auto-restart for long-running workloads	Agent records container exit as “completion” and stops. A `kind=service` should restart crashed containers (with backoff).
P3	Two credit stores reconciliation	`/credits/balance` reads `users.credits` only. Batch-job-only tenants see 0 despite a funded settlement ledger. Either merge the stores or make the endpoint settlement-aware.
P4	Admin attach-user-to-tenant path	`POST /admin/tenants` creates a tenant with no user → API-key auth can’t resolve an acting identity. Seed a user at tenant-creation time or add an explicit attach endpoint.
P5	mTLS cert rotation automation	Wire the existing `CertRotator` (`ServerTLSWithRotation` / `ClientTLSWithRotation`) for hands-off rotation. Certificates expire 2027-06.
P6	Remote-accessible OCI registry	Expose the internal registry over TLS via Caddy (`registry.cp.seppelabs.com`) so remote agents can pull private images. Today: `localhost:5000` only; workaround is Docker Hub.
P7	Workload-aware encryption	Encrypt customer secrets to the assigned host’s age pubkey; mount plaintext only in tmpfs (`/run/openinfra/secrets`). Output re-encryption helper so results are encrypted before any provider syscall sees plaintext.
P8	Second provider node (Track A)	The scheduler supports multi-host, but with only Kuma there’s no rescheduling target. Adding even a small second node proves the multi-host path and removes the single-provider risk.
P9	SSE for mobile (Cognito)	Processing Queue SSE works for web. Mobile (Expo) needs Cognito-authenticated EventSource so async coach results arrive on-phone.
P10	Migrate `platform_fee_bps` to governance config	Currently an env var. Schema migration + tokenomics bounds enforcement.
P11	Provider profile fields	`pricing.credits_per_cpu_hour`, `credits_per_gb_hour`, `geo`, `bandwidth_gbps`, `reputation_score` — needed for multi-provider scheduling quality.
P12	OSS hygiene: scrub LAN IPs	`192.168.0.155`/`192.168.0.224` defaults in `deploy/local-pilot/` and `examples/coledex/deploy/staging/`. Replace with `localhost`/env-var-only. `deploy/kuma` + `provision-tenant.sh` already done.
P13	Deploy Prometheus metrics profile on CP	When the CP box is sized up (or on a separate tiny instance). `--profile metrics` adds ~100 MB — fine on 2 GB. Buys real visibility into the running system.
P14	`gosec` in CI	Static security analysis as a blocking CI job. Already in `docs/SECURITY-MODEL.md` §7 backlog.
P15	Edge rate limiting	Per-IP rate limiting before requests hit the api-server. Caddy `rate_limit` already does this for `analyzer.seppelabs.com`; extend to `cp.seppelabs.com` API routes.

Tier 3 — DePIN Vision (long-term, post-beta)

#	Item	Context
D1	Public beta launch	Open registration, public catalog, real user workloads. Gated on operational resilience (Tier 1) + platform maturity (Tier 2).
D2	Horizontal scaling of control plane	Multiple api-server replicas behind a load balancer. `ListUnsettledOnchain` already designed for `SKIP LOCKED` upgrade.
D3	Firecracker microVMs	VM-level isolation for untrusted multi-tenant workloads. 125ms boot, ~5MB overhead. Gated on having untrusted providers.
D4	Cosmos SDK chain	Decentralized ledger: host registry, staking, rewards, governance on a sovereign chain. CometBFT P2P gossip for host discovery. Currently: centralized PostgreSQL ledger.
D5	Public provider registration + staking/slashing	Open host onboarding with economic security. Gated on Firecracker isolation + on-chain ledger.
D6	GPU workload support	GPU passthrough to containers/VMs, GPU-specific pricing and scheduling.
D7	Geographic region support + latency-based routing	Multi-region host pools, latency-aware scheduling, geo-replicated data.
D8	Storage-only and DBaaS host modes	Hosts that provide only storage (IPFS pinning) or only managed databases, no compute.
D9	Mainnet token economics	OINFRA SPL deployment on Solana mainnet, Squads multisig treasury, inflation curve, vesting, credit↔OINFRA unpeg mechanism. Spec in `docs/TOKENOMICS.md` §8.
D10	Fiat payouts	Stripe Connect or token cashout for providers who want USD, not crypto.
D11	SEV-SNP / TDX attestation	Hardware-attested confidential computing for memory-sensitive workloads.
D12	Agent self-update mechanism	Signed + checksummed releases, agent pulls and applies updates. Gated on `goreleaser` + reproducible builds.
D13	Formal security audit	Third-party audit covering Firecracker isolation, mTLS, secrets envelope, host trust boundary.

Tier 4 — SeppeLabs Platform Integration

#	Item	Context
S1	Wire OIDC `/auth/callback` for each SPA	Cognito pool `us-east-1_ugXwpvt6Q` ready. Portal + ChessLabs need PKCE callback handling.
S2	Coledex: containerize + offload more heavy jobs	`catalog-sync`, `price-poller`, etc. Each needs a catalog service + env_template + `OPENINFRA_NETWORK_MODE`. Pattern proven with `sets-health-check`.
S3	ChessLabs: interactive per-match analyzer/review board	Phase 9 follow-up — review board from user’s perspective on already-stored per-ply eval/best/classification data.
S4	ChessLabs: mobile Cognito → mobile SSE	Expo app receives async coach job notifications via the Processing Queue.
S5	Second batch workload on OpenInfra	poketrace graded sync or similar — proves the pattern generalizes beyond ChessLabs + Coledex.
S6	Retire orphan OpenInfra scanner	`openinfra-0fcf77de` on Kuma (:18000) — self-health-checks only, no real traffic.

Open-source readiness — done (2026-06-18)

Item	Notes
Apache-2.0 `LICENSE` + `NOTICE`	Was “TBD”. Patent grant; standard for infra/DePIN.
`SECURITY.md`, `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`	Private-disclosure policy scoped to the multi-tenant trust model.
`.github/ISSUE_TEMPLATE/*` + `PULL_REQUEST_TEMPLATE.md`	Bug/feature templates, security advisory link, PR checklist (incl. “no tenant code in core”).
README narrative realigned	Front page now distinguishes works today (centralized control plane) from roadmap (Firecracker / WireGuard / IPFS / on-chain). Go pin 1.24→1.25.
Coledex coupling relocated → `examples/coledex/`	Moved `images/coledex-`, `manifests/coledex-`, `docs/COLEDEX-.md`, `deploy/coledex-staging`, `deploy/kuma/coledex-tunnel`. Repointed `Makefile`, `run-pilot.sh`, and the 3 `register-coledex-.sh` manifest paths.

Open-source readiness — done (2026-06-19)

Item	Notes
In-source tenant decoupling: new `internal/secretrules` package	Removed `internal/api/handlers/coledex_validation.go` (hardcoded `coledex-` service-name switch + Coledex-specific secret/SSRF rules). Replaced with a manifest-driven, tenant-agnostic validator: a `Spec` parsed from `services.config.secret_validation` (JSONB), with generic rule kinds (`pattern`, `enum`, `object_key`, `url`, `url_list`, `int_range`) + the SSRF private-IP deny-list as a reusable primitive. `Compile()` precompiles regexes/validates shape; `Validate()` fails closed with user-safe 400 messages; nil-spec = no-op. Rules moved into `examples/coledex/manifests/coledex-data-export.yaml` (8 required + 3 rules) and `coledex-webhook-fanout.yaml` (4 required + 6 rules). The generic `customer_event_id` idempotency-key format check stays in `workloads.go` as a core (non-tenant) pattern. New package 89.9% coverage; behaviors ported 1:1 from the old validator tests; full suite + `go vet` clean. Note:* services already seeded into local-pilot/Kuma DBs before this change carry no `secret_validation` block (the image still validates at container start); re-running the `register-coledex-*.sh` upsert scripts picks up the manifest rules.
Reference single-node deployment model: Kuma = node, Coledex = tenant	Settled the public framing: `deploy/kuma` is the canonical “run a complete billing OpenInfra node on one Linux+Docker host” reference (control plane + co-located agent), and `examples/coledex/` is the reference tenant whose workloads run on it. Genericized `deploy/kuma/docker-compose.yml` in place — replaced the personal-box header (LAN IP / ZimaOS / `/DATA/AppData`) with a generic single-node description, and made provider capacity env-tunable (`OPENINFRA_MAX_VCPUS/MEMORY_MB/DISK_GB`, current values as defaults so the live box is unchanged). Added `deploy/kuma/.env.example` + `deploy/kuma/README.md`. New top-level `docs/QUICKSTART.md` walks clone → node up → register an example service → fire a workload → settle, pointing at the real `run-pilot.sh`/`register-coledex-*.sh` for exact payloads. Fixed a stale `examples/coledex/README.md` “contributor note” that still described the now-deleted `coledex_validation.go` as in-core tech debt. Genericized the `provision-tenant.sh` default API URL (was a personal LAN IP). Build + full test suite green; `docker compose config` validates.
Split-topology deployment: control plane on a public VPS, agent at home	New `deploy/seppelabs/` — control-plane-only stack (postgres + api-server + registry + Caddy) for a small always-on VPS (`cp.seppelabs.com`). Caddy auto-TLS terminates HTTPS for the REST API; the gRPC control channel is exposed on `:9090` with mTLS (`tls.RequireAndVerifyClientCert`). New `deploy/kuma/docker-compose.agent-only.yml` + `.env.agent-only.example` run Kuma as a provider-only node that dials the remote control plane. Zero code changes — mTLS was already fully wired (`certs.ServerTLS`/`ClientTLS`, env-gated via `TLS_ENABLED`/`TLS_SERVER_NAME`); only certs (`cmd/gen-certs`) + env flags are new. The agent verifies the server against the baked-in SAN `openinfra-server`, so the public hostname need not be in the cert. Runbook in `deploy/seppelabs/README.md` (DNS, firewall: only 80/443/9090, cert minting, agent onboarding, Coledex provisioning). Postgres + registry are compose-internal (not host-published). `.env` + `certs/` gitignored on both sides. Verified: both composes `config`-validate; `gen-certs` produces correct SANs; end-to-end cert inspection confirmed. This supersedes the parked Hetzner cutover and the Tailscale/Headscale routing blocker. Added a `.dockerignore` (none existed) so the build context excludes `web/node_modules` (124 MB) + secrets.
Control plane LIVE on AWS Lightsail (`cp.seppelabs.com`)	Provisioned end-to-end via the AWS API: Lightsail `Amazon_Linux_2023-1` (micro, 1 GB/2 vCPU) + free static IP `52.20.186.176` + Route53 A record (zone `seppelabs.com`) + instance firewall opened to 22/80/443/9090 only. Box prep: 2 GB swap (1 GB box), Docker 25 + Compose v2 plugin (AL2023 ships neither). Image built locally (`openinfra-api:prod`, 45 MB) and `docker save
Coledex tenant provisioned on the live CP (2026-06-19)	Bootstrapped an admin (`admin@seppelabs.com`, registered then promoted via `UPDATE users SET role='admin'` — fresh DB had none). Ran `provision-tenant.sh` against `https://cp.seppelabs.com`: created the Coledex tenant+user (`tenant=aebc8c63…`, `user=32c534cf…`), seeded 10,000,000 credits (verified in the `transactions` ledger: `internal:user:32c534cf… ⇐ internal:platform:mint = 10000000`), and issued a scoped key `oi_prd_…` (scopes `workloads:write,workloads:read,credits:read`). Verified the key end-to-end: authed `GET /credits/balance` + `/workloads` → 200, unauth → 401. Secrets saved gitignored: `examples/coledex/deploy/staging/.secrets/{api-key.txt,admin.json,coledex-login.json}`; Kuma client certs at `deploy/kuma/certs/{ca,client,client-key}.pem`. Remaining (not AWS): onboard the Kuma agent (home box — copy the client certs, `docker-compose.agent-only.yml`), then swap admin JWT → `oi_prd_…` key in `tcg-proj-amplify` prod.
Kuma agent onboarded to the live CP (2026-06-20)	Connected to Kuma (ZimaOS v1.5.4, Docker 27.5.1, `ssh kuma` = `seppemarotta@192.168.0.224`, key `~/.ssh/kuma_devnet`). Found the old all-in-one local control plane dead for 6 days (`kuma-openinfra-postgres` Exited 127 → api-server + agent crash-looping) — exactly the home-hosted fragility the VPS move fixes. Removed the 3 dead local-CP containers; left the live Coledex stack (13 containers incl. `coledex-scanner-live` :18002, `kuma-coledex-catalog` :5435, tcgdex, mediamtx, telegram-bot, `coledex-tailscale` + `coledex-headscale` overlays) and the registry untouched. Ran a fresh agent-only container (reused `kuma-agent:latest`, fresh data volume + 24h invite) dialing `cp.seppelabs.com:9090` over mTLS; certs delivered via a docker volume (ZimaOS `$HOME=/DATA` isn’t user-writable). Verified: agent logs “mTLS enabled” + “registered” (`host_id=6dd0cc95…`); CP `/admin/hosts` shows it online, heartbeating. Note: the OpenInfra-managed scanner `openinfra-0fcf77de` (:18000, `coledex-scanner:local-test`) survived as an orphan (long-running detached container) but only self-health-checks — no real traffic; candidate for retirement (pending user OK). Remaining: register Coledex services in the new CP catalog + fire an end-to-end test workload; swap admin JWT → `oi_prd_…` key in `tcg-proj-amplify` prod.
Follow-up: two credit stores	Batch jobs settle via the settlement-engine ledger (`transactions`, `internal:user:` accounts); hourly `kind: service` workloads check `users.credits` (a separate column). `GET /credits/balance` reads only `users.credits`, so a batch-job-only tenant (like Coledex) sees `0` even with a funded settlement ledger — misleading. Reconcile the two (or make `/credits/balance` report the settlement balance for tenants) so the dashboard isn’t misleading.

Control-plane resilience — done (2026-06-21)

Scaling track B: turn the single live VPS from a silent single-point-of-failure into something that survives its own disk loss and degradation. No Go changes — all deploy/seppelabs/ config; full Go suite green, both composes config-validate, shell scripts sh -n/bash -n clean.

Item	Notes
Off-box automatic backups	Added a `pg-backup` sidecar to the core `deploy/seppelabs/docker-compose.yml` (ported from the parked `deploy/hetzner` stack). Streams `pg_dump → gzip -9 → (optional age encrypt) → rclone → b2:$B2_BUCKET/control-plane/<ts>.sql.gz`, once on boot then every 24h. The micro keeps no local copy (streams straight to B2) so a disk loss can’t take the backup with it. alpine idle-sleeps between runs (few-MB steady state). Required env `B2_ACCOUNT_ID/B2_APPLICATION_KEY/B2_BUCKET` (sidecar refuses to boot without them — backups are mandatory on a single-VPS deployment); optional `BACKUP_AGE_RECIPIENT` for at-rest encryption. Same pipeline as the customer-facing `coledex-pg-backup` batch_job, so bugs surface in our own backups first.
Tested restore (`restore-drill.sh`)	The piece everyone skips — an untested backup is not a backup. Pulls the newest (or a named) B2 dump into a throwaway `postgres:16-alpine`, restores it, runs sanity row-counts (users/tenants/services/workloads/transactions), and fails if the restored DB has 0 users. Never touches the live DB; runs entirely in scratch containers (needs only Docker + `.env`). Handles age-decrypt via `BACKUP_AGE_IDENTITY`. The README documents the real disaster-recovery path (rebuild a dead box: fresh `postgres` → `rclone … \| gunzip \| psql`), which the drill exercises 1:1.
Monitoring overlay (opt-in)	New `deploy/seppelabs/docker-compose.monitoring.yml` + `prometheus.yml` + `alerts.yml` + `grafana-provisioning/`. Kept OUT of the core stack so it never burdens the 1 GB micro by default. Profile-gated: `--profile metrics` = Prometheus + alert rules only (~100 MB, fine on the micro); add `--profile dashboard` for Grafana (heavier — bump box to 2 GB first). Both bind `127.0.0.1` only — reached by SSH tunnel (9091/3000), so no public Grafana vhost, no extra cert, no Caddy change, no info-disclosure surface. Prometheus retention capped (15d / 512 MB) so metrics can’t fill the disk. Note: compose interpolates all services’ required vars before profile filtering, so `GRAFANA_ADMIN_PASS` uses an empty default (not `:?`) to keep the lean metrics-only path usable without it.
Alert rules grounded in real metrics	`alerts.yml`: ControlPlaneDown (`up{job="openinfra-api"} == 0`), NoHostsOnline (`openinfra_hosts_total{status="online"} == 0`), DBPoolExhausted (`openinfra_db_pool_idle_conns == 0`), and the Backup{Stale,OneMissed,WatcherErroring,WatcherStuck} family driven by the api-server’s existing `openinfra_backup_age_seconds` freshness gauge (backup family ported 1:1 from `deploy/local-pilot/alerts.yml`). Alertmanager/paging routing is operator-side and still deferred — rules are visible in Prometheus `/alerts` + Grafana Alerting until then. README also recommends an external uptime pinger (on-box monitoring can’t report the box itself dying).

LIVE on the box — done (2026-06-21): deployed the pg-backup sidecar to the running Lightsail stack. The box’s ~/openinfra is not a git checkout (it was scp’d, and its compose has build: swapped to image: openinfra-api:prod), so the sidecar block + pg-backup-entrypoint.sh + restore-drill.sh were shipped via scp and the block awk-inserted into the box compose in place (original saved as docker-compose.yml.bak-preB2). B2 backups reuse the existing Backblaze account via a key scoped to a new private bucket openinfra (the Coledex image bucket is tcgproj, untouched); creds appended to the box .env (read from the Coledex repo .env, piped over SSH, never echoed). First dump uploaded immediately (control-plane/20260621T165705Z.sql.gz, 105 KB). restore-drill.sh PASSED on the box — round-tripped B2 → throwaway postgres, restored 3 users / 4 tenants / 9 workloads / 13 ledger transactions. The other 4 containers were untouched; box still has ~436 MB RAM free. Monitoring overlay deliberately NOT deployed (would swap-thrash the 1 GB box) — it’s committed and ready for when the box is sized up.

Still pending on resilience (not blocking): (1) wire an Alertmanager + a real notification channel (email/Telegram/PagerDuty) so the rules actually page — currently visible-only (and Prometheus itself isn’t running on the micro yet); (2) enable BACKUP_AGE_RECIPIENT once a hardware-stored age key exists (burn-in runs unencrypted for easier verification); (3) an external uptime check (UptimeRobot/healthchecks.io) on /healthz — NOT /api/health, which falls through to the static SPA and returns 200 text/html regardless of api-server health; (4) a B2 lifecycle rule to expire dumps >30d (bucket is “keep all versions”; cost is trivial so low priority). Track A (second provider node) deferred per user — single node for now.

Track D — CI/security hardening — done (2026-06-21)

CI was already mature (lint, test, web-test, build, e2e, release) but had gaps that let regressions slip:

Change	Notes
`-race` in CI	The `test` job ran plain `go test -count=1`; the race detector (in the Makefile) never ran in CI. Now `go test -race -count=1 -covermode=atomic -coverprofile=coverage.out ./...` (cgo + gcc ship on `ubuntu-latest`).
Coverage ratchet gate	New step computes total coverage via `go tool cover -func` and fails below `MIN_COVERAGE` (set to 35%, a regression floor — measured baseline is ~39%). It must only ever go UP; the 80% target (CLAUDE.md / `rules/common/testing.md`) stays aspirational. Profile uploaded as a CI artifact.
Stale migration list fixed	The `test` job applied only `001…005` of 015 migrations (hand-kept list gone stale). Replaced with a glob loop over `internal/db/migrations/*.up.sql` so new migrations are never silently missed.
`govulncheck` job (blocking)	New job scans for CVEs reachable from our code; added to the `release` job’s `needs` so a tagged release can’t ship with a reachable vuln.
`docs/SECURITY-MODEL.md`	Full threat model — trust boundaries B1–B7, assets, per-boundary mitigations + honest residual/accepted risks, a per-endpoint security checklist, and a tracked open-items backlog (rate limiting, backup encryption, workload isolation, cert revocation, gosec).

Security finding from govulncheck (actionable): the scan flagged 28 reachable stdlib vulnerabilities under go1.25.0, two reachable on the live service — GO-2025-4006 (net/mail.ParseAddress CPU-DoS) via the public /register endpoint, and GO-2025-4007 (quadratic crypto/x509 name-constraint check). Fixed by pinning the build toolchain: go.mod now carries toolchain go1.25.11, which govulncheck confirms clears all reachable vulns (0 reachable, exit 0). CI’s setup-go: "1.25" floats to the latest patch, and the Dockerfile floats golang:1.25-alpine, so the live binary clears this on its next rebuild+redeploy (the running binary, built on 1.25.0, is still exposed until then — a redeploy is the remediation). All 29 test packages pass under go1.25.11.

Track D follow-ups (in docs/SECURITY-MODEL.md §7): redeploy the live binary on the patched toolchain (clears the running exposure); edge rate limiting; age-encrypt backups; workload isolation hardening before untrusted third-party images; agent cert revocation + CA custody before third-party providers; gosec in CI.

SeppeLabs platform — portal hub + ChessLabs tenant — foundations (2026-06-21)

Scaling track E: stop treating OpenInfra as a standalone project and make it the backbone for all SeppeLabs apps. The thesis (cost minimization): host SeppeLabs’ own stateful compute on OpenInfra provider nodes the user already runs, instead of paid cloud; keep the static front-ends on the cheapest thing possible. ChessLabs Coach becomes the second tenant after Coledex.

Topology settled this session:

seppelabs.com ─ Portal (static hub)
   ├─ openinfra.seppelabs.com   OpenInfra docs/landing
   ├─ chess.seppelabs.com       ChessLabs Coach PWA (static)
   │     └─ remoteUrl → analyzer.seppelabs.com = analyzer-worker hosted ON OpenInfra
   └─ cp.seppelabs.com          control plane (live)

Static sites (portal + chess PWA) → served by Caddy on the CP box (capacity is a non-issue; the real trade-off is coupling to the CP box — mitigate later with a free Cloudflare proxy in front). The analyzer-worker (Stockfish + shared SQLite cache, the one stateful piece, today pinned to the user’s LAN PC) → an OpenInfra workload.

Item	Notes
Portal scaffolded (`seppelabs-portal/`, new)	Astro static site, repositioned “open-source apps on open infrastructure” (the 2023 ML-store/VR-lab copy was stale). Data-driven project cards (`src/data/projects.ts`) — OpenInfra (live) + ChessLabs Coach (beta) — reusing the existing logo. `npm run build` green (static `dist/`, 1 page, ~1.5s). Not yet a git repo / not pushed.
ChessLabs analyzer-worker manifest (`examples/chesslabscoach/`)	New reference tenant mirroring `examples/coledex/`. Manifest wraps the chesslabscoach `services/analyzer-worker/Dockerfile` (Node 22 + Stockfish 17, port 8787, SQLite cache volume) as `kind=batch_job + runtime=container + OPENINFRA_LONG_RUNNING=1` — the proven long-running-service pattern. Env grounded in the worker’s actual `main.ts` defaults (`THREADS/HASH_MB/ANALYZE_DEPTH/LICHESS_*`). README documents the build→register→deploy→repoint-app flow. No core (`internal/`) changes — manifest is tenant data.
Caddy site blocks staged (`deploy/seppelabs/Caddyfile.sites-example`)	Portal + chess + openinfra + analyzer blocks written but kept OUT of the live `Caddyfile` (Caddy attempts ACME for every site name on load — a block for an un-pointed domain would spam failed challenges on the live CP). Documents the compose volume mounts needed to serve the static bundles. Merge per-domain once DNS points at the box and the bundle is shipped.

ChessLabs analyzer-worker — DEPLOYED LIVE on OpenInfra (2026-06-21)

ChessLabs Coach is now tenant #2 with a real running workload — the first non-coledex tenant to run compute on the platform.

Step	Done
Image	Built `localhost:5000/chesslabs-analyzer:v1` on the Kuma node + pushed to its registry (source synced from `chesslabscoach/services/analyzer-worker`, the `auth.ts` build).
Service	Registered `chesslabs-analyzer-worker` v1 in the live CP (`examples/chesslabscoach/deploy/register-analyzer.sh`; `config` carries `secret_validation` so a malformed token is rejected at deploy time).
Tenant	Provisioned `chesslabscoach` tenant + scoped `oi_prd_` key + 10M seed credits (reused `provision-tenant.sh`).
Workload	Deployed pinned to the Kuma host (`fire-analyzer.sh`); `analyzer_token` minted (`openssl rand`) and passed inline as the `{{secret:analyzer_token}}` value. Container up; Stockfish ready; auth ENABLED. Verified on the node: `/health`→200 (open), `/manifest`→401 without token / 200 with token.
Ingress	Kuma is NAT’d → autossh reverse tunnel binds the analyzer port to the CP-box loopback (`examples/chesslabscoach/deploy/analyzer-cp-tunnel`), a host-networked `socat` (`analyzer-ingress-bridge`) republishes it on the `seppelabs_default` gateway (172.18.0.1), and the CP Caddy reverse-proxies `analyzer.seppelabs.com` there. No sshd / GatewayPorts change needed. ✅ LIVE (2026-06-22): Route53 A-record added (`→ 52.20.186.176`, INSYNC), Caddy reloaded (zero CP-API downtime), Let’s Encrypt cert issued (`CN=analyzer.seppelabs.com`, valid → 2026-09-20).
App	`DEFAULT_WORKER_URL` → `https://analyzer.seppelabs.com`; token injected at build via `EXPO_PUBLIC_ANALYZER_TOKEN` (not committed); persist migrated v2→v3 so existing installs move off the LAN IP.

Endpoint status: ✅ PUBLIC & VERIFIED (2026-06-22) — https://analyzer.seppelabs.com/health → 200 over the full chain (Caddy → socat bridge → autossh tunnel → analyzer container); bearer auth enforced (no token → 401, valid token → routed); per-IP rate limit live (120 req/min, /health exempt — custom Caddy image openinfra-caddy:ratelimit-2.11.4, caddy-ratelimit plugin; verified 121st req → 429). Endpoint hardening complete.

ChessLabs PWA — LIVE at https://chess.seppelabs.com (2026-06-22). Expo web export (apps/mobile, output:"single" SPA, pnpm web:export → dist-web) served statically by the CP Caddy: bundle shipped to deploy/seppelabs/sites/chess/ and mounted ./sites:/srv:ro; Caddy block root * /srv/chess + try_files {path} /index.html (SPA routing) + file_server (serves SQLite + Stockfish WASM with application/wasm). No analyzer token baked in — on-device engine works immediately, remote analyzer is opt-in via the in-app Settings token field. Verified: / 200, WASM MIME correct, SPA fallback 200, Let’s Encrypt cert valid. Deploy gotcha: the box docker-compose.yml is hand-maintained with image: openinfra-api:prod (NOT build:); never scp the repo compose over it — edit the box compose in place (clobbered once during this deploy, recovered from backup).

OpenInfra docs site — LIVE at https://openinfra.seppelabs.com (2026-06-22). Static landing + docs built with Astro Starlight (docs-site/, pnpm build → dist/), served by the CP Caddy from sites/openinfra (reuses the ./sites mount — no compose change, zero-downtime caddy reload). Pages: landing splash + Overview, Quickstart, Home-lab onboarding, Local pilot, Security model, Tokenomics (sourced from docs/*.md; HETZNER-MIGRATION.md deliberately excluded as internal-ops). Pagefind search + sitemap included. No api-server dependency (the old plan to reverse-proxy the auth-gated dashboard SPA is dropped — the dashboard is an operator console, not a public homepage). Verified: all routes 200, search 200, custom 404, LE cert valid.

Portal hub — LIVE at https://seppelabs.com (2026-06-22). Astro static site (seppelabs-portal, branch chore/astro-rebuild; output:"static" + directory format for clean URLs) served by the CP Caddy from sites/portal (same ./sites mount, zero-downtime reload). Apex landing linking out to the live project subdomains (OpenInfra, ChessLabs Coach). www.seppelabs.com → 301 redirect to apex; both have Let’s Encrypt certs.

Platform DNS + ingress COMPLETE (2026-06-22): all five names live on the CP box (52.20.186.176) behind one Caddy — cp (control plane API), analyzer (rate-limited worker), chess (PWA), openinfra (docs), seppelabs.com+www (portal). Each served from deploy/seppelabs/sites/<name>/ except cp/analyzer (reverse-proxied). Future hardening (optional): front the static sites with a free Cloudflare proxy to decouple from the single CP box; age-encrypt backups; the broader Track-D security backlog in docs/SECURITY-MODEL.md §7.

ChessLabs analysis Phase 2 — scale-to-zero batch + always-on broker (2026-06-23)

LIVE end-to-end. Split compute from cache: the always-on engine-less broker on the CP box owns cache.db + serves the unchanged HTTP API; on a cache miss it fires a scale-to-zero analyze-batch workload (8 vCPU / 5 GB on Kuma) that computes the FENs and POSTs evals back. Client unchanged at analyzer.seppelabs.com; adaptive in-browser fallback covers cold-start/failure.

Item	Notes
analyze-batch compute entrypoint	`chesslabscoach@6f7a66c` — `services/analyzer-worker/src/batch/`, reads FEN list from args, runs Stockfish, POSTs results to broker’s `/cache/insert`
BROKER_MODE in analyzer-worker	`chesslabscoach@edb9127` — `openinfraClient.ts` + `brokerDispatch.ts` with tests; broker serves HTTP API + dispatches batch when cache misses
Manifest + deploy	`manifests/chesslabs-analyze-batch.yaml` + `deploy/register-analyze-batch.sh`; `batch_job`, secrets-driven FEN list, no tailnet — POSTs over public HTTPS
Broker compose	`deploy/broker/` docker-compose + README — runs on CP box alongside the core stack; image shares the analyzer-worker Dockerfile
Client `/capabilities` reconcile	`chesslabscoach@e24cd5c` — broker → always-remote, self-hosted worker → adaptive probe
Deploy	Reused existing ChessLabs tenant key (`oi_prd_ucxd5…`); batch service `aefbc431-d41b-4f07-ae29-8f098753d67b`; `chesslabs-analyzer:v1` built + shipped to CP box + Kuma registry
Cut over	Caddy `analyzer.seppelabs.com` → `chesslabs-broker:8787` (was `172.18.0.1:18787`); verified — `/capabilities`→broker, novel position browser→broker→Kuma batch→cache→~4 s with `source:"server" depth:22`
Old analyzer kept	Socat→tunnel→Kuma long-running `openinfra-631ec89e` as instant rollback (revert Caddy line + restart)
Caddy gotcha	Edit Caddyfile inode-preserving or `docker restart openinfra-cp-caddy`; `sed -i` swaps the inode and the reload reads stale
Portal docs updated	`deploy/` docs PR has been merged, `seppelabs.com` docs reflect the Phase 2 architecture

ChessLabs coach batch — scale-to-zero LLM service (2026-06-23)

Anthropic-powered coach explanations as a scale-to-zero OpenInfra batch workload, same pattern as analyze-batch. The broker owns the /coach endpoint and dispatches to a coach-batch workload on Kuma, then stashes the generated note in the broker cache.

Item	Notes
Coach service catalog entry	`manifests/chesslabs-coach-batch.yaml` registered in the live CP (`chesslabscoach@85c1b92`) — `batch_job` + `runtime=container`, carries `ANTHROPIC_API_KEY` as a required secret with `secret_validation.enum`
Broker `/coach` dispatch	Broker compose wired with `OPENINFRA_COACH_SERVICE_ID` env var (`chesslabscoach@fd8b82a`); on cache miss the broker dispatches a coach-batch workload via the same `openinfraClient` pattern
Sync service manifest	`examples/chesslabscoach/deploy/chesslabs-sync.yaml` + broker deploy bundle (`ec97806`) — synchronous data sync from CP DB to the broker’s local cache
Auth	Broker auth uses the same `oi_prd_` tenant key pattern; coach-batch manifest validates `ANTHROPIC_API_KEY` format at deploy time

SeppeLabs Cognito IaC — shared identity pool (2026-06-23)

Shared AWS Cognito user pool for all SeppeLabs apps, deployed via CloudFormation.

Item	Notes
Pool	`us-east-1_ugXwpvt6Q` (us-east-1), shared across seppelabs.com, chess.seppelabs.com, and future portals
OIDC clients	Portal + ChessLabs PKCE clients with proper callback/logout URL configuration
Hosted UI	`seppelabs.auth.us-east-1.amazoncognito.com` with custom domain
IaC	`deploy/seppelabs/cognito/seppelabs-cognito.yaml` (CloudFormation) + `deploy-cognito.sh` CLI wrapper; committed `91e2bfa`
Pool isolation	Coledex’s own pool (`LTN-UserPool-Prod`, 30 users) kept separate — future merge via User Migration Lambda documented in a code comment
Next	Wire each SPA’s OIDC `/auth/callback` flow

OpenInfra Data Layer — Track 2 (2026-06-23/24)

Content-addressed, tenant-scoped user-data primitive built on IPFS pinning. Provides a push→pin→reward pipeline so arbitrary data blobs can be stored, replicated across hosts, and verified.

Item	Date	Notes
Phase 1 — Content-addressed store	8f6e6bc	`internal/data/` — tenant-scoped data CRUD with content-addressed addressing (CID-based), sqlc-backed, compatible with the existing IPFS integration
Phase 2a — Multi-host pinning model	a3a857e	Control-plane model for pinning data across multiple hosts: `pins` table, pin policy, SQL queries
Phase 2b — Proto + agent kubo pinning	238c70e	Protobuf definitions for pinning RPCs; agent-side kubo (IPFS) pinning client with mTLS transport
Phase 2c — Push→multi-host pin coordinator	147486e	Agent coordinator that receives a push from the CP and fans out pin operations to multiple kubo nodes per policy
Phase 2d — Two-node push→pin e2e	19524a1, f49ee51	Proven end-to-end: CP pushes data → two host agents pin on their respective kubo nodes → verified by integration test
Phase 3 — Pinning rewards	0d81336	Hourly reward engine metered by pinned bytes (rate per GB-hour), integrated with the existing settlement ledger
Phase 4A — Per-route body cap	6925445	Configurable per-route request body size limit for the data layer HTTP endpoints, preventing oversized blob attacks
Sync service deploy	a35fbea	`deploy/` wiring for the data-layer sync backend service on the CP box

Processing Queue service — LIVE (2026-06-25)

A standalone Go service (cmd/processing-queue/) providing a unified Postgres-backed async job queue with SSE result delivery. Built for ChessLabs coach async but designed as a general-purpose primitive.

Item	Notes
Phase 1 — Core queue (72b0427)	Go service with Postgres-backed queue (`SELECT … FOR UPDATE SKIP LOCKED`), SSE job-update stream, configurable timeouts. 24 unit + integration tests.
Phase 2 — SERVICE_TOKEN auth + broker SSE (c31fa51)	Pre-shared bearer token authentication for the processing-queue HTTP endpoints; broker SSE client wiring so broker actions are async-safe.
Broker dual-write (4d2ab05)	Web app’s SSE frontend streams from the processing queue for real-time job updates; broker dual-writes job status. SSE `/events` endpoint authenticates via the broker’s own Cognito token for mobile compatibility.
Caddy ingress (f46fe82)	`queue.seppelabs.com` route added to the CP Caddyfile; service runs as a container in the CP docker-compose
Endpoint splits (84b115d)	`UpdateJobStatus` split into `CompleteJob`/`FailJob`/`CancelJobStatus` for clearer semantics; `result` field added to SSE payloads (41a0272)
Coach async integration	`/coach` endpoint now accepts `Prefer: async` header → returns `202` with `jobId` → coachDispatch loop enqueues → SSE delivers result to the web app’s `processingQueueStore`
Images	PQ `v4`, broker `v3` deployed; service running on `queue.seppelabs.com`
Next	Mobile Cognito → mobile SSE (so the Expo app receives async job notifications)