Upgrade k1m1 node image v24.2.4 -> v24.3.33
Rolling major upgrade of the k1m1 CRDB StatefulSet node. Note: cluster will not finalize to 24.3 until the k3m1 node is also upgraded (tracked separately).
Drop the decommissioned k2m1 cluster section
k2m1 (10.8.2.1:6443) is decommissioned and unreachable; its steps lived in the same deploy_services job and failed it (missing kubeconfig / no route) even when ...
Indent heredoc bodies so the startup.sh configmap is valid YAML
The startup.sh block scalar embedded 5 shell heredocs (rtp.conf, http.conf, an SQL block, WebRTC endpoints, freepbx.conf) whose bodies sat at column 0 — less in...
Don't let a missing kubectl cache dir kill the whole job
The kubectl-install step runs 'set -eo pipefail' then find **************** -name kubectl ... | head -1 When that cache dir is absent (cleared between runner ...
Upgrade v3.0 -> v3.3 (fix intermittent WebSocket disconnect)
v3.0.0 intermittently completes the WS 101 upgrade then closes the socket before any frame flows — the recurring 'disconnect on load' in the universe game clien...
Move Dovecot maildirs to a local-path PVC (off rclone/MinIO FUSE)
Maildirs were on hostPath /mnt/shared-files/email/mail — the rclone FUSE mount backed by MinIO object storage, which can't safely host a maildir (no atomic rena...
Put MySQL datadir on a local-path PVC instead of rclone/MinIO FUSE
The mail MySQL datadir (and maildirs) were on hostPath /mnt/shared-files, an rclone FUSE mount backed by MinIO object storage, which cannot durably or consisten...
Backup verified working — DB-scoped backup, MinIO region us-west-ca-s-1, EXTERNALIOIMPLICITACCESS grant in bootstrap
Drop WITH revision_history from backup (enterprise-only); core BACKUP is OSS
node-config: document k1m1 forgejo-runner config (--add-host for submodule DNS)
Job containers on k1m1 fall back to public DNS, so the fleet-swept http://k1m1:3000 .gitmodules URLs fail to clone there. Pin k1m1 into job containers via conta...
node-config: document k1m1 docker daemon insecure-registries
The k1m1 forgejo-runner's builds push to the HTTP Forgejo registry; the daemon.json shipped with only the :5000 entries, so every build job landing on k1m1 fail...
K1m1 image registry replacing retired k0m1:5000 (same credentials)
Loki 6.x SingleBinary needs zeroed scalable targets; crdb exec via localhost (cert SANs)
Kamo-universe namespace+quota+valkey, CockroachDB backups, observability stack
- universe/: namespace + ResourceQuota, valkey-universe (AOF, hostPath k1m1), nightly CockroachDB backup CronJob, operator bootstrap script - monitoring/: pin...
Join shared two-node cluster with k3m1 over LAN; account isolation
Retire WireGuard routes (10.8.x.x) for LAN routes (192.168.4.22 <-> 192.168.4.36 :6222) with creds embedded in the route URLs. Rename cluster to kamo-ratestack-...
Add k3m1 node + migrate cluster peering to LAN IPs
- statefulset: advertise/join over 192.168.4.x (k1m1=.22, k3m1=.36); WireGuard dropped (k3m1 wg0 10.8.0.1 collided with k0m1's advertise IP). - init-job + CI ...
Refresh TLSStore every loop so newly-issued certs get served
update_tls_store() only ran at startup and on shared-storage imports, so a cert freshly issued by cert-manager (HTTP-01) was Ready with a tls-* secret but never...
Give redirect-only route a backend so Traefik v3 enables it
Traefik v3 disables an IngressRoute router with services: [] when allowEmptyServices is off (as it is here), so the domain-agnostic HTTP->HTTPS redirect router ...
Redirect all HTTP hosts to HTTPS + gate auto-cert to real org domains
Custom org domains (e.g. login.sign.pink) returned Traefik's bare "404 page not found" over plain HTTP: the only port-80 redirect router was hardcoded to *.kamo...
Pre-seed sign.pink + www.sign.pink in STATIC_DOMAINS
sign.pink is a new apex domain (consumer e-signature brand / white-label Kamo tenant org). Pre-seed apex + www so a Let's Encrypt cert exists before first HTTPS...
Update subdomain map comment KamoLOS → KamoMLOS
The /Next.js/KamoLOS project was renamed to KamoMLOS (Kamo Mortgage Loan Origination System). The KamoLOS name is now used for the new personal-loan service. Th...
node-config: raise k1m1 kubelet max-pods 110 -> 200
Default RKE2 max-pods=110 was being hit, blocking kamolos rolling-update deploys (maxSurge:1,maxUnavailable:0 needs slot to create new pod before killing old; d...
Document myloan alias for KamoLOS
Adds `myloan` to the SUBDOMAINS recognition list and rewrites the comment to make the list's purpose explicit (it's documentary; the dynamic Traefik log scan is...
Preserve full FQDN for subdomains deeper than two labels
organize_by_root() was extracting only the first label and the last two labels from each Traefik-logged hostname. For the common two-level case (login.b11capita...
Manage mail.kamocrm.com TLS end-to-end
Auto-cert now provisions and maintains the Let's Encrypt cert for mail.kamocrm.com (HTTP-01 via Traefik on port 80) and keeps the postfix consumer in sync acros...
Remove decommissioned Janus WebRTC gateway
Janus is no longer in use. Removes the orphaned KlusterServices/janus/ manifests (Janus was host-installed, not cluster-deployed, and is not running on k1m1). ...
Advertise new public IP 47.181.8.83 for external media
JVB-k1m1 was advertising 75.83.113.11 and coturn was advertising the node's LAN IP via status.hostIP — both broke external WebRTC media now that the public WAN ...
Replace heredoc with grouped echos so YAML literal block parses
The /etc/odbc.ini heredoc body sat at column 1, which dedented out of the surrounding startup.sh: | literal-block scalar and broke `kubectl apply` of the Config...
Follow rclone remounts via HostToContainer mount propagation
Without slave propagation the pod captures the FUSE bind mount once at container start; if rclone-mount.service restarts on the host, the pod keeps pointing at ...
Reverse-proxy priceturbo.com apex + wildcard via k1m1 with TLS
k1m1 Traefik now terminates HTTPS for priceturbo.com and any *.priceturbo.com subdomain, forwarding plain HTTP to the upstream Traefik at 192.168.4.36:80 with t...
Scale to 2 replicas to absorb sync + runtime traffic
A single replica is the bottleneck for the dictionary sync workflow: each gunicorn worker handles one request then auto-restarts (~5-10s window of connection-re...
Reverse-proxy rate.loans/stack.loans hosts via k1m1 with TLS
k1m1 Traefik now terminates HTTPS for api.rate.loans, www.rate.loans, and www.stack.loans, forwarding plain HTTP to the upstream Traefik at 192.168.4.36:80. Aut...
Drop unrecognized --workers arg causing crashloop
The libretranslate CLI does not accept --workers, only the gunicorn process inside it does. Passing it caused the pod to fail argparse on boot and CrashLoopBack...
Add pricing@kamocrm.com mailbox and external IMAP NodePort
- New Service mail/dovecot-imap-external (NodePort 30143 -> 143) so IMAP can be reached from outside the cluster on $NODE:30143. - Seed pricing@kamocrm.com in...
Probe timeout 10s and 2 gunicorn workers
LibreTranslate's /languages probe blocks on the gunicorn worker event loop. With a single worker translating a long chat message, the probe times out at the def...
Scope concurrency to build jobs, not Kubernetes deploys
Workflow-level concurrency serialized the whole pipeline (including kubectl rollouts). Move the group to the build job only so deploy jobs from different runs c...
Rewrite /etc/odbc.ini to point at $MARIADB_HOST on every boot
The escomputers/freepbx:17 image bakes /etc/odbc.ini with Server=db (the docker-compose link target). On this Kubernetes deployment MariaDB lives at \$MARIADB_H...
Configure STUN so browser ICE can complete bidirectionally
Two-party browser-to-browser calls connected at the SIP layer but had no audio either direction. Voicemail (one-way Asterisk → browser) worked because Asterisk'...
Seed pbx.k1.kluster.kamocrm.com as a static domain
The browser softphone needs SIP-over-WSS to land on a hostname with a real TLS cert. Add the FreePBX admin / WS hostname to STATIC_DOMAINS so the auto-cert seed...
Host browser SIP WSS on internal.kamocrm.com (real cert)
Browsers refused the WebSocket connection to **************** with close code 1006 because Traefik was serving its self-signed default cert for that hostname (n...
Make WebRTC config survive pod restart
The previous approach (http_custom.conf with [general] override) didn't actually take effect because Asterisk's INI parser doesn't honor multiple [general] sect...
Enable Asterisk built-in HTTP server on :8089 for WebRTC /ws
A chan_pjsip `protocol=ws` transport rides on top of res_http_websocket, which in turn requires Asterisk's built-in HTTP server to be running. The default FreeP...
Add WSS WebRTC transport for browser softphones
Asterisk's chan_pjsip now binds a plain WebSocket transport on 0.0.0.0:8089 and Traefik terminates TLS at **************** (port 443) so kamo-internal can regis...
Expose port 8088 directly on ClusterIP service
Service only had port 80→8088 mapping. Internal services using **************** (e.g. VOIPService API token requests) were timing out because port 8088 had no l...
Add missing fail2ban-config ConfigMap for k1m1
Deployment references fail2ban-config ConfigMap (jail.local + asterisk-security.conf) but it did not exist, blocking the pod at MountVolume.SetUp and preventing...
Allow /domain/ path prefix on theme IngressRoute for CORS
The theme config at theme.kamocrm.com/domain/... was not matched by the existing route rules (only /public/ and /internal/ were allowed), so Traefik returned a ...
Production fail2ban with PJSIP jails and persistent state
- Add fail2ban-configmap.yaml with [asterisk] and [freepbx-web] jails, incremental ban escalation (1h→24h→30d ceiling), LAN ignoreip, and a custom asterisk-...
Replace Coqui TTS with Piper TTS (en_US-norman-medium)
Coqui VCTK model has no named speakers — replace with Piper TTS which has the exact norman[medium] voice. Piper binary (~3 MB) and model (~60 MB) are downloaded...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.