Pin CockroachDB to v24.2.4 to avoid v24.3 no-license txn throttle
v24.3 (and 24.2.5+) enforce the new CockroachDB Software License; with no license key the cluster throttles to 5 concurrent open transactions cluster-wide. A v2...
Pin RDP color-depth + reap xrdp sessions so god reuses one desktop
xrdp 0.9.24 keys sessions on <user,bpp> (Policy=Default) and never reaps disconnected ones (KillDisconnected=false, DisconnectedTimeLimit=0). The RDP user is al...
Stop expired/orphaned secrets shadowing live per-host certs
update_tls_store() rebuilt the TLSStore from a plain Secret listing, bucketed only on name prefix and type, with no check that a Certificate still owned the sec...
Explicit priority so the OTA updates route wins over the general theme route
Traefik ranks by rule-string length; the general /public/ theme route is longer, so the expo-updates header injection never ran. priority:100000 forces the win.
Raise guacamole memory 1Gi→2Gi to stop OOM mid-RDP-session
The Tomcat webapp idles near ~800Mi, so a 1Gi ceiling left almost no headroom. Opening an RDP session pushed RSS over 1Gi and the container was OOMKilled (exit ...
Move data dir off MinIO FUSE mount to local disk
rspamd's /var/lib/rspamd was hostPath-backed by the rclone FUSE mount of the MinIO shared-files bucket. Every hyperscan recompile / map refresh rewrote files th...
/dev/shm 2Gi (headless Chrome needs >=2GB) + 6Gi mem limit
Jibri's 10-config init aborts with '/dev/shm must be at least 2GB' when shm is 512Mi, crash-looping the recorder. Bump the tmpfs to 2Gi and the memory limit to ...
Correct Guacamole branding manifest version field
Manifest used guacManifestVersion:1.0; the bundled extensions use guacamoleVersion. The wrong field made Guacamole reject the branding extension ('Unable to rea...
Move guacd to pod network (MTU fix) + unicast WoL
guacd on hostNetwork (k1m1 eno49 MTU 1500) vs guacamole pod on cilium overlay (MTU 1420): RDP connected but large frames were dropped across the boundary -> 'Gu...
Watch the desktop namespace for IngressRoutes
desktop.kamocrm.com IngressRoute lives in the desktop ns, but Traefik's CRD provider only watched **************** — so the route was ignored and requests fell ...
Import VM disk as Filesystem, not Block
CDI's block importer crashlooped: 'blockdev: cannot open /dev/cdi-block-volume: Permission denied' — it can't access the raw Longhorn block device. Filesystem v...
Correct KubeVirt VM cloud-init field + modernize spec
- cloudInitNoCloud userdata secret field is 'secretRef' (not userDataSecretRef); KubeVirt v1.8.4 pruned the wrong field -> 'must have at least one userdatasou...
Proactively provision browser-only aliases (media.*)
The log-scan discovery mechanism can never bootstrap the `media` alias: it is only ever contacted by the browser over a WebSocket/XHR subresource (SockJS /ws/in...
Correct FPBXCONF heredoc + repoint universe Scylla
- freepbx: the FPBXCONF here-document closing delimiter was indented (8 spaces); bash requires <<WORD delimiters at column 0, so startup.sh died with 'unexpec...
Pin kamo Traefik to k1m1; make auto-cert single-cluster
SSL errors: kamo Traefik is a LoadBalancer with externalIPs 192.168.4.22 (k1m1) + externalTrafficPolicy=Local, so it MUST run on k1m1 — the pod rebalance moved ...
Indent heredoc bodies so the startup.sh configmap is valid YAML
The startup.sh block scalar embedded 5 shell heredocs (rtp.conf, http.conf, an SQL block, WebRTC endpoints, freepbx.conf) whose bodies sat at column 0 — less in...
Move Dovecot maildirs to a local-path PVC (off rclone/MinIO FUSE)
Maildirs were on hostPath /mnt/shared-files/email/mail — the rclone FUSE mount backed by MinIO object storage, which can't safely host a maildir (no atomic rena...
Put MySQL datadir on a local-path PVC instead of rclone/MinIO FUSE
The mail MySQL datadir (and maildirs) were on hostPath /mnt/shared-files, an rclone FUSE mount backed by MinIO object storage, which cannot durably or consisten...
Backup verified working — DB-scoped backup, MinIO region us-west-ca-s-1, EXTERNALIOIMPLICITACCESS grant in bootstrap
Drop WITH revision_history from backup (enterprise-only); core BACKUP is OSS
Loki 6.x SingleBinary needs zeroed scalable targets; crdb exec via localhost (cert SANs)
Refresh TLSStore every loop so newly-issued certs get served
update_tls_store() only ran at startup and on shared-storage imports, so a cert freshly issued by cert-manager (HTTP-01) was Ready with a tls-* secret but never...
Give redirect-only route a backend so Traefik v3 enables it
Traefik v3 disables an IngressRoute router with services: [] when allowEmptyServices is off (as it is here), so the domain-agnostic HTTP->HTTPS redirect router ...
Redirect all HTTP hosts to HTTPS + gate auto-cert to real org domains
Custom org domains (e.g. login.sign.pink) returned Traefik's bare "404 page not found" over plain HTTP: the only port-80 redirect router was hardcoded to *.kamo...
Preserve full FQDN for subdomains deeper than two labels
organize_by_root() was extracting only the first label and the last two labels from each Traefik-logged hostname. For the common two-level case (login.b11capita...
Advertise new public IP 47.181.8.83 for external media
JVB-k1m1 was advertising 75.83.113.11 and coturn was advertising the node's LAN IP via status.hostIP — both broke external WebRTC media now that the public WAN ...
Replace heredoc with grouped echos so YAML literal block parses
The /etc/odbc.ini heredoc body sat at column 1, which dedented out of the surrounding startup.sh: | literal-block scalar and broke `kubectl apply` of the Config...
Follow rclone remounts via HostToContainer mount propagation
Without slave propagation the pod captures the FUSE bind mount once at container start; if rclone-mount.service restarts on the host, the pod keeps pointing at ...
Scale to 2 replicas to absorb sync + runtime traffic
A single replica is the bottleneck for the dictionary sync workflow: each gunicorn worker handles one request then auto-restarts (~5-10s window of connection-re...
Drop unrecognized --workers arg causing crashloop
The libretranslate CLI does not accept --workers, only the gunicorn process inside it does. Passing it caused the pod to fail argparse on boot and CrashLoopBack...
Probe timeout 10s and 2 gunicorn workers
LibreTranslate's /languages probe blocks on the gunicorn worker event loop. With a single worker translating a long chat message, the probe times out at the def...
Rewrite /etc/odbc.ini to point at $MARIADB_HOST on every boot
The escomputers/freepbx:17 image bakes /etc/odbc.ini with Server=db (the docker-compose link target). On this Kubernetes deployment MariaDB lives at \$MARIADB_H...
Configure STUN so browser ICE can complete bidirectionally
Two-party browser-to-browser calls connected at the SIP layer but had no audio either direction. Voicemail (one-way Asterisk → browser) worked because Asterisk'...
Host browser SIP WSS on internal.kamocrm.com (real cert)
Browsers refused the WebSocket connection to **************** with close code 1006 because Traefik was serving its self-signed default cert for that hostname (n...
Make WebRTC config survive pod restart
The previous approach (http_custom.conf with [general] override) didn't actually take effect because Asterisk's INI parser doesn't honor multiple [general] sect...
Enable Asterisk built-in HTTP server on :8089 for WebRTC /ws
A chan_pjsip `protocol=ws` transport rides on top of res_http_websocket, which in turn requires Asterisk's built-in HTTP server to be running. The default FreeP...
Expose port 8088 directly on ClusterIP service
Service only had port 80→8088 mapping. Internal services using **************** (e.g. VOIPService API token requests) were timing out because port 8088 had no l...
Add missing fail2ban-config ConfigMap for k1m1
Deployment references fail2ban-config ConfigMap (jail.local + asterisk-security.conf) but it did not exist, blocking the pod at MountVolume.SetUp and preventing...
Allow /domain/ path prefix on theme IngressRoute for CORS
The theme config at theme.kamocrm.com/domain/... was not matched by the existing route rules (only /public/ and /internal/ were allowed), so Traefik returned a ...
Replace Coqui TTS with Piper TTS (en_US-norman-medium)
Coqui VCTK model has no named speakers — replace with Piper TTS which has the exact norman[medium] voice. Piper binary (~3 MB) and model (~60 MB) are downloaded...
Add local_net for split-DNS / LAN client RTP fix
Phones on the same LAN as k1m1 connect via split DNS (domain resolves to 192.168.4.22 internally). Without local_net, Asterisk still advertises the external dom...
Persist sorcery contact=memory fix and block SIP scanner in startup.sh
- Inject [res_pjsip] contact=memory into sorcery.conf before Asterisk starts; the default sorcery-astdb SQLite backend silently drops all REGISTER contacts - ...
Move hostPath from MinIO (rclone FUSE) to local /var/lib/pbx
MinIO via rclone FUSE does not support POSIX chmod or symlink() — both required by FreePBX's module system. Changed all FreePBX and MariaDB hostPath volumes fro...
Wrap fwconsole/amportal via php for MinIO FUSE hostPath
/var/lib/asterisk is backed by MinIO via rclone FUSE which doesn't support POSIX chmod, so fwconsole scripts can't be made executable on that volume. startup.sh...
Add preStop hook and terminationGracePeriodSeconds
Containers were getting stuck in Terminating because background Asterisk processes kept running after apache2ctl (PID 1) exited. preStop hook sends SIGTERM to A...
Add bootstrap.php require to regenerated /etc/freepbx.conf
The installer writes freepbx.conf with datasource='' and a require_once for bootstrap.php, which config.php depends on to load the FreePBX class. Our regenerate...
Regenerate /etc/freepbx.conf on restart
/etc is ephemeral (not a hostPath), so /etc/freepbx.conf written by the installer is lost on every pod restart. Added else branch to regenerate it from env vars...
Bypass start_asterisk, wait for control socket instead
start_asterisk uses `pidof asterisk` which returns a false positive on pod restarts, causing the wait-for-Asterisk loop to hang. Now startup.sh starts Asterisk ...
Start Asterisk in background, wait for it, then run installer
The entrypoint.sh does not run the FreePBX installer — it only starts services. startup.sh now replicates the entrypoint setup steps, starts Asterisk in backgro...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.