Keep FreeSWITCH's log readable while it is crash-looping
FreeSWITCH runs with -nc, so mod_logfile is the only record of what it did, and on the container filesystem that record dies with the container — precisely when...
Stop offering 8 unroutable ICE candidates on every call
Callers hear several seconds of one-way audio at the start of a call — the other party says "hi" three or four times before the softphone user hears anything. ...
SIP registration works end to end
An endpoint can now REGISTER from the public internet and Kazoo's API reports it: **************** 200 OK via 47.181.8.87:5060. Four separate faults, each of w...
Zone must be named 'local' to match the node
The config.ini was loading correctly and the AMQP URI was right, but every node logged 'no local zone configured, adding default AMQP' and then retried ********...
Distinct Erlang node names for apps and ecallmgr
One image runs as two nodes. They need different names or they refuse to coexist, and both need the shared cookie that FreeSWITCH's mod_kazoo also uses.
Run the OTP release, not make release
The image now ships a relx release rather than the source tree, so passing 'make release' to the entrypoint failed with 'No rule to make target release'. foreg...
K3m1 is 47.181.8.87 (pbx.k3.kluster.kamocrm.com)
Reverts an incorrect change of mine. I had propagated 42.181.8.87 and edited coturn/deployment.yaml's comment, which was right all along. Verified by DNS: pbx...
Bound the Go heap with GOMEMLIMIT and declare a real memory request
Root cause of the crash-loop was not a leak. The 485Mi steady state that kept tripping the old 512Mi limit was two normal things stacked: ~350Mi Go heap at ...
Raise memory limit to 1Gi and relax liveness probe timeout
Traefik sat at ~485Mi against a 512Mi limit (95%). The cgroup recorded 47836 memory.max hits with oom_kill 0 — never OOM-killed, because enough of the footprint...
Allowlist Telnyx in both fail2ban jails
Adds the Telnyx US signaling addresses (192.76.120.10, 64.16.250.10 — sip.telnyx.com) to ignoreip now that the carrier trunk is provisioned. Also overrides ign...
Make fail2ban actually run — enable Asterisk's security log
fail2ban has never started on this PBX. Its asterisk-security jail watches /var/log/asterisk/security, a file Asterisk never created: FreePBX generates logger_l...
The auto-reconnect reload-looped every page into a blank screen
I shipped this and it took the desktop down: a blank white page and a tab loading forever. Disabled in-cluster immediately; this is the real fix. `ng-switch` s...
The auto-reconnect was watching for a class that never appears
It keyed on **************** That class exists in Guacamole's stylesheet and in none of its templates, so the selector matched nothing: the script loaded, ran, ...
Cap Guacamole's heap, and unbreak the deploy that was blocking it
Two faults, one of them mine. MINE FIRST: the `node --test` gate added with the auto-reconnect script pointed at a DIRECTORY, and the runner's node resolves a ...
Stop the canary crying wolf after every legitimate restart
session_durable treated a REJECTED token as a durability failure. That is not the outage it guards: a gateway restart, or Guacamole's own inactivity timeout, in...
Stop every deploy from throwing everyone off their desktop
"An error has occurred and this action cannot be completed" after leaving a desktop open was not an idle timeout and not a browser problem. It was CI. deploy-s...
Run a dispatched session in the workspace's common parent
sage's VS Code is a multi-root workspace holding all 63 repositories, so taking the first workspace folder rooted a bug about the login site in aiservice — the ...
Say what a dead KamoDesktop connection actually means
Guacamole's stock CLIENT.ERROR_TUNNEL_204 is "The requested connection does not exist. Please check the connection name and try again." There is no connection n...
Submit the bug report without simulating a keystroke
vscode:// URI handler and sent a synthetic Return. Measured on desktop-1, that cannot work: KDE's screen locker holds an exclusive X input grab whenever the scr...
Cache CORS preflights for two hours, not 100 seconds
`accessControlMaxAge: 100` meant browsers re-ran the preflight roughly once per visit, so every page load on the marketing site paid two extra OPTIONS round-tri...
Stop the edge rate limit from 429ing every cold KamoDesktop load
Opening an assigned Linux instance always landed on Guacamole's generic "An error has occurred and this action cannot be completed" modal, and always worked aft...
Make the HLS playlists immutable, and say what that costs
The playlists were the last entry in Lighthouse's cache-lifetime audit — 300s originally, then an hour, still reported. Everything else in the ladder was alread...
Compress the theme host, which was serving every text asset raw
MinIO returns object bytes exactly as stored — it does not negotiate Content-Encoding — and the two theme IngressRoutes carried `strip-hsts` and `cors` but no c...
Pin and verify the kubectl download instead of falling back to a broken wget
The deploy died at "Install kubectl". Three separate faults lined up: 1. curl aborted mid-stream with (56) "decryption failed or bad record mac" — the corru...
Name channels live/<name>, use Recreate, feed the loop a uniform GOP
Three defects found by exercising the deployed service. 1. The /live/ route 404'd through Traefik. MediaMTX answers the first HLS request with a 302 to `?co...
Move the encode job's decision logic out of the mc container
The plan stage died on `sed: command not found`. The minio/mc image is minimal to the point of having no sed, awk, grep, wget or curl — it can move objects and ...
Upload segments with an explicit glob, not --recursive
mc cp --recursive takes a directory; the segments were being passed as a glob, which is the wrong shape for that flag. Segments and the playlist beside them nee...
Preserve the client IP so the KAMOTEAM export ACL matches
Mounts failed with 'access denied by server'. A packet capture inside the guest showed the connection arriving from 10.42.0.151 — a pod-network address — not th...
Make the workstation fstab rewrite safe
Backs up /etc/fstab, normalises a missing trailing newline (a plain append would otherwise splice onto the last entry), and refuses to install a file that lost ...
Stop xrdp restarts orphaning sessions and causing black screens
systemctl restart xrdp makes xrdp-sesman forget its in-memory session registry while the Xorg processes keep running. The next login cannot find the session so ...
Allow SecurityService egress to the dev machine agent, and apply policies from CI
Cilium egress is default-deny for every endpoint it selects, and toEndpoints:[{}] is scoped to the policy's OWN namespace — so SecurityService could not reach d...
Pin the VM MAC and add RDP readiness/liveness probes
KubeVirt generates a random MAC per VMI when macAddress is absent. The guest's netplan matched on the first-boot MAC, so every VMI recreate silently orphaned th...
Repair the kamouniverse backup, broken since the Yugabyte cutover
Renaming the dead CockroachBackupMissing alert in the previous commit made it fire immediately: no universe backup had succeeded since the 2026-08-09 migration....
Raise CPU limit off the 2-core ceiling
Both replicas sat pinned at 1997m/1999m against a 2-core limit for ~47h. That starved /languages enough that the liveness and readiness probes timed out (443 an...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Stop the 60s entrypoint read timeout capping uploads
respondingTimeouts.readTimeout is the deadline for reading a WHOLE request, body included, and Traefik v3 defaults it to 60s where v2 had no limit. That silentl...
Pin the remaining hostPath workloads to the node holding their data
Same fault as embedding-model, found by auditing every hostPath volume in the repo. A hostPath is node-local and nothing replicates it, so an unpinned pod can b...
Pin to the node its model weights are actually on
The model is a hostPath under /var/lib/embedding-model on k1m1, and nothing replicates it. When k3m1 joined, the new ReplicaSet's pod was scheduled there and ne...
Stop the passdb from splitting a mailbox across two maildirs
password_query returned `email as user`, which overwrote the %Lu-lowercased login name with the row's stored casing. user_query then interpolated %n from the re...
STUN answered on the wrong node, so kamocrm.com:3478 was dead
coturn is hostNetwork with no nodeSelector, so it was scheduled on k3m1 (public 47.181.8.87). Its own --external-ip, and every client's `stun:kamocrm.com:3478`,...
Bridge advertised only private ICE candidates — no meet had media for 35 days
The bridge never completed a single ICE connection in the 35 days this pod has been up (0 successes, 18 terminations). Every conference joined over XMPP fine an...
Liveness probe could never pass — cause of 920 restarts in 35 days
Root cause: the liveness probe was `tcpSocket: port 10000`, but Scylla's REST API binds to api_address 127.0.0.1 (its default) while a tcpSocket probe dials the...
Resync drifted manifests with live before enabling full reconciles
Auditing what a full reconcile would apply turned up two manifests that had drifted far enough from the cluster that re-applying them would have caused an outag...
Stop the avatar-resolver policy blocking every login
This policy took the whole platform's login offline for hours. Cilium enforces egress against the RESOLVED BACKEND, after service DNAT, so the port it matches ...
Master Machine uses VNC (x11vnc), not RDP
KRdp/RDP is incompatible with Guacamole: guacd has no RDP Graphics Pipeline (EGFX) and KRdp is H.264-only, so the connection authenticated but died with GRAPHIC...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.