A cloud session asks for its task in the terminal
description") unless the account has the interactive cloud backend, which this one does not; given one, it creates the session, prints its task for the cloud se...
ASCII in finalize.sh's log lines — jibri's logger printed the em dash as ???
Xrdp was told its session backend was itself, so nothing ever rendered
Opening a computer gave a black screen that never became a desktop. The cause is one unanchored sed in this file: sed -i 's/^port=.*/port=3389/' /etc/xrdp/...
Recording had never worked — jicofo never looked for jibri, and jibri never arrived
Four separate faults, each enough on its own to stop every recording: - jicofo had no ENABLE_RECORDING, so the image wrote no `jibri {}` block: it never watc...
Recreate, and probe the XMPP session — a surge left meetings with no focus for 18h
Every jicofo logs in to prosody as the same full JID, focus@auth.meet.Meet/focus. The maxSurge rollout from 9da0998 runs two of them at once, and prosody answer...
Two threads rebuilt the initrd at once and destroyed the machine
A computer provisioned, came up, and panicked on its next boot: Run /init as init process Failed to execute /init (error -2) Kernel panic - not syn...
The restart could land mid-initramfs and leave a computer unbootable
A brand new computer provisioned, came up, reported AWAKE, and ninety seconds later panicked: Kernel panic - not syncing: No working init found. I caused ...
A resumed computer went straight back to sleep
Thirty seconds after the first real resume, from the machine's own log: kamo-hc-agent: idle 13202s >= 1800s — going to sleep PM: hibernation: hibernati...
Kamouniverse.com redirect was losing to the www.* catch-all
www.kamouniverse.com was served by the kamocrm.com marketing app instead of redirecting: kamo-marketing-route's HostRegexp(\`^www[.].+\$\`) at priority 24 match...
The login greeter was counting as a member using the computer
rdp_sessions() collected every Xorg process with a display argument. The image installs lightdm and lightdm-gtk-greeter, so lightdm runs its own Xorg on :0 on e...
A stale resume_offset would hibernate into the wrong place
resume_configured() checked that `resume=` and `resume_offset=` were present on the running kernel's command line. Present, not correct — and the difference is ...
Root could not read a member's idle time, so nothing ever slept
The first computer reported its idle time as 13.9, then 29.2, then 14.4 seconds over three minutes while nobody was touching it. That is not a person — it is a ...
The swap file was smaller than the check that measures it
With the restart landing, the first computer came up with resumeConfigured true and stopped on the next precondition — which turns out to be unsatisfiable by co...
A computer could never restart into being able to sleep
The first real Hosted Computer reached AWAKE and then stayed awake forever. Hibernation reported resumeConfigured: false, IdleWatcher only arms when readiness i...
Start probing at 15s, not 190 — nothing could open on its first wake
A woken computer is only reachable once its Service has endpoints, and that happens when the readiness probe on 3389 first passes. initialDelaySeconds was 180, ...
Xorg may not start from a remote session, so no computer opened
Guacamole connected, authenticated, and was hung up on: guacd: Security mode: Negotiate (ANY) guacd: Loading keymap "en-us-qwerty" guacd: RDP serve...
Traefik could not see the gateway, and it served at the wrong path
Opening a computer landed on the redirect service's fallback page — "the request reached the fallback page instead of being redirected. Check the Host header." ...
Let auto-cert mirror the gateway certificate into this namespace
Listing computers.kamocrm.com for issuance is only half of it. Traefik requires a TLS secret in the IngressRoute's OWN namespace, so the cert is issued in `kamo...
A computer that has never restarted can never sleep
prepare-sleep writes the swap file and the resume wiring, then reports readiness — and readiness reads `resume=` from /proc/cmdline, which the kernel fixed at b...
Reinstall grub after virt-resize, or the guest boots to a rescue prompt
virt-resize renumbers partitions. Canonical's image is laid out p14 (BIOS boot), p15 (ESP), p16 (/boot) and then p1 (root) physically last, and virt-resize rewr...
The golden disk has to be RAW, or nothing boots and nothing says so
The job wrote the compressed qcow2 the build produces straight into the golden claim. KubeVirt attaches a Filesystem-PVC disk with <driver type='raw'> — always,...
Only the platform reaches the database's and NATS's admin ports
YugabyteDB and NATS run on k1m1's host network, so every pod in every namespace could open every port they listen on. The client ports authenticate (YSQL :5433 ...
NATS refuses anonymous clients — no_auth_user is gone
Second half of 926986d. Every client now presents the kamo_svc login: the Java services through the shared library's NatsConfig (NATS_USERNAME / NATS_PASSWORD f...
Three RBAC grants no provision had ever needed
Nothing had ever been provisioned, so every grant past the namespace was untested. Each of these refused at exactly the point the previous fix unblocked. `patc...
NATS clients authenticate — first half of retiring no_auth_user
NATS runs on the host network of k1m1 and listens on 0.0.0.0, and its config mapped every credential-less connection to the KAMO account (no_auth_user: anon). S...
Only the platform can reach the session Redis
The `kamo` Redis holds every *** session and OTK and has no password. Nothing restricted who could connect to it: from a pod in the `desktop` namespace (where t...
Five reasons the golden image had never actually been built
The build had never been run. Running it found one bug per attempt, and the first three would each have produced a plausible-looking image with something missin...
The three templates that never got committed
service-template.yaml, cloud-init.userdata.yaml and cloud-init.networkdata.yaml. They were written into a second worktree and committed from neither — the earli...
A reply to a sending domain is refused as the recipient, in words that fit NoReply
Campaigns now send Reply-To: NoReply@<their From's domain> by default, and those replies reach postfix-k3m1-inbound. The refusal said "Sender address rejected: ...
Prune unused images on k3m1 too, and stop bulk removals timing out
k3m1 had no prune job, and kubelet image GC waits for 85% disk, so every deploy's SHA-tagged image piled up: 3,368 images (539G), 86 in use. Add a daily k3m1-im...
This relay only sends as kamocrm.com
Anything in mynetworks (every pod, and the LAN) could relay through 47.181.8.84 as any domain, which receivers treat as spoofing and charge to this IP; that is ...
The delivery-events sidecar drops a batch EmailService rejects as invalid
A 400/413/422 means the batch itself will never be accepted; retrying it forever would hold back every result queued behind it. It is logged and dropped. 401/40...
Keep hello.kamocrm.com off this relay, and stop three errors on every start
- A @hello.kamocrm.com sender is refused relay here with a 4xx, so bulk mail can only leave through postfix-bulk; delivery to our own domains is unaffected....
Strip identity headers on myloan.*/api/ before the portal BFF relays them
KamoMLOS's BFF routes under /api/ (the /api/proxy/* passthrough, /api/session/extend, /api/chat/routing, /api/chat/subject-state) copy every client header excep...
Delete identity headers at the edge before header-trusting backends
Add a strip-identity-headers Middleware. A customRequestHeaders value of "" deletes the header, so X-Org-Id, X-Member-Id, X-User-Id, X-God-Mode-Active, X-Member...
Four workers on a node that can hold them, not two on one that cannot
Correcting the previous commit, which was half right and caused a short outage. The memory diagnosis was correct: four workers, each loading its own ~3GB copy ...
Give the translation models room to stay resident, and refuse a scanner
libretranslate ran 4 gunicorn workers against a 12Gi limit, and each worker loads its own ~3GB copy of the argos model set — 11.8GB resident, 98% of the limit, ...
The terminal had no scrollback to scroll, so turn tmux's mouse on
for anything else. I built it against a bare xterm.js writing its own output and never against the real pipeline, which is the whole of the mistake. tmux is a ...
Send To AI must not open a tab in sage's editor
A Send-to-AI hand-off is hosted in a Remote Terminal the operator is already watching in the console. On top of that, the agent also handed the session to VS Co...
Restarting the agent must not kill every member's shells
mid-work today, and would have taken every other member's terminals with it. systemd's default is KillMode=control-group: on stop or restart it SIGKILLs every ...
A budget must not be able to deadlock a node drain
Switches every PodDisruptionBudget from minAvailable: 1 to maxUnavailable: 1. On a two-replica Deployment the two are identical — one pod evictable at a time. ...
Stop running the platform's entire edge on one pod
Traefik ran a single replica, so every restart of it — a rollout, an OOM, an eviction — took the whole platform's TLS down. 192.168.4.22:443 has no local endpoi...
The AI hand-off raced tmux, so nothing was typed and nothing named
Caught by running it on the machine: the terminal opened at the right path, with no command in it and no title. One race, both symptoms. `bootstrap` runs strai...
The terminal list was always empty — tmux escapes control chars
The field separator was a unit separator (0x1f), which is the obvious choice and silently wrong: tmux OCTAL-ESCAPES control characters in `-F` format output, so...
The terminal's IngressRoute was in a directory nothing applies
It was written into securityservice/k8s/ beside the deployment, which reads as the obvious home and is not one: that repo's workflow applies exactly configmap.y...
Drop `su` — it swallowed SIGWINCH, so no terminal ever resized
Caught by running the protocol against the live machine, not by reading it: after a resize control frame, `tput cols` in the shell still reported the size the t...
Add 8777 to the WebRTC endpoint roster
Aiden Perry's extension was created from KamoCRM, which cannot write this file — so it registered over WSS with no DTLS or ICE and its calls would have carried ...
Probe the bridge's real health, not a port that is always open
kamo-meet let one person into a meeting and then ended it the moment a second joined. The bridge had been hard-unhealthy for 41 hours: an in-place container res...
Set the VM's MTU to the pod network's, and let the desktop apply updates
Two separate reasons the "Code" update could not be installed. THE UPDATE COULD NOT BE DOWNLOADED. enp1s0 came up at MTU 1500 while every other workload on thi...
Stop unattended upgrades restarting xrdp under a live session
The black screen was apt. apt-daily-upgrade.service upgraded xrdp at 06:49 UTC on 2026-08-27 and restarted it while a desktop from 2026-08-25 was running. xrdp-...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.