Root could not read a member's idle time, so nothing ever slept
The first computer reported its idle time as 13.9, then 29.2, then 14.4 seconds over three minutes while nobody was touching it. That is not a person — it is a ...
The swap file was smaller than the check that measures it
With the restart landing, the first computer came up with resumeConfigured true and stopped on the next precondition — which turns out to be unsatisfiable by co...
A computer could never restart into being able to sleep
The first real Hosted Computer reached AWAKE and then stayed awake forever. Hibernation reported resumeConfigured: false, IdleWatcher only arms when readiness i...
Start probing at 15s, not 190 — nothing could open on its first wake
A woken computer is only reachable once its Service has endpoints, and that happens when the readiness probe on 3389 first passes. initialDelaySeconds was 180, ...
Rename apps DNS alias to apply
auto-cert's SUBDOMAINS list, the availability health-check URL and the middleware comment now match the apps -> apply subdomain rename in kamo-internal and secu...
Xorg may not start from a remote session, so no computer opened
Guacamole connected, authenticated, and was hung up on: guacd: Security mode: Negotiate (ANY) guacd: Loading keymap "en-us-qwerty" guacd: RDP serve...
Traefik could not see the gateway, and it served at the wrong path
Opening a computer landed on the redirect service's fallback page — "the request reached the fallback page instead of being redirected. Check the Host header." ...
An org's own computers host gets a certificate and a route
computers is a standard alias now — every white-label org is told to point one at Kamo on /setup/dns, the way it points meet and sign — so two things had to fol...
Apply the auto-cert mirror Role on deploy
Applied to the cluster by hand when the gateway certificate was set up; this is the same grant in the deploy step, so a rebuilt cluster does not have to redisco...
Let auto-cert mirror the gateway certificate into this namespace
Listing computers.kamocrm.com for issuance is only half of it. Traefik requires a TLS secret in the IngressRoute's OWN namespace, so the cert is issued in `kamo...
Issue and mirror the certificate for computers.kamocrm.com
Opening a Hosted Computer produced a browser tab that appeared and vanished. The launch link is correct and so is the token: SecurityService builds ************...
A computer that has never restarted can never sleep
prepare-sleep writes the swap file and the resume wiring, then reports readiness — and readiness reads `resume=` from /proc/cmdline, which the kernel fixed at b...
Reinstall grub after virt-resize, or the guest boots to a rescue prompt
virt-resize renumbers partitions. Canonical's image is laid out p14 (BIOS boot), p15 (ESP), p16 (/boot) and then p1 (root) physically last, and virt-resize rewr...
The golden disk has to be RAW, or nothing boots and nothing says so
The job wrote the compressed qcow2 the build produces straight into the golden claim. KubeVirt attaches a Filesystem-PVC disk with <driver type='raw'> — always,...
Only the platform reaches the database's and NATS's admin ports
YugabyteDB and NATS run on k1m1's host network, so every pod in every namespace could open every port they listen on. The client ports authenticate (YSQL :5433 ...
NATS refuses anonymous clients — no_auth_user is gone
Second half of 926986d. Every client now presents the kamo_svc login: the Java services through the shared library's NatsConfig (NATS_USERNAME / NATS_PASSWORD f...
Three RBAC grants no provision had ever needed
Nothing had ever been provisioned, so every grant past the namespace was untested. Each of these refused at exactly the point the previous fix unblocked. `patc...
NATS clients authenticate — first half of retiring no_auth_user
NATS runs on the host network of k1m1 and listens on 0.0.0.0, and its config mapped every credential-less connection to the KAMO account (no_auth_user: anon). S...
Only the platform can reach the session Redis
The `kamo` Redis holds every *** session and OTK and has no password. Nothing restricted who could connect to it: from a pod in the `desktop` namespace (where t...
Five reasons the golden image had never actually been built
The build had never been run. Running it found one bug per attempt, and the first three would each have produced a plausible-looking image with something missin...
A way to actually build the golden image
build-golden-image.sh said "Run by CI (see ../../.forgejo/workflows/)". No such workflow was ever written. It needs libguestfs, qemu-img and virt-customize, non...
The three templates that never got committed
service-template.yaml, cloud-init.userdata.yaml and cloud-init.networkdata.yaml. They were written into a second worktree and committed from neither — the earli...
Apply the shared infrastructure on deploy
The namespace, the single-replica StorageClass, the clusterwide seal, computeservice's RBAC and the tenant gateway. Guarded on the hosted-computers/ directory c...
Per-org VM islands, suspend-to-disk, and a white-labeled golden image
A member's own Linux computer, running as a KubeVirt VM in its organization's namespace and opened from the browser. Separate from desktop/ in every way that ma...
Encode the kamo14 reel
The marketing reel was recut (new closing logo shot) and uploaded as public/kamo14.mp4. It gets a new name instead of replacing kamo13 in place because the ladd...
A reply to a sending domain is refused as the recipient, in words that fit NoReply
Campaigns now send Reply-To: NoReply@<their From's domain> by default, and those replies reach postfix-k3m1-inbound. The refusal said "Sender address rejected: ...
universe backup: pull mc from quay.io, docker.io/minio/mc is gone [skip ci]
MinIO withdrew its Docker Hub images, so the backup's upload container sat in ImagePullBackOff from 2026-09-12. With concurrencyPolicy Forbid that one stuck job...
node-config: registry prune must not garbage-collect with --delete-untagged [skip ci]
The nightly prune on k1m1 lived only on the host. Its garbage-collect ran with --delete-untagged, which deletes the per-platform manifests an OCI image index re...
auto-cert: stop provisioning www.stack.loans
The ratestack estate has been scaled to 0 since 2026-08-04, so nothing answers HTTP-01 for stack.loans. Its Certificates expired on 2026-08-02 and their orders ...
Disable node-exporter's xfs collector [skip ci]
It cannot parse this kernel's /proc/fs/xfs/stat ("xpc") and logged an error on every scrape, ~240/h per node. Filesystem usage is unaffected (filesystem collect...
Kamo_app owns the ocr_* tables [skip ci]
OCRService (web and worker) alters its own ocr_* tables at startup and refuses to start without ownership; after the switch to kamo_app the worker crash-looped ...
Non-superuser kamo_app role for all application workloads [skip ci]
Services connected as the superuser kamo. Add an idempotent script that creates kamo_app (no superuser, not a member of kamo so it cannot SET ROLE back) with gr...
Bounces for organizations' sending domains reach EmailService through postfix-k3m1-inbound
A campaign sent through the shared relay has its From at the organization's sending domain as its envelope sender, and that domain is a CNAME to contact.kamocrm...
Scrape YugabyteDB tserver/YSQL and SecurityService pool metrics [skip ci]
Connection sizing had no data behind it: nothing scraped the database or the Hikari pools. Add a curated server-level tserver job (the full endpoint is ~3.25M l...
Postmaster@ and abuse@contact.kamocrm.com reach KamoCRM — the shared relay's name receives its checks
contact.kamocrm.com is the shared org relay's name (47.181.8.86's PTR, MX contact.kamocrm.com), and its MX is 47.181.8.86 → k3m1, where postfix-k3m1-inbound ans...
Postfix-orgs, the shared bulk relay every organization sends through as 47.181.8.86
KamoCRM's own campaign mail keeps leaving through postfix-bulk (47.181.8.87) and regular mail through postfix (47.181.8.84). Every other organization's campaign...
node-config(k3m1-ext): eno1 on DHCP as k3m1-ext, used only by traffic from its own address
k3m1's third cable is the port the shared bulk relay (postfix-orgs, 47.181.8.86) sends through. DHCP gives it an address and nothing else, so the main table is ...
Prune unused images on k3m1 too, and stop bulk removals timing out
k3m1 had no prune job, and kubelet image GC waits for 85% disk, so every deploy's SHA-tagged image piled up: 3,368 images (539G), 86 in use. Add a daily k3m1-im...
Sign Feedback-ID in DKIM signatures
Gmail counts a campaign message toward its Feedback Loop only when the Feedback-ID header is covered by the sending domain's DKIM signature. local.d/dkim.conf k...
Receive postmaster@/abuse@stack.loans on k3m1 for the bulk IP's checks
47.181.8.87's reverse DNS is mail.stack.loans and the router forwards its port 25 to k3m1, but nothing there accepted mail, so a provider verifying who runs the...
Route fbl@/abuse@hello.kamocrm.com complaint reports to EmailService
Spam-complaint feedback-loop reports (ARF, RFC 5965) and abuse reports sent to fbl@ and abuse@hello.kamocrm.com are rewritten to complaints-mail.invalid and han...
Mail to unsubscribe@hello.kamocrm.com reaches EmailService
Every bulk and shared-article message offers an unsubscribe by email (List-Unsubscribe mailto), but mail to that address went to postmaster and was never acted ...
Postfix-bulk accepts any sender from EmailService [skip ci]
The relay is now a platform outbound host that every organization's campaign mail may go through, as whatever From that organization sends as. Only the null sen...
This relay only sends as kamocrm.com
Anything in mynetworks (every pod, and the LAN) could relay through 47.181.8.84 as any domain, which receivers treat as spoofing and charge to this IP; that is ...
The delivery-events sidecar drops a batch EmailService rejects as invalid
A 400/413/422 means the batch itself will never be accepted; retrying it forever would hold back every result queued behind it. It is logged and dropped. 401/40...
Deploy postfix-bulk with the rest of the mail server [skip ci]
The bulk relay's four manifests were applied by hand because the mail step applies its files by name. It now applies them too, refuses to continue if the sideca...
Sign hello.kamocrm.com with its own customer key, by the From domain itself
Campaign mail leaves as hello.kamocrm.com through postfix-bulk. It gets its own DKIM key (selector customer, published as customer._domainkey.hello), and use_es...
Postfix-bulk, a second outbound relay for campaign mail, with delivery results fed back
Campaign mail now leaves through its own relay on k3m1 (egress 47.181.8.87) as hello.kamocrm.com, so complaints about bulk mail never touch the reputation of ma...
Keep hello.kamocrm.com off this relay, and stop three errors on every start
- A @hello.kamocrm.com sender is refused relay here with a 4xx, so bulk mail can only leave through postfix-bulk; delivery to our own domains is unaffected....
Strip identity headers on myloan.*/api/ before the portal BFF relays them
KamoMLOS's BFF routes under /api/ (the /api/proxy/* passthrough, /api/session/extend, /api/chat/routing, /api/chat/subject-state) copy every client header excep...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.