Restore bergamot with working model sources and correct image
- replicas: 1 - Init container (alpine): downloads 4 pairs from data.statmt.org (tarballs for enes/enfr/ende/enpl) and 2 from firefox-translations-models GitH...
Scale bergamot to 0, update image to mozilla/translation-service
The Google Cloud Storage model download bucket (bergamot-models-sandbox) returns 404 for all files and the firefox-translations-models repo was archived Dec 202...
Scale bergamot back to 1 replica — translation failures caused by missing pod
Bergamot was at 0 replicas with its service still registered. The ProviderRouter in TranslateService had no circuit breaker, so every request for es/fr/de/pt/it...
Remove --load-only restriction from LibreTranslate
All needed language packages are pre-installed on the hostPath volume. --load-only was preventing some packages from loading. Without the flag, LibreTranslate l...
Use startupProbe for LibreTranslate to handle long model downloads
startupProbe with 60 attempts x 30s = 30min startup window for initial model downloads. After startup completes, normal liveness and readiness probes take over ...
Restrict LibreTranslate to only load supported language models
Use --load-only with the 22 supported language codes instead of downloading all available models (~hundreds of GB). Removes --update-models flag which would dow...
Remove invalid --load-awesome-ai-models flag from LibreTranslate
Flag does not exist in current libretranslate image. Replace with --update-models to ensure language packs are downloaded on startup.
Add LibreTranslate permission init container, scale Bergamot to 0
LibreTranslate was crashing with PermissionError on packages subdir. Added busybox init container to mkdir + chmod 777 the mounted hostPath before the main cont...
Deploy LibreTranslate and Bergamot in workflow
The manifests were committed but never wired into the deploy steps. Add kubectl apply for both services on K1M1 and K2M1.
Add concurrency group to prevent duplicate workflow runs
When a push triggers a workflow and workflow_dispatch is also fired (or two pushes arrive in quick succession), both runs would execute simultaneously. Adding a...
JVB_ADVERTISE_IPS must be public IP, not Wireguard IP
Was 10.8.1.1 (Wireguard private) — guests' browsers can't reach this. Changed to 75.83.113.11 (public IP). This is why guests kept getting ICE restarts every ~1...
Disable mod_token_verification via custom plugin override to stop guest kicks
The MUC token_verification module re-validates tokens and demotes guests from PARTICIPANT to VISITOR after ~60 seconds, causing Jicofo to kick them. The module ...
Add guest domain to token_verification_allowlist, increase smacks timeout
Guests authenticate via Meet-anonymous on guest.meet.Meet but the MUC token_verification module was re-checking them and demoting their role from PARTICIPANT to...
Disable Jicofo auth module to prevent 1-minute participant kick
Jicofo's AbstractAuthAuthority has a hardcoded 1-minute auth session lifetime that ignores the config file setting. After 60 seconds, Prosody demotes participan...
Use room names (not full JIDs) for all brewery MUC configs
Both Jicofo and JVB Docker images append @XMPP_INTERNAL_MUC_DOMAIN to the brewery room name automatically. Using full JIDs like **************** caused double-d...
Use TCP probe on port 9090 for JVB health checks instead of HTTP on 8080
JVB's REST API port 8080 conflicts with CockroachDB's admin UI on the same hostNetwork. The JVB_OHTTP_SERVER_PORT env var is not recognized by this JVB image ve...
Resolve JVB crash loop — wrong MUC JID + port 8080 conflict
Two issues causing JVB to crash in a loop for 43+ hours: 1. JVB_BREWERY_MUC was **************** (full JID) but JVB appends XMPP_INTERNAL_MUC_DOMAIN automat...
JVB_BREWERY_MUC should be room name only, not full JID
Was **************** which caused JVB to construct a double-domain JID **************** with an @ in the domain part — crashing JVB with XmppStringprepException...
Commit pending infrastructure config changes across auto-cert, cockroachdb, coturn, janus, mail, nats, and traefik
Add dnsPolicy ClusterFirstWithHostNet to JVB StatefulSets so hostNetwork pods can resolve cluster DNS
Configure Prosody, Jicofo, and secrets for Jibri recording support
Enable recording in Prosody, add Jibri XMPP passwords to secrets, and configure Jicofo brewery MUC for Jibri instance discovery.
Add Jibri deployment manifests for Meet meeting recording
Kubernetes deployment, service, and configmap for Jibri (Meet Broadcasting Infrastructure) to enable meeting recording with MinIO storage integration.
Reference *** instead of non-existent turn-secret in Prosody deployment
Add Meet service deployment steps to CI/CD pipeline for k1m1 and k2m1
Add distributed JVB pool with per-node StatefulSets on k1m1 and k2m1
Add Jicofo conference focus with JVB brewery and lobby enforcement
Add Prosody XMPP server deployment with JWT auth and lobby support
Enable auto-truncate on embedding model to prevent 413 errors
Add --auto-truncate flag so inputs exceeding the 256 token limit are silently truncated instead of rejected with a validation error.
Load embedding model from local hostPath to prevent HuggingFace download failures
The embedding-model pod was in CrashLoopBackOff because it couldn't download the model from HuggingFace on startup (network/DNS issue). Mount the model from a h...
Auto-cert service not renewing expired SSL certificates
The auto-cert service was importing expired certs from shared storage and skipping Certificate CR creation, so cert-manager couldn't manage renewals. Also fixed...
Add Qdrant vector DB and MiniLM embedding model K8s manifests
- Qdrant v1.12.1 deployment + ClusterIP service (ports 6333/6334) - HuggingFace TEI with all-MiniLM-L6-v2 deployment + service (port 8080) - Updated deploy-serv...
Fix kubectl not found in pod, add docs alias, proactive SUBDOMAINS
- Mount /usr/local/bin/kubectl from host via hostPath instead of failing apt-get install (kubectl not in default Debian repos) - Add "docs" to SUBDOMAINS list...
Add wopi-*** for DocsService
Adds WOPI JWT signing secret used by DocsService for Docs access tokens. Deployed to both K1M1 and K2M1 clusters.
Remove forced WebSocket headers from media middleware
Traefik v3 handles WebSocket Upgrade/Connection headers natively. Remove forced headers that broke non-WebSocket API requests.
Add system account to all NATS configs
Add $SYS account with admin credentials to k0m1, k1m1, and k2m1 NATS configs. Enables nats CLI cluster management operations (peer removal, step-down, etc.) for...
Resilient probes so cluster stays up when one node is offline
- Liveness: tcpSocket on 4222 (do not depend on JetStream meta leader) - Readiness: initialDelay 90s, period 15s, failureThreshold 10 so k0m1+k1m1 can form qu...
Update Traefik routes: themes.* assets now only load from k1m1 minio instead of balancing across k0m1, k1m1, and k2m1
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.