Session 0011: v15 Deploy, Debug & Full Verification

Date: 2026-02-25 Status: Completed Origin: MDF Webseiten session 0054

Work Done

• [x] Deployed ops dashboard v15 to server (rsync + container rebuild)
• [x] Fixed missing OPS_CLI path in run_job() — nsenter couldn't find the ops command
• [x] Fixed same OPS_CLI bug in restore.py _stream_to_job()
• [x] Fixed Python stdout buffering through nsenter pipe — added PYTHONUNBUFFERED=1 to _NSENTER_PREFIX
• [x] Fixed stdin inheritance — added stdin=asyncio.subprocess.DEVNULL to prevent docker run -i blocking
• [x] Fixed schedule.py hard-coded /usr/local/bin/ops path — replaced with OPS_CLI constant
• [x] Added logging to run_job() (command start, subprocess PID, exit code)
• [x] Fixed terminal docker exec missing -it flags — shell was exiting immediately with code 0
• [x] Fixed MDF backup timer — gen-timers wasn't expanding {env} in custom command templates
• [x] Verified backup (dev + int): lines streaming in real-time
• [x] Verified disconnect/reconnect: output replay from offset works
• [x] Verified restart mdf/dev: 2 containers restarted successfully
• [x] Verified terminal: WebSocket handshake + interactive shell working

Key Decisions / Learnings

| Bug | Root Cause | Fix | |-----|-----------|-----| | nsenter: can't execute 'backup' | run_job() missing OPSCLI prefix | Added `[OPSCLI] to full_args` | | Backup produces 0 lines | Python stdout buffered through pipe | Added PYTHONUNBUFFERED=1 to nsenter prefix | | docker run -i hangs | stdin inherited from server process | stdin=asyncio.subprocess.DEVNULL | | Terminal exits immediately (code 0) | docker exec missing -it flags | Added -it to exec command | | MDF backups not running (2 nights) | gen-timers: {env} never expanded in custom command | Loop over envs + .replace("{env}", env) |

• PYTHONUNBUFFERED=1 is essential whenever running Python via nsenter pipe — buffering silently swallows all output
• stdin=asyncio.subprocess.DEVNULL is required for non-interactive subprocess calls from async context

Commits

• 9e13f76 — feat: ops dashboard v15 — persistent jobs + container terminal (Webseiten repo)
• 4e65e9e — fix: gen-timers expand {env} placeholder in custom backup commands (infrastructure repo)

Tags: #Session #OpsDashboard #Debug #Deployment