# Session 0011: v15 Deploy, Debug & Full Verification

**Date:** 2026-02-25
**Status:** Completed
**Origin:** MDF Webseiten session 0054

---

## Work Done

- [x] Deployed ops dashboard v15 to server (rsync + container rebuild)
- [x] Fixed missing `OPS_CLI` path in `run_job()` — nsenter couldn't find the `ops` command
- [x] Fixed same `OPS_CLI` bug in `restore.py` `_stream_to_job()`
- [x] Fixed Python stdout buffering through nsenter pipe — added `PYTHONUNBUFFERED=1` to `_NSENTER_PREFIX`
- [x] Fixed stdin inheritance — added `stdin=asyncio.subprocess.DEVNULL` to prevent `docker run -i` blocking
- [x] Fixed `schedule.py` hard-coded `/usr/local/bin/ops` path — replaced with `OPS_CLI` constant
- [x] Added logging to `run_job()` (command start, subprocess PID, exit code)
- [x] Fixed terminal `docker exec` missing `-it` flags — shell was exiting immediately with code 0
- [x] Fixed MDF backup timer — `gen-timers` wasn't expanding `{env}` in custom command templates
- [x] Verified backup (dev + int): lines streaming in real-time
- [x] Verified disconnect/reconnect: output replay from offset works
- [x] Verified restart mdf/dev: 2 containers restarted successfully
- [x] Verified terminal: WebSocket handshake + interactive shell working

## Key Decisions / Learnings

| Bug | Root Cause | Fix |
|-----|-----------|-----|
| `nsenter: can't execute 'backup'` | `run_job()` missing OPS_CLI prefix | Added `[OPS_CLI]` to `full_args` |
| Backup produces 0 lines | Python stdout buffered through pipe | Added `PYTHONUNBUFFERED=1` to nsenter prefix |
| `docker run -i` hangs | stdin inherited from server process | `stdin=asyncio.subprocess.DEVNULL` |
| Terminal exits immediately (code 0) | `docker exec` missing `-it` flags | Added `-it` to exec command |
| MDF backups not running (2 nights) | `gen-timers`: `{env}` never expanded in custom command | Loop over envs + `.replace("{env}", env)` |

- `PYTHONUNBUFFERED=1` is essential whenever running Python via nsenter pipe — buffering silently swallows all output
- `stdin=asyncio.subprocess.DEVNULL` is required for non-interactive subprocess calls from async context

## Commits

- `9e13f76` — feat: ops dashboard v15 — persistent jobs + container terminal (Webseiten repo)
- `4e65e9e` — fix: gen-timers expand {env} placeholder in custom backup commands (infrastructure repo)

---

**Tags:** #Session #OpsDashboard #Debug #Deployment