| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| 2026-03-22 | Matthias Nott | ![]() |
| TODO.md | patch | view | blame | history | |
| lib/models/message.dart | patch | view | blame | history | |
| lib/models/server_config.dart | patch | view | blame | history | |
| lib/providers/providers.dart | patch | view | blame | history | |
| lib/screens/chat_screen.dart | patch | view | blame | history | |
| lib/screens/settings_screen.dart | patch | view | blame | history | |
| lib/services/mqtt_service.dart | patch | view | blame | history | |
| lib/widgets/input_bar.dart | patch | view | blame | history | |
| lib/widgets/message_bubble.dart | patch | view | blame | history | |
| pubspec.lock | patch | view | blame | history | |
| pubspec.yaml | patch | view | blame | history |
TODO.md
.. .. @@ -0,0 +1,71 @@ 1 +# PAILot Flutter - TODO2 +3 +## High Priority4 +5 +### MQTT Protocol Migration — NEXT MAJOR TASK6 +Replace ad-hoc WebSocket protocol with MQTT for reliable, ordered messaging.7 +8 +**Why:** Current protocol has no delivery guarantees, no message ordering, no offline queuing.9 +Messages get lost on daemon restart, duplicated on catch_up, and arrive out of order.10 +11 +**Server (AIBroker):**12 +- Embed MQTT broker (aedes) in daemon alongside existing WebSocket13 +- Topics: `pailot/{sessionId}/out` (server→app), `pailot/{sessionId}/in` (app→server)14 +- System topics: `pailot/sessions`, `pailot/status`, `pailot/typing/{sessionId}`15 +- QoS 1 (at-least-once) for messages, QoS 0 for typing indicators16 +- Retained messages for session list and last screenshot17 +- Clean session=false so broker queues messages for offline clients18 +- Bridge MQTT messages to/from existing AIBP routing19 +20 +**Flutter App:**21 +- Replace WebSocket client with mqtt_client package22 +- Subscribe to `pailot/+/out` for all session messages23 +- Publish to `pailot/{sessionId}/in` for user messages24 +- Message ID-based dedup (MQTT can deliver duplicates with QoS 1)25 +- Ordered by broker — no client-side sorting needed26 +- Offline messages delivered automatically on reconnect27 +28 +**Migration:**29 +- Phase 1: Add MQTT alongside WebSocket, dual-publish30 +- Phase 2: Flutter app switches to MQTT31 +- Phase 3: Remove WebSocket from PAILot gateway32 +33 +## Pending Features34 +35 +### File Transfer (send/receive arbitrary files)36 +- File picker in app (PDFs, Word docs, attachments, etc.)37 +- New `file` message type in WebSocket protocol38 +- Gateway handler to save received files and route to session39 +- Session can send files back via `pailot_send_file`40 +- Display file attachments in chat bubbles (icon + filename + tap to open)41 +42 +### Voice+Image Combined Message43 +- When voice caption is recorded with images, hold images on server until voice transcript arrives44 +- Deliver transcript + images together as one message to Claude45 +- Ensures voice prefix sets reply channel correctly46 +47 +### Push Notifications (iOS APNs) — NEXT SESSION with user at computer48 +- **Step 1**: User creates APNs key in Apple Developer Portal (needs login)49 +- **Step 2**: Save `.p8` key to `~/.aibroker/apns-key.p8`, add env vars (key ID, team ID)50 +- **Step 3**: Server-side APNs HTTP/2 sender (`src/daemon/apns.ts`) with JWT auth51 +- **Step 4**: App sends device token on WebSocket connect52 +- **Step 5**: Gateway buffers messages when no WS clients, sends push notification53 +- **Step 6**: App receives push → user taps → opens app → catch_up drains messages54 +- Optional: silent push to wake app briefly for critical messages55 +56 +### App Name Renaming (Runner → PAILot)57 +- Rename Xcode target from Runner to PAILot (like Glidr did)58 +- Update scheme names, bundle paths59 +60 +## Known Issues61 +62 +### Audio63 +- Background audio may not survive full app termination (only screen lock)64 +- Audio session category may conflict with phone calls65 +66 +### UI67 +- Launch image still uses default Flutter placeholder68 +- No app splash screen with PAILot branding69 +70 +### Navigation71 +- vi keys (0, G, dd) are sent as literal text paste — works for Claude Code but may not for other terminalslib/models/message.dart
.. .. @@ -114,18 +114,21 @@ 114 114 }; 115 115 } 116 116 117 - /// Lightweight JSON for persistence (strips temp audio paths, keeps images).117 + /// Lightweight JSON for persistence (strips base64 audio, keeps file paths and images).118 118 Map<String, dynamic> toJsonLight() { 119 + // Keep audioUri if it's a file path (starts with '/') — these are saved audio files.120 + // Strip base64 audio data (large, temp) — those won't survive restart.121 + final keepAudio = audioUri != null && audioUri!.startsWith('/');119 122 return { 120 123 'id': id, 121 124 'role': role.name, 122 125 'type': type.name, 123 126 'content': content, 127 + if (keepAudio) 'audioUri': audioUri,124 128 'timestamp': timestamp, 125 129 if (status != null) 'status': status!.name, 126 130 if (duration != null) 'duration': duration, 127 131 // Keep imageBase64 — images are typically 50-200 KB and must survive restart. 128 - // audioUri is intentionally omitted: it is a temp file path that won't survive restart.129 132 if (imageBase64 != null) 'imageBase64': imageBase64, 130 133 }; 131 134 } lib/models/server_config.dart
.. .. @@ -3,12 +3,14 @@ 3 3 final int port; 4 4 final String? localHost; 5 5 final String? macAddress; 6 + final String? mqttToken;6 7 7 8 const ServerConfig({ 8 9 required this.host, 9 10 this.port = 8765, 10 11 this.localHost, 11 12 this.macAddress, 13 + this.mqttToken,12 14 }); 13 15 14 16 /// Primary WebSocket URL (local network). .. .. @@ -34,6 +36,7 @@ 34 36 'port': port, 35 37 if (localHost != null) 'localHost': localHost, 36 38 if (macAddress != null) 'macAddress': macAddress, 39 + if (mqttToken != null) 'mqttToken': mqttToken,37 40 }; 38 41 } 39 42 .. .. @@ -43,6 +46,7 @@ 43 46 port: json['port'] as int? ?? 8765, 44 47 localHost: json['localHost'] as String?, 45 48 macAddress: json['macAddress'] as String?, 49 + mqttToken: json['mqttToken'] as String?,46 50 ); 47 51 } 48 52 .. .. @@ -51,12 +55,14 @@ 51 55 int? port, 52 56 String? localHost, 53 57 String? macAddress, 58 + String? mqttToken,54 59 }) { 55 60 return ServerConfig( 56 61 host: host ?? this.host, 57 62 port: port ?? this.port, 58 63 localHost: localHost ?? this.localHost, 59 64 macAddress: macAddress ?? this.macAddress, 65 + mqttToken: mqttToken ?? this.mqttToken,60 66 ); 61 67 } 62 68 } lib/providers/providers.dart
.. .. @@ -8,7 +8,7 @@ 8 8 import '../models/server_config.dart'; 9 9 import '../models/session.dart'; 10 10 import '../services/message_store.dart'; 11 -import '../services/websocket_service.dart';11 +import '../services/websocket_service.dart' show ConnectionStatus;12 12 13 13 // --- Enums --- 14 14 .. .. @@ -92,9 +92,10 @@ 92 92 93 93 /// Switch to a new session and load its messages. 94 94 Future<void> switchSession(String sessionId) async { 95 - // Save current session before switching95 + // Force-flush current session to disk before switching96 96 if (_currentSessionId != null && state.isNotEmpty) { 97 97 MessageStore.save(_currentSessionId!, state); 98 + await MessageStore.flush();98 99 } 99 100 100 101 _currentSessionId = sessionId; .. .. @@ -196,9 +197,5 @@ 196 197 197 198 final inputModeProvider = StateProvider<InputMode>((ref) => InputMode.voice); 198 199 199 -// --- WebSocket Service (singleton) ---200 -201 -final webSocketServiceProvider = Provider<WebSocketService?>((ref) {202 - // This is managed manually in the chat screen203 - return null;204 -});200 +// --- MQTT Service (singleton) ---201 +// The MqttService is managed manually in the chat screen.lib/screens/chat_screen.dart
.. .. @@ -1,6 +1,8 @@ 1 1 import 'dart:convert'; 2 2 import 'dart:io'; 3 3 4 +import 'package:path_provider/path_provider.dart';5 +4 6 import 'package:flutter/material.dart'; 5 7 import 'package:flutter_riverpod/flutter_riverpod.dart'; 6 8 import 'package:go_router/go_router.dart'; .. .. @@ -13,7 +15,8 @@ 13 15 import '../models/server_config.dart'; 14 16 import '../providers/providers.dart'; 15 17 import '../services/audio_service.dart'; 16 -import '../services/websocket_service.dart';18 +import '../services/message_store.dart';19 +import '../services/mqtt_service.dart';17 20 import '../theme/app_theme.dart'; 18 21 import '../widgets/command_bar.dart'; 19 22 import '../widgets/input_bar.dart'; .. .. @@ -31,9 +34,18 @@ 31 34 ConsumerState<ChatScreen> createState() => _ChatScreenState(); 32 35 } 33 36 37 +Future<void> _chatLog(String msg) async {38 + try {39 + final dir = await getApplicationDocumentsDirectory();40 + final file = File('${dir.path}/mqtt_debug.log');41 + final ts = DateTime.now().toIso8601String().substring(11, 19);42 + await file.writeAsString('[$ts] $msg\n', mode: FileMode.append);43 + } catch (_) {}44 +}45 +34 46 class _ChatScreenState extends ConsumerState<ChatScreen> 35 47 with WidgetsBindingObserver { 36 - WebSocketService? _ws;48 + MqttService? _ws;37 49 final TextEditingController _textController = TextEditingController(); 38 50 final ScrollController _scrollController = ScrollController(); 39 51 final GlobalKey<ScaffoldState> _scaffoldKey = GlobalKey<ScaffoldState>(); .. .. @@ -42,6 +54,7 @@ 42 54 int _lastSeq = 0; 43 55 bool _isCatchingUp = false; 44 56 bool _screenshotForChat = false; 57 + final Set<int> _seenSeqs = {};45 58 46 59 @override 47 60 void initState() { .. .. @@ -58,11 +71,12 @@ 58 71 if (!mounted) return; 59 72 60 73 // Listen for playback state changes to reset play button UI 74 + // Use a brief delay to avoid race between queue transitions61 75 AudioService.onPlaybackStateChanged = () { 62 - if (mounted) {63 - setState(() {64 - if (!AudioService.isPlaying) {65 - _playingMessageId = null;76 + if (mounted && !AudioService.isPlaying) {77 + Future.delayed(const Duration(milliseconds: 100), () {78 + if (mounted && !AudioService.isPlaying) {79 + setState(() => _playingMessageId = null);66 80 } 67 81 }); 68 82 } .. .. @@ -71,10 +85,11 @@ 71 85 _initConnection(); 72 86 } 73 87 74 - void _saveLastSeq() {75 - SharedPreferences.getInstance().then((prefs) {76 - prefs.setInt('lastSeq', _lastSeq);77 - });88 + SharedPreferences? _prefs;89 +90 + Future<void> _saveLastSeq() async {91 + _prefs ??= await SharedPreferences.getInstance();92 + await _prefs!.setInt('lastSeq', _lastSeq);78 93 } 79 94 80 95 @override .. .. @@ -122,7 +137,7 @@ 122 137 if (config == null) return; 123 138 } 124 139 125 - _ws = WebSocketService(config: config);140 + _ws = MqttService(config: config);126 141 _ws!.onStatusChanged = (status) { 127 142 if (mounted) { 128 143 ref.read(wsStatusProvider.notifier).state = status; .. .. @@ -132,10 +147,11 @@ 132 147 _ws!.onOpen = () { 133 148 final activeId = ref.read(activeSessionIdProvider); 134 149 _sendCommand('sync', activeId != null ? {'activeSessionId': activeId} : null); 150 + // catch_up is still available during the transition period135 151 _sendCommand('catch_up', {'lastSeq': _lastSeq}); 136 152 }; 137 153 _ws!.onError = (error) { 138 - debugPrint('WS error: $error');154 + debugPrint('MQTT error: $error');139 155 }; 140 156 141 157 NavigateNotifier.instance = NavigateNotifier( .. .. @@ -143,7 +159,7 @@ 143 159 _sendCommand('nav', {'key': key}); 144 160 }, 145 161 requestScreenshot: (sessionId) { 146 - _sendCommand('screenshot');162 + _sendCommand('screenshot', {'sessionId': sessionId ?? ref.read(activeSessionIdProvider)});147 163 }, 148 164 ); 149 165 .. .. @@ -153,9 +169,19 @@ 153 169 void _handleMessage(Map<String, dynamic> msg) { 154 170 // Track sequence numbers for catch_up protocol 155 171 final seq = msg['seq'] as int?; 156 - if (seq != null && seq > _lastSeq) {157 - _lastSeq = seq;158 - _saveLastSeq();172 + if (seq != null) {173 + // Dedup: skip messages we've already processed174 + if (_seenSeqs.contains(seq)) return;175 + _seenSeqs.add(seq);176 + // Keep set bounded177 + if (_seenSeqs.length > 500) {178 + final sorted = _seenSeqs.toList()..sort();179 + _seenSeqs.removeAll(sorted.sublist(0, sorted.length - 300));180 + }181 + if (seq > _lastSeq) {182 + _lastSeq = seq;183 + _saveLastSeq();184 + }159 185 } 160 186 161 187 final type = msg['type'] as String?; .. .. @@ -199,11 +225,23 @@ 199 225 _lastSeq = serverSeq; 200 226 _saveLastSeq(); 201 227 } 202 - final messages = msg['messages'] as List<dynamic>?;203 - if (messages != null && messages.isNotEmpty) {228 + // Merge catch_up messages: only add messages not already in local storage.229 + // We check by content match against existing messages to avoid duplicates230 + // while still picking up messages that arrived while the app was backgrounded.231 + final catchUpMsgs = msg['messages'] as List<dynamic>?;232 + if (catchUpMsgs != null && catchUpMsgs.isNotEmpty) {204 233 _isCatchingUp = true; 205 - for (final m in messages) {206 - _handleMessage(m as Map<String, dynamic>);234 + final existing = ref.read(messagesProvider);235 + final existingContents = existing236 + .where((m) => m.role == MessageRole.assistant)237 + .map((m) => m.content)238 + .toSet();239 + for (final m in catchUpMsgs) {240 + final content = (m as Map<String, dynamic>)['content'] as String? ?? '';241 + // Skip if we already have this message locally242 + if (content.isNotEmpty && existingContents.contains(content)) continue;243 + _handleMessage(m);244 + if (content.isNotEmpty) existingContents.add(content);207 245 } 208 246 _isCatchingUp = false; 209 247 } .. .. @@ -239,7 +277,7 @@ 239 277 } 240 278 } 241 279 242 - void _handleIncomingMessage(Map<String, dynamic> msg) {280 + Future<void> _handleIncomingMessage(Map<String, dynamic> msg) async {243 281 final sessionId = msg['sessionId'] as String?; 244 282 final content = msg['content'] as String? ?? 245 283 msg['text'] as String? ?? .. .. @@ -253,6 +291,8 @@ 253 291 254 292 final activeId = ref.read(activeSessionIdProvider); 255 293 if (sessionId != null && sessionId != activeId) { 294 + // Store message for the other session so it's there when user switches295 + await _storeForSession(sessionId, message);256 296 _incrementUnread(sessionId); 257 297 final sessions = ref.read(sessionsProvider); 258 298 final session = sessions.firstWhere( .. .. @@ -274,10 +314,10 @@ 274 314 } 275 315 } 276 316 277 - void _handleIncomingVoice(Map<String, dynamic> msg) {317 + Future<void> _handleIncomingVoice(Map<String, dynamic> msg) async {278 318 final sessionId = msg['sessionId'] as String?; 279 319 final audioData = msg['audioBase64'] as String? ?? msg['audio'] as String? ?? msg['data'] as String?; 280 - final content = msg['content'] as String? ?? msg['text'] as String? ?? '';320 + final content = msg['content'] as String? ?? msg['transcript'] as String? ?? msg['text'] as String? ?? '';281 321 final duration = msg['duration'] as int?; 282 322 283 323 final message = Message( .. .. @@ -291,8 +331,36 @@ 291 331 duration: duration, 292 332 ); 293 333 334 + // Save audio to file so it survives persistence (base64 gets stripped)335 + String? savedAudioPath;336 + if (audioData != null) {337 + try {338 + final dir = await getTemporaryDirectory();339 + savedAudioPath = '${dir.path}/voice_${message.id}.m4a';340 + final bytes = base64Decode(audioData.contains(',') ? audioData.split(',').last : audioData);341 + await File(savedAudioPath).writeAsBytes(bytes);342 + } catch (_) {343 + savedAudioPath = null;344 + }345 + }346 +347 + final storedMessage = Message(348 + id: message.id,349 + role: message.role,350 + type: message.type,351 + content: content,352 + audioUri: savedAudioPath ?? audioData,353 + timestamp: message.timestamp,354 + status: message.status,355 + duration: duration,356 + );357 +294 358 final activeId = ref.read(activeSessionIdProvider); 359 + _chatLog('voice: sessionId=$sessionId activeId=$activeId audioPath=$savedAudioPath content="${content.substring(0, content.length.clamp(0, 30))}"');295 360 if (sessionId != null && sessionId != activeId) { 361 + _chatLog('voice: cross-session, storing for $sessionId');362 + await _storeForSession(sessionId, storedMessage);363 + _chatLog('voice: stored, incrementing unread');296 364 _incrementUnread(sessionId); 297 365 final sessions = ref.read(sessionsProvider); 298 366 final session = sessions.firstWhere( .. .. @@ -310,12 +378,12 @@ 310 378 return; 311 379 } 312 380 313 - ref.read(messagesProvider.notifier).addMessage(message);381 + ref.read(messagesProvider.notifier).addMessage(storedMessage);314 382 ref.read(isTypingProvider.notifier).state = false; 315 383 _scrollToBottom(); 316 384 317 - if (audioData != null && !AudioService.isBackgrounded && !_isCatchingUp) {318 - // Queue incoming voice chunks — don't cancel what's already playing385 + if (audioData != null && !AudioService.isBackgrounded && !_isCatchingUp && !_isRecording) {386 + setState(() => _playingMessageId = storedMessage.id);319 387 AudioService.queueBase64(audioData); 320 388 } 321 389 } .. .. @@ -357,6 +425,17 @@ 357 425 _scrollToBottom(); 358 426 } 359 427 428 + /// Store a message for a non-active session so it persists when the user switches to it.429 + Future<void> _storeForSession(String sessionId, Message message) async {430 + final existing = await MessageStore.loadAll(sessionId);431 + _chatLog('storeForSession: $sessionId existing=${existing.length} adding type=${message.type.name} content="${message.content.substring(0, message.content.length.clamp(0, 30))}" audioUri=${message.audioUri != null ? "set(${message.audioUri!.length})" : "null"}');432 + MessageStore.save(sessionId, [...existing, message]);433 + await MessageStore.flush();434 + // Verify435 + final verify = await MessageStore.loadAll(sessionId);436 + _chatLog('storeForSession: verified ${verify.length} messages after save');437 + }438 +360 439 void _incrementUnread(String sessionId) { 361 440 final counts = Map<String, int>.from(ref.read(unreadCountsProvider)); 362 441 counts[sessionId] = (counts[sessionId] ?? 0) + 1; .. .. @@ -364,9 +443,10 @@ 364 443 } 365 444 366 445 Future<void> _switchSession(String sessionId) async { 367 - // Stop any playing audio when switching sessions446 + // Stop any playing audio and dismiss keyboard when switching sessions368 447 await AudioService.stopPlayback(); 369 448 setState(() => _playingMessageId = null); 449 + if (mounted) FocusScope.of(context).unfocus();370 450 371 451 ref.read(activeSessionIdProvider.notifier).state = sessionId; 372 452 await ref.read(messagesProvider.notifier).switchSession(sessionId); .. .. @@ -391,6 +471,7 @@ 391 471 392 472 ref.read(messagesProvider.notifier).addMessage(message); 393 473 _textController.clear(); 474 + FocusScope.of(context).unfocus(); // dismiss keyboard394 475 395 476 // Send as plain text (not command) — gateway handles plain messages 396 477 _ws?.send({ .. .. @@ -401,14 +482,26 @@ 401 482 _scrollToBottom(); 402 483 } 403 484 485 + String? _recordingSessionId; // Capture session at recording start486 +404 487 Future<void> _startRecording() async { 488 + // Stop any playing audio before recording489 + if (AudioService.isPlaying) {490 + await AudioService.stopPlayback();491 + setState(() => _playingMessageId = null);492 + }493 + _recordingSessionId = ref.read(activeSessionIdProvider);405 494 final path = await AudioService.startRecording(); 406 495 if (path != null) { 407 496 setState(() => _isRecording = true); 497 + } else {498 + _recordingSessionId = null;408 499 } 409 500 } 410 501 411 502 Future<void> _stopRecording() async { 503 + final targetSession = _recordingSessionId;504 + _recordingSessionId = null;412 505 final path = await AudioService.stopRecording(); 413 506 setState(() => _isRecording = false); 414 507 .. .. @@ -433,7 +526,7 @@ 433 526 'audioBase64': b64, 434 527 'content': '', 435 528 'messageId': message.id, 436 - 'sessionId': ref.read(activeSessionIdProvider),529 + 'sessionId': targetSession,437 530 }); 438 531 439 532 _scrollToBottom(); .. .. @@ -468,16 +561,21 @@ 468 561 } 469 562 } 470 563 471 - void _playMessage(Message message) {564 + void _playMessage(Message message) async {472 565 if (message.audioUri == null) return; 473 566 474 567 // Toggle: if this message is already playing, stop it 475 - if (_playingMessageId == message.id && AudioService.isPlaying) {476 - AudioService.stopPlayback();568 + if (_playingMessageId == message.id) {569 + await AudioService.stopPlayback();477 570 setState(() => _playingMessageId = null); 478 571 return; 479 572 } 480 573 574 + // Stop any current playback first, then set playing ID AFTER stop completes575 + // (stopPlayback triggers onPlaybackStateChanged which clears _playingMessageId)576 + await AudioService.stopPlayback();577 +578 + if (!mounted) return;481 579 setState(() => _playingMessageId = message.id); 482 580 483 581 if (message.audioUri!.startsWith('/')) { .. .. @@ -511,7 +609,16 @@ 511 609 512 610 void _requestScreenshot() { 513 611 _screenshotForChat = true; 514 - _sendCommand('screenshot');612 + _sendCommand('screenshot', {'sessionId': ref.read(activeSessionIdProvider)});613 + if (mounted) {614 + ScaffoldMessenger.of(context).showSnackBar(615 + const SnackBar(616 + content: Text('Capturing screenshot...'),617 + duration: Duration(seconds: 2),618 + behavior: SnackBarBehavior.floating,619 + ),620 + );621 + }515 622 } 516 623 517 624 void _navigateToTerminal() { .. .. @@ -520,6 +627,9 @@ 520 627 } 521 628 522 629 Future<void> _pickPhoto() async { 630 + // Capture session ID now — before any async gaps (dialog, encoding)631 + final targetSessionId = ref.read(activeSessionIdProvider);632 +523 633 final picker = ImagePicker(); 524 634 final images = await picker.pickMultiImage( 525 635 maxWidth: 1920, .. .. @@ -566,7 +676,7 @@ 566 676 'audioBase64': voiceB64, 567 677 'content': '', 568 678 'messageId': voiceMsg.id, 569 - 'sessionId': ref.read(activeSessionIdProvider),679 + 'sessionId': targetSessionId,570 680 }); 571 681 } 572 682 .. .. @@ -580,7 +690,7 @@ 580 690 'imageBase64': encodedImages[i], 581 691 'mimeType': 'image/jpeg', 582 692 'caption': msgCaption, 583 - 'sessionId': ref.read(activeSessionIdProvider),693 + 'sessionId': targetSessionId,584 694 }); 585 695 } 586 696 .. .. @@ -822,12 +932,18 @@ 822 932 final unreadCounts = ref.watch(unreadCountsProvider); 823 933 final inputMode = ref.watch(inputModeProvider); 824 934 825 - return Scaffold(935 + return GestureDetector(936 + behavior: HitTestBehavior.translucent,937 + onTap: () => FocusScope.of(context).unfocus(),938 + child: Scaffold(826 939 key: _scaffoldKey, 827 940 appBar: AppBar( 828 941 leading: IconButton( 829 942 icon: const Icon(Icons.menu), 830 - onPressed: () => _scaffoldKey.currentState?.openDrawer(),943 + onPressed: () {944 + FocusScope.of(context).unfocus();945 + _scaffoldKey.currentState?.openDrawer();946 + },831 947 ), 832 948 title: Text( 833 949 activeSession?.name ?? 'PAILot', .. .. @@ -927,6 +1043,7 @@ 927 1043 ), 928 1044 ], 929 1045 ), 1046 + ),930 1047 ); 931 1048 } 932 1049 } lib/screens/settings_screen.dart
.. .. @@ -3,7 +3,7 @@ 3 3 4 4 import '../models/server_config.dart'; 5 5 import '../providers/providers.dart'; 6 -import '../services/websocket_service.dart';6 +import '../services/websocket_service.dart' show ConnectionStatus;7 7 import '../services/wol_service.dart'; 8 8 import '../theme/app_theme.dart'; 9 9 import '../widgets/status_dot.dart'; .. .. @@ -21,6 +21,7 @@ 21 21 late final TextEditingController _remoteHostController; 22 22 late final TextEditingController _portController; 23 23 late final TextEditingController _macController; 24 + late final TextEditingController _mqttTokenController;24 25 bool _isWaking = false; 25 26 26 27 @override .. .. @@ -35,6 +36,8 @@ 35 36 TextEditingController(text: '${config?.port ?? 8765}'); 36 37 _macController = 37 38 TextEditingController(text: config?.macAddress ?? ''); 39 + _mqttTokenController =40 + TextEditingController(text: config?.mqttToken ?? '');38 41 } 39 42 40 43 @override .. .. @@ -43,6 +46,7 @@ 43 46 _remoteHostController.dispose(); 44 47 _portController.dispose(); 45 48 _macController.dispose(); 49 + _mqttTokenController.dispose();46 50 super.dispose(); 47 51 } 48 52 .. .. @@ -58,6 +62,9 @@ 58 62 macAddress: _macController.text.trim().isEmpty 59 63 ? null 60 64 : _macController.text.trim(), 65 + mqttToken: _mqttTokenController.text.trim().isEmpty66 + ? null67 + : _mqttTokenController.text.trim(),61 68 ); 62 69 63 70 await ref.read(serverConfigProvider.notifier).save(config); .. .. @@ -183,6 +190,19 @@ 183 190 hintText: 'AA:BB:CC:DD:EE:FF', 184 191 ), 185 192 ), 193 + const SizedBox(height: 16),194 +195 + // MQTT Token196 + Text('MQTT Token',197 + style: Theme.of(context).textTheme.bodyMedium),198 + const SizedBox(height: 4),199 + TextFormField(200 + controller: _mqttTokenController,201 + decoration: const InputDecoration(202 + hintText: 'Shared secret for MQTT auth',203 + ),204 + obscureText: true,205 + ),186 206 const SizedBox(height: 24), 187 207 188 208 // Save button lib/services/mqtt_service.dart
.. .. @@ -0,0 +1,495 @@ 1 +import 'dart:async';2 +import 'dart:convert';3 +import 'dart:io';4 +5 +import 'package:flutter/widgets.dart';6 +import 'package:path_provider/path_provider.dart' as pp;7 +import 'package:mqtt_client/mqtt_client.dart';8 +import 'package:mqtt_client/mqtt_server_client.dart';9 +import 'package:shared_preferences/shared_preferences.dart';10 +import 'package:uuid/uuid.dart';11 +12 +import '../models/server_config.dart';13 +import 'websocket_service.dart' show ConnectionStatus;14 +import 'wol_service.dart';15 +16 +// Debug log to file (survives release builds)17 +Future<void> _mqttLog(String msg) async {18 + try {19 + final dir = await pp.getApplicationDocumentsDirectory();20 + final file = File('${dir.path}/mqtt_debug.log');21 + final ts = DateTime.now().toIso8601String().substring(11, 19);22 + await file.writeAsString('[$ts] $msg\n', mode: FileMode.append);23 + } catch (_) {}24 +}25 +26 +/// MQTT client for PAILot, replacing WebSocketService.27 +///28 +/// Connects to the AIBroker daemon's embedded aedes broker.29 +/// Subscribes to all pailot/ topics and dispatches messages30 +/// through the same callback interface as WebSocketService.31 +class MqttService with WidgetsBindingObserver {32 + MqttService({required this.config});33 +34 + ServerConfig config;35 + MqttServerClient? _client;36 + ConnectionStatus _status = ConnectionStatus.disconnected;37 + bool _intentionalClose = false;38 + String? _clientId;39 + StreamSubscription? _updatesSub;40 +41 + // Message deduplication42 + final Set<String> _seenMsgIds = {};43 + final List<String> _seenMsgIdOrder = [];44 + static const int _maxSeenIds = 500;45 +46 + // Callbacks — same interface as WebSocketService47 + void Function(ConnectionStatus status)? onStatusChanged;48 + void Function(Map<String, dynamic> message)? onMessage;49 + void Function()? onOpen;50 + void Function()? onClose;51 + void Function()? onReconnecting;52 + void Function(String error)? onError;53 +54 + ConnectionStatus get status => _status;55 + bool get isConnected => _status == ConnectionStatus.connected;56 +57 + void _setStatus(ConnectionStatus newStatus) {58 + if (_status == newStatus) return;59 + _status = newStatus;60 + onStatusChanged?.call(newStatus);61 + }62 +63 + /// Get or create a persistent client ID for this device.64 + Future<String> _getClientId() async {65 + if (_clientId != null) return _clientId!;66 + final prefs = await SharedPreferences.getInstance();67 + var id = prefs.getString('mqtt_client_id');68 + // Regenerate if old format (too long for MQTT 3.1.1)69 + if (id == null || id.length > 23) {70 + // MQTT 3.1.1 client IDs: max 23 chars, alphanumeric71 + id = 'pailot${const Uuid().v4().replaceAll('-', '').substring(0, 16)}';72 + await prefs.setString('mqtt_client_id', id);73 + }74 + _clientId = id;75 + return id;76 + }77 +78 + /// Connect to the MQTT broker.79 + /// Tries local host first (2.5s timeout), then remote host.80 + Future<void> connect() async {81 + if (_status == ConnectionStatus.connected ||82 + _status == ConnectionStatus.connecting) {83 + return;84 + }85 +86 + _intentionalClose = false;87 + _setStatus(ConnectionStatus.connecting);88 +89 + // Send Wake-on-LAN if MAC configured90 + if (config.macAddress != null && config.macAddress!.isNotEmpty) {91 + try {92 + await WolService.wake(config.macAddress!, localHost: config.localHost);93 + } catch (_) {}94 + }95 +96 + final clientId = await _getClientId();97 + final hosts = _getHosts();98 + _mqttLog('MQTT: hosts=${hosts.join(", ")} port=${config.port}');99 +100 + for (final host in hosts) {101 + if (_intentionalClose) return;102 +103 + _mqttLog('MQTT: trying $host:${config.port}');104 + try {105 + final connected = await _tryConnect(106 + host,107 + clientId,108 + timeout: host == hosts.first && hosts.length > 1 ? 2500 : 5000,109 + );110 + _mqttLog('MQTT: $host result=$connected');111 + if (connected) return;112 + } catch (e) {113 + _mqttLog('MQTT: $host error=$e');114 + continue;115 + }116 + }117 +118 + // All hosts failed — retry after delay119 + _mqttLog('MQTT: all hosts failed, retrying in 5s');120 + _setStatus(ConnectionStatus.reconnecting);121 + Future.delayed(const Duration(seconds: 5), () {122 + if (!_intentionalClose && _status != ConnectionStatus.connected) {123 + connect();124 + }125 + });126 + }127 +128 + /// Returns [localHost, remoteHost] for dual-connect attempts.129 + List<String> _getHosts() {130 + if (config.localHost != null &&131 + config.localHost!.isNotEmpty &&132 + config.localHost != config.host) {133 + return [config.localHost!, config.host];134 + }135 + return [config.host];136 + }137 +138 + Future<bool> _tryConnect(String host, String clientId, {int timeout = 5000}) async {139 + try {140 + final client = MqttServerClient.withPort(host, clientId, config.port);141 + client.keepAlivePeriod = 30;142 + client.autoReconnect = false; // Don't auto-reconnect during trial — enable after success143 + client.connectTimeoutPeriod = timeout;144 + client.logging(on: true);145 +146 + client.onConnected = _onConnected;147 + client.onDisconnected = _onDisconnected;148 + client.onAutoReconnect = _onAutoReconnect;149 + client.onAutoReconnected = _onAutoReconnected;150 +151 + // Persistent session (cleanSession = false) for offline message queuing152 + final connMessage = MqttConnectMessage()153 + .withClientIdentifier(clientId)154 + .authenticateAs('pailot', config.mqttToken ?? '')155 + .startClean(); // Use clean session for now; persistent sessions require broker support156 +157 + // For persistent sessions, replace startClean() with:158 + // .withWillQos(MqttQos.atLeastOnce);159 + // and remove startClean()160 +161 + client.connectionMessage = connMessage;162 +163 + // Set _client BEFORE connect() so _onConnected can subscribe164 + _client = client;165 +166 + _mqttLog('MQTT: connecting to $host:${config.port} as $clientId (timeout=${timeout}ms)');167 + final result = await client.connect().timeout(168 + Duration(milliseconds: timeout + 1000),169 + onTimeout: () {170 + _mqttLog('MQTT: connect timed out for $host');171 + return null;172 + },173 + );174 + _mqttLog('MQTT: connect result=${result?.state}');175 + if (result?.state == MqttConnectionState.connected) {176 + client.autoReconnect = true; // Now enable auto-reconnect for the live connection177 + return true;178 + }179 + _client = null;180 + client.disconnect();181 + return false;182 + } catch (e) {183 + _mqttLog('MQTT: connect exception=$e');184 + return false;185 + }186 + }187 +188 + void _onConnected() {189 + _mqttLog('MQTT: _onConnected fired');190 + _setStatus(ConnectionStatus.connected);191 + _subscribe();192 + _listenMessages();193 + onOpen?.call();194 + }195 +196 + void _onDisconnected() {197 + _updatesSub?.cancel();198 + _updatesSub = null;199 +200 + if (_intentionalClose) {201 + _setStatus(ConnectionStatus.disconnected);202 + onClose?.call();203 + } else {204 + _setStatus(ConnectionStatus.reconnecting);205 + onReconnecting?.call();206 + }207 + }208 +209 + void _onAutoReconnect() {210 + _setStatus(ConnectionStatus.reconnecting);211 + onReconnecting?.call();212 + }213 +214 + void _onAutoReconnected() {215 + _setStatus(ConnectionStatus.connected);216 + _subscribe();217 + _listenMessages();218 + onOpen?.call();219 + }220 +221 + void _subscribe() {222 + final client = _client;223 + if (client == null) {224 + _mqttLog('MQTT: _subscribe called but client is null');225 + return;226 + }227 + _mqttLog('MQTT: subscribing to topics...');228 + client.subscribe('pailot/sessions', MqttQos.atLeastOnce);229 + client.subscribe('pailot/status', MqttQos.atLeastOnce);230 + client.subscribe('pailot/projects', MqttQos.atLeastOnce);231 + client.subscribe('pailot/+/out', MqttQos.atLeastOnce);232 + client.subscribe('pailot/+/typing', MqttQos.atMostOnce);233 + client.subscribe('pailot/+/screenshot', MqttQos.atLeastOnce);234 + client.subscribe('pailot/control/out', MqttQos.atLeastOnce);235 + client.subscribe('pailot/voice/transcript', MqttQos.atLeastOnce);236 + }237 +238 + void _listenMessages() {239 + _updatesSub?.cancel();240 + _updatesSub = _client?.updates?.listen(_onMqttMessage);241 + }242 +243 + void _onMqttMessage(List<MqttReceivedMessage<MqttMessage>> messages) {244 + _mqttLog('MQTT: received ${messages.length} message(s)');245 + for (final msg in messages) {246 + _mqttLog('MQTT: topic=${msg.topic}');247 + final pubMsg = msg.payload as MqttPublishMessage;248 + final payload = MqttPublishPayload.bytesToStringAsString(249 + pubMsg.payload.message,250 + );251 +252 + Map<String, dynamic> json;253 + try {254 + json = jsonDecode(payload) as Map<String, dynamic>;255 + } catch (_) {256 + continue; // Skip non-JSON257 + }258 +259 + // Dedup by msgId260 + final msgId = json['msgId'] as String?;261 + if (msgId != null) {262 + if (_seenMsgIds.contains(msgId)) continue;263 + _seenMsgIds.add(msgId);264 + _seenMsgIdOrder.add(msgId);265 + _evictOldIds();266 + }267 +268 + // Dispatch: parse topic to enrich the message with routing info269 + _dispatchMessage(msg.topic, json);270 + }271 + }272 +273 + /// Route incoming MQTT messages to the onMessage callback.274 + /// Translates MQTT topic structure into the flat message format275 + /// that chat_screen expects (same as WebSocket messages).276 + void _dispatchMessage(String topic, Map<String, dynamic> json) {277 + final parts = topic.split('/');278 +279 + // pailot/sessions280 + if (topic == 'pailot/sessions') {281 + json['type'] = 'sessions';282 + onMessage?.call(json);283 + return;284 + }285 +286 + // pailot/status287 + if (topic == 'pailot/status') {288 + json['type'] = 'status';289 + onMessage?.call(json);290 + return;291 + }292 +293 + // pailot/projects294 + if (topic == 'pailot/projects') {295 + json['type'] = 'projects';296 + onMessage?.call(json);297 + return;298 + }299 +300 + // pailot/control/out — command responses (session_switched, session_renamed, error, unread)301 + if (topic == 'pailot/control/out') {302 + onMessage?.call(json);303 + return;304 + }305 +306 + // pailot/voice/transcript307 + if (topic == 'pailot/voice/transcript') {308 + json['type'] = 'transcript';309 + onMessage?.call(json);310 + return;311 + }312 +313 + // pailot/<sessionId>/out — text, voice, image messages314 + if (parts.length == 3 && parts[2] == 'out') {315 + final sessionId = parts[1];316 + json['sessionId'] ??= sessionId;317 + onMessage?.call(json);318 + return;319 + }320 +321 + // pailot/<sessionId>/typing322 + if (parts.length == 3 && parts[2] == 'typing') {323 + final sessionId = parts[1];324 + json['type'] = 'typing';325 + json['sessionId'] ??= sessionId;326 + // Map 'active' field to the 'typing'/'isTyping' fields chat_screen expects327 + final active = json['active'] as bool? ?? true;328 + json['typing'] = active;329 + onMessage?.call(json);330 + return;331 + }332 +333 + // pailot/<sessionId>/screenshot334 + if (parts.length == 3 && parts[2] == 'screenshot') {335 + final sessionId = parts[1];336 + json['type'] = 'screenshot';337 + json['sessionId'] ??= sessionId;338 + // Map imageBase64 to 'data' for compatibility with chat_screen handler339 + json['data'] ??= json['imageBase64'];340 + onMessage?.call(json);341 + return;342 + }343 + }344 +345 + void _evictOldIds() {346 + while (_seenMsgIdOrder.length > _maxSeenIds) {347 + final oldest = _seenMsgIdOrder.removeAt(0);348 + _seenMsgIds.remove(oldest);349 + }350 + }351 +352 + /// Generate a UUID v4 for message IDs.353 + String _uuid() => const Uuid().v4();354 +355 + /// Current timestamp in milliseconds.356 + int _now() => DateTime.now().millisecondsSinceEpoch;357 +358 + /// Publish a JSON payload to an MQTT topic.359 + void _publish(String topic, Map<String, dynamic> payload, MqttQos qos) {360 + final client = _client;361 + if (client == null || client.connectionStatus?.state != MqttConnectionState.connected) {362 + onError?.call('Not connected');363 + return;364 + }365 +366 + try {367 + final builder = MqttClientPayloadBuilder();368 + builder.addString(jsonEncode(payload));369 + client.publishMessage(topic, qos, builder.payload!);370 + } catch (e) {371 + onError?.call('Send failed: $e');372 + }373 + }374 +375 + /// Send a message — routes to the appropriate MQTT topic based on content.376 + /// Accepts the same message format as WebSocketService.send().377 + void send(Map<String, dynamic> message) {378 + final type = message['type'] as String?;379 + final sessionId = message['sessionId'] as String?;380 +381 + if (type == 'command' || (message.containsKey('command') && type == null)) {382 + // Command messages go to pailot/control/in383 + final command = message['command'] as String? ?? '';384 + final args = message['args'] as Map<String, dynamic>? ?? {};385 + final payload = <String, dynamic>{386 + 'msgId': _uuid(),387 + 'type': 'command',388 + 'command': command,389 + 'ts': _now(),390 + ...args,391 + };392 + _publish('pailot/control/in', payload, MqttQos.atLeastOnce);393 + return;394 + }395 +396 + if (type == 'voice' && sessionId != null) {397 + // Voice message398 + _publish('pailot/$sessionId/in', {399 + 'msgId': _uuid(),400 + 'type': 'voice',401 + 'sessionId': sessionId,402 + 'audioBase64': message['audioBase64'] ?? '',403 + 'messageId': message['messageId'] ?? '',404 + 'ts': _now(),405 + }, MqttQos.atLeastOnce);406 + return;407 + }408 +409 + if (type == 'image' && sessionId != null) {410 + // Image message411 + _publish('pailot/$sessionId/in', {412 + 'msgId': _uuid(),413 + 'type': 'image',414 + 'sessionId': sessionId,415 + 'imageBase64': message['imageBase64'] ?? '',416 + 'mimeType': message['mimeType'] ?? 'image/jpeg',417 + 'caption': message['caption'] ?? '',418 + 'ts': _now(),419 + }, MqttQos.atLeastOnce);420 + return;421 + }422 +423 + if (type == 'tts' && sessionId != null) {424 + // TTS request — route as command425 + _publish('pailot/control/in', {426 + 'msgId': _uuid(),427 + 'type': 'command',428 + 'command': 'tts',429 + 'text': message['text'] ?? '',430 + 'sessionId': sessionId,431 + 'ts': _now(),432 + }, MqttQos.atLeastOnce);433 + return;434 + }435 +436 + // Default: plain text message (content + sessionId)437 + if (sessionId != null) {438 + final content = message['content'] as String? ?? '';439 + _publish('pailot/$sessionId/in', {440 + 'msgId': _uuid(),441 + 'type': 'text',442 + 'sessionId': sessionId,443 + 'content': content,444 + 'ts': _now(),445 + }, MqttQos.atLeastOnce);446 + return;447 + }448 +449 + onError?.call('Cannot send message: missing sessionId');450 + }451 +452 + /// Disconnect intentionally.453 + void disconnect() {454 + _intentionalClose = true;455 + _updatesSub?.cancel();456 + _updatesSub = null;457 +458 + try {459 + _client?.disconnect();460 + } catch (_) {}461 + _client = null;462 +463 + _setStatus(ConnectionStatus.disconnected);464 + onClose?.call();465 + }466 +467 + /// Update config and reconnect.468 + Future<void> updateConfig(ServerConfig newConfig) async {469 + config = newConfig;470 + disconnect();471 + await Future.delayed(const Duration(milliseconds: 100));472 + await connect();473 + }474 +475 + /// Dispose all resources.476 + void dispose() {477 + disconnect();478 + }479 +480 + // App lifecycle integration481 + @override482 + void didChangeAppLifecycleState(AppLifecycleState state) {483 + switch (state) {484 + case AppLifecycleState.resumed:485 + if (_status != ConnectionStatus.connected && !_intentionalClose) {486 + connect();487 + }488 + case AppLifecycleState.paused:489 + // Keep connection alive — MQTT handles keepalive natively490 + break;491 + default:492 + break;493 + }494 + }495 +}lib/widgets/input_bar.dart
.. .. @@ -111,8 +111,7 @@ 111 111 const EdgeInsets.symmetric(horizontal: 16, vertical: 10), 112 112 isDense: true, 113 113 ), 114 - textInputAction: TextInputAction.send,115 - onSubmitted: (_) => onSendText(),114 + textInputAction: TextInputAction.newline,116 115 maxLines: 4, 117 116 minLines: 1, 118 117 ), lib/widgets/message_bubble.dart
.. .. @@ -1,5 +1,6 @@ 1 1 import 'dart:convert'; 2 2 import 'dart:math'; 3 +import 'dart:typed_data';3 4 4 5 import 'package:flutter/material.dart'; 5 6 import 'package:flutter/services.dart'; .. .. @@ -8,6 +9,9 @@ 8 9 import '../models/message.dart'; 9 10 import '../theme/app_theme.dart'; 10 11 import 'image_viewer.dart'; 12 +13 +// Cache decoded image bytes to prevent flicker on widget rebuild14 +final Map<String, Uint8List> _imageCache = {};11 15 12 16 /// Chat message bubble with support for text, voice, and image types. 13 17 class MessageBubble extends StatelessWidget { .. .. @@ -208,11 +212,13 @@ 208 212 return const Text('Image unavailable'); 209 213 } 210 214 211 - final bytes = base64Decode(212 - message.imageBase64!.contains(',')213 - ? message.imageBase64!.split(',').last214 - : message.imageBase64!,215 - );215 + // Cache decoded bytes to prevent flicker on rebuild216 + final bytes = _imageCache.putIfAbsent(message.id, () {217 + final raw = message.imageBase64!;218 + return Uint8List.fromList(base64Decode(219 + raw.contains(',') ? raw.split(',').last : raw,220 + ));221 + });216 222 217 223 return Column( 218 224 crossAxisAlignment: CrossAxisAlignment.start, .. .. @@ -232,6 +238,7 @@ 232 238 width: 260, 233 239 height: 180, 234 240 fit: BoxFit.cover, 241 + gaplessPlayback: true,235 242 errorBuilder: (_, e, st) => const SizedBox( 236 243 width: 260, 237 244 height: 60, pubspec.lock
.. .. @@ -161,6 +161,14 @@ 161 161 url: "https://pub.dev" 162 162 source: hosted 163 163 version: "7.0.3" 164 + event_bus:165 + dependency: transitive166 + description:167 + name: event_bus168 + sha256: "1a55e97923769c286d295240048fc180e7b0768902c3c2e869fe059aafa15304"169 + url: "https://pub.dev"170 + source: hosted171 + version: "2.0.1"164 172 fake_async: 165 173 dependency: transitive 166 174 description: .. .. @@ -504,6 +512,14 @@ 504 512 url: "https://pub.dev" 505 513 source: hosted 506 514 version: "2.0.0" 515 + mqtt_client:516 + dependency: "direct main"517 + description:518 + name: mqtt_client519 + sha256: fd22ea00a4c7b5623e01000a91a256d62a8bacba38e9812170458070c52affed520 + url: "https://pub.dev"521 + source: hosted522 + version: "10.11.9"507 523 native_toolchain_c: 508 524 dependency: transitive 509 525 description: pubspec.yaml
.. .. @@ -26,6 +26,7 @@ 26 26 share_plus: ^12.0.1 27 27 udp: ^5.0.3 28 28 intl: ^0.20.2 29 + mqtt_client: ^10.6.029 30 uuid: ^4.5.1 30 31 collection: ^1.19.1 31 32