Browse In-App Agent

RevRag Flutter SDK — AI Agent Integration Master Prompt

How to use: paste this entire document into your AI coding agent (Copilot, Cursor, Claude Code, etc.) inside the Flutter app you want to integrate. The agent will inspect your app, install and configure embed_flutter, ask you the few decisions only you can make, then validate the integration end-to-end and report.

Every rule below is derived from the SDK's source (v0.4.0). Where this prompt and a generic Flutter tutorial disagree, this prompt is correct for this SDK. Flutter is materially simpler to integrate than the React Native SDK: there is no native bootstrap code (no MainApplication/AppDelegate edits, no registerGlobals, no Babel/Metro/worklets/Gesture-Handler setup) — the livekit_client plugin registers its own native side.


You are an expert Flutter integration agent. Take this application from zero → working → validated RevRag SDK integration with minimal manual work.

Prime directives

  1. Inspect first, modify second. Never assume Flutter/Dart version, navigation library, state management, or native setup. Read the project.
  2. Preserve host behavior. Merge into existing config; never blindly overwrite AndroidManifest.xml, Info.plist, Podfile, main.dart, or the root widget.
  3. Never claim success without running the validation in §16–§21 and observing the log lines named there.
  4. Ask the developer for the decisions in §13 — do not guess them.
  5. Never hide a crash or error. The SDK ships its own crash-guard that suppresses only LiveKit/WebRTC async errors (§21); everything else must surface. Report SDK-side issues you cannot fix.
  6. Do not add dependencies, permissions, or config the SDK does not require.

1. Inspect the existing application

Record each of these before changing anything:

WhatHow to find it
Flutter / Dart versionflutter --version; environment: in pubspec.yaml
NavigationMaterialApp named routes / onGenerateRoute, MaterialApp.router + GoRouter, Navigator 2.0, or a tab shell (ShellRoute / BottomNavigationBar / IndexedStack)
Where runApp mountslib/main.dart — is there an auth/login gate before the app mounts?
Root widgetthe widget passed to runApp(...); what wraps MaterialApp
Existing LiveKit / WebRTCgrep -n "livekit_client|flutter_webrtc" pubspec.yaml pubspec.lock — note versions
Existing shared peerspermission_handler, shared_preferences, path_provider, http, lottie in pubspec.yaml (conflict risk, §4/§22)
Providers above the rootany global overlay/portal/theme wrapper that sits above MaterialApp (occlusion risk, §22)
Permissions already declaredandroid/app/src/main/AndroidManifest.xml, ios/Runner/Info.plist
App version sourcea build constant, --dart-define, CI-stamped value, or package_info — the SDK does NOT auto-derive it (§10)
Platform targetsandroid/app/build.gradle minSdkVersion; ios/Podfile platform :ios; is web/desktop a target? (unsupported, §2)

VALIDATE: you can state, in one sentence each, the navigation type, the runApp location, whether a login gate exists, and where the app version comes from.


2. Determine compatibility

Supported: Flutter ≥ 3.29 with Dart ≥ 3.7 (pubspec.yaml declares sdk: ">=3.7.0 <4.0.0", flutter: ">=3.29.0"). The floor is real, not cosmetic: the SDK reads SemanticsData.role, which first shipped in Flutter 3.29. Android minSdkVersion ≥ 21 and compileSdk 34+ (floors from flutter_webrtc/livekit_client, not the SDK's Dart), iOS deployment target 13.0. Web and desktop are not supported (LiveKit mobile stack) — Android + iOS only. Reference-tested on Flutter 3.41.x.

TargetFloorNotes
Flutter≥ 3.29Needs SemanticsData.role (3.29). Below this, flutter pub get refuses to resolve ^0.4.0.
Dart≥ 3.7 (< 4.0)Bundled with Flutter 3.29+.
AndroidminSdk 21, compileSdk 34+Non-negotiable WebRTC floor.
iOS13.0platform :ios, '13.0' in the Podfile.
Web / desktopunsupportedAndroid + iOS only.

If any platform target is below floor: explain, raise it, re-validate. Do not upgrade/downgrade unrelated dependencies.

If the host's Flutter is older than 3.29 (check flutter --version and the environment: block): do not silently upgrade the host's Flutter toolchain — that is a project-wide change. Ask the developer which they want:

  1. Upgrade Flutter to ≥ 3.29 (recommended; 0.3.0+ has the result-accuracy, screen-naming and release-build fixes in §14/§17/§19–§21, and 0.4.0 the call-safety fixes), then use embed_flutter: ^0.4.0; or
  2. Stay on embed_flutter: 0.2.0 (pin exactly — embed_flutter: 0.2.0, not ^0.2.0, which can never resolve to 0.4.0 anyway). 0.2.0 supports Flutter ≥ 3.0 / Dart ≥ 3.0 and the same integration API (embedInitialize, EmbedWidget, EmbedNavigatorObserver, USER_DATA). The 0.3.0+ behaviour described in this prompt (route-name-first screen naming, the new failure codes, the new wire fields, and everything new in 0.4.0) does not apply on 0.2.0 — say so in your report.

3. Install the SDK

# pubspec.yaml
dependencies:
  embed_flutter: ^0.4.0
flutter pub get

VALIDATE: flutter pub get resolves with no version-solve error. If it fails on livekit_client / flutter_webrtc / permission_handler / lottie, an existing host pin is conflicting — reconcile the range (§4); do not fork the plugin.


4. Install and configure dependencies

There is nothing to configure. embed_flutter pulls livekit_client (→ flutter_webrtc), permission_handler, shared_preferences, path_provider, http, and lottie transitively; pub's solver shares one copy of each. Unlike React Native, there is no Babel plugin, no Metro config, no worklets, no Gesture-Handler root view, and no registerGlobals.

Verify a single resolved copy of the native stack:

flutter pub deps | grep -E "livekit_client|flutter_webrtc|permission_handler|lottie"

Expected: exactly one resolved version of each. A second copy of livekit_client pulled in by the host is a version-solve failure, not a silent double-runtime — reconcile the range.


5. Android native integration

  1. Permissions: add nothing. From 0.4.0 embed_flutter is a Flutter plugin whose own AndroidManifest.xml declares every permission a call needs (INTERNET, ACCESS_NETWORK_STATE, RECORD_AUDIO, MODIFY_AUDIO_SETTINGS, WAKE_LOCK, BLUETOOTH_CONNECT, and BLUETOOTH up to Android 11); it is merged into the host at build time. Do not add them to the host manifest (prime directive 6). Entries the host already has are harmless — leave them. The SDK requests the runtime ones itself: the mic at call start, and BLUETOOTH_CONNECT (Android 12+) only when a Bluetooth headset is connected; a Bluetooth denial never blocks the call.
  2. No MainApplication / MainActivity code is required. The plugin auto-registers; MainActivity stays a plain FlutterActivity. (There is no audioRecordSamplesDispatcher-style manual-setup trap as in RN.)
  3. android/app/build.gradle: minSdkVersion 21 (or higher), compileSdkVersion at the Flutter default (34+). The mic runtime prompt is done by the SDK during startCall via permission_handler.
  4. Build: flutter build apk --debug. On failure decide SDK-related vs pre-existing; fix; rebuild. Never suppress errors.

VALIDATE: APK builds; the merged manifest (build/app/intermediates/merged_manifests/.../AndroidManifest.xml) contains RECORD_AUDIO and BLUETOOTH_CONNECT (from embed_flutter) without any host edit.


6. iOS native integration

  1. ios/Runner/Info.plist — NSMicrophoneUsageDescription is mandatory (iOS kills the app on a mic access without it):
    <key>NSMicrophoneUsageDescription</key>
    <string>This app needs microphone access for voice calls.</string>
  2. ios/Podfile — set the platform and add the permission_handler macro. The SDK requests the mic through permission_handler (Permission.microphone.request()), so PERMISSION_MICROPHONE=1 is required — without it the dialog never appears and the permission is always reported denied:
    platform :ios, '13.0'
    
    post_install do |installer|
      installer.pods_project.targets.each do |target|
        flutter_additional_ios_build_settings(target)
        target.build_configurations.each do |config|
          config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
            '$(inherited)', 'PERMISSION_MICROPHONE=1',
          ]
        end
      end
    end
    Then cd ios && pod install.
  3. No AppDelegate.swift edits are required — there is no LivekitReactNative.setup analog.
  4. The bundled examples/BajajFinance/ios/Podfile omits both the macro and the platform :ios, '13.0' line — do not copy it verbatim, or iOS mic requests are auto-denied and the WebRTC pods may fail to build below iOS 13.

VALIDATE: pod install completes; flutter build ios --no-codesign succeeds; Info.plist has NSMicrophoneUsageDescription.


7. Build & analyzer validation

flutter pub get
dart analyze              # host app must stay at its own baseline
flutter build apk --debug
flutter build ios --no-codesign

VALIDATE: both platform builds compile. A red analyzer error introduced by the integration (e.g. a wrong embedInitialize signature) must be fixed here, not deferred.


8. SDK initialization (exact contract)

Call embedInitialize once, before runApp. All parameters are named; apiKey and appVersion are required:

import 'package:embed_flutter/embed_flutter.dart';

void main() {
  embedInitialize(
    apiKey: API_KEY,                     // REQUIRED (named), from the app's config/env
    appVersion: APP_VERSION,             // REQUIRED (named) — see §10
    embedUrl: EMBED_URL,                 // optional; RevRag-provided host, defaults to prod
    onResult: (bool success, String? error) {
      if (!success) debugPrint('Embed init failed: $error');
    },
  );
  runApp(const MyApp());
}
  • embedInitialize is the only place credentials and app version enter the SDK. There is no EmbedProvider widget and no positional apiKey — embedInitialize('key', ...) will not compile.
  • It is synchronous and fail-open: it never throws at your call site. Do NOT block the app on init; render the app — the widget simply won't appear until config arrives.
  • It installs the host-crash safety net (EmbedCrashGuard, §21) before any LiveKit work.

Then wrap the tree — EmbedWidget must be the outermost widget (above MaterialApp, above any portal/overlay/theme wrapper, or the FAB and highlights end up underneath):

EmbedWidget(
  enabledRoutes: const ['home', 'orders', 'cart', 'checkout'],
  child: MaterialApp(
    navigatorObservers: [EmbedNavigatorObserver()], // REQUIRED, §14
    // ...
  ),
);

VALIDATE: the call compiles with named apiKey: / appVersion:; onResult fires. On the first run with a valid key you see [ApiService] ← GET /initialize status=200 and (debug builds only) [Embed][ui_config] … full payload: {…}.


9. API key validation

  • The key is sent as the X-Revrag-Embedded-Key header on GET /embedded-agent/initialize. A wrong or missing key still compiles and runs — the fetch returns non-200, so the SDK logs [ApiService] ← GET /initialize status=<code> then [ApiService] ⚠️ /initialize returned non-200 (<code>) — UI config is NULL, widget will use SDK defaults; onResult fires (false, "Failed to fetch widget configuration and no cached configuration was available.") (when nothing is cached), and the agent button never appears.
  • Do not hard-code secrets in source if an env system exists. If missing → stop, report "API key missing", ask the developer.

VALIDATE: with the real key you see [ApiService] ← GET /initialize status=200 and onResult(success=true). A non-200 (401 / 403) means the key or header is wrong — do not proceed to §11 until fixed.


10. App version validation

  • appVersion is required in embedInitialize and is the only intake — there is no auto-derivation from package_info and no post-init setter. It is sent as ?app_version=<value> on /initialize and attached to every outbound event in the sdk block as app_version.
  • Derive it from a single source of truth (a build constant, a --dart-define, or the value your CI stamps). Never pass "" or a stale literal.

VALIDATE: the /initialize URL query shows ?app_version=<your value>, and outbound event payloads carry the same value in the sdk block.


11. USER_DATA validation

The agent stays hidden and no call can connect until the SDK knows the user. After login, send:

import 'package:embed_flutter/embed_flutter.dart';

embedEvent(
  EventKeys.USER_DATA,
  UserEventPayload(
    app_user_id: userId,                     // required — your stable user id
    data: {'name': 'Priya', 'plan': 'gold'}, // optional context
  ),
);

Facts the integration must respect:

  • Place it immediately after auth succeeds. For a no-auth/demo app, send it in the first screen's initState.
  • Without it, a call's token fetch is skipped — the SDK logs [ApiService] token request skipped: no app_user_id yet. Send a USER_DATA event … after login, before the agent is expected to connect. — so the call cannot connect.
  • No need to wait for config: with a user set, events go out whatever the widget is doing (§18). Before any user is set, events are not sent and embedEvent returns false.
  • The user is kept across app restarts (0.4.0) until logout. Wire await EmbedWidget.clearUserSession(); into the host's logout flow — it ends any call, wipes the SDK's screen cache and form values, and forgets the user (also with no widget mounted). Skipping it leaves the next person identified as the previous user. embedClearStorageCache() also logs out. Re-send USER_DATA for the next user.
  • embedEvent returns a Future<bool> and never throws.

Required order: embedInitialize → init success → USER_DATA → widget config → FAB shown on an enabled route → interactions → call.


12. Widget configuration validation

The FAB renders only after /initialize returns a widget_config (position, colors, avatar, collapsed view, inactivity nudge). It is dashboard-driven — there are no host props for position or colors. The SDK merges it and type-coerces every field, so a malformed value falls back to a default (it never crashes a build).

CaseExpectedObserve
A. config existsFAB appears on allowed routes; agent_visible fires[Embed][ui_config] … full payload: {…}
B. config null / {}default styling (black button, #B391F3 gradient), no crash[Embed][ui_config] … (button_color→black, gradient→#B391F3, text→white)
C. init failsno FAB, app fully functionalonResult(false, …); [ApiService] ⚠️ /initialize returned non-200
D. delayed configapp must not assume config is ready; FAB appears when it landson a cold start the FAB may need the next route change to mount

Also confirm the native cache store: on-device UI-graph writes log under the [EmbedCache] tag; [EmbedCache] store unavailable, memory-only: … means the file store failed and the cache is in memory only.


13. Ask the developer (do not guess)

After inspecting, list the actual route names you found (the names the navigator reports — e.g. home, checkout; not display titles), then ask:

  1. Which routes should the RevRag widget appear on? (Offer: all-except via showOnAllRoutes: true + disabledRoutes, or an explicit enabledRoutes list.)
  2. Which routes must never show the agent / must end a call on entry (login, OTP, payment, regulated data)? → disabledRoutes (entering one ends a live call). This is also the only way to stop the agent's missions from looking at a screen (§17). Ask too which dialogs / bottom sheets are sensitive (OTP, card entry): give them a RouteSettings(name:) and list that name — it blocks capture while open but does not hide the FAB.
  3. Should a live call survive navigation between agent routes? → continuity: EmbedButtonContinuity.keepVisible vs reset (default) (§15).
  4. Do different route groups need different show/delay behavior? → buttonVisibility groups (EmbedButtonVisibilityConfig, §15).
  5. App version source if it can't be inferred; API key location.
  6. Any WebView-only, native-picker, or custom-navigation flows (see §22).

Wait for answers before finalizing screen integration.


14. Screen integration (how the SDK actually detects screens)

Register the observer; EmbedWidget wraps MaterialApp. Detection rules by navigator:

  • Named routes / onGenerateRoute — register EmbedNavigatorObserver() on MaterialApp.navigatorObservers. Always pass settings: settings in onGenerateRoute so routes have names.
  • GoRouter — put EmbedNavigatorObserver() in GoRouter(observers: [...]) and use each route's name: (not path:) in enabledRoutes.
  • Tabs / ShellRoute / BottomNavigationBar / IndexedStack — tab bodies live in a nested navigator the top observer can't see. Wrap each tab body in EmbedRouteListener(routeName: 'home', child: …) whose routeName matches enabledRoutes.
  • Anonymous pushes — a MaterialPageRoute with no settings reports a null name and is invisible. Give it settings: RouteSettings(name: '…'), or supply a custom EmbedNavigatorObserver(nameExtractor: (route) => …).
  • Matching — routeMatchMode is exact (default), startsWith, or contains (RouteMatchMode). Leading slashes are normalized ('home' == '/home'). This is the analog of RN's screen-matching tiers.
  • Named popups — dialogs / bottom sheets opened with routeSettings: RouteSettings(name: '…') never change the page route. Listing such a name in disabledRoutes blocks capture (missions and telemetry) while it is open, but does not hide the FAB — the button follows the page underneath. Named popups still emit SCREEN_STATE route-change analytics on open/close.

14.1 Stable screen names (0.3.0)

Separately from FAB visibility, every UI capture is filed under a screen name. It keys the on-device UI-graph cache, prefixes every element's stable ID, and is how the agent knows a tap changed screens. 0.3.0 resolves it in this order:

  1. The current route name — RouteSettings.name, your nameExtractor result, or the name set by EmbedRouteListener(routeName:). If a tab shell (IndexedStack / PageView) is in front, the active tab is appended: route/tab (e.g. home/portfolio), so every tab of a single-route shell gets its own screen.
  2. Stable widget-tree signals only — used for the tab part, and as the whole name when the top page is unnamed:
    • IndexedStack tab, in order: a ValueKey<String> on the tab's root widget, a Semantics(identifier:) at the top of the tab, then the tab's first heading (Semantics(header: true) — an AppBar title);
    • PageView(children: …) page: a ValueKey<String> on the page child only;
    • unnamed top page: its first heading. A heading only counts when it is short (≤ 4 words, ≤ 32 chars) and digit-free. Names are slugged ("My Orders" → my_orders).
  3. Unnamed pushed pages with no stable signal are named by their depth above the last named route: route/page_N (e.g. checkout/page_1) — they never inherit the page below them.
  4. Otherwise the active visibility group, then unknown_flow.

Class names are never used — not in debug, not in release — so debug, release and --obfuscate builds produce identical screen names, stable-ID prefixes and cache keys.

What to ask / fix in the host (in priority order):

  • Name every route — settings: RouteSettings(name: '…') on every MaterialPageRoute / onGenerateRoute result, name: on every GoRoute. route/page_N in the logs means an unnamed push: name it.
  • Give each tab root a ValueKey<String> in IndexedStack / PageView shells (IndexedStack(children: [HomeTab(key: ValueKey('home')), …])), or wrap tab bodies in EmbedRouteListener. Do not rely on headings for tabs whose title carries a number, a greeting or a user's name — those are rejected as names.
  • Never derive names from runtimeType / toString() in a custom nameExtractor — release builds strip toString() of Flutter types and --obfuscate renames classes.

VALIDATE: EmbedWidget(debug: true) turns on the SDK's verbose logging (EmbedLog), which narrates config load, route changes, and FAB-visibility decisions; navigate to an enabled route and confirm the FAB appears (after USER_DATA). For screen names, set WireLog.enabled = true in a debug build (it is forced off in release) and check that each [Embed ▶ SEND] type=ui_diff … screen=<name> line shows a meaningful route name (or route/tab), not unknown_flow or …/page_N. Switch every tab and confirm each gets a distinct name.


15. Continuous vs per-screen — what to verify

Host-side visibility props (RN parity, single-agent):

PropEffect
enabledRoutes: List<String>Routes the FAB shows on (default timing).
showOnAllRoutes: boolShow everywhere; pair with disabledRoutes.
disabledRoutes: List<String>Always hidden; the call ends when the FAB hides here. Also the only list that keeps the agent's missions from looking at a screen (§17). A named popup listed here blocks capture while open but does not hide the FAB.
continuity: EmbedButtonContinuitykeepVisible = FAB (and call) stay up across enabled routes; reset (default) = re-show/re-delay each route.
embedButtonDelayMs / buttonDelayPre-show delay + EmbedButtonDelayPolicy (perScreen / oncePerGroupEntry / oncePerAppSession).
buttonVisibility: EmbedButtonVisibilityConfigVisibility groups — {defaultDelayMs, groups}, each group {id, screens, continuity, delayMs?, delayPolicy?}; first match wins (see INTEGRATION.md §7.2).

Behavior to verify:

  • Under showOnAllRoutes: true, a call is continuous across screens — it is not disconnected on each navigation. Add a route to disabledRoutes to force a disconnect there (RN's endCallWhenHiddenByVisibility is automatic in Flutter: leaving the allow-list ends the call).
  • A group's screens join the allow-list — a route listed in a buttonVisibility group shows the FAB even if it is not in enabledRoutes.

VALIDATE: navigate across two enabled routes with continuity: keepVisible during a live call — the call persists; navigate to a disabledRoutes route — the call ends (agent_conversation_ended).


16. Event ordering validation

Confirm this order in the logs on a healthy first session, and that nothing fires out of order:

  1. [ApiService] ← GET /initialize status=200 → [Embed][ui_config] … full payload
  2. USER_DATA sent (after login) → token fetch unlocked
  3. agent_visible on an enabled route
  4. on FAB tap / startCall: [CallTiming] t=0ms startCall() invoked → [CallTiming] t=<ms>ms … → ROOM CONNECTED — call is live → agent_conversation_started (carries a flat call_id)
  5. on end: agent_conversation_ended (same call_id)

Invalid patterns to catch: a call before USER_DATA ([ApiService] token request skipped: no app_user_id yet …), agent_visible never firing (re-check §11 and §14), a second EmbedWidget in the tree.


17. UI tree / snapshot validation

For the agent to act (tap/scroll/type), it reads your Flutter Semantics tree — the same data screen readers use. Mission frames arrive on the LiveKit data channel; element highlights log [Embed HIGHLIGHT] pulse stableId=…; the durable on-device UI-graph writes under [EmbedCache]. To see the wire traffic itself, set WireLog.enabled = true in a debug build (e.g. WireLog.enabled = kDebugMode; before runApp): you then get [Embed ◀ RECV] type=mission …, [Embed ▶ SEND] type=ui_diff … screen=…, and [EmbedWidget] 📩 LiveKit DataReceived … headers. WireLog is forced off in release builds whatever the host sets, and inbound set_text values are redacted in the debug log. Deep guidance lives in HOST_INTEGRATION.md; the essentials:

  • Icon-only buttons need Semantics(label: …, button: true, …). Without a label the SDK falls back to a name from what the control draws — an image's asset file name (ic_home_active.png → home), a common Material/Cupertino icon's literal name (Icons.pie_chart → pie chart), or, for an unlabelled tab, Tab 2 of 4. These are fallbacks: a real label always wins, and "pie chart" is not "Portfolio".
  • Chips / toggles / custom radios / custom tabs must surface selection (Semantics(selected: …) / Semantics(checked: …), or use ChoiceChip / FilterChip / Radio / Switch). Without it the SDK infers selection — a custom tab row reports selected only when exactly one item is drawn differently (else from the tab the session saw lead to the current screen); a custom option group whose items each draw a small round indicator becomes role radio with checked only when exactly one has a filled dot. Explicit semantics always win; inference returns nothing when unsure.
  • Custom bottom-navigation rows (≥ 3 short tappables aligned in the bottom ~20% of the screen, not inside a scrollable, not a Material button bar, grid or keypad) are reported with role tab and kept first under the 15 KB snapshot budget.
  • Critical CTAs (login, pay, submit) should carry Semantics(identifier: 'checkout.pay') — the SDK matches identifier first.
  • Disabled controls (onPressed: null, TextField(enabled: false)) are captured with disabled: true; a tap or set_text on them fails at once with target_disabled.
  • Secure fields (TextField(obscureText: true) → SemanticsFlag.isObscured) are redacted at capture and stay redacted (sticky) — typed secrets never leave the device in the UI snapshot (secure: true, no value).
  • Who may look (0.3.0). Agent missions (find / tap / scroll / read / verify and their responses) are blocked only by explicit host exclusions: disabledRoutes (matching the current route or the detected screen name) and an open named popup listed there. action_config.flags.dataCapture: false, a hidden FAB, and a route missing from enabledRoutes / the platform allowlist do not blind a running mission — dataCapture: false stops telemetry only (live ui_diff pushes, the mission_exploration audit, the UI-graph cache). Telemetry also continues while the FAB is hidden (flags.transmitOnHiddenWidget defaults on; only an explicit false turns it off). A refused mission response is replaced by a failure frame (screen_not_capturable / payload_too_large) — never dropped silently. Host query: EmbedWidget.isWidgetTreeSnapshotEnabled.

What a healthy node looks like on the wire (one entry of a ui_diff baseline's nodes; keys are omitted when empty/false):

{
  "id": "checkout.group_0.full_name",      // stable ID, prefixed by the screen name
  "role": "input_field",                     // Android vocabulary, see below
  "text": "Full Name",
  "bounds": [16, 220, 376, 276],             // logical px
  "visible": true,                           // ≥ 5% of the node on screen
  "visible_fraction": 1.0,
  "has_action": true,
  "parent_id": "checkout.group_0",           // nearest kept ancestor
  "actions": ["tap", "setText"],             // setText on every writable field
  "editable": true,
  "placeholder": "As on PAN card",           // input hint (editable nodes only)
  "disabled": false                          // measured; absent when unknown
}

Other 0.3.0 node keys: range: {min, max, step?} on sliders when the host provides bounds, selected: true, checked: true|false|"mixed", secure: true, identifier, layer (> 0 inside a modal / sheet), subtree_text. Roles use Android's vocabulary: button, card, text, heading, input_field, checkbox, radio, switch, slider, link, image, tab, navigation, listitem, row, group, scrollable, modal, overlay, none — raw Flutter role names never appear. Snapshot frames also carry app_user_id (after USER_DATA), app_version and sdk.

A healthy snapshot for a key screen: the screen name is the route name (or route/tab); every tappable the user would name has a non-empty text (icon-only items labelled); the active tab / chip / radio has selected / checked; text fields carry setText and their placeholder; disabled CTAs carry disabled: true; password/OTP fields carry secure: true and no value; nothing is called unknown_flow.

What the tree cannot see: anything with no Semantics (raw CustomPaint, some third-party UI kits), platform views / WebViews, and native pickers.

VALIDATE: turn on TalkBack/VoiceOver and tab through a key screen — anything the reader can't name is invisible to the agent. Missions failing with target_not_found point here; ambiguous_target means two controls carry the same label (give them distinct labels or identifiers). A screen file over the cache upload budget logs [EmbedCache] screen "…" is <N> bytes, over the <M> batch budget; sending alone.


18. Event / click-capture validation

Automatic capture. The SDK reports taps on every tappable control (buttons, GestureDetector, InkWell, ListTile, checkboxes, switches, radios, tabs) by finding the tapped node in the Semantics tree, and watches TextField controllers for text changes — all gated by the dashboard clickTracking flag (ActionGates.clickTrackingEnabled), only while the widget is active (EmbedWidget.isActive) and only once a user is set. Click event_names follow the convention shared with Android, RN and web (0.4.0): <name>_<suffix> (pressed / toggled / selected / opened), no screen suffix (the screen is in context.screen). <name> = the control's ValueKey<String> → Semantics(identifier:) → tooltip / visible text → the image or icon it draws → role. Rapid taps (3 within 1 s) emit a rage_click analytics event.

The agent's targeting id (stableId) is separate from analytics: it resolves Semantics(identifier:) → text → role. Put a stable identifier on every meaningful control:

Semantics(identifier: 'submit_interest', button: true, child: MyButton(onTap: submit));

Explicit context. Send app context so the planner disambiguates screens/actions:

embedEvent(EventKeys.SCREEN_STATE,
  ScreenEventPayload(screen: 'checkout_step_2', data: {'step': 2}));
embedEvent(EventKeys.CUSTOM_EVENT,
  CustomEventPayload(data: {'action': 'plan_selected', 'plan_id': 'gold'}));
embedEvent(EventKeys.ANALYTICS_DATA,
  AnalyticsDataEventPayload(event_name: 'payment_completed', data: {'amount': 999}));

Observe agent events with embedOnAgent(...) / SdkEventName (see INTEGRATION.md §9). rage_click and screen_view are analytics only — they don't reach embedOnAgent; use embedOn for them.

VALIDATE: with USER_DATA in place, events are sent immediately and return true / 2xx whatever the widget is doing (hidden, delayed, excluded route, config still loading). Before any user is set they are not queued: embedEvent returns false. Failed analytics are retried one by one from a persisted queue (at most 100).


19. Call validation

Preconditions the SDK enforces: config loaded (FAB present), mic permission granted (the SDK prompts via permission_handler; denied → EmbedWidget(onPermissionStatusChanged:) gets false + microphone_permission_denied), and USER_DATA stored (else the token fetch is skipped, §11). Call start is single-flight (startCall no-ops while already active/connecting).

Programmatic API (see INTEGRATION.md §10):

await EmbedWidget.startCall();                                   // co-pilot (default)
await EmbedWidget.startCall(agentTriggerMode: AgentTriggerMode.WORKFLOW);
await EmbedWidget.endCall();
EmbedWidget.isCallActive;                                        // bool
EmbedWidget.collapseWidget();  EmbedWidget.expandWidget();       // bar <-> FAB
await EmbedWidget.pauseAgent();  await EmbedWidget.resumeAgent(); // keep the call, mute the agent
EmbedWidget.isAgentPaused;                                       // bool
await EmbedWidget.clearUserSession();                            // on logout
  • startCall() does not auto-expand the bar; pair with expandWidget() if you start from your own UI.
  • AgentTriggerMode: CO_PILOT (default — agent drives your screens, panels dropped) vs WORKFLOW (Revrag-owned panel/avatar surface).
  • pauseAgent() / resumeAgent() / isAgentPaused (0.4.0; also embedPauseAgent() / embedResumeAgent() / embedIsAgentPaused()): the agent stops speaking and listening and the mic is muted; resume restores the mic. They return false with no call or no agent answer.
  • Not present in the Flutter SDK (unlike RN / Android): isWidgetExpanded. Do not reference it.
  • A user touch on the host app while the agent is acting cancels that action (failure_reason: user_interrupted).

VALIDATE (real device): startCall() → [CallTiming] t=0ms startCall() invoked → ROOM CONNECTED — call is live → agent_conversation_started → isCallActive true → endCall() → agent_conversation_ended. Then a second call starts cleanly.

19.1 Agent actions (missions) — reading the result frames

During a co-pilot call the backend sends mission frames (one or more steps: click, navigate, set_text, scroll_to, check, select, set_slider, back, …) and the SDK answers on the data channel. To read them, run a debug build with WireLog.enabled = true and filter on [Embed (bodies longer than ~900 bytes are split into numbered [Embed body#N i/n] chunks — reassemble them). Per step you will see mission_phase (searching → executing → verifying, plus disambiguating with candidates), one verification_result, and runner lines such as [Embed VERIFY mid=… step=… success=… evidence=[…] reason=…]. Every mission ends with exactly one terminal mission_status: succeeded, failed or cancelled.

0.3.0 result honesty. A step is succeeded only when the intended target was acted on and verification supports it; otherwise it is failed with a specific failure_reason (the terminal vocabulary itself is unchanged). How to read the frames:

FieldWhereMeaning
step_indexevery terminal mission_status (on succeeded: the last step)Which chained step ended the mission.
steps_completed / steps_totalterminal mission_statusHow far a multi-step mission got.
steps_skippedterminal failed / cancelledSteps that never ran.
steps_failedterminal, continue_on_failure missionsSteps that failed while the mission kept going (each also emits step_failed).
requested_stable_id / requested_textverification_result; mirrored on failure framesWhat the step asked for (requested_text is the label, ≤ 60 chars, never typed text, omitted for obscured fields).
acted_stable_id / acted_roleverification_result (acted_stable_id mirrored on failure)What was actually acted on — absent when nothing was. Compare with requested_*.
match_sourceverification_resultHow the target was found (stable_id, identifier, text_exact, text_token, redirect_label, twin, role, bounds, …).
verifiedverification_resultThe verdict rests on an observation of the intended target.
value_matchedverification_result, set_text onlyThe field's value matches the request — a boolean, never the value itself.
redirectedverification_result, set_text onlyThe text went to a field bound to the matched label.
value / strategyverification_result, set_sliderThe achieved slider value and how it was set.

Second look (observation only). When a dispatched tap / click / navigate / back cannot be confirmed right away, the SDK waits ~1.5 s, re-captures once and re-runs the verifier (log: [Embed SECOND_LOOK mid=… confirmed=… (no re-dispatch)]). A late, network-driven update then becomes a verified success. The action is never pressed a second time; still unconfirmed → the original failure is reported. A tap with no visible effect is likewise never retried, so there are no double side effects (double payments, double submits).

Optional step param params.expect_screen (navigate): when the backend sends it, the landed screen must match it or the step fails target_mismatch — which makes stable screen names (§14.1) matter.

VALIDATE (real device, debug build): ask the agent to do one tap, one navigation and one text entry on a key screen. Each must end succeeded with verified: true and acted_stable_id equal to (or clearly the same control as) requested_*. Any failed frame: read its failure_reason in the §20 mission table and fix the host side it points to before declaring success.


20. Error / failure conditions to test explicitly

ConditionExpected (correct) behavior
Invalid apiKeyNon-200 (e.g. 401/403), UI config NULL, onResult(false, …), FAB hidden, no crash.
appVersion blank/stale/initialize still fires with whatever was passed; supply a real value (§10).
USER_DATA never sent (and no user kept from a previous session)FAB never shows; token fetch skipped (token request skipped: no app_user_id yet); events are not sent and embedEvent returns false.
Mic permission deniedonPermissionStatusChanged(false) / microphone_permission_denied; no crash.
Token fetch fails when the user taps StartThe FAB stays and shows "Couldn't connect — tap to retry"; tapping again retries. A failed pre-call prep at sign-in is retried automatically (backoff, up to 5 times).
User leaves for a hidden screen while a call is still connectingThe call is cancelled; it never goes live without a visible End button.
Network switch mid-call (full LiveKit reconnect)The call continues; a muted or paused mic stays muted, and a running mission is not dropped.
Bluetooth headset connected / disconnected mid-callAudio follows the headset, then returns to the user's speaker choice; the SDK never forces the loudspeaker over a headset.
Backend widget_config = null/{}SDK uses default styling; FAB still works.
Wrong-typed backend config fieldCoerced to a default; no CastError.
Navigate to disabledRoutes mid-callCall ends cleanly.
Bad embedEvent payload typeReturns false (does not throw).
Forced LiveKit renegotiation errorSuppressed by the crash-guard; host process survives (§21).
Backend sends action_config.effects.*Ms: 0Clamped to 1 ms — highlight / border / awakening animations run imperceptibly, no crash.
Backend sends flags.nativeBridge: falseOnly the synthetic-pointer tap fallback is off; accessibility-action taps still work.
Agent asked to act on a disabledRoutes screen / an open excluded named popupNothing on that screen is captured or sent: the mission sees no screen, and a response that would carry its content is replaced by a screen_not_capturable failure frame. The mission still ends with a terminal status — it is never left hanging.

Mission failure codes — how to read them (0.3.0). These arrive as failure_reason on a failed verification_result / terminal mission_status (§19.1). Most point at a host fix; none is a crash.

failure_reasonWhat happenedNothing acted on?Host-side fix
target_not_foundNo element matched, even after scroll-and-recapture.yesLabel the control (Semantics(label:)), surface it in Semantics, check the screen is the one the agent expects (§17).
ambiguous_targetTwo comparable controls match equally well (candidates ride the disambiguating phase). Side-effect verbs refuse to guess.yesGive duplicates distinct labels or Semantics(identifier:).
target_mismatchThe only reachable element names something else — a field labelled "PIN" for "Full Name", "Nominee Name" for "Name", a card whose only button isn't the request, or expect_screen not reached after a navigate.yes (except the navigate expect_screen case)Make each input's label sit in its own container / directly above or left of the field; label fields explicitly; name routes (§14.1).
target_disabledThe target is measured disabled (onPressed: null, enabled: false).yesExpected behaviour — the control really is disabled.
verification_unavailableThe action could not be verified (no capture after it, an unreadable obscured field, or — evidence unattributed_change — something changed elsewhere on screen that can't be tied to the target). Not success and not "nothing happened".no (dispatched)Make the effect visible on the target: a state change on the tapped control, its option group or its card, a route change, or an overlay. Surface selection with Semantics(selected:/checked:).
tap_had_no_observable_effect / navigate_had_no_observable_effectDispatched; even after the ~1.5 s second look nothing tied to the target changed. Never retried.no (dispatched)Check the handler actually fires for semantics taps (a bare GestureDetector needs Semantics(button: true, onTap: …)); a change only in a counter / badge / status line outside the tapped card does not count.
back_had_no_observable_effectBack was reported handled but nothing popped (e.g. a PopScope veto).noWire EmbedWidget(onBackRequested:) for custom back handling.
check_not_applied / select_not_appliedThe checkbox/switch or chosen option is not shown in the new state afterwards.noExpose Semantics(checked:) / Semantics(selected:) on the custom control.
field_has_no_textread_field on an empty field (no longer answered with the label).n/aExpected.
invalid_params / value_out_of_range / range_unknown / set_slider_not_appliedSlider step could not be applied.variesUse a Material Slider or report min/max so range is known.
screen_not_capturable / payload_too_largeA mission response was refused by a host exclusion (disabledRoutes / excluded popup), or could not fit the 15 KB transport cap even after trimming.—Expected for excluded screens; for size, reduce very heavy screens.
invalid_mission / exceptionMalformed step from the backend / unexpected SDK error.—Report to Revrag with the wire log.

21. Crash visibility — grep for these

The SDK is engineered so nothing it does can crash the host (INTEGRATION.md §12): public API returns safe defaults, every timer/listener/post-frame is guarded, config is type-coerced, and a chained PlatformDispatcher.onError (installed by embedInitialize) swallows only livekit_client/flutter_webrtc async errors (e.g. a renegotiation NegotiationError), delegating everything else to the host's handler untouched.

Grep your crash reporting (Sentry / Crashlytics / logs) for these real SDK lines:

  • [EmbedSDK] suppressed voice-stack async error — the guard working as intended (informational; a LiveKit/WebRTC error was kept from crashing the host).
  • [ApiService] ⚠️ /initialize returned non-200 / [ApiService] ⚠️ /initialize fetch FAILED — config load failed; FAB won't show (key/URL/network).
  • [ApiService] token request skipped: no app_user_id yet — USER_DATA not sent (§11).
  • [EmbedCache] sync rejected status=…; pass stopped (401/403 — credentials), [EmbedCache] "…" rejected 3 times (…); quarantined until it changes (one screen file the backend keeps refusing — the rest still sync), [EmbedCache] store unavailable, memory-only — UI-graph sync/store problems (non-fatal).
  • Embed init failed (from your onResult) — surfaces config-load failures.

Release builds are quiet by design (0.3.0): WireLog and inbound-frame logging are forced off in release whatever the host sets, the /initialize response body and the [Embed][ui_config] … lines are printed in debug builds only, and debug logs redact secrets (API keys, tokens, PINs/OTPs, passwords, phone/email) and inbound set_text values. Mission-level lines ([Embed VERIFY …], [Embed SECOND_LOOK …], ▶ SEND / ◀ RECV) therefore exist only in a debug build with WireLog.enabled = true — do not expect them in Crashlytics / Sentry from production. Tracker IDs no longer collapse to Instance of … in release.

Native (platform) failures to classify — not SDK crashes: attempted to access privacy-sensitive data without a usage description (iOS NSMicrophoneUsageDescription missing), a mic prompt that never appears (iOS PERMISSION_MICROPHONE=1 missing, §6), uses-sdk:minSdkVersion … cannot be smaller than version 21 (Android floor, §5). Never swallow a native crash; classify app-side vs SDK-side.


22. Unknown-unknowns — inspect the host for these (detect → why → do)

Work through every row. "Ask" = a developer decision.

UI / layout

  • EmbedWidget not outermost → a global overlay/portal/theme wrapper above it can occlude the FAB and highlights; keep EmbedWidget above MaterialApp; verify on a device.
  • Bottom sheets / dialogs / overlays drawn above the FAB → the agent's highlight and the FAB can end up under them; verify z-order; surface the sheet with Semantics(container: true) so the agent sees it as its own subtree.
  • SafeArea / keyboard insets → the FAB position and highlight bounds can drift near the bottom while the keyboard is open; verify.

Components / design system (Semantics)

  • Icon-only buttons / icon-only nav items without Semantics(label:) → 0.3.0 names them from what they draw (asset file name, a common Material/Cupertino icon's literal name, or Tab i of n), which is the icon, not the meaning ("pie chart", not "Portfolio"; "Tab 3 of 4", not "Orders"). Label them and set button: true.
  • Custom GestureDetector tappables with no semantics → not tappable by the agent; wrap in Semantics(button: true, onTap: …) or use a Material button.
  • Selection shown by style only (custom radios/chips/tabs) → 0.3.0 infers it only when exactly one item is drawn differently (radios: exactly one filled dot) or from tab taps seen this session, and returns nothing when unsure — so check / select / tab taps can end verification_unavailable / select_not_applied. Add Semantics(selected:) / Semantics(checked:) (or use ChoiceChip/Radio/Switch).
  • Custom bottom-navigation bars (a Row of GestureDetectors) → reported as role tab; without Semantics(selected:) the active tab is inferred as above. Mark it explicitly: Semantics(label: 'Portfolio', selected: i == current, child: …).
  • Input labels far from their field (label in another column, or two labelled fields stacked with one caption) → set_text refuses to guess and fails target_mismatch rather than typing into the wrong field. Put each label in the field's own container (or use InputDecoration(labelText:)).
  • CustomPaint / canvas UIs / third-party kits with no Semantics → invisible to the agent (§17); wrap the meaningful bits.

Inputs / forms

  • Uncontrolled TextField (no controller) → harder to verify typed text and to detect changes for auto-capture; prefer a TextEditingController.
  • Password / OTP fields → mark them obscureText: true so SemanticsFlag.isObscured redacts them at capture (sticky). Redaction is per node — mark each secret field, not the wrapper.
  • Read-only field used as a picker trigger → the agent may try to type into it; use a tappable + Text instead.

Lists / performance

  • Lazy lists (ListView.builder, Sliver…) → off-screen rows aren't in the tree (not captured) until scrolled into view; the find ladder scrolls-and-recaptures, but give rows a stable Semantics(identifier:) so they're targetable when visible.
  • Very heavy screens → capture measures the tree; large trees can hit the cache byte budget ([EmbedCache] screen "…" is <N> bytes, over the …) and get trimmed.

Navigation-adjacent

  • Anonymous routes — MaterialPageRoute without settings ⇒ no route name ⇒ invisible to the observer for FAB visibility, and its screen name falls back to its heading or parentRoute/page_N (§14.1). Give every route a name.
  • --obfuscate / --split-debug-info release builds → 0.3.0 never uses class names, so screen names, stable IDs and cache keys are identical to debug. Verify any custom nameExtractor doesn't use runtimeType / toString() (obfuscated or stripped in release).
  • GoRouter observer omitted — must be on GoRouter(observers: […]), not only on an inner MaterialApp; use route name: not path:.
  • Tabs via IndexedStack / PageView with no ValueKey<String> on the tab roots, no Semantics(identifier:) and no short heading → every tab shares the shell's route name (one screen for all tabs, cache and stable IDs collide). Give each tab root a ValueKey<String> (→ route/tab), or wrap each tab body in EmbedRouteListener (also needed for per-tab FAB visibility).
  • Route names differing between enabledRoutes and EmbedRouteListener — they must be identical strings.
  • disabledRoutes intent — a disabled route ends an active call; confirm that's desired (e.g. a payment WebView screen).

Native / platform

  • Backgrounded calls (iOS) → the verified setup covers foreground calls. If a call must keep running while the app is backgrounded, iOS additionally needs audio under UIBackgroundModes in Info.plist; the SDK and example do not set it — confirm the requirement and add it deliberately (don't assume it's handled). Ask.
  • Other audio-session users (audio players, video, VoIP) → the SDK drives the audio session per call; a host that also owns it may conflict. Ask.
  • onPermissionStatusChanged not wired → an EmbedWidget callback prop; if the host doesn't pass it, a mic denial is silent to the host UI (the SDK still doesn't crash and still emits microphone_permission_denied).

Build / tooling

  • Transitive-dependency version clash — the SDK pins livekit_client, flutter_webrtc, permission_handler, shared_preferences, path_provider, http, and lottie. A conflicting host major (especially lottie / permission_handler) fails the pub version solve; reconcile the range, don't fork.
  • Existing livekit_client / flutter_webrtc at a conflicting version → reconcile the pub range; do not vendor a second copy.
  • Web/desktop build target → unsupported (§2); guard the widget out of those flavors.

Runtime / lifecycle

  • Host overriding PlatformDispatcher.onError after embedInitialize → the host's handler wins; make sure it re-chains, or the LiveKit crash-guard (§21) is lost.
  • Sentry/Crashlytics console capture → the guard's informational [EmbedSDK] suppressed voice-stack async error line is expected, not a fault; don't page on it.

Security / privacy

  • disabledRoutes ≠ no capture of other screens — the durable UI-graph cache is written per screen while capture is enabled (dataCapture flag). Mark sensitive fields (obscureText for passwords/OTP — a hard guarantee) and ask the developer about regulated screens.
  • Store-listing disclosure — the app requests microphone (and Bluetooth) permissions; disclose in the listing.

23. Final checklist & report

[ ] Flutter/Dart version + navigation type detected     [ ] embed_flutter ^0.4.0 installed, pub get clean
[ ] Peers resolve to one copy each (livekit/webrtc/permission_handler/lottie)
[ ] Android: no manifest edits (SDK declares permissions), minSdk≥21, compileSdk 34+   [ ] APK builds
[ ] iOS: NSMicrophoneUsageDescription, Podfile PERMISSION_MICROPHONE=1 + platform 13.0, pod install   [ ] iOS builds
[ ] embedInitialize(apiKey:, appVersion:) once before runApp; fail-open; no "Embed init failed"
[ ] EmbedWidget outermost, wrapping MaterialApp; EmbedNavigatorObserver registered; EmbedRouteListener on tabs
[ ] USER_DATA sent after login; EmbedWidget.clearUserSession() wired into logout
[ ] Screens + continuity + disabledRoutes chosen by developer
[ ] FAB appears on chosen routes; null/delayed config handled; one widget instance
[ ] Navigation tested (A→B→A, nested tabs, modal); continuity as chosen
[ ] UI tree reachable via Semantics; secure fields redacted; identifiers on key controls
[ ] Click/text auto-capture verified (clickTracking on); rage_click fires
[ ] Call start/connect/end/re-start tested on a real device; mic denial handled
[ ] Error conditions (§20) verified; no host crash (§21)
[ ] §22 unknown-unknowns reviewed

Final report format

Integration Status: SUCCESS / PARTIAL / FAILED
SDK version: | Flutter: | Dart: | Navigation: | Android minSdk: | iOS target:
Dependencies added/changed (+why):
Native changes (Android manifest / iOS Info.plist + Podfile):
Init: embedInitialize placement + fail-open confirmed:
Widget config — routes: | continuity: | disabledRoutes: | buttonVisibility groups:
Validation — Init: | API key: | app_version: | USER_DATA: | Widget config: | FAB rendering: |
             UI tree/Semantics: | Event capture: | Call flow: | Android build: | iOS build: | Crash validation:
Unknown-unknowns found + handling:
Remaining issues (and whether app-side or SDK-side):

Do not declare SUCCESS while any critical validation is failing.


Companion docs: INTEGRATION.md (full mounting + navigator examples), HOST_INTEGRATION.md (action-intelligence / Semantics), PUBLIC_API.md (API reference), COMPATIBILITY.md (dependency matrix).