# Auth Source: https://docs.mentraglass.com/app-devs/core-concepts/auth Call your miniapp's backend as the signed-in Mentra user. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). If your miniapp has its own backend, `session.auth` lets you call it as the current Mentra user, with no login screen. The host hands your miniapp a signed token; your backend verifies it and gets the user's id. Two steps. ## Get started ### 1. Call your backend from the miniapp Use `session.auth.fetch` exactly like `fetch`. It attaches the user's token for you. ```typescript src/background/index.ts theme={null} const res = await session.auth.fetch("https://api.example.com/notes"); const notes = await res.json(); ``` ### 2. Verify the token on your backend Install [`@mentra/auth`](https://www.npmjs.com/package/@mentra/auth) and add its middleware. The verified user lands on the request. ```typescript theme={null} import { createMentraAuth, type MentraAuthVariables } from "@mentra/auth"; import { Hono } from "hono"; const mentraAuth = createMentraAuth({ packageName: "com.example.myapp" }); const app = new Hono<{ Variables: MentraAuthVariables }>(); app.use("/api/*", mentraAuth.hono()); app.get("/api/notes", (c) => { const { mentraUserId } = c.get("mentraAuth"); return c.json({ userId: mentraUserId }); }); ``` That's it. Every request to `/api/*` now arrives with a verified Mentra user id, and a token minted for another miniapp can't be used against yours (its audience is pinned to your package). ## On the client `session.auth.fetch(input, init)` is the easy path. If you bring your own HTTP client, grab the header or token instead: | Method | Returns | Description | | - | - | - | | `fetch(input, init?)` | `Promise` | Standard `fetch` with the `Authorization` header added. | | `getAuthHeader(options?)` | `Promise` | The string `"Bearer "`. | | `getToken(options?)` | `Promise` | The raw token. | | `onUpdate(handler)` | `() => void` | Fires when the host rotates the token. Returns an unsubscribe function. | ```typescript theme={null} const header = await session.auth.getAuthHeader(); await axios.get("https://api.example.com/notes", { headers: { Authorization: header } }); ``` The token rotates, so read it through `fetch`/`getAuthHeader`/`getToken` each time rather than caching the string. Those three take a `minTtlMs` option, the minimum lifetime the token must still have (default `30000`); if the current token has less left, the call waits for the next refresh. ```typescript theme={null} const res = await session.auth.fetch("https://api.example.com/sync", { minTtlMs: 120_000 }); ``` `session.auth.current` holds the active token and who it's for (or `null` before the host issues one). `session.auth.mentraUserId` and `session.auth.oemId` are getter shortcuts. | Field | Type | Description | | - | - | - | | `mentraUserId` | `string` | The Mentra user this token is for. | | `oemId` | `string \| undefined` | The OEM that provisioned the user, when present. | | `token` | `string` | The signed token sent to your backend. | | `expiresAt` | `number` | Expiry as a Unix timestamp in milliseconds. | ## On your backend `createMentraAuth({ packageName })` checks the signature against MentraOS's JWKS (only the miniapp token key, `mentra-miniapp-1`), the issuer, and the audience (your `packageName`), and rejects expired tokens. Other tokens MentraOS signs, which share that key set, never pass. `packageName` can also come from the `MENTRA_PACKAGE_NAME` environment variable. Not on Hono? Verify directly: ```typescript theme={null} await mentraAuth.verifyRequest(request); // a Web Request await mentraAuth.verifyAuthHeader(headerValue); // an Authorization header value await mentraAuth.verifyToken(token); // a bare token ``` Each resolves to a `VerifiedMentraAuth` (or throws `MentraAuthError`): | Field | Type | Description | | - | - | - | | `mentraUserId` | `string` | The Mentra user (the JWT `sub`). | | `oemId` | `string?` | The OEM that provisioned the user, when present. | | `email` | `string?` | The user's verified Mentra account email, when available. Absent for OEM accounts and unverified addresses. It can change, so store data against `mentraUserId`. | | `packageName` | `string` | Your miniapp package (the verified audience). | | `tokenId` | `string?` | The token's `jti`, if set. | | `expiresAt` | `number?` | `exp`, in Unix seconds. | | `issuedAt` | `number?` | `iat`, in Unix seconds. | | `claims` | `JWTPayload` | The raw verified claims. | ### Pointing at a non-production Core By default it verifies against production Core's JWKS (`https://core.mentraglass.com/.well-known/jwks.json`). For local, staging, or self-hosted Core, pass `jwksUrl` (or set `MENTRA_AUTH_JWKS_URL`): ```typescript theme={null} const mentraAuth = createMentraAuth({ packageName: "com.example.myapp", jwksUrl: "http://localhost:3000/.well-known/jwks.json", }); ``` # Blob Source: https://docs.mentraglass.com/app-devs/core-concepts/blob Persistent binary storage for files, images, and audio. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.blob` stores arbitrary bytes (PDFs, EPUBs, audio, video, model files, caches) as files on the phone. It's scoped to the user and your miniapp, so your keys never collide with another miniapp's, and the files survive restarts. Where [`session.storage`](/app-devs/core-concepts/storage) holds small strings, blob holds binary data. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp(async (session) => { await session.blob.set("logo", pngBytes, { mimeType: "image/png" }); const meta = await session.blob.get("logo"); // meta.uri -> "file://…", meta.bytes, meta.mimeType }); ``` Blob needs no permission and no hardware. Every method is async and round-trips to the phone. A stored blob's `uri` is a `file://` path in your miniapp's private storage, which you can hand to `session.speaker.play`, `session.blob.share`, or another host capability. ## Reading and writing `set` takes a `Uint8Array`, an `ArrayBuffer`, or a base64 string, and returns the stored blob's [metadata](#blobmeta). `get` returns that metadata (including the `uri`) or `null`. `bytes` reads the whole file into memory. ```typescript theme={null} await session.blob.set("note.pdf", pdfBytes, { mimeType: "application/pdf", name: "note.pdf", }); const meta = await session.blob.get("note.pdf"); // BlobMeta, or null if absent const raw = await session.blob.bytes("note.pdf"); // Uint8Array, or null if absent await session.blob.delete("note.pdf"); const exists = await session.blob.has("note.pdf"); // false ``` `bytes` throws once a blob crosses `BLOB_READ_ALL_MAX_BYTES` (32 MB). Read anything larger as a [stream](#large-files-streaming). ### Downloading and importing without crossing the bridge `setFromUrl` downloads a URL straight to disk host-side, and `importFile` opens the OS file picker and stores the chosen file. In both cases the bytes never pass through your JS context, which keeps large files off the bridge. ```typescript theme={null} // Host downloads the URL to disk. opts.headers can carry an auth token. const cover = await session.blob.setFromUrl( "cover", "https://example.com/cover.jpg", { mimeType: "image/jpeg" }, ); // OS file picker. Resolves to BlobMeta, or null if the user cancels. const picked = await session.blob.importFile({ mimeType: "audio/wav" }); if (picked) { await session.speaker.play({ audioUrl: picked.uri }); } ``` ### Usage and quota `usage` reports how much you've stored and the quota ceiling, all in bytes. ```typescript theme={null} const { bytes, count, quotaBytes } = await session.blob.usage(); ``` ## Large files: streaming For files past the in-memory caps, stream them. `createWriteStream` opens a `BlobWriter`: call `write` or `writeBase64` as many times as you need (each call auto-splits into bridge-safe chunks of `BLOB_WRITE_CHUNK_BYTES`, 1 MB raw), then `close` to publish or `abort` to discard the partial blob. ```typescript theme={null} const writer = await session.blob.createWriteStream("recording.pcm", { mimeType: "audio/L16", }); session.mic.onAudioChunk(async (b64) => { await writer.writeBase64(b64); // base64 straight from the mic }); // later, when capture ends: const meta = await writer.close({ source: "mic" }); // optional meta merges in ``` `createReadStream` opens a `BlobReader`. Call `read` for the next chunk until `done` is true, then `close`. ```typescript theme={null} const reader = await session.blob.createReadStream("recording.pcm"); for (;;) { const { bytes, done } = await reader.read(); // default chunk = BLOB_WRITE_CHUNK_BYTES process(bytes); if (done) break; } await reader.close(); ``` ### BlobWriter | Method | Returns | Notes | | - | - | - | | `write(chunk)` | `Promise` | Append a `Uint8Array`/`ArrayBuffer`. Auto-chunked. | | `writeBase64(b64)` | `Promise` | Append base64 bytes (e.g. from `mic.onAudioChunk`). | | `writeAt(offset, chunk)` | `Promise` | Overwrite within already-written bytes (a seek write). Advanced. | | `close(meta?)` | `Promise` | Finalize and return the metadata. `meta` merges into the record. | | `abort()` | `Promise` | Discard the partial blob. Idempotent. | | `key` | `string` | The key being written. | ### BlobReader | Method | Returns | Notes | | - | - | - | | `read(maxBytes?)` | `Promise<{ bytes: Uint8Array; done: boolean }>` | Read up to `maxBytes` (default one chunk). `done` is true at end of file. | | `close()` | `Promise` | Close the reader. | | `handle` | `string` | The reader's handle. | | `meta` | `BlobMeta` | Metadata for the blob being read. | ## Method reference | Method | Returns | Notes | | - | - | - | | `set(key, data, opts?)` | `Promise` | Store `Uint8Array`/`ArrayBuffer`/base64 bytes. | | `setFromUrl(key, url, opts?)` | `Promise` | Host downloads `url` to disk. `opts.headers` for auth. | | `importFile(opts?)` | `Promise` | OS file picker. `null` if cancelled. | | `createWriteStream(key, opts?)` | `Promise` | Streaming writer for large payloads. | | `get(key)` | `Promise` | Metadata (incl. `uri`), or `null` if absent. | | `stat(key)` | `Promise` | Alias of `get`. | | `has(key)` | `Promise` | Whether a blob is stored under `key`. | | `keys()` | `Promise` | Every key, newest first. | | `list()` | `Promise` | Every blob you own, newest first. | | `createReadStream(key)` | `Promise` | Streaming reader. | | `bytes(key)` | `Promise` | Read whole blob into memory. Throws past the cap. | | `usage()` | `Promise<{ bytes: number; count: number; quotaBytes: number }>` | Per-app usage and quota ceiling, in bytes. | | `delete(key)` | `Promise` | Delete the blob. No-op if absent. | | `clear()` | `Promise` | Delete every blob you own. | | `share(key)` | `Promise<{ success: boolean; cancelled?: boolean }>` | Hand the file to the OS share sheet (host shares from disk). | ## BlobMeta `get`, `stat`, `list`, and `close` resolve to a `BlobMeta`. | Field | Type | Description | | - | - | - | | `key` | `string` | The key it's stored under. | | `name` | `string` | Optional display/file name (e.g. the original filename for an import). | | `mimeType` | `string` | MIME type, e.g. `"application/pdf"`. | | `bytes` | `number` | Size on disk in bytes. | | `createdAt` | `number` | Epoch ms first written. | | `updatedAt` | `number` | Epoch ms last written. | | `md5` | `string` | Content md5 (lowercase hex). Skipped for very large blobs. | | `uri` | `string` | `file://` URI. Feed to `session.speaker.play`, `session.blob.share`, etc. | | `meta` | `Record` | App-defined metadata persisted with the blob. | ## Size limits | Constant | Value | Meaning | | - | - | - | | `BLOB_READ_ALL_MAX_BYTES` | 32 MB | Max `bytes()` buffers before throwing. Read larger blobs as a stream. | | `BLOB_WRITE_CHUNK_BYTES` | 1 MB | Raw bytes per chunked write, and the default `read()` chunk size. | ## Errors | Message | When | | - | - | | `BlobWriter is already closed/aborted` | Writing to a `BlobWriter` after `close()` or `abort()`. | | `BlobReader is closed` | Reading from a `BlobReader` after `close()`. | | `Blob "" exceeds the in-memory read cap` | `bytes(key)` read a blob past `BLOB_READ_ALL_MAX_BYTES`. Stream it with `createReadStream()` instead. | > [!NOTE] > > For small strings (counters, flags, serialized settings), use > [`session.storage`](/app-devs/core-concepts/storage) instead. To share or export > a file you didn't store as a blob, see `session.system`. # Photos Source: https://docs.mentraglass.com/app-devs/core-concepts/camera/photos Capture photos on camera glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.camera.takePhoto` captures a still on glasses that have a camera, like Mentra Live. It resolves after the photo is captured and uploaded, and hands you back a download URL. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { session.input.onButtonPress(async () => { if (!session.capabilities?.hasCamera) return; const photo = await session.camera.takePhoto(); session.display.render([{type: "image", id: "photo", box: {x: 144, y: 44, w: 288, h: 200}, data: photo.photoUrl}]); }); }); ``` Camera needs the `CAMERA` permission and glasses with a camera. Declare a `CAMERA` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest) so the call is allowed, and a `CAMERA` hardware requirement so the miniapp only installs on glasses that can capture. Check `session.capabilities.hasCamera` before calling so you don't fire a request the glasses can't satisfy. ## Take a photo `takePhoto` sends the request, waits while the glasses capture and the phone uploads, then resolves with the photo metadata. The returned `photoUrl` is a signed download URL with a short TTL (about 30 minutes), so fetch or display it soon after. ```typescript theme={null} const photo = await session.camera.takePhoto({ size: "high", sound: false }); // photo.photoUrl – signed URL, ~30 min TTL // photo.mimeType – e.g. "image/jpeg" // photo.size – bytes ``` All options are optional. The defaults are listed below. | Option | Type | Default | What it does | | - | - | - | - | | `size` | `"low" \| "medium" \| "high" \| "max"` | `"medium"` | Capture resolution. | | `transferMethod` | `"auto" \| "direct" \| "ble"` | `"auto"` | Use `"direct"` for webhook upload without BLE fallback, or `"ble"` to always relay through the phone. | | `compress` | `"none" \| "low" \| "medium" \| "high"` | `"none"` | JPEG compression applied before upload. | | `sound` | `boolean` | `true` | Play a shutter sound on the glasses. | | `saveToGallery` | `boolean` | `false` | Keep a copy in the glasses gallery. | | `exposureTimeNs` | `number` | auto | Manual shutter time in nanoseconds. Honored only on cameras with manual exposure; ignored otherwise. Omit to auto-expose. | Unknown runtime `transferMethod` values are rejected instead of being treated as `"auto"`. The result is a `PhotoTaken`: | Field | Type | Description | | - | - | - | | `requestId` | `string` | Correlates the request with ASG status and upload logs. | | `photoUrl` | `string` | Signed download URL for the captured image (\~30 min TTL). | | `mimeType` | `string` | MIME type of the image, e.g. `"image/jpeg"`. | | `size` | `number` | Image size in bytes. | ## Field of view `setFov` applies a temporary FOV/ROI override owned by the calling miniapp and resolves after the ASG client reports it was applied. It applies to all captures (photos and video) while that miniapp is active. When the miniapp closes, the host restores the previous live miniapp override or the persistent base setting. The factory/missing base is the centered `standard` preset (102 degrees); existing saved user values are preserved. ```typescript theme={null} // A named preset. The simplest option, always center ROI. await session.camera.setFov({ preset: "wide" }); // Or an explicit horizontal FOV with a region. await session.camera.setFov({ fov: 102, roiPosition: "bottom" }); ``` The argument is one of two shapes: | Field | Type | Notes | | - | - | - | | `preset` | `"narrow" \| "standard" \| "wide"` | Named preset. Uses center ROI. Mutually exclusive with `fov`/`roiPosition`. | | `fov` | `number` | Horizontal FOV in degrees. Pair with `roiPosition`. | | `roiPosition` | `"center" \| "bottom" \| "top"` | Region position for the `fov` form. Defaults to `"center"`. | It resolves with a `CameraFovResult`: | Field | Type | Description | | - | - | - | | `requestId` | `string` | Correlates the request with ASG logs. | | `fov` | `number` | The applied horizontal FOV in degrees. | | `roiPosition` | `"center" \| "bottom" \| "top"` | The applied region position. | | `timestamp` | `number` | When the setting was applied. | ## Errors Every method returns a promise that rejects with `{ code, message }` on failure. Wrap calls in `try`/`catch` and read `code`. | Code | When | | - | - | | `PERMISSION_NOT_DECLARED` | The miniapp didn't declare the `CAMERA` permission. | | `REQUEST_ABORTED` | The request timed out, or the session was torn down before it completed. | | `INTERNAL` | The phone-side handler threw. The glasses have no camera, capture or upload failed, or another device error occurred. Read `message`. | ```typescript theme={null} try { const photo = await session.camera.takePhoto(); session.display.render([{type: "image", id: "photo", box: {x: 144, y: 44, w: 288, h: 200}, data: photo.photoUrl}]); } catch (err) { if (err.code === "PERMISSION_NOT_DECLARED") { session.display.render([ {type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Camera permission not granted"}, ]); } } ``` > [!NOTE] > > For video, see [Videos](/app-devs/core-concepts/camera/videos). For live video > streaming, see [Streaming](/app-devs/core-concepts/stream). ### Breaking change: photo compression Photo compression now accepts exactly `none`, `low`, `medium`, and `high`, sent unchanged across the Mentra Miniapp SDK, Bluetooth SDKs, and glasses. The default is `none` everywhere. Glasses encode these as JPEG Q95, Q88, Q78, and Q60 respectively; `size` controls dimensions separately. `none` still uses lossy JPEG encoding. Unknown values are rejected. Cloud only allocates upload/download URLs: it takes no photo options and ignores any request body. Photo compression is handled by the phone and glasses. The former `heavy` spelling is removed; callers must use `high`. Android's native default changes from `medium` to `none`. Existing miniapps may therefore produce different JPEG quality or payload sizes. There are no compatibility aliases or wire translations. # Videos Source: https://docs.mentraglass.com/app-devs/core-concepts/camera/videos Record video on camera glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.camera` records video on glasses that have a camera, like Mentra Live. `startVideoRecording` returns a `recordingId`; pass it to `stopVideoRecording` to end the clip. ```typescript src/background/index.ts theme={null} const { recordingId } = await session.camera.startVideoRecording({ fps: 5, save: true }); // … later … await session.camera.stopVideoRecording(recordingId); ``` Camera needs the `CAMERA` permission and glasses with a camera. Declare a `CAMERA` permission and a `CAMERA` hardware requirement in your [manifest](/app-devs/core-concepts/miniapp-manifest), and check `session.capabilities.hasCamera` before calling. ## Record `startVideoRecording` resolves once the glasses report that recording started. Resolution and frame rate are optional. Omit them to use the device's saved button-video settings. A lower `fps` (for example `5`) keeps the glasses cooler and the file smaller on long recordings. | Option | Type | Default | What it does | | - | - | - | - | | `width` | `number` | device default | Video width in pixels. | | `height` | `number` | device default | Video height in pixels. | | `fps` | `number` | device default | Frames per second. | | `sound` | `boolean` | `true` | Play start/stop capture sounds. | | `save` | `boolean` | `false` | Keep the recording on the glasses after it finishes. | It resolves with a `VideoRecordingStarted`: | Field | Type | Description | | - | - | - | | `recordingId` | `string` | Identifier for this recording. Pass it to `stopVideoRecording`. | ## Upload the recording `stopVideoRecording(recordingId)` resolves to `void`. By default the clip stays on the glasses. To send it somewhere, pass an `uploadUrl` and the glasses upload the finished recording there with a multipart POST (the file is the `video` part): ```typescript theme={null} await session.camera.stopVideoRecording(recordingId, { uploadUrl: "https://your-server.example/clips", uploadAuthToken: "optional-bearer-token", // sent as the Authorization header if set }); ``` | Option | Type | What it does | | - | - | - | | `uploadUrl` | `string` | Endpoint the finished recording is POSTed to. Omit to keep it on the glasses. | | `uploadAuthToken` | `string` | Bearer token for the upload's `Authorization` header, if your endpoint needs one. | Unlike a photo, the recording bytes don't come back over the bridge: with an `uploadUrl` they go straight from the glasses to your endpoint, otherwise they stay on the device. ## Errors Every method returns a promise that rejects with `{ code, message }` on failure. Wrap calls in `try`/`catch` and read `code`. | Code | When | | - | - | | `PERMISSION_NOT_DECLARED` | The miniapp didn't declare the `CAMERA` permission. | | `REQUEST_ABORTED` | The request timed out, or the session was torn down before it completed. | | `INTERNAL` | The phone-side handler threw. The glasses have no camera, or another device error occurred. Read `message`. | > [!NOTE] > > For stills, see [Photos](/app-devs/core-concepts/camera/photos), which also covers > [field of view](/app-devs/core-concepts/camera/photos#field-of-view). For live > video streaming, see [Streaming](/app-devs/core-concepts/stream). # Cloud status Source: https://docs.mentraglass.com/app-devs/core-concepts/cloud Read whether the phone is connected to MentraOS cloud. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.cloud` reports whether the phone currently has a connection to MentraOS cloud. The phone owns that connection. Your miniapp does not open, configure, or close it. This module only observes and reports the current state, so you can render honest UX (an online/offline indicator, a "reconnecting" hint). ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { if (session.cloud.connected) { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Online"}]); } session.cloud.onStatusChanged((status) => { // status.status: "connected" | "connecting" | "reconnecting" | "disconnected" // status.audioTransport: "udp" | "ws" | "offline" | "none" }); }); ``` This module needs no permission. It's read-only. ## Reading status `status` returns the latest `CloudClientStatus` snapshot. `connected` is `true` only when `status.status` is `"connected"`. `isConnected()` is a method alias for `connected`. ```typescript theme={null} const snapshot = session.cloud.status; // { status, audioTransport } const online = session.cloud.connected; // boolean const same = session.cloud.isConnected(); // same value, function-style ``` | Member | Returns | Notes | | - | - | - | | `status` | `CloudClientStatus` | Latest snapshot pushed by the phone. | | `connected` | `boolean` | `true` when `status.status === "connected"`. | | `isConnected()` | `boolean` | Method form of `connected`. | | `onStatusChanged(handler)` | `UnsubscribeFn` | Subscribe to changes. Call the returned function to unsubscribe. | ## Watching for changes `onStatusChanged` delivers the current value to your handler immediately, then calls it again on every change. You don't need a separate read to hydrate UI. ```typescript theme={null} const stop = session.cloud.onStatusChanged((status) => { render(status.status, status.audioTransport); }); stop(); // unsubscribe ``` ## CloudClientStatus ```typescript theme={null} interface CloudClientStatus { status: "connected" | "connecting" | "reconnecting" | "disconnected"; audioTransport: "udp" | "ws" | "offline" | "none"; } ``` `status` is the connection state: | Value | Meaning | | - | - | | `connected` | The cloud-client handshake is complete. | | `connecting` | A first connection is in progress. | | `reconnecting` | A dropped connection is being re-established. | | `disconnected` | No cloud connection. | `audioTransport` is how audio reaches the cloud when a connection exists: | Value | Meaning | | - | - | | `udp` | Audio over UDP. | | `ws` | Audio over WebSocket. | | `offline` | Connected, but no audio transport available. | | `none` | No audio transport (the default before any status arrives). | > [!NOTE] > > Some capabilities depend on the cloud connection. > [`session.transcription`](/app-devs/core-concepts/microphone/speech-to-text), > [`session.translation`](/app-devs/core-concepts/translation), and cloud > text-to-speech on [`session.speaker`](/app-devs/core-concepts/speakers/text-to-speech) > degrade or stop while `status.status` is not `"connected"`. Use `session.cloud` to > tell the user why, not to control the connection. # Display Source: https://docs.mentraglass.com/app-devs/core-concepts/display/layouts Render scenes of positioned text, images, and shapes on the glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.display.render()` draws to the glasses HUD. Each call describes the whole frame: everything that should be on screen right now. The phone diffs it against the previous frame, so elements with a stable `id` update in place (no flicker), anything you leave out of the new frame comes off the screen, and `render([])` clears it. Your job is to describe the frame; the phone works out what changed. ```typescript theme={null} session.display.render([ {type: "text", id: "stats", box: {x: 12, y: 9, w: 200, h: 40}, text: "863 m · 11 min"}, {type: "image", id: "map", box: {x: 335, y: 14, w: 150, h: 150}, data: mapPng}, {type: "rect", id: "frame", box: {x: 4, y: 4, w: 490, h: 210}, style: {border: 2, radius: 6}}, ]); ``` Display needs no permission, but the glasses do need a screen. Declare a `DISPLAY` hardware requirement in your [manifest](/app-devs/core-concepts/miniapp-manifest) so the miniapp only installs on glasses that can show anything. ## Elements | Element | Fields | Shows | | - | - | - | | `{type: "text", box, text, style?, id?}` | `style`: `border`, `radius`, `overflow`, `breakMode`, `maxLines`, `textWindow`, `verticalAlign` | Wrapped text inside the box. | | `{type: "image", box, data, id?}` | `data`: base64 PNG/JPEG | A bitmap, converted to the glasses' native format by the phone. | | `{type: "rect", box, style?, id?}` | `style`: `border`, `radius` | An empty bordered box. | ## Screen sizes `box` is `{x, y, w, h}` in raw device pixels. Coordinates map 1:1 to the device canvas, with the origin at the top-left: | Glasses | Canvas | Positioned elements | | - | - | - | | Even Realities G2 | 576 × 288 | Yes | | NIMO | 500 × 220 | Yes | | Even Realities G1 | 640 × 200 panel | Text fallback | | Vuzix Z100 | 640 × 480 panel | Text fallback | **We recommend designing your layout for 500 × 220.** A frame that fits 500 × 220 renders as-is on every positioning device; on the G2's larger canvas it sits in the top-left with some margin. Boxes that run past the edge are clamped, and elements past the device's budget are dropped tail-first. Both show up in the render result, so you find out. When you want an exact fit instead (a full-screen caption, an element pinned to the bottom edge), compute boxes from the connected device's real canvas. `session.capabilities.display` carries it, populated on the `"ready"` event: ```typescript theme={null} const d = session.capabilities?.display; session.display.render([ {type: "text", id: "caption", box: {x: 0, y: 0, w: d?.width ?? 576, h: d?.height ?? 288}, text}, ]); ``` ## Older glasses Even Realities G1 and Vuzix Z100 show text but can't place elements at coordinates, and their displays can't show bitmaps. Render the same scene anyway: the phone converts it for them. Text elements collapse into a single full-view text layout in reading order (top to bottom, then left to right), and image and rect elements are dropped. The result tells you what happened (`degraded: true`, plus the dropped ids), and `session.capabilities.display.canPosition` tells you up front which kind of device you're on. In practice a text-first miniapp works everywhere with zero branching: a caption app's single full-canvas text element reads the same on every device. If your layout leans on positioning or images, like a HUD with a minimap, check `canPosition` and render a text-only variant for these devices. ## Updating and clearing Give any element that changes over time a stable `id`. Re-rendering with the same id updates that element in place on the glasses, and unchanged elements never re-cross Bluetooth, so pushing the whole frame on every change is cheap. To take something off screen, leave it out of the next frame. To clear everything: ```typescript theme={null} session.display.render([]); ``` ## Options and results `render(elements, options?)` takes `{view, durationMs}` — `view` picks `"main"` (default) or `"dashboard"`, and `durationMs` auto-clears the frame after that many milliseconds. Awaiting is opt-in. A plain fire-and-forget call is fine, and the returned promise never rejects: ```typescript theme={null} const result = await session.display.render(elements); // result.status — "displayed" (sent to the device) or "blocked" (another // app owns the display; result.reason says why) // result.degraded — true when the phone adjusted the scene for the device // result.dropped — ids of elements the phone had to drop ``` ## Just showing text The simplest useful frame is one text element covering the whole canvas — a status line, a caption, a "Saved" confirmation: ```typescript theme={null} const d = session.capabilities?.display; session.display.render([ {type: "text", id: "message", box: {x: 0, y: 0, w: d?.width ?? 576, h: d?.height ?? 288}, text: "Saved"}, ], {durationMs: 2000}); ``` Per-element `style.breakMode` (`"character" | "character-no-hyphen" | "word" | "strict-word"`) controls how the text wraps inside its box. ## Fitting text The host wraps text using the connected glasses' font metrics. The box is the available area; optional `style.maxLines` imposes an additional line limit. Leave it out to use the box's capacity. Miniapps do not need device font tables. `style.textWindow` chooses which end stays visible when text overflows: `"start"` (default) keeps the beginning; `"end"` keeps the newest content. `style.overflow` remains `"clip"` (default) or `"ellipsis"`. An ellipsis appears on the omitted side. These controls require Mentra App 3.2.1 or newer. ```typescript theme={null} session.display.render([{ type: "text", id: "caption", box: captionArea, text: recentTranscript, style: {maxLines: 3, textWindow: "end", breakMode: "word"}, }]); ``` Each call still supplies the complete content. The host does not append to history or accumulate text behind an element id. Optional `style.verticalAlign: "top" | "bottom"` places the line region inside its box. With `maxLines`, that region reserves the selected number of rows so bottom captions don't jump as partial transcripts grow. The outer box is always a hard bound. On devices without calibrated text positioning, placement falls back to the top and the result reports degradation for a bottom request. G1/Z100 use their text-only row budgets; position and images degrade as above. ### Text feedback Pass `{includeTextLayout: true}` as the second render argument when a preview or line-based progression needs the actual host-computed text: ```typescript theme={null} const result = await session.display.render(elements, {includeTextLayout: true}); const caption = result.textLayout?.caption; // caption.lines: visible {text, start, end} entries // caption.lineStarts: source starts of all wrapped lines // caption.capacity: effective line limit; caption.truncated: whether text was omitted ``` Offsets use JavaScript UTF-16 string indices into that element's supplied text; `end` is exclusive. Source spans describe wrapped lines before an ellipsis is applied. Inserted hyphens and ellipses do not consume source characters. `lineStarts` lets a teleprompter replace its text at a host-computed line boundary without measuring fonts. Feedback is keyed by the element id, or `text[index]` for unnamed text. Dropped elements have no feedback. It describes the frame the host compiled, not a hardware acknowledgment. Ignore older responses when a newer frame has been submitted. Waiting for feedback is optional. ## Dashboard view `render(elements, {view: "dashboard"})` targets the dashboard view, the glasses' status view that the user brings up with a glasses-specific gesture (head-up on G1, a double-tap on others). The default is `"main"`. # Glasses Source: https://docs.mentraglass.com/app-devs/core-concepts/glasses Battery level and connection state of the glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.glasses` reports on the glasses hardware: battery level plus charging state, and whether the glasses are connected. Both are subscriptions, so you get a callback whenever the value changes. ```typescript src/background/index.ts theme={null} session.glasses.onBattery(({ level, charging }) => { if (level <= 15 && !charging) { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Glasses battery low"}]); } }); ``` There's no synchronous getter for the current value. You learn the state by subscribing and waiting for the first event. ## Battery `onBattery` fires when the glasses report a new battery reading. ```typescript theme={null} session.glasses.onBattery((data) => { console.log(`${data.level}%`, data.charging ? "charging" : "on battery"); }); ``` Each `BatteryData` has: | Field | Type | Notes | | - | - | - | | `level` | `number` | Battery percentage, 0 to 100. | | `charging` | `boolean` | Whether the glasses are charging. | ## Connection `onConnection` fires when the glasses connect or disconnect. ```typescript theme={null} session.glasses.onConnection((data) => { if (!data.connected) { // The glasses dropped. Display calls won't reach the HUD until they return. } }); ``` Each `ConnectionData` has: | Field | Type | Notes | | - | - | - | | `connected` | `boolean` | Whether the glasses are currently connected. | | `modelName` | `string?` | Glasses model name, when reported. May be absent. | `modelName` is optional, so guard for it before reading: ```typescript theme={null} session.glasses.onConnection(({ connected, modelName }) => { if (connected && modelName) { console.log(`Connected to ${modelName}`); } }); ``` ## Cleaning up Both methods return an unsubscribe function: ```typescript theme={null} const off = session.glasses.onBattery(handler); off(); // stop listening ``` > [!NOTE] > > `session.glasses` covers the glasses hardware. The phone has its own battery and > connection events on [`session.phone`](/app-devs/core-concepts/phone). For what > the connected glasses can *do* (display, camera, sensors), read > `session.capabilities` once the session is [ready](/app-devs/core-concepts/session#lifecycle). # Heading Source: https://docs.mentraglass.com/app-devs/core-concepts/heading The phone's compass heading, as a continuous stream of degrees. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.heading` reports the phone's compass heading. Subscribe with `onUpdate()` to get a `HeadingData` every time the heading changes. ```typescript src/background/index.ts theme={null} const off = session.heading.onUpdate((data) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: `${Math.round(data.degrees)}°`}]); }); ``` Heading needs the `LOCATION` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest), the same key [`session.location`](/app-devs/core-concepts/location) uses. ## Continuous updates `onUpdate()` delivers a `HeadingData` on each heading change. It returns an unsubscribe function. ```typescript theme={null} const off = session.heading.onUpdate((data) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: `${Math.round(data.degrees)}°`}]); }); off(); // stop receiving updates ``` ## HeadingData | Field | Type | Notes | | - | - | - | | `degrees` | `number` | Compass heading in degrees. `0` = north, `90` = east. | ## Permission `session.heading.hasPermission` is `true` when `LOCATION` is declared in your manifest. It reports the manifest declaration, not the OS grant: if the user denied the system location prompt, `onUpdate()` produces no reading. See [Permissions](/app-devs/core-concepts/permissions). > [!NOTE] > > `session.heading` works on both Android and iOS. It shares the `LOCATION` manifest > key with [`session.location`](/app-devs/core-concepts/location). # IMU Source: https://docs.mentraglass.com/app-devs/core-concepts/imu Head-position events from the glasses inertial measurement unit. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.imu` reports which way the wearer is looking. The glasses' inertial measurement unit resolves head pose into two states, `"up"` and `"down"`, and fires an event each time that state changes. Use it to wake content when someone glances up, or to dismiss it when they look back down. This module needs the `IMU` hardware requirement in your [manifest](/app-devs/core-concepts/miniapp-manifest). Mark it `REQUIRED` and glasses without an IMU won't list your miniapp; mark it `OPTIONAL` and they still run it without head-position events. There is no runtime permission to request. ## Head position ```typescript src/background/index.ts theme={null} const off = session.imu.onHeadPosition(({ position }) => { if (position === "up") session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Welcome back"}]); else session.display.render([]); }); ``` Each `HeadPositionData` has: | Field | Type | Notes | | - | - | - | | `position` | `"up" \| "down"` | The new head pose. Fires on each transition, not continuously. | ## Raw accelerometer `onAccel` streams raw accelerometer readings, `{x, y, z}` in g plus a timestamp. This is G2-only today. On glasses without an exposed IMU stream the subscription succeeds and no events arrive. ```typescript src/background/index.ts theme={null} const off = session.imu.onAccel(({ x, y, z, timestamp }) => { // x, y, z in g; timestamp is Unix ms. }); ``` Each `AccelData` has: | Field | Type | Notes | | - | - | - | | `x`, `y`, `z` | `number` | Accelerometer axes in g (gravity-normalized). | | `timestamp` | `number` | Unix ms timestamp of the reading. | Subscribing turns the sensor on and unsubscribing turns it off, so most callers never touch it directly. `session.imu.setEnabled(true)` drives the sensor by hand when you want to, for instance, a diagnostic toggle. It's a no-op on glasses without an exposed IMU stream. Gyroscope, magnetometer, and fused orientation aren't exposed yet. ## Cleaning up Both methods return an unsubscribe function: ```typescript theme={null} const off = session.imu.onHeadPosition(handler); off(); // stop listening ``` Subscriptions are ref-counted, so the SDK only talks to the phone on the first subscribe and last unsubscribe of a stream. # Input Source: https://docs.mentraglass.com/app-devs/core-concepts/input Glasses button presses and touchpad gestures. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.input` delivers the physical controls on the glasses: button presses and touchpad gestures. The starter scaffold wires a button press to the display, so a short press shows a text wall. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { session.input.onButtonPress((press) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: `${press.buttonId}: ${press.pressType}`}]); }); }); ``` Button events only reach glasses that have a button. Declare a `BUTTON` hardware requirement in your [manifest](/app-devs/core-concepts/miniapp-manifest) to gate the miniapp to those glasses. ## Button presses `onButtonPress` fires on every press of a physical button: ```typescript theme={null} session.input.onButtonPress((press) => { if (press.pressType === "long") session.display.render([]); }); ``` Each `ButtonPressData` has: | Field | Type | Notes | | - | - | - | | `buttonId` | `string` | Which button fired. | | `pressType` | `"short" \| "long"` | Press duration. | ## Touch gestures `onTouch` subscribes to touchpad gestures. Pass a handler for every gesture, a single gesture name to filter, or an array of names for one subscription across several gestures. ```typescript theme={null} // Every touch gesture. session.input.onTouch((touch) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: touch.kind}]); }); // One gesture. session.input.onTouch("single_tap", () => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Tap"}]); }); // Several gestures, one subscription. session.input.onTouch(["swipe_up", "swipe_down"], (touch) => { // touch.kind tells you which one. }); ``` Each `TouchData` has: | Field | Type | Notes | | - | - | - | | `kind` | `string` | The gesture. See the values below. | The gesture values the runtime reports: | `kind` | Gesture | | - | - | | `"single_tap"` | A single tap on the touchpad. | | `"double_tap"` | Two quick taps. | | `"triple_tap"` | Three quick taps. | | `"long_press"` | A press and hold. | | `"swipe_up"` | A swipe up. | | `"swipe_down"` | A swipe down. | Glasses can send other `kind` strings, so `kind` is typed as `string`. Match the values you handle and ignore the rest. ## Cleaning up `onButtonPress` and every `onTouch` form return an unsubscribe function: ```typescript theme={null} const off = session.input.onTouch("single_tap", handler); off(); // stop listening ``` Calling `onTouch` with an array returns one unsubscribe function that drops all of the gestures it registered. # LED Source: https://docs.mentraglass.com/app-devs/core-concepts/led/overview Drive the indicator LED on glasses that have one. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.led` controls the indicator LED on supported glasses. You give it a color and a blink pattern, and it tells the LED to turn on or off. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { session.transcription.on(async () => { // Blink green twice when speech comes in. await session.led.blink("green", 200, 200, 2); }); }); ``` The LED needs glasses with a `LIGHT` hardware capability. Declare it in your [manifest](/app-devs/core-concepts/miniapp-manifest) so the miniapp only installs on glasses that have an LED. There's no runtime permission to request. ## Methods ```typescript theme={null} // Turn the LED on with an explicit pattern. await session.led.turnOn({ color: "blue", ontime: 500, offtime: 500, count: 3 }); // Blink: color, on-ms, off-ms, repeat count. await session.led.blink("red", 200, 200, 4); // Solid: hold one color for a duration in ms. await session.led.solid("white", 2000); // Off. await session.led.turnOff(); ``` | Method | Signature | What it does | | - | - | - | | `turnOn` | `turnOn(options?: LedControlOptions)` | Turns the LED on with the given color and pattern. | | `turnOff` | `turnOff()` | Turns all LEDs off. | | `blink` | `blink(color, ontime, offtime, count)` | Repeats an on/off cycle `count` times. Wraps `turnOn`. | | `solid` | `solid(color, duration)` | Holds one color for `duration` ms (one cycle, no off time). Wraps `turnOn`. | `LedControlOptions` is the shape `turnOn` accepts: | Field | Type | Default | What it does | | - | - | - | - | | `color` | `LedColor` | `"red"` | Which color to light. | | `ontime` | `number` | `1000` | On duration per cycle, in ms. | | `offtime` | `number` | `0` | Off duration per cycle, in ms. | | `count` | `number` | `1` | How many on/off cycles to run. | ## Colors `LedColor` is a fixed set of names. The phone maps each name to the per-device LED index, so the same name can light different physical LEDs across glasses. | Value | | - | | `"red"` | | `"green"` | | `"blue"` | | `"orange"` | | `"white"` | > [!NOTE] > > Every method returns a `Promise` that resolves once the command is dispatched to > the glasses, not when the blink pattern finishes playing. A `blink("red", 200, > 200, 4)` call resolves right after the command goes out, well before the LED has > run all four cycles. Don't `await` the call as a way to wait for the LED to go > dark; if you need timing, track it yourself. # Location Source: https://docs.mentraglass.com/app-devs/core-concepts/location The user's GPS position, as a one-shot fix or a continuous stream. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.location` reports the phone's GPS position. Get a single fix with `getOnce()`, or subscribe to a stream with `onUpdate()`. ```typescript src/background/index.ts theme={null} const fix = await session.location.getOnce(); session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: `${fix.lat.toFixed(4)}, ${fix.lng.toFixed(4)}`}]); ``` Location needs the `LOCATION` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest). Without it, the phone rejects both paths with `PERMISSION_NOT_DECLARED` and no reading arrives. ## A single fix `getOnce()` resolves with the next available reading from the phone. Use it at app load to seed your UI before a stream starts. ```typescript theme={null} const fix = await session.location.getOnce(); ``` It returns a `Promise`. ## Continuous updates `onUpdate()` delivers a `LocationData` every time the position changes. It returns an unsubscribe function. ```typescript theme={null} const off = session.location.onUpdate((data) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: `${data.lat}, ${data.lng}`}]); }); off(); // stop receiving updates ``` ## LocationData Both paths hand you the same shape: | Field | Type | Notes | | - | - | - | | `lat` | `number` | Latitude in degrees. | | `lng` | `number` | Longitude in degrees. | | `accuracy` | `number?` | Accuracy in meters, when the platform reports it. | | `timestamp` | `number?` | Unix ms timestamp of the fix. | | `correlationId` | `string?` | Set when the event answers a single-location request. | ## Permission `session.location.hasPermission` is `true` when `LOCATION` is declared in your manifest. It reports the manifest declaration, not the OS grant: if the user denied the system location prompt, `getOnce()` and `onUpdate()` produce no reading. See [Permissions](/app-devs/core-concepts/permissions). > [!NOTE] > > `LOCATION` and `BACKGROUND_LOCATION` are separate manifest keys, and > `hasPermission` checks for `LOCATION` only. For turn-by-turn directions built on > position, see [`session.navigation`](/app-devs/core-concepts/session). # Microphone Source: https://docs.mentraglass.com/app-devs/core-concepts/microphone/audio-chunks Raw audio frames and voice-activity detection from the glasses mic. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.mic` is the low-level audio input: raw frames and a voice-activity signal. If you want text, use [`session.transcription`](/app-devs/core-concepts/microphone/speech-to-text) instead. Reach for `mic` when you need the audio itself, like recording a clip or running your own audio processing. Both subscriptions need the `MICROPHONE` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest), or the phone rejects them with `PERMISSION_NOT_DECLARED`. ## Raw audio ```typescript theme={null} session.mic.onAudioChunk((chunk) => { // chunk.data is base64-encoded PCM or LC3. buffer.push(chunk.data); }); ``` Each `AudioChunkData` has: | Field | Type | Notes | | - | - | - | | `data` | `string` | Base64-encoded audio. PCM or LC3 depending on the phone's mic mode. | | `sampleRate` | `number?` | Sample rate, when reported. | | `format` | `string?` | The encoding, when reported. | ## Voice activity `onVoiceActivity` fires when the user starts and stops speaking, without giving you the audio: ```typescript theme={null} session.mic.onVoiceActivity(({ status }) => { // status: true while speaking, false when silent. if (status) session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Listening…"}]); }); ``` ## Cleaning up Both methods return an unsubscribe function, and `stop()` tears down everything this module owns: ```typescript theme={null} const off = session.mic.onAudioChunk(handler); off(); // drop this one session.mic.stop(); // drop all of them ``` `session.mic.hasPermission` reports whether `MICROPHONE` is declared in your manifest, not whether the user granted the OS prompt. If they denied it, the subscription succeeds but no chunks arrive. > [!NOTE] > > `mic`, `transcription`, and `translation` are independent. Subscribing to raw > audio does not give you transcripts, and vice versa. Audio *output* (playback and > text-to-speech) lives on [`session.speaker`](/app-devs/core-concepts/speakers/text-to-speech). # Transcription Source: https://docs.mentraglass.com/app-devs/core-concepts/microphone/speech-to-text Real-time speech-to-text from the glasses microphone. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.transcription` turns what the user says into text. Register a handler and you start getting results; the SDK handles the subscription for you. ```typescript theme={null} session.transcription.on((data) => { if (data.isFinal) { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: data.text}]); } }); ``` Transcription needs the `MICROPHONE` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest). Without it, the phone rejects the subscription with `PERMISSION_NOT_DECLARED` and no events arrive. ## Results Each event is a `TranscriptionData`: | Field | Type | Notes | | - | - | - | | `text` | `string` | The transcribed text so far. | | `isFinal` | `boolean` | `false` for interim results that keep updating, `true` once the utterance is settled. | | `language` | `string?` | The detected language (when auto-detecting). | Interim results stream as the user speaks and get replaced; wait for `isFinal` if you only want settled text. ## Targeting languages `on()` auto-detects the language. To listen for specific ones, use `forLanguage()` with BCP-47 tags: ```typescript theme={null} // One language. session.transcription.forLanguage("en-US", (data) => { /* … */ }); // Several at once. session.transcription.forLanguage(["en-US", "es-ES"], (data) => { /* … */ }); ``` Each call is independent, and you can have several active at the same time. ## Requiring on-device transcription Pass `{forceLocal: true}` as the final argument to route that subscription through on-device STT and suppress cloud transcript delivery: ```typescript theme={null} session.transcription.on(handler, {forceLocal: true}); session.transcription.forLanguage("en-US", handler, {forceLocal: true}); ``` This requires a downloaded local STT model. Cloud transcripts are suppressed for the local-only subscription, even while other miniapps continue using cloud transcription. Routing is per listener, so a default listener on the same stream can continue receiving cloud transcription. ## Configuring `configure()` tunes the recognizer. It's fire-and-forget and applies to subsequent results: ```typescript theme={null} session.transcription.configure({ languageHints: ["en", "ja"], // ISO 639-1 hints to improve detection vocabulary: ["MentraOS", "HIPAA"], // boost domain terms diarization: true, // tag speakers when the provider supports it }); ``` ## Cleaning up Every subscription returns an unsubscribe function. `stop()` tears down every subscription this module owns at once: ```typescript theme={null} const off = session.transcription.on(handler); off(); // drop this one session.transcription.stop(); // drop all of them ``` `session.transcription.hasPermission` tells you whether `MICROPHONE` is declared in your manifest. It does not tell you whether the user granted the OS prompt: if they denied it, your handler simply never fires. See [Permissions](/app-devs/core-concepts/permissions). > [!NOTE] > > Transcription is for text. For raw audio frames or voice-activity detection, use > [`session.mic`](/app-devs/core-concepts/microphone/audio-chunks). For translated > text, use [`session.translation`](/app-devs/core-concepts/translation). Subscribing > to one does not subscribe you to the others. # Actions Source: https://docs.mentraglass.com/app-devs/core-concepts/miniapp-interop Expose typed actions from your miniapp so Mentra AI can call them. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). Your miniapp can expose **actions**: typed, described capabilities that Mentra AI can call to drive it. You declare each action in `miniapp.json` and handle it in your background layer. Mentra AI reads your declarations and decides when to call them. Actions map 1:1 onto [MCP](https://modelcontextprotocol.io) tools (`id` to `name`, `description`, `parameters` to `inputSchema`, and `outputSchema`), so the same declarations can be surfaced to external agents later. ## Declare an action List your actions in `miniapp.json`. The `description` is the contract Mentra AI reads, so say *when* the action should be used. ```json miniapp.json theme={null} { "actions": [ { "id": "add_todo", "description": "Add an item to the user's todo list. Use when the user asks to remember, note, or add something.", "parameters": { "type": "object", "properties": { "text": { "type": "string", "description": "The todo item text" } }, "required": ["text"] }, "outputSchema": { "type": "object", "properties": { "ok": { "type": "boolean" }, "count": { "type": "number" } }, "required": ["ok", "count"] } } ] } ``` | Field | Rules | | - | - | | `id` | `^[a-z][a-z0-9_]*$`, unique within your miniapp, ≤64 chars. | | `description` | Required, non-empty. The AI-facing contract: say when to use it. | | `parameters` | A JSON-Schema object (the MCP subset): top-level `type: "object"`, `properties` typed `string` / `number` / `boolean` / `array` (of primitives), plus `enum`, `description`, `items`, and a top-level `required` array. Use `number`, not `integer`. | | `outputSchema` | Optional JSON Schema describing the handler's structured return value. Declaring it lets result-aware agents interpret the result reliably instead of guessing from arbitrary JSON. | | `lifecycle` | Optional. `"persistent"` (default) starts your miniapp as normal activity that keeps running after the call. `"transient"` wakes it invisibly for the call only: it does not show as running and stops when the call settles, unless the user opens it meanwhile. | | `audience` | Optional. `"system"` (default) lets system miniapps such as Mentra AI find and call the action. `"host"` reserves it for the Mentra App itself; it is never listed to miniapps. | ## Handle it Register a handler in your background layer with `session.actions.handle`. Its return value goes back to the caller. ```ts background/index.ts theme={null} session.actions.handle("add_todo", async (params, ctx) => { // ctx.callerPackageName is host-stamped, so it's trustworthy. await todos.add(String(params.text)); return { ok: true, count: todos.size }; }); ``` * One handler per `id`. Registering the same `id` twice throws. * A thrown handler rejects the caller, and your error message comes back to them. * The return value is serialized to the caller (max 256 KB). * Keep results structured and consistent with `outputSchema`. Do not return logs, secrets, access tokens, or instructions intended for the agent. * `handle` returns a function that deregisters the handler. Mentra AI executes actions before producing its final answer. It sends the structured result through a result-finalization step, which turns values such as `{ "ok": true, "count": 3 }` into concise user-facing language. A thrown error or a result that explicitly reports failure produces a failure answer instead of an optimistic success message. ## How a call reaches you When Mentra AI invokes one of your actions, the host **headless-wakes** your miniapp if it isn't already running: it spawns your background context (no UI, no foreground change, so whatever the user is doing is undisturbed), waits for your handlers to register, delivers the call, and returns your handler's result. Your miniapp keeps running afterward until it's stopped, unless the action is `"transient"`, in which case the host stops the context it woke once the call settles. Because an inbound call waits only briefly for a just-woken miniapp to register, call `session.actions.handle` at the top of your [`registerMiniapp`](/app-devs/core-concepts/two-layer-architecture#the-background-layer) handler rather than after async setup. > [!NOTE] > > Exposing actions with `handle` is open to every miniapp, so your miniapp is > callable by Mentra AI today. The *calling* side (discovering, launching, and > invoking other miniapps) is restricted to system miniapps. See > [System Miniapp APIs](/app-devs/reference/system-apis). # The manifest (miniapp.json) Source: https://docs.mentraglass.com/app-devs/core-concepts/miniapp-manifest Declare your miniapp's identity, entry points, permissions, and hardware requirements. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). Every miniapp has a `miniapp.json` at its root. It declares who the miniapp is, where its two layers are built to, what permissions it needs, and what hardware it requires. The CLI validates it on every `dev`, `pack`, and `release`: bad values fail on your laptop, not on the phone. ```json miniapp.json theme={null} { "$schema": "./node_modules/@mentra/miniapp-cli/schema/miniapp.schema.json", "packageName": "com.example.notes", "version": "1.0.0", "name": "Notes", "description": "Hands-free note taking", "icon": "icon.png", "type": "standard", "sdkVersion": "0.3.0", "minHostVersion": "1.42.0", "entry": { "background": "background/index.js", "ui": "ui/index.html" }, "permissions": [ { "type": "MICROPHONE", "description": "Capture notes you dictate." } ], "hardwareRequirements": [ { "type": "DISPLAY", "level": "REQUIRED", "description": "Shows your notes." }, { "type": "MICROPHONE", "level": "REQUIRED" } ] } ``` ## Fields **Required:** `packageName`, `version`, `name`, `hardwareRequirements`. Everything else is optional. | Field | Type | Notes | | - | - | - | | `packageName` | string | Reverse-DNS, e.g. `com.example.notes`. Must match `^[a-zA-Z][a-zA-Z0-9_]*(\.[a-zA-Z][a-zA-Z0-9_]*)+$`. Your miniapp's unique identity. | | `version` | string | Semver, e.g. `1.0.0`. | | `name` | string | Display name in the launcher and store. | | `hardwareRequirements` | array | What the glasses must (or may) have. See below. **Required.** | | `description` | string | Short summary. | | `icon` | string | Path to the icon, relative to the project root (e.g. `icon.png`). Packed into the bundle. | | `type` | string | `"standard"`. | | `sdkVersion` | string | The `@mentra/miniapp` version you built against. | | `minHostVersion` | string | Minimum Mentra App version required to run this miniapp. | | `entry` | object | `{ "background": "background/index.js", "ui": "ui/index.html" }`: built output paths for the two layers. Declaring `entry.background` is what makes a miniapp two-layer (and gives you `session.ui`). | | `port` | number | Dev-server starting port (default `3000`). Used only by `dev`; `release` picks its own port. | | `permissions` | array | Runtime permissions you request. See below. | | `actions` | array | Capabilities other miniapps / Mentra AI can invoke. See [Interop & Actions](/app-devs/core-concepts/miniapp-interop). | The `$schema` line points editors at the JSON Schema shipped with the CLI, so you get autocomplete and validation as you type. `create-mentra-miniapp` adds it for you; you can also print the schema with `mentra-miniapp schema print`. ## Permissions Each entry is `{ "type": "...", "description": "..." }`. The optional description is shown to the user in the OS permission prompt. | `type` | Grants | | - | - | | `MICROPHONE` | Audio input, transcription, translation | | `CAMERA` | Photos and video on camera glasses | | `CALENDAR` | Calendar events via `session.phone.calendar` | | `LOCATION` | GPS via `session.location` (and `session.navigation` / `session.heading`) | | `BACKGROUND_LOCATION` | Location while the user isn't in your miniapp | | `READ_NOTIFICATIONS` | Phone notifications via `session.phone.notifications` | | `POST_NOTIFICATIONS` | Posting notifications | > [!NOTE] > > Declaring a permission is not the same as the user granting it. > `session.permissions.has(type)` reports what you **declared** in the manifest, not > the live OS grant. If the user denied the OS prompt, your subscriptions simply > receive nothing. See [Permissions](/app-devs/core-concepts/permissions). ## Hardware requirements Each entry is `{ "type": "...", "level": "...", "description"?: "..." }`. | `type` | `LEVEL` | | - | - | | `CAMERA`, `DISPLAY`, `MICROPHONE`, `SPEAKER`, `IMU`, `BUTTON`, `LIGHT`, `WIFI` | `REQUIRED` or `OPTIONAL` | * **`REQUIRED`**: glasses without this hardware can't install the miniapp (when Store distribution becomes available, it will be hidden in the Mentra Miniapp Store and launcher on incompatible devices). * **`OPTIONAL`**: incompatible glasses still run the miniapp, in a degraded state. > [!NOTE] > > `MICROPHONE` means that the miniapp needs audio input. MentraOS satisfies this > requirement with the connected glasses microphone when one is available and > falls back to the phone microphone otherwise. A miniapp that requires audio is > therefore compatible with glasses such as the Vuzix Z100 that do not have an > onboard microphone. ## Editing the manifest You can edit `miniapp.json` by hand, or use the CLI's guided commands: ```bash theme={null} mentra-miniapp manifest # interactive wizard mentra-miniapp permission add MICROPHONE mentra-miniapp hardware add DISPLAY REQUIRED ``` See the [CLI reference](/app-devs/reference/cli) for all of them. # Navigation Source: https://docs.mentraglass.com/app-devs/core-concepts/navigation Turn-by-turn directions on the glasses, driven by the phone's GPS. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.navigation` runs a turn-by-turn trip. Call `start()` with a destination, then read live maneuvers from `onUpdate()` as the user moves. The phone owns the trip lifecycle; your miniapp starts it, listens, and stops it. ```typescript src/background/index.ts theme={null} const off = session.navigation.onUpdate((update) => { if (update.kind === "maneuver" && update.instruction) { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: update.instruction}]); } }); session.navigation.start({lat: 37.7764, lng: -122.4242, mode: "walking"}); ``` Navigation needs the `LOCATION` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest). The gated methods (`requestPermission`, `start`, `computeRoute`) check the declaration before doing anything; without it they resolve `{ok: false, error}` and never reach the phone. > [!NOTE] > > Navigation works on both iOS and Android. Every method resolves with an > `{ok, error?}` shape and nothing throws, so branch on `result.ok` rather than > wrapping in try/catch. ## Permission `session.navigation.hasPermission` is `true` when `LOCATION` is declared in your manifest. It reports the declaration, not the OS grant: if the user denied the system location prompt, a trip still won't get fixes. `requestPermission()` confirms the navigation SDK's terms up front, before the user hits start. On platforms with a terms gate it shows the dialog; where there isn't one it resolves immediately. It's idempotent, and resolves a `NavPermissionResult`: | Field | Type | Notes | | - | - | - | | `ok` | `boolean` | `false` when `LOCATION` isn't declared. | | `accepted` | `boolean` | `true` once the user accepts the SDK terms (always `true` where there's no terms gate). | | `error` | `string?` | Set when `ok` is `false`. | ```typescript theme={null} const result = await session.navigation.requestPermission(); if (result.ok && result.accepted) { // ready to start a trip } ``` ## Starting a trip `start(options)` accepts a single destination (`lat` / `lng`) or an ordered list of `stops`. A resolved `{ok: true}` means the phone accepted the request, not that a route exists yet. Watch `onUpdate()` and `onRoute()` for the route and live maneuvers. ```typescript theme={null} await session.navigation.start({ stops: [ {lat: 37.7764, lng: -122.4242}, {lat: 37.7858, lng: -122.4065}, ], mode: "walking", avoid: {highways: true}, }); ``` | Option | Type | Notes | | - | - | - | | `lat` / `lng` | `number` | Single-destination shorthand. Rewritten to `stops: [{lat, lng}]`. | | `stops` | `LatLng[]` | Ordered stops; the last is the final destination. Needs at least one entry. | | `mode` | `TravelMode` | Defaults to `"driving"`. | | `avoid` | `RouteAvoidances` | `highways` / `tolls` / `ferries`, each defaulting to `false`. | | `simulate` | `boolean` | Dev/testing only. Fakes walking the route. | | `speedMultiplier` | `number` | Simulation speed factor. Defaults to `1`. | | `pivots` | `PivotOptions` | Override pivot-detection radii for this trip. | | `missedTurnRerouteMeters` | `number` | Reroute once the user is this many meters past a missed turn. Omit or `0` for host defaults. | ### Travel modes `mode` is one of: | Value | | | - | - | | `"driving"` | The default. | | `"walking"` | | | `"cycling"` | | | `"two_wheeler"` | | ## Live updates `onUpdate(handler)` delivers a `NavUpdate` every time the trip changes. Discriminate on `kind`. It returns an unsubscribe function. ```typescript theme={null} const off = session.navigation.onUpdate((update) => { switch (update.kind) { case "maneuver": // update.instruction, update.distanceMeters, update.nextStepRoad break; case "off_route": // update.offRouteDistanceMeters break; case "rerouting": break; case "arrived": off(); break; case "error": // update.message break; } }); ``` | `kind` | Carries | | - | - | | `"maneuver"` | The current step: `maneuverType`, `distanceMeters`, `instruction`, `nextStepRoad`, plus distance/time to destination and current speed/limit. | | `"off_route"` | `offRouteDistanceMeters`: perpendicular distance from the route. | | `"rerouting"` | Nothing. The host is computing a new route. | | `"arrived"` | Nothing. The trip is done. | | `"error"` | `message`. | On a `"maneuver"`, prefer `instruction` (the host's verbatim text, render as-is) and `nextStepRoad` (the street after the turn) over the legacy `toRoad` field. Distance and time fields use `-1` for unknown; the speed and heading fields use `null`. ## The route polyline `onRoute(handler)` fires once per route build, on the initial start and again on each reroute, with the full path. It returns an unsubscribe function. ```typescript theme={null} const off = session.navigation.onRoute((route) => { // route.points, route.totalDistanceMeters, route.steps }); ``` A `NavRoute` carries `points` (the polyline as `LatLng[]`), optional `totalDistanceMeters` and `totalDurationSeconds`, and optional `steps` (ordered `NavStep` segments, each ending in a maneuver). ## Reading current state `getState()` resolves a `NavState` snapshot of the running trip, or `null` when no trip is active. It's shaped like the streaming events, so a miniapp opened mid-trip can hydrate without waiting for the next update. ```typescript theme={null} const state = await session.navigation.getState(); if (state?.active) { // state.mode, state.currentStopIndex, state.route, state.maneuver } ``` ## Stopping `stop()` ends the active trip and detaches pivot tracking. It's fire-and-forget and a no-op when nothing is running. ```typescript theme={null} session.navigation.stop(); ``` ## Computing a route without starting `computeRoute(options)` returns route geometry without starting a trip. Use it to preview a route or show distance and time before the user commits. ```typescript theme={null} const result = await session.navigation.computeRoute({ origin: {lat: 37.7764, lng: -122.4242}, stops: [{lat: 37.7858, lng: -122.4065}], mode: "walking", alternatives: 2, }); if (result.ok && result.routes) { const primary = result.routes[0]; // alternates follow } ``` | Option | Type | Notes | | - | - | - | | `origin` | `LatLng` | Start coordinate. | | `stops` | `LatLng[]` | At least one entry; the last is the final destination. | | `mode` | `TravelMode` | Defaults to `"driving"`. | | `avoid` | `RouteAvoidances` | Same flags as `start()`. | | `alternatives` | `number` | Up to N routes when the engine offers them. Defaults to `1`. | It resolves `{ok, error?, routes?}`. Each `ComputedRoute` has `points`, `totalDistanceMeters`, `totalDurationSeconds`, an optional `summary`, and optional `steps`. The primary route is first; alternates follow. ## Pivots A pivot is a real turn along the route (a left or right), derived once per route build and re-derived on reroute. Pivots let you cue the user as a turn approaches rather than reacting to raw maneuver updates. `onPivot(handler)` subscribes to pivot events and returns an unsubscribe function. Each pivot fires `approaching` → `entered` → `exited`, once per kind, then the cursor advances to the next pivot. ```typescript theme={null} const off = session.navigation.onPivot((event) => { if (event.kind === "approaching") { // event.pivot.direction, event.distanceMeters } }); ``` Three reads of the pivot list are available without subscribing: | Method | Returns | | - | - | | `getPivots()` | The full `Pivot[]` for the active route. Empty before the first `onRoute`. | | `getActivePivot()` | The pivot the user is currently inside (between `entered` and `exited`), or `null`. | | `getUpcomingPivot()` | The next pivot ahead, or `null` once all pivots are passed. | A `Pivot` carries its `index`, `lat` / `lng`, `direction` (`"left"` or `"right"`), the `fromRoad` and `toRoad` names (each `null` when the engine has none), the `maneuver`, the `distanceAlongRouteMeters` from trip start, and the `radiusMeters` that counts as "turning now". Tune detection per trip with `pivots` on `start()`: `radiusMeters` sets the "turning now" radius and `approachThresholdMeters` sets the "approaching" radius. Both fall back to mode-aware defaults when omitted. ## Simulation helpers `session.navigation.dev` holds simulator-only helpers for testing trips, including `deviate(offsetMeters)`, which nudges the simulated position off-route to trigger a reroute. These are dev tooling, not part of a normal trip flow. # Permissions Source: https://docs.mentraglass.com/app-devs/core-concepts/permissions Query the permissions your miniapp declared, and react to changes. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.permissions` tells you which permissions your miniapp declared in its [`miniapp.json`](/app-devs/core-concepts/miniapp-manifest). `has(type)` is a synchronous boolean, `getAll()` returns the full record, and `onUpdate` / `onPermissionError` let you react to changes and to typed permission errors. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { if (session.permissions.has("microphone")) { session.transcription.on((data) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: data.text}]); }); } }); ``` > [!WARNING] > > `has()` and `getAll()` report what you **declared** in the manifest, not whether > the user granted the OS prompt. `has("microphone") === true` only means your > manifest lists `MICROPHONE`. If the user denied the OS permission, that > subscription stays silent and receives nothing. The way to know a permission was > actually granted is whether events arrive (and `onPermissionError` for the typed > declaration error). See [the manifest](/app-devs/core-concepts/miniapp-manifest) > for how to declare permissions. ## Methods | Method | Returns | Notes | | - | - | - | | `has(type)` | `boolean` | Whether `type` is declared in the manifest. | | `getAll()` | `PermissionRecord` | Fresh copy of the full declared record. | | `onUpdate(handler)` | `UnsubscribeFn` | Fires when the declared record changes. | | `onPermissionError(handler)` | `UnsubscribeFn` | Fires on a typed `PERMISSION_NOT_DECLARED` error. | `onUpdate` does not fire with the current value when you subscribe. Call `getAll()` yourself if you want the starting state. The record changes on the connect handshake (`CONNECT_ACK`), or when the phone pushes an update (for example a dev miniapp re-scanned with an updated manifest). The handler receives the full record, not a list. ```typescript theme={null} const stop = session.permissions.onUpdate((perms) => { // perms: { microphone: true, location: false, camera: false, ... } if (!perms.microphone) { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: "Microphone not declared"}]); } }); ``` ## Permission types `has()` and the keys of `PermissionRecord` use these canonical names. Each maps from the UPPER\_CASE manifest `type` values. | `PermissionType` | Declared by manifest `type` | | - | - | | `microphone` | `MICROPHONE` | | `camera` | `CAMERA` | | `location` | `LOCATION`, `BACKGROUND_LOCATION` | | `notifications` | `READ_NOTIFICATIONS`, `POST_NOTIFICATIONS` | | `calendar` | `CALENDAR` | `location` is true if you declared either `LOCATION` or `BACKGROUND_LOCATION`, and `notifications` is true if you declared either notification permission. `PermissionRecord` is `Record`, so `getAll()` always returns all five keys with a boolean each. ## Permission errors If you call a capability whose permission you never declared, the phone runtime rejects with `PERMISSION_NOT_DECLARED`. `onPermissionError` is a filtered view of the session `error` event that surfaces only those. ```typescript theme={null} session.permissions.onPermissionError((err) => { // err.permission – the manifest permission name (UPPER_CASE) that was missing // err.subscription – the subscription that triggered the rejection // err.operation – the operation that triggered the rejection }); ``` | Field | Type | Notes | | - | - | - | | `code` | `string` | `PERMISSION_NOT_DECLARED` today; reserved for future codes. | | `message` | `string` | Human-readable error message. | | `permission` | `string \| undefined` | The missing manifest permission (UPPER\_CASE). | | `subscription` | `string \| undefined` | The subscription that triggered it. | | `operation` | `string \| undefined` | The operation that triggered it. | This error means the permission is missing from your manifest, which is a fix you make in `miniapp.json`. It is separate from the user denying the OS prompt, which produces no error and instead leaves the relevant stream silent. > [!NOTE] > > Checking OS-grant state directly (`isGranted`) and prompting for a permission > (`request`) are not implemented yet. They are planned to land on this same module > alongside `has` / `getAll`, without renaming today's methods. For now, treat the > arrival of events as the signal that a permission was granted. # Phone Source: https://docs.mentraglass.com/app-devs/core-concepts/phone Phone Wi-Fi, notifications, calendar events, and battery. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.phone` reports state from the user's phone: notifications as they post, calendar events, and the phone's own battery. Each concern is its own sub-module. Register a handler and the events start arriving. ```typescript src/background/index.ts theme={null} session.phone.notifications.on((data) => { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: `${data.app}: ${data.title}`}]); }); ``` Notifications need the `READ_NOTIFICATIONS` permission and calendar events need the `CALENDAR` permission, both declared in your [manifest](/app-devs/core-concepts/miniapp-manifest). Phone battery needs no permission. ## Notifications `session.phone.notifications.on()` fires when a notification posts on the phone. ```typescript theme={null} session.phone.notifications.on((data) => { // data: PhoneNotificationData }); ``` Each event is a `PhoneNotificationData`: | Field | Type | Notes | | - | - | - | | `notificationId` | `string` | Stable id from the phone's notification listener. | | `app` | `string` | Human app name (e.g. `"Messages"`). | | `title` | `string` | The notification title. | | `content` | `string` | The notification body. | | `priority` | `string` | Android priority string; empty on iOS. | | `timestamp` | `number` | Unix ms timestamp. | | `packageName` | `string` | Reverse-DNS package/bundle id of the originating app. | ### Dismissals `session.phone.notifications.onDismissed()` fires when the user swipes away or clears a notification. ```typescript theme={null} session.phone.notifications.onDismissed((data) => { // data: NotificationDismissedData }); ``` Each event is a `NotificationDismissedData`: | Field | Type | Notes | | - | - | - | | `notificationId` | `string` | Same id as the matching post event from `notifications.on(...)`. | | `notificationKey` | `string?` | Android NotificationKey. More stable than `notificationId` across reposts. | | `packageName` | `string?` | Reverse-DNS package/bundle id of the originating app. | | `timestamp` | `number` | Unix ms timestamp of the dismissal. | > [!NOTE] > > `onDismissed` is **Android only**. iOS does not expose dismiss callbacks to apps > (an Apple privacy restriction), so subscribing on iOS succeeds but no events ever > fire. The matching `notifications.on()` post-event works on both platforms. ## Calendar `session.phone.calendar.listEvents()` reads a current snapshot from all event calendars on the phone. Declare `CALENDAR` as a required permission; the Mentra App requests access before opening the miniapp. ```typescript theme={null} const result = await session.phone.calendar.listEvents({ startsAt: new Date(), endsAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000), limit: 50, }); ``` The date window may span at most 31 days. `limit` defaults to 50 and may not exceed 100. The result contains `events` and a `truncated` flag. Each event has: | Field | Type | Notes | | - | - | - | | `id` | `string` | Unique occurrence id, including recurring instances. | | `calendarId` | `string` | Id of the source calendar. | | `title` | `string` | The event title. | | `startsAt` | `string` | ISO 8601 start time. All-day events preserve their calendar timezone offset. | | `endsAt` | `string` | ISO 8601 end time. All-day events preserve their calendar timezone offset. | | `timezone` | `string?` | The event's timezone. | | `allDay` | `boolean` | Whether it's an all-day event. | | `location` | `string?` | The event location. | | `notes` | `string?` | The event notes. | | `url` | `string?` | The platform calendar URL field, when available. | | `links` | `string[]` | Deduplicated HTTPS links found in URL, location, and notes. | ## Phone battery `session.phone.onBattery()` reports the phone's battery. It stays flat (not sub-namespaced) because it's a single event. ```typescript theme={null} session.phone.onBattery((data) => { // data: BatteryData }); ``` Each event is a `BatteryData`: | Field | Type | Notes | | - | - | - | | `level` | `number` | Battery level. | | `charging` | `boolean` | Whether the phone is charging. | ## Cleaning up Every subscription returns an unsubscribe function. The `notifications` sub-module also has a `stop()` that tears down every notification subscription at once. Calendar snapshots do not create a subscription. ```typescript theme={null} const off = session.phone.notifications.on(handler); off(); // drop this one session.phone.notifications.stop(); // drop all notification subscriptions ``` `session.phone.notifications.hasPermission` tells you whether `READ_NOTIFICATIONS` is declared in your manifest, and `session.phone.calendar.hasPermission` tells you whether `CALENDAR` is declared. Neither tells you whether the user granted the OS prompt. A calendar request rejects with `PERMISSION_DENIED` if access was later revoked. See [Permissions](/app-devs/core-concepts/permissions). ## Phone Wi-Fi Use these APIs before a user starts a feature that connects the phone to the glasses hotspot. They concern the **phone**, independently of `session.glasses.requestWifiSetup()`, which connects the glasses to a network. Neither method requires a miniapp manifest permission. ```typescript theme={null} const enabled = await session.phone.isWifiEnabled() if (enabled === false) { const result = await session.phone.requestWifiEnable( "Live video needs your phone’s Wi-Fi to connect to your glasses.", ) if (result.cancelled || result.enabled === false) return } // Continue the requested call/stream. The host handles joining the glasses hotspot. ``` `isWifiEnabled(): Promise` reads the Android radio switch, including when Wi-Fi is on but not connected to a network. On iOS it returns `true` for an observed Wi-Fi connection and `null` otherwise: iOS has no public API to read the switch. Unknown does not mean off or that a hotspot join will fail. `requestWifiEnable(reason?): Promise<{enabled: boolean | null; cancelled: boolean}>` shows a Mentra App prompt with your reason. On Android the action opens the system Wi-Fi panel, falling back to Wi-Fi settings. On iOS it opens the app's Settings page and explains how to reach Settings → Wi-Fi. No private Settings URLs are used. The miniapp stays mounted; the promise resolves after the user returns and the host rechecks Wi-Fi. Already-enabled Wi-Fi skips the prompt. Cancellation, or a Settings visit exceeding five minutes, returns `cancelled: true`. The normal short SDK request timeout does not apply while waiting for the user. A `null` result after returning on iOS lets you attempt the normal system hotspot join; it does not verify the radio is enabled. Call from a user-initiated flow while the Mentra App is active. Concurrent prompts reject with `REQUEST_ABORTED`. Older hosts or hosts without this capability reject with `NOT_IMPLEMENTED`. Only prompt automatically when the state is `false`. When it is `null`, try the normal hotspot join; call `requestWifiEnable()` from an explicit recovery action if the connection fails. Neither function switches Wi-Fi on itself, joins an SSID, or requires internet access through Wi-Fi. # The session Source: https://docs.mentraglass.com/app-devs/core-concepts/session Your background-layer handle to everything on the glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). When your miniapp connects, the SDK hands your [`registerMiniapp`](/app-devs/core-concepts/two-layer-architecture#the-background-layer) handler a `session`. It lives in the **background layer**, and every capability (display, microphone, sensors, storage) is a module on it. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { // session.userId – who this user is // session.packageName – your miniapp's package name // session.capabilities – what the connected glasses can do (null until ready) session.transcription.on((data) => { session.display.render([ { type: "text", id: "caption", box: { x: 0, y: 0, w: 576, h: 288 }, text: data.text }, ]); }); }); ``` You don't create or connect the session yourself. The host does it and runs your handler. Everything you do with the glasses goes through `session`. (For advanced or testing use you can `new MiniappSession()` and call `connect()` directly, but a normal miniapp never needs to.) ## Modules Type `session.` in your editor and autocomplete shows everything. Each module is covered in its own page. ### Output | Module | What it does | | - | - | | [`session.display`](/app-devs/core-concepts/display/layouts) | Scenes of positioned text, images, and shapes on the glasses HUD. | | [`session.speaker`](/app-devs/core-concepts/speakers/text-to-speech) | Play audio and text-to-speech. | | [`session.led`](/app-devs/core-concepts/led/overview) | The LED on supported glasses. | ### Input & audio | Module | What it does | | - | - | | [`session.transcription`](/app-devs/core-concepts/microphone/speech-to-text) | Real-time speech-to-text. | | [`session.translation`](/app-devs/core-concepts/translation) | Real-time translation. | | [`session.mic`](/app-devs/core-concepts/microphone/audio-chunks) | Raw audio chunks and voice-activity detection. | | [`session.input`](/app-devs/core-concepts/input) | Glasses button presses and touchpad gestures. | ### Sensors & device | Module | What it does | | - | - | | [`session.glasses`](/app-devs/core-concepts/glasses) | Battery and connection state of the glasses. | | [`session.location`](/app-devs/core-concepts/location) | GPS: latest fix or a stream. | | [`session.heading`](/app-devs/core-concepts/heading) | Compass heading. | | [`session.imu`](/app-devs/core-concepts/imu) | Head position, and raw accelerometer on G2. | | [`session.navigation`](/app-devs/core-concepts/navigation) | Turn-by-turn navigation. | ### Phone & system | Module | What it does | | - | - | | [`session.phone`](/app-devs/core-concepts/phone) | Notifications, calendar events, and phone battery. | | [`session.system`](/app-devs/core-concepts/system) | Share sheet, downloads, clipboard, open URL. | | [`session.permissions`](/app-devs/core-concepts/permissions) | Which permissions your miniapp declared, and changes to them. | ### Storage & data | Module | What it does | | - | - | | [`session.storage`](/app-devs/core-concepts/storage) | Persistent string key-value storage. | | [`session.blob`](/app-devs/core-concepts/blob) | Persistent binary storage (files, images, audio). | | [`session.auth`](/app-devs/core-concepts/auth) | A miniapp-scoped backend token and an authenticated `fetch`. | ### Glasses with cameras | Module | What it does | | - | - | | [`session.camera`](/app-devs/core-concepts/camera/photos) | Photos and video on camera glasses. | | [`session.stream`](/app-devs/core-concepts/stream) | Live streaming on camera glasses. | ### Interop & UI | Module | What it does | | - | - | | [`session.actions`](/app-devs/core-concepts/miniapp-interop) | Expose actions Mentra AI can call. | | [`session.ui`](/app-devs/core-concepts/two-layer-architecture#talking-between-layers) | The message bus to your UI layer. | | [`session.cloud`](/app-devs/core-concepts/cloud) | Cloud-connection status. | ## Lifecycle Subscribe to lifecycle events with `session.on(event, handler)`. Each returns an unsubscribe function. ```typescript theme={null} registerMiniapp((session) => { session.on("ready", () => { // CONNECT_ACK received. session.capabilities / userId are now populated. }); session.on("visibility", (state) => { // "foreground" | "background": the user opened or left your miniapp. }); session.on("disconnect", (reason) => { // The session ended. }); }); ``` > [!WARNING] > > `session.capabilities`, `session.userId`, and the user's permissions are > populated from the connect handshake, which lands **after** your handler first > runs. Subscribe to events immediately, but read `capabilities` inside > `session.on("ready", …)` (or `session.onCapabilitiesChange`). It's `null` before > then. If the background context crashes, the host respawns it and re-runs your handler with backoff. See [crash recovery](/app-devs/core-concepts/two-layer-architecture#crash-recovery). ## Subscriptions Registering a handler subscribes; calling the returned function unsubscribes. You never manage subscriptions by hand. ```typescript theme={null} const stop = session.transcription.on((data) => { session.display.render([ { type: "text", id: "caption", box: { x: 0, y: 0, w: 576, h: 288 }, text: data.text }, ]); }); stop(); // unsubscribe ``` Every `.on()`, `.onUpdate()`, `.onChunk()`, `.onButtonPress()`, and the rest return this cleanup function. Subscriptions are ref-counted internally: the SDK only talks to the phone on the first subscribe and last unsubscribe of a given stream. > [!NOTE] > > A successful request means **accepted**, not **done**. For streaming capabilities, > watch the stream for the real result rather than assuming the call already > succeeded. ## Properties | Property | Type | Description | | - | - | - | | `session.userId` | `string` | The user's ID (available after `ready`). | | `session.packageName` | `string` | Your miniapp's package name. | | `session.capabilities` | `GlassesCapabilities \| null` | What the connected glasses can do. `null` until `ready`. | | `session.visibility` | `"foreground" \| "background"` | Whether the user is currently in your miniapp. | | `session.colorScheme` | `"light" \| "dark"` | The host's current color scheme. | | `session.ready` | `boolean` | Whether the connect handshake has completed. | ## Next steps > **[Display](/app-devs/core-concepts/display/layouts)** > > Show text and graphics on the glasses. > **[Transcription](/app-devs/core-concepts/microphone/speech-to-text)** > > Turn speech into text in real time. > **[Storage](/app-devs/core-concepts/storage)** > > Persist data across sessions. > **[Actions](/app-devs/core-concepts/miniapp-interop)** > > Let Mentra AI call your miniapp. # Speaker Source: https://docs.mentraglass.com/app-devs/core-concepts/speakers/text-to-speech Play URLs, text-to-speech, and live PCM audio through the phone or connected glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.speaker` is audio output: play a URL, speak text, or stream live PCM. Playback follows the phone's media route, including connected Bluetooth audio glasses. For audio *input* (mic frames, voice activity), use [`session.mic`](/app-devs/core-concepts/microphone/audio-chunks) instead. ```typescript src/background/index.ts theme={null} import {registerMiniapp} from "@mentra/miniapp/background" registerMiniapp((session) => { session.transcription.on(async (data) => { await session.speaker.speak(`You said: ${data.text}`) }) }) ``` ## Methods | Method | Returns | Notes | | - | - | - | | `play(options)` | `Promise` | Play a URL. Resolves when playback completes on the phone. | | `speak(text, options?)` | `Promise` | Speak text. Resolves when playback completes; rejects on TTS failure. | | `createStream(options?)` | `Promise` | Open a bounded live PCM stream. Android and iOS. | | `stop()` | `void` | Stop any audio this miniapp is playing. Fire-and-forget. | | `state` | `SpeakerState` | Current playback state (sync getter). | | `isPlaying` | `boolean` | `true` when `state === "playing"`. | | `onStateChange(handler)` | `UnsubscribeFn` | Fires on every state transition. | ## Playing a URL `play` streams an audio URL through the phone's playback service and resolves when it finishes. ```typescript theme={null} await session.speaker.play({ audioUrl: "https://example.com/chime.mp3", volume: 0.8, // optional stopOtherAudio: true, // optional, defaults to false }) ``` `PlayAudioOptions`: | Field | Type | Notes | | - | - | - | | `audioUrl` | `string` | The URL to play. | | `volume` | `number?` | Playback volume. | | `stopOtherAudio` | `boolean?` | Stop other audio first. Defaults to `false`. | ## Text-to-speech `speak` sends the text to the phone, which streams cloud TTS when connected and falls back to local offline TTS when the cloud is unavailable. It resolves when playback completes. Offline playback may segment longer text internally to reduce latency; callers use the same `speak()` API and receive the same interruption behavior in either mode. By default, the Mentra App converts display-oriented markup and symbols into natural spoken words before either TTS path runs. For example, `68°F (20°C)` becomes "68 degrees Fahrenheit or 20 degrees Celsius." Set `enableSanitization: false` when the TTS engine must receive the exact input, such as code, equations, or character-by-character content. ```typescript theme={null} const result = await session.speaker.speak("Turn left in 200 meters", { voice_id: "rachel", volume: 1.0, }) if (!result.completed) { // Playback was interrupted before finishing. } ``` `SpeakOptions`: | Field | Type | Notes | | - | - | - | | `voice_id` | `string?` | Voice to use. | | `voice_settings` | `Record?` | Provider-specific voice tuning. | | `volume` | `number?` | Playback volume. | | `stopOtherAudio` | `boolean?` | Stop other audio first. Defaults to `false`. | | `enableSanitization` | `boolean?` | Normalize markup and symbols. Defaults to `true`. | | `forceLocal` | `boolean?` | Require on-device TTS; never call cloud TTS. | `SpeakResult` has one field, `completed`: `true` if playback finished, `false` if it was interrupted. ## Live PCM streaming Use `createStream()` when audio arrives incrementally and cannot be exposed as a complete file or URL—for example, live meeting audio received over a WebSocket. The stream accepts signed 16-bit little-endian mono PCM on Android and iOS. > [!WARNING] > > Live PCM requires Mentra App 2.13.0 or newer. Set `minHostVersion: "2.13.0"` in a miniapp that depends on > `createStream()` so older hosts reject installation instead of failing during a call. ```typescript src/background/index.ts theme={null} const writer = await session.speaker.createStream({ sampleRate: 24000, volume: 1, stopOtherAudio: true, }) try { for await (const pcmChunk of livePcmChunks) { // Await every write. The host applies backpressure when its native queue // reaches roughly two seconds, keeping memory bounded and audio ordered. const {bufferedMs} = await writer.write(pcmChunk) console.debug("speaker buffer", bufferedMs) } await writer.close() // drains queued audio before resolving } catch (error) { await writer.abort() // immediately drops queued audio throw error } ``` `SpeakerStreamOptions`: | Field | Type | Notes | | - | - | - | | `sampleRate` | `16000 \| 24000 \| 48000` | Defaults to `16000`. | | `channels` | `1` | Version 1 supports mono only. | | `volume` | `number?` | `0` through `1`. Defaults to `1`. | | `stopOtherAudio` | `boolean?` | Stop other playback before opening. Defaults to `true`. | The returned `SpeakerStreamWriter` has: | Method | Behavior | | - | - | | `write(Uint8Array \| ArrayBuffer)` | Encodes, chunks, orders, and writes raw PCM. Returns `{bufferedMs}`. | | `writeBase64(string)` | Writes already base64-encoded PCM without decoding it in the miniapp. | | `close()` | Drains scheduled audio and returns `{durationMs?}`. | | `abort()` | Immediately stops playback and drops the queue. Idempotent. | > [!WARNING] > > Always await writes and call either `close()` or `abort()`. The host also aborts the stream when the miniapp > disconnects, but explicit teardown gives the best call-end latency. Compressed MP3, AAC, and Opus frames are not > accepted; decode them to PCM before writing. ### Errors `speak` rejects with an object carrying a `code` field on cloud-side TTS failure. Catch it and branch on `code`: ```typescript theme={null} try { await session.speaker.speak(longText) } catch (err) { if (err.code === "TTS_TEXT_TOO_LONG") { // split the text and retry } } ``` | Code | Meaning | | - | - | | `TTS_TEXT_TOO_LONG` | The text exceeded the provider's length limit. | | `TTS_INVALID_VOICE` | The `voice_id` isn't a known voice. | | `TTS_UPSTREAM_ERROR` | The TTS provider returned an error. | | `TTS_LOCAL_UNAVAILABLE` | `forceLocal` was requested without a ready local model. | | `INTERNAL` | Any other failure. | ## State Five states, read synchronously via `state` or `isPlaying`, or observed as transitions via `onStateChange`: | State | Meaning | | - | - | | `idle` | Nothing has played yet. | | `loading` | A `speak()`, `play()`, or `createStream()` request is being prepared. | | `playing` | Audio is playing. | | `stopped` | Playback ended, by completion or `stop()`. | | `error` | A failure occurred. Transient. | A request moves `idle` → `loading` → `playing` → `stopped`. On failure the speaker fires `error` once with `errorCode` set, then settles to `stopped` so `isPlaying` reads `false`. ```typescript theme={null} const off = session.speaker.onStateChange((event) => { if (event.state === "error") { console.warn(event.errorCode, event.errorMessage) } if (event.state === "stopped") { console.log("ran for", event.durationMs, "ms") } }) off() // unsubscribe ``` `SpeakerStateEvent`: | Field | Type | Notes | | - | - | - | | `state` | `SpeakerState` | The new state. | | `errorCode` | `string?` | Set when `state === "error"` (`TTS_*` or `INTERNAL`). | | `errorMessage` | `string?` | Human-readable detail on error. | | `durationMs` | `number?` | Set when `state === "stopped"`: how long playback ran (best-effort). | > [!NOTE] > > `onStateChange` does not fire immediately with the current value. Read `state` separately if you want the starting > value. Audio *input* (mic frames and voice-activity detection) lives on > [`session.mic`](/app-devs/core-concepts/microphone/audio-chunks). # Storage Source: https://docs.mentraglass.com/app-devs/core-concepts/storage Persistent key-value storage scoped to your miniapp. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.storage` is a phone-local key-value store. It's scoped to the user and your miniapp, so your keys never collide with another miniapp's, and the data survives restarts. Keys and values are both strings. ```typescript theme={null} await session.storage.set("lastNote", "buy milk"); const note = await session.storage.get("lastNote"); // "buy milk" ``` Storage needs no permission and no hardware. Every method is async and round-trips to the phone. ## Reading and writing ```typescript theme={null} await session.storage.set("count", "3"); const count = await session.storage.get("count"); // "3", or null if unset await session.storage.delete("count"); const exists = await session.storage.has("count"); // false ``` | Method | Returns | Notes | | - | - | - | | `get(key)` | `Promise` | `null` when the key isn't set. | | `set(key, value)` | `Promise` | Overwrites silently. | | `delete(key)` | `Promise` | No-op if the key isn't set. | | `has(key)` | `Promise` | Whether the key is set. | | `keys()` | `Promise` | Every key in your namespace. | | `clear()` | `Promise` | Drops every key in your namespace. | ## Bulk operations For hydrating state on startup or saving several keys at once: ```typescript theme={null} const all = await session.storage.getAll(); // { count: "3", lastNote: "…" } await session.storage.setMultiple({ a: "1", b: "2" }); // one round-trip ``` | Method | Returns | Notes | | - | - | - | | `getAll()` | `Promise>` | Every key/value pair. | | `setMultiple(values)` | `Promise` | Writes many pairs in one round-trip. | > [!NOTE] > > Keep the total stored payload small (think under \~1 MB). `getAll()` pulls > everything across the bridge at once, and a large namespace will stall the > background context while it resolves. For images, audio, or files, use > [`session.blob`](/app-devs/core-concepts/blob) instead. ## Storing objects Values are strings, so serialize structured data yourself: ```typescript theme={null} await session.storage.set("notes", JSON.stringify(notes)); const notes = JSON.parse((await session.storage.get("notes")) ?? "[]"); ``` # Streaming Source: https://docs.mentraglass.com/app-devs/core-concepts/stream Live video streaming from camera glasses. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.stream` starts and stops a live video stream from camera glasses like Mentra Live. One method, `startStream`, covers both ways to stream: Mentra hosts it for you (the default), or you publish straight to a URL you own. ```typescript src/background/index.ts theme={null} // Managed: Mentra hosts the stream and gives you playback URLs. const stream = await session.stream.startStream(); session.display.showReferenceCard("Live", stream.hlsUrl); ``` ## Managed streaming (default) Call `startStream()` with no `direct` URL and Mentra provisions a hosted stream, handles transport and reconnection, and returns playback URLs. This is the right choice for most apps. ```typescript src/background/index.ts theme={null} // Hosted stream, optionally fanned out to your own destinations. const stream = await session.stream.startStream({ destinations: [{ url: "rtmp://a.rtmp.youtube.com/live2/KEY", name: "youtube" }], }); // stream.hlsUrl, stream.dashUrl, stream.webrtcUrl ``` | Option | Type | What it is | | - | - | - | | `destinations` | `Array` | Restream targets the relay fans out to (YouTube, Twitch, …). Bare URLs or `{ url, name? }` objects, mixed freely. | | `ingest` | `"srt" \| "whip"` | Latency/durability trade. `"srt"` (default): \~10-20s latency, HLS/DASH playback, recording. `"whip"`: sub-second WebRTC playback, no HLS/DASH, no recording. | | `video` | `StreamVideoConfig` | `width`, `height`, `bitrate`, `fps` (all optional). | | `audio` | `StreamAudioConfig` | `bitrate`, `sampleRate`, `echoCancellation`, `noiseSuppression` (all optional). | | `sound` | `boolean` | Play glasses-side start/stop sounds. Defaults to `true`. | ## Direct streaming Pass `direct` with your own ingest URL (rtmp/rtmps/srt/whip) and the glasses publish straight to it, with no Mentra relay. You manage the endpoint, and no playback URLs come back. ```typescript src/background/index.ts theme={null} const stream = await session.stream.startStream({ direct: "rtmp://example.com/live/STREAM-KEY", video: { fps: 30 }, }); // stream.streamId, pass it to stop() ``` `video`, `audio`, and `sound` work the same as managed. `destinations` and `ingest` are ignored for direct streams. ## Result `startStream` resolves to a `StreamResult`: | Field | Type | What it is | | - | - | - | | `streamId` | `string` | ID for this stream. Pass it to `stop`. | | `status` | `string` | The stream's reported state. | | `resolvedConfig` | `StreamResolvedConfig` | The transport and the video/audio settings actually applied. Optional. | | `hlsUrl` | `string` | HLS playback URL. Managed streams only. | | `dashUrl` | `string` | DASH playback URL. Managed streams only. | | `webrtcUrl` | `string` | WHEP playback URL. Managed streams in `"whip"` mode. | | `liveInputId` | `string` | Hosted live-input UID, for building player URLs. Managed streams only. | | `mode` | `"hls" \| "webrtc"` | Which playback the stream serves. Managed streams only. | ## Stop `stop` ends the stream. Pass the `streamId` from the start result, or omit it to stop the active stream. ```typescript src/background/index.ts theme={null} await session.stream.stop(stream.streamId); ``` # Stream preview Source: https://docs.mentraglass.com/app-devs/core-concepts/stream-preview Show the live video of a call inside your miniapp's UI. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). A stream preview draws raw decoded video frames from a source straight into your miniapp's WebView. The frames never leave the phone and never pass through your background as JSON, so the preview costs no network and adds no encode step. In this release the only source is `"call"`: the outgoing video of a phone-native meeting your miniapp started with `session.meeting.join()`. A preview of the glasses camera outside a call is not available yet; asking for `"glasses"` rejects with `unsupported`. ```typescript src/background/index.ts theme={null} // After the call connects. Never await this on the connect path. session.stream .preview({source: "call"}) .then((handle) => { previewHandle = handle handle.onStatus((status) => { if (status.state === "ended") previewHandle = null }) }) .catch((error) => console.warn("preview unavailable", error.code)) ``` ```tsx src/ui/CallScreen.tsx theme={null} import {StreamPreview} from "@mentra/miniapp/react" export function CallPreview({visible}: {visible: boolean}) { return visible ? console.warn(code)} /> : null } ``` ## Requirements * Declare the `CAMERA` permission in `miniapp.json`. * Your miniapp must own the active meeting. Previewing another miniapp's call is not supported. * Only one preview lease exists at a time. A second `preview()` rejects with `preview_busy`, and the first keeps delivering. ## Who owns what A preview has three parts with different lifetimes. Keeping them apart is what lets the UI come and go freely without interrupting anything. | Part | Owned by | Created | Ends | | - | - | - | - | | **Lease** (`PreviewHandle`) | Your background | `session.stream.preview()` | `handle.stop()`, the meeting ending, or your background stopping | | **Connection** | The UI document | The first `` mount | A reload or navigation, or the lease ending | | **Renderer** | The `` component | Mount | Unmount | The lease survives the UI closing and the WebView being destroyed. Reopening the UI and mounting `` again picks the preview back up without a new `preview()` call. ## Hide versus release * **Hide:** unmount `` (or let it size to zero). The phone stops producing frames, but the lease and the document's connection stay open, so mounting it again shows video immediately. Use this for an eye toggle. * **Release:** call `handle.stop()`. The lease ends and every connection to it closes. Do this when the call ends or the user turns the preview off for good. Ending the meeting releases the lease automatically. The Mentra App also pauses production while it is in the background and resumes it when it returns. Your component receives `paused: true` in `onStatus` in the meantime. ## `session.stream.preview(options?)` | Option | Type | What it is | | - | - | - | | `source` | `"call" \| "glasses"` | Defaults to `"call"`. `"glasses"` rejects with `unsupported`. | Resolves with a `PreviewHandle`: | Member | What it is | | - | - | | `source` | The source the lease is on. | | `handleId` | Identifies this lease. | | `previewTraceId` | Correlation id stamped on every `PREVIEW_TRACE` log line for this lease. Include it in bug reports. | | `state` | `"held"` or `"ended"`. | | `stop()` | Releases the lease. Idempotent. | | `onStatus(cb)` | Fires `{state: "ended", reason}` exactly once. Returns an unsubscribe function. | Rejects with a `PreviewError` whose `code` is one of the codes below. ## `` | Prop | Type | What it is | | - | - | - | | `fit` | `"contain" \| "cover"` | `contain` (default) letterboxes; `cover` fills and crops. The picture is never stretched. | | `onStatus` | `(status) => void` | Connection state (`waiting_for_lease`, `open`, `error`, ...), `paused`, and once-a-second counters in `stats`. | | `onError` | `(code) => void` | A typed error code, below. | | `className`, `style` | | Applied to the canvas, which fills its container by default. | Mount the component before or after `preview()` resolves; if the lease is not held yet it reports `waiting_for_lease` and connects on its own as soon as it is. The picture is sized to the rendered box and capped at 640×360 at 15 fps; resizing only reconfigures the phone when the size tier changes. ## Error and status codes | Code | Meaning | | - | - | | `permission_denied` | The miniapp does not declare `CAMERA`. | | `not_meeting_owner` | There is no active meeting, or another miniapp owns it. | | `preview_busy` | Another preview already holds the source. | | `waiting_for_lease` | Status, not an error: the UI is ready but the background has not taken the lease yet. | | `source_ended` | The meeting ended, before or after `preview()` resolved. | | `unsupported` | This device or WebView cannot carry the preview, or the source is not available. | | `ack_timeout`, `transport_failed` | The frame link dropped. The component reconnects on its own a few times. | | `pack_failed` | The phone could not prepare frames. Remount the component to try again. | | `paused_background` | Status: the Mentra App is in the background. | A preview failure never affects the call. Treat the preview as optional: keep showing your own call UI when it is `waiting_for_lease` or in an error state. # System Source: https://docs.mentraglass.com/app-devs/core-concepts/system Phone-side OS actions: share sheet, downloads, clipboard, open URL. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.system` runs OS-level actions on the phone: open the native share sheet, save a file through the system downloader, copy text to the clipboard, or open a URL in the phone browser. None of these need a permission or any hardware. ```typescript src/background/index.ts theme={null} const result = await session.system.share({ text: "Check out my run", url: "https://maps.app/abc" }); if (result.cancelled) { // the user dismissed the share sheet } ``` ## Share `share(options)` opens the phone's native share sheet. Pass text and a URL for a link share, or pass `base64` with a `mimeType` to share file contents. The promise resolves once the sheet closes. ```typescript theme={null} // Share a link await session.system.share({ title: "My route", text: "Today's walk", url: "https://maps.app/abc", }); // Share file contents await session.system.share({ base64: pngBytes, mimeType: "image/png", filename: "route.png", }); ``` | `ShareOptions` field | Type | Notes | | - | - | - | | `text` | `string` | Body text for the share. | | `url` | `string` | A URL to share. | | `title` | `string` | Title shown by the share sheet. | | `base64` | `string` | Base64-encoded file data for file sharing. | | `mimeType` | `string` | MIME type when sharing `base64` data. | | `filename` | `string` | Filename when sharing `base64` data. | | `ShareResult` field | Type | Notes | | - | - | - | | `success` | `boolean` | Whether the share completed. | | `cancelled` | `boolean` | `true` when the user dismissed the sheet. Optional. | ## Download `download(options)` saves a file. It opens the share sheet so the user picks where to save. Give it a `url` to fetch from, or `base64` data you already have. ```typescript theme={null} // Download from a URL const a = await session.system.download({ url: "https://example.com/report.pdf", filename: "report.pdf", }); // Save base64 data const b = await session.system.download({ base64: csvBytes, mimeType: "text/csv", filename: "export.csv", }); ``` | `DownloadOptions` field | Type | Notes | | - | - | - | | `url` | `string` | URL to download from, or use `base64`. | | `base64` | `string` | Base64 file data, as an alternative to `url`. | | `filename` | `string` | Suggested filename. | | `mimeType` | `string` | MIME type of the data. | | `DownloadResult` field | Type | Notes | | - | - | - | | `success` | `boolean` | Whether the download completed. | | `cancelled` | `boolean` | `true` when the user dismissed the save sheet. Optional. | > [!NOTE] > > To export data your miniapp holds as binary, read it from > [`session.blob`](/app-devs/core-concepts/blob), base64-encode it, and pass it to > `share` or `download`. ## Clipboard `copyToClipboard(text)` copies a string to the system clipboard. It resolves when the copy is done. ```typescript theme={null} await session.system.copyToClipboard("https://maps.app/abc"); ``` ## Open URL `openUrl(url)` opens a URL in the phone's browser. It's fire-and-forget: it returns `void` immediately and doesn't report whether the browser opened. Dangerous schemes (`javascript:`, `file:`) are blocked. ```typescript theme={null} session.system.openUrl("https://docs.mentra.glass"); ``` | Method | Returns | Notes | | - | - | - | | `share(options)` | `Promise` | Opens the native share sheet. | | `download(options)` | `Promise` | Saves a file via the system downloader. | | `copyToClipboard(text)` | `Promise` | Copies text to the clipboard. | | `openUrl(url)` | `void` | Opens a URL in the phone browser. Blocks `javascript:`/`file:`. | # Translation Source: https://docs.mentraglass.com/app-devs/core-concepts/translation Real-time translation of speech from the glasses microphone. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `session.translation` turns spoken speech into translated text. Register a handler for the language pair you care about and results start arriving; the SDK handles the subscription for you. ```typescript theme={null} session.translation.to("es", (data) => { if (data.isFinal) { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: data.text}]); } }); ``` Translation needs the `MICROPHONE` permission in your [manifest](/app-devs/core-concepts/miniapp-manifest). Without it the phone drops the subscription and no events arrive. The rejection is silent: there is no error event, your handler just never fires. ## Results Each event is a `TranslationData`: | Field | Type | Notes | | - | - | - | | `text` | `string` | The translated text so far. | | `isFinal` | `boolean` | `false` for interim results that keep updating, `true` once the utterance is settled. | | `sourceLanguage` | `string` | Language the speech was recognized in. | | `targetLanguage` | `string` | Language `text` was translated into. | | `originalText` | `string?` | Source-language text of the same utterance, when the provider supplies it. | | `utteranceId` | `string?` | Stable id correlating interim and final results of one utterance. | | `speakerId` | `string?` | Speaker id when the provider reports diarization. | Interim results stream as the user speaks and get replaced; wait for `isFinal` if you only want settled text. Use `utteranceId` to match an interim result to the final one that supersedes it. ## Choosing what to listen for Three methods select which translations reach your handler. They differ only in how they pin the source and target language. | Method | Listens for | | - | - | | `on(handler)` | Every translation event, any source, any target. | | `to(target, handler)` | Any source, translated to `target`. Pass an array of targets to fan one handler across several. | | `fromTo(source, target, handler)` | A specific `source` to `target` pair. Pass an array for `target` to fan one source across several targets. | Each returns an unsubscribe function. `source` and `target` are language tags (for example `"en"`, `"es"`). ```typescript theme={null} // Any language the user hears, translated to Spanish. session.translation.to("es", handler); // Several targets at once, one handler. session.translation.to(["es", "fr"], handler); // A specific direction: English speech to Spanish text. session.translation.fromTo("en", "es", handler); // One source fanned across multiple targets. session.translation.fromTo("en", ["es", "fr"], handler); ``` `on()` is the broad case. It registers cheaply on your side but asks the cloud to fan every active pair out to you, so reach for `to` or `fromTo` once you know the language(s) you want. When the provider supplies `originalText`, you can show the source and the translation together: ```typescript theme={null} session.translation.to("es", (data) => { if (data.originalText) { session.display.showDoubleTextWall(data.originalText, data.text); } else { session.display.render([{type: "text", id: "msg", box: {x: 0, y: 0, w: 576, h: 288}, text: data.text}]); } }); ``` > [!NOTE] > > `forLanguagePair(fromLang, toLang, handler)` is a deprecated alias for > `fromTo(source, target, handler)`. Use `fromTo` in new code; the alias will be > removed in a future release. ## Cleaning up Every subscription returns an unsubscribe function. `stop()` tears down every subscription this module owns at once: ```typescript theme={null} const off = session.translation.to("es", handler); off(); // drop this one session.translation.stop(); // drop all of them ``` `session.translation.hasPermission` tells you whether `MICROPHONE` is declared in your manifest. It does not tell you whether the user granted the OS prompt: if they denied it, your handler simply never fires. See [Permissions](/app-devs/core-concepts/permissions). > [!NOTE] > > Translation gives you translated text. For the untranslated transcript in the > spoken language, use > [`session.transcription`](/app-devs/core-concepts/microphone/speech-to-text). > Subscribing to one does not subscribe you to the other. # Two-layer architecture Source: https://docs.mentraglass.com/app-devs/core-concepts/two-layer-architecture How a miniapp's always-on background and on-demand UI layers work and talk to each other. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). Every Mentra miniapp has two layers. Understanding the split is the key to the whole SDK: it determines where your code runs, what it can reach, and how the two halves communicate. | Layer | Lifetime | Has DOM? | Hardware access | Import from | | - | - | - | - | - | | **Background** | Always on while the miniapp is enabled | No | Full: owns the `session` | `@mentra/miniapp/background` | | **UI** | Only while the user has your tile open | Yes (React) | None: talks to background | `@mentra/miniapp/ui`, `@mentra/miniapp/react` | ## File layout ``` my-miniapp/ ├── miniapp.json # declares both entry points ├── build.ts # bundles both layers in one pass └── src/ ├── background/ │ └── index.ts # registerMiniapp(session => …) ├── ui/ │ ├── index.html │ ├── main.tsx # mounts , calls mentra.ready() │ └── App.tsx └── shared/ └── channels.ts # typed channel registry: both sides import it ``` `miniapp.json` points at the built output of each layer: ```json theme={null} { "entry": { "background": "background/index.js", "ui": "ui/index.html" } } ``` ## The background layer The host evaluates your background bundle inside a per-miniapp JavaScript context and calls your `registerMiniapp` handler when the background starts. This is where every `session.*` call lives. ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; import "../shared/channels"; registerMiniapp(async (session) => { // Subscribe to glasses events here. The runtime tears them down on disable. session.transcription.on((data) => { session.display.render([ { type: "text", id: "caption", box: { x: 0, y: 0, w: 576, h: 288 }, text: data.text }, ]); }); }); ``` The background runs **even when no UI is open**. Register your subscriptions in the handler; they live for as long as the miniapp is enabled. ### Startup: return your setup promise The Mentra App keeps your miniapp's splash screen up, and holds every message your UI sends, until the promise returned by your `registerMiniapp` handler settles. Your `session.ui.handle(...)` handlers therefore exist before the UI's first `mentra.request(...)` arrives. * **Return your setup.** Return or `await` it; never discard it with `void`. Otherwise the UI can open before your handlers are registered, and its first requests fail with `no handler registered`. * **Register UI handlers before slow work.** The splash stays up while the promise is pending. Register `session.ui.handle` / `session.ui.onOpen` first, and keep network calls out of the startup path. * **Failures still open the UI.** If the handler throws or rejects, or takes longer than 10 seconds, the Mentra App opens the UI anyway. ```typescript src/background/index.ts theme={null} registerMiniapp(async (session) => { const settings = await loadSettings(); // fast, local session.ui.handle("getHistory", () => history.list()); session.ui.onOpen(() => session.ui.send("settings", settings)); void refreshFromNetwork(session); // don't hold the UI for this }); ``` The same applies after a crash: requests the UI sends while the background restarts wait for the new background to be ready. Only requests the crashed background was already handling fail. > [!NOTE] > > The background context is not a browser and not Node: it's a bare JS engine > (JavaScriptCore on iOS, QuickJS on Android). There's no `window`, no DOM, and no > module resolver at runtime, which is why `build.ts` bundles the SDK *into* your > background output rather than leaving it external. ### Background runtime APIs Do not infer background support from what TypeScript or Bun accepts at build time. The background bundle targets a small, cross-platform runtime. MentraOS currently provides these browser-like globals: | Available | Scope and limits | | - | - | | `console` | `log`, `info`, `warn`, `error`, `debug`, and `trace` | | Timers | `setTimeout`, `clearTimeout`, `setInterval`, `clearInterval`, and `queueMicrotask` | | `fetch` | String, `ArrayBuffer`, and typed-array request bodies; `text()`, `json()`, and byte-exact `arrayBuffer()` responses over the native HTTP bridge | | `WebSocket` | Native WebSocket bridge | | `localStorage` | Per-miniapp string key/value storage | | `crypto` | `getRandomValues` and `randomUUID`; `crypto.subtle` is not implemented | | Encoding | `TextEncoder`, `TextDecoder`, `atob`, and `btoa` | | Cancellation | The SDK-supported `AbortController` / `AbortSignal` subset | Common browser and Node APIs are **not** available in the background. This includes `window`, `document`, DOM elements, `performance`, `XMLHttpRequest`, Node built-ins such as `fs` and `path`, `process`, and runtime `require()` / dynamic module resolution. Put rendering, DOM work, and browser UI libraries in `src/ui/`. Put `session.*` calls, hardware subscriptions, durable state, and logic that must continue after the UI closes in `src/background/`. Exchange only serializable data through `mentra.*` / `session.ui.*`. > [!WARNING] > > If an AI coding agent introduces `performance.now()`, DOM access, or a Node API > under `src/background/`, move that code to the UI or replace it with a supported > primitive. For elapsed time in the background, use `Date.now()`. > > Projects created with `create-mentra-miniapp` also check background source and > its imported shared files during `bun run build`. Common unsupported browser or > Node APIs fail the build with the source location and a supported alternative. ## The UI layer The UI is a normal React app. It has **zero** direct native access: it reaches the glasses only by sending messages to its own background layer. It mounts inside `` and must call `mentra.ready()` on boot so the host knows it's mounted. ```tsx src/ui/main.tsx theme={null} import { createRoot } from "react-dom/client"; import { MentraProvider } from "@mentra/miniapp/ui"; import { App } from "./App"; createRoot(document.getElementById("root")!).render( , ); mentra.ready(); // flush any buffered messages from background ``` ## Talking between layers The two layers never share memory: they pass messages over a typed bus. You declare every channel once in `src/shared/channels.ts`; both halves import it, so names and payload shapes are enforced at compile time. ```typescript src/shared/channels.ts theme={null} import type { MentraTyped } from "@mentra/miniapp/ui"; export interface Channels { ping: { at: number }; // UI → background pong: { at: number; roundtripMs: number }; // background → UI } declare global { var mentra: MentraTyped; } ``` **From the UI**, use the injected `mentra` global: ```tsx theme={null} mentra.send("ping", { at: Date.now() }); // fire-and-forget mentra.on("pong", ({ roundtripMs }) => { /* … */ }); // subscribe const res = await mentra.request("getNotes", {}); // request/response (RPC) ``` **From the background**, use `session.ui`: ```typescript theme={null} session.ui.onOpen(() => session.ui.send("pong", { at: Date.now(), roundtripMs: 0 })); session.ui.on("ping", ({ at }) => { /* … */ }); session.ui.handle("getNotes", async () => ({ notes })); // respond to mentra.request ``` The two sides are intentionally asymmetric: `mentra.send` from the UI **buffers** until the WebView calls `ready()`, while `session.ui.send` from the background **drops silently** when no WebView is open (there's no one to receive it). There's no runtime channel registry: channel names are opaque strings on the wire, validated only by TypeScript. See [Interop & Actions](/app-devs/core-concepts/miniapp-interop) for the separate, cross-miniapp RPC layer (`session.actions`). ## Lifecycle 1. **Install**: the host unzips the bundle, validates the manifest, spawns the background JS context, and runs your `registerMiniapp` handler. Background is alive. 2. **UI opens**: the user taps your tile; the host spawns a WebView, injects the `mentra` shim, and loads `ui/index.html`. The WebView calls `mentra.ready()`. Once your background is ready (see [Startup](#startup-return-your-setup-promise)), the splash hides and `session.ui.onOpen` fires in the background. 3. **UI closes**: the user navigates away; the host destroys the WebView. The background context stays alive and `session.ui.onClose` fires. 4. **Disable / uninstall**: the host tears down your subscriptions and kills the background context. ## Crash recovery If your background layer crashes, the host restarts it automatically. Repeated crashes back off and eventually show the user a "try again" prompt, so aim to reach a stable state quickly on startup. A UI request that was in progress when the background crashed fails with `BACKGROUND_RESTARTED` and is never replayed; retry it yourself if it is safe to repeat. ## Next steps > **[The manifest](/app-devs/core-concepts/miniapp-manifest)** > > Declare entries, permissions, and hardware in `miniapp.json`. > **[The session](/app-devs/core-concepts/session)** > > The background-side handle to every glasses capability. > **[The UI layer](/app-devs/core-concepts/webviews/react-webviews)** > > React hooks, the capsule menu, and safe areas. > **[Interop & Actions](/app-devs/core-concepts/miniapp-interop)** > > Let other miniapps (and Mentra AI) call yours. # Back navigation Source: https://docs.mentraglass.com/app-devs/core-concepts/webviews/back-navigation Make the phone's back gesture step back through your miniapp's screens and dialogs before it minimizes the miniapp. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). The Mentra App sends the phone's back gesture (the left-edge swipe on iOS, the back button or gesture on Android) to your UI WebView's browser history first: * **While your WebView has history**, back runs `history.back()` inside your miniapp. Your router or `useHistoryState` hook sees the `popstate` and shows the previous screen. * **When the history is empty**, back minimizes your miniapp. The UI WebView unmounts, but your [background layer](/app-devs/core-concepts/session) keeps running with its session. When the user reopens the miniapp, the UI starts again on its first screen with empty history. Keep anything that must survive in the background layer and send it to the UI when it reopens. The host can only see browser history. A screen or dialog that exists only in React state (`useState`, `MemoryRouter`, `createMemoryRouter`, or wouter's `memoryLocation`) never creates a history entry, so back skips past it and minimizes the whole miniapp. Anything the user should be able to go back from must push a history entry. ## Pages: a router in hash mode For page-level navigation, use your router's hash mode. Each navigation becomes a real history entry, and hash URLs keep working when the host reloads your bundle from disk. ```tsx react-router theme={null} import { HashRouter, Route, Routes } from "react-router-dom"; } /> } /> ``` ```tsx wouter theme={null} import { Router } from "wouter"; import { useHashLocation } from "wouter/use-hash-location"; ``` A back button in your own header should also go back, not forward to the parent route. Otherwise the next back gesture returns the user to the page they just left. ```tsx theme={null} import { MiniappHeader } from "@mentra/miniapp/ui"; history.back()} /> ``` ## Tabs, dialogs, and screens: `useHistoryState` For UI state that is not a route, such as tabs, dialogs, sheets, pickers, or a multi-step screen, use `useHistoryState` from `@mentra/miniapp/ui`. It works like `useState`, except that the phone's back gesture undoes each change. ```tsx theme={null} import { useHistoryState } from "@mentra/miniapp/ui"; export function App() { const [tab, setTab] = useHistoryState<"home" | "settings">("tab", "home"); const [pickerOpen, setPickerOpen] = useHistoryState("languagePicker", false); return ( <> {tab === "home" ? setPickerOpen(true)} /> : } {pickerOpen && setPickerOpen(false)} />} ); } ``` | Action | Result | | - | - | | Set a new value | Pushes a history entry. Back restores the previous value. | | Set the value it had before its latest change, such as `setPickerOpen(false)` from the picker's close button | Goes back instead of pushing, so the next back gesture does not reopen the picker. | | Set the same value again | Does nothing, so repeated taps do not stack entries. | Rules for keys and values: * **`key` names the value inside the history entry.** Use a different key for each piece of state on the page. Two components that pass the same key share one value. * **Values are stored in `history.state`,** so they must survive structured clone: strings, numbers, booleans, `null`, and plain objects and arrays. For a dialog about one item, store the item's id rather than an object with methods. * **The value belongs to the current history entry.** A component that remounts, or a page that reloads, reads it back. Minimizing the miniapp discards it along with the WebView. `useHistoryState` and a hash router can live in the same app. A route change starts a fresh entry, so hook values fall back to their initial values on the new page and come back when the user returns. ## Check it on a phone 1. Open a nested page or dialog, then swipe back. You land on its parent. 2. Open a dialog, close it with its own button, then swipe back. The dialog does not reopen. 3. From the first screen, swipe back. The miniapp minimizes. On Android, repeat each step with the system back button. ## Next steps > **[The UI layer](/app-devs/core-concepts/webviews/react-webviews)** > > The `mentra` bridge, hooks, and components. > **[Safe areas & layout](/app-devs/core-concepts/webviews/safe-areas)** > > Keep content clear of the capsule menu. # The UI layer Source: https://docs.mentraglass.com/app-devs/core-concepts/webviews/react-webviews Build the miniapp's WebView UI with React, talking to the background over the injected mentra global. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). The UI layer is the on-demand half of a [two-layer miniapp](/app-devs/core-concepts/two-layer-architecture). It is a normal React app that mounts when the user opens your tile and is destroyed when they leave. It has zero direct hardware access. To reach the glasses it sends messages to its own [background layer](/app-devs/core-concepts/session), which holds the `session` and translates those messages into `session.*` calls. The bridge is a single injected global named `mentra`. The entry point is small: create a React root, wrap it in ``, and call `mentra.ready()`. ```tsx src/ui/main.tsx theme={null} import { createRoot } from "react-dom/client"; import { MentraProvider } from "@mentra/miniapp/ui"; import "../shared/channels"; import { App } from "./App"; createRoot(document.getElementById("root")!).render( , ); mentra.ready(); // flush any messages background buffered while the WebView mounted ``` UI imports come from two sub-paths. `@mentra/miniapp/ui` re-exports everything (the `mentra` global types, the React hooks, ``, ``), and `@mentra/miniapp/react` exports the hooks and components directly. There is no `@mentra/sdk` here and no hosted server: the UI never authenticates, never opens a port, and never owns a session. > [!NOTE] > > The UI layer must not construct a `MiniappSession`. That class is the background (JSContext) API. Inside a WebView its `CONNECT` never gets acked by the host's UI router, so `session.connect()` times out with `CONNECT_ACK timeout`. UI code reaches the glasses only through `mentra`, and reads host state through the hooks below. ## The `mentra` global The host injects `mentra` into every UI WebView before your content loads. It carries the typed bus to background plus the two lifecycle signals. ```tsx theme={null} mentra.send("ping", { at: Date.now() }); // fire-and-forget, UI → background const off = mentra.on("pong", ({ roundtripMs }) => { /* … */ }); // subscribe, returns an unsubscribe fn const res = await mentra.request("getNotes", {}); // request/response (RPC) mentra.ready(); // call once on bootstrap ``` | Method | What it does | | - | - | | `mentra.send(channel, payload)` | Broadcast a message to background. Buffered until `ready()` is acked, then flushes; after that it fires immediately. Compile-error on an RPC channel. | | `mentra.on(channel, cb)` | Subscribe to broadcasts from background. Returns an unsubscribe function. Compile-error on an RPC channel. | | `mentra.request(channel, payload, options?)` | Call a background `session.ui.handle(channel, …)` handler and await its return. No default timeout; pass `options.timeout` and `options.signal` to bound it. | | `mentra.onOpen(cb)` | Fires when the bridge is open and `send` is flushable. Fires immediately if you subscribe after `ready()`. Returns an unsubscribe function. | | `mentra.onClose(cb)` | Fires when the host is about to destroy the WebView. Snapshot state into background via `send` here if you want it to survive. | | `mentra.ready()` | Call once on bootstrap. Tells the host the WebView is mounted; background's `session.ui.onOpen` handlers fire after this lands. Idempotent. | The two sides are asymmetric on purpose. `mentra.send` from the UI buffers until `ready()`, while `session.ui.send` from the background drops silently when no WebView is open. See [Talking between layers](/app-devs/core-concepts/two-layer-architecture#talking-between-layers) for the background side (`session.ui.handle`, `on`, `send`). `mentra.request` rejects with an `Error` whose `name` is `MentraRpcError` (the handler threw), `MentraRpcTimeoutError` (the `timeout` elapsed), or `AbortError` (the `signal` aborted). Branch on `err.name`, not `instanceof`: these errors come from the WebView's bare runtime scope, so `instanceof` checks are unreliable across the bridge. ## Typed channels Declare every channel once in `src/shared/channels.ts`. Both halves import this file, so channel names and payload shapes are checked at compile time on both sides of the bridge. `MentraTyped` types the global; importing the file for its `declare global` side effect is what wires the types in. ```typescript src/shared/channels.ts theme={null} import type { MentraTyped } from "@mentra/miniapp/ui"; export interface Channels { ping: { at: number }; // UI → background pong: { at: number; roundtripMs: number }; // background → UI } declare global { // eslint-disable-next-line no-var var mentra: MentraTyped; } ``` For request/response channels, mark the entry as an RPC shape so `mentra.request` (and `useRpc`) infer the request and response types. `send`/`on` then reject that channel at compile time, and `request` rejects the broadcast channels. ## React hooks Each hook is a thin read over host-injected state or the `mentra` bridge. The host injects safe-area and color-scheme values once before content loads, so those hooks read at mount and do not update at runtime (a theme or orientation change forces a reload). | Hook | Signature | Returns | | - | - | - | | `useConnected` | `useConnected(): boolean` | `true` while the `mentra` bridge is open (after `ready()`, before close). Tracks `onOpen`/`onClose`. | | `useColorScheme` | `useColorScheme(): MiniappColorScheme` | The host's color scheme, `"light"` or `"dark"`, read once at mount. | | `useSafeArea` | `useSafeArea(): UseSafeAreaResult` | `{ insets, capsuleMenu }`: pixel insets to pad your root with, plus the capsule menu rect (`null` when the host renders none). | | `useCapsuleHeaderStyle` | `useCapsuleHeaderStyle(options?): CSSProperties` | Ready-to-spread CSS that vertically aligns a header row with the floating capsule menu. | | `useCapabilities` | `useCapabilities(): GlassesCapabilities \| null` | Always `null` in the UI. The glasses profile is background-only; relay it over a channel instead (read `session.capabilities`, `session.ui.send` it, then `mentra.on` it). | | `useRpc` | `useRpc(channel): RpcCallable` | A stable callable around `mentra.request(channel, …)` for one RPC channel, plus `.abort()`. | | `useHistoryState` | `useHistoryState(key, initialValue): [T, SetHistoryState]` | `useState` backed by browser history, so the phone's back gesture undoes each change. Use it for tabs, dialogs, and screens that are not routes. See [Back navigation](/app-devs/core-concepts/webviews/back-navigation). | `useRpc` recreates its `AbortController` per call: calling it again aborts the previous in-flight request (good for per-keystroke autocomplete), and unmount aborts everything pending. A caller-provided `options.signal` is merged with the internal one. ```tsx theme={null} import { useRpc } from "@mentra/miniapp/react"; const autocomplete = useRpc("places:autocomplete"); const results = await autocomplete({ query: "blue bot" }); autocomplete.abort(); // cancel the latest in-flight call ``` > [!NOTE] > > `useCapabilities` returns `null` by design. It does not construct a `MiniappSession` (which would time out in a WebView). When a UI page needs the glasses profile, read `session.capabilities` in a background `session.ui.onOpen` handler, `session.ui.send` it on a channel, and pick it up in the UI with `mentra.on`. ## Components ### `` The optional root provider. Today it keeps `` in sync with the host's color scheme so shared theme CSS switches automatically. The toggle runs synchronously during render, so children paint with the right class on first mount and there is no theme flash. Pass `syncColorScheme={false}` if your app owns the `dark` class itself (for example when integrating a theming library). ```tsx theme={null} ``` ### `` A drop-in header that aligns with the host's capsule menu, leaving the top-right clear. It has three slots: `left`, `title`, and `right`. A string `title` renders as a semantic `

`; pass a node for custom markup. Set `onBack` to render a back chevron in the left slot (ignored if `left` is set). The component layers `useCapsuleHeaderStyle` underneath, so its `UseCapsuleHeaderStyleOptions` (`leftPadding`, `rightGap`, `fallbackHeight`, `fallbackMarginTop`) are accepted as props. ```tsx theme={null} import { MiniappHeader } from "@mentra/miniapp/react"; history.back()} right={Connected} /> ``` Use `useCapsuleHeaderStyle` directly when you want full control over header markup but want the alignment math done for you. For the capsule-menu rect, the safe-area insets, and where you may not place clickable content, see [Safe Areas & Layout](/app-devs/core-concepts/webviews/safe-areas). ## Next steps > **[Two-layer architecture](/app-devs/core-concepts/two-layer-architecture)** > > The bus model and how background and UI talk. > **[Safe areas & layout](/app-devs/core-concepts/webviews/safe-areas)** > > Keep content clear of the capsule menu. > **[Back navigation](/app-devs/core-concepts/webviews/back-navigation)** > > Make the back gesture step through your screens and dialogs. > **[The session](/app-devs/core-concepts/session)** > > The background-side `session` your UI sends to. > **[Interop & Actions](/app-devs/core-concepts/miniapp-interop)** > > Let other miniapps call yours. # Safe areas & the capsule menu Source: https://docs.mentraglass.com/app-devs/core-concepts/webviews/safe-areas Lay out your webview UI around the host chrome: notch insets and the floating capsule menu. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). Your miniapp UI runs in a WebView inside the Mentra App, and the host draws its own chrome on top. Two things overlap your content: the device safe-area insets (notch, status bar, rounded corners) and a floating **capsule menu** the host parks in the top-right. If you lay out edge-to-edge, your header sits under the notch and your top-right button sits under the capsule, where taps go to the host instead of your app. `useSafeArea()` reads both regions so your layout can pad around them. It imports from `@mentra/miniapp/react`, the [UI layer](/app-devs/core-concepts/webviews/react-webviews) of the [two-layer architecture](/app-devs/core-concepts/two-layer-architecture). ```tsx theme={null} import { useSafeArea } from "@mentra/miniapp/react"; function App() { const { insets } = useSafeArea(); return (

My Miniapp

); } ``` The host injects these values before your content loads. The hook reads them once at mount. They don't change at runtime today, so there's nothing to re-render on. ## `useSafeArea()` Returns an object with two fields. | Field | Type | Description | | - | - | - | | `insets` | `MiniappSafeAreaInsets` | Pixel insets around the WebView content. Apply as padding on your root element. | | `capsuleMenu` | `MiniappCapsuleMenuRect \| null` | Bounding rect of the host's floating capsule menu. `null` when the host doesn't render one (e.g. older builds). | `insets` always has all four sides, defaulting to `0` when the host provides none (for example, a browser preview): | `insets` field | Type | Description | | - | - | - | | `top` | `number` | Inset from the top edge. | | `bottom` | `number` | Inset from the bottom edge. | | `left` | `number` | Inset from the left edge. | | `right` | `number` | Inset from the right edge. | `capsuleMenu` is the rect of the floating control in the top-right. Use it to keep interactive elements out from under the menu, since taps there reach the host. | `capsuleMenu` field | Type | Description | | - | - | - | | `top` | `number` | Distance from the top of the viewport to the menu. | | `right` | `number` | Distance from the right of the viewport to the menu. | | `bottom` | `number` | Distance from the top to the menu's bottom edge. | | `left` | `number` | Distance from the left to the menu's left edge. | | `width` | `number` | Menu width in CSS pixels. | | `height` | `number` | Menu height in CSS pixels. | > [!NOTE] > > Coordinates are in CSS pixels. Always handle `capsuleMenu === null` so your layout > still works outside the Mentra App and on hosts that don't draw a menu. ## `useCapsuleHeaderStyle()` When you want your own header markup but want the capsule alignment done for you, `useCapsuleHeaderStyle()` returns a `CSSProperties` object you spread onto your header element. It positions the row vertically centered on the capsule menu and reserves room on the right so your content doesn't slide under it. ```tsx theme={null} import { useCapsuleHeaderStyle } from "@mentra/miniapp/react"; function Header() { const style = useCapsuleHeaderStyle(); return (

Hello

); } ``` It accepts an options object. All four are optional and have fallbacks used when the host doesn't render a capsule menu. | Option | Type | Default | Description | | - | - | - | - | | `leftPadding` | `number` | `20` | Extra left padding beyond the safe area. | | `rightGap` | `number` | `16` | Space between the header content and the capsule menu. | | `fallbackHeight` | `number` | `32` | Header height when no capsule menu is present. | | `fallbackMarginTop` | `number` | `16` | Top margin when no capsule menu is present. | The returned style sets `display: flex` with `justifyContent: "space-between"`, so a left child and a right child split to opposite ends of the row. ## `` For a stock header, `` renders the aligned row for you. It has three slots (`left`, `title`, `right`) and respects the safe area and capsule menu the same way. It also accepts every `useCapsuleHeaderStyle()` option as a prop. Render it inside a root that is already padded by `insets`, as in the first example. ```tsx theme={null} import { MiniappHeader, useSafeArea } from "@mentra/miniapp/react"; function App() { const { insets } = useSafeArea(); return (
history.back()} right={Connected} /> {/* page content */}
); } ``` | Prop | Type | Description | | - | - | - | | `title` | `ReactNode` | A string renders as a semantic `

`. Pass a node for custom markup. | | `left` | `ReactNode` | Left slot, typically a back button or logo. Overrides `onBack` if both are set. | | `onBack` | `() => void` | Renders a back chevron in the left slot that calls this handler. Ignored when `left` is set. | | `right` | `ReactNode` | Right slot, typically a badge or action buttons. Sits to the left of the capsule. | | `className` | `string` | Applied to the header element. | | `style` | `CSSProperties` | Inline overrides merged over the computed layout style. | | `bottomSpacer` | `boolean` | Adds an 8px spacer below the header so the next content isn't flush against it. Default `true`. | `` also forwards `leftPadding`, `rightGap`, `fallbackHeight`, and `fallbackMarginTop` through to the underlying `useCapsuleHeaderStyle()`. The component ships only the layout, so use `className` or `style` for colors and fonts. ## Status bar and home indicator colors The WebView fills the whole screen, so the Mentra App draws no bars of its own. Whatever your page paints at its top and bottom edges shows behind the status bar and the home indicator. * Put your background (a color, gradient or image) on the root element you pad with `insets`, and make that element fill the screen. Padding, unlike margin, keeps the background under the status bar and the home indicator. * The status-bar text follows the Mentra App's light or dark theme. Wrap your app in ``, which mirrors that theme as ``, and define colors for both themes so the status bar stays readable. ```css theme={null} :root { --background: #ffffff; --foreground: #111111; } :root.dark { --background: #0b0b0b; --foreground: #f5f5f5; } .app { box-sizing: border-box; /* keeps the inset padding inside the 100vh */ min-height: 100vh; background: var(--background); color: var(--foreground); } ``` ## Next steps > **[React webviews](/app-devs/core-concepts/webviews/react-webviews)** > > The UI layer overview and its hooks. > **[Two-layer architecture](/app-devs/core-concepts/two-layer-architecture)** > > How the UI and background layers split and talk. # AI IDE Integration Source: https://docs.mentraglass.com/app-devs/getting-started/ai-ide-integration Connect your AI coding assistant to MentraOS docs for smarter development > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). > [!NOTE] > > **TL;DR** - Get your AI coding assistant to understand MentraOS instantly. Works with Zed, Cursor, VS Code, Windsurf, Claude Code, and more. ## Why Connect Your AI IDE? When you connect your AI coding assistant to MentraOS docs, it can: * Answer questions about the Mentra Miniapp SDK * Help you write smart glasses apps faster * Look up API references automatically * Debug your code with context about how MentraOS works ## MCP Server URL All setups below use this URL: ``` https://docs.mentraglass.com/mcp ``` This is the **docs MCP** (SDK reference and guides). It is not the Developer Console. *** ## Setup by IDE ### Zed Add this to your `settings.json` (open with `Cmd+,` or `zed: open settings`): ```json theme={null} { "context_servers": { "mentraos-docs": { "command": "npx", "args": ["-y", "mcp-remote", "https://docs.mentraglass.com/mcp"], "env": {} } } } ``` Restart Zed and you're done. > **How to verify it's working** > > Open the Agent Panel and check the MCP servers indicator in settings. A green dot means the server is active. You can also type `/` in a chat to see available prompts from the server. *** ### Cursor Add this to your Cursor MCP config file at `~/.cursor/mcp.json` (create it if it doesn't exist): ```json theme={null} { "mcpServers": { "mentraos-docs": { "url": "https://docs.mentraglass.com/mcp" } } } ``` Then **restart Cursor** and approve the MCP server if prompted. > **Alternative: Project-level config** > > You can also create `.cursor/mcp.json` in your project root to enable MentraOS docs only for that project. *** ### VS Code (with GitHub Copilot) VS Code 1.99+ has built-in MCP support with GitHub Copilot. 1. Open Command Palette (`Cmd+Shift+P` / `Ctrl+Shift+P`) 2. Run **"MCP: Add Server"** 3. Select **"HTTP"** as the transport type 4. Enter URL: `https://docs.mentraglass.com/mcp` 5. Name it `mentraos-docs` > **Manual configuration** > > Create `.vscode/mcp.json` in your project and add: > > ```json theme={null} > { > "servers": { > "mentraos-docs": { > "type": "http", > "url": "https://docs.mentraglass.com/mcp" > } > } > } > ``` *** ### Windsurf 1. Open Windsurf Settings (`Cmd+,` / `Ctrl+,`) 2. Search for "MCP" and click to open the Manage Plugins panel 3. Click **"View raw config"** to open `mcp_config.json` 4. Add the following configuration: ```json theme={null} { "mcpServers": { "mentraos-docs": { "command": "npx", "args": ["-y", "mcp-remote", "https://docs.mentraglass.com/mcp"] } } } ``` 5. Click **Refresh** in the Manage Plugins panel > **Config file location** > > * **macOS**: `~/.codeium/windsurf/mcp_config.json` > * **Windows**: `%USERPROFILE%\.codeium\windsurf\mcp_config.json` *** ### Claude Code (CLI) Run this in your terminal: ```bash theme={null} claude mcp add --transport http mentraos-docs https://docs.mentraglass.com/mcp ``` > **Verify it worked** > > Run `claude mcp list` to see your connected servers. You should see `mentraos-docs` in the list. *** ### Other MCP-Compatible Tools For tools that support remote HTTP MCP servers, use this URL directly: ``` https://docs.mentraglass.com/mcp ``` If your tool only supports command-based local MCP servers, use `mcp-remote`: ```bash theme={null} npx -y mcp-remote https://docs.mentraglass.com/mcp ``` *** ## Test It Out After setup, try asking your AI assistant: * *"How do I display text on smart glasses with MentraOS?"* * *"Show me how to use the camera API"* * *"What modules are available on the miniapp session?"* * *"How do I subscribe to transcription in a specific language?"* Your AI should now give you accurate, up-to-date answers using our docs. *** ## Troubleshooting > **Server not connecting** > > If your setup uses `mcp-remote`, make sure you have Node.js 18+ installed. > > ```bash theme={null} > node --version # Should be v18.0.0 or higher > ``` > **npx command not found** > > Install Node.js from [nodejs.org](https://nodejs.org/) which includes npm and npx. > **Permission errors** > > Fix your Node.js package permissions with a version manager or npm prefix configuration instead of running downloaded packages with `sudo`. *** ## What's Next? > **[Quickstart](/app-devs/getting-started/quickstart)** > > Build your first smart glasses app in 15 minutes > **[Example Apps](/app-devs/getting-started/example-apps)** > > Clone a working example and start hacking # Distribution Source: https://docs.mentraglass.com/app-devs/getting-started/distribution Prepare and install miniapp builds for local testing. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). A miniapp is a **bundle that runs on the user's phone**: there's no server to host, no domain, no uptime to manage. MentraOS 3.0 supports local development and testing, but it does not provide a supported way to distribute a miniapp to users. > [!NOTE] > > **Looking for the `@mentra/sdk` cloud SDK?** Apps used to be cloud-hosted > servers built with `@mentra/sdk`. They're now built on-device with the Mentra > Miniapp SDK (`@mentra/miniapp`), which these docs cover. The older cloud SDK > docs live at [mentraglass.com/legacy](https://mentraglass.com/legacy). ## Dev install (you, iterating) While you're building, [`mentra-miniapp dev`](/app-devs/reference/cli#dev) serves your project over the LAN with hot reload. Scan the QR from the Mentra App and your changes reload on save. This is the inner loop; see the [Quickstart](/app-devs/getting-started/quickstart). Dev mode is not an installation: the computer and CLI must remain running to serve each miniapp's runtime bundle. The Mentra App keeps a separate dev entry for each manifest package name and caches its name and icon, so you can scan and test multiple dev miniapps together. Rescanning the same package updates that entry. Use a release install when you want the miniapp to run without the computer. On a phone signed in to a company workspace, the Miniapp Developer tools appear only if the workspace allows miniapp development. A dev build runs as the package named in its `miniapp.json`, so it can stand in for an installed copy of that package. Dev code is unsigned: the Mentra App runs it in place of an unsigned install, but refuses it while the package is installed from a bundle signed with a publisher key. Uninstall the miniapp from the phone first, then scan again. `session.auth` works in a dev build the same way it does for the installed miniapp. ## Release install (local testing over LAN) To put a real, installed build on one or more test phones on your local network, run: ```bash theme={null} mentra-miniapp release ``` This builds, packs, and serves the bundle behind a `miniapp://release` QR. Anyone on the same network can scan it to install. The miniapp installs on the device, **runs offline, and persists across restarts**: no laptop required once it's installed. This is intended for local testing, demos, and dogfooding; it is not a supported distribution channel. Each install is logged in your terminal, and the server stays up so multiple devices can install from the same QR. The miniapp opens on the phone as soon as it installs. A release install is a fixed snapshot, like a React Native release build. After each change, run `mentra-miniapp release` again and rescan. While you develop, use `mentra-miniapp dev` instead: scan once, and every saved change reloads on the phone. ## Future distribution In a future release, developers will upload miniapps through the [Mentra Developer Console](https://console.mentraglass.com), and users will download them from the [Mentra Miniapp Store](https://apps.mentraglass.com). The Developer Console and Mentra Miniapp Store do not support Miniapp SDK distribution in MentraOS 3.0. The CLI can already produce the ZIP artifact intended for that future workflow: ```bash theme={null} mentra-miniapp pack ``` The artifact is written to `build/-.zip`. Make sure your [`miniapp.json`](/app-devs/core-concepts/miniapp-manifest) is accurate before you pack: its `hardwareRequirements` will decide which glasses can see your miniapp, and its `permissions` will drive the OS prompts users see. > [!NOTE] > > Camera, photo, and live-streaming features run through the on-glasses Bluetooth > stack, not a cloud server. See the [Bluetooth SDK docs](/bluetooth-sdk/overview) > for those workflows. ## Versioning Bump `version` in `miniapp.json` for every release build; it must be a semantic version such as `1.2.0`. The CLI names the artifact `-.zip`. The Mentra App installs a release only when its version is the same as or newer than the installed one. Installing the same version again replaces it; an older version is refused, whether it comes from a release QR or is bundled with the Mentra App. To go back to an older build, publish it again under a newer version. ## Signing Signing is optional. `mentra-miniapp pack --sign` (or `release --sign`) embeds a publisher signature in the bundle; see [Signing](/app-devs/reference/cli#signing) for keys and CI. The Mentra App follows the Android model: the installed signer decides which bundles may replace a miniapp. | Installed miniapp | Incoming bundle | Result | | - | - | - | | none, or unsigned | unsigned | installs | | none, or unsigned | signed | installs, and pins the miniapp to that key | | signed with key A | signed with key A | installs | | signed with key A | signed with key B | refused | | signed with key A | unsigned, or a dev build | refused: uninstall first | Uninstalling the miniapp clears its recorded key. The same rule applies to every way a miniapp reaches the phone; there is no exception for first-party miniapps. ## Next steps > **[CLI reference](/app-devs/reference/cli)** > > `dev`, `release`, and `pack` in detail. > **[The manifest](/app-devs/core-concepts/miniapp-manifest)** > > Get permissions and hardware requirements right before you build a release. # Example miniapps Source: https://docs.mentraglass.com/app-devs/getting-started/example-apps Working miniapps to start from: the scaffolder starter, the in-repo reference, and first-party apps shipped in this repo. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). There are three ways to get working code in front of you: scaffold a fresh project, read the in-repo reference miniapp, or browse the first-party miniapps that ship in this repo. ## Start from the scaffolder The fastest path is `create-mentra-miniapp`. It writes a ready-to-run project with both layers wired together. The [Quickstart](/app-devs/getting-started/quickstart) walks through it end to end. ```bash theme={null} bunx create-mentra-miniapp my-app ``` It asks for a project name and the kind of glasses you're targeting. You can skip the prompt with a flag: ```bash theme={null} bunx create-mentra-miniapp my-app --type=display # Even Realities G1/G2, Vuzix Z100 bunx create-mentra-miniapp my-app --type=camera # Mentra Live ``` The `--type` choice only changes the manifest's `hardwareRequirements` (display vs. camera plus microphone). The generated code is the same either way: a glasses button press calls `session.display.render(...)`, and the WebView UI sends a `ping` that the background answers with a `pong` carrying a roundtrip timestamp. That gives you a button-to-display path and a UI-to-background path to edit from. ## The in-repo reference miniapp If you've cloned the monorepo, `miniapps/example-miniapp` is the reference app that consumes the SDK. Its manifest names it "Mentra Example", and it exercises more of the surface than the starter does: transcription, text-wall display, TTS, button events, and an "SDK Tester" UI that drives each session module (`session.camera`, `session.location`, notifications, calendar) page by page. Run it from the repo to see the SDK working against your phone: ```bash theme={null} cd miniapps/example-miniapp bun install bun dev ``` `bun dev` validates the manifest, builds both layers, serves them over your LAN, and prints a QR code. In the Mentra App, scan it from **Settings → Miniapp Developer Settings → Scan Miniapp QR Code** (phone and laptop on the same Wi-Fi, or `bun dev --usb` on Android to go over the cable instead). ## First-party miniapps in this repo These are apps Mentra ships, all under `miniapps/`. Read them when you want a real feature rather than a demo. Each builds with `bun dev` the same way the example does. > **Teleprompter** > > `miniapps/teleprompter`: paste a script and read it off the glasses; > scrolling follows your voice or runs at a set speed. > **Mentra Maps** > > `miniapps/navigation`: turn-by-turn walking navigation with compass heading > and Google Maps, driven by `session.location`. > **Recorder** > > `miniapps/recorder`: capture the glasses microphone to a WAV file on your > phone, play it back, and export it. Built on `session.recorder` + > `session.blob`. > **Captions** > > `miniapps/captions`: live transcription rendered on the display, with a > WebView for language and display settings. > **Translation** > > `miniapps/translation`: translate a detected source language into a target > language and render it on the display. > [!NOTE] > > New here? Read the [Quickstart](/app-devs/getting-started/quickstart) first, then > come back and open whichever app above is closest to what you're building. # Overview Source: https://docs.mentraglass.com/app-devs/getting-started/overview Build on-device apps for smart glasses with the Mentra Miniapp SDK. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). MentraOS is the operating system for smart glasses. A **miniapp** is an app that runs on the user's phone, inside the Mentra App, and drives the glasses through the Mentra Miniapp SDK (`@mentra/miniapp`). You write a bundle that runs on the user's device. Public distribution through the Mentra Miniapp Store is planned for a future release and is not available in MentraOS 3.0. > [!NOTE] > > Building a mobile app that connects directly to glasses over Bluetooth? Use the > [Bluetooth SDK docs](/bluetooth-sdk/overview) instead. > [!NOTE] > > **Looking for the `@mentra/sdk` cloud SDK?** Apps used to be cloud-hosted > servers built with `@mentra/sdk`. They're now built on-device with the Mentra > Miniapp SDK (`@mentra/miniapp`), which these docs cover. The older cloud SDK > docs live at [mentraglass.com/legacy](https://mentraglass.com/legacy). ## How a miniapp is built Every miniapp has **two layers**: * **Background**: an always-on JavaScript context (no DOM) that owns all the glasses logic: subscribing to the microphone, drawing to the display, reacting to button presses. It holds your `session`. * **UI**: an on-demand WebView (React + your styles) that opens when the user taps your miniapp's tile, and is torn down when they leave. It has no direct hardware access; it talks to the background layer over a typed message bus. This split exists so dozens of miniapps can stay resident on a phone without each one holding a heavyweight WebView in memory. See [Two-layer architecture](/app-devs/core-concepts/two-layer-architecture) for the full picture. ## Hello world This background entry shows text on the glasses when the user presses a button: ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; registerMiniapp((session) => { // Runs once the miniapp connects. `session` is your handle to the glasses. // display.render() replaces the frame; a stable id updates the element in // place. Boxes are raw device px — see session.capabilities.display. const hud = { x: 0, y: 0, w: 576, h: 288 }; session.input.onButtonPress(() => { session.display.render([{ type: "text", id: "hud", box: hud, text: "Hello from MentraOS!" }]); }); // React to what the user says. session.transcription.on((data) => { session.display.render([{ type: "text", id: "hud", box: hud, text: data.text }]); }); }); ``` You don't `new` a server or call `start()`. The host evaluates your background bundle and hands you a `session`. Everything you do with the glasses goes through [`session`](/app-devs/core-concepts/session). ## The toolchain Three npm packages, but you only ever invoke one of them directly: | Package | Role | Where it lives | | - | - | - | | [`create-mentra-miniapp`](https://www.npmjs.com/package/create-mentra-miniapp) | Scaffolds a new project. Run once via `bunx`, never installed. | N/A | | [`@mentra/miniapp`](https://www.npmjs.com/package/@mentra/miniapp) | The runtime SDK you `import` (`session.*`, React hooks). | your `dependencies` | | [`@mentra/miniapp-cli`](https://www.npmjs.com/package/@mentra/miniapp-cli) | The `mentra-miniapp` CLI (`dev` / `pack` / `release`). | your `devDependencies` | `create-mentra-miniapp` writes a `package.json` that already pins the other two, so a single `bun install` brings everything down. ## What you could build * **Live captions** that show spoken words on the HUD * **Real-time translation** for conversations * **Hands-free note-taking** triggered by a button or your voice * **Navigation** prompts on supported glasses * **Teleprompters**, fitness coaches, inspection checklists, and more ## Next steps > **[Quickstart](/app-devs/getting-started/quickstart)** > > Scaffold, run, and load a miniapp on your phone in minutes. > **[Two-layer architecture](/app-devs/core-concepts/two-layer-architecture)** > > How the background and UI layers work and talk to each other. > **[The session](/app-devs/core-concepts/session)** > > The object that connects you to the glasses. > **[Mobile Bluetooth apps](/bluetooth-sdk/overview)** > > Connecting to glasses directly over BLE instead. # Quickstart Source: https://docs.mentraglass.com/app-devs/getting-started/quickstart Scaffold a Mentra miniapp and run it on your phone in a few minutes. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). This Quickstart goes from zero to a miniapp running on your glasses. You need a phone, [Bun](https://bun.sh/docs/installation), and basic TypeScript. You can do the whole thing **without glasses**: the display just won't have anywhere to draw. > [!NOTE] > > Building a mobile app that connects directly to glasses over Bluetooth? Use the > [Bluetooth SDK Quickstart](/bluetooth-sdk/quickstart) instead. > [!NOTE] > > **Looking for the `@mentra/sdk` cloud SDK?** Apps used to be cloud-hosted > servers built with `@mentra/sdk`. They're now built on-device with the Mentra > Miniapp SDK (`@mentra/miniapp`), which these docs cover. The older cloud SDK > docs live at [mentraglass.com/legacy](https://mentraglass.com/legacy). ### Step 1: Install the Mentra App Install the Mentra App on your Android or iPhone from [MentraGlass.com/OS](https://mentraglass.com/os), sign in, and connect your glasses (if you have them). ### Step 2: Scaffold a project ```bash theme={null} bunx create-mentra-miniapp my-miniapp ``` The scaffolder asks for a project name and the kind of glasses you're targeting (**display** glasses like Even Realities G1/G2 and Vuzix Z100, or **camera** glasses like Mentra Live), then generates a ready-to-run two-layer project. ```bash theme={null} cd my-miniapp bun install ``` `bun install` pulls in [`@mentra/miniapp`](https://www.npmjs.com/package/@mentra/miniapp) (the runtime) and [`@mentra/miniapp-cli`](https://www.npmjs.com/package/@mentra/miniapp-cli) (the `mentra-miniapp` tooling) automatically. ### Step 3: Start the dev server ```bash theme={null} bun dev ``` This validates your `miniapp.json`, builds both layers, serves them over your LAN, and prints a **QR code** plus a `miniapp://dev?...` URL. It hot-reloads on save and forwards the miniapp's console logs back to your terminal. ### Step 4: Load it on your phone In the Mentra App, go to **Settings → Miniapp Developer Settings → Scan Miniapp QR Code** and scan the QR from your terminal. > [!NOTE] > > Your phone and your laptop must be on the **same Wi-Fi network**. `bun dev` > watches for Wi-Fi changes and reprints the QR if your LAN IP moves. > > On Android you can skip the Wi-Fi requirement entirely: plug the phone in and > run `bun dev --usb`. The CLI sets up `adb reverse` and points the QR at the > phone's own loopback address, so guest networks and client isolation stop > mattering. The miniapp installs and starts. In the starter project, pressing a glasses button shows text on the display, and opening the miniapp's UI tile shows a round-trip latency demo between the UI and background layers. ## Understanding the starter A scaffolded miniapp has two entry points wired together by a shared, typed channel registry: ``` my-miniapp/ ├── miniapp.json # manifest: package name, permissions, hardware, entries ├── build.ts # bundles both layers └── src/ ├── background/index.ts # always-on glasses logic: owns the session ├── ui/ # the on-demand WebView (React) └── shared/channels.ts # typed messages between the two layers ``` **Background**: owns the `session` and all glasses logic: ```typescript src/background/index.ts theme={null} import { registerMiniapp } from "@mentra/miniapp/background"; import "../shared/channels"; registerMiniapp((session) => { session.input.onButtonPress(() => { // render() replaces the whole frame — a full-canvas text element here. // Box coordinates are raw device px from session.capabilities.display. const d = session.capabilities?.display; session.display.render([ { type: "text", id: "hello", box: { x: 0, y: 0, w: d?.width ?? 576, h: d?.height ?? 288 }, text: "Hello from MentraOS" }, ]); }); // Reply to a "ping" from the UI with a "pong". session.ui.on("ping", ({ at }) => { session.ui.send("pong", { at: Date.now(), roundtripMs: Date.now() - at }); }); }); ``` **UI**: a normal React app that talks to the background over `mentra`. It has no direct hardware access: ```tsx src/ui/App.tsx theme={null} export function App() { useEffect(() => mentra.on("pong", ({ roundtripMs }) => { /* update UI */ }), []); return ; } ``` **Shared channels**: the single source of truth for what flows between the two layers, enforced at compile time: ```typescript src/shared/channels.ts theme={null} export interface Channels { ping: { at: number }; // UI → background pong: { at: number; roundtripMs: number }; // background → UI } ``` ## Next steps > **[Two-layer architecture](/app-devs/core-concepts/two-layer-architecture)** > > How background and UI work and communicate. > **[The session](/app-devs/core-concepts/session)** > > Every capability you can reach on the glasses. > **[The manifest](/app-devs/core-concepts/miniapp-manifest)** > > Permissions, hardware requirements, and entry points. > **[Test a release](/app-devs/getting-started/distribution)** > > Pack a release and install it on local test devices. # CLI reference (mentra-miniapp) Source: https://docs.mentraglass.com/app-devs/reference/cli The author CLI for developing, packing, and releasing Mentra miniapps. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). `@mentra/miniapp-cli` provides the `mentra-miniapp` command. It's installed as a dev dependency in every scaffolded project, so your `package.json` scripts call it directly (`"dev": "mentra-miniapp dev"`). ```bash theme={null} mentra-miniapp ``` Run with no arguments to print the usage table. > [!NOTE] > > The CLI is **Bun-only**: it ships as TypeScript and runs under Bun. Use `bun` > (or `bunx`), not `npx`/Node. | Command | What it does | | - | - | | [`dev`](#dev) | Dev server with hot reload; prints a QR to load the miniapp on a phone. | | [`release`](#release) | Builds, packs, and serves a QR to install a release on phones over LAN. | | [`pack`](#pack) | Validates the manifest and zips `dist/` into `-.zip`. | | [`keys`](#keys) | Creates, shows, imports and exports publisher signing keys. | | [`manifest`](#manifest) | Interactive wizard for editing `miniapp.json`. | | [`permission`](#permission-and-hardware) | Add / remove / list manifest permissions. | | [`hardware`](#permission-and-hardware) | Add / remove / list manifest hardware requirements. | | [`schema print`](#schema) | Prints the canonical `miniapp.json` JSON Schema. | ## `dev` ```bash theme={null} mentra-miniapp dev ``` Validates `miniapp.json`, builds both layers, serves them over your LAN, and prints a QR (and a `miniapp://dev?...` URL). It also runs a **dev sidecar** WebSocket on the next port up for live reload and console-log forwarding back to your terminal (non-fatal if it can't start). It watches for Wi-Fi/LAN-IP changes every 10 seconds and reprints the QR. The starting port comes from `"port"` in `miniapp.json` (default `3000`); if it or its sidecar neighbor is busy, the CLI scans upward for a free adjacent pair. **On the phone:** In the Mentra App, open **Settings → Miniapp Developer Settings → Scan Miniapp QR Code**. Phone and laptop must share Wi-Fi. `Ctrl+C` stops everything. > [!WARNING] > > `dev` is live and temporary. Keep the CLI and computer running: the Mentra App > loads the runtime bundle from that LAN server. Dev miniapps are keyed by their > manifest package name, so you can scan and test several side by side; rescanning > the same package updates only that entry. The Mentra App caches each entry's name > and icon. Use `bun run release` for an installed miniapp that works without the > computer. ## `release` ```bash theme={null} mentra-miniapp release mentra-miniapp release --no-cache # force rebuild even if the cached zip is fresh mentra-miniapp release --sign # sign with the package's stored publisher key mentra-miniapp release --signing-key key.json # sign with a key file (not stored) ``` The all-in-one path for putting a build on real devices: validate → build → pack → serve behind a QR so you can install on as many phones as you want. It reuses a cached `build/-.zip` when every source file is older than the zip, otherwise rebuilds (auto-detecting your package manager from the lockfile). It serves the bundle, manifest, and icon over HTTP on `0.0.0.0` and prints a `miniapp://release?...` QR, then stays up so multiple devices can install. Each install logs a line. Installed miniapps run offline and persist across restarts; no laptop needed after install. The bundle is unsigned unless you pass `--sign` or `--signing-key`; see [Signing](#signing). A cached ZIP is reused only when it matches the signing you asked for. ## `pack` ```bash theme={null} mentra-miniapp pack mentra-miniapp pack --no-build # zip the existing dist/ as-is mentra-miniapp pack --sign # sign with the package's stored publisher key mentra-miniapp pack --signing-key key.json # sign with a key file (not stored) ``` Produces the release artifact only (`release` calls this internally). It builds with `NODE_ENV=production`, verifies `dist/`, validates the manifest, copies `miniapp.json` + `icon.png` into `dist/`, and zips to `build/-.zip`, replacing an older ZIP only after the new one is complete. It prints the bundle's SHA-256 and its publisher key, or `unsigned`. In a future release, this ZIP will be uploaded through the Mentra Developer Console for distribution in the Mentra Miniapp Store. That distribution workflow is not available in MentraOS 3.0. > [!NOTE] > > `pack` requires the `zip` binary on `PATH` (preinstalled on macOS and most Linux; > on Windows use WSL). `build/` self-ignores from git. ### Signing Bundles are unsigned by default. `--sign` signs with the package's stored publisher key (see [`keys`](#keys)); `--signing-key ` signs with a key file without storing it, for CI. CI can also set `MENTRA_MINIAPP_SIGNING_KEY_FILE` or `MENTRA_MINIAPP_SIGNING_KEY_JSON` together with `--sign`. A signed ZIP carries `META-INF/MENTRA.SIG`: the package's Ed25519 public key and a signature over its package name, version, manifest and every file. > [!WARNING] > > Signing is permanent for a package. The first signed bundle a phone installs > records the publisher key, and from then on that phone accepts only bundles > signed with the same key. An unsigned bundle or a different key is refused until > the user uninstalls the miniapp. There is no key rotation, so back up the key > (`keys export`) before you ship a signed bundle. Signing needs a lowercase > reverse-DNS package name (letters, digits and underscores). ## `keys` ```bash theme={null} mentra-miniapp keys create com.example.myapp mentra-miniapp keys show com.example.myapp mentra-miniapp keys export com.example.myapp ./publisher-key.json mentra-miniapp keys import com.example.myapp ./publisher-key.json ``` Manages one publisher signing key per package. Keys are stored in the OS keychain, or in a mode-`0600` file under `~/.mentra/cli-v2/signing-keys/` when the keychain refuses the write. If the keychain can't be read, key commands stop without creating or changing a key; on machines without a keychain, such as CI, sign with `--signing-key`. `show` prints the key's `sha256:` fingerprint. `export` writes the private key to a new file; keep it secret and in a secure backup. ## `manifest` ```bash theme={null} mentra-miniapp manifest ``` An interactive wizard for `miniapp.json`: edit permissions and hardware requirements, show the current manifest, or finish. It saves after each confirmed change, so `Ctrl+C` never loses a saved edit. ## `permission` and `hardware` Object-verb edits that share the wizard's validation: ```bash theme={null} mentra-miniapp permission list mentra-miniapp permission add [TYPE] # interactive if TYPE omitted mentra-miniapp permission remove [TYPE] mentra-miniapp hardware list mentra-miniapp hardware add [TYPE] [LEVEL] mentra-miniapp hardware remove [TYPE] ``` Permission `TYPE`: `MICROPHONE`, `CAMERA`, `CALENDAR`, `LOCATION`, `BACKGROUND_LOCATION`, `READ_NOTIFICATIONS`, `POST_NOTIFICATIONS`. Hardware `TYPE`: `CAMERA`, `DISPLAY`, `MICROPHONE`, `SPEAKER`, `IMU`, `BUTTON`, `LIGHT`, `WIFI`; `LEVEL`: `REQUIRED`, `OPTIONAL`. See [the manifest](/app-devs/core-concepts/miniapp-manifest) for what each means. ## `schema` ```bash theme={null} mentra-miniapp schema print ``` Prints the canonical `miniapp.json` JSON Schema to stdout: useful for CI validation or IDE config. The same schema ships at `node_modules/@mentra/miniapp-cli/schema/miniapp.schema.json`, which the `$schema` line in a scaffolded `miniapp.json` references for editor autocomplete. # System Miniapp APIs Source: https://docs.mentraglass.com/app-devs/reference/system-apis APIs restricted to system miniapps for interop and privileged device controls. > [!WARNING] > > **Mentra Miniapp SDK beta** > > The SDK is in beta, so its APIs may change before general availability. > > There is currently no way to distribute a miniapp built with the Miniapp SDK. > In a future release, developers will upload miniapps through the > [Mentra Developer Console](https://console.mentraglass.com), and users will > download them from the > [Mentra Miniapp Store](https://apps.mentraglass.com). Neither service supports > Miniapp SDK distribution in MentraOS 3.0. > > Only use the Miniapp SDK if you are comfortable with these limitations. > > Developing on Mentra Live? We recommend using the > [Mentra Bluetooth SDK](/bluetooth-sdk/overview). > > The developer tools are available in the Mentra App under **Settings → > Miniapp Developer Settings**. > > Share feedback with an in-app bug report, on > [Discord](https://discord.gg/5ukNvkEAqT), or by email at > [help@mentra.glass](mailto:help@mentra.glass). > [!WARNING] > > These APIs are restricted to **system miniapps** (Mentra AI today). A regular > miniapp that calls them gets `NOT_PERMITTED`. They're documented here for > reference. To make your own miniapp callable, expose actions instead, see > [Actions](/app-devs/core-concepts/miniapp-interop). System miniapps can discover other miniapps, control their lifecycle, invoke the actions those miniapps expose, and use a small set of privileged device controls. ## Discover and control miniapps `session.miniapps` lists installed miniapps and starts or stops them. ```ts theme={null} // Compatible miniapps only by default; each action can carry input and output schemas. const apps = await session.miniapps.list(); const all = await session.miniapps.list({ includeIncompatible: true }); // start() runs a miniapp in the BACKGROUND: it spawns the background context and // reports as running, without changing phone navigation or foregrounding anything // (the WebView mounts only if the user opens it). await session.miniapps.start("com.mentra.todo"); await session.miniapps.open("com.mentra.todo"); // start and bring its phone UI forward await session.miniapps.stop("com.mentra.todo"); ``` | Method | Returns | Notes | | - | - | - | | `list(options?)` | `Promise` | `options.includeIncompatible` defaults to `false`. Each entry carries its declared `actions`. | | `start(packageName)` | `Promise` | Background-spawn; no foreground change. | | `open(packageName)` | `Promise` | Start and foreground the miniapp's phone UI. | | `stop(packageName)` | `Promise` | Stop a running miniapp. | Each `MiniappInfo` also reports `system` (the Mentra App build ships this miniapp) and, when a bundled Store maintains the installed release, `storeOwnerPackageName` or `systemStoreOwnerPackageName`. Actions declared with `"audience": "host"` are not listed. ### Installing miniapps `session.miniapps.install`, `checkInstallCompatibility`, `onInstallProgress` and `uninstall` are accepted only from a Store miniapp bundled with the Mentra App. Other callers, system miniapps included, get `NOT_PERMITTED`. The host downloads the release, checks its SHA-256, archive, publisher signature, version and compatibility, and installs it through the same installer as every other source. An automatic update passes `onlyIfStopped: true`: it never stops a running miniapp by itself and answers `APP_RUNNING` instead. With `askIfRunning: true` as well, the Mentra App stages the release and asks the user whether to update now; declining, or a question the app does not show (it is in the background, another question is open, or it asked about this miniapp within the last day), still answers `APP_RUNNING`, and the update installs once the miniapp stops. ## Invoke an action `session.actions.invoke` calls an action another miniapp exposed via `session.actions.handle`. It headless-wakes the target if it's stopped, runs the handler, and returns the result. ```ts theme={null} const res = await session.actions.invoke( "com.mentra.todo", "add_todo", { text: "milk" }, { timeoutMs: 30000 }, // default 30s, max 120s ); ``` `invoke()` rejects with `{ code, message }` where `code` is one of: | Code | Meaning | | - | - | | `NOT_PERMITTED` | The caller isn't a system miniapp. | | `APP_NOT_FOUND` | No miniapp with that package name. | | `APP_NOT_COMPATIBLE` | The target can't run on the connected glasses. | | `ACTION_NOT_FOUND` | The target doesn't declare that action. | | `NO_ACTION_HANDLER` | The action is declared but no handler is registered. | | `WAKE_FAILED` | The target couldn't be headless-woken. | | `ACTION_TIMEOUT` | The handler didn't resolve within `timeoutMs`. | | `PAYLOAD_TOO_LARGE` | Params or result exceeded 256 KB. | A thrown handler in the target surfaces its own error message to the caller. The action descriptor returned by `session.miniapps.list()` includes an optional `outputSchema`, which describes the structured result for result-aware callers. ## Privileged device controls ### Wi-Fi ADB (Mentra Live) `session.glasses.setWifiAdbState` toggles wireless debugging on Mentra Live. The preference is persisted on the glasses and reapplied at boot (default **off**). Non-system callers get `NOT_PERMITTED`. ```ts theme={null} await session.glasses.setWifiAdbState(true); // enable await session.glasses.setWifiAdbState(false); // disable ``` | Method | Returns | Notes | | - | - | - | | `setWifiAdbState(enabled)` | `Promise` | Mentra Live only; other models no-op. SYSTEM-only. | # AI IDE Integration Source: https://docs.mentraglass.com/bluetooth-sdk/ai-ide-integration Connect your AI coding assistant to MentraOS docs for smarter development > [!NOTE] > > **TL;DR** - Get your AI coding assistant to understand MentraOS instantly. Works with Zed, Cursor, VS Code, Windsurf, Claude Code, and more. ## Why Connect Your AI IDE? When you connect your AI coding assistant to MentraOS docs, it can: * Answer questions about the MentraOS SDK * Help you write smart glasses apps faster * Look up API references automatically * Debug your code with context about how MentraOS works ## MCP Server URL All setups below use this URL: ``` https://docs.mentraglass.com/mcp ``` *** ## Setup by IDE ### Zed Add this to your `settings.json` (open with `Cmd+,` or `zed: open settings`): ```json theme={null} { "context_servers": { "mentraos-docs": { "command": "npx", "args": ["-y", "mcp-remote", "https://docs.mentraglass.com/mcp"], "env": {} } } } ``` Restart Zed and you're done. > **How to verify it's working** > > Open the Agent Panel and check the MCP servers indicator in settings. A green dot means the server is active. You can also type `/` in a chat to see available prompts from the server. *** ### Cursor Add this to your Cursor MCP config file at `~/.cursor/mcp.json` (create it if it doesn't exist): ```json theme={null} { "mcpServers": { "mentraos-docs": { "url": "https://docs.mentraglass.com/mcp" } } } ``` Then **restart Cursor** and approve the MCP server if prompted. > **Alternative: Project-level config** > > You can also create `.cursor/mcp.json` in your project root to enable MentraOS docs only for that project. *** ### VS Code (with GitHub Copilot) VS Code 1.99+ has built-in MCP support with GitHub Copilot. 1. Open Command Palette (`Cmd+Shift+P` / `Ctrl+Shift+P`) 2. Run **"MCP: Add Server"** 3. Select **"HTTP"** as the transport type 4. Enter URL: `https://docs.mentraglass.com/mcp` 5. Name it `mentraos-docs` > **Manual configuration** > > Create `.vscode/mcp.json` in your project and add: > > ```json theme={null} > { > "servers": { > "mentraos-docs": { > "type": "http", > "url": "https://docs.mentraglass.com/mcp" > } > } > } > ``` *** ### Windsurf 1. Open Windsurf Settings (`Cmd+,` / `Ctrl+,`) 2. Search for "MCP" and click to open the Manage Plugins panel 3. Click **"View raw config"** to open `mcp_config.json` 4. Add the following configuration: ```json theme={null} { "mcpServers": { "mentraos-docs": { "command": "npx", "args": ["-y", "mcp-remote", "https://docs.mentraglass.com/mcp"] } } } ``` 5. Click **Refresh** in the Manage Plugins panel > **Config file location** > > * **macOS**: `~/.codeium/windsurf/mcp_config.json` > * **Windows**: `%USERPROFILE%\.codeium\windsurf\mcp_config.json` *** ### Claude Code (CLI) Run this in your terminal: ```bash theme={null} claude mcp add --transport http mentraos-docs https://docs.mentraglass.com/mcp ``` > **Verify it worked** > > Run `claude mcp list` to see your connected servers. You should see `mentraos-docs` in the list. *** ### Other MCP-Compatible Tools For tools that support remote HTTP MCP servers, use this URL directly: ``` https://docs.mentraglass.com/mcp ``` If your tool only supports command-based local MCP servers, use `mcp-remote`: ```bash theme={null} npx -y mcp-remote https://docs.mentraglass.com/mcp ``` *** ## Test It Out After setup, try asking your AI assistant: * *"How do I stream video from Mentra Live?"* * *"Show me how to use the camera API"* * *"What events can I listen to in an AppSession?"* * *"How do I listen for metric system setting changes?"* Your AI should now give you accurate, up-to-date answers using our docs. *** ## Troubleshooting > **Server not connecting** > > If your setup uses `mcp-remote`, make sure you have Node.js 18+ installed. > > ```bash theme={null} > node --version # Should be v18.0.0 or higher > ``` > **npx command not found** > > Install Node.js from [nodejs.org](https://nodejs.org/) which includes npm and npx. > **Permission errors** > > Fix your Node.js package permissions with a version manager or npm prefix configuration instead of running downloaded packages with `sudo`. *** ## What's Next? > **[Quickstart](/bluetooth-sdk/quickstart)** > > Run a complete example app and connect to Mentra Live > **[Build from Scratch](/bluetooth-sdk/build-from-scratch)** > > Add the SDK to your own mobile app # Android Source: https://docs.mentraglass.com/bluetooth-sdk/android Use the Mentra Bluetooth SDK from a native Android app. Native Android apps use the Maven artifact `com.mentraglass:bluetooth-sdk`. ## Requirements * Android min SDK `28` or newer. * Java 17. * Android Studio or Gradle. * A physical Android phone for Bluetooth, camera, microphone, direct phone photo, and direct phone WebRTC testing. ## Gradle Setup ```kotlin theme={null} // settings.gradle.kts dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) repositories { google() mavenCentral() } } ``` ```kotlin theme={null} // app/build.gradle.kts android { defaultConfig { minSdk = 28 } packaging { jniLibs { pickFirsts += "**/libc++_shared.so" } } } dependencies { implementation("com.mentraglass:bluetooth-sdk:3.3.0") } ``` The SDK is published to Maven Central. Keep `google()` and `mavenCentral()` configured. The SDK POM pulls in the companion `com.mentraglass:lc3Lib` codec artifact automatically; do not add it manually unless you have a specific override reason. The packaging rule avoids duplicate `libc++_shared.so` conflicts when your app also includes native media or camera dependencies. ## Permissions Declare the platform permissions your app uses: ```xml theme={null} ``` Request runtime permissions before scanning. Some Android 12+ devices require Location permission and Location services before BLE scan callbacks are delivered. ## Basic Flow ```kotlin theme={null} import android.content.Context import com.mentra.bluetoothsdk.Device import com.mentra.bluetoothsdk.DeviceModel import com.mentra.bluetoothsdk.GlassesRuntimeState import com.mentra.bluetoothsdk.MentraBluetoothSdk import com.mentra.bluetoothsdk.MentraBluetoothSdkCallback class GlassesController(context: Context) : MentraBluetoothSdkCallback() { private val sdk = MentraBluetoothSdk.create( context = context.applicationContext, listener = this, ) private var selectedDevice: Device? = null fun scan() { sdk.scan(DeviceModel.MENTRA_LIVE, timeoutMs = 10_000) { devices -> renderDevicePicker(devices, onSelect = { selectedDevice = it }) } } fun connect() { selectedDevice?.let { sdk.connect(it) } } fun refreshStatus() { sdk.requestVersionInfo() val glasses = sdk.getGlasses() if (glasses is GlassesRuntimeState.Connected) { val model = glasses.device.deviceModel?.deviceType ?: "glasses" val battery = glasses.battery.level?.toString() ?: "unknown" println("Connected to $model, battery=$battery%") } } override fun onGlassesChanged(glasses: GlassesRuntimeState) { // Keep app UI derived from SDK status. println("Glasses changed: $glasses") } private fun renderDevicePicker(devices: List, onSelect: (Device) -> Unit) { // Render devices in SDK-provided order and call onSelect with the user's choice. } fun close() { sdk.close() } } ``` ## Local SDK Override Use this when testing SDK source changes from a local MentraOS checkout. Run the publish command from the SDK Android module with a local Gradle install, then consume the Maven-local artifact from your app: ```bash theme={null} cd /path/to/MentraOS/mobile/modules/bluetooth-sdk/android gradle :lc3Lib:publishToMavenLocal :publishToMavenLocal -PmentraPublicSdk=true ``` Then include `mavenLocal()` before `mavenCentral()` in the app's repositories while testing the local artifact. # API Reference Source: https://docs.mentraglass.com/bluetooth-sdk/api-reference Lifecycle, status, commands, events, and shared types for the Mentra Bluetooth SDK. The Mentra Bluetooth SDK exposes the same core glasses lifecycle across Android, iOS, and React Native: * Scan for a supported glasses model. * Connect to a discovered `Device` or an app-restored default device. * Read typed lifecycle state through the public surface for your platform. * Subscribe to typed hardware events. * Send camera, stream, audio, Wi-Fi, hotspot, LED, and settings commands for Mentra Live. ## Packages And Imports | Platform | Install package | Import | | - | - | - | | Android | `com.mentraglass:bluetooth-sdk` | `import com.mentra.bluetoothsdk.*` | | iOS | `MentraBluetoothSDK` Swift package | `import MentraBluetoothSDK` | | React Native / Expo | `@mentra/bluetooth-sdk` | `import BluetoothSdk, {DeviceModels} from '@mentra/bluetooth-sdk'` | | React Native hooks | `@mentra/bluetooth-sdk` | `import {useMentraBluetooth} from '@mentra/bluetooth-sdk/react'` | | React Native photo receiver | `@mentra/bluetooth-sdk` | `import MentraPhotoReceiver from '@mentra/bluetooth-sdk/photo-receiver'` | Only documented imports are part of the supported app developer API. Undocumented package subpaths or symbols with a leading underscore can change without notice. ## Lifecycle **React Native** ```tsx theme={null} import {useBluetoothEvent, useMentraBluetooth} from "@mentra/bluetooth-sdk/react" function DeviceScreen() { const mentra = useMentraBluetooth() useBluetoothEvent("button_press", (event) => { console.log(event.buttonId, event.pressType) }) console.log(mentra.glasses.connection.state) } ``` **Android** ```kotlin theme={null} val sdk = MentraBluetoothSdk.create( context = applicationContext, config = MentraBluetoothSdkConfig(), listener = listener, ) sdk.close() ``` **iOS** ```swift theme={null} let sdk = MentraBluetoothSDK(configuration: .default) sdk.delegate = delegate sdk.invalidate() ``` Keep one SDK instance per active app session. The SDK owns Bluetooth connection state, native event delivery, and cleanup. Your app owns user identity, UI state, and whether a default device record is persisted across app restarts. ## Mentra SDK Usage Analytics The SDK reports anonymous usage events to Mentra's PostHog project by default so Mentra can understand SDK adoption, glasses connections, and how Mentra Live features are used: | Event | Fires When | | - | - | | `bluetooth_sdk_started` | Once per app runtime after the native SDK starts. | | `bluetooth_sdk_glasses_connected` | When SDK status transitions from not connected to connected. Failed scans and failed connection attempts are not counted. | | `bluetooth_sdk_glasses_identified` | Once per connection after the glasses report a valid manufacturing serial, and once per new reporting day (`America/Los_Angeles`) while that connection stays up (`event_kind=glasses_heartbeat`). Carries `glasses_device_id` (the hardware serial, not a user or phone id) and the glasses-side firmware versions of identified glasses. | | `bluetooth_sdk_glasses_disconnected` | When a connection that was reported as connected ends. Carries `session_duration_ms`, `disconnect_reason` (`user_disconnect`, `forget`, `cancel`, `bluetooth_off`, `link_lost`, or `unknown`), and the battery levels seen during the connection. | | `bluetooth_sdk_photo_taken` | When a `requestPhoto` call settles, or a photo taken with the glasses button fails. Carries `trigger`, `success`, `error_code`, `latency_ms`, the requested size, mode, compression, gallery save and transfer method, and `file_size_bytes` when known. | | `bluetooth_sdk_video_recorded` | When a recording (app- or button-started) stops or fails. Carries `trigger`, `success`, `error_code`, `duration_ms`, requested resolution and frame rate, gallery save, maximum duration, and `stopped_by_app`. | | `bluetooth_sdk_stream_started` | When a `startStream` call settles. Carries `success`, `error_code`, `protocol` (`rtmp`, `srt`, `whip`, or `unknown`, from the URL scheme only), requested video settings, `capture_audio`, `glasses_network`, and `latency_ms`. | | `bluetooth_sdk_stream_ended` | When that stream ends. Carries `protocol`, `duration_ms`, `end_state`, `stopped_by_app`, `reconnect_attempts`, `error_code`, and the last reported bitrate, frame rate, dropped frames, and temperature when the stream reported statistics. | | `bluetooth_sdk_mic_session` | When the active microphone turns off or changes. Carries `duration_ms`, `mic_source`, and `local_transcription`. | | `bluetooth_sdk_wifi_setup`, `bluetooth_sdk_hotspot_started` | When `sendWifiCredentials` or `setHotspotState(true)` settles. Carry `success`, `error_code`, and `latency_ms`. | | `bluetooth_sdk_ota_finished` | When a glasses software update completes or fails. Carries `success`, the failed step and error, `total_steps`, `duration_ms`, and the glasses versions the update started from. | | `bluetooth_sdk_request_failed` | When a glasses command times out (`error_code=request_timeout`), with the command as `operation`. At most 100 per SDK start. | Every event carries the host app's `app_version`, `app_build`, `app_build_type` (`debug` / `release`), and `app_install_source` (`play_store`, `app_store`, `testflight`, `adhoc_or_dev`, `simulator`, `sideload`, a named third-party store, `other_store`, or `unknown` when the platform gave no usable evidence). Apps that ship one package id through several lanes can add `app_environment` through the `analytics.environment` plugin option, the `com.mentra.bluetoothsdk.analytics.environment` Android metadata, or the `MentraBluetoothSdkAnalyticsEnvironment` `Info.plist` key (trimmed and lowercased; must start with a letter or digit, then `[a-z0-9_-]`, at most 32 characters). Mentra treats an install as production only when `app_install_source` is `app_store` or `play_store`, `app_build_type` is `release`, and, when the host declares a lane, `app_environment` is `prod`; store installs without a declared lane are reported as an unclassified cohort. Analytics delivery never blocks Bluetooth SDK behavior: events are submitted asynchronously. A failed upload is kept in a small on-device queue (at most 100 events, 7 days) and retried on the next successful send or SDK start (payloads PostHog rejects outright are dropped); each event carries its own id and capture time, so retries never double count. Mentra's PostHog project API key is embedded in the SDK as a public analytics write token, not a private PostHog personal API key. Apps do not need to configure PostHog to send these Mentra SDK usage events. **React Native** Disable analytics before native SDK startup through the Expo config plugin: ```json theme={null} { "expo": { "plugins": [ [ "@mentra/bluetooth-sdk", { "analytics": false } ] ] } } ``` **Android** ```kotlin theme={null} val sdk = MentraBluetoothSdk.create( context = applicationContext, config = MentraBluetoothSdkConfig( analytics = BluetoothSdkAnalyticsConfig.disabled() ), listener = listener, ) ``` Android apps can also set application metadata before SDK startup: ```xml theme={null} ``` **iOS** ```swift theme={null} let sdk = MentraBluetoothSDK( configuration: MentraBluetoothSDKConfiguration( analytics: .disabled ) ) ``` iOS apps can also set `MentraBluetoothSdkAnalyticsDisabled` to `true` in `Info.plist` before SDK startup. Apps do not configure the analytics destination; these SDK usage events are always sent to Mentra's PostHog project unless analytics are disabled. ### Update check Separately from analytics, the SDK checks for a newer SDK release every time it starts, whether or not analytics are disabled. The check is one HTTPS request to `core.mentraglass.com` carrying the SDK install id, the app identifier, the app version and build facts listed above, the SDK version, platform and surface, and the OS version. Mentra records each check. When a newer release exists, the SDK prints a warning with the update command to the developer console: Logcat on Android, the Xcode console on iOS, and a `console.warn` in React Native (a LogBox warning in debug builds). It repeats every 15 minutes while the SDK runs. Nothing is shown in your app's UI. A network that blocks `core.mentraglass.com` loses only the warning; SDK behavior is unchanged. Captured properties are intentionally narrow: | Property | Meaning | | - | - | | `event_source` | Always `mentra_bluetooth_sdk`. | | `sdk_platform` | Native platform that sends the event: `android` or `ios`. | | `sdk_surface` | Public SDK surface: `react_native`, `android`, or `ios`. | | `sdk_version` | Bluetooth SDK version. | | `app_package` / `app_bundle_identifier` | Consuming app identifier. | | `os_platform`, `os_version` | Phone OS metadata. | | `event_kind` | `sdk_started`, `glasses_connected`, `glasses_identified`, or `glasses_disconnected`. | | `phone_manufacturer`, `phone_model`, `locale`, `timezone` | Phone hardware model, language and time zone. | | `app_version`, `app_build`, `app_build_type`, `app_install_source`, `app_environment` | Host build facts described above (`app_installer_package` on Android carries the raw installer package). | | `fully_booted` | Connection and identification events; whether the connected status already reports fully booted. | | `glasses_model`, `glasses_is_simulated` | Connection and identification events; the model is included only when the SDK has a non-empty value. | | `glasses_device_id`, `glasses_device_id_type` | Identification event only; the glasses manufacturing serial (`manufacturing_serial`). | | `glasses_firmware_version`, `glasses_bes_firmware_version`, `glasses_mtk_firmware_version`, `glasses_android_version`, `glasses_app_version`, `glasses_build_number` | Identification event only; included when known. | | `glasses_model`, `glasses_device_id`, `glasses_build_number` on usage events | Photo, video, stream, mic, setup, OTA and timeout events carry the connected glasses' model, serial and build when known. | Apart from the glasses manufacturing serial, the SDK does not upload BLE MAC addresses, CoreBluetooth identifiers, Bluetooth device names, user ids, tokens, Wi-Fi names or credentials, stream, webhook or upload URLs, microphone data, photos, videos, or transcripts, and it never reads device location. Error details have URLs removed before they are sent. PostHog derives an approximate location (country, region, city) from the IP address each event is sent from. It stores a locally generated anonymous SDK install id and sends it as PostHog `distinct_id`; events include `$process_person_profile: false` so PostHog does not create person profiles for these SDK usage pings. ## Connection **React Native** ```ts theme={null} const devices = await BluetoothSdk.scan(DeviceModels.MentraLive, { timeoutMs: 10_000, onResults: (nextDevices) => renderDevicePicker(nextDevices), }) const device = await chooseDevice(devices) await BluetoothSdk.connect(device, {saveAsDefault: false}) await BluetoothSdk.setDefaultDevice(device) async function reconnectSavedDevice() { const defaultDevice = await BluetoothSdk.getDefaultDevice() if (defaultDevice) { await BluetoothSdk.connectDefault() } } async function clearSavedDevice() { await BluetoothSdk.clearDefaultDevice() } async function stopConnection() { await BluetoothSdk.cancelConnectionAttempt() await BluetoothSdk.disconnect() await BluetoothSdk.forget() } ``` **Android** ```kotlin theme={null} val devices = mutableListOf() val scanSession = sdk.scan(DeviceModel.MENTRA_LIVE, timeoutMs = 10_000) { nextDevices -> devices.clear() devices.addAll(nextDevices) renderDevicePicker(nextDevices, onSelect = { device -> sdk.connect(device) sdk.setDefaultDevice(device) }) } fun reconnectSavedDevice() { val defaultDevice = sdk.getDefaultDevice() if (defaultDevice != null) { sdk.connectDefault() } } fun clearSavedDevice() { sdk.clearDefaultDevice() } fun stopConnection() { sdk.cancelConnectionAttempt() sdk.disconnect() sdk.forget() } ``` For an explanation when a scan completes empty, use the callback overload: ```kotlin theme={null} sdk.scan( model = DeviceModel.MENTRA_LIVE, callback = object : MentraBluetoothScanCallback() { override fun onResults(devices: List) { renderDevicePicker(devices) } override fun onDiagnostic(diagnostic: ScanDiagnostic) { showScanHint(diagnostic.message) } override fun onComplete(devices: List) { finishDevicePicker(devices) } }, ) ``` `onDiagnostic` is advisory and runs before `onComplete`; it does not call `onError`. The `device_connected_on_phone` hint means a device matching the scanned model is already connected to this phone. Neither platform identifies the owning app, so another app is a possible explanation, not a confirmed cause. This check cannot see connections held by another phone. Diagnostics are skipped for cancelled or nonempty scans, simulated devices, known SDK connections or connection attempts, and unavailable Bluetooth access. Existing `ScanCallback` implementations keep their original completion behavior; implement `ScanDiagnosticCallback` or extend `MentraBluetoothScanCallback` to receive hints. iOS uses CoreBluetooth service-filtered connected peripherals; models without a supported service filter or a matching identity stay silent. **iOS** ```swift theme={null} var devices: [Device] = [] let scanSession = try sdk.scan(model: .mentraLive, timeout: 10) { nextDevices in devices = nextDevices renderDevicePicker(nextDevices) { device in do { try sdk.connect(to: device) sdk.setDefaultDevice(device) } catch { handleConnectionError(error) } } } func reconnectSavedDevice() throws { if sdk.getDefaultDevice() != nil { try sdk.connectDefault() } } func clearSavedDevice() { sdk.clearDefaultDevice() } func stopConnection() { sdk.cancelConnectionAttempt() sdk.disconnect() sdk.forget() } ``` Prefer connecting to a `Device` returned by SDK scan callbacks. If your app wants `connectDefault()` to work after restart, persist a small default-device record in app storage and restore it with `setDefaultDevice()` before calling `connectDefault()`. Optional empty-scan hints are also available on React Native and iOS: ```ts theme={null} await BluetoothSdk.scan(DeviceModels.MentraLive, { onResults: renderDevicePicker, onDiagnostic: hint => showScanHint(hint.message), }) // React hooks expose the same advisory as useBluetoothScan().diagnostic // or useMentraBluetooth().scan.diagnostic. ``` ```swift theme={null} let session = try sdk.scan( model: .mentraLive, timeout: 10, onResults: renderDevicePicker, onDiagnostic: { hint in showScanHint(hint.message) } ) ``` The React Native promise still resolves to an empty device list; hints do not reject the scan. Hosts using an older native binary simply omit the hint until rebuilt with this SDK. Clear the displayed hint when retrying, changing models, connecting, or receiving results. The Mentra App and Starter Kit examples show these hints alongside their retry controls. Use `scan()` for user-facing device pickers. The progressive result callback is for UI: render the nearby-device list every time it changes during scanning. The returned final result is for control flow: after the timeout/completion, choose a device from the last list and connect. In multi-device environments, do not auto-connect to the first nearby glasses; present an explicit picker. Mentra Live connections on iOS require Apple Notification Center Service (ANCS) authorization by default. Apps that do not use notification relay can set `requiresAncs: false` in the `connect` or `connectDefault` options to skip that system authorization requirement. The option defaults to `true` and is an intentional no-op on Android. Native iOS callers pass `ConnectOptions(requiresAncs: false)`. ## Device Identity `Device.id` is the stable app-facing key for a scan result, within the limits of the platform identifier available to the SDK. Use it as a list key, selected-device key, and persisted default-device key. Do not parse `id` for model, name, or address information. Use the typed fields instead: | Field | Meaning | | - | - | | `model` | Glasses family reported by the SDK, expected to be Mentra Live in these flows. | | `name` | Bluetooth name reported during scan. | | `address` / `identifier` | Platform device handle when available. Android commonly provides a Bluetooth address. iOS commonly provides a CoreBluetooth identifier. React Native exposes this value as `address`. | | `rssi` | Optional signal strength from scan results. It may be undefined at first discovery and appear in a later scan update when the platform reports RSSI metadata. | When the platform does not provide an address or identifier, the SDK falls back to a `model:name` key. Do not require `rssi` for picker rows. Use SDK-provided stable discovery order by default, and treat RSSI as supplemental signal-strength metadata when it is present. ## Status **React Native** ```tsx theme={null} import {useMentraBluetooth} from "@mentra/bluetooth-sdk/react" const mentra = useMentraBluetooth() ``` React Native exposes `mentra.glasses.connection` as a discriminated union: ```ts theme={null} type GlassesConnectionStatus = | {state: "disconnected"} | {state: "scanning"} | {state: "connecting"} | {state: "bonding"} | {state: "connected"; fullyBooted: boolean} ``` Use `mentra.glasses.connection.state` for link-layer progress. `fullyBooted` only exists when `state === 'connected'`. The hook exposes the React app state as `glasses`, `sdk`, and `scan`. **Android** ```kotlin theme={null} val state = sdk.getState() val glasses = sdk.getGlasses() val sdkState = sdk.getSdkState() val scan = sdk.getScanState() ``` Android exposes `GlassesRuntimeState` as `Connected` or `Disconnected`. The full state is grouped as `MentraBluetoothState(glasses, sdk, scan)`. **iOS** ```swift theme={null} let state = sdk.state let glasses = sdk.glasses let sdkState = sdk.sdkState let scan = sdk.scanState ``` iOS exposes `GlassesRuntimeState` as `.connected(...)` or `.disconnected(...)`. The full state is grouped as `MentraBluetoothState(glasses, sdk, scan)`. Status snapshots are safe to read at any time. For continuous subsystems, keep UI state derived from status callbacks or hook state. Request/response commands document their exact promise resolution and error criteria below. Public status is grouped the same way across platforms: | Field | Meaning | | - | - | | `glasses` | Connected-glasses runtime state. Includes connection/readiness, connected device identity, battery, firmware, Wi-Fi, hotspot, and signal metadata when connected. | | `sdk` | Phone-side SDK runtime state. Includes default device, gallery mode, microphone route, mic ranking, Wi-Fi scan results, scan activity, system mic availability, other Bluetooth audio status, and recent SDK log lines. | | `scan` | User-facing scan state. Includes whether a scan is active, whether controller scanning is active, and the stable-order discovered `Device[]` list. | Use `glasses.connected` / `mentra.glasses.connected` before reading connected-only fields. Native Android uses `GlassesRuntimeState.Connected`; native iOS uses `GlassesRuntimeState.connected(...)`; React Native exposes `mentra.glasses` with the same grouped concepts in React-friendly objects. ## React Native Public Surface These are the supported React Native app developer entrypoints: | Area | Methods | | - | - | | Status and subscriptions | `useMentraBluetooth`, `useBluetoothScan`, and `useBluetoothEvent` for React components; `addListener` is available as the lower-level non-React subscription API | | Default device | `getDefaultDevice`, `setDefaultDevice`, `clearDefaultDevice` | | Connection | `scan`, `startScan`, `stopScan`, `connect`, `connectDefault`, `cancelConnectionAttempt`, `disconnect`, `forget` | | Display | `displayText`, `clearDisplay`, `setDashboardContent`, `showDashboard`, `setDashboardPosition`, `setHeadUpAngle`, `setScreenDisabled` | | Wi-Fi and hotspot | `requestWifiScan`, `getSavedWifiNetworks`, `sendWifiCredentials`, `forgetWifiNetwork`, `setHotspotState` | | Camera and gallery | `requestPhoto`, `warmUpCamera`, `stopCameraWarmUp`, `queryGalleryStatus`, `setGalleryModeEnabled`, `setPhotoCaptureDefaults` (deprecated), `setVideoRecordingDefaults`, `setMaxVideoRecordingDuration`, `setCameraFov`, `setCameraFovOverride`, `releaseCameraFovOverride`, `startVideoRecording`, `stopVideoRecording`, `queryVideoRecordingStatus` | | Streaming | `startStream`, `stopStream` | | Audio | `setMicState`, `setVoiceActivityDetectionEnabled`, `setLoudnessGateEnabled`, `setPreferredMic`, `setOwnAppAudioPlaying`, `getGlassesMediaVolume`, `setGlassesMediaVolume` | | LED, version, and OTA | `rgbLedControl`, `requestVersionInfo`, `checkForOtaUpdate`, `startOtaUpdate` | React Native helper exports include `DeviceModels`, `WifiForgetResult`, `WifiForgetOutcome`, `SavedWifiNetworksResult`, `SavedWifiNetworksOutcome`, `isConnectedGlassesConnectionStatus`, `isReadyGlassesConnectionStatus`, `isBusyGlassesConnectionStatus`, `isConnectedWifiStatus`, and `isEnabledHotspotStatus`. The React subpath exports `useMentraBluetooth`, `useBluetoothScan`, and `useBluetoothEvent`. For React Native status UI, use `useMentraBluetooth()` from `@mentra/bluetooth-sdk/react`. It returns `mentra.glasses`, `mentra.sdk`, and `mentra.scan` for connection, battery, Wi-Fi, hotspot, scan, and SDK runtime state. Important defaults: * `scan(model, options)` reports progressive results through `options.onResults`, resolves with the final matching `Device[]`, and times out after 15 seconds unless `options.timeoutMs` is set. * `connect` and `connectDefault` default `saveAsDefault`, `cancelExistingConnectionAttempt`, and `requiresAncs` to `true`. `requiresAncs` only changes Mentra Live connections on iOS; Android accepts it as a no-op. * `setDashboardContent(content)` keeps the standard time, date, and battery header. It stores content for the current SDK session and renders it immediately only when the contextual dashboard is visible; otherwise it appears on the next head-up. Pass the exact empty string to restore the status-only dashboard. * `setMicState(enabled)` defaults to glasses microphone on, transcript events off, and LC3 events off. Microphone audio events are continuous while capture is enabled. Use `setVoiceActivityDetectionEnabled(...)` for glasses-side Voice Activity Detection and `setLoudnessGateEnabled(...)` for Mentra Live's center-microphone loudness gate ("Barrier"). With Barrier enabled, quiet input is replaced with silence while the glasses speaker plays, and audio frames continue. The standalone public Bluetooth SDK defaults are VAD disabled and Barrier enabled, and both settings are reapplied after reconnect. The Mentra App manages separate persisted OS-level microphone preferences. `voice_activity_detection_status` reports whether VAD is enabled, and `speaking_status` reports speaking/not-speaking when supported. Microphone audio events include the latest `voiceActivityDetectionEnabled` value. * `requestPhoto({authToken, ...})` omits the `Authorization` header when `authToken` is `null` or empty. * `requestPhoto(...)` and `warmUpCamera(...)` generate unique request IDs when `requestId` is omitted or blank. Pass an explicit `requestId` only when your app needs to know it before the response, for example to poll a predictable upload-status URL. * `requestPhoto({exposureTimeNs, ...})` uses auto exposure when `exposureTimeNs` is omitted or `null`; pass a positive nanosecond value for one-shot manual exposure. * `requestPhoto({exposureTimeNs, iso, ...})` uses `iso` only when `exposureTimeNs` enables one-shot manual exposure. Omit `iso` / pass `null` for auto ISO selection. * `requestPhoto(...)` capture fields apply to that one request. `setPhotoCaptureDefaults(...)` is deprecated; it still persists gallery-mode action-button photo defaults, but prefer per-request `requestPhoto(...)` options (e.g. `mode: "text"`) instead. * `warmUpCamera({requestId, size, exposureTimeNs, durationMs})` opens/configures Mentra Live's camera and holds it warm without capturing or uploading media. Holds default to 15 seconds and are capped at 60 seconds. Use an explicit request ID when the owner may need to call `stopCameraWarmUp(requestId)` during teardown. The promise resolves on `camera_status.state === "ready"`; the later `stopped` event marks lease expiry/teardown for listeners and does not settle the already-resolved promise. * Request/response commands now resolve from the glasses response rather than only updating local SDK state. React Native returns success-only values for commands whose raw events can also carry errors: `requestPhoto(...)` returns terminal `PhotoSuccessResponseEvent` after capture and delivery finish, `startVideoRecording(...)` returns `VideoRecordingStartedStatusEvent`, `stopVideoRecording(...)` returns `VideoRecordingStoppedStatusEvent`, and `rgbLedControl(...)` returns `RgbLedControlSuccessResponseEvent`. When `stopVideoRecording(...)` includes a webhook URL, the promise resolves after the video upload succeeds. The corresponding raw events still include error variants for listeners. * `setCameraFov({fov, roiPosition})` clamps `fov` to 62-118 degrees and accepts `roiPosition` as `"center"`, `"bottom"`, or `"top"`. You can also call `setCameraFov({preset: "narrow" | "standard" | "wide"})`; presets map to 82, 102, and 118 degrees with center ROI. On Mentra Live this persists the setting, restarts the camera, and resolves with `CameraFovResult` only after the ASG client reports the setting was applied to camera hardware after the restart cooldown. The promise rejects if the glasses report an error, persist the setting without hardware application, or time out. Treat FOV as a framing/ROI control; output resolution and effective detail can vary by capture path, firmware, and camera mode. * `startStream` optional `video` and `audio` configs are omitted unless supplied, so the connected glasses use their model defaults. Stream video input fields are `width`, `height`, `bitrate`, and `fps`; stream status reports the resolved effective frame rate as `resolvedConfig.video.fps`. The SDK sends stream keep-alives automatically and reports timeout/error state through `stream_status`. * Photo capture, video recording, and streaming always enable the camera light as a privacy indicator. * The SDK sends `set_system_time` once shortly after each glasses connection becomes ready, after the initial command burst has drained. The guard resets on disconnect, so reconnecting synchronizes again. There is no periodic clock correction during a long-lived connection; compare `requestVersionInfo().systemTimeMs` with the phone clock if your app needs to monitor later skew. ## Head-Up Dashboard Content Use `setDashboardContent` to place persistent, glanceable text below the standard dashboard status header. The SDK keeps the raw status placeholders and expands them every time the dashboard renders, so time, date, and battery remain current. The call does not force the dashboard open. If the contextual dashboard is hidden, the content appears on the next head-up. **React Native** ```ts theme={null} await BluetoothSdk.setDashboardContent('Next meeting at 2 PM'); await BluetoothSdk.setDashboardContent(''); ``` **Android** ```kotlin theme={null} sdk.setDashboardContent("Next meeting at 2 PM") sdk.setDashboardContent("") ``` **iOS** ```swift theme={null} await sdk.setDashboardContent("Next meeting at 2 PM") await sdk.setDashboardContent("") ``` Only the exact empty string resets the custom content; whitespace and newlines are otherwise preserved. Dashboard content lasts for the current SDK process. Persist it in the host app and reapply it after launch when it must survive an app restart. Gate this API by display capability because Mentra Live does not have a display. ## Promise Results And Errors One-shot commands resolve only after the matching ASG client response arrives. They reject/throw when the ASG response is an error, when the SDK cannot send the command, when an incompatible command is already in flight, or when the matching response times out. Keep listeners for streaming progress and status; use the returned value for the one-shot decision. React Native exposes the narrowest return types for clear branching after `await`. Android and iOS async APIs throw `BluetoothException` / `BluetoothError` for the same error cases, so successful calls return the success payload even when the native event struct can also represent raw listener errors. | API | Resolves With | Rejection / Throw Criteria | | - | - | - | | `requestWifiScan()` | Final `WifiSearchResult[]` from `wifi_scan_result` where `scanComplete: true`, including `[]` when no networks are found. | Scan response timeout, send failure, or another Wi-Fi scan already in flight. | | `getSavedWifiNetworks()` | `SavedWifiNetworksResult`. `outcome: "confirmed"` includes exact saved SSIDs from a reliable enumeration backend. `outcome: "unsupported"` includes `networks: []` on K900 and legacy firmware. Whitespace is preserved so a returned SSID can be passed back to `forgetWifiNetwork`. Capability-unknown calls wait for `version_info_1`; negotiation and response share one 15-second deadline. Only protocol version 1 is supported. | Capability-negotiation timeout, request timeout, session reset/disconnect, send failure, or another saved-list request in flight. Glasses-reported `unsupported` and `failed` are typed results, not transport exceptions. | | `sendWifiCredentials(ssid, password)` | `WifiStatusChangeEvent` whose state is `connected` for the requested SSID. Success typically resolves within a few seconds. | Glasses-reported provisioning failure, with the reason as the error code: `wrong_password` (typically \~7s), `connect_timeout` (\~12s, network unreachable or join never completed), or `connected_to_other_network` (the glasses ended up on a different SSID than requested). Also request timeout, send failure, or another Wi-Fi status command in flight. Routine `disconnected` events during association do **not** reject — the glasses briefly report disconnected while switching networks. Glasses running older software never send failure reasons; failed attempts there surface as request timeouts. | | `forgetWifiNetwork(ssid)` | `WifiForgetResult` with `outcome` of `confirmed`, `dispatched`, `not_found`, `unsupported`, `failed`, or `legacy_unverified`. Capability-unknown calls wait for `version_info_1`; negotiation and response share one 15-second deadline. A discovery timeout never selects legacy. Modern results are correlated by request ID, glasses session ID, protocol version, and exact requested SSID; `wifi_status_change` cannot settle them. K900 returns `dispatched`, meaning its asynchronous SystemUI broadcast was queued, not that removal was verified. `connected` is an optional diagnostic snapshot and is omitted, rather than fabricated as `false`, if ASG cannot read link state. Legacy firmware returns `legacy_unverified` immediately after accepted native dispatch, without a fabricated link snapshot. Transport correlation remains internal. This return type intentionally replaces the previous `WifiStatusChangeEvent`. | Empty SSID, capability-negotiation timeout, request timeout, session reset/disconnect, send failure, or another Wi-Fi status command in flight. Glasses-reported semantic failures are typed results. | | `setHotspotState(enabled)` | `HotspotStatusChangeEvent` matching the requested enabled/disabled state. | `hotspot_error`, timeout, send failure, or another hotspot command in flight. | | `requestVersionInfo()` | `VersionInfoResult` from the ASG `version_info` response. | Timeout, send failure, or another version query in flight. | | Gallery capture settings | `SettingsAckSuccessEvent` with a non-failure status such as `applied` or `ready`; the SDK local settings store updates only after this ASG ack resolves. | `settings_ack.status` of `error`, `failed`, `failure`, or `rejected`; timeout; or send failure. | | `setCameraFov(...)` | `CameraFovResult` with `fov`, `roiPosition`, `requestId`, and `timestamp` after the ASG client reports the setting was applied to camera hardware after the restart cooldown; the SDK local FOV setting updates only after this result resolves. | Glasses-side FOV/settings error, persist-only/no hardware-application ack, timeout, or send failure. | | `queryGalleryStatus()` | `GalleryStatusEvent`, including `cameraBusy` and optional `cameraBusyReason`. | Timeout, send failure, or another gallery status query in flight. | | `requestPhoto(...)` | Terminal response after capture and delivery finish: React Native `PhotoSuccessResponseEvent`; Android/iOS successful `PhotoResponseEvent`. The success includes `uploadUrl`, and may include webhook-returned `photoUrl`, `statusUrl`, `contentType`, or `fileSizeBytes`. Progress follows through `photo_status`. | Raw `photo_response.state === "error"` from glasses/phone, fallback upload failure, terminal response timeout, or send failure. | | `warmUpCamera(...)` | `CameraStatusEvent` whose `state` is `ready`, meaning the camera is open/configured and held warm for the requested TTL. Progress follows through `camera_status`; `stopped` is a listener-only lifecycle event after `ready`. | `camera_status.state === "error"` such as `camera_busy`, `camera_restart_cooldown`, `photo_capture_in_progress`, `ble_transfer_in_progress`, `media_capture_service_unavailable`, or `camera_warm_up_failed`; timeout; or send failure. | | `startVideoRecording(...)` | React Native `VideoRecordingStartedStatusEvent` with `status: "recording_started"`; Android/iOS successful `VideoRecordingStatusEvent` with `status: "recording_started"`. | `success: false` statuses such as `already_recording`, timeout, send failure, or duplicate request ID. | | `stopVideoRecording(requestId, webhookUrl?, authToken?)` | React Native `VideoRecordingStoppedStatusEvent` with `status: "recording_stopped"`; Android/iOS successful `VideoRecordingStatusEvent` with `status: "recording_stopped"`. When `webhookUrl` is supplied, the promise waits for the video upload to finish successfully. | `success: false` statuses such as `not_recording`, video upload failure, timeout, send failure, or duplicate request ID. | | `queryVideoRecordingStatus(requestId)` | `VideoRecordingStatusEvent` with `status: "recording_status"`, the current `data.recording` value, and elapsed `data.duration_ms` when available. | Disconnected glasses, timeout, send failure, or a duplicate video request ID. | | `startStream(...)` | `StreamStatusEvent` for the requested stream once `status` is `streaming`. | `stream_status` error/reconnect failure, timeout, or send failure. | | `stopStream()` | `StreamStatusEvent` once `status` is `stopped`; an already-stopped / not-streaming glasses reply resolves as a normalized stopped event. | Real stop error, timeout, send failure, or another stream stop in flight. | | `rgbLedControl(...)` | React Native `RgbLedControlSuccessResponseEvent`; Android/iOS successful `RgbLedControlResponseEvent`. | Raw `rgb_led_control_response.state === "error"`, timeout, or send failure. | | `checkForOtaUpdate()` | `boolean`, true when the configured OTA manifest has an ASG APK, MTK, or BES update for the connected glasses; false only when the manifest was checked successfully and no update is available. | Rejects when the glasses are disconnected, version info is unavailable, the manifest cannot be fetched, or the manifest response is invalid/missing required ASG app version fields. | | `startOtaUpdate()` | `OtaStartAckEvent` from `ota_start_ack`. | Known glasses battery below 5% (`battery_low` before ack), start-ack timeout, send failure, or another OTA start in flight. | ## Gallery Mode Mentra Live has a gallery mode for the right action button. When gallery mode is enabled, a short press takes a photo, a long press starts video recording, and a short press stops the active video recording. Button and touch events are still reported to the SDK. Use `setGalleryModeEnabled(true)` to enable local button capture, and `setGalleryModeEnabled(false)` to report button events without triggering local gallery capture while the glasses are connected. These setting calls return `SettingsAckSuccessEvent` from the ASG client and reject on failure statuses. Raw `settings_ack` listeners still receive `SettingsAckEvent` because listener events can include both success and failure statuses. The SDK can also configure the capture settings used by gallery mode: | Setting | API | | - | - | | Photo capture defaults (deprecated) | `setPhotoCaptureDefaults(...)` — prefer per-request `requestPhoto(...)` | | Video resolution and frame rate | `setVideoRecordingDefaults({width, height, fps})` | | Maximum local video length | `setMaxVideoRecordingDuration(minutes)` | | Camera field of view | `setCameraFov(...)` | `requestPhoto(...)` accepts optional `requestId` plus `size`, `webhookUrl`, `authToken`, `compress`, `save`, `sound`, `exposureTimeNs`, `iso`, `aeExposureDivisor`, `isoCap`, `zsl`, `mfnr`, `noiseReduction`, `edgeEnhancement`, `ispDigitalGain`, and `ispAnalogGain`. ZSL buffering and MFNR still capture are independent flags, both enabled by default; pass `zsl: false` and/or `mfnr: false` to opt out. Photo, video, and stream capture always enable the camera light as a privacy indicator. Manual `iso` is used only when `exposureTimeNs` enables manual exposure. `warmUpCamera(...)` accepts optional `requestId` plus `size`, `exposureTimeNs`, and `durationMs`. Omit or pass a non-positive `durationMs` for the default 15 second warm hold; values above 60 seconds are capped. `camera_status` reports `warming`, `ready`, `stopped`, or `error`; the method resolves on `ready` and rejects on `error`. Call `stopCameraWarmUp(requestId)` to release a known lease. Cancelling before `ready` rejects the pending warm-up with `camera_warm_up_cancelled`; cancelling after `ready` emits `stopped`. Compatible ready leases share the camera and expire independently. `setPhotoCaptureDefaults(...)` is **deprecated**. It still accepts `size`, `zsl`, `mfnr`, `noiseReduction`, `edgeEnhancement`, `ispDigitalGain`, `ispAnalogGain`, `aeExposureDivisor`, `isoCap`, `compress`, `sound`, and `resetCaptureTuning`, but prefer per-request `requestPhoto(...)` options (for example `mode: "text"` for text sensor size/crop, or explicit `aeExposureDivisor`) instead of sticky button-photo presets. Its independent `zsl` and `mfnr` values are persisted for physical camera-button photos; when either is omitted, that field inherits the corresponding global device default. Omitted fields leave existing action-button photo defaults unchanged. `resetCaptureTuning: true` clears the optional action-button tuning overrides without changing global ZSL/MFNR settings or photo size unless `size` is also provided. ## Common Commands | Area | Commands | | - | - | | Connection | `scan`, `startScan`, `stopScan`, `connect`, `connectDefault`, `disconnect`, `forget` | | Display | `displayText`, `clearDisplay`, `setDashboardContent`, `showDashboard`, dashboard position, and head-up angle | | Wi-Fi and hotspot | `requestWifiScan`, `getSavedWifiNetworks`, `sendWifiCredentials`, `forgetWifiNetwork`, `setHotspotState` | | Camera | `requestPhoto`, gallery mode, photo capture defaults, video recording defaults, camera field of view | | Streaming | `startStream`, `stopStream` | | Audio | `setMicState`, Voice Activity Detection setting and events, audio callbacks, local transcription, media volume | | Settings | Gallery mode, camera options, microphone route, Wi-Fi, hotspot, and OTA | | LEDs | `rgbLedControl` on supported glasses | | System | `requestVersionInfo`, `checkForOtaUpdate`, `startOtaUpdate` | Mentra Live has camera, microphone, speaker, Wi-Fi, LED, and OTA capabilities. Gate UI by connection readiness, permissions, and the latest SDK status. ## OTA Updates Each Bluetooth SDK release has a matching Mentra Live glasses software release with the same version number. When your app checks for an update, the SDK uses the OTA manifest for its own release. For the current `3.3.0` software and complete installation instructions, see [Update Mentra Live](/bluetooth-sdk/software-update). For React Native apps, use Mentra Engine instead of assembling these primitives yourself. `MentraLiveOtaFlow` supplies the exact Mentra App pages, and `useMentraLiveOta` supports custom presentation over the same tested coordinator. See [Update Mentra Live](/bluetooth-sdk/software-update#add-the-ota-requirement-to-your-app). The APIs below are low-level building blocks used by native Android/iOS apps and by Mentra Engine. They are not a second React Native OTA coordinator. Mentra Live OTA installation is glasses-owned. The SDK checks its release-specific manifest on the phone, then sends `ota_start` with that same manifest URL when the user accepts the update: | API | Purpose | | - | - | | `checkForOtaUpdate()` | Fetch the configured manifest and resolve `true` when an ASG APK, MTK, or BES update is available. | | `startOtaUpdate()` | Send `ota_start` with the configured manifest URL after your app presents the update and the user accepts it. Resolves with `OtaStartAckEvent` from the ASG client. | | `queryOtaStatus()` | Query and return the active OTA session status after a reconnect or interrupted response. React Native only. | Use the boolean check result for update prompts and keep `ota_status` listeners/delegates for install progress and terminal `complete` / `failed` state. Availability is exposed through `checkForOtaUpdate()` rather than a public event; React Native receives `ota_start_ack` and `ota_status` events during the install flow. Android listeners receive `onOtaStartAck` and `onOtaStatus`; iOS delegates receive `.otaStartAck` and `.otaStatus` through `BluetoothEvent`. OTA requires Mentra Live glasses firmware that supports the ASG OTA protocol and network access from the glasses. During install, normal BLE traffic can be interrupted and the glasses may restart; keep the app connected and avoid sending unrelated commands until `ota_status.status` is `complete` or `failed`. Treat `complete` as terminal even though the connection may drop immediately after it: when the final step of an update is the BES firmware, the glasses power-cycle right after emitting `complete`, so no further `ota_status` follows. If the BES firmware transfer step stalls on the glasses side, it aborts itself after about 30 seconds and surfaces as `failed`; the APK and MTK steps use their own timeout paths. On such an abort the BES stays on its current firmware — earlier APK or MTK steps of the same update may already have completed — and retrying the whole update from the app is the intended recovery. ## Events React Native components should use `useBluetoothEvent()` for hardware events. The hook keeps the callback typed and removes the native subscription when the component unmounts. Native apps receive the same event categories through listener/delegate methods: **React Native** ```tsx theme={null} import {useBluetoothEvent} from "@mentra/bluetooth-sdk/react" export function HardwareEventLogger() { useBluetoothEvent("button_press", (event) => console.log(event)) useBluetoothEvent("touch_event", (event) => console.log(event)) useBluetoothEvent("photo_status", (event) => console.log(event.status, event.resolvedConfig)) useBluetoothEvent("stream_status", (event) => console.log(event.resolvedConfig?.video?.fps)) useBluetoothEvent("ota_status", (event) => console.log(event.overall_percent)) useBluetoothEvent("speaking_status", (event) => console.log(event.speaking)) useBluetoothEvent("mic_pcm", (event) => { console.log(event.sampleRate, event.bitsPerSample, event.channels, event.encoding) console.log(event.pcm) }) return null } ``` **Android** ```kotlin theme={null} class GlassesEvents : MentraBluetoothSdkCallback() { override fun onButtonPress(event: ButtonPressEvent) { Log.d("Mentra", "${event.buttonId} ${event.pressType}") } override fun onPhotoResponse(event: PhotoResponseEvent) { Log.d("Mentra", "Photo request ${event.requestId}: ${event.response.state}") } override fun onMicPcm(event: MicPcmEvent) { Log.d("Mentra", "${event.sampleRate} Hz ${event.bitsPerSample}-bit PCM") } override fun onOtaStatus(event: OtaStatusEvent) { Log.d("Mentra", "OTA ${event.status}: ${event.overallPercent}%") } } ``` **iOS** ```swift theme={null} final class GlassesEvents: MentraBluetoothSDKDelegate { func mentraBluetoothSDK(_ sdk: MentraBluetoothSDK, didReceive event: BluetoothEvent) { switch event { case let .buttonPress(button): print("\(button.buttonId) \(button.pressType)") case let .photoResponse(photo): print("Photo request \(photo.requestId): \(photo.response.state.rawValue)") case let .otaStatus(status): print("OTA \(status.status): \(status.overallPercent)%") default: break } } func mentraBluetoothSDK(_ sdk: MentraBluetoothSDK, didReceiveMicPcm event: MicPcmEvent) { print("\(event.sampleRate) Hz \(event.bitsPerSample)-bit PCM") } } ``` For non-React modules, `BluetoothSdk.addListener(...)` is the low-level subscription API. Keep the returned subscription and call `remove()` when the listener is no longer needed. The React Native event surface is typed through `BluetoothSdkEventMap`. These are the public event names accepted by `useBluetoothEvent()` and `BluetoothSdk.addListener()`: | Event name | Payload type | When it fires | | - | - | - | | `log` | `LogEvent` | SDK diagnostic log line. | | `device_discovered` | `Device` | A supported glasses device is discovered during scan. | | `default_device_changed` | `{device?: Device}` | The SDK default device changes. | | `glasses_not_ready` | `GlassesNotReadyEvent` | A command needs ready glasses but the connected device is not ready. | | `button_press` | `ButtonPressEvent` | Glasses button press. | | `touch_event` | `TouchEvent` | Glasses touch or swipe gesture. | | `head_up` | `HeadUpEvent` | Head-up state changes. | | `voice_activity_detection_status` | `VoiceActivityDetectionStatusEvent` | Glasses-side Voice Activity Detection is enabled or disabled. | | `speaking_status` | `SpeakingStatusEvent` | Glasses-side Voice Activity Detection reports speaking or not speaking. | | `battery_status` | `BatteryStatusEvent` | Battery update from glasses. | | `local_transcription` | `LocalTranscriptionEvent` | SDK local transcription text update. | | `wifi_status_change` | `WifiStatusChangeEvent` | Glasses Wi-Fi connection state changes. When the event is the verdict of a failed `sendWifiCredentials` attempt it carries an `error` reason (`wrong_password`, `connect_timeout`, `connected_to_other_network`); the reason describes the attempt, not the link, so `connected_to_other_network` arrives on a `connected` state. Routine link updates carry no `error`. | | `wifi_scan_result` | `WifiScanResultEvent` | Wi-Fi scan results. Intermediate events can have `scanComplete: false`; the final event has `scanComplete: true` and is what `requestWifiScan()` awaits. | | `wifi_forget_result` | `WifiForgetResultEvent` | Discriminated raw result union. `mode: "modern"` is session-correlated and carries semantic `outcome`; K900 `outcome: "dispatched"` does not prove the asynchronous vendor receiver removed the credential. `mode: "legacy"` preserves an older frame's `dispatched` boolean without inventing missing correlation fields. In either mode, optional `connected` is omitted when the wire frame has no diagnostic link snapshot. Delivery is at least once; raw listeners should deduplicate modern retries by `(requestId, sid)`. Promise APIs ignore duplicates after the first matching terminal result. | | `saved_wifi_networks` | `SavedWifiNetworksEvent` | Session-correlated saved-SSID result from a backend that advertised support. Delivery is at least once; raw listeners should deduplicate retries by `requestId` and `sid`. Promise APIs ignore duplicates after the first matching terminal result. | | `version_info` | `VersionInfoEvent` | Glasses version metadata response. `version_info_1` can include `wifiForgetResultVersion` and `savedWifiNetworksVersion`; absent/0 means unsupported for that glasses session. Also returned by `requestVersionInfo()`. | | `hotspot_status_change` | `HotspotStatusChangeEvent` | Glasses hotspot state changes. | | `hotspot_error` | `HotspotErrorEvent` | Hotspot operation fails. | | `photo_response` | `PhotoResponseEvent` | Terminal photo result. Success means capture and webhook delivery completed; error means the glasses or phone rejected/failed the request. `requestPhoto(...)` resolves with the success case and rejects on the error case; progress follows through `photo_status`. | | `photo_status` | `PhotoStatusEvent` | Photo capture progress changes. May include `resolvedConfig`, `requestedCaptureConfig`, `meteredPreview`, or `captureMetadata` depending on status. | | `camera_status` | `CameraStatusEvent` | Camera warm-up lifecycle. `warmUpCamera(...)` resolves on `ready`, rejects on `error`, and listeners can observe `warming` and `stopped` progress. `stopped` is emitted after `ready` when the lease expires or the camera closes; pending warm-ups receive `error` instead if teardown happens before `ready`. | | `video_recording_status` | `VideoRecordingStatusEvent` | Video recording start/stop succeeds or fails, or a status query reports `status: "recording_status"` with `data.recording`. `startVideoRecording(...)` resolves from `recording_started`; `stopVideoRecording(...)` resolves from `recording_stopped` unless a webhook upload was requested, in which case it waits for upload success. | | `media_success` / `media_error` | `MediaUploadEvent` | Raw glasses media upload completion events. Video webhook stops use these events to resolve or reject the `stopVideoRecording(...)` promise after recording has stopped. | | `gallery_status` | `GalleryStatusEvent` | Gallery content/camera-busy status changes. `cameraBusyReason` is included when the glasses report a busy camera reason such as `video` or `stream`. | | `settings_ack` | `SettingsAckEvent` | Raw gallery/button/FOV setting acknowledgements from the glasses. Gallery and button setting methods return `SettingsAckSuccessEvent` and reject failure statuses; `setCameraFov(...)` resolves a friendlier `CameraFovResult` after the raw ack reports camera hardware application. | | `compatible_glasses_search_stop` | `CompatibleGlassesSearchStopEvent` | Compatible-glasses search stops for a model. | | `swipe_volume_status` | `SwipeVolumeStatusEvent` | Swipe-volume setting changes. | | `switch_status` | `SwitchStatusEvent` | Glasses switch status changes. | | `rgb_led_control_response` | `RgbLedControlResponseEvent` | RGB LED command succeeds or fails. `rgbLedControl(...)` resolves with the success case and rejects on the error case. | | `pair_failure` | `PairFailureEvent` | Bluetooth pairing fails. | | `audio_pairing_needed` | `AudioPairingNeededEvent` | The phone needs Bluetooth audio pairing for the device. | | `audio_connected` | `AudioConnectedEvent` | Bluetooth audio connects. | | `audio_disconnected` | `AudioDisconnectedEvent` | Bluetooth audio disconnects. | | `mic_pcm` | `MicPcmEvent` | PCM microphone frame arrives. | | `mic_lc3` | `MicLc3Event` | LC3 microphone frame arrives. | | `mic_health` | `MicHealthEvent` | Native LC3 processing detects a `sequence_gap` or `decode_failure`. Includes cumulative current-connection counters and last LC3/PCM receipt timestamps. | | `stream_status` | `StreamStatusEvent` | Camera stream lifecycle, reconnect, or error state changes. May include `resolvedConfig` with effective stream settings: encoded output size in `video.width` / `video.height`, native camera capture size in `video.captureWidth` / `video.captureHeight`, video bitrate and `fps`, plus audio bitrate, sample rate, echo cancellation, and noise suppression. | | `ota_start_ack` | `OtaStartAckEvent` | Mentra Live acknowledges `startOtaUpdate()`. Also returned by `startOtaUpdate()`. | | `ota_status` | `OtaStatusEvent` | OTA step, phase, status, progress, or error changes. | For request/response commands, prefer the returned value from the method, such as the `PhotoSuccessResponseEvent` returned by React Native `requestPhoto(...)` or the operation-specific video result returned by React Native `startVideoRecording(...)` / `stopVideoRecording(...)` / `queryVideoRecordingStatus(...)`. `requestPhoto(...)` waits for the terminal `photo_response`: it resolves after the photo is captured and delivered to the webhook, and rejects if the glasses reject the request, the phone-side fallback upload fails, or the terminal response times out. `stopVideoRecording(...)` keeps the same end-to-end shape for video webhook stops: it waits for recording stop plus upload success, and rejects on upload failure. Photo capture/upload/Bluetooth fallback progress follows through `photo_status`; raw video upload completion arrives through `media_success` and `media_error` for listeners that need the glasses event stream directly. Use event listeners for ongoing hardware streams and status updates: for example, `useBluetoothEvent('mic_pcm', ...)` receives a `MicPcmEvent`. `MicPcmEvent` includes `sampleRate`, `bitsPerSample`, `channels`, `encoding`, and `voiceActivityDetectionEnabled`; `MicLc3Event` includes `sampleRate`, `channels`, `encoding`, `frameDurationMs`, `frameSizeBytes`, `bitrate`, `packetizedFromGlasses`, and `voiceActivityDetectionEnabled`. `mic_health` reports native sequence/decode failures that cannot be reconstructed from PCM and includes `sequenceGapEvents`, `decodeFailures`, `lastLc3ReceivedAt`, and `lastPcmProducedAt`. `speaking_status` is separate from microphone audio frames so apps can use continuous audio while still reacting to speech activity. Public React Native event payload fields usually use camelCase. OTA events intentionally mirror the glasses firmware field names, such as `overall_percent` and `version_name`. For example, touch events expose `deviceModel` and `gestureName`, successful photo responses expose `uploadUrl` and may include webhook-returned `photoUrl`, `statusUrl`, `contentType`, and `fileSizeBytes`, successful media upload events expose `requestId`, `mediaUrl`, and `mediaType`, hotspot errors expose `errorMessage`, Wi-Fi scan events expose `scanComplete`, and gallery status exposes `hasContent`, `cameraBusy`, and optional `cameraBusyReason`. ### Photo Status Metadata `PhotoStatusEvent` reports progress through capture and transfer. Capture metadata is attached to the stage where the glasses know that data: | Status | Optional field | Description | | - | - | - | | `configuring` | `resolvedConfig` | Effective JPEG dimensions, quality, requested size, source (`sdk` or `button`), transfer method, compression, and manual exposure fields when present. | | `capturing` | `requestedCaptureConfig` | Camera2 still request values submitted to the HAL, such as exposure time, ISO, frame duration, AE mode, AE lock, exposure compensation, target FPS range, AF mode, and ZSL. | | `capturing` | `meteredPreview` | Latest preview auto-exposure estimate before the still capture, including exposure time, ISO, and a light proxy when available. | | `captured` | `captureMetadata` | Actual still capture result returned by the HAL, including applied exposure time, ISO, frame duration, AE state/name, noise reduction mode, edge mode, sensor timestamp, and MFNR hint when available. | Transport statuses such as `uploading`, `compressing`, `ble_fallback_compression`, `ready_for_transfer`, and `transferring` describe upload or BLE progress only and do not carry capture metadata. `ble_fallback_compression` means the direct Wi-Fi/webhook upload failed and the glasses are compressing the already-captured photo for Bluetooth fallback delivery. Apps should read actual capture values from `event.captureMetadata` on the `captured` status, not from upload statuses. Local action-button photos emitted by the glasses use the same `photo_status` shape when the phone SDK is connected. Those local captures set `resolvedConfig.source` to `button` and `resolvedConfig.transferMethod` to `local`. Android and iOS expose typed callbacks/delegate methods instead of the React Native string event API. Android uses `MentraBluetoothSdkListener` methods such as `onStateChanged`, `onGlassesChanged`, `onSdkStateChanged`, `onScanChanged`, `onDeviceDiscovered`, `onButtonPress`, `onSpeakingStatus`, `onPhotoResponse`, `onPhotoStatus`, `onMicPcm`, `onMicHealth`, `onStreamStatus`, and `onOtaStatus`. iOS uses `MentraBluetoothSDKDelegate` methods such as `mentraBluetoothSDK(_:didUpdate:)`, `mentraBluetoothSDK(_:didUpdateGlasses:)`, `mentraBluetoothSDK(_:didUpdateSdkState:)`, `mentraBluetoothSDK(_:didUpdateScan:)`, `mentraBluetoothSDK(_:didDiscover:)`, `mentraBluetoothSDK(_:didReceive:)`, `mentraBluetoothSDK(_:didReceiveMicPcm:)`, and `mentraBluetoothSDK(_:didReceiveMicLc3:)`; microphone health arrives through `didReceive` as `.micHealth`, while OTA install progress arrives as `.otaStartAck` or `.otaStatus`. Microphone audio callbacks use `MicPcmEvent` and `MicLc3Event` objects with the same metadata as React Native. | Event area | Examples | | - | - | | Input | Button press, touch, swipe, head-up | | Device state | Battery, case, Wi-Fi, hotspot, connection, RSSI | | Camera | Photo request result, camera warm-up state, gallery state, camera settings | | Streaming | Stream initializing, active, stopped, error | | OTA | Update available, start acknowledged, step/progress status | | Audio | Microphone PCM, LC3, local transcription, audio route state | | Diagnostics | SDK log events | ### Combined JavaScript and Native Logs The React Native SDK automatically forwards Android and iOS SDK diagnostics to the JavaScript console after import. Native entries are prefixed with `[native:android]` or `[native:ios]` and appear alongside your app's JS logs. The `log` event remains available for custom consumers; an additional listener that calls `console.log` will duplicate the SDK's automatic output. Credential-bearing messages are redacted before console output, and each native message is capped at 16,384 characters. The Mentra App attaches this combined console stream to incident reports using its existing phone log buffer and retention limits. Native console output remains available too. On Android, forwarding to JavaScript is capped at 100 messages per second. Each forwarded message holds a JNI global reference until the JavaScript thread drains it, so an unbounded log source can exhaust the process-wide reference table and abort the app rather than merely slow it down. Messages above the cap are withheld from the JavaScript stream only — they are still written to the native console — and the next forwarded message is preceded by a notice saying how many were withheld. Per-frame microphone payload events are not traced at all; audio faults are still reported through the `mic_health` event. This covers SDK-owned diagnostics while the JavaScript runtime is active. It does not collect arbitrary OS or third-party native logs or replay logs from before the subscription. Rebuild the native app to pick up the routing changes. ## Version Fields Call `requestVersionInfo()` after connection when your app wants the glasses to refresh version metadata. Updated values arrive through the normal glasses-status callback and are also available in the next status snapshot. | Field | Meaning | | - | - | | `firmwareVersion` | Generic glasses firmware version when the connected model reports one. | | `deviceFirmwareVersion` | Device firmware version for models that report device info as a structured payload. | | `leftFirmwareVersion` / `rightFirmwareVersion` | Per-side firmware versions for glasses that report left/right firmware separately. | | `besFirmwareVersion` | Mentra Live BES firmware version. | | `mtkFirmwareVersion` | Mentra Live MTK/system OTA firmware version. | | `appVersion` | Glasses-side companion app version. On Mentra Live this is the ASG client APK version, not firmware. | | `androidVersion` | Android OS version on Android-based glasses. This is not firmware. | Different glasses models expose different version fields, so apps should prefer the generic firmware field when present, then fall back to model-specific firmware fields. Keep app and OS versions visibly labeled as app/OS versions. # Audio Source: https://docs.mentraglass.com/bluetooth-sdk/audio Use microphone input and route normal phone audio playback to supported glasses speakers. The Bluetooth SDK exposes microphone input controls, microphone audio callbacks, local transcription events, and Mentra Live media-volume helpers. Speaker playback is separate: your app plays audio with normal Android, iOS, or React Native audio APIs, and the phone OS routes that audio to Mentra Live when the glasses are connected as the Bluetooth media output. > [!NOTE] > > Mentra Live has microphone input and speaker playback support. ## Microphone Input Use `setMicState` when your app wants microphone audio from the connected glasses or phone microphone path. `enabled` turns microphone capture on or off. `useGlassesMic` selects the glasses microphone when `true`, or the phone microphone when `false`. Microphone audio events are emitted continuously from the selected microphone path while capture is enabled. The SDK does not apply phone-side Voice Activity Detection gating to PCM or LC3 events. Glasses-side Voice Activity Detection can be enabled or disabled with `setVoiceActivityDetectionEnabled(...)`; the current setting is reported through `voice_activity_detection_status`. Mentra Live's center-microphone loudness gate ("Barrier") is controlled separately with `setLoudnessGateEnabled(...)`. When supported by the connected glasses, `speaking_status` reports whether the glasses currently detect speech. Microphone events also include the latest `voiceActivityDetectionEnabled` value. **React Native** ```ts theme={null} await BluetoothSdk.setMicState(true); await BluetoothSdk.setVoiceActivityDetectionEnabled(false); await BluetoothSdk.setLoudnessGateEnabled(true); ``` **Android** ```kotlin theme={null} sdk.setMicState(enabled = true) sdk.setVoiceActivityDetectionEnabled(false) sdk.setLoudnessGateEnabled(true) ``` **iOS** ```swift theme={null} sdk.setMicState(enabled: true) try await sdk.setVoiceActivityDetectionEnabled(false) try await sdk.setLoudnessGateEnabled(true) ``` Your app owns user-facing microphone permission copy, should ask for microphone access only when the feature needs it, and should provide a visible way to disable microphone streaming. ## Local Transcription **React Native** ```tsx theme={null} import {useBluetoothEvent} from '@mentra/bluetooth-sdk/react'; export function TranscriptLogger() { useBluetoothEvent('local_transcription', (event) => { console.log(`${event.text} final=${event.isFinal}`); }); return null; } ``` **Android** ```kotlin theme={null} override fun onLocalTranscription(event: LocalTranscriptionEvent) { Log.d("Mentra", "${event.text} final=${event.isFinal}") } ``` **iOS** ```swift theme={null} func mentraBluetoothSDK(_ sdk: MentraBluetoothSDK, didReceive event: BluetoothEvent) { if case let .localTranscription(transcription) = event { print("\(transcription.text) final=\(transcription.isFinal)") } } ``` ## Microphone Audio Events React Native can listen for microphone PCM events. PCM events are self-describing so apps can pass the buffer directly to STT, WAV writing, or playback code without reading native SDK sources: ```tsx theme={null} import {useBluetoothEvent} from '@mentra/bluetooth-sdk/react'; export function MicPcmLogger() { useBluetoothEvent('mic_pcm', (event) => { console.log(event.sampleRate, event.bitsPerSample, event.channels, event.encoding); console.log(event.voiceActivityDetectionEnabled); console.log(event.pcm); }); useBluetoothEvent('mic_health', (event) => { console.log(event.reason, event.sequenceGapEvents, event.decodeFailures); console.log(event.lastLc3ReceivedAt, event.lastPcmProducedAt); }); return null; } ``` `mic_pcm` emits continuous 16 kHz, 16-bit, mono, signed little-endian PCM while microphone capture is enabled. `mic_lc3` emits SDK-encoded LC3 frames with `sampleRate`, `channels`, `frameDurationMs`, `frameSizeBytes`, `bitrate`, `packetizedFromGlasses`, and `voiceActivityDetectionEnabled`. For SDK-encoded LC3 events, `packetizedFromGlasses` is `false`: the SDK decodes glasses microphone LC3 and re-encodes it to the SDK's canonical LC3 output format before emitting app-facing LC3 events. Native Android and iOS apps receive `MicPcmEvent` and `MicLc3Event` objects with the same metadata through the SDK listener/delegate surfaces. They also receive typed `MicHealthEvent` events through `onMicHealth` on Android and `.micHealth` on iOS. `mic_health` fires only when native processing detects an LC3 `sequence_gap` or `decode_failure`. It includes cumulative `sequenceGapEvents` and `decodeFailures` counters for the current connection, plus `lastLc3ReceivedAt`, `lastPcmProducedAt`, and the event `timestamp` in Unix milliseconds when available. Use the timestamps with your own stall policy; the SDK does not emit a separate stall event. Disable microphone audio callbacks when the app finishes using them. ## Voice Activity Detection Voice Activity Detection means the glasses firmware is detecting whether the microphone signal currently contains speech. Use it as a signal, not as a replacement for microphone capture: * `setVoiceActivityDetectionEnabled(false)` disables VAD packet suppression. App-facing PCM and LC3 remain continuous for external STT, recording, and voice-agent use. * `voice_activity_detection_status` reports whether glasses-side Voice Activity Detection is enabled. * `speaking_status` reports the current speaking/not-speaking state when the connected glasses provide it. ```tsx theme={null} import {useBluetoothEvent} from '@mentra/bluetooth-sdk/react'; export function SpeakingLogger() { useBluetoothEvent('speaking_status', (event) => { console.log(event.speaking ? 'speaking' : 'not speaking'); }); return null; } ``` ## Barrier `setLoudnessGateEnabled(enabled)` controls Mentra Live's center-microphone loudness gate, also called Barrier. It keeps the microphone from picking up the glasses' own speaker: * When enabled, quiet center-microphone input is replaced with silence while the glasses speaker is playing, and for a moment after. Audio frames keep flowing, so the audio timeline stays continuous. The rest of the time, input passes through unchanged. * When disabled, input always passes through unchanged. * The default is enabled, both in the standalone Bluetooth SDK and in the Mentra App, which keeps its own persisted microphone preferences. * The SDK retains the setting and reapplies it when Mentra Live reconnects. For a voice-to-voice agent that needs continuous audio without hearing its own replies, disable VAD and keep Barrier enabled: ```ts theme={null} await BluetoothSdk.setVoiceActivityDetectionEnabled(false); await BluetoothSdk.setLoudnessGateEnabled(true); ``` ## Mentra Live Speaker Playback Mentra Live uses two Bluetooth paths: * The SDK's BLE connection controls glasses features and receives device events. * The phone OS's Bluetooth media connection carries speaker audio from the phone to the glasses. There is no special Bluetooth SDK method for "play this sound through the speaker." Play audio the same way you would play audio from any mobile app. If Mentra Live is the active Bluetooth media output, the sound comes from the glasses. On Android, the system usually creates the Bluetooth audio bond after the BLE connection flow. If the phone shows a pairing dialog, the user should accept it. On iOS, apps cannot programmatically pair or select the Bluetooth media output; ask the user to open Settings > Bluetooth and connect/select Mentra Live before playback. **React Native** ```ts theme={null} import {createAudioPlayer, setAudioModeAsync} from 'expo-audio'; await setAudioModeAsync({ playsInSilentMode: true, interruptionMode: 'duckOthers', }); const player = createAudioPlayer({uri: 'https://example.com/reply.wav'}); await BluetoothSdk.setOwnAppAudioPlaying(true); player.addListener('playbackStatusUpdate', (status) => { if (status.didJustFinish) { void BluetoothSdk.setOwnAppAudioPlaying(false); } }); player.play(); ``` **Android** ```kotlin theme={null} val player = MediaPlayer().apply { setAudioAttributes( AudioAttributes.Builder() .setUsage(AudioAttributes.USAGE_MEDIA) .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH) .build() ) setDataSource(audioUrl) prepare() } sdk.setOwnAppAudioPlaying(true) player.setOnCompletionListener { sdk.setOwnAppAudioPlaying(false) it.release() } player.start() ``` **iOS** ```swift theme={null} let session = AVAudioSession.sharedInstance() try session.setCategory(.playback, mode: .default, options: [.duckOthers]) try session.setActive(true) let item = AVPlayerItem(url: audioUrl) let player = AVPlayer(playerItem: item) var playbackEndObserver: NSObjectProtocol? playbackEndObserver = NotificationCenter.default.addObserver( forName: .AVPlayerItemDidPlayToEndTime, object: item, queue: .main ) { _ in sdk.setOwnAppAudioPlaying(false) if let observer = playbackEndObserver { NotificationCenter.default.removeObserver(observer) } } sdk.setOwnAppAudioPlaying(true) player.play() ``` `setOwnAppAudioPlaying(true)` does not route audio by itself. It tells the SDK that your app is playing audio so the Mentra Live microphone path can avoid conflicting with speaker playback. Set it back to `false` when playback finishes or stops. ## Mentra Live Media Volume Mentra Live exposes step-based media volume helpers where supported: **React Native** ```ts theme={null} const current = await BluetoothSdk.getGlassesMediaVolume(); console.log(current.level); // 0-15 await BluetoothSdk.setGlassesMediaVolume(8); ``` **Android** ```kotlin theme={null} val current = sdk.getGlassesMediaVolume() sdk.setGlassesMediaVolume(8) ``` **iOS** ```swift theme={null} let current = try await sdk.getGlassesMediaVolume() try await sdk.setGlassesMediaVolume(8) ``` ## Production Notes * Always provide a user-visible microphone permission explanation. * Let users disable microphone streaming. * For Mentra Live speaker features, explain that users may need to pair/select the glasses as the phone's Bluetooth media output. * Expect model availability to differ by platform, locale, and app configuration. * Keep cloud upload and retention policies explicit in your privacy disclosures. * Disable microphone audio callbacks when the app finishes using them. # Build from Scratch Source: https://docs.mentraglass.com/bluetooth-sdk/build-from-scratch Add the Mentra Bluetooth SDK to your own mobile app. This is how to add the Mentra Bluetooth SDK to your own app from scratch. Choose your platform below for the complete installation, permissions, configuration, and connection flow. If you want to try the SDK before integrating it, start with the [Quickstart](/bluetooth-sdk/quickstart) and run a complete example app. > [!WARNING] > > Use a physical phone for Bluetooth testing. Simulators and emulators are useful > for UI and compile checks, but they do not provide the real Bluetooth path. > [!WARNING] > > Bluetooth SDK `3.3.0` requires the matching Mentra Live > `3.3.0` glasses software. Every SDK release requires the Mentra Live > software release with the same version number. ## Choose Your Platform > **[React Native / Expo](/bluetooth-sdk/react-native-expo)** > > Install `@mentra/bluetooth-sdk`, configure its Expo plugin and permissions, > create a native build, and connect. > **[Android](/bluetooth-sdk/android)** > > Add the Maven package, configure Android permissions and native libraries, > and connect from Kotlin. > **[iOS](/bluetooth-sdk/ios)** > > Add the Swift package, configure iOS permissions and background modes, and > connect from Swift. > **[Native macOS](/bluetooth-sdk/macos)** > > Build an AppKit, SwiftUI, or react-native-macos host with the shared Apple SDK. ## Integration Sequence Each platform guide walks through the same core sequence: 1. Add the platform package to your existing app. 2. Configure Bluetooth, microphone, local-network, and notification permissions required by the features you use. 3. Scan for Mentra Live, let the user choose a device, and connect. 4. Read SDK status and keep your UI derived from the latest connection state. 5. Check for and install the [matching Mentra Live software](/bluetooth-sdk/software-update) before testing SDK features. ## Continue Building Once your app connects, use the guides below to add glasses features. > **[API Reference](/bluetooth-sdk/api-reference)** > > Explore lifecycle methods, commands, events, and status models. > **[Camera and Streaming](/bluetooth-sdk/camera-streaming)** > > Capture photos and video or stream to your own ingest endpoint. > **[Audio](/bluetooth-sdk/audio)** > > Use microphone input, transcription events, and speaker playback. > **[Hardware](/bluetooth-sdk/hardware)** > > Handle buttons, touch, LEDs, battery, and capability state. # Camera And Streaming Source: https://docs.mentraglass.com/bluetooth-sdk/camera-streaming Request photos, record videos, and start RTMP, SRT, or WebRTC streams from supported smart glasses. Camera and streaming features depend on Mentra Live being connected, ready, and permitted to capture. Gate UI by SDK status before exposing photo, video, or stream controls. ## Photo Upload Photo upload sends a JPEG from supported glasses to a webhook URL. In the starter apps, the default Camera flow starts a receiver on the phone and sends the photo from the glasses to that phone URL. Turn on **Use cloud server** only when you want to send the upload to an external endpoint, such as your production API, a staging API, or the optional starter-kit helper. **React Native** ```ts theme={null} const photo = await BluetoothSdk.requestPhoto({ size: 'medium', webhookUrl: 'https://api.example.com/mentra/photo', authToken: 'optional-token', compress: 'medium', sound: true, exposureTimeNs: null, iso: null, }); console.log('photo delivered', photo.photoUrl ?? photo.uploadUrl); ``` **Android** ```kotlin theme={null} val photo = sdk.requestPhoto( PhotoRequest( size = PhotoSize.MEDIUM, webhookUrl = "https://api.example.com/mentra/photo", authToken = "optional-token", compress = PhotoCompression.MEDIUM, sound = true, exposureTimeNs = null, iso = null, ) ) println("photo delivered: ${photo.requestId}") ``` **iOS** ```swift theme={null} let photo = try await sdk.requestPhoto( PhotoRequest( size: .medium, webhookUrl: "https://api.example.com/mentra/photo", authToken: "optional-token", compress: .medium, sound: true, exposureTimeNs: nil, iso: nil ) ) print("photo delivered: \(photo.requestId)") ``` `requestPhoto(...)` resolves when the full photo action reaches terminal success: capture completed and the JPEG was delivered to the webhook, either directly from the glasses over Wi-Fi or through the phone's Bluetooth fallback relay. It rejects or throws when the glasses report a terminal `photo_response` error, when phone-side fallback upload fails, when the SDK cannot send the command, or when no terminal response arrives before the command timeout. Use `photo_status` for intermediate progress such as `accepted`, `configuring`, `capturing`, `captured`, `uploading`, `ble_fallback_compression`, `ready_for_transfer`, and `transferring`; `photo_response` is the final success/error result. The returned success includes `uploadUrl`, and may include webhook-returned metadata such as `photoUrl`, `statusUrl`, `contentType`, or `fileSizeBytes`. The webhook should accept multipart form data with a `photo` file and `requestId`. The SDK generates a unique photo `requestId` when you omit one; pass your own only when your app needs to know it before the terminal response, such as when polling a predictable `/uploads/.json` endpoint. If `authToken` is provided, the uploader adds `Authorization: Bearer `. The capture light is always enabled for photo, video, and stream capture as a privacy indicator. For one-shot manual capture tuning, pass `exposureTimeNs` and `iso` together. `exposureTimeNs` is sensor exposure time in nanoseconds; `iso` is sensor ISO. If `exposureTimeNs` is omitted, `null` / `nil`, invalid, or unsupported by the connected glasses, the camera uses auto exposure and ignores `iso`. ### Photo Parameters `requestPhoto(...)` is a one-shot request. Its capture fields apply only to that request and do not change the action-button photo defaults used by gallery mode. | Parameter | Applies to | Behavior | | - | - | - | | `requestId` | React Native, Android, iOS | Optional ID used to correlate status and final response events. The SDK generates one when omitted or blank. | | `size` | React Native, Android, iOS | JPEG size tier: `low`, `medium`, `high`, or `max`. | | `webhookUrl` | React Native, Android, iOS | Upload URL that receives the multipart `photo` file and `requestId`. | | `authToken` | React Native, Android, iOS | Optional bearer token for the upload request. Omit it or pass `null` / `nil` to send no `Authorization` header. | | `compress` | React Native, Android, iOS | Upload compression: `none`, `low`, `medium`, or `high`. Omitted native values default to `none`. | | `save` | React Native, Android, iOS | Keeps the captured photo in the glasses gallery after delivery when `true`. When `false`, the photo is only used for the requested delivery. Defaults to `false`. | | `sound` | React Native, Android, iOS | Plays the shutter sound when `true`; defaults to `true`. | | `exposureTimeNs` | React Native, Android, iOS | Optional manual sensor exposure time in nanoseconds. Omit or pass `null` / `nil` for auto exposure. | | `iso` | React Native, Android, iOS | Optional manual ISO. Used only when `exposureTimeNs` enables manual exposure. | | `aeExposureDivisor` | React Native, Android, iOS | Optional scan-style auto-exposure divisor. Values must be greater than `1`. | | `isoCap` | React Native, Android, iOS | Optional scan-style ISO cap. Values must be greater than `0`. | | `zsl` | React Native, Android, iOS | ZSL preview/capture buffering. Enabled by default; pass `false` to disable. Honored in text/scan/manual exposure modes. | | `mfnr` | React Native, Android, iOS | Vendor multi-frame noise reduction on still capture. Enabled by default; pass `false` for the single-frame pipeline. Honored in text/scan/manual exposure modes. MFNR arms ZSL buffering in preview even when `zsl` is false. | | `noiseReduction` | React Native, Android, iOS | Optional per-request noise-reduction preference. Some firmware may report that this tuning is not implemented. | | `edgeEnhancement` | React Native, Android, iOS | Optional per-request edge-enhancement preference. Some firmware may report that this tuning is not implemented. | | `ispDigitalGain` | React Native, Android, iOS | Optional per-request ISP digital gain hint. Some firmware may report that this tuning is not implemented. | | `ispAnalogGain` | React Native, Android, iOS | Optional per-request ISP analog gain hint. Some firmware may report that this tuning is not implemented. | Listen for `photo_status` to render progress and inspect the effective settings the glasses used: ```ts theme={null} import {useBluetoothEvent} from '@mentra/bluetooth-sdk/react'; useBluetoothEvent('photo_status', (event) => { if (event.status === 'configuring') { console.log('effective JPEG config', event.resolvedConfig); } if (event.status === 'capturing') { console.log('requested still config', event.requestedCaptureConfig); console.log('latest metered preview', event.meteredPreview); } if (event.status === 'captured') { console.log('actual still capture', event.captureMetadata); } }); ``` `photo_status` metadata is stage-specific: | Status | Optional metadata | When to use it | | - | - | - | | `configuring` | `resolvedConfig` | Show the effective JPEG dimensions, quality, requested size, source, transfer method, compression, and manual exposure fields when present. | | `capturing` | `requestedCaptureConfig`, `meteredPreview` | Compare the Camera2 still request with the latest auto-exposure preview estimate before the capture is fired. | | `captured` | `captureMetadata` | Read the actual HAL-applied still capture values, including exposure time, ISO, frame duration, AE state/name, noise reduction mode, edge mode, sensor timestamp, and MFNR hint when available. | Upload and BLE transfer statuses such as `uploading`, `compressing`, `ble_fallback_compression`, `ready_for_transfer`, and `transferring` are transport progress only. They do not carry capture metadata, so use the `captured` event when you need the actual exposure/ISO/frame data for debugging or UI. `ble_fallback_compression` means the direct Wi-Fi/webhook upload failed and the glasses are compressing the already-captured photo for Bluetooth fallback delivery. ## Camera Warm-Up Use `warmUpCamera(...)` when your UI is about to offer photo capture and you want the next same-configuration `requestPhoto(...)` to skip most of the camera open/configure latency. A warm-up opens and configures the Mentra Live camera with the same size and optional manual exposure path as a photo request, starts preview/AE settling, and holds that session warm for a short TTL without taking a photo or uploading media. Because warm-up does not capture or transmit media, it does not pulse the capture light; photo, video, and stream capture still enable the light as the privacy indicator. In foreground camera UI, call warm-up when the screen or tab becomes active, then refresh it before the TTL expires. The starter apps use a `30_000ms` warm-up and refresh about every `20_000ms` while the Camera tab is foreground. **React Native** ```ts theme={null} await BluetoothSdk.warmUpCamera({ size: 'medium', exposureTimeNs: null, durationMs: 30_000, }); ``` **Android** ```kotlin theme={null} sdk.warmUpCamera( size = PhotoSize.MEDIUM, exposureTimeNs = null, durationMs = 30_000, ) ``` **iOS** ```swift theme={null} try await sdk.warmUpCamera( size: .medium, exposureTimeNs: nil, durationMs: 30_000 ) ``` `warmUpCamera(...)` returns the matching `camera_status` `ready` event. It throws or rejects when the glasses send `camera_status` `error`, when the command cannot be sent, or when the response times out. Listen for `camera_status` if you want progress events: `warming` means the warm-up was accepted, `ready` means the camera is open/configured for reuse, `stopped` means the warm lease expired or the camera closed, and `error` includes `errorCode` / `errorMessage` when available. Warm-up holds default to 15 seconds and are capped at 60 seconds. When your UI leaves the foreground, call `stopCameraWarmUp(requestId)` with the explicit request ID used to start the lease. Stopping while the camera is still opening rejects the original warm-up with `camera_warm_up_cancelled`; stopping a ready lease emits `stopped`. Compatible request-owned leases share the camera, so releasing one does not close it while another remains. The `stopped` event is not a promise settlement; it is the lifecycle end for a warm lease that already reached `ready`. If multiple callers warmed or refreshed the same camera session and received `ready`, each request ID receives its own `stopped` event when that lease expires or the camera service tears down. If the camera service tears down before a warm-up reaches `ready`, the pending warm-up rejects with `error` instead. Warm-up reuse is configuration-sensitive: the next photo should use the same `size` and `exposureTimeNs` value. A different size or manual-exposure setting may force the camera to reconfigure. If video recording or streaming already owns the camera, warm-up reports `ready` immediately because the camera is already active. If a photo capture, Bluetooth transfer, camera restart, or another warm-up is in flight, the glasses may report a retryable `camera_busy`, `photo_capture_in_progress`, `ble_transfer_in_progress`, or `camera_restart_cooldown` error; retry on the next foreground refresh rather than blocking capture UI. If a `requestPhoto(...)` arrives while warm-up is opening the session, the glasses finish parking the warm session first, then dispatch the queued photo on that same warm camera configuration. ## Video Recording Use `startVideoRecording(...)` when your app needs a clip instead of a still image. Pass per-recording settings only when you want to override the saved video recording defaults; `maxRecordingTimeMinutes` is an auto-stop guard, and `0` or an omitted value records until your app stops recording or the glasses interrupt it. **React Native** ```ts theme={null} const requestId = `video-${Date.now()}`; const started = await BluetoothSdk.startVideoRecording(requestId, true, true, { width: 1280, height: 720, fps: 30, maxRecordingTimeMinutes: 1, }); console.log('video started', started.status); // Reconcile state after app resume, a timeout, or a reconnect. const current = await BluetoothSdk.queryVideoRecordingStatus(`status-${Date.now()}`); console.log('recording now', current.data?.recording, current.data?.duration_ms); const stopped = await BluetoothSdk.stopVideoRecording( requestId, 'https://api.example.com/mentra/video', 'optional-token', ); console.log('video uploaded', stopped.status); ``` **Android** ```kotlin theme={null} val requestId = "video-${System.currentTimeMillis()}" val started = sdk.startVideoRecording( VideoRecordingRequest( requestId = requestId, save = true, sound = true, width = 1280, height = 720, fps = 30, maxRecordingTimeMinutes = 1, ) ) println("video started: ${started.status}") val current = sdk.queryVideoRecordingStatus("status-${System.currentTimeMillis()}") println("recording now: ${current.data?.get("recording")}") val stopped = sdk.stopVideoRecording( requestId = requestId, webhookUrl = "https://api.example.com/mentra/video", authToken = "optional-token", ) println("video uploaded: ${stopped.status}") ``` **iOS** ```swift theme={null} let requestId = "video-\(Date().timeIntervalSince1970)" let started = try await sdk.startVideoRecording( VideoRecordingRequest( requestId: requestId, save: true, sound: true, width: 1280, height: 720, fps: 30, maxRecordingTimeMinutes: 1 ) ) print("video started: \(started.status)") let current = try await sdk.queryVideoRecordingStatus(requestId: "status-\(Date().timeIntervalSince1970)") print("recording now: \(current.data?["recording"] ?? false)") let stopped = try await sdk.stopVideoRecording( requestId: requestId, webhookUrl: "https://api.example.com/mentra/video", authToken: "optional-token" ) print("video uploaded: \(stopped.status)") ``` `startVideoRecording(...)` resolves from `recording_started`. `stopVideoRecording(...)` without a webhook resolves from `recording_stopped`; when you pass a webhook URL, it waits for `recording_stopped` plus video `media_success`, and rejects on video `media_error`. `queryVideoRecordingStatus(requestId)` sends the glasses' existing status query and resolves with the current `recording` state and elapsed `duration_ms`, so use it to reconcile after a timeout, reconnect, or app resume. Use a fresh request ID for each query. Raw `video_recording_status`, `media_success`, and `media_error` events remain available for listeners that need the glasses event stream directly. The video upload webhook should accept multipart form data with a `video` field and `requestId`. Pass the webhook URL and auth token at stop time so upload credentials can be fresh when the recording ends. ## Gallery Mode Mentra Live has gallery mode for the right action button. When gallery mode is enabled, a short press takes a photo, a long press starts video recording, and a short press stops the active video recording. Button and touch events are still reported to your app. > [!WARNING] > > `setPhotoCaptureDefaults(...)` in the samples below is **deprecated**. Prefer per-request `requestPhoto(...)` options for capture tuning. The call still configures action-button size for now. **React Native** ```ts theme={null} await BluetoothSdk.setGalleryModeEnabled(true); await BluetoothSdk.setPhotoCaptureDefaults({size: 'medium'}); // deprecated await BluetoothSdk.setVideoRecordingDefaults({width: 1280, height: 720, fps: 30}); await BluetoothSdk.setMaxVideoRecordingDuration(3); const cameraFov = await BluetoothSdk.setCameraFov({fov: 102, roiPosition: 'center'}); console.log(`Camera ready at ${cameraFov.fov}°`); ``` **Android** ```kotlin theme={null} sdk.setGalleryModeEnabled(true) @Suppress("DEPRECATION") sdk.setPhotoCaptureDefaults(PhotoCaptureDefaults(size = PhotoSize.MEDIUM)) // deprecated sdk.setVideoRecordingDefaults(VideoRecordingDefaults(width = 1280, height = 720, fps = 30)) sdk.setMaxVideoRecordingDuration(minutes = 3) val cameraFov = sdk.setCameraFov(CameraFov(fov = 102, roiPosition = CameraRoiPosition.CENTER)) println("Camera ready at ${cameraFov.fov}°") ``` **iOS** ```swift theme={null} try await sdk.setGalleryModeEnabled(true) try await sdk.setPhotoCaptureDefaults(PhotoCaptureDefaults(size: .medium)) // deprecated try await sdk.setVideoRecordingDefaults(VideoRecordingDefaults(width: 1280, height: 720, fps: 30)) try await sdk.setMaxVideoRecordingDuration(minutes: 3) let cameraFov = try await sdk.setCameraFov(CameraFov(fov: 102, roiPosition: .center)) print("Camera ready at \(cameraFov.fov)°") ``` Use `setGalleryModeEnabled(false)` when you want the action button to report events without triggering local gallery capture while the glasses are connected. Action-button photos emitted by the glasses use the same `photo_status` metadata shape when the phone SDK is connected. These local captures report `resolvedConfig.source: "button"` and `resolvedConfig.transferMethod: "local"`, then expose `requestedCaptureConfig` / `meteredPreview` on `capturing` and `captureMetadata` on `captured`. ### Gallery Photo Request IDs In gallery mode, photos are saved on the glasses instead of being uploaded to a webhook. Because nothing is uploaded, there is no response that returns the `requestId`. Instead, read the `requestId` from `photo_status` events. Action-button captures set `resolvedConfig.source` to `"button"`, so you can filter for them. When your own app starts the capture, you set the `requestId` yourself by passing it to `requestPhoto({requestId, save: true, webhookUrl: null})`. **React Native** ```ts theme={null} import {useBluetoothEvent} from '@mentra/bluetooth-sdk/react'; useBluetoothEvent('photo_status', (event) => { if (event.resolvedConfig?.source === 'button') { console.log('gallery photo requestId', event.requestId); } }); ``` **Android** ```kotlin theme={null} override fun onPhotoStatus(event: PhotoStatusEvent) { if (event.resolvedConfig?.source == "button") { Log.d("Mentra", "gallery photo requestId: ${event.requestId}") } } ``` **iOS** ```swift theme={null} func mentraBluetoothSDK(_ sdk: MentraBluetoothSDK, didReceive event: BluetoothEvent) { if case let .photoStatus(status) = event, status.resolvedConfig?.source == "button" { print("gallery photo requestId: \(status.requestId)") } } ``` To find the saved photo file, connect to the glasses Wi-Fi hotspot and request the gallery list. Each photo includes `request_id`, `name`, `url`, `mime_type`, and `size`. Match on `request_id` to find your capture: ```ts theme={null} const gallery = await fetch(`${galleryBaseUrl}/api/gallery?limit=100`).then((res) => res.json()); const savedPhoto = gallery.data.photos.find((p) => p.request_id === requestId); console.log('saved as', savedPhoto?.name, 'at', savedPhoto?.url); ``` > [!WARNING] > > The gallery server is only reachable over the glasses Wi-Fi hotspot. Connect the phone to that hotspot before querying `/api/gallery`. ### Action-Button Photo Defaults > [!WARNING] > > `setPhotoCaptureDefaults(...)` is **deprecated**. Prefer per-request `requestPhoto(...)` options (for example `mode: "text"` for text sensor size/crop, or explicit `aeExposureDivisor`) instead of sticky button-photo presets. The method still works but will be removed in a future release. `setPhotoCaptureDefaults(...)` controls action-button photo defaults. Omitted fields leave the existing defaults unchanged. Pass `resetCaptureTuning: true` to restore the optional tuning fields to standard capture behavior; this does not change photo size unless you also pass `size`. If you include other fields in the same call, those fields become the new defaults after the reset. | Default field | Behavior | | - | - | | `size` | Action-button photo size tier: `low`, `medium`, `high`, or `max`. | | `zsl` | Persisted ZSL override for physical camera-button photos. When omitted, button photos inherit the global ZSL default. | | `mfnr` | Persisted MFNR override for physical camera-button photos. When omitted, button photos inherit the global MFNR default. | | `noiseReduction` | Noise-reduction preference for action-button photos. | | `edgeEnhancement` | Edge-enhancement preference for action-button photos. | | `ispDigitalGain` | ISP digital gain hint for action-button photos. | | `ispAnalogGain` | ISP analog gain hint for action-button photos. | | `aeExposureDivisor` | Scan-style auto-exposure divisor for action-button photos. Values must be greater than `1`. | | `isoCap` | Scan-style ISO cap for action-button photos. Values must be greater than `0`. | | `compress` | Upload compression for action-button photos: `none`, `low`, `medium`, or `high`. | | `sound` | Shutter sound preference for action-button photos. | | `resetCaptureTuning` | Restores optional tuning fields to standard capture behavior before applying any fields included in the same call. | `setCameraFov` accepts FOV degrees from 62 to 118 and ROI position `"center"`, `"bottom"`, or `"top"` on React Native, `CameraRoiPosition` on Android, and `CameraRoiPosition` on iOS. React Native also accepts `{preset: "narrow" | "standard" | "wide"}`; presets map to 82, 102, and 118 degrees with center ROI. On Mentra Live, applying FOV/ROI restarts the camera; the returned `CameraFovResult` resolves from the ASG client only when the setting has been applied to camera hardware after the restart cooldown, and the promise rejects on glasses-side error, persist-only/no hardware-application ack, or timeout. Treat FOV as a framing/ROI control; output resolution and effective detail can vary by capture path, firmware, and camera mode. ## Streaming Streaming sends camera video from supported glasses to an ingest URL. In the starter apps, the default Stream flow uses the phone as the receiver where the platform example supports it. Turn on **Use cloud server** when you want the glasses to stream to an external RTMP, SRT, or WHIP/WebRTC ingest endpoint. Use your own server in production; the local helper below is only a demo convenience for testing external endpoints from the starter apps. **React Native** ```ts theme={null} const streamId = `stream-${Date.now()}`; await BluetoothSdk.startStream({ type: 'start_stream', streamUrl: 'http://192.168.1.42:8889/mentra-live/whip', streamId, video: {fps: 15}, }); await BluetoothSdk.stopStream(); ``` **Android** ```kotlin theme={null} val streamId = "stream-${System.currentTimeMillis()}" sdk.startStream( StreamRequest( streamUrl = "http://192.168.1.42:8889/mentra-live/whip", streamId = streamId, video = StreamVideoConfig(fps = 15), ) ) sdk.stopStream() ``` **iOS** ```swift theme={null} let streamId = "stream-\(Date().timeIntervalSince1970)" try await sdk.startStream( StreamRequest( streamUrl: "http://192.168.1.42:8889/mentra-live/whip", streamId: streamId, video: StreamVideoConfig(fps: 15) ) ) try await sdk.stopStream() ``` Use `rtmp://` or `rtmps://` for RTMP, `srt://` for SRT, and `http://` or `https://` for WHIP/WebRTC ingest. `startStream()` resolves once glasses report `status: "streaming"`; `stopStream()` resolves once glasses report `status: "stopped"` and also resolves with a normalized stopped event when glasses report that no stream was active. The SDK sends stream keep-alives automatically while streaming and reports keep-alive failures through `stream_status`. The camera light is always enabled while streaming. ## Optional Local Demo Helper The starter kit includes a local helper server so developers can try the **Use cloud server** path without first deploying their own webhook or streaming service: ```bash theme={null} git clone https://github.com/Mentra-Community/Mentra-Bluetooth-SDK-Starter-Kit.git cd Mentra-Bluetooth-SDK-Starter-Kit python3 examples/local-demo-cloud/server.py ``` In the starter apps, leave **Use cloud server** off for the default glasses-to-phone flow. Turn it on when you want to test an external endpoint, then paste the printed LAN `/upload` URL into the Camera screen or the printed RTMP, SRT, or WHIP publish URL into the Stream screen. You can skip this helper entirely when you already have reachable upload and streaming endpoints. > [!WARNING] > > Do not use `localhost` from the mobile app. The glasses, phone, and computer must be on a network where the glasses can reach the printed LAN address. ## Direct Phone Capture And Streaming The starter kit examples also include direct phone photo and WebRTC receive flows: * Android can receive photos directly on the phone and host a GStreamer WHIP receiver. * iOS can receive photos directly on the phone and host a GStreamer WHIP receiver. * React Native includes local native companion modules for Android and iOS direct phone photo and WebRTC demos. Use real hardware for these flows. Keep the glasses Wi-Fi active and make sure the phone and glasses are on a reachable local network. For direct phone photo upload in React Native, start the phone receiver first and pass its `uploadUrl` as the photo webhook: ```ts theme={null} import BluetoothSdk from '@mentra/bluetooth-sdk'; import MentraPhotoReceiver from '@mentra/bluetooth-sdk/photo-receiver'; const receiver = await MentraPhotoReceiver.startPhotoReceiver(); const uploadSubscription = MentraPhotoReceiver.addListener('photoUpload', (event) => { console.log(event.requestId, event.fileUri, event.byteCount); }); try { const photo = await BluetoothSdk.requestPhoto({ size: 'medium', webhookUrl: receiver.uploadUrl, authToken: null, compress: 'medium', sound: true, }); console.log('photo delivered', photo.photoUrl ?? photo.uploadUrl); } catch (error) { uploadSubscription.remove(); await MentraPhotoReceiver.stopPhotoReceiver(); throw error; } // Later: uploadSubscription.remove(); await MentraPhotoReceiver.stopPhotoReceiver(); ``` ## Stream Status Subscribe to stream status before starting a stream. Treat stream start as accepted, then drive UI from status events: **React Native** ```tsx theme={null} import {useBluetoothEvent} from '@mentra/bluetooth-sdk/react'; export function StreamStatusLogger() { useBluetoothEvent('stream_status', (event) => { console.log(event); console.log(event.resolvedConfig?.video?.fps); }); return null; } ``` **Android** ```kotlin theme={null} override fun onStreamStatus(event: StreamStatusEvent) { Log.d("Mentra", "Stream status: ${event.status}") Log.d("Mentra", "Resolved FPS: ${event.resolvedConfig?.video?.fps}") } ``` **iOS** ```swift theme={null} func mentraBluetoothSDK(_ sdk: MentraBluetoothSDK, didReceive event: BluetoothEvent) { if case let .streamStatus(status) = event { print("Stream status: \(status)") print("Resolved FPS: \(status.resolvedConfig?.video?.fps ?? -1)") } } ``` Status values include initializing, streaming, stopped, reconnecting, and error states. When a status event includes `resolvedConfig`, it contains the effective settings used by the glasses after defaults, clamps, and camera-mode selection. In `resolvedConfig.video`, `width` and `height` are the encoded output dimensions, `captureWidth` and `captureHeight` are the native camera buffer dimensions before crop or downscale, `bitrate` is the encoded video bitrate in bits per second, and `fps` is the resolved capture/encode frame rate. `resolvedConfig.audio` reports the audio bitrate, sample rate, echo cancellation, and noise suppression settings when audio is active. # Hardware Integration Source: https://docs.mentraglass.com/bluetooth-sdk/hardware Handle Mentra Live capabilities and status-driven hardware behavior. Mentra Live is the camera glasses model supported by this SDK tab. Apps should still treat hardware features as status-driven because firmware version, permissions, and connection state affect what is available at any moment. ## Recommended Flow 1. Ask the user which glasses model they are pairing. 2. Call `scan()` for that model. 3. Present typed discovered devices from the progressive scan results. 4. Connect using the discovered device or default-device helper. 5. Read shaped glasses state, firmware fields, and hardware-related status before enabling advanced features. 6. Keep app UI derived from SDK status rather than from command success alone. **React Native** ```ts theme={null} import BluetoothSdk, {DeviceModels} from '@mentra/bluetooth-sdk'; const devices = await BluetoothSdk.scan(DeviceModels.MentraLive, { timeoutMs: 10_000, onResults: (nextDevices) => renderDevicePicker(nextDevices), }); await BluetoothSdk.connect(await chooseDevice(devices)); ``` **Android** ```kotlin theme={null} sdk.scan(DeviceModel.MENTRA_LIVE, timeoutMs = 10_000) { nextDevices -> renderDevicePicker(nextDevices, onSelect = { device -> sdk.connect(device) }) } ``` **iOS** ```swift theme={null} try sdk.scan(model: .mentraLive, timeout: 10) { nextDevices in renderDevicePicker(nextDevices) { device in do { try sdk.connect(to: device) } catch { handleConnectionError(error) } } } ``` In multi-device environments, keep the picker explicit. Do not auto-connect to the first scan result; let the user choose the glasses they expect. ## Capability Areas * Input: button, touch, head-up, and switch events. * Audio: PCM, LC3, audio pairing, preferred microphone, speaker playback, and local transcription. * Camera: photo, gallery, video recording, and streaming. * Network: Wi-Fi scan, credentials, and hotspot state. * Maintenance: Mentra Live version info and OTA status. ## Mentra Live Hardware Mentra Live is built for visual capture, streaming, and audio-first interactions. * Camera: 1080p with streaming support. * Microphone: yes. * Speaker: yes. * Buttons: yes, including press, double press, and long press events. * LEDs: RGB feedback plus white privacy light. * Wi-Fi: yes. ```ts theme={null} { modelName: "Mentra Live", hasCamera: true, hasMicrophone: true, hasSpeaker: true, hasButton: true, hasLight: true, hasWifi: true } ``` Unsupported operations are recoverable SDK errors. Reconcile command failures with the latest status snapshot before showing destructive UI. ## Connection Resilience * Subscribe to status callbacks before connecting. * Retry scans manually from the UI instead of scanning forever in the background. * On mobile OS background transitions, expect Bluetooth behavior to vary by platform. * Provide a visible "forget device" path that clears both the SDK default device and any app-persisted default-device record. * If your app persists a default device, restore it with `setDefaultDevice()` before calling `connectDefault()`. # iOS Source: https://docs.mentraglass.com/bluetooth-sdk/ios Use the Mentra Bluetooth SDK from a native iOS app. Native iOS apps use the `MentraBluetoothSDK` Swift package. The same package also supports [native macOS with AppKit or SwiftUI](/bluetooth-sdk/macos). The permissions and background modes below apply to iOS, including iOS apps running on Apple silicon Macs. Native Mac apps use the separate macOS setup. ## Requirements * iOS deployment target `15.1` or newer. * Xcode 15 or newer. * Swift Package Manager. * A physical iPhone for Bluetooth, camera, microphone, direct phone photo, and direct phone WebRTC testing. ## Install The public SwiftPM repository is: ```text theme={null} https://github.com/Mentra-Community/mentra-bluetooth-sdk-ios.git ``` In Xcode, choose **File > Add Package Dependencies**, enter that URL, select version `3.3.0`, and add the `MentraBluetoothSDK` product to your app target. For `Package.swift` consumers: ```swift theme={null} .package( url: "https://github.com/Mentra-Community/mentra-bluetooth-sdk-ios.git", from: "3.3.0" ) ``` Then add the product to your target: ```swift theme={null} .product(name: "MentraBluetoothSDK", package: "mentra-bluetooth-sdk-ios") ``` For local SDK development, add the package folder directly in Xcode: ```text theme={null} /path/to/MentraOS/mobile/modules/bluetooth-sdk ``` > [!WARNING] > > The SwiftPM package currently builds the core Bluetooth SDK only. It does not build optional local STT or tar.bz2 extraction code paths. ## Permissions Add usage descriptions to `Info.plist`: ```xml theme={null} NSBluetoothAlwaysUsageDescription This app connects to your smart glasses over Bluetooth. NSMicrophoneUsageDescription This app uses the microphone when you enable audio features. NSLocalNetworkUsageDescription This app can connect to optional local demo servers on your network. ``` ## Background Operation Declare Core Bluetooth background mode in your app's `Info.plist`. It is required, not optional: iOS reads background modes only from the app's own `Info.plist`, so the Swift package cannot add it for you. ```xml theme={null} UIBackgroundModes bluetooth-central ``` Mentra Live drops its Bluetooth link for short periods in normal use. During a software update, the glasses restart their Bluetooth chip and are unreachable for about 45 seconds; walking out of range or powering the glasses off has the same effect. When the link drops, the SDK keeps a pending connection to the same glasses open, and iOS completes it as soon as they advertise again. With `bluetooth-central`, iOS completes that reconnection and wakes your app even while the phone is locked in a pocket. Without it, iOS suspends your app and nothing reconnects until the user opens it again, so a software update started on a locked phone stalls after the Bluetooth chip restarts. App Review asks apps that declare `bluetooth-central` to need it; a companion app for connected glasses does. At startup the SDK checks your `Info.plist` and logs a warning when `bluetooth-central` is missing. Read `MentraBluetoothSDK.supportsBackgroundReconnect` to check it from code, for example in a debug assertion or a setup screen. If your app also keeps microphone capture or an audio session active in the background, add `audio` as well: ```xml theme={null} UIBackgroundModes bluetooth-central audio ``` Configure your `AVAudioSession` before starting continuous microphone or playback work. Start continuous microphone capture while the app is foregrounded; iOS background mode lets an active session continue, but the SDK does not start the phone microphone from the background. For example: ```swift theme={null} let session = AVAudioSession.sharedInstance() try session.setCategory(.playAndRecord, mode: .default, options: [.allowBluetooth, .defaultToSpeaker]) try session.setActive(true) ``` `bluetooth-central` lets iOS deliver BLE events to your app and complete its pending reconnection in the background, subject to iOS scheduling and power-management limits. `audio` is required only for apps that intentionally keep recording or audio playback active after backgrounding. > [!WARNING] > > This SDK version does not enable terminated-app Core Bluetooth state restoration with `CBCentralManagerOptionRestoreIdentifierKey`. If iOS terminates the app, relaunch the app and reconnect from your normal startup flow. ## Basic Flow ```swift theme={null} import MentraBluetoothSDK @MainActor final class GlassesController: NSObject, MentraBluetoothSDKDelegate { private let sdk = MentraBluetoothSDK() private var selectedDevice: Device? override init() { super.init() sdk.delegate = self } func scan() throws { try sdk.scan(model: .mentraLive, timeout: 10) { [weak self] devices in self?.renderDevicePicker(devices) { device in self?.selectedDevice = device } } } func connect() throws { guard let selectedDevice else { return } try sdk.connect(to: selectedDevice) } func refreshStatus() { sdk.requestVersionInfo() let glasses = sdk.glasses if let device = glasses.device { let battery = glasses.battery?.level.map(String.init) ?? "unknown" print("Connected to \(device.deviceModel?.deviceType ?? "glasses"), battery=\(battery)%") } } func mentraBluetoothSDK(_ sdk: MentraBluetoothSDK, didUpdateGlasses glasses: GlassesRuntimeState) { // Keep app UI derived from SDK status. print("Glasses changed: \(glasses)") } private func renderDevicePicker(_ devices: [Device], onSelect: @escaping (Device) -> Void) { // Render devices in SDK-provided order and call onSelect with the user's choice. } deinit { sdk.invalidate() } } ``` ## Local SDK Override Use this when testing native Swift SDK source changes locally: ```text theme={null} /path/to/MentraOS/mobile/modules/bluetooth-sdk ``` Add that folder as a local Swift package in Xcode. Keep local paths in your workspace settings or CI environment, not in committed project settings. # macOS Source: https://docs.mentraglass.com/bluetooth-sdk/macos Use the Bluetooth SDK in native AppKit, SwiftUI, or react-native-macos apps. The Apple SDK supports native macOS apps through the same `MentraBluetoothSDK` Swift package used on iOS. The npm package exposes that implementation to `react-native-macos` through Expo Modules. > [!NOTE] > > Native macOS support is new on the `dev` branch. Until a coordinated SDK release > containing this change is published, use the local source overrides below. > Older published packages are iOS-only. ## Native macOS Versus iOS On Mac The iOS example distributed through TestFlight can run on compatible Apple silicon Macs. That is an iOS app running on Mac, not an AppKit or `react-native-macos` build. It continues to use the iOS SDK implementation. Use the native examples below when building a macOS application. CoreBluetooth, camera requests, OTA commands, status, and events share the existing Apple SDK implementation. The Mac adapters use CoreAudio for host microphone input and ImageIO for image decoding instead of iOS-only APIs. ## Swift: AppKit Or SwiftUI Requirements: macOS 13 or newer, Xcode with Swift 5.9 or newer, and a Mac with Bluetooth for device testing. Add the `MentraBluetoothSDK` product to your macOS app target from: ```text theme={null} https://github.com/Mentra-Community/mentra-bluetooth-sdk-ios.git ``` The repository keeps its existing name for compatibility; it supplies both Apple platforms. For unpublished changes, add this folder as a local package in Xcode instead: ```text theme={null} /path/to/MentraOS/mobile/modules/bluetooth-sdk ``` The [Swift connection API](/bluetooth-sdk/ios#basic-flow) is unchanged. Own the SDK on the main actor in your AppKit controller or SwiftUI model, observe its delegate, and call `invalidate()` when the owning session ends. Do not configure `AVAudioSession` or `UIBackgroundModes` in a native Mac app. ### Permissions And Sandbox Add usage descriptions to your app's `Info.plist`: ```xml theme={null} NSBluetoothAlwaysUsageDescription Connect to your smart glasses. NSMicrophoneUsageDescription Capture audio when microphone recording is enabled. NSLocalNetworkUsageDescription Access glasses and media servers on your local network. ``` For a sandboxed app, enable these capabilities in the app target, not the SDK: ```xml theme={null} com.apple.security.app-sandbox com.apple.security.device.bluetooth com.apple.security.device.audio-input com.apple.security.network.client com.apple.security.network.server ``` Audio input is only needed for the Mac microphone. Network server access is only needed when hosting a local photo receiver or OTA server. Request host microphone permission from your app, for example with `await AVCaptureDevice.requestAccess(for: .audio)`, before enabling that input. The SDK checks permission but does not show a microphone permission dialog. For local HTTP photo webhooks, allow local networking with `NSAppTransportSecurity.NSAllowsLocalNetworking`. Prefer HTTPS for remote servers. ### Run The AppKit Example From Source From the Starter Kit `dev` branch: ```bash theme={null} cd examples/macos export MENTRA_BLUETOOTH_SDK_PACKAGE_PATH=/absolute/path/to/MentraOS/mobile/modules/bluetooth-sdk bash run.sh ``` The script builds and ad-hoc signs a sandboxed `.app` with usage descriptions. Running the raw SwiftPM executable is not equivalent to running this app bundle. The focused sample demonstrates scan/connect, photo upload and preview, and OTA availability checking. It does not duplicate the complete phone example OTA UI. ## React Native macOS Use `examples/react-native-macos` in the Starter Kit, not `expo run:ios` and not the phone example's generated project. This native Mac host currently pairs React Native macOS 0.81 with React Native 0.81 and Expo Modules from Expo SDK 54; it requires macOS 14 or newer. Keep the native host, React Native, and Expo versions compatible when upgrading them. ```bash theme={null} cd examples/react-native-macos bun install --frozen-lockfile export MENTRA_BLUETOOTH_SDK_PACKAGE_PATH=/absolute/path/to/MentraOS/mobile/modules/bluetooth-sdk cd macos pod install cd .. bun run macos ``` Use the same source override in both the `pod install` shell and the Metro shell. After the native-macOS SDK is published, remove the override to use the package pinned in `package.json`. The example's Podfile enables Expo Modules autolinking for the macOS target, sets `MENTRA_BLUETOOTH_SDK_INCLUDE_EXPO_ADAPTER=1`, and uses the `react-native-macos` native dependency scripts. Its Metro configuration resolves `react-native` to `react-native-macos`. The existing JavaScript imports and SDK React hooks are unchanged: ```tsx theme={null} import BluetoothSdk from '@mentra/bluetooth-sdk'; import {useMentraBluetooth} from '@mentra/bluetooth-sdk/react'; ``` Expo Go and Expo's iOS/Android prebuild plugin do not generate this Mac host. Microsoft documents Expo Modules support on `react-native-macos` as [experimental](https://microsoft.github.io/react-native-macos/docs/guides/installing-expo-modules). This is support for the Bluetooth SDK, not a claim that all Mentra Engine dependencies or arbitrary Expo packages support native macOS. ## Platform Differences * Bluetooth testing requires a real Mac. Sleeping the Mac can interrupt the connection; reconnect through the normal SDK lifecycle after wake. * Select the Mac's Bluetooth audio output in System Settings. The SDK does not change the system-wide audio route. Host microphone selection uses CoreAudio for the SDK's own recording engine. Bluetooth microphone modes use the Bluetooth input selected in System Settings > Sound > Input, not the first connected headset. Internal microphone mode uses a built-in input only. Input changes are observed while recording; glasses audio availability is tracked independently of the selected output. * macOS has no iOS `AVAudioSession` audio-focus contract. The SDK tracks explicit own-app playback, but does not detect every other app's playback to suspend the glasses microphone. A host app must coordinate simultaneous audio use. * A playback device reconfiguration rejects pending PCM stream operations; reopen the stream after selecting the new route. * iOS ANCS notification relay is not available on native macOS. * Optional MentraOS local STT/TTS, Vuzix, and Mentra Nex dependencies are not part of this native macOS package. * Glasses Wi-Fi/hotspot commands work through the shared SDK. Joining that hotspot from the Mac remains a host/system Wi-Fi action. The Android-only `otaLocalNetwork` scoped networking adapter remains unavailable on macOS. * Source-built SDKs do not gain a production OTA pin. Use the existing `debug.setOtaVersionUrl(...)` override when deliberately testing OTA from source, and use a published coordinated SDK for its matching release pin. # Mentra Bluetooth SDK Source: https://docs.mentraglass.com/bluetooth-sdk/overview Connect mobile and native macOS apps directly to Mentra Live over Bluetooth. The Mentra Bluetooth SDK connects Android, iOS, macOS, and React Native apps to Mentra Live directly over Bluetooth. It provides scanning, pairing, device status, microphone audio, camera capture, speaker playback, streaming, Wi-Fi, and hardware events. See the [native macOS setup and platform differences](/bluetooth-sdk/macos) for AppKit, SwiftUI, and react-native-macos. Use this SDK when your app owns the mobile experience and needs a direct phone-to-glasses connection. ## Packages | Platform | Package | Import | | - | - | - | | Android | `com.mentraglass:bluetooth-sdk` | `import com.mentra.bluetoothsdk.*` | | iOS / macOS | `MentraBluetoothSDK` Swift package | `import MentraBluetoothSDK` | | React Native / Expo / react-native-macos | `@mentra/bluetooth-sdk` | `import BluetoothSdk from '@mentra/bluetooth-sdk'` | ## Quickstart The fastest way to try the Mentra Bluetooth SDK is to run one of the complete Android, iOS, or React Native example apps in the Mentra Bluetooth SDK Starter Kit. > **[Run an example app](/bluetooth-sdk/quickstart)** > > Clone the starter kit, choose your platform, and connect to Mentra Live. You can also [open the starter kit directly on GitHub](https://github.com/Mentra-Community/Mentra-Bluetooth-SDK-Starter-Kit). ## What You Can Build * Native Android, iOS, macOS, or React Native apps that scan for and connect to Mentra Live. * Apps that receive button, touch, swipe, head-up, battery, Wi-Fi, hotspot, stream, photo, audio, and SDK log events. * Apps that control Mentra Live features such as gallery mode, camera settings, speaker playback, RGB LEDs, Wi-Fi, hotspot, microphone, camera, and streaming. * Photo upload to your own webhook and RTMP, SRT, or WebRTC streaming to your own ingest endpoint. The starter kit also includes optional local demo helpers for trying these flows on a LAN. ## Requirements * The Bluetooth SDK and Mentra Live glasses software must use the same release version. The latest matching pair is `3.3.0`. * A physical Android phone, iPhone, or Mac for real Bluetooth testing. * Android min SDK `28` or newer. * iOS deployment target `15.1` or newer. * Native macOS deployment target `13` or newer; the react-native-macos example requires `14` or newer. * React Native / Expo apps must use a development build or production native build. Expo Go cannot load the native SDK. * Bluetooth permissions, plus Android location permission where BLE scanning requires it. > [!WARNING] > > Bluetooth connection alone does not confirm version compatibility. Before testing SDK features, [update Mentra Live to the glasses software matching your SDK version](/bluetooth-sdk/software-update). An app using Bluetooth SDK `3.3.0` must use the matching Mentra Live `3.3.0` software release. ## Next Steps > **[Build from Scratch](/bluetooth-sdk/build-from-scratch)** > > Add the SDK to your own Android, iOS, macOS, or React Native app. > **[Update Mentra Live](/bluetooth-sdk/software-update)** > > Install the glasses software that matches your Bluetooth SDK release. > **[API Reference](/bluetooth-sdk/api-reference)** > > Review lifecycle, commands, events, status fields, and Mentra Live behavior. > **[Troubleshooting](/bluetooth-sdk/troubleshooting)** > > Fix native build, permission, scan, stream, and Expo issues. # Production Checklist Source: https://docs.mentraglass.com/bluetooth-sdk/production-checklist Validate a mobile app before shipping with the Mentra Bluetooth SDK. Use this checklist before shipping an app with the Mentra Bluetooth SDK. ## Analytics * Mentra Bluetooth SDK usage analytics are enabled by default and use Mentra's embedded PostHog project write key. * If your app disables SDK analytics, do it before SDK startup with the Expo config plugin, Android manifest metadata, iOS `Info.plist`, or native SDK configuration. * Verify `bluetooth_sdk_started` appears after SDK startup and `bluetooth_sdk_glasses_connected` appears only after a successful glasses connection. * If you ship one package id through several lanes, set `analytics.environment` (or the native metadata equivalent) so store builds are separable from dev and staging builds; store detection (`app_install_source`) is automatic. * Confirm your privacy copy reflects anonymous SDK usage analytics if needed for your app's distribution channel. * The SDK checks for a newer release on every start, separately from analytics. If a debug build prints a Mentra Bluetooth SDK update warning, update before shipping. ## App Setup * Android, iOS, and/or React Native native builds are configured for the app stack you ship. * iOS deployment target is `15.1` or newer. * Android min SDK is `28` or newer. * Android builds use Java 17. * Android native packaging includes any `pickFirst` rules required by the app's own native dependencies. * Bluetooth permission copy is user-friendly and specific. * Microphone, camera, local network, notification, and location permissions are requested only when needed. * The app integrates through the documented Android, iOS, or React Native SDK APIs. ## SDK Integration * App subscribes to typed glasses and Bluetooth status callbacks. * App handles disconnect, reconnect, scan stopped, and pair failure states. * App derives controls from Mentra Live status, permissions, and firmware behavior. * App cleans up SDK listeners/delegates and calls `close()` / `invalidate()` when sessions end. * React Native apps remove every SDK event subscription when screens/session owners unmount. * App uses typed settings APIs for gallery mode, camera, LED, Wi-Fi, streaming, and microphone settings. * Advanced controls are gated behind SDK capability/status checks. ## Privacy And Compliance * Audio capture behavior is clearly disclosed. * Camera capture behavior is clearly disclosed. * Bluetooth and location permission copy explains device discovery. * Cloud upload behavior is documented in the app privacy policy. * Logs avoid sensitive user data. ## Release Validation * Fresh bare Android app install using the published Maven artifact. * Fresh bare iOS app install using the published Swift package. * Fresh React Native development or production build using the published JavaScript package, if shipping React Native. * Android sample app in the Starter Kit builds. * iOS sample app in the Starter Kit resolves SwiftPM dependencies and builds. * React Native sample app in the Starter Kit runs `bun install`, `bunx expo prebuild`, and at least one native platform build. * Fresh install test on iOS. * Fresh install test on Android. * Pairing test with Mentra Live. * Reconnect test after app restart. * Reconnect test after Bluetooth toggle. * Camera/gallery/streaming test with Mentra Live. * Speaker playback test with Mentra Live. * Hardware settings test for save-in-gallery mode, camera options, LED color/pattern, Wi-Fi, and hotspot behavior. * Button/touch event test. * Microphone audio/transcription test if enabled. ## Restricted-Network Deployments * The app pins one exact completed Mentra release identity rather than a moving package tag. * The self-hosted OTA bundle is the exact bundle selected by the completed release record for the pinned SDK release. * Mentra SDK analytics are disabled before native SDK startup. * Your network policy allows or blocks the SDK's startup update check to `core.mentraglass.com`; the SDK works the same either way. * The imported OTA ZIP matches the SHA-256 recorded in the completed release record. * The final `version.json` and every artifact are reachable from both the phone and Mentra Live. * The complete update and SDK feature set are tested with public internet access blocked. * Older inventory is factory- or legacy-bootstrapped to ASG build 39 or newer. # Quickstart Source: https://docs.mentraglass.com/bluetooth-sdk/quickstart Run a complete Mentra Bluetooth SDK example app on a physical phone. The fastest way to get started with the Mentra Bluetooth SDK is to run one of the complete example apps in the [Mentra Bluetooth SDK Starter Kit](https://github.com/Mentra-Community/Mentra-Bluetooth-SDK-Starter-Kit). Choose Android, iOS, or React Native / Expo and connect to Mentra Live without first adding the SDK to your own app. > **[Open the Starter Kit on GitHub](https://github.com/Mentra-Community/Mentra-Bluetooth-SDK-Starter-Kit)** > > Browse the example apps and source code. > [!WARNING] > > Use a physical phone for Bluetooth testing. Simulators and emulators do not > provide the real Bluetooth path. ## 1. Clone the Starter Kit ```bash theme={null} git clone https://github.com/Mentra-Community/Mentra-Bluetooth-SDK-Starter-Kit.git cd Mentra-Bluetooth-SDK-Starter-Kit ``` The examples include the OTA flow required to install the Mentra Live glasses software that matches their Bluetooth SDK version. The React Native example owns a forkable `src/ota/` presentation layer over Mentra Engine's tested controller, so you can change its cosmetics without reimplementing OTA sequencing. For `3.3.0` integration instructions and available prebuilt example apps, see [Update Mentra Live](/bluetooth-sdk/software-update). ## 2. Run an Example App ### Android ```bash theme={null} cd examples/android ./gradlew installDebug ``` The Android example is a Kotlin / Jetpack Compose app. It installs the SDK as `com.mentraglass:bluetooth-sdk`. ### iOS ```bash theme={null} cd examples/ios open MentraExample.xcodeproj ``` Choose a physical iPhone as the run destination in Xcode. The iOS example is a SwiftUI app that installs the `MentraBluetoothSDK` Swift package. ### Native macOS Use the Starter Kit `dev` branch's `examples/macos` (AppKit/SwiftPM) or `examples/react-native-macos` for a native Mac host. See [macOS setup](/bluetooth-sdk/macos) for source overrides, permissions, compatible versions, and the distinction from running the iOS TestFlight example on Apple silicon. ### React Native / Expo ```bash theme={null} cd examples/react-native bun install bunx expo prebuild bunx expo run:ios # or bun run android:dev ``` The React Native example installs `@mentra/bluetooth-sdk` and demonstrates the same Device, Camera, Stream, System, and Console flows as the native examples. ## 3. Connect and Explore Open the example app, scan for Mentra Live, select your glasses, and connect. The examples demonstrate: * Connecting, disconnecting, and reconnecting to a saved/default device. * Checking for and installing the glasses software release that matches the SDK. * Reading typed glasses and Bluetooth status snapshots. * Handling button, touch, swipe, head-up, battery, Wi-Fi, hotspot, stream, photo, video upload, audio, and SDK log events. * Controlling gallery mode, capture settings, speaker playback, RGB LEDs, Wi-Fi, hotspot, microphone, camera, and streaming. * Running glasses-to-phone photo and streaming demos. ## Optional Local Media and Streaming Helper The Camera and Stream screens default to a glasses-to-phone flow. You do not need a separate server for that default path. Use the helper only when you turn on **Use cloud server** in the example app and want to test an external endpoint without deploying your own media upload webhook or streaming server. From the repository root, run: ```bash theme={null} python3 examples/local-demo-cloud/server.py ``` Paste the printed LAN `/upload`, RTMP, SRT, or WHIP URL into the corresponding example screen. If Docker is unavailable, the helper starts the media upload webhook and skips streaming with a warning. > [!WARNING] > > Do not use `localhost` in the app. The glasses, phone, and computer must be on a > network where the glasses can reach the printed LAN address. ## Add the SDK to Your App Ready to integrate the SDK into an existing project? Continue to [Build from Scratch](/bluetooth-sdk/build-from-scratch). # React Native / Expo Source: https://docs.mentraglass.com/bluetooth-sdk/react-native-expo Use @mentra/bluetooth-sdk from a React Native or Expo development build. React Native apps use the JavaScript package `@mentra/bluetooth-sdk`. The package contains native Android and iOS code, so it must run in a development build or production native build. For a native Mac app, use the [react-native-macos setup](/bluetooth-sdk/macos#react-native-macos). The phone setup below also runs on compatible Apple silicon Macs as an iOS app; that is distinct from native macOS support. > [!WARNING] > > Expo Go cannot load `@mentra/bluetooth-sdk` because Expo Go does not include the SDK's native modules. > [!WARNING] > > SDK `3.3.0` requires the matching Mentra Live `3.3.0` glasses software. [Update Mentra Live before testing SDK features](/bluetooth-sdk/software-update). ## Install ```bash theme={null} bun add @mentra/bluetooth-sdk@3.3.0 bunx expo install expo-build-properties ``` ## Configure The Expo Plugin Add the SDK plugin and Android native-library packaging rules to your Expo app config. In most Expo apps this is the root `app.json`; in TypeScript-configured apps it is usually the root `app.config.ts`. Merge these keys into the existing top-level `expo` object instead of creating a second config file or replacing unrelated settings: ```json theme={null} { "expo": { "plugins": [ [ "@mentra/bluetooth-sdk", { "node": true } ], [ "expo-build-properties", { "android": { "minSdkVersion": 28, "packagingOptions": { "pickFirst": [ "**/libc++_shared.so", "**/libonnxruntime.so", "**/libonnxruntime4j_jni.so" ] } } } ] ] } } ``` The plugin configures native module registration and the companion Android `lc3Lib` module during prebuild. ## Run ```bash theme={null} bunx expo prebuild bunx expo run:ios # or bunx expo run:android ``` Use a physical phone for Bluetooth, camera, microphone, direct phone photo, and direct phone WebRTC testing. ## Permissions Configure permissions in the same root Expo app config. If your app already has `android.permissions` or `ios.infoPlist`, append or merge these values: ```json theme={null} { "expo": { "android": { "permissions": [ "android.permission.BLUETOOTH_SCAN", "android.permission.BLUETOOTH_CONNECT", "android.permission.ACCESS_FINE_LOCATION", "android.permission.RECORD_AUDIO", "android.permission.POST_NOTIFICATIONS" ] }, "ios": { "infoPlist": { "NSBluetoothAlwaysUsageDescription": "This app connects to your smart glasses over Bluetooth.", "NSMicrophoneUsageDescription": "This app uses the microphone when you enable audio features.", "NSLocalNetworkUsageDescription": "This app can connect to optional local demo servers on your network." } } } } ``` ## Background Operation On iOS The SDK config plugin adds `bluetooth-central` to `UIBackgroundModes` during prebuild, merging it with any modes your app already declares. You do not need to add it yourself. It lets iOS keep delivering BLE events and reconnect to the glasses while the phone is locked, including when the glasses restart their Bluetooth chip during a software update. If the app also keeps microphone capture or an audio session active in the background, add `audio` yourself: ```json theme={null} { "expo": { "ios": { "infoPlist": { "UIBackgroundModes": [ "audio" ], "NSBluetoothAlwaysUsageDescription": "This app connects to your smart glasses over Bluetooth.", "NSMicrophoneUsageDescription": "This app uses the microphone when you enable audio features." } } } } ``` Configure the iOS audio session before continuous mic capture. If your app does not already depend on `expo-audio`, install it with `bunx expo install expo-audio`. Then call `setAudioModeAsync` once during app startup or before enabling the microphone: ```ts theme={null} import {setAudioModeAsync} from 'expo-audio'; await setAudioModeAsync({ allowsRecording: true, allowsBackgroundRecording: true, shouldPlayInBackground: true, playsInSilentMode: true, interruptionMode: 'duckOthers', }); ``` Start continuous microphone capture while the app is foregrounded; iOS background mode lets an active session continue, but the SDK does not start the phone microphone from the background. Use `audio` background mode only when the app genuinely records or plays audio in the background. `bluetooth-central` is the BLE requirement; `audio` is the audio-session requirement for continuous mic/audio behavior. > [!WARNING] > > This SDK version does not enable terminated-app Core Bluetooth state restoration with `CBCentralManagerOptionRestoreIdentifierKey`. If iOS terminates the app, relaunch the app and reconnect from your normal startup flow. ## Basic Usage ```ts theme={null} import BluetoothSdk, {DeviceModels} from '@mentra/bluetooth-sdk'; const devices = await BluetoothSdk.scan(DeviceModels.MentraLive, { timeoutMs: 10_000, onResults: (nextDevices) => { console.log('Nearby glasses:', nextDevices); }, }); const device = await chooseDevice(devices); if (!device) { throw new Error('No Mentra Live glasses selected'); } await BluetoothSdk.connect(device); await BluetoothSdk.requestVersionInfo(); ``` Use the React hooks below to render connection and status state in components. ## React Hooks React Native apps can use the optional hooks subpath for common lifecycle plumbing while keeping the root SDK available for direct commands: | Hook | Use when | | - | - | | `useMentraBluetooth` | You want one React session object for status, scan, connect/disconnect, default-device state, and gallery mode. | | `useBluetoothScan` | You want a focused scan picker without the full connection/session helper. | | `useBluetoothEvent` | You want React-managed subscription cleanup for hardware events such as button presses, touch, battery, Wi-Fi, hotspot, photo, stream, speaking, or microphone events. | ```tsx theme={null} import {Button, Text, View} from 'react-native'; import {DeviceModels} from '@mentra/bluetooth-sdk'; import {useBluetoothEvent, useMentraBluetooth} from '@mentra/bluetooth-sdk/react'; export function DeviceScreen() { const mentra = useMentraBluetooth({ defaultModel: DeviceModels.MentraLive, scanTimeoutMs: 10_000, }); useBluetoothEvent('button_press', (event) => { console.log('Glasses button:', event.buttonId, event.pressType); }); useBluetoothEvent('speaking_status', (event) => { console.log('Speaking:', event.speaking); }); return ( {mentra.glasses.connected ? 'Connected' : 'Disconnected'}