Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.mentraglass.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

Hardware modules provide access to glasses and phone capabilities. Always check session.capabilities before using hardware features.

​
Camera Module

Access the glasses camera through session.camera:

​
Request Photo

// Simple photo capture
const photo = await session.camera.requestPhoto();
console.log(`Photo type: ${photo.mimeType}`);
// photo.photoData is an ArrayBuffer

// With metadata
const photo = await session.camera.requestPhoto({
  metadata: { 
    purpose: 'document-scan',
    timestamp: Date.now()
  }
});

​
Video Streaming

// RTMP streaming
const result = await session.camera.startLocalLivestream({
  rtmpUrl: 'rtmp://your-server.com/live/stream-key'
});

if (result.success) {
  console.log(`Streaming started: ${result.streamId}`);
}

// Stop streaming
await session.camera.stopLocalLivestream();

// Managed streaming (creates URLs for viewers)
const managed = await session.camera.startLivestream({
  title: 'Live Tour',
  description: 'Walking through the museum'
});

console.log(`HLS URL: ${managed.hlsUrl}`);
console.log(`DASH URL: ${managed.dashUrl}`);

​
Stream Status

session.camera.onLocalLivestreamStatus((status) => {
  console.log(`Stream status: ${status.state}`);
  if (status.error) {
    console.error(`Stream error: ${status.error}`);
  }
});

​
Camera FOV / ROI

Control the camera’s field of view and crop region with session.camera.setFov(). The fov value (82—118 degrees) directly sets the FOV of the raw image the camera produces — the crop happens at the sensor level, not in software after capture. At fov: 118 the camera uses its full sensor with no crop. Values below 118 tell the sensor to apply an ROI (Region of Interest) crop so the output image is that narrower angle. Lower values mean a narrower view (more zoomed in). roiPosition controls where the crop window sits vertically on the sensor: "top" shifts it upward, "bottom" shifts it downward, and "center" keeps it in the middle. This lets apps pan the visible area up or down without the user physically tilting the glasses. At fov: 118 there is no crop, so roiPosition has no effect. Output resolution vs. ROI: The sensor crop gets smaller as FOV decreases, but the delivered frame often stays at the same output size (for example 1080p / 1920×1080 in typical video paths). The pipeline scales the cropped region up to that resolution, so the picture stays full-frame and “HD,” but you are not gaining unlimited real detail as you zoom in—past roughly 102°, angular resolution (actual sharpness per degree in the scene) often stops improving, and at the narrowest FOVs you may see softer or noisier results because fewer native sensor samples are stretched across the same pixel grid, in addition to limits from optics and the ISP. 
// Narrow the FOV to 92 degrees (defaults to center crop)
await session.camera.setFov({ fov: 92 });

// Narrow FOV with the crop shifted to the top of the sensor
await session.camera.setFov({ fov: 92, roiPosition: 'top' });

// Full sensor, no crop
await session.camera.setFov({ fov: 118 });
Parameters:
  • fov (required) — number between 82 and 118. The FOV in degrees of the camera output. 118 = full sensor, no crop.
  • roiPosition (optional) — "center" | "top" | "bottom". Where the crop window sits vertically. Defaults to "center". Ignored when fov is 118.
Behavior notes:
  • Requires CAMERA permission.
  • Fire-and-forget: the promise resolves when the message is sent, not when the hardware confirms the change.
  • When fov is 118, roiPosition is forced to "center" internally (no crop means no positioning).
// Error handling
try {
  await session.camera.setFov({ fov: 92, roiPosition: 'bottom' });
} catch (error) {
  console.error('Failed to set camera FOV:', error);
}

​
Audio Module

Play audio on glasses through session.audio:

​
Play Audio File

// Play from URL
await session.audio.play('https://example.com/sound.mp3');

// With options
await session.audio.play('https://example.com/sound.mp3', {
  volume: 0.8  // 0-1
});

// Stop playback
await session.audio.stop();

​
Text-to-Speech

// Simple TTS
await session.audio.speak('Hello world');

// With options
await session.audio.speak('Hola mundo', {
  language: 'es-ES',
  voice: 'female'
});

​
Location Module

Access device location through session.location:

​
Continuous Updates

// Subscribe to location updates
const cleanup = await session.location.subscribeToStream({
  accuracy: 'high',  // 'high' | 'medium' | 'low'
  updateInterval: 5000  // milliseconds
}, (location) => {
  console.log(`Location: ${location.lat}, ${location.lng}`);
  if (location.accuracy) {
    console.log(`Accuracy: ${location.accuracy} meters`);
  }
});

// Later: unsubscribe
await session.location.unsubscribeFromStream();

​
One-time Location

// Get current location once
const location = await session.location.getLatestLocation({
  accuracy: 'high'
});

console.log(`Current location: ${location.lat}, ${location.lng}`);

​
Dashboard Module

Control the always-visible dashboard through session.dashboard:

​
Update Content

// Write to dashboard
await session.dashboard.write({
  text: '3 new messages'
});

// Target specific views
await session.dashboard.write({
  text: 'Navigation: 0.5 mi'
}, {
  targets: ['dashboard', 'always_on']
});

​
Dashboard Mode

// Listen for mode changes
session.dashboard.onModeChange((mode) => {
  console.log(`Dashboard enabled: ${mode.enabled}`);
  console.log(`Always on: ${mode.alwaysOn}`);
});

// For system apps only:
const content = await session.dashboard.systemDashboard.read();

​
Settings Module

Manage app settings through session.settings:

​
Read Settings

// Get single setting
const theme = await session.settings.get('theme', 'light');

// Check if setting exists
const hasTheme = await session.settings.has('theme');

​
Listen for Changes

// Specific setting
session.settings.on('theme', (newValue) => {
  console.log(`Theme changed to: ${newValue}`);
  applyTheme(newValue);
});

// Any setting change
session.settings.onChange((key, newValue) => {
  console.log(`${key} changed to ${newValue}`);
});

​
Capability Checking

Always check capabilities before using hardware: 
async function capturePhoto(session: AppSession) {
  if (!session.capabilities.hasCamera) {
    await session.layouts.showTextWall('Camera not available');
    return null;
  }
  
  try {
    const photo = await session.camera.requestPhoto();
    return photo;
  } catch (error) {
    console.error('Photo capture failed:', error);
    await session.layouts.showTextWall('Failed to capture photo');
    return null;
  }
}

​
Best Practices

try {
  const location = await session.location.getLatestLocation();
  processLocation(location);
} catch (error) {
  console.error('Location error:', error);
  // Use fallback or cached location
  useFallbackLocation();
}
Hardware access requires permissions. Request only what you need:
// Only subscribe to location if needed
if (userWantsNavigation) {
  await session.location.subscribeToStream({}, handleLocation);
}

// Store cleanup functions
const cleanups = [];

// Add location subscription
const locationCleanup = await session.location.subscribeToStream(
  {}, 
  handleLocation
);
cleanups.push(locationCleanup);

// Clean up when done
session.events.onDisconnected(() => {
  cleanups.forEach(cleanup => cleanup());
});

​
Common Patterns

​
Photo with Feedback

async function takePhotoWithFeedback(session: AppSession) {
  // Show capturing message
  await session.layouts.showTextWall('Capturing photo...');
  
  try {
    const photo = await session.camera.requestPhoto();
    
    // Show success
    await session.layouts.showTextWall('✓ Photo captured', {
      durationMs: 2000
    });
    
    return photo;
  } catch (error) {
    // Show error
    await session.layouts.showTextWall('✗ Photo failed', {
      durationMs: 2000
    });
    throw error;
  }
}

​
Location-based Features

class LocationTracker {
  private lastLocation: any;
  private totalDistance = 0;
  
  constructor(private session: AppSession) {}
  
  async start() {
    await this.session.location.subscribeToStream({
      accuracy: 'high',
      updateInterval: 5000
    }, (location) => {
      if (this.lastLocation) {
        // Calculate distance
        const distance = this.calculateDistance(
          this.lastLocation, 
          location
        );
        this.totalDistance += distance;
        
        // Update display
        this.session.layouts.showReferenceCard({
          title: 'Distance Tracker',
          text: `Total: ${this.totalDistance.toFixed(2)} km`
        });
      }
      
      this.lastLocation = location;
    });
  }
  
  private calculateDistance(loc1: any, loc2: any): number {
    // Haversine formula for distance between coordinates
    // ... implementation
    return distance;
  }
}