Device Control

Control device hardware using session.audio, session.camera, session.led, and session.location. Each manager provides methods for specific hardware features.

Audio Control

Control audio playback and text-to-speech on glasses.

Text-to-Speech

Convert text to speech:

// Basic TTS
await session.audio.speak('Hello from your app!');

// With custom voice settings
await session.audio.speak('Welcome back!', {
  voice_id: 'your_voice_id',
  voice_settings: {
    stability: 0.5,
    speed: 1.2
  },
  volume: 0.8
});

Play Audio Files

Play audio from URL:

const result = await session.audio.playAudio({
  audioUrl: 'https://example.com/notification.mp3',
  volume: 0.8,
  stopOtherAudio: true
});

if (result.success) {
  session.logger.info('Audio played successfully');
} else {
  session.logger.error('Audio failed:', result.error);
}

Stop Audio

// Stop all currently playing audio
session.audio.stopAudio();

Audio Options

Option	Type	Default	Description
`audioUrl`	string	required	URL to audio file
`volume`	number	1.0	Volume level (0.0-1.0)
`stopOtherAudio`	boolean	true	Stop other audio before playing

TTS Options:

Option	Type	Description
`voice_id`	string	Voice ID to use
`model_id`	string	Model ID (default: eleven_flash_v2_5)
`voice_settings`	object	Voice parameters (stability, speed, etc.)
`volume`	number	Volume level (0.0-1.0)

Camera Control

Capture photos and stream video from camera-equipped glasses.

Check Camera Availability

if (session.capabilities?.hasCamera) {
  // Camera available
  await session.camera.requestPhoto();
} else {
  session.logger.warn('Camera not available on this device');
}

Take Photos

Basic photo capture:

const photo = await session.camera.requestPhoto();
session.logger.info('Photo captured:', photo.filename);

With options:

const photo = await session.camera.requestPhoto({
  saveToGallery: true,
  size: 'large',
  compress: 'medium'
});

Photo data returned:

Field	Type	Description
`filename`	string	Photo filename
`url`	string	Photo URL
`timestamp`	Date	When photo was taken

Photo Options

Option	Type	Default	Description
`saveToGallery`	boolean	false	Save to device gallery
`size`	string	’medium'	'small’, ‘medium’, or ‘large’
`compress`	string	’none'	'none’, ‘medium’, or ‘heavy’
`customWebhookUrl`	string	-	Override default webhook URL
`authToken`	string	-	Auth token for custom webhook

RTMP Streaming

Start streaming:

await session.camera.startStream({
  rtmpUrl: 'rtmp://live.example.com/stream/key',
  video: {
    resolution: '1280x720',
    framerate: 30,
    bitrate: 2500000
  },
  audio: {
    enabled: true,
    bitrate: 128000
  }
});

session.logger.info('Stream started');

Monitor stream status:

session.camera.onStreamStatus((status) => {
  session.logger.info('Stream status:', status.status);
  // status values: 'starting', 'streaming', 'stopped', 'error'

  if (status.status === 'error') {
    session.logger.error('Stream error:', status.error);
  }
});

Stop streaming:

await session.camera.stopStream();

Managed Streaming

Zero-infrastructure streaming (MentraOS handles RTMP):

// Start managed stream
const result = await session.camera.startManagedStream({
  video: {
    resolution: '1920x1080',
    framerate: 30
  }
});

session.logger.info('Streaming at:', result.streamUrl);
session.logger.info('Watch at:', result.viewerUrl);

// Stop managed stream
await session.camera.stopManagedStream();

LED Control

Control RGB LEDs on supported glasses.

Check LED Availability

if (session.capabilities?.hasLight) {
  const lightInfo = session.capabilities.light;
  session.logger.info(`Device has ${lightInfo.count} LEDs`);

  // Check for RGB support
  const hasRGB = lightInfo.lights?.some(led => led.isFullColor);
  if (hasRGB) {
    // Can use any color
  }
}

Turn LED On

Basic usage:

await session.led.turnOn({
  color: 'blue',
  ontime: 2000 // milliseconds
});

With brightness:

await session.led.turnOn({
  color: 'green',
  brightness: 128, // 0-255
  ontime: 1000
});

Turn LED Off

await session.led.turnOff();

LED Blink Pattern

await session.led.blink({
  color: 'red',
  ontime: 500,
  offtime: 500,
  count: 5 // blink 5 times
});

LED Options

Option	Type	Description
`color`	string	’red’, ‘green’, ‘blue’, ‘white’, etc.
`brightness`	number	0-255 (optional)
`ontime`	number	Duration in milliseconds
`offtime`	number	Off duration for blinking
`count`	number	Number of blinks

Location Access

Access GPS location from the user’s device. Continuous location updates:

session.location.subscribeToStream({ accuracy: 'high' }, (location) => {
  session.logger.info('Location:', location.lat, location.lng);
  session.logger.info('Accuracy:', location.accuracy, 'meters');

  // Use location data
  session.layouts.showTextWall(
    `Lat: ${location.lat.toFixed(4)}\n` +
    `Lng: ${location.lng.toFixed(4)}`
  );
});

Get Latest Location

One-time location fetch:

const location = await session.location.getLatestLocation();
if (location) {
  session.logger.info('Current location:', location.lat, location.lng);
}

Location Data

Field	Type	Description
`lat`	number	Latitude
`lng`	number	Longitude
`accuracy`	number	Accuracy in meters
`altitude`	number	Altitude in meters (optional)
`timestamp`	number	When location was captured

Accuracy Levels

Level	Description
`high`	Best accuracy, higher battery usage
`medium`	Balanced accuracy and battery
`low`	Lower accuracy, battery efficient

Common Patterns

Audio Feedback

protected async onSession(session: AppSession, sessionId: string, userId: string) {
  // Welcome message
  await session.audio.speak('Welcome to the app!');

  // Listen for voice commands
  session.events.onTranscription(async (data) => {
    if (data.isFinal) {
      // Acknowledge command
      await session.audio.speak('Processing your request');

      // Process...
      const result = await this.processCommand(data.text);

      // Speak result
      await session.audio.speak(result);
    }
  });
}

Photo on Button Press

protected async onSession(session: AppSession, sessionId: string, userId: string) {
  if (!session.capabilities?.hasCamera) {
    session.layouts.showTextWall('No camera available');
    return;
  }

  session.events.onButtonPress(async (data) => {
    if (data.button === 'select') {
      session.layouts.showTextWall('Capturing photo...');

      try {
        const photo = await session.camera.requestPhoto({
          saveToGallery: true
        });

        session.layouts.showTextWall('Photo saved!');
        await session.audio.speak('Photo captured successfully');
      } catch (error) {
        session.logger.error('Photo failed:', error);
        session.layouts.showTextWall('Photo failed');
      }
    }
  });
}

LED Notifications

async function showNotification(session: AppSession, type: 'info' | 'success' | 'error') {
  if (!session.capabilities?.hasLight) return;

  const colors = {
    info: 'blue',
    success: 'green',
    error: 'red'
  };

  await session.led.turnOn({
    color: colors[type],
    ontime: 1000
  });
}

// Usage
session.events.onTranscription(async (data) => {
  if (data.isFinal) {
    await showNotification(session, 'info');
    // Process command...
    await showNotification(session, 'success');
  }
});

Location-Based Alerts

protected async onSession(session: AppSession, sessionId: string, userId: string) {
  const targetLocation = { lat: 37.7749, lng: -122.4194 };
  const alertRadius = 0.1; // km

  session.location.subscribeToStream({ accuracy: 'high' }, async (location) => {
    const distance = this.calculateDistance(
      location.lat, location.lng,
      targetLocation.lat, targetLocation.lng
    );

    // Update display
    session.dashboard.content.writeToMain(`Distance: ${distance.toFixed(2)} km`);

    // Alert when nearby
    if (distance < alertRadius) {
      await session.audio.speak('You are near your destination');

      if (session.capabilities?.hasLight) {
        await session.led.blink({
          color: 'green',
          ontime: 200,
          offtime: 200,
          count: 3
        });
      }
    }
  });
}

private calculateDistance(lat1: number, lng1: number, lat2: number, lng2: number): number {
  const R = 6371; // Earth radius in km
  const dLat = (lat2 - lat1) * Math.PI / 180;
  const dLng = (lng2 - lng1) * Math.PI / 180;
  const a = Math.sin(dLat/2) * Math.sin(dLat/2) +
            Math.cos(lat1 * Math.PI / 180) * Math.cos(lat2 * Math.PI / 180) *
            Math.sin(dLng/2) * Math.sin(dLng/2);
  return R * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
}

Streaming Control

class StreamingApp extends AppServer {
  private isRecording = false;

  protected async onSession(session: AppSession, sessionId: string, userId: string) {
    if (!session.capabilities?.hasCamera) {
      session.layouts.showTextWall('Camera not available');
      return;
    }

    session.layouts.showTextWall('Say "start" or "stop"');

    session.events.onTranscription(async (data) => {
      if (!data.isFinal) return;

      const text = data.text.toLowerCase();

      if (text.includes('start') && !this.isRecording) {
        await this.startRecording(session);
      } else if (text.includes('stop') && this.isRecording) {
        await this.stopRecording(session);
      }
    });

    // Monitor stream status
    session.camera.onStreamStatus((status) => {
      session.dashboard.content.writeToMain(`Stream: ${status.status}`);
    });
  }

  private async startRecording(session: AppSession) {
    try {
      const result = await session.camera.startManagedStream({
        video: { resolution: '1280x720', framerate: 30 }
      });

      this.isRecording = true;
      session.layouts.showTextWall('Recording started');
      await session.audio.speak('Recording');

      session.logger.info('Stream URL:', result.viewerUrl);
    } catch (error) {
      session.logger.error('Failed to start stream:', error);
      await session.audio.speak('Failed to start recording');
    }
  }

  private async stopRecording(session: AppSession) {
    await session.camera.stopManagedStream();
    this.isRecording = false;

    session.layouts.showTextWall('Recording stopped');
    await session.audio.speak('Recording stopped');
  }
}

Best Practices

Check Capabilities First

Always verify hardware availability before using device features:

// ✅ Good
if (session.capabilities?.hasCamera) {
  await session.camera.requestPhoto();
}

// ❌ Avoid - may crash on devices without camera
await session.camera.requestPhoto();

Handle Errors Gracefully

Device operations can fail - always handle errors:

// ✅ Good
try {
  const photo = await session.camera.requestPhoto();
  session.layouts.showTextWall('Success!');
} catch (error) {
  session.logger.error('Photo failed:', error);
  session.layouts.showTextWall('Photo failed');
}

Provide User Feedback

Let users know what’s happening:

// ✅ Good
session.layouts.showTextWall('Taking photo...');
const photo = await session.camera.requestPhoto();
session.layouts.showTextWall('Photo captured!');
await session.audio.speak('Photo saved');

// ❌ Avoid - silent operations confuse users
await session.camera.requestPhoto();

Clean Up Resources

Stop streams and unsubscribe when done:

protected async onStop(sessionId: string, userId: string, reason: string) {
  // Streams stop automatically on disconnect
  // Location subscriptions clean up automatically
  session.logger.info('Session ended');
}

Permissions Required

Device features require specific permissions:

Feature	Permission	Notes
Audio playback	None	Always available
Camera (photo/video)	CAMERA	Set in dev console
LED control	None	If hardware available
Location	LOCATION	Set in dev console

Set permissions in the Developer Console. Users approve them when installing your app.

Next Steps

Capabilities

Learn about device capabilities

Permissions

Understand permission system

Audio Manager

Complete audio API reference

Camera Manager

Complete camera API reference

Getting Started

Core Concepts

SDK Reference

Audio Control

Text-to-Speech

Play Audio Files

Stop Audio

Audio Options

Camera Control

Check Camera Availability

Take Photos

Photo Options

RTMP Streaming

Managed Streaming

LED Control

Check LED Availability

Turn LED On

Turn LED Off

LED Blink Pattern

LED Options

Location Access

Get Latest Location

Location Data

Accuracy Levels

Common Patterns

Audio Feedback

Photo on Button Press

LED Notifications

Location-Based Alerts

Streaming Control

Best Practices

Permissions Required

Next Steps

Capabilities

Permissions

Audio Manager

Camera Manager

Getting Started

Core Concepts

SDK Reference

​Audio Control

​Text-to-Speech

​Play Audio Files

​Stop Audio

​Audio Options

​Camera Control

​Check Camera Availability

​Take Photos

​Photo Options

​RTMP Streaming

​Managed Streaming

​LED Control

​Check LED Availability

​Turn LED On

​Turn LED Off

​LED Blink Pattern

​LED Options

​Location Access

​Subscribe to Location Updates

​Get Latest Location

​Location Data

​Accuracy Levels

​Common Patterns

​Audio Feedback

​Photo on Button Press

​LED Notifications

​Location-Based Alerts

​Streaming Control

​Best Practices

​Permissions Required

​Next Steps

Capabilities

Permissions

Audio Manager

Camera Manager

Audio Control

Text-to-Speech

Play Audio Files

Stop Audio

Audio Options

Camera Control

Check Camera Availability

Take Photos

Photo Options

RTMP Streaming

Managed Streaming

LED Control

Check LED Availability

Turn LED On

Turn LED Off

LED Blink Pattern

LED Options

Location Access

Subscribe to Location Updates

Get Latest Location

Location Data

Accuracy Levels

Common Patterns

Audio Feedback

Photo on Button Press

LED Notifications

Location-Based Alerts

Streaming Control

Best Practices

Permissions Required

Next Steps