> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mentraglass.com/llms.txt
> Use this file to discover all available pages before exploring further.

# MentraOS ASG Client Guidelines

> Overview and development guidelines for the MentraOS ASG (Android Smart Glasses) Client, which runs on Android-based smart glasses like Mentra Live.

# MentraOS Android Smart Glasses (ASG) Client Development Guidelines

This guide provides an overview and development guidelines for the MentraOS ASG (Android Smart Glasses) Client, which runs on Android-based smart glasses like Mentra Live.

## Overview

The MentraOS ASG Client is an Android application that runs directly on Android-based smart glasses hardware. It serves as the bridge between the glasses' hardware capabilities and the MentraOS ecosystem, enabling features like camera capture, display rendering, Bluetooth communication, and network management.

## Architecture

### Core Components

1. **AsgClientService** - The main service that coordinates all functionality:
   * Parses Bluetooth messages from the MentraOS mobile app
   * Handles command processing from those messages
   * Parses messages from Mentra Live's microcontroller (button presses, swipes)
   * Triggers actions like taking pictures, videos, and starting RTMP streams
   * Manages network state and battery status reporting

2. **Manager Systems** - Interface-based factory pattern for device abstraction:
   * **NetworkManager** - WiFi and hotspot control
   * **BluetoothManager** - BLE communication with phone app
   * **MediaCaptureService** - Photo/video capture and upload
   * **RtmpStreamingService** - Live video streaming

### Network Manager System

The NetworkManager uses a factory pattern to support different device types:

```java theme={null}
// Interface defining network operations
public interface INetworkManager {
    void setWifiEnabled(boolean enabled);
    void connectToWifi(String ssid, String password);
    void startHotspot(String ssid, String password);
    // ... other network operations
}
```

**Implementations:**

* **K900NetworkManager** - For Mentra Live (K900) devices using proprietary broadcasts
* **SystemNetworkManager** - For devices with system permissions using reflection
* **FallbackNetworkManager** - For regular devices, prompts user for manual configuration

### Bluetooth Manager System

Similarly structured for Bluetooth/BLE operations:

```java theme={null}
// Interface for Bluetooth operations
public interface IBluetoothManager {
    void initialize();
    void startAdvertising();
    void sendData(byte[] data);
    boolean isConnected();
    // ... other BLE operations
}
```

**Implementations:**

* **K900BluetoothManager** - Uses serial port communication with BES2700 chip
* **StandardBluetoothManager** - Full BLE GATT server implementation for standard Android devices

### Media System

The media system handles camera button presses and media capture:

**Button Press Flow**:

* Physical button press detected by microcontroller
* MCU sends command to ASG Client (`cs_pho` for short press, `cs_vdo` for long press)
* ASG Client checks button press mode configuration:
  * **PHOTO mode**: Takes photo/video locally
  * **APPS mode**: Sends button event to phone/apps
  * **BOTH mode**: Does both actions
* Photos are captured and queued for upload when connected

### RTMP Streaming

Supports live video streaming with four main commands:

1. **start\_rtmp\_stream** - Initiates streaming to specified RTMP URL
2. **stop\_rtmp\_stream** - Terminates active stream
3. **keep\_rtmp\_stream\_alive** - Must be sent every 15 seconds to prevent 60-second timeout
4. **get\_rtmp\_status** - Queries current streaming status

The stream automatically stops if no keep-alive is received for 60 seconds. The system uses ACK-based reliability where each keep-alive must be acknowledged by the glasses.

## Development Guidelines

### Code Style

* **Java**:
  * Use Java SDK 17
  * Classes: PascalCase
  * Methods: camelCase
  * Constants: UPPER\_SNAKE\_CASE
  * Member variables: mCamelCase (with m prefix)
  * 2-space indentation
  * EventBus for component communication

### Adding Support for New Glasses

To add support for new Android-based smart glasses:

1. **Fix Device Detection** (REQUIRED):

   ```java theme={null}
   // In NetworkManagerFactory.java and BluetoothManagerFactory.java
   // Change this:
   if (true || isK900Device()) {  // Currently forced to K900

   // To this:
   if (isK900Device()) {  // Proper device detection
   ```

2. **Create a SmartGlassesCommunicator** in augmentos\_core:
   * Extend `SmartGlassesCommunicator` base class
   * Implement BLE client to connect to StandardBluetoothManager
   * Handle display, audio, and sensor capabilities
   * See `ANDROID_FIRMWARE_GUIDE.md` for detailed instructions

3. **Update Device Detection**:

   ```java theme={null}
   private boolean isYourGlassesDevice() {
       String model = Build.MODEL.toLowerCase();
       return model.contains("your-device-identifier");
   }
   ```

4. **Configure Manager Selection**:
   * StandardBluetoothManager will work for most devices
   * Choose appropriate NetworkManager based on permissions
   * Test with both System and Fallback network managers

### Important Notes on Current State

**StandardBluetoothManager**: A fully implemented BLE GATT server exists with:

* Complete BLE GATT server functionality
* Auto-pairing capabilities
* MTU negotiation
* Serial-like data exchange over characteristics

However, it's not currently used because:

* The K900 (Mentra Live) uses proprietary serial communication via K900BluetoothManager
* The MentraLiveSGC.java in augmentos\_core is designed specifically for K900's serial protocol
* To use StandardBluetoothManager, you'd need a new SmartGlassesCommunicator that acts as a BLE client

### Building and Testing

1. **Environment Setup**:

   ```bash theme={null}
   # Copy environment file
   cp .env.example .env

   # For local development, modify .env:
   MENTRAOS_HOST=localhost  # or your computer's IP
   MENTRAOS_PORT=8002
   MENTRAOS_SECURE=false
   ```

2. **Build Requirements**:
   * Android Studio with Java SDK 17
   * Set Gradle JDK to version 17 in Android Studio settings

3. **Testing on Mentra Live Glasses**:

   Mentra Live only supports ADB over WiFi (no USB ADB). To connect:

   a. **Setup WiFi ADB Connection**:

   ```bash theme={null}
   # 1. Pair Mentra Live with the MentraOS mobile app
   # 2. In the app, connect glasses to your WiFi network
   # 3. Note the IP address shown on the "Glasses" screen
   # 4. Connect via ADB (computer must be on same WiFi network):
   adb connect [IP_ADDRESS]:5555

   # Example:
   adb connect 192.168.1.123:5555

   # Verify connection:
   adb devices
   # Should show: 192.168.1.123:5555    device
   ```

   b. **Build and Install**:

   ```bash theme={null}
   # Build the APK
   ./gradlew assembleDebug

   # Install on glasses
   adb install app/build/outputs/apk/debug/app-debug.apk

   # For local development server, forward ports:
   adb reverse tcp:8002 tcp:8002
   ```

### Message Processing

The client processes JSON commands from the cloud:

```java theme={null}
// Example command processing in AsgClientService
switch (type) {
    case "take_photo":
        String requestId = dataToProcess.optString("requestId", "");
        mMediaCaptureService.takePhotoAndUpload(photoFilePath, requestId);
        break;

    case "start_stream":
        String streamId = dataToProcess.optString("streamId", "");
        String streamUrl = dataToProcess.optString("streamUrl", "");
        // Protocol auto-detected from URL scheme (srt://, rtmp://, https://)
        StreamCommandHandler.handleCommand("start_stream", dataToProcess);
        break;

    case "display_text":
        // For glasses with displays
        showTextOnDisplay(text);
        break;
}
```

## Compatible Devices

Currently supported:

* **Mentra Live** (K900)

The ASG Client could potentially be adapted to work on other Android-based smart glasses with sufficient modifications, such as TCL Rayneo X2/X3, INMO Air 2/3, and others.

## Next Steps for Contributors

1. **Fix the K900 detection** in both factory classes
2. **Create device-specific SmartGlassesCommunicators** in augmentos\_core
3. **Add device detection** for your specific glasses model
4. **Test the StandardBluetoothManager** with your device
5. **Submit PRs** with your device support additions

## Resources

* [MentraOS Mobile App Guidelines](/contributing/mentraos-manager-guidelines) - Mobile app development guide