Measure App KPIs

• Use the KPI Visualizer
• Measure and understand your app’s key performance metrics
• Set up measurements
• Interpret results

​

Prerequisites

1. Install the Vega Performance API module:
   npm install @amazon-devices/kepler-performance-api
   
2. Read the following sections:
   • What each KPI measures and its target guidelines. See App KPI metrics and guidelines.
   • How apps launch in different scenarios. See Launch scenarios.
   • When to set performance markers. See Fully drawn marker.
3. To measure foreground memory, video fluidity, or UI fluidity, follow the instructions in Measure Fluidity and Foreground Memory.
4. Choose your measurement method:
   • Measure KPIs in Visual Studio (VS) Code
   • Measure KPI with command-line interface (CLI)

​

App KPI metrics and guidelines

​

Launch scenarios

• Cool start - When a user launches an app for the first time, and the system loads all resources and dependencies into memory.
• Warm start - When a user moves an app from background (an inactive state) to foreground (an active state) with some resources and dependencies already in memory.

​

Fully drawn marker

• Indicates when your app completes loading its essential components.
• Marks when users can start interacting with your app.
• Helps measure TTFD performance.

1. Add the useReportFullyDrawn hook to your app.
2. Place markers at these key points:
   • Cool start - After loading initial data and rendering the main screen.
   • Warm start - When your app becomes responsive after foregrounding.

import { useReportFullyDrawn } from '@amazon-devices/kepler-performance-api';
import React, { useCallback, useEffect, useState } from 'react';
import { useKeplerAppStateManager } from '@amazon-devices/react-native-kepler';
...
...
export const App = () => {
  const reportFullyDrawnCallback = useReportFullyDrawn();
  const KeplerAppStateManager = useKeplerAppStateManager();
  const [appState, setAppState] = useState(KeplerAppStateManager.getCurrentState());

  // Using a useEffect Hook to have the fully drawn reporting performed
  // post first render after cool launch.

  // If the app performs additional asynchronous processing
  // that needs to be completed before it is fully drawn, pass the
  // completion state in the array of dependencies and check the state
  // inside the hook.
  useEffect(() => {
    reportFullyDrawnCallback();
  }, [reportFullyDrawnCallback]);

  // Emit fully drawn marker on the first draw after warm launch.
  const handleAppStateChange = useCallback((stateChange: any) => {
    if (

     appState.match(/^(inactive|background)$/) &&
     stateChange === 'active'
    ) {
      reportFullyDrawnCallback();
    }
    if (stateChange.match(/^(inactive|background|active|unknown)$/)) {
      setAppState(stateChange)
    }
  }, [appState, reportFullyDrawnCallback]);

  useEffect(() => {
    const changeSubscription = KeplerAppStateManager.addAppStateListener(
      'change',
      handleAppStateChange,
    );
    return () => {
      changeSubscription.remove();
    }
  }, [handleAppStateChange]);
  ...
  ...
  return (
    <View style={styles.container}>
      ...
      ...
    </View>
  );
};

​

Measure KPIs in VS Code

​

Step 1: Open the command palette in VS Code

• For Mac: Shift+Command ⌘+P
• For Linux: Ctrl+Shift+P

​

Step 2: Add Vega: Launch App KPI Visualizer and press Enter

​

Step 3: Select a use case

• For memory testing - Include video playback or image loading.
• For streaming - Include typical video watching patterns.

​

Cool start KPIs

• Launch the test app on the device.
• Wait for 10 seconds for the test app to completely load.
• Close the test app.
• Process the KPIs.

​

Warm start KPIs

• Launch the test app on the device.
• Launch another app on the device, moving the test app to the background.
• Launch the test app, moving it to foreground.
• Wait for 15 s for the test app to completely load.
• Close the test app.
• Process the KPIs.

​

Foreground memory KPIs

• Launch the test app on the device.
• Perform the specified steps in test scenario, and capture the KPIs for analysis and reporting.
• Close the test app.
• Process the KPIs.

​

Background memory KPIs

• Launch the test app on the device.
• Launch another app on device, moving the test app to the background.
• Wait for 15 s to collect the KPIs of the test app.
• Close the test app.
• Process the KPIs.

​

UI fluidity KPIs

• Launch the test app on the device.
• Choose your test method:
  • Custom test - Use your own UI interaction scenarios
  • Default test - Uses standard scrolling patterns
    • 2 sets of horizontal scrolls (5 left, 5 right)
    • 2 sets of vertical scrolls (5 down, 5 up)
    • 900 ms between actions
• Select the Cancel button or Esc key to end the test.
• Capture the KPIs for analysis and reporting.
• Close the test app.
• Process the KPIs.

​

Video playback fluidity KPIs

• Launch the test app on the device.
• Perform the steps in video playback test scenario, and capture the KPIs for analysis and reporting.
• Close the test app.
• Process the KPIs.

​

Step 4: Choose whether to ignore trace loss during the test

​

Step 5: View KPI scores in the visualizer window

​

Step 6: Assess results

​

Measure KPIs with CLI commands

1. At the command prompt, run the vega exec perf doctor command to check if your host and target devices are ready.
   vega exec perf doctor [--app-name]
   
   Replace --app-name with the package ID from the manifest.toml file. If multiple devices are running (including Vega Virtual Device), specify the device serial number in the perf commands using the --device-serial-number parameter. Example if only single device is connected:
   # Single device
   vega exec perf doctor --app-name=com.amazondeveloper.keplervideoapp.main
   
   Firmware: Stable build (<device-user> OS 1.1 (TV Mainline/1387)).
   ✅ Network: Connected
   ✅ Free disk space: 43.31 GB available.
   ✅ Appium: Installed (version 2.2.2)
   ✅ Appium driver for Vega: Installed - kepler@3.18.0 [installed (linked from /Users/.../AppiumVegaDriver)]
   ✅ com.amazondeveloper.keplervideoapp.main is installed.
   
   Collecting CPU (4 cores) and Memory data...
   ❌ Max User CPU usage at 241.20%. Check for unwanted processes.
   ❌ Max System CPU usage at 222.80%. Check for unwanted processes.
   ✅ Average User CPU usage at 166.16%
   ✅ Average System CPU usage at 101.84%
   ✅ Min memory available at 30.80%
   ✅ Average memory available at 32.16%
   
   ! Device: Not ready for performance testing. Please review lines with X (error) and ! (warnings) symbols.
   
   Example if multiple devices are connected:
   # Multiple devices
   vega exec perf doctor --app-name=com.amazondeveloper.keplervideoapp.main --device-serial-number=G000XX1234567890
   
2. Start the Vega App KPI Visualizer:
   vega exec perf kpi-visualizer --app-name=[app-name]
   
   For kpi-visualizer, app-name is the only required parameter. Replace [app-name] with the id of default interactive component from the manifest.toml file.
3. (Optional) Specify the default number of iterations. Default iterations: 3 or 30 if –certification is used.
   --iteration [number]
   
4. (Optional) Specify which KPI to measure:
   --kpi [kpi name]
   
   Without this parameter, the visualizer measures cool start TTFF and TTFD KPIs by default.
5. View all available options:
   kpi-visualizer --help
   
   Example:
   vega exec perf kpi-visualizer --help
   
   NAME:
   
   KPI Visualizer Tool
   
   DESCRIPTION:
   
   Measures key performance metrics like app launch times, memory usage, and UI responsiveness to optimize your app's user experience.
   
   SYNOPSIS:
   
   vega exec perf kpi-visualizer [parameters]
   
   Use 'vega exec perf command --help' to retrieve information for a specific command.
   
   PARAMETERS:
   
   --iterations ITERATIONS
         Sets the number of times to run the test. This overrides .conf setting.
   --record-cpu-profiling
         Enables CPU profile recording during test execution.
   --sourcemap-file-path SOURCEMAP_FILE_PATH
         Specifies the path to the source map file.
   --grpc-port port
         Specifies the port number for the gRPC server.
   --certification
         Runs tests in certification mode using 30 iterations and 90th percentile aggregation.
   --expected-video-fps EXPECTED_VIDEO_FPS
         Specifies the target FPS for the app under test.
   --kpi KPI
         (Optional) Specifies the performance metric to measure.
   
         Supported scenarios:
         1. cool-start-latency -  Measures app launch latency from cool start. Includes both TTFF and TTFD by default.
         2. ui-fluidity - Measures smoothness of UI interactions.
         3. warm-start-latency - Measures first frame display latency when resuming an app from background to foreground.
         4. foreground-memory -  Measures app memory usage while in foreground state.
         5. background-memory -  Measures app memory usage while in background state.
         6. video-fluidity - Measures smoothness of video playback. Requires a test scenario (--test-scenario) that initiates video playback.
   
   --test-scenario TEST_SCENARIO
         Specifies the Python script that defines the UI test scenario. Use the generate-test-template command to create a test scenario template.
   
   --monitor-processes MONITOR_PROCESSES [MONITOR_PROCESSES ...]
         Specifies additional services to monitor during testing.
   
         Example:
         --monitor-processes webview.renderer_service
   
   --ignore-trace-loss
         Skips trace data loss verification during test.
   
   --help
         Shows this help message.
   
   --version, -v
         Shows current version of this perf tool.
   %
   

Performance Analyzer KPI Report

Firmware version: Device OS 1.1 (VegaMainlineTvIntegration/XXXX), serial number: XXXXXXXXXXXXXXXX
Date: 01/09/2025, test: app-background, iterations requested: 3, iterations completed: 3, duration: 15 seconds

Memory Statistics
                            | n   | min    | mean    | max    | stdev  | ci (+/-)
App Resident Memory (kB)    | 104 | 131044 | 132139.0| 133136 | 865.2  | 140.8 √

--ignore-trace-loss <true/false>

​

Understand the performance report

1. Test information
   • Date - The date when the system captures the KPI data.
   • Device Serial ID - The unique identifier of the device running the app.
   • App - The name of the app for which the KPI data appears.
   • Number of iterations - The number of times the KPI measurement process runs.
2. KPI Name - The name of the KPI with its unit.
3. Test Name - The name of the test or the completed use case.
4. KPI Health - A color-coded system representing the health of each KPI:
   • 🟢 (Green) - Meets the recommended guideline.
   • 🟡 (Yellow)- Within 10% of the guideline.
   • 🔴 (Red) - Exceeds the guideline by more than 10%.
5. KPI Score - The numeric value of the KPI, which appears in the same unit as specified in the KPI name. If the visualizer can’t calculate the KPIs, it displays “N/A” for the KPI score and list the KPI Health as “Unknown.”
6. Guideline - The recommended value or range for the KPI score based on industry standards or performance targets.

​

Example results

​

Use case: Cool start TTFF and TTFD

​

Use case: Warm start TTFF and TTFD

​

Use case: Foreground memory

​

Use case: Background memory

​

Use case: UI fluidity

​

Use case: Video playback fluidity

​

Open a KPI report

• aggregated-kpi-report-[timestamp].json - Consolidates KPI data from all test scenarios.
• [test-name]-kpi-report-[timestamp].json - Creates one file for each individual test scenario. The [test-name] identifies the specific scenario.

​

Option 1 (Preferred): Use Quick Open

​

From VS Code

1. Find the KPI report file (example: app-launch-kpi-report-[timestamp].json) using VS Code Quick Open or in the project’s generated directory.
2. Select the file once to preview or twice to open in active mode.

​

From the CLI

1. Open a terminal window and enter:
   code <<path-to-recording-file>>
   
   If your terminal doesn’t recognize the code command, open the VS Code Command Palette:
   • Mac: Cmd+Shift+P
   • Linux: Ctrl+Shift+P
2. Run “Shell Command: Install code command in PATH”.
3. Retry the command.

​

Option 2: Use the VS Code Command Palette

1. Open VS Code.
2. Open the command palette:
   • Mac: Cmd+Shift+P
   • Linux: Ctrl+Shift+P
3. Enter Vega: Open Recording View.
4. Select the file you want to open, such as app-launch-kpi-report-[timestamp].json.

​

Related topics

• ▶️ Improve App Performance on Vega
• Measure Fluidity and Foreground Memory
• Identify UI Rendering Issues
• Monitor CPU Usage
• Monitor and Record Memory
• Detect Overdraw