Implement Closed Captions and Subtitles in Vega

In this article, you’ll implement closed captions and subtitles in your Vega app using the W3C-compliant text track APIs. Closed captions provide audio transcriptions to help viewers with disabilities or ambient noise, while subtitles translate dialogue into different languages. Text tracks can be broadly divided into two categories:

In-band: Text tracks delivered within the media segment, for example, CEA captions and MP4 text tracks (WebVTT/TTML as separate tracks).
Out-of-band (OOB): Text tracks delivered as a separate file from the media stream. In ABR streaming, these files are referenced in the playlist or manifest file. In some cases, caption files are associated with media content through a database and aren’t present in the manifest file.

Prerequisites

Vega development environment set up
Basic knowledge of React Native and TypeScript
@amazon-devices/react-native-w3cmedia package version 2.1.14 or later
Vega SDK v0.10 or later

Add the required privilege

To add the accessibility privilege, add the following to manifest.toml.

[[wants.privilege]]
id = "com.amazon.devconf.privilege.accessibility"

Install the VTTCue polyfill

The @amazon-devices/react-native-w3cmedia package provides a polyfill for VTTCue, which is an implementation of the TextTrackCue interface. Install the polyfill during your app initialization.

import { VTTCue } from '@amazon-devices/react-native-w3cmedia';

// Inside your polyfill class:
static install() {
    console.log('Installing W3CMedia polyfills');
    global.window.VTTCue = global.VTTCue = VTTCue;
    if(!window.VTTCue) {
      console.log("VTTCue not polyfilled");
    }
}

Add the `KeplerCaptionsView` component

The KeplerCaptionsView component renders closed captions and subtitles on screen. Use this component when operating in pre-buffering mode with VideoPlayer and AudioPlayer components. Add the component to your render tree.

import { KeplerCaptionsView } from '@amazon-devices/react-native-w3cmedia';

const onCaptionViewCreated = (captionsHandle: string): void => {
    videoPlayer.setCaptionViewHandle(captionsHandle);
}

// Inside your component's return statement:
<View style={{ backgroundColor: "white", alignItems: "stretch",
    width: deviceWidth, height: deviceHeight}}>
        <KeplerCaptionsView
            onCaptionViewCreated={onCaptionViewCreated}
            show={true}
            style={{ width: '100%',
            height: '100%',
            top: 0,
            left: 0,
            position: 'absolute',
            backgroundColor: 'transparent',
            flexDirection: 'column',
            alignItems: 'center',
            zIndex: 2}}
        />
</View>

Make sure to set the show property to true to render the captions.

Add a text track for app-parsed captions

When your app handles fetching and parsing caption data directly, add a text track to your HTMLMediaElement implementation and populate it with VTTCue objects. This approach applies to any text track that your app parses, whether the source is in-band or out-of-band. If you use a JavaScript player such as Shaka Player, it manages track creation internally and you don’t need to follow this step.

const textTrack = videoPlayer.addTextTrack('subtitles', 'SampleTextTrack', 'en');

Add cues to the text track

After parsing caption data, add cues to the text track using VTTCue.

const vttCue = new VTTCue(startTime, endTime, payload);
vttCue.lineAlign = lineAlign;
vttCue.positionAlign = positionAlign;
if (size) {
    vttCue.size = size;
}
textTrack.addCue(vttCue);

Listen for natively parsed text tracks

The platform automatically detects in-band captions like CEA 608 and 708. When media segments are appended to the SourceBuffer, the platform parses these captions in the native layer, which is more efficient than JavaScript-based parsing. Register for the addtrack event to receive notifications when text tracks are detected.

videoPlayer.textTracks.addEventListener("addtrack", onTextTrackAdded);

const onTextTrackAdded = (event: Event): void => {
    const textTrackList: TextTrackList | undefined | null = media.current?.textTracks;
    if(textTrackList === undefined || textTrackList === null) {
        return;
    }
    for (let i = 0; i < textTrackList.length; i++) {
        const textTrack = textTrackList[i];
        console.log(`text track id = ${textTrack.id} \
            kind = ${textTrack.kind} \
            language = ${textTrack.language} \
            label = ${textTrack.label} \
            mode = ${textTrack.mode}`);
   }
}

Add a text track with a caption file URL

If your app has the caption file URL but lacks parsing capability, the platform can fetch, parse, and render known subtitle formats. Pass the caption file URL and its MIME type when adding the text track. This approach is not W3C-compliant, is not the recommended method, and is on the deprecation path. When possible, use app-side or JavaScript player-based parsing instead.

const textTrack = videoPlayer.addTextTrack("subtitles", 'SampleTextTrack', 'en',
    contentUri, contentMimeType);

The contentUri and contentMimeType parameters are not standard W3C parameters. The following MIME types are supported.

application/ttml+xml
text/vtt
application/x-subtitle
application/x-subtitle-sami
application/x-subtitle-tmplayer
application/x-subtitle-mpl2
application/x-subtitle-dks
application/x-subtitle-qttext
application/x-subtitle-lrc
application/x-subtitle-vtt

Select and display a text track

Set the text track mode to showing to render it on screen.

textTrack.mode = 'showing';

By default, text tracks are added with mode set to hidden.

Use ShakaPlayer for automatic caption parsing

ShakaPlayer can automatically parse and render captions from manifest files and media segments. When using ShakaPlayer, the player handles text track creation and cue management internally, so you don’t need to manually call addTextTrack() or addCue(). Configure ShakaPlayer to enable text track rendering.

shakaPlayer.configure({
    autoShowText: shaka.config.AutoShowText.ALWAYS,
});

After loading content, set text track visibility to true to display captions.

async load(content: any) {
    await shakaPlayer.load(content.uri);
    shakaPlayer.setTextTrackVisibility(true);
}

Make sure you’re using Vega SDK v0.21 or later with the ShakaPlayer patches from shaka-rel-v4.6.18-r2.5.tar.gz or newer.

Test the implementation

Build and run your app on a Fire TV device or emulator:

Navigate to a screen with video playback.
Start playing media that contains captions.
Verify that captions appear on screen.
Test switching between different caption tracks if available.

Known limitations

Vertical rendering of subtitles is not supported.

const cue = new VTTCue(startTime, endTime, text);
cue.vertical = ""; // "rl" or "lr" not supported

Custom cue window size is not supported. Captions window size is auto only.

const cue = new VTTCue(startTime, endTime, text);
cue.size = 20; // Custom cue size is not supported.

If position is not provided in VTTCue, subtitles and captions are positioned at the bottom of the screen to prevent overlapping.
Multiple KeplerCaptionsView instances within the same process are not supported.
Only up to one CEA 708 native track is supported.
Roll-up mode for native captions is not supported. For more details, see Roll-up captions.
Native caption tracks don’t include language metadata. The language parameter contains channel identifiers such as CC1, CC2, CC3, and CC4.
VTTRegion is not supported.
CSS styling of subtitles and captions is not supported.

FAQ

Do apps need to parse in-band captions if the platform supports native CEA 608/708 parsing?

Ideally, native parsing of CEA 608/708 should be enabled and JavaScript-based parsing should be avoided. Apps should parse captions only when native parsing doesn’t work or the app needs access to the caption text.

The ShakaPlayer API doesn't report natively parsed caption tracks. How can the app detect them?

Use a combined track management strategy:

Use a JavaScript player API (for example, ShakaPlayer) for subtitle tracks.
Use the VideoPlayer (MediaElement) from the W3C Media API for caption tracks.

The following example queries all available text tracks using this approach.

getTextTracks(): TextTrackInfo[] {
    let textTrackInfoList: TextTrackInfo[] = [];

    if (this.player) {
        let shakaTextTrackList = this.player.getTextTracks();
        for (let track of shakaTextTrackList) {
            textTrackInfoList.push({
                id: 'shaka:' + track.id.toString(),
                kind: track.kind,
                language: track.language,
                label: track.label,
                mode: (track.active === true ? 'showing' : 'hidden'),
                playerTrackData: ({ type: 'shaka', track: track }),
            });
        }
    }

    if (this.mediaElement) {
        let nativeTextTrackList = this.mediaElement.textTracks;
        for (let track of nativeTextTrackList) {
            if (track.label == 'Shaka Player TextTrack') {
                continue;
            }
            textTrackInfoList.push({
                id: 'native:' + track.id.toString(),
                kind: track.kind,
                language: track.language,
                label: track.label,
                mode: track.mode,
                playerTrackData: ({ type: 'native', track: track }),
            });
        }
    }
    return textTrackInfoList;
}

How do you enable or disable text tracks?

Use the same combined track management strategy. The following example shows how to select a text track.

setTextTrack(newTrack: TextTrackInfo | null, currTrack: TextTrackInfo | null): void {
    if (currTrack) {
        if (currTrack.playerTrackData.type == 'native') {
            currTrack.playerTrackData.track.mode = 'hidden';
        } else if (currTrack.playerTrackData.type == 'shaka') {
            this.player?.setTextTrackVisibility(false);
        }
    }
    if (newTrack) {
        if (newTrack.playerTrackData.type == 'native') {
            newTrack.playerTrackData.track.mode = 'showing';
        } else if (newTrack.playerTrackData.type == 'shaka') {
            this.player?.selectTextTrack(newTrack.playerTrackData.track);
            this.player?.setTextTrackVisibility(true);
        }
    }
}

Enabling caption parsing by the JavaScript player reports duplicate text tracks. How can this be avoided?

When relying on native caption parsing, disable the JavaScript player’s in-band caption parsers to prevent duplicate text tracks.To disable in-band caption parsing in ShakaPlayer, unregister the media segment caption parsers.

shaka.media.ClosedCaptionParser.unregisterParser('video/mp4');
shaka.media.ClosedCaptionParser.unregisterParser('video/mp2t');

If relying on JavaScript in-band parsing instead, use JavaScript player APIs for text track management:

Use getTextTracks() to retrieve available text tracks.
Use setTextTrackVisibility(isVisible) to enable or disable text tracks.

Is subtitle and caption styling supported?

Yes. The following styling tags are supported:

Class span: text color and text background color
Italics span
Bold span
Underline span

For more details, see the WebVTT specification.

Is VTTRegion supported?

VTTRegion is not supported.

Is CSS styling of subtitles and captions supported?

CSS styling of subtitles and captions is not supported.

​Prerequisites

​Add the required privilege

​Install the VTTCue polyfill

​Add the KeplerCaptionsView component

​Add a text track for app-parsed captions

​Add cues to the text track

​Listen for natively parsed text tracks

​Add a text track with a caption file URL

​Select and display a text track

​Use ShakaPlayer for automatic caption parsing

​Test the implementation

​Known limitations

​FAQ

​Related content