Integrate

The live stream

A session's device streams its screen as live video over one WebSocket and takes touch, keys and buttons back over the same socket. This page covers tickets, watching and driving, the stream client and input.

The stream's delay from glass to glass is 17 to 34 ms at the median, measured on one Mac.

How it works

1. You mint a ticket for a session with streams.ticket. The answer is { url, ticket }.
2. The viewer opens a WebSocket to url and presents the ticket.
3. The server sends video. A viewer whose ticket allows it sends input.

On a local runtime url is the runtime's own /v1/stream. In Troupe Cloud it is the device host's listener: video and input travel between the viewer and the host, never through the API.

iOS Simulators and Android Emulators stream H.264. Mac Catalyst has no stream, and streams.ticket is refused there with CAPABILITY_UNSUPPORTED.

Tickets

await troupe.call('sessions.wait', { sessionId, until: 'stream' })
const { url, ticket } = await troupe.call('streams.ticket', { sessionId, cap: 'drive' })

• Capability. cap: 'watch' receives video. cap: 'drive' may also send input. A watch ticket needs the devices:read scope; a drive ticket needs devices:drive too.
• Lifetime. A ticket is valid for at most 10 minutes and is checked only when the socket opens. An open socket keeps running past it. The server sends fresh tickets over the socket, and the stream client presents the newest one when it reconnects.
• Scope. A ticket works for one session. When the session ends, every ticket for it stops working.
• Secrecy. A ticket is a credential. It travels in the WebSocket subprotocol offer, never in a URL, a query string or a cookie. Do not log it.

The stream opens before the session is ready: once the device is booted and its screen is captured, while the driver still starts. until: 'stream' waits for that moment. Before it, streams.ticket fails with SESSION_NOT_READY. See Wait.

Watch and drive

One viewer drives a session at a time. Others watch: up to 64 watching sockets per session by default.

A waiting socket becomes the driver by sending takeover. The displaced driver becomes a watcher and needs a new drive ticket to drive again. When the driver closes, the oldest waiting socket is promoted.

A session held by a troupe test worker has no driver: the test holds the device, and a takeover is refused.

Live input and planned actions share one queue per device. A live gesture holds its turn from its first finger down to its last finger up, so an input.tap from a test or an agent never lands between two live fingers.

The stream client

@alivelabs/troupe-client/stream is the browser client. It needs WebSocket and WebCodecs.

import {
  StreamConnection,
  StreamDecoder,
  browserCodecs,
  canvasSink,
} from '@alivelabs/troupe-client/stream'

const getTicket = (cap?: 'watch' | 'drive') =>
  troupe.call('streams.ticket', { sessionId, cap: cap ?? 'drive' })

const connection = new StreamConnection({ source: await getTicket(), getTicket })
const decoder = new StreamDecoder({
  sink: canvasSink(canvas),
  backend: browserCodecs,
  onKeyframeNeeded: () => connection.requestKeyframe(),
})

connection.subscribe((event) => {
  if (event.type === 'frame') decoder.push(event.frame)
  if (event.type === 'status') console.log(event.status.state, event.status.role)
})
connection.start()

StreamConnection keeps the socket alive. StreamDecoder paints the frames on a canvas. In a React app, DeviceScreen does all of this for you.

The connection moves through the states idle, connecting, playing, reconnecting, failed and closed. Its reconnect rules are fixed:

• A dropped socket reconnects after 250 ms, doubling to 4 seconds, with jitter.
• A refused ticket asks getTicket for a new one at 0, 1, 2, 4 and 8 seconds, then the connection is failed.
• Silence for two and a half heartbeats counts as a drop.

connection.status has what a view needs:

close() stops for good. restart() starts over after failed. takeover() asks to drive.

Call decoder.suspend() when the page is hidden and decoder.resume() when it is shown again: the decoder releases WebCodecs meanwhile and restarts from a keyframe.

Input

Only the driver's input is sent; from any other role send returns false. Check status.input first: a device says which controls work now, and offers none while its live input is unavailable.

Three helpers turn DOM events into stream messages:

import {
  KeyboardInput,
  PointerInput,
  browserClock,
  containRect,
} from '@alivelabs/troupe-client/stream'

const rect = () =>
  containRect(canvas.getBoundingClientRect(), { width: canvas.width, height: canvas.height })

const pointer = new PointerInput({
  rect,
  send: (message) => connection.send(message),
  maxContacts: () => connection.status.input?.maxContacts ?? 0,
  timestamps: () => connection.status.input?.touchTimestamps === true,
  clock: browserClock,
})
canvas.addEventListener('pointerdown', (event) => pointer.down(event))
canvas.addEventListener('pointermove', (event) => pointer.move(event))
canvas.addEventListener('pointerup', (event) => pointer.up(event))
canvas.addEventListener('pointercancel', (event) => pointer.cancel(event))

const keyboard = new KeyboardInput({
  send: (message) => connection.send(message),
  mode: () => connection.status.input?.keyboard ?? 'none',
})
// The canvas needs a tabindex to take keyboard focus.
canvas.addEventListener('keydown', (event) => {
  if (keyboard.keydown(event)) event.preventDefault()
})
canvas.addEventListener('keyup', (event) => keyboard.keyup(event))

Buttons and rotation are single messages:

connection.send({ type: 'button', name: 'home', phase: 'down' })
connection.send({ type: 'button', name: 'home', phase: 'up' })
connection.send({ type: 'rotate', orientation: 'landscapeLeft' })

When the server refuses an input, the connection emits an inputRefused event with a reason such as busy, view-only or not-ready.

On the wire

A client in another language needs these facts.

• Upgrade. Offer the subprotocols troupe-stream.v1 and troupe.ticket.<ticket>. The server answers with troupe-stream.v1 alone. A missing or bad ticket is an accepted upgrade that is closed with a code.
• Hello. Send { "type": "hello", "v": 1, "codecs": ["h264"] } first, within 5 seconds. The server answers with its own hello (role, codec, device, input), then config, then frames.
• Video. Binary messages are one tag byte and a payload: 0x01 the H.264 decoder description (avcC), 0x02 a keyframe, 0x03 a delta frame, 0x04 a JPEG image for the first paint. Ignore other tags.
• Control. Text messages are JSON with a type. Ignore types you do not know.
• Heartbeats. Send heartbeat when you have sent nothing for the heartbeatMs the server's hello names.

Next

• The React components: DeviceScreen wraps the stream client
• Sessions: create the session a stream belongs to
• Hosted device sessions: the same stream from Troupe Cloud