Testing

Running in CI

troupe test behaves a little differently in CI: the runtime lives inside the test process, failed tests are retried, and output is compact. This page covers those defaults, the reports to keep, shards, and exit codes.

What changes when CI is set

Troupe treats the run as CI when the CI environment variable is set to anything but an empty string, 0 or false. Most CI systems set it for you.

	On your machine	In CI
Runtime	The daemon, started in the background and left running	Inside the `troupe test` process; its devices are shut down when the run ends
`retries`	0	2
`reporter`	`list`	`dot`
`webServer` and `bundler`	Reuse a server that already answers	Refuse to start if the address already answers

--no-daemon gives you the in-process runtime without setting CI. Each value in the config wins over these defaults.

The in-process runtime holds the same lock as the daemon, so it refuses to start while a daemon or another in-process run uses the same Troupe home. On one machine, run jobs one after another.

The machine

A CI runner needs what a developer's Mac needs: macOS on Apple silicon, Bun, and for iOS a full Xcode with a simulator runtime, or for Android the SDK with an emulator image. Install has the list. Your job also has to produce the app build that use.app names before the tests start.

A job

Shell

bun install --frozen-lockfile
bunx troupe install ios         # checks, the XCUITest runner, and one device as proof
bunx troupe test --forbid-only

Run troupe install first: a broken environment then fails there, with its fix printed, and not in the middle of the suite. --forbid-only fails the run when someone left a test.only in.

Keep two directories as artifacts of the job, whether it passed or not:

troupe-report/: the HTML report, with a copy of every kept trace.
test-results/: the traces themselves, and any file a reporter wrote there.

Download the report and open it with bunx troupe show-report <dir>.

Reports

Name the reporters in the config, because --reporter on the command line cannot pass options such as an output file:

TypeScript

import { defineConfig } from '@alivelabs/troupe'

const ci = Boolean(process.env.CI)

export default defineConfig({
  forbidOnly: ci,
  reporter: ci
    ? [['dot'], ['junit', { outputFile: 'test-results/junit.xml' }], ['html']]
    : [['list'], ['html']],
})

The JUnit file reports an infrastructure problem as an <error> and a broken test as a <failure>. See reporters.

Retries and flaky tests

With retries: 2 a failed test runs up to twice more. A test that fails and then passes is flaky: the run still passes, and the report lists it. Set failOnFlakyTests: true to fail the run instead.

The default trace mode keeps a trace of every failed attempt, the first included, so a flaky test's failure can be opened afterwards. See traces.

Option	Use
`--retries <n>`	Override the retries for this run
`-x`, `--max-failures <n>`	Stop the run after `n` failed tests
`globalTimeout`	End the whole run after this many milliseconds, with exit code 1
`--repeat-each <n>`	Run each test `n` times, to see whether a new test is stable

Shards

A shard is one part of the suite. --shard 2/3 runs the second of three parts; three machines that each run one shard run the whole suite between them.

Shell

bunx troupe test --shard 1/3

Tests are split in order into nearly equal parts. Tests that belong together, such as the tests of one file or of a serial describe, stay in one shard.

Shards are for several machines. On one machine, more workers do the same job: the runtime runs two devices per platform by default.

To get one report from all shards, have each write a blob and merge them in a last job. In GitHub Actions:

YAML

jobs:
  e2e:
    runs-on: [self-hosted, macOS, ARM64]
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    steps:
      - uses: actions/checkout@v7
      - uses: oven-sh/setup-bun@v2
      - run: bun install --frozen-lockfile
      # Build or download the app that use.app names here.
      - run: bunx troupe install ios
      - run: bunx troupe test --shard ${{ matrix.shard }}/3 --reporter dot,blob
      # Upload blob-report/ and test-results/ as artifacts of this job.

  report:
    needs: e2e
    if: ${{ !cancelled() }}
    runs-on: [self-hosted, macOS, ARM64]
    steps:
      - uses: actions/checkout@v7
      - uses: oven-sh/setup-bun@v2
      - run: bun install --frozen-lockfile
      # Download every shard's blob-report/ into ./all-blobs here.
      - run: bunx troupe merge-reports ./all-blobs --reporter html
      # Upload troupe-report/ as an artifact.

Each shard's own exit code fails its job. merge-reports exits 0 when the merge worked, whatever the tests did.

Exit codes

Code	Meaning
0	Every test had its expected outcome
1	A test failed or timed out, the run passed `globalTimeout`, a spec file did not load, or no test was found
2	Usage or configuration: an unknown flag, no config file, an invalid config, a `test.only` under `--forbid-only`, an option the platform cannot honour, or an `xcodeBuild()` app that is not built yet
3	Infrastructure or environment: a failed environment check, a `webServer` that did not start, a runtime that did not start, or tests that ended `errored` while none failed
130	The run was canceled with Ctrl+C

Codes 1 and 3 tell you who has to act. Code 1 is your test or your app. Code 3 is the machine: read the fix that was printed, or run bunx troupe doctor.

After a job that was killed

A CI run that ends normally shuts its devices down. When the process is killed instead, a small watchdog process that the runtime started removes the devices it left about a minute later, unless it was killed too. To clean up yourself, for example at the start of the next job:

Shell

bunx troupe runtime sweep --apply

It removes only devices and processes that Troupe created. See the runtime daemon.

Running in CI

What changes when CI is set

The machine

A job

Reports

Retries and flaky tests

Shards

Exit codes

After a job that was killed

Next