Testing

Reporters

A reporter turns a run into output: lines in the terminal, an HTML report, a JUnit file for CI. This page covers the built-in reporters, the HTML report, blob reports and merging, and writing your own.

Choosing reporters

Set reporter in the config to a name, or to a list when you want several or need options:

TypeScript

export default defineConfig({
  reporter: [['list'], ['html'], ['junit', { outputFile: 'test-results/junit.xml' }]],
})

bunx troupe test --reporter dot,html replaces the config's reporters for one run. Reporters named on the command line take no options.

Reporter	Writes	Options
`list`	One line per test as it ends, then the failures and the counts. The default	none
`line`	A progress line per test, each failure as it happens, then the counts	none
`dot`	One character per test. The default when `CI` is set	none
`html`	The report folder, `troupe-report/`	`outputFolder`
`json`	The whole run as one JSON document	`outputFile`; without it, standard output
`junit`	JUnit XML	`outputFile`; without it, standard output
`blob`	A zip of the run's events for `merge-reports`, in `blob-report/`	`outputDir`

In the terminal

list prints each test with its project, file, line and title. This is a sample run of six tests, with its durations left out:

Output

Running 6 tests using 2 workers

  ✓  1 [ios] › e2e/cart.e2e.ts:5:3 › cart › adds an item (…)
  ✘  2 [ios] › e2e/cart.e2e.ts:12:3 › cart › applies a coupon (…)
  ✓  3 [ios] › e2e/cart.e2e.ts:12:3 › cart › applies a coupon (retry #1) (…)
  ✘  4 [ios] › e2e/cart.e2e.ts:20:3 › cart › shows the total (…)
  ✘  5 [ios] › e2e/cart.e2e.ts:20:3 › cart › shows the total (retry #1) (…)
  ✘  6 [ios] › e2e/cart.e2e.ts:28:3 › cart › pays [errored] (…)
  -  7 [ios] › e2e/cart.e2e.ts:35:3 › cart › gift wrap
  ✘  8 [ios] › e2e/cart.e2e.ts:40:3 › cart › waits for delivery (…)

  2 failed
    [ios] › e2e/cart.e2e.ts:20:3 › cart › shows the total
    [ios] › e2e/cart.e2e.ts:40:3 › cart › waits for delivery
  1 errored
    [ios] › e2e/cart.e2e.ts:28:3 › cart › pays
  1 flaky
    [ios] › e2e/cart.e2e.ts:12:3 › cart › applies a coupon
  1 skipped
  1 passed (…)

A ✓ is an attempt that ended as expected, a ✘ one that did not, and a - a skipped test. Between the lines and the counts, every failure is printed in full, as when a test fails shows. The counts keep three things apart:

failed: your test or your app.
errored: infrastructure, such as a lost device. It does not mean your app is wrong.
flaky: failed, then passed on a retry.

dot prints · for a pass, F for a failure, T for a timeout, E for an errored test, × for a failure that will be retried and ° for a skip.

The HTML report

The html reporter writes a folder at the end of the run. outputFolder names it; the default is troupe-report, beside the config.

Output

troupe-report/
  index.html, assets/     the viewer
  report.json             the run
  data/                   traces, screenshots and other attachments, named by their SHA-256

Open it with troupe show-report, which serves the folder on a loopback port and prints the address:

Shell

bunx troupe show-report              # troupe-report/
bunx troupe show-report ./ci-report  # another folder

The report must be served over HTTP: a browser does not let the page read its data from file://. Neither the reporter nor show-report opens a browser.

The report's first page shows the run's status and duration, a count per outcome that filters the list, a table by project, and the tests grouped by file with a search box. A test's page has one tab per attempt, each with its errors (the message, the source line and the accessibility excerpt), its steps as a tree with timings, its attachments, its output, and a link to its trace.

The reporter replaces the folder on each run, but only when the folder is empty or holds a previous report. A folder with anything else in it is refused, not deleted.

JSON and JUnit

json writes the run as one document: status, startTime, durationMs, stats (expected, unexpected, flaky, skipped, errored), the projects, and every test with its outcome and its results, down to steps, errors and attachments.

junit writes one <testsuite> per project and file. A failed test is a <failure> of type FAILURE, a timed-out one a <failure> of type TIMEOUT, and an errored one an <error> whose type is the error code, so a CI dashboard can tell an infrastructure problem from a broken test.

Blob reports and merging

When a suite runs as shards on several machines, each shard writes a blob and one command merges them into a single report.

TypeScript

// troupe.config.ts
export default defineConfig({
  reporter: process.env.CI ? [['dot'], ['blob']] : [['list'], ['html']],
})

Each shard writes blob-report/report-<i>-of-<n>.zip. Collect the zips into one directory, then:

Shell

bunx troupe merge-reports ./all-blobs --reporter html
bunx troupe merge-reports ./all-blobs --reporter junit > junit.xml

merge-reports replays the shards as one run through any reporters; without --reporter it uses list. It exits 0 when the merge worked, whatever the tests did: let each shard's own exit code fail the job.

A blob holds the run's events, not its files. Small attachments travel inside the events. Traces and other attachments on disk stay in each shard's test-results/, and the merged HTML report includes those it finds at the paths the shards recorded. Copy each shard's test-results/ to the same path on the merging machine to keep them.

A reporter of your own

A reporter is a module whose default export is an object, or a class that is constructed with its options. Name it by path in the config:

TypeScript

// reporters/infrastructure.ts
import type { Reporter } from '@alivelabs/troupe/reporter'

const reporter: Reporter = {
  onTestEnd(params, run) {
    if (params.status !== 'errored') return
    const test = run.cases.get(params.testId)
    console.log(`infrastructure problem in: ${test?.summary.title}`)
  },
  onEnd(params) {
    const { expected, errored } = params.stats
    console.log(`${params.status}: ${expected} as expected, ${errored} errored`)
  },
}

export default reporter

TypeScript

export default defineConfig({ reporter: [['list'], ['./reporters/infrastructure.ts']] })

Method	Called
`onBegin(params, run)`	Once, with the config, the projects and every test
`onTestBegin`, `onTestEnd`	For each attempt of a test
`onStepBegin`, `onStepEnd`	For each step: `test.step`, assertions, fixtures, hooks and runtime calls
`onStdOut`, `onStdErr`	For output from a test or a worker
`onError`	For an error outside any test
`onEnd(params, run)`	Once, with the run's status and counts
`onExit()`	After everything, for writing files
`printsToStdio()`	To say whether the reporter writes to the terminal

Every method is optional. Each event method receives the event's fields and the run as built so far. A reporter written for Playwright's reporter API works through playwrightAdapter from the same entry point; it sees an errored test as failed with an annotation of type troupe-errored.

Reporters

Choosing reporters

In the terminal

The HTML report

JSON and JUnit

Blob reports and merging

A reporter of your own

Next