Handling timeouts in Vitest on Bazel
Recently I have been trying to get a handle on flakiness in our test infrastructure at work. The most common thing left was timeouts while running Vitest. However, Vitest when receiving a SIGTERM just cleans up and calls process.exit without any information on what was running.
We run Vitest in Bazel, so my first attempt was looking for a way to pass down the timeout Bazel will use onto Vitest. However, it doesn't support a global timeout (only per test, hook, etc). Then I looked into any other mechanism it might have, but came up short.
Finally I realised I could hook into it's rather nice Reporter interface. It's slightly hacky because we need to register a SIGTERM handler before Vitest does. In practice though it works.
Out of interest Vitest's logger does provide a onTerminalCleanup
that could be useful here. However, it will be called for both SIGTERM and exit without anyway to distinguish the two.
Also I don't really write typescript code, so no idea if this looks like decent TS code. Also stored in a gist
import type { TestCase, TestSuite, Vitest } from 'vitest/node'
import type { Reporter } from 'vitest/reporters'
/**
* Catches SIGTERM from Bazel so we can report what tests are currently
* running. Vitest currently does not support reporting this information nor
* configuring a global timeout.
*/
class TimeoutReporter implements Reporter {
private runningTests = new Set<{ id: string; name: string; file: string; startTime: number }>()
private runningSuites = new Set<{ name: string; startTime: number }>()
public onInit(_: Vitest): void {
this.runningTests.clear()
this.runningSuites.clear()
// Vitest registers a SIGTERM listener in logger.ts which calls
// process.exit(). So we need to make sure we run before it.
process.prependOnceListener('SIGTERM', () => {
const now = Date.now()
process.stderr.write('\n\nSIGTERM received!\n')
for (const test of this.runningTests) {
const duration = now - test.startTime
process.stderr.write(`Currently running test ${test.name} (${test.file}) - running for ${duration}ms\n`)
}
for (const suite of this.runningSuites) {
const duration = now - suite.startTime
process.stderr.write(`Currently running suite ${suite.name} - running for ${duration}ms\n`)
}
process.stderr.write('Vitest was terminated by Bazel due to timeout\n')
})
}
public onTestCaseReady(testCase: TestCase): void {
this.runningTests.add({
id: testCase.id,
name: testCase.fullName,
file: testCase.module.moduleId,
startTime: Date.now(),
})
}
public onTestCaseResult(testCase: TestCase): void {
for (const test of this.runningTests) {
if (test.id === testCase.id) {
this.runningTests.delete(test)
break
}
}
}
public onTestSuiteReady(testSuite: TestSuite): void {
this.runningSuites.add({ name: testSuite.fullName, startTime: Date.now() })
}
public onTestSuiteResult(testSuite: TestSuite): void {
for (const suite of this.runningSuites) {
if (suite.name === testSuite.fullName) {
this.runningSuites.delete(suite)
break
}
}
}
}