Skip to content

Check code coverage in CI#3788

Open
mbg wants to merge 2 commits intomainfrom
mbg/test/code-coverage
Open

Check code coverage in CI#3788
mbg wants to merge 2 commits intomainfrom
mbg/test/code-coverage

Conversation

@mbg
Copy link
Copy Markdown
Member

@mbg mbg commented Mar 27, 2026

Adds a dev dependency on c8 and uses it to generate code coverage information for the tests (when the new test-coverage script is run). The results can then be checked against the thresholds with npm run coverage. Both new scripts are now used in the pr-checks workflow. The previous npm run test script does not calculate code coverage so that local testing is not slowed down.

The thresholds are configured to roughly match our current code coverage. Increasing the code coverage (and thresholds) is outside the scope of this PR.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Environments:

  • Testing/None - This change does not impact any CodeQL workflows in production.

How did/will you validate this change?

  • Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).
  • End-to-end tests - I am depending on PR checks (i.e. tests in pr-checks).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Development/testing only - This change cannot cause any failures in production.

How will you know if something goes wrong after this change is released?

  • Other - Please provide details.

Are there any special considerations for merging or releasing this change?

  • No special considerations - This change can be merged at any time.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change.
  • Confirm the readme and docs have been updated if necessary.

@mbg mbg self-assigned this Mar 27, 2026
@github-actions github-actions bot added the size/XS Should be very easy to review label Mar 27, 2026
@mbg mbg marked this pull request as ready for review March 27, 2026 18:59
@mbg mbg requested a review from a team as a code owner March 27, 2026 18:59
Copilot AI review requested due to automatic review settings March 27, 2026 18:59
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds code coverage measurement and enforcement to CI for the repo’s AVA unit tests, while keeping the default local npm test workflow unchanged for speed.

Changes:

  • Add c8 as a dev dependency and introduce test-coverage / coverage npm scripts.
  • Update the PR checks workflow to run coverage-enabled tests and enforce coverage thresholds.
  • Ignore generated coverage output in .gitignore and increase a flaky test timeout.

Reviewed changes

Copilot reviewed 3 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/artifact-scanner.test.ts Increases timeout for a non-Windows integration-style test.
package.json Adds c8, new scripts, and coverage threshold configuration.
package-lock.json Locks new c8 dependency and transitive packages.
.gitignore Ignores coverage/ output directory.
.github/workflows/pr-checks.yml Runs test-coverage and then checks/enforces coverage in CI.

Comment on lines +98 to 102
"c8": {
"functions": 80,
"lines": 80,
"branches": 80
}
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c8 is currently run without --all, so the coverage check only applies to files actually loaded during tests; completely untested modules won’t count as 0% and can slip through. Consider enabling all and narrowing src/include/exclude (e.g., to the transpiled build/ output) so coverage reflects the whole codebase under test.

Copilot uses AI. Check for mistakes.
Comment on lines 55 to +61
- name: Run unit tests
if: always()
run: npm test
run: npm run test-coverage

- name: Check code coverage
if: always()
run: npm run coverage
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The coverage check step is marked if: always(), so it will run even when the preceding test run fails. In that case it may fail due to missing/partial coverage data and add noise on top of the real test failure; consider running the coverage check only when the test-coverage step succeeds (default success()), or give the test step an id and gate on its outcome.

See below for a potential fix:

        id: test-coverage
        if: always()
        run: npm run test-coverage

      - name: Check code coverage
        if: steps.test-coverage.outcome == 'success'

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS Should be very easy to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants