Running in CI

A schedule that works#

GitHub Actions#

The full file: examples/github-workflow.yml. The core:

name: uiscout
on: pull_request
permissions: { contents: read, pull-requests: write }

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }                 # history for --affected
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with: { node-version: 24, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec playwright install --with-deps chromium
      - run: pnpm build --mode test               # with the uiscout plugin, for --affected

      - name: Serve the build
        run: |
          pnpm preview --port 4173 --strictPort &
          for i in $(seq 1 30); do curl -sf http://localhost:4173/ >/dev/null && break; sleep 1; done

      - name: uiscout
        run: pnpm exec uiscout check --url http://localhost:4173/ --mode replay --affected origin/${{ github.base_ref }}

      - name: Comment on the pull request
        if: always() && hashFiles('.uiscout/report.md') != ''
        env: { GH_TOKEN: ${{ github.token }} }
        run: gh pr comment ${{ github.event.pull_request.number }} --edit-last --create-if-none --body-file .uiscout/report.md

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: uiscout
          include-hidden-files: true   # .uiscout/ starts with a dot: skipped otherwise
          path: |
            .uiscout/
            !.uiscout/screens/
            !.uiscout/server.log

On the pull request#

• A comment (from .uiscout/report.md), updated in place on each push:
  • the verdict (✅ no errors or ❌ N errors);
  • graph changes as a diff;
  • the errors;
  • intent coverage;
  • the full report, folded.
• The job summary: the same content, no token needed.
• The artifact: .uiscout/, including graph.html to download and open (without screenshots or the server log).
• A link to the hosted graph, with the step below.

A hosted report per pull request#

The artifact has to be downloaded to be looked at. uiscout site turns the run into a static site instead (the graph page as index.html, the reports; screenshots only with --screenshots; never the server log), to deploy anywhere static. examples/pr-report-vercel.yml deploys it to a Vercel preview on each push and links it from the comment:

      - name: Host the report
        if: always() && hashFiles('.uiscout/graph.html') != '' && env.VERCEL_TOKEN != ''
        run: |
          pnpm exec uiscout site --to uiscout-site
          url=$(npx --yes vercel@latest deploy uiscout-site --yes --token "$VERCEL_TOKEN")
          printf '\n[Open the graph for this pull request](%s)\n' "$url" >> .uiscout/report.md

• The graph page holds each screen's text. Turn on Vercel's Deployment Protection (or use any host only the team can reach) unless the app's screens are public.
• Pull requests from forks get no secrets: the step is skipped and the comment has no link.
• Netlify, Cloudflare Pages or GitHub Pages work the same way: deploy the uiscout-site directory.

When a pull request changes screens on purpose#

The job fails with app.graph.json is out of date or button "X" is gone. On your machine:

uiscout check --mode replay --update
git diff uiscout/              # review the change
git add uiscout/ && git commit -m "Update uiscout baseline"

Notes for CI#

• Screenshots are off by default when CI is set. Turn them on with --screenshots only when the app shows no personal data.
• Run against a build (vite preview): faster and closer to production than a dev server.
• Use replay, so CI needs no backend.
• Block every external service (block; uiscout init lists the ones it finds in the code).
• With auth, give the test account's credentials as repository secrets mapped to the variables the config reads: env: { SCOUT_EMAIL: ${{ secrets.SCOUT_EMAIL }}, SCOUT_PASSWORD: ${{ secrets.SCOUT_PASSWORD }} }.
• webServer works in CI too; against a build, set its command to the preview server (pnpm preview --port 4173 --strictPort).

Edit this page on GitHub