How verification works

This is the step that decides whether a migration is a candidate for a PR at all, so it's worth understanding exactly what it does and doesn't prove.

What happens

Once Sunset writes a migration for a matched call site, it applies the diff to an isolated git worktree, not your working branch, and runs your repo's own test suite against a test environment pinned to the new Shopify API version. If your tests pass, the migration is a candidate for a draft PR. If they fail, Sunset either retries the migration with the failure as feedback, or, if it can't resolve it, marks the case low-confidence and reports why rather than opening a PR against failing tests.

What "green" actually means

A passing test run means the migration didn't break anything your test suite currently checks for. That's a real, useful signal, and it's also bounded by how good your test suite already is. If your repo has thin coverage around the code Sunset touched, a green result is weaker evidence than it would be on a well-tested repo. We report the test result plainly in every PR rather than presenting a checkmark as proof of correctness it isn't.

What it doesn't prove

Verification runs against a test environment, not your live store, and it validates against your existing tests, not against every possible real-world case a client's live traffic might hit. It's not a substitute for your own review. Every Sunset PR is a draft precisely because verification is a strong gate, not a guarantee.

Confidence scoring

Where a call site is unambiguous, the field or endpoint clearly matches something on Shopify's sunset schedule, and tests pass cleanly, Sunset reports high confidence. Where the API version is set dynamically, a helper function is used in more than one context, or the changelog itself is vague about the exact behavior change, Sunset reports low confidence and says specifically what's uncertain, rather than either guessing silently or refusing to flag it at all.

Why we don't trust tests alone as an oracle

An earlier version of this system leaned on mocked API responses that validate request and response shape but not real provider behavior. That's a weak gate, since it can pass without proving the migrated code actually works against the real API. Verification now runs against a genuine Shopify development store pinned to the target version, closer to how the code will actually be exercised in production, and we're honest in the report about where that fidelity still has gaps.