Key Takeaways
- Frameworks die of flake, not missing features, architecture, not discipline, keeps a suite trusted.
- Selector strategy is a contract with the frontend team: data-testid everywhere, enforced in code review.
- Definition of done for handover: the client team merges their own new test without asking you anything.
An Automation framework survives when a green run is trusted enough to gate a release, and that trust is an architecture outcome, not a discipline outcome. These are the decisions we make in every Playwright build so the suite is still alive in a year.
Selector strategy is a contract, not a habit
Standardize on data-testid attributes agreed with the frontend team, enforce them in code review, and ban text/CSS-position selectors in the framework lint rules. Flake starts where selectors depend on layout.
Design for parallel from day one
- Every test creates its own data, no shared users, no shared carts
- API-based setup and teardown; the UI is for the behavior under test, not for arranging state
- Storage-state login once per worker, never a UI login per test
Project structure that survives growth
Structure is the difference between a suite Engineers extend and one they route around. The layout we hand over, boring on purpose:
e2e/
├─ tests/ # specs by feature, mirroring the product's nav
│ ├─ checkout/
│ ├─ accounts/
│ └─ admin/
├─ fixtures/ # auth, data factories, custom test extensions
├─ pages/ # page objects: locators + intent methods only
├─ api/ # typed API clients for setup/teardown
├─ data/ # builders, no static JSON blobs
└─ playwright.config.ts
# rules that keep it clean:
# pages/ contain NO assertions; tests/ contain NO selectors
# every helper is <50 lines or it becomes a fixtureSharding and the merge gate
Past a few hundred tests, one machine is the bottleneck. Playwright's built-in sharding splits the suite across parallel CI machines, four shards typically turn a 40-minute wall into an 11-minute one, which is the difference between a gate people respect and a gate people bypass. Merge the blob reports into one HTML artifact so a failure is one click from its trace, and set the gate policy explicitly: stable suite green to merge, quarantine suite reported but non-blocking, full cross-browser sweep nightly rather than per-commit. Speed is a trust feature.
The fixture architecture that makes it work
The decisions above become concrete in the fixture layer. Authentication happens once per worker via storage state; every test builds its own data through the API; the UI is reserved for the behavior actually under test:
// fixtures.ts, the framework's real API
export const test = base.extend<{ order: Order }>({
// storage-state login: once per worker, never per test
storageState: "auth/buyer.json",
// every test gets ITS OWN order, created via API, deleted after
order: async ({ request }, use) => {
const order = await createOrder(request, { items: 2 });
await use(order);
await deleteOrder(request, order.id);
},
});
test("buyer can cancel an order", async ({ page, order }) => {
await page.goto(`/orders/${order.id}`);
await page.getByRole("button", { name: "Cancel order" }).click();
await expect(page.getByText("Cancelled")).toBeVisible();
});Anti-patterns we remove first on rescue engagements
- UI login in every test, the single biggest time and flake tax; storage state removes it in an afternoon.
- Shared 'test user 47', one seeded account touched by forty tests is a race condition with a username.
- XPath tied to layout, selectors that break on a div wrap are maintenance invoices, not tests.
- Assertions on implementation ('spinner disappears') instead of outcomes ('order appears in list').
- sleep() anywhere, every hardcoded wait is either too long (slow suite) or too short (flaky suite), usually both across environments.
Treat flake as a defect class
Quarantine flaky tests automatically, track a flake-rate metric per suite, and fix or delete, a suite with 2% random failure is a suite nobody trusts at 2 a.m. on release night.
Reporting people actually read
A failing suite nobody can interpret gets bypassed just as fast as a flaky one. The reporting bar we build to: a failure is one click from its trace (Playwright's HTML report with trace-on-retry does this natively); failures are grouped by root cause, not alphabetically, so one broken login fixture reads as one problem rather than forty; and the summary a non-engineer sees leads with what matters, which user-facing flows are at risk, not raw counts. Post the one-line digest to the team channel on every main-branch run: green with duration, or red with the top failure and its owner. When the report answers 'what broke and who's on it' in ten seconds, the suite stays part of the team's nervous system instead of a tab nobody opens.
Hand over the framework, not a dependency
Documentation, naming conventions, and a 'write your first test' guide are deliverables. Our definition of done: the client team merges their own new test without asking us anything.
The handover package, itemized
A framework isn't delivered until the client team runs it alone. Our handover checklist: a README that gets a new Engineer to a green local run in under fifteen minutes; a 'write your first test' guide with a worked example; documented selector and data conventions; CI configuration with artifact retention; a flake triage runbook; and one pairing session where the client team, not us, merges a new test. That last item is the acceptance test for the whole engagement, and it's the standard our Automation practice is contractually held to.
Want us to run this on your product?
A free 30-minute assessment. We'll tell you what's working, what's costing you time, and where to start. Findings delivered within days.
Get a Free QA Assessment