Dog Scraping — Playwright-First Patterns

RULE: All browser automation uses Playwright. Never use fetch, axios, requests, cheerio, or puppeteer.

When to Activate

• Scraping data from any website
• Browser automation (login flows, form submission, navigation)
• Taking screenshots of pages
• Extracting structured data from dynamic/JS-rendered pages
• Testing web interactions programmatically

Core Pattern

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({
  userAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/120",
});

// Block unnecessary resources for speed
await context.route("**/*.{png,jpg,gif,svg,woff,woff2,ico}", r => r.abort());

const page = await context.newPage();

try {
  await page.goto(url, { waitUntil: "networkidle", timeout: 30_000 });
  const data = await page.evaluate(() => {
    return Array.from(document.querySelectorAll(".target")).map(el => ({
      text: el.textContent?.trim(),
    }));
  });
  return data;
} catch (err) {
  await page.screenshot({ path: "debug.png", fullPage: true });
  throw err;
} finally {
  await browser.close();
}

Selector Strategy (best → worst)

1. page.getByRole("button", { name: "Submit" }) — ARIA roles
2. page.getByLabel("Email") — form labels
3. page.getByText("Sign in") — visible text
4. page.locator("[data-testid='target']") — test IDs
5. CSS class selectors — fragile, avoid if possible

Pagination

async function scrapeAllPages(page: Page, startUrl: string) {
  const results: unknown[] = [];
  let url: string | null = startUrl;

  while (url) {
    await page.goto(url, { waitUntil: "domcontentloaded" });
    results.push(...await extractItems(page));

    const next = await page.locator("a[rel='next']").getAttribute("href").catch(() => null);
    url = next;
    if (url) await page.waitForTimeout(800 + Math.random() * 700); // polite delay
  }
  return results;
}

Authentication

// Cookie injection (fastest, no re-login)
await context.addCookies([{
  name: "session",
  value: process.env.SESSION_TOKEN!,
  domain: "app.example.com",
  path: "/",
}]);

// Form login
await page.fill('[name="email"]', process.env.EMAIL!);
await page.fill('[name="password"]', process.env.PASSWORD!);
await page.click('[type="submit"]');
await page.waitForURL(/dashboard/);

// Save state for reuse across runs
await context.storageState({ path: ".auth/state.json" });
// Next run: newContext({ storageState: ".auth/state.json" })

Output

// JSON
await Bun.write("output.json", JSON.stringify(data, null, 2));

// CSV
const headers = Object.keys(data[0]).join(",");
const rows = data.map(r => Object.values(r).join(",")).join("\n");
await Bun.write("output.csv", `${headers}\n${rows}`);

Performance Tips

• waitUntil: "domcontentloaded" is faster than "networkidle" when JS hydration isn't needed
• Use page.evaluate() for bulk DOM extraction (one round trip vs many)
• Reuse browser context across pages in the same domain
• Block images/fonts when only extracting text

Setup

bun add playwright
bunx playwright install chromium