# Collecting with a browser — Crawlee + stealth

> **What this chapter covers**
> How to attach `puppeteer-extra-plugin-stealth` to Crawlee's `PuppeteerCrawler` —
> plus **what to check before you attach it**, and what you will certainly run
> into afterwards.

## First — do you actually need a browser?

The collection step in [Case study — the content factory](/guide/ht-case-factory)
is a single `terminal` command. Most collection ends there. Work down this list.

| What the site gives you | What to use |
|---|---|
| An official API | The API |
| RSS · sitemap · JSON endpoint | That |
| HTML the server already rendered | `fetch` + an HTML parser |
| A screen that only appears once scripts run | Now Crawlee |

A browser is the **most expensive option**. One Chrome tab costs hundreds of
megabytes, adds seconds per page, and above all **breaks the most often** — a
small markup change on their side stops you.

```mermaid
graph TD
  A["You need data"] --> B{"Is there an API?"}
  B -->|Yes| C["API — done here"]
  B -->|No| D{"Already in the HTML?"}
  D -->|Yes| E["fetch + parser"]
  D -->|No| F["Browser — Crawlee"]
```

## Then — are you allowed to?

This check comes **before** the technical one. Break these and the problem is not
getting blocked, it is contracts and law. The stealth plugin solves none of them.

- **robots.txt and terms of service.** If automated collection is disallowed, it
  does not matter whether it is technically possible.
- **Behind a login or paywall.** That is account-terms territory. "Pulling my own
  data with my own account" and "getting around the barrier" are different things.
- **Personal data.** Public does not mean collectable. You need a basis.
- **Load on their servers.** Cap concurrency and pace yourself — that is what
  Crawlee's `maxConcurrency` and `maxRequestsPerMinute` are for.

**The easy case is your own site, your own data, or somewhere that has explicitly
agreed.** Checking how your own service renders, or gathering your own product
listings, sidesteps most of the above. For anything else, asking first is usually
faster than engineering around it.

## The wiring

Install:

```
npm install crawlee puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
```

The key is `launchContext.launcher` — this is where you tell Crawlee to launch
the browser with puppeteer-extra **instead of** plain puppeteer.

```js
import { PuppeteerCrawler } from "crawlee";
import puppeteerExtra from "puppeteer-extra";
import stealthPlugin from "puppeteer-extra-plugin-stealth";

// Register **before** creating the crawler.
puppeteerExtra.use(stealthPlugin());

const crawler = new PuppeteerCrawler({
  launchContext: {
    launcher: puppeteerExtra,
    launchOptions: { headless: true },
  },
  maxConcurrency: 2,
  maxRequestsPerMinute: 30,
  async requestHandler({ page, request, log }) {
    const title = await page.title();
    log.info(`${title} — ${request.url}`);
  },
});

await crawler.run(["https://example.com"]);
```

That is the whole wiring. Everything hard is on either side of it.

## The trap — Crawlee is already doing this

**This is the most important paragraph in the chapter.**

Crawlee ships its own **browser fingerprint generator**, and it is **on by
default** in `PuppeteerCrawler` and `PlaywrightCrawler`. It replaced the old
`stealth` option. So with zero configuration, fingerprints are already being
generated and injected.

Stack the stealth plugin on top and **two tools overwrite the same values** —
`navigator.webdriver`, User-Agent, language, screen size. When they disagree you
get an inconsistent combination that stands out **more than using neither**.
That is the usual explanation behind "I added stealth and got blocked more."

Pick one.

```js
const crawler = new PuppeteerCrawler({
  // If you're going with the stealth plugin, turn Crawlee's fingerprints off.
  browserPoolOptions: { useFingerprints: false },
  launchContext: { launcher: puppeteerExtra },
  // ...
});
```

**Before that, check whether you need stealth at all.** Crawlee's default
fingerprints are often enough, which drops two dependencies and all of the
upkeep below. The order is: try it bare → only then add one of the two.

## Upkeep — this is not build-once

| What | How it moves |
|---|---|
| The stealth plugin | **Trails** puppeteer releases. Upgrade and combinations break |
| Detection | Keeps changing. What works today fails next month |
| Their markup | Changes without notice |

So **never build a pipeline that assumes success.** Silently emitting an empty
result is the worst outcome — every later step keeps running on the blank.

The principle from [Building pipelines with schedules](/guide/ht-pipelines)
applies unchanged:

```
[Done when]
- If the item count is 0, do not write the result file — report failure
- If it is under half the usual count, keep it but warn
```

## Running it in HyperTeams

Collection has no judgement in it, so it belongs in a **`terminal`** job, not a
`task`. A fixed command writes a fixed format and stops.

```
Daily 05:30  ·  kind: terminal
node collect.mjs
```

- Keep the script and its output inside the
  [working directory](/guide/ht-working-directory). This job launches a browser
  and writes files, so a narrow scope is worth having.
- Leave interpretation to the next `task`. Mixing collection and judgement into
  one step means you cannot tell which half failed.
- Headless browsers are memory-hungry. Start with a low `maxConcurrency`.

## Summary

1. API → static HTML → browser. Check **in that order**; Crawlee last.
2. "Are we allowed" comes before "is it possible." Stealth answers neither.
3. The wiring is `launchContext.launcher`.
4. **Do not stack** Crawlee's fingerprints and stealth. Stacked, you stand out more.
5. Build for breakage, and always report a zero-item run.