Collecting with a browser — Crawlee + stealth

What this chapter covers How to attach puppeteer-extra-plugin-stealth to Crawlee's PuppeteerCrawler — plus what to check before you attach it, and what you will certainly run into afterwards.

First — do you actually need a browser?

The collection step in Case study — the content factory is a single terminal command. Most collection ends there. Work down this list.

A browser is the most expensive option. One Chrome tab costs hundreds of megabytes, adds seconds per page, and above all breaks the most often — a small markup change on their side stops you.

graph TD
  A["You need data"] --> B{"Is there an API?"}
  B -->|Yes| C["API — done here"]
  B -->|No| D{"Already in the HTML?"}
  D -->|Yes| E["fetch + parser"]
  D -->|No| F["Browser — Crawlee"]

Then — are you allowed to?

This check comes before the technical one. Break these and the problem is not getting blocked, it is contracts and law. The stealth plugin solves none of them.

• robots.txt and terms of service. If automated collection is disallowed, it does not matter whether it is technically possible.
• Behind a login or paywall. That is account-terms territory. "Pulling my own data with my own account" and "getting around the barrier" are different things.
• Personal data. Public does not mean collectable. You need a basis.
• Load on their servers. Cap concurrency and pace yourself — that is what Crawlee's maxConcurrency and maxRequestsPerMinute are for.

The easy case is your own site, your own data, or somewhere that has explicitly agreed. Checking how your own service renders, or gathering your own product listings, sidesteps most of the above. For anything else, asking first is usually faster than engineering around it.

The wiring

Install:

The key is launchContext.launcher — this is where you tell Crawlee to launch the browser with puppeteer-extra instead of plain puppeteer.

That is the whole wiring. Everything hard is on either side of it.

The trap — Crawlee is already doing this

This is the most important paragraph in the chapter.

Crawlee ships its own browser fingerprint generator, and it is on by default in PuppeteerCrawler and PlaywrightCrawler. It replaced the old stealth option. So with zero configuration, fingerprints are already being generated and injected.

Stack the stealth plugin on top and two tools overwrite the same values — navigator.webdriver, User-Agent, language, screen size. When they disagree you get an inconsistent combination that stands out more than using neither. That is the usual explanation behind "I added stealth and got blocked more."

Pick one.

Before that, check whether you need stealth at all. Crawlee's default fingerprints are often enough, which drops two dependencies and all of the upkeep below. The order is: try it bare → only then add one of the two.

Upkeep — this is not build-once

So never build a pipeline that assumes success. Silently emitting an empty result is the worst outcome — every later step keeps running on the blank.

The principle from Building pipelines with schedules applies unchanged:

Running it in HyperTeams

Collection has no judgement in it, so it belongs in a terminal job, not a task. A fixed command writes a fixed format and stops.

• Keep the script and its output inside the working directory. This job launches a browser and writes files, so a narrow scope is worth having.
• Leave interpretation to the next task. Mixing collection and judgement into one step means you cannot tell which half failed.
• Headless browsers are memory-hungry. Start with a low maxConcurrency.

Summary

1. API → static HTML → browser. Check in that order; Crawlee last.
2. "Are we allowed" comes before "is it possible." Stealth answers neither.
3. The wiring is launchContext.launcher.
4. Do not stack Crawlee's fingerprints and stealth. Stacked, you stand out more.
5. Build for breakage, and always report a zero-item run.