Batch scrape multiple URLs

Trigger

Scraping multiple web pages at once — comparing products, collecting articles, or gathering data from several sources.

Workflow

1. Verify anakin-cli is authenticated by running anakin status.
2. Create the output directory if it doesn't exist:

   mkdir -p .anakin
   

3. Run the batch scrape:

   anakin scrape-batch "$ARGUMENTS" -o .anakin/batch-results.json
   

4. If the user provides more than 10 URLs, split into batches of 10:

   anakin scrape-batch "<url1>" ... "<url10>" -o .anakin/batch-1.json
   anakin scrape-batch "<url11>" ... "<url20>" -o .anakin/batch-2.json
   

5. Read the output and present a summary of all scraped pages.

Commands

# Batch scrape multiple URLs
anakin scrape-batch "<url1>" "<url2>" "<url3>" -o .anakin/batch-results.json

# With headless browser for JS-heavy sites
anakin scrape-batch "<url1>" "<url2>" --browser -o .anakin/batch-spa.json

# Geo-targeted batch
anakin scrape-batch "<url1>" "<url2>" --country de -o .anakin/batch-de.json

# Extended timeout for many pages
anakin scrape-batch "<url1>" "<url2>" "<url3>" --timeout 300 -o .anakin/batch.json

Options

• --browser — Use headless browser (for JS-rendered pages)
• --country <code> — Country code for geo-targeting (default: us)
• --timeout <seconds> — Max wait time (default: 180)
• -o, --output <path> — Save to file

Limit

Max 10 URLs per request. For more URLs, split into multiple batch calls.

Guardrails

• Maximum 10 URLs per scrape-batch command — split larger lists into batches.
• Always quote every URL to prevent shell interpretation.
• If individual URLs fail within a batch, report which ones failed and suggest retrying with anakin scrape using --browser.
• Always use -o to save output to a file rather than flooding the terminal.

Output

• Combined results from all URLs in a single JSON file
• Summary listing each URL with its scrape status