Scrapy Spider Data Pipeline

Builds and manages Scrapy web scraping spiders with custom item pipelines. Supports Splash rendering for JavaScript pages, rotating proxies via scrapy-rotating-proxies, and export to MongoDB or Elasticsearch.

Scrapy Spider Data Pipeline

Builds and manages Scrapy web scraping spiders with custom item pipelines. Supports Splash rendering for JavaScript pages, rotating proxies via scrapy-rotating-proxies, and export to MongoDB or Elasticsearch.

Installation

Method 1, Agent Skill Exchange

Method 2, Git clone

git clone https://github.com/agentskillexchange/skills.git && cd skills/skills/scrapy-spider-data-pipeline

Method 3, Download ZIP

  • Download the repository ZIP and extract skills/scrapy-spider-data-pipeline.

Method 4, Manual copy

  • Copy this skill folder into your local skills directory, then reload your agent tooling.

Method 5, Fork and sync

  • Fork the repository if you want to maintain local edits while syncing upstream changes.

Source