Scrapy Pipeline Manager

Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.

Installation

Method 1, Agent Skill Exchange

• Install from the marketplace listing: https://agentskillexchange.com/skills/scrapy-pipeline-manager/

Method 2, Git clone

git clone https://github.com/agentskillexchange/skills.git && cd skills/skills/scrapy-pipeline-manager

Method 3, Download ZIP

• Download the repository ZIP and extract skills/scrapy-pipeline-manager.

Method 4, Manual copy

• Copy this skill folder into your local skills directory, then reload your agent tooling.

Method 5, Fork and sync

• Fork the repository if you want to maintain local edits while syncing upstream changes.

Source

• Agent Skill Exchange