Common Crawl Index Query Agent

Queries the Common Crawl Index API for large-scale web archive research and data extraction. Uses the CDX Server API, WARC record parsing with warcio, and the Common Crawl S3 bucket for bulk data access.

Common Crawl Index Query Agent

Queries the Common Crawl Index API for large-scale web archive research and data extraction. Uses the CDX Server API, WARC record parsing with warcio, and the Common Crawl S3 bucket for bulk data access.

Installation

Method 1, Agent Skill Exchange

Method 2, Git clone

git clone https://github.com/agentskillexchange/skills.git && cd skills/skills/common-crawl-index-query-agent

Method 3, Download ZIP

  • Download the repository ZIP and extract skills/common-crawl-index-query-agent.

Method 4, Manual copy

  • Copy this skill folder into your local skills directory, then reload your agent tooling.

Method 5, Fork and sync

  • Fork the repository if you want to maintain local edits while syncing upstream changes.

Source