grobid-pdf-extraction

Extract structured metadata, text, and citations from academic PDFs using machine learning. GROBID converts unstructured PDFs into TEI XML format with recognized logical structure. Use when: (1) extracting metadata (title, authors, affiliations) from research papers, (2) identifying document sections and structure, (3) parsing reference lists and citations, (4) building indexes or knowledge bases from academic PDFs, or (5) batch processing directories of papers. Not suitable for: scanned image PDFs without OCR, non-academic documents with unusual layouts, binary content (tables/figures), or simple text extraction tasks.

Extract Structured Data from Academic PDFs

GROBID (GeneRation Of BIbliographic Data) is a machine learning service that parses academic PDFs and outputs semi-structured TEI XML. It performs document understanding—recognizing logical document structure (sections, citations, affiliations) even when PDF encoding is purely visual.

Set Up GROBID Service

Deploy GROBID using Docker with the official image: grobid/grobid:0.7.3

docker run -p 8070:8070 grobid/grobid:0.7.3

Critical: Always verify service readiness before sending PDFs:

curl http://localhost:8070/api/isalive  # Wait for HTTP 200

For detailed deployment configuration, timeouts, resource allocation, and performance tuning, see deployment.md.

Submit PDFs and Extract Results

Use the provided scripts to process PDFs end-to-end or implement API calls directly.

Quick Start

python quickstart.py input.pdf output_dir/

This processes a single PDF, extracts metadata/sections/references, and saves JSON + TEI XML.

Batch Processing

python batch_processor.py pdf_directory/ output_directory/

Processes all PDFs in a directory with error recovery and progress tracking.

API Usage

Send PDFs via POST /api/processFulltextDocument:

response = requests.post(
    'http://localhost:8070/api/processFulltextDocument',
    files={'input': open('paper.pdf', 'rb')},
    data={'consolidateHeader': '1', 'consolidateCitations': '1'},
    timeout=300  # CRITICAL: 300+ seconds for large PDFs
)
tei_xml = response.text

Timeout requirement: GROBID runs ML inference. Set timeout to 300+ seconds (5+ minutes); some large documents need even more.

Parse TEI XML Output

GROBID outputs TEI (Text Encoding Initiative) XML—a scholarly standard requiring namespace-aware parsing.

The provided script extract_metadata.py implements three functions:

  • extract_metadata(tei_xml) — Returns title, authors, abstract, keywords, DOI

    • ⭐ Key pattern: Author extraction is scoped to document header only (avoids bibliography pollution of 50-100+ cited authors)
    • Uses .itertext() for nested elements (abstracts contain nested <div> and <p> tags)

  • extract_sections(tei_xml) — Returns dictionary of {section_heading: text}

    • Iterates hierarchical body structure
    • Concatenates paragraphs within each section

  • extract_references(tei_xml) — Returns list of reference dictionaries

    • Extracts title, authors, year, venue, DOI for each citation

For detailed parsing patterns, critical gotchas, and namespace handling, see tei_xml_parsing.md.

Quick reference — use these patterns:

# Define namespace
ns = {'tei': 'http://www.tei-c.org/ns/1.0'}

# Authors: ALWAYS scope to header (not entire document)
header = root.find('.//tei:teiHeader', ns)
authors = header.findall('.//tei:author', ns)

# Nested text: Use .itertext(), NOT .text
abstract_text = ''.join(abstract_el.itertext()).strip()

Provided Scripts

  • grobid_client.py — Reusable HTTP client with health checks and configurable timeout
  • extract_metadata.py — TEI XML parsing (implements critical namespace scoping and nested text patterns)
  • quickstart.py — Single-PDF example with UTF-8 encoding setup for Windows
  • batch_processor.py — Batch directory processing with error recovery and progress tracking

How to Use This Skill

  1. Deploy GROBID — See "Set Up GROBID Service" section above
  2. Choose workflow — Single PDF (quickstart.py) or batch (batch_processor.py)
  3. Parse results — Use extract_metadata.py functions or implement custom parsing
  4. Handle errors — See troubleshooting.md for failure modes and recovery

Workflow Patterns

Extract Metadata Only

For metadata-only use cases, use the faster header-only endpoint:

client = GrobidClient()
tei_xml, status = client.process_pdf_header_only('paper.pdf')
metadata = extract_metadata(tei_xml)

Batch Processing Best Practices

  • Process sequentially (not parallel) — GROBID is CPU-intensive; concurrency exhausts CPU without speedup
  • Implement per-document error handling — wrap each PDF in try/except to prevent one bad document from stopping the batch
  • Validate sample output — spot-check 5-10% of results against original PDFs to catch systematic parsing errors
  • Keep container running for multiple batches (avoid restart overhead)

Additional Resources

Reference Documentation:

  • deployment.md — Docker setup, timeouts, resource allocation, performance tuning
  • tei_xml_parsing.md — XML namespace handling, parsing patterns, critical gotchas
  • troubleshooting.md — Error recovery, character encoding, batch resilience, document-type considerations

External Resources: