social-data

Search and fetch social media data from Twitter/X, Reddit, and Hacker News.

Social Data

Access social media content from multiple platforms. Some require API keys, some are free.

Hacker News (free, no key)

Top Stories

curl -s "https://hacker-news.firebaseio.com/v0/topstories.json" | head -c 200

Returns array of item IDs. Fetch individual items:

curl -s "https://hacker-news.firebaseio.com/v0/item/12345678.json"

Search (via Algolia, free)

curl -s "https://hn.algolia.com/api/v1/search?query=rust+programming&tags=story&hitsPerPage=10"
  • tags: story, comment, ask_hn, show_hn, poll
  • numericFilters: created_at_i>1700000000 (Unix timestamp)
  • Response: hits[] array with title, url, author, points, num_comments

Front Page

curl -s "https://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=30"

Reddit (free, no key for public data)

Subreddit Posts

curl -s "https://www.reddit.com/r/programming/hot.json?limit=10" \
  -H "User-Agent: bitterbot/1.0"

Sort options: hot, new, top, rising For top: add ?t=hour|day|week|month|year|all

Search Reddit

curl -s "https://www.reddit.com/search.json?q=rust+async&sort=relevance&limit=10" \
  -H "User-Agent: bitterbot/1.0"

Post Comments

curl -s "https://www.reddit.com/r/programming/comments/POST_ID.json" \
  -H "User-Agent: bitterbot/1.0"

Response is array of two listings: [0] = post, [1] = comments tree.

User Profile (public)

curl -s "https://www.reddit.com/user/USERNAME/submitted.json?limit=10" \
  -H "User-Agent: bitterbot/1.0"

Twitter/X (requires API key)

Requires TWITTER_BEARER_TOKEN env var (from X Developer Portal).

Search Recent Tweets

curl -s "https://api.twitter.com/2/tweets/search/recent?query=from:elonmusk&max_results=10&tweet.fields=created_at,public_metrics" \
  -H "Authorization: Bearer $TWITTER_BEARER_TOKEN"

Query operators:

  • from:username — tweets by user
  • to:username — replies to user
  • #hashtag — hashtag search
  • "exact phrase" — exact match
  • -is:retweet — exclude retweets
  • lang:en — language filter
  • has:media — tweets with media

User Lookup

curl -s "https://api.twitter.com/2/users/by/username/elonmusk?user.fields=public_metrics,description,created_at" \
  -H "Authorization: Bearer $TWITTER_BEARER_TOKEN"

User Timeline

curl -s "https://api.twitter.com/2/users/USER_ID/tweets?max_results=10&tweet.fields=created_at,public_metrics" \
  -H "Authorization: Bearer $TWITTER_BEARER_TOKEN"

Tips

  • Always include User-Agent header for Reddit (they block default agents).
  • Reddit JSON endpoints append .json to any Reddit URL.
  • HN Algolia API is the most reliable for search; Firebase API for real-time data.
  • Twitter free tier allows 500K tweets/month read access.
  • For rate-limited APIs, cache results and avoid repeated identical queries.
  • Use web_fetch for HTML pages, exec + curl for JSON APIs.