Use case · 7 recipes

News and content

Headlines, front pages, newsletters and feeds. 7 recipes, one API call each.

Every recipe for news and content

google-newsverifiedHeadlines for a query from Google News: title, the publishing source, Google's link to the article, time and a short summary.news.google.com · 2 units · 5 fields hackernews-frontverifiedThe stories on the Hacker News front page, newest first: title, link, points, author, comment count.news.ycombinator.com · 2-4 units · 8 fields hackernews-searchverifiedStories or comments on Hacker News that match a query, from the public Algolia API: title, link, points, author, comment count, time, and the Hacker News page.hn.algolia.com · 2 units · 9 fields producthunt-todayverifiedThe launches on Product Hunt's home page today, in the site's own rank order: name, tagline, the score on the vote button, the first topic and the product link.producthunt.com · 2 units · 5 fields rss-feedverifiedThe items of any RSS 2.0 or Atom feed: title, link, publication time, author, the first 300 characters of the summary, and the item's id.Any RSS feed · 2 units · 6 fields substack-postsverifiedThe latest posts of a Substack publication, from its public archive: title, subtitle, link, date, author, likes, comment count, whether it is behind the…Substack · 2 units · 9 fields medium-postsstagedThe latest posts of a Medium tag, author or publication, from Medium's public RSS feed: title, link (the ?source= tracking query removed), author, publication…medium.com · 2 units · 6 fields

google-newsGoogle News

verified

Headlines for a query from Google News: title, the publishing source, Google's link to the article, time and a short summary.

Market research5 fields

2 units · fresh 5 minTry it →

hackernews-frontHacker News

verified

The stories on the Hacker News front page, newest first: title, link, points, author, comment count.

Startup and investor intel8 fields

2-4 units · fresh 10 minTry it →

hackernews-searchHacker News

verified

Stories or comments on Hacker News that match a query, from the public Algolia API: title, link, points, author, comment count, time, and…

News and content9 fields

2 units · fresh 5 minTry it →

producthunt-todayProduct Hunt

verified

The launches on Product Hunt's home page today, in the site's own rank order: name, tagline, the score on the vote button, the first topic…

Startup and investor intel5 fields

2 units · fresh 1 hTry it →

rss-feedAny RSS feed

verified

The items of any RSS 2.0 or Atom feed: title, link, publication time, author, the first 300 characters of the summary, and the item's id.

News and content6 fields

2 units · fresh 10 minTry it →

substack-postsSubstack

verified

The latest posts of a Substack publication, from its public archive: title, subtitle, link, date, author, likes, comment count, whether it…

News and content9 fields

2 units · fresh 10 minTry it →

medium-postsMedium

staged

The latest posts of a Medium tag, author or publication, from Medium's public RSS feed: title, link (the ?source= tracking query removed)…

News and content6 fields

2 units · fresh 10 minTry it →

Other use cases

Lead generation · 8 Competitor intelligence · 8 Market research · 9 Hiring and talent · 7 E-commerce and catalogs · 12 Social listening · 24 Startup and investor intel · 9 Travel and real estate · 5 Local and maps · 6 Video and creators · 8 Research and knowledge · 5 Developers and open source · 7

For developers

Get a free API key and run your first crawl.

1,000 credits every month. No credit card. Your self-hosted setup keeps working.

Get a free API key

For teams

Talk to us about volume, regions and an SLA.

A 30-minute call with the team that writes the crawler. DPA on request, invoice billing, your choice of US, EU or Singapore.

Book a call hello@crawl4ai.com