The ultimate guide to where to search for online published news articles. open-sourced & free tools only. Use it to build a News API alternative.
An ultimate guide to where to search for online published news articles. Open-sourced & free tools only.
Build a News API alternative with free and open-source tools only.
Why News API Alternative?
Data scientists and NLP enthusiasts love working with news data because there are many real-world use cases:
- topic clustering
- named entity recognition
- trend detection
- sentiment analysis
- trading on news
While building NewsCatcher News API, we discovered many open-sourced & free tools, services, libraries that help you find & parse online-published news articles.
We even published a Python package that helps work with news data: pygooglenews, your own Google News API alternative in Python.
While there are a few paid options (including what NewsCatcher does), some non-commercial use cases might be satisfied with open-sourced & free options. So you don't have to pay to News API providers.
Who is this list for?
- Students/portfolio builders Are you searching for a data science/data engineering job? You'd need to prove you can deliver some results. Here's one example of a data engineering project from Damian Kliś. Off-topic comment: one thing that makes Damian's GitHub repository stand out is that he made a clear and concise README. If you add any GitHub repository to your CV, you'd better explain well what is inside. No-README repo might harm you more than help you find a job: it shows you don't care about documenting & illustrating your work. It's a red sign for people who'd consider hiring you.
- Side-project This list might inspire you for your next side-project here. For example, you may build another news aggregator.
- Indie hackers Don't have money to pay news data providers? Try building your tool.
1. NewsMCP
NewsMCP is the free news API we built for AI agents. Instead of a list of articles, it returns stories: every outlet covering the same event is grouped into one result, with a count of the newsrooms that reported it.
Each story comes with a headline, a short summary and links to its sources, but not the full article text. It covers roughly the past week of news, or two weeks with a free key. You can connect it to an agent as an MCP server or call it as a REST API, and it works without an account.
- Documentation
- Playground, to run a search in your browser
Pros:
- free, and commercial use is allowed on every plan
- one result per story, with how many newsrooms covered it
- works without an account or a key
- connects to AI agents as an MCP server, or to your code as a REST API
Cons:
- headlines, summaries and source links only, no article text
- looks back days, not years
- calls without a key are blocked from cloud servers, so a hosted app needs the free key
2. GDELT 2.0 Global Knowledge Graph
GDELT analyses news articles published online. They apply Natural Language Processing to understand what news is being written worldwide. In addition, the GKG dataset allows you to find the links to newly published news articles.
Pros:
- ~400,000 news articles/day
- updates every 15 minutes
- worldwide multi-language coverage
- a news archive back to February 2015
Cons:
- just URL to the article: you have to scrape & extract it yourself
- not consistent in terms of delay and coverage
You might think that a list of URLs isn't much, but I bet you might be wrong. It's a half job done. For example, you could use the newspaper4k Python package, the maintained successor to newspaper3k, for parsing news by its URL or HTML.
3. News Crawl by Common Crawl
Common Crawl crawls the web and open-source all of the online pages they could have found. They are non-profit, so I highly encourage you to donate to them if you'll end up using their solution.
In 2016, Common Crawl decided to decouple the news crawl part from their primary dataset. News Crawl uses RSS & news sitemaps to parse the news. This part of Crawl is separately open-sourced.
Pros:
- ~600,000 news articles/day
- worldwide multi-language coverage
- few years of history
- full HTML of a page
Cons:
- updates multiple times a day
- you still need to parse the content from the HTML
4. RSS Feeds
RSS feeds still exist. Our beta version used to rely solely on RSS feeds. You can read the full story in How we built a news API beta in 60 days.
In Python, feedparser reads any RSS or Atom feed.
Pros:
- partially structured & contains some data points (title, published date)
Cons:
- you have to find the RSS feed – it's not a trivial task when you need it at scale
- news provider can turn off RSS feed at any time
5. Google News (RSS) - Google News API Alternative
Google News is the biggest UI-first news aggregator.
Google News has an RSS for any UI page. This RSS is lightweight, and you will not get blocked for accessing it many times a day.
We wrote a Python library that helps you parse any Google News RSS page. Even if you are not a Python person, you can use this repository as an unofficial Google News RSS documentation (there is no official one).
Conclusion
This list is a good starting point if you'd like to experiment with news data.
Building your News API alternative may teach you more about the subject itself.
Plus, it's a great data engineering exercise.
.png)
.png)
