How to retrieve more than 10,000 articles

The Newscatcher API limits results to 10,000 articles per search query. The Python SDK provides special methods that automatically split your search across multiple time periods to bypass the limit and retrieve all articles relevant to your query.

​

Understanding the article limit

When your query matches more than 10,000 articles, the API returns "total_hits": 10000 as a hard limit, and you cannot retrieve more through standard pagination.

from newscatcher import NewscatcherApi

client = NewscatcherApi(api_key="YOUR_API_KEY")

response = client.search.post(
    q="technology",
    from_="7d",
    to="now"
)

print(f"Total hits: {response.total_hits}")
print(f"Is result capped: {response.total_hits == 10000}")  # True if limit reached

​

Using time-chunking methods

The SDK provides two special methods to retrieve large volumes of articles:

• get_all_articles
• get_all_headlines

Both methods available for synchronous and asynchronous clients.

​

Get all articles

from newscatcher import NewscatcherApi

client = NewscatcherApi(api_key="YOUR_API_KEY")

articles = client.get_all_articles(
    q="renewable energy",
    from_="30d",
    to="now",
    time_chunk_size="1d",
    max_articles=50000,
    show_progress=True,
)

print(f"Retrieved {len(articles)} articles")

​

Get all headlines

headlines = client.get_all_headlines(
    when="30d",
    time_chunk_size="1d",
    max_articles=20000,
    show_progress=True
)

print(f"Retrieved {len(headlines)} headlines")

​

How time-chunking works

Time-chunking divides your date range into smaller intervals, making separate API calls for each period and combining the results. Each interval can return up to 10,000 articles.

For example, with time_chunk_size="1d" over 5 days, the method makes 5 API calls, one for each day, with auto pagination to potentially retrieve up to 50,000 articles.

​

Choosing the right chunk size

The optimal chunk size depends on how many articles your query returns:

response = client.search.post(
    q="renewable energy",
    from_="1d",
    to="now"
)

print(f"Articles for one day: {response.total_hits}")

if response.total_hits == 10000:
    response_6h = client.search.post(
        q="renewable energy",
        from_="6h",
        to="now"
    )
    print(f"Articles for 6 hours: {response_6h.total_hits}")

​

Method parameters

​

Common issues and solutions

​

See also

• Advanced query syntax
• API rate limits
• Python SDK