The tech driving our API

Our process begins with a proprietary scheduling algorithm that monitors the publication frequencies of different sources over a week. This data informs our crawlers, allowing us to efficiently gather new article links without overwhelming system resources. This method ensures an optimal balance between timeliness and resource utilization.

We fetch and store the raw webpage for each article link. This archival strategy provides the flexibility to enhance data extraction methods retrospectively as new techniques become available, ensuring continuous improvement in data quality.

We utilize five distinct extraction methods to retrieve article data, including two advanced adaptations of open-source technologies and three proprietary techniques developed in-house. This diverse toolkit enables us to handle a wide range of article formats and data types effectively.

Our process begins with a proprietary scheduling algorithm that monitors the publication frequencies of different sources over a week. This data informs our crawlers, allowing us to efficiently gather new article links without overwhelming system resources. This method ensures an optimal balance between timeliness and resource utilization.

The next phase involves a comprehensive data cleaning process. We use a detailed directory of patterns to identify and remove irrelevant information. This meticulous approach significantly enhances the quality of the information.

Cleaned articles are processed through an advanced Natural Language Processing (NLP) pipeline. This stage includes summarizing the content, classifying articles into broad news topics, detecting named entities, and assessing sentiment. This enriches the articles, making them more actionable and insightful for users.

Processed articles are indexed in our main production ES clusters for querying. We also distribute specific datasets to dedicated client clusters and shared cloud storage to ensure high availability and performance.

Our system dynamically filters and groups articles based on user queries, employing sophisticated algorithms to cluster similar articles and deliver highly relevant results swiftly and efficiently.