Most RAG demos look great on day one. You load some documents, ask a question, and the model answers with citations. Everyone’s impressed.
Then three months pass. Nobody has touched the knowledge base since launch, and users start asking about things that happened last week. The model either says it doesn’t know, or it pulls up an old document and answers confidently with outdated information. The second case is worse, because it looks correct.
That’s the quiet failure mode of retrieval-augmented generation. The retrieval part is only as good as what you put in the index, and most indexes go stale. In this post I’ll walk through building a RAG pipeline on news data that stays current, handles questions about the past properly, and gives credit to the sources it uses.
Why News Is Ideal RAG Material
News has a few qualities that make it a natural fit for RAG.
First, it’s exactly the kind of information language models don’t have. A model’s training data stops at a cutoff date, and news is, by definition, what happened after that. So retrieval adds real value here rather than just repeating what the model already knows.
Second, news articles are well structured. They come with a title, a date, a source and usually a clear topic. That metadata is gold for retrieval, because it lets you filter by time and publisher instead of relying only on vector similarity.
Third, articles are a manageable size. Most fit into a handful of chunks, which keeps embedding costs low and retrieval precise.
The one tricky part is getting the data in the first place, reliably and continuously. That’s where a news data API comes in, and it’s what we’ll use below.
Architecture Overview
The pipeline has five stages. None of them are complicated on their own.
- Fetch. Pull articles from a news API, either on a schedule or on demand.
- Chunk. Split each article into smaller passages that fit well into an embedding model.
- Embed. Turn each chunk into a vector and store it, along with metadata like date and source.
- Retrieve. When a user asks a question, find the most relevant chunks, filtered by date where it makes sense.
- Generate. Pass those chunks to the LLM as context and ask it to answer, citing sources.
For the examples I’m using Python, AllNewsAPI for the data and Chroma as the vector store. Chroma is handy for tutorials because it runs locally and embeds text for you with a default model. You can swap in Pinecone, pgvector, Weaviate or anything else later without changing the overall design.
pip install requests chromadb
Pulling News Data via API
Here’s a simple fetch function. It searches for a topic within a date range and returns the article list.
import os
import requests
API_KEY = os.getenv(“ALLNEWSAPI_KEY”)
def fetch_news(query, start_date, end_date=None, page=1):
params = {
“apikey”: API_KEY,
“q”: query,
“startDate”: start_date,
“lang”: “en”,
“max”: 50,
“page”: page,
“content”: “true”,
}
if end_date:
params[“endDate”] = end_date
resp = requests.get(“https://api.allnewsapi.com/search”, params=params, timeout=30)
resp.raise_for_status()
return resp.json()
A quick note on content=true. Full article text is a paid feature. On the free plan you’ll get titles and descriptions, which is still enough to build and test the pipeline, just with thinner context. The max value also depends on your plan, so adjust it to match.
Using AI summaries and entities to enrich chunks
This is the part most tutorials skip, and it makes a noticeable difference to retrieval quality.
Each article can come back with an AI-generated summary and a list of named entities, meaning the people, organisations and places mentioned. You can use both. The summary gives you a short, dense passage that’s often the best single chunk for retrieval. The entities make great metadata for filtering, so you can restrict a search to articles that actually mention a company rather than hoping the embeddings figure it out.
Another small trick: add a header line to every chunk with the title, source and date. When the LLM reads a chunk out of context, it immediately knows where it came from and how old it is.
import hashlib
from datetime import datetime
def chunk_text(text, size=800, overlap=100):
chunks, start = [], 0
while start < len(text):
chunks.append(text[start:start + size])
start += size – overlap
return chunks
def prepare_article(a):
date = a[“publishedAt”][:10]
header = f'{a[“title”]} | {a[“source”][“name”]} | {date}\n’
body = a.get(“content”) or a.get(“description”) or “”
pieces = []
if a.get(“ai_summary”):
pieces.append(a[“ai_summary”])
pieces.extend(chunk_text(body))
entities = “, “.join(e[“name”] for e in a.get(“ai_entities”) or [])
ts = int(datetime.fromisoformat(a[“publishedAt”].replace(“Z”, “+00:00”)).timestamp())
base_id = hashlib.md5(a[“url”].encode()).hexdigest()
ids, docs, metas = [], [], []
for i, piece in enumerate(pieces):
ids.append(f”{base_id}-{i}”)
docs.append(header + piece)
metas.append({
“url”: a[“url”],
“source”: a[“source”][“name”],
“published_ts”: ts,
“entities”: entities,
})
return ids, docs, metas
Chroma only accepts simple values in metadata, which is why the entity list is joined into a string and the date is stored as a Unix timestamp. The timestamp matters for the next section.
Historical data for time-aware answers
Users don’t only ask about today. They ask “how did coverage of this change over the past year?” or “what was being said about this company before the merger?” A RAG system with only a week of data can’t answer those.
AllNewsAPI’s archive goes back to January 2016, with how far you can reach depending on your plan. The free plan covers the last 30 days, and paid plans extend that to anywhere from one to ten years. For a backfill, loop through pages until nextPage comes back empty:
import chromadb
client = chromadb.PersistentClient(path=”./news_index”)
collection = client.get_or_create_collection(“news”)
def backfill(query, start_date, end_date):
page = 1
while page:
data = fetch_news(query, start_date, end_date, page)
for article in data[“articles”]:
ids, docs, metas = prepare_article(article)
collection.upsert(ids=ids, documents=docs, metadatas=metas)
page = data.get(“nextPage”)
Because every chunk has a timestamp, retrieval can now respect time. A question about last month only searches last month:
def retrieve(question, since_ts=None, k=6):
where = {“published_ts”: {“$gte”: since_ts}} if since_ts else None
return collection.query(query_texts=[question], n_results=k, where=where)
This one change fixes a surprising number of wrong answers. Without it, a question about “the latest” earnings might retrieve a very similar article from two years ago, simply because the wording matches.
From here, generation is the usual pattern. Take the retrieved documents, put them in the prompt along with their URLs, and instruct the model to answer only from that context and cite the sources it used. Any LLM works for this step.
Keeping the Index Fresh with Scheduled Fetches
Backfilling is a one-time job. Staying fresh is the ongoing one, and it’s where most RAG projects quietly fall apart.
The simplest approach is a scheduled job, a cron entry or a cloud function, that runs every hour or so and fetches anything published since the last run. Since the API is a real-time news API, new articles show up in the index soon after publishers release them, so hourly updates keep you close to live.
# run hourly via cron
from datetime import datetime, timedelta, timezone
def refresh(topics):
since = (datetime.now(timezone.utc) – timedelta(hours=2)).strftime(“%Y-%m-%dT%H:%M:%SZ”)
for topic in topics:
data = fetch_news(topic, since)
for article in data[“articles”]:
ids, docs, metas = prepare_article(article)
collection.upsert(ids=ids, documents=docs, metadatas=metas)
Notice the two-hour window on an hourly job. The overlap means you don’t miss articles if a run fails or is delayed. Duplicates aren’t a problem, because the IDs are based on the article URL and upsert simply overwrites them.
Keep an eye on your request budget as you add topics. Each topic is at least one request per run. Twenty topics refreshed hourly adds up to 480 requests a day, which is well past a free plan but easily handled by a paid one.
For all the parameters, including date formats and pagination details, read the documentation.
RAG vs MCP: Which Approach Fits Your Use Case?
There’s another way to give an LLM current news that doesn’t involve building an index at all. AllNewsAPI offers a news API MCP server, powered by the same news API, which lets an AI assistant search news live, at the moment you ask a question.
So which should you use? It depends on what you’re building.
| RAG pipeline | MCP server | |
| Setup effort | Moderate, you build and maintain an index | Minimal, add one config block |
| Freshness | As fresh as your last scheduled fetch | Live at query time |
| Control over data | Full: your chunking, filters and ranking | Limited to the API’s search results |
| Deep historical analysis | Strong, once you’ve backfilled | Possible, but query by query |
| Best for | Products, research tools, custom search | Assistants, agents, quick exploration |
In practice, many teams use both. MCP is great for prototyping, because you can test what questions users ask and what sources matter before building anything. RAG takes over when you need consistent behaviour, custom ranking or analysis across years of data. And since both run on the same API key, moving from one to the other is straightforward.
Handling Copyright and Attribution Properly
This part isn’t optional, so it’s worth doing right from the start.
News articles belong to their publishers. Using them to answer questions internally is one thing. Republishing large parts of them to your users is another, and it can cause real legal problems. AllNewsAPI itself advises crediting original publishers and not presenting their content as your own.
A few habits keep you on the right side:
- Always show sources. Every answer should link back to the original articles it used. The metadata we stored makes this easy.
- Summarise, don’t reproduce. Instruct the model to answer in its own words and avoid quoting long passages.
- Keep excerpts short. If you show snippets in your interface, keep them to a sentence or two.
- Check your use case. If you’re building a commercial product that displays news content, get proper legal advice for your market.
Good attribution also builds trust. Users are much more likely to believe an answer when they can click through and see where it came from.
FAQs
What is RAG with news data?
It’s a setup where an LLM answers questions using news articles retrieved from an index, instead of relying only on its training data. The system finds relevant articles for each question and gives them to the model as context, so answers reflect recent events.
How do I keep a RAG system up to date?
Run a scheduled job that fetches new articles from a news API and adds them to your index, typically every hour or so. Use stable IDs based on article URLs so repeat fetches update existing entries instead of creating duplicates.
Should I use RAG or an MCP server for live news?
Use an MCP server if you want an AI assistant to look up news on demand with almost no setup. Use RAG if you need full control over the data, custom ranking or deep historical analysis. Many projects start with MCP and add RAG later.
Can I build a custom news dataset for machine learning?
Yes. You can pull articles by topic, date range, country and language through the API and store them in your own format. Paid plans also support CSV and Excel exports. Just make sure your use of the content respects publisher rights, especially if you share the dataset.
Final Thoughts
A RAG system doesn’t fail loudly when its data goes stale. It just gets a little more wrong every week. Building freshness into the pipeline from day one, with scheduled fetches, timestamps and time-aware retrieval, is what separates a demo from something people can actually rely on.
If you want to try the code above, grab a free API key, backfill a single topic and ask it a few questions. You’ll see the difference between a fresh index and a frozen one pretty quickly.