Keyworx

by Claude

Keyworx: An Intelligent News Aggregator Built for the Local AI Era

In an age where information overload is the default state of the internet, the challenge is no longer finding news — it's making sense of it. Hundreds of RSS feeds, thousands of articles per day, the same story told a dozen times by a dozen outlets. Most news aggregators solve this by curation (human editors pick what matters) or by algorithmic ranking (engagement signals float the loudest content to the top). Keyworx takes a different approach: it uses machine learning to cluster related stories into unified articles, runs everything on local hardware, and presents the result through an interactive canvas that treats news as geography rather than a list.

The Problem With Lists

The traditional RSS reader is essentially a sorted inbox. Items arrive chronologically, stack up, and the user scrolls or marks-as-read until the anxiety of unread counts drives them to give up. Even aggregators that attempt deduplication typically do it with simple title hashing or URL matching — they catch the obvious reposts but miss the subtler case where Reuters, BBC, and AP all cover the same developing story with entirely different headlines and URLs.

The deeper problem is that a list of articles is a flat representation of a structured thing. News is not flat. Events have geography, they have temporal arcs, they have clusters of related reporting. Displaying it as a reverse-chronological feed throws all of that structure away, leaving the reader to reassemble it manually.

Keyworx was built to preserve that structure. Rather than presenting individual RSS items, it surfaces articles — synthetic documents that represent a story as covered by multiple sources — and organizes those articles across time windows and an interactive map. The pipeline that produces those articles is where the interesting engineering lives.

The Pipeline

At the heart of Keyworx is a five-step worker pipeline that runs on a five-minute tick loop. Each step is independent and stateful; the pipeline is designed to be restartable and to handle partial failures gracefully.

Step 1: RSS Fetch. The pipeline begins by pulling configured RSS sources. Rather than polling every source on every tick, each source carries its own VisitIntervalMinutes — a per-source rate determined by how frequently that outlet actually publishes. A wire service publishing 200 items a day gets polled every 30 minutes; a niche blog publishing twice a week gets polled daily. This rate is not configured manually. A separate SourceRateStep runs weekly, counts each source's actual output over a 14-day lookback, and sets the interval automatically. The result is a self-tuning polling schedule that reduces unnecessary network traffic without sacrificing freshness.

Step 2: Source Audit. Also on a weekly cadence, SourceAuditStep verifies that each source is still appropriate to aggregate. It checks the source's robots.txt for disallow rules, inspects the HTML head for <link rel="license"> declarations, and reads the feed's own <copyright> field. Sources that disallow crawling, or that carry Creative Commons licenses with non-commercial or no-derivatives restrictions, are automatically inactivated with a recorded reason. This is a legal hygiene step that most aggregators skip entirely.

Step 3: Embedding. Raw RSS items, once fetched, are passed through an embedding model to produce dense vector representations. Keyworx uses nomic-embed-text-v2-moe, a mixture-of-experts embedding model served locally via Ollama. The embedding captures semantic meaning — two sentences about the same news event will produce similar vectors even if they share no words. This is the foundation on which clustering is built.

The embedding step is parallelized across a pool of Ollama servers distributed across the local network, dispatched through a channel-based concurrent worker system. This allows the pipeline to saturate multiple machines simultaneously rather than waiting on a single GPU.

Step 4: Clustering. Once items are embedded, the clustering step groups them into articles. This is done by a Python sidecar process — a small FastAPI server running scikit-learn's DBSCAN algorithm — rather than in .NET directly. The choice of Python for this step is pragmatic: the scientific Python ecosystem (numpy, scikit-learn) has mature, well-optimized implementations of density-based clustering that would be painful to replicate in C#.

DBSCAN was chosen over k-means or hierarchical clustering because it does not require specifying the number of clusters in advance. News events are unpredictable; on a slow day there might be three major stories; on a breaking-news day there might be fifty. DBSCAN finds however many clusters the data supports, a — items that don't fit any cluster are discarded as singletons rather than forced into a groupwhere they don't belong.

The metric is cosine distance, not Euclidean. This is a subtle but important detail. Embedding vectors have high dimensionality, and in high-dimensional space, Euclidean distance becomes unreliable — all points tend toroughly equidistant from each other. Cosine distance measures the angle between vectors rather akes it the correct metric for comparing normalized embeddings. The current ClusterEps value of0.30 in cosine space means that two items must share approximately 70% cosine similarity to be grouped together — a tight threshold that keeps clusters clean without missing obviously related items.

Between full re-cluster runs, an incremental path handles items that arrive mid-session. New items are compared against existing article centroids using a similarity threshold; if close enough, they're assigned to the existing article rather than triggering a full DBSCAN pass. This keeps the pipeline responsive ompute.

Step 5: Enrichment. The clustering step produces groups of raw RSS items. The enrichment step tshed article. An LLM — currently qwen2.5:7b served via Ollama — reads the headlines and strippeddescriptions from all items in a cluster and produces a synthesized title, a two-to-three sentence journalistic summary, a URL-safe slug, a keyword list, and a geographical extraction (city, country, latitude/longitude of

the most prominent location in the story).

The enrichment step is also parallelized across multiple Ollama servers, with a configurable con-parameter model was chosen after evaluation of smaller options: a 3-billion-parameter modelproduced correct but uninspired output — single-sentence summaries that restated the headline — while the 7B model reliably produced the kind of brief, informative summary a journalist would write. The trade-off in

inference time is acceptable given the multi-server distribution.

The Session Model

Articles in Keyworx are not organized chronologically in a flat list. They're organized into se by configurable schedule rows in the database. A session might be a day, a week, a month. Theclustering step creates a session for each period that has RSS activity, then runs DBSCAN within that session's items. This means that clustering is bounded by time as well as semantic similarity: a story from last

Tuesday won't accidentally merge with a similar but unrelated story from this Tuesday just beca

Sessions become the primary navigation unit in the web app. Rather than scrolling through a fees, and within a session they move between articles on a canvas.

The Web Application

The front end is an ASP.NET MVC application with Google OAuth for authentication and a canvas-b. The canvas approach treats a session's articles as objects in a space rather than items in alist, which maps well to the geographical nature of much news content.

Geographical enrichment data — the extracted city, country, and coordinates from each article — powers a map layer. Articles with location data appear as pins on the map; selecting a region of the map filters the canvas

to articles from that geography. For users trying to follow events in a specific part of the wore useful than a keyword search.

Client-side code is written in TypeScript, compiled to JavaScript via esbuild. The build pipelielopment workflow: npm run build from the web project directory produces development output; npm

Client-side code is written in TypeScript, compiled to JavaScript via esbuild. The build pipelielopment workflow: npm run build from the web project directory produces development output; npmrun build:min produces production-ready bundles. Razor views use inline scripts only to bridge server-side data to the client — actual logic lives in compiled TypeScript modules.

Local-First Architecture

What distinguishes Keyworx from cloud-based news aggregators is that every component runs on local hardware. Ollama serves the embedding model and LLM from machines on the local network. The database is a local SQL Server instance managed by Entity Framework Core with code-first migrations. The Python sidecar is a l is sent to an external API; there are no per-request costs, no rate limits imposed bythird-party services, no dependency on external availability.

This architecture has real costs. Running multiple Ollama servers across a LAN requires hardware. The pipeline is more complex to operate than one that simply calls a hosted API. Tuning embedding and clustering paramerequires understanding the underlying mathematics.

But it also has real benefits. Latency is bounded by local hardware rather than external servics unconstrained — any model that Ollama can serve can be swapped in without changing code.Privacy is absolute: no article text or embedding vector ever leaves the local network. And the per-article enrichment cost, measured in electricity rather than API tokens, is effectively zero at scale.

Operational Tooling

The pipeline includes a set of backfill commands for operational scenarios — recovering from bugs, re-processing after parameter changes, or bootstrapping new installations. Re-stripping HTML from stored descriptions,regenerating geographical extractions, generating slugs for articles that predate the slug feated-cluster-enrich pipeline over the last month's data can all be triggered with command-lineflags. These aren't afterthoughts; they reflect the reality that a pipeline processing live data will occasionally need to be rewound and rerun without destroying what's already been produced.

What It Represents

Keyworx is a practical demonstration of what becomes possible when capable language models and embedding models can be run locally on commodity hardware. The same pipeline that would have required cloud API calls and meaningful per-request costs two years ago now runs on machines that fit under a desk. DBSCAN odings, with a capable LLM doing enrichment, produces article clusters that are genuinely useful — not perfect, but consistently better than what a naive deduplication approach would yield.

The result is a news reader that does less of what news readers usually do (showing you everything) and more of what they rarely do (helping you understand what's happening). For someone trying to follow the world without drowning in it, that's the right trade.