Skip to content
From Static to Current pt 2

[Editor’s note: From Static to Current pt 1 covers the specifics of which Data Subscriptions are migrating to Grids and the new benefits — what we’ve built and why it’s better. This post is a technical overview of how we built Grids.]

Understanding our transition from static Data Subscriptions to dynamic Grids requires a Halcyon history lesson. Eighteen months ago, we began developing our Gas Power Plant Tracker by dogfooding our internal APIs to our data scientists. Each cell in the excel files we delivered to customers was the result of an iterative research process: We developed targeted searches and queries, smashed together the outputs, and then meticulously checked the sources in order to develop new searches and queries.

This process was slow and ad-hoc, so monthly updates made sense; outputs weren’t predictable, so humans manually curated spreadsheets. These Data Subscriptions were a great demonstration of our platform’s capabilities, but they weren’t flexible, scalable, or customizable. Over the past year, we’ve developed solutions to these issues.

Abstract Syntax Trees (ASTs)

One reason that we couldn’t initially build data subscriptions in-app — and required an ad-hoc series of queries to construct — is that we couldn’t realistically express the universe of documents corresponding to a project (like a gas power plant) using a single search. We developed Abstract Syntax Trees to provide a structured format for describing a series of Halcyon searches and the relationships between them. They say a hokey engineering diagram is worth a thousand words, so here is the AST representing the Plato North Power Generation Project:

Blog - Nov 19 - Image 01

ASTs allow us to combine disjoint, hyper-specific searches: in this example, a series of dockets that directly concern the Plato North plant along with various combinations of publisher and keyword search constraints that might indicate relevant information about the project. ASTs also allow us to filter out confounding terms that might pollute search results: for instance, we’d like to exclude documents about the nearby (but distinct!) ‘Plato South’ project.

Implementing AST searches across our web app and APIs wasn’t trivial. Search sits at the core of most Halcyon products, so vast swathes of our codebase that previously dealt with simple search objects now had to deal with trees. If you’re up for a little brain-teaser (and a peek into the life of a Halcyon engineer), consider whether the following AST guarantees that all satisfying documents contain the keyword “Microsoft”:

Blog - Nov 19 - Image 02

If you answered yes, congratulations on applying De Morgan’s law: documents must contain “Microsoft”, but cannot contain “Microsoft” and “Amazon” together. These sorts of puzzles aren’t academic: we rebuilt our query pipeline to support interrogating every set of search constraints correctly in order to accurately rewrite queries and re-rank search results. We also built to accommodate far more complex scenarios involving multidimensional constraint nodes, and then addressed challenges of efficiently persisting large trees to a database.

Alerts

While ASTs enabled us to precisely target searches at arbitrary cross-sections of our catalog, Alerts allowed us to periodically run queries against those searches, enabling the always-on intelligence that Grids provide. Put more simply: each row in a grid is a search; each column is a query; and each cell is an alert, populating the cell with its latest output according to a defined schedule:

Blog - Nov 19 - Image 03

Customizable Columns

Because rows (searches), columns (queries), and cells (alerts) all represent primitives in our platform, we are excited to launch a feature that static excel files couldn’t support: custom user-generated columns. When you add a column, you’re defining a query; when you execute the column, you’re generating an alert for each row in the grid, powered by complex AST search constraints, humming along at your specified cadence, and automatically populating cells with the latest responses.

Asking a question of 500+ gas power plant projects manually would take weeks. Doing it one query at a time via Halcyon’s web app would take days. Adding a column to a grid and asynchronously executing all those queries takes hours or minutes.

Metadata Searches

The final piece of the puzzle powering the Data Subscriptions-to-Grids migration — technology that we’ll soon enable for customers — is a domain-specific language for performing complex aggregations of document metadata. We’re pretty excited about this, because it addresses longstanding limitations of Retrieval Augmented Generation (RAG) pipelines in responding to certain types of questions. Let’s compare the two approaches using a simple query: “What are the titles of all the documents?”

Blog - Nov 19 - Image 04

RAG’s limitations in responding to this query become apparent pretty quickly. First, we’re searching our corpus for text semantically similar to the “What are the titles of all the documents?”, which, for the purpose of answering the question, will surface more-or-less random chunks of text. Second, semantic search returns only the top-k results, so we’re limited to k titles at best. And third, the chunks of text usually won’t contain any titles — instead, we rely on augmenting document chunks with metadata before prompting an LLM to generate the answer. However, this augmentation is one-size-fits-all: we’re forced to augment each and every query’s chunks with a predetermined set of useful metadata in the hopes it’ll help answer a few questions here or there. In most cases, the metadata simply wastes tokens in the LLM’s context window.

Now, consider the metadata search: we can fetch all the documents matching the criteria, extract only the metadata requested by the question (in this case, titles), and then perform an aggregation. We can do all this on our search cluster in hundreds of milliseconds, and we don’t have to force the answer through a lossy LLM. Best of all, we can support complex aggregations such as “Who is the most common author per publisher?” or “What is the filing type of the most recently published document per docket?”

While we still believe RAG is the answer for most queries, metadata searches cover many of the remainder, so we’re hard at work designing a slick UI and API surface that will enable you to ask these sorts of questions.