Last week we introduced Halcyon Search. It’s free, try it now: https://app.halcyon.io/search

My colleague Sam wrote about why we built it, features and capabilities, and common use cases. In the interest of building in public (and, well, because we think it’s cool and we hope you do too), today I will share a bit about how we built Halcyon Search. 

Precision vs Recall

Most people are familiar with the concept of search on the internet, originally via Google and more recently via whatever LLM-du-jour you might fancy. Fundamentally, search is about the trade-off between two things: precision (showing the right thing) and recall (not missing things). Most people have an idea of a very precise thing: “Show me stuff related to ‘vegetation management’.” This is what folks think of as “keyword” search.

But what happens when you misspell ‘vegetaation’? Or, what happens when the source material that you wanted to see misspells ‘managesment’? 

This reveals one of the fundamental challenges inherent in search: to be good at returning what people are actually looking for, search providers need to know a lot about what’s being asked. You need to know that someone who types ‘vegetaation’ actually means ‘vegetation’; you need to adjust for plurals. Natural language processing (NLP) models have largely solved this problem; when you misspell a Google search, there’s a lot happening under the hood to figure out your actual intent. 

Semantic Commodification

As search evolved, more advanced methods of improving relevance became possible, such as transformer models (one of the technologies employed by large language models). Improved relevance has ushered in the current familiar era of “vibe searching” as the default search experience on the web. Instead of trying to figure out exactly what someone means by ‘vegetaation’, search purveyors use a model to infer a more general representation, and then match documents in their catalogs based on similarity to that — which might surface results like “tree trimming,” for example. This conceptual association, rather than literal (mis)interpretation, is the main advantage of “semantic” search relative to keyword search: it’s very helpful to see tree trimming-related results when you’re interested in vegetation management!

The rise of AI and the subsequent open sourcing of these transformer models means that they have become better, faster, and cheaper than ever before. The result has been the effective commodification of semantic search, starting with text embedding (the abstract semantic representations that computers can understand) generation. New algorithms that better match embeddings to content are similarly commoditized. This opens the door to a wide range of new use cases: Similar Products recommendations on commerce sites; source content for complex Retrieval Augmented Generation (RAG) applications; or to implement ‘memory’ for the next generation of AI agents.

Fuzzy Constraints

While semantic search excels at understanding intent, it can also be frustratingly unpredictable. 

Consider the search phrase “data centers in Wisconsin.” On its own, semantic search might return results about renewable energy projects in Minnesota because the model has learned these concepts are related. Sometimes that’s great, other times, it’s completely unhelpful. This is a core challenge with semantic search in isolation: users have little control over how “fuzzy” their results get. 

This is where the distinction between search constraints and search terms becomes critical — and why it fundamentally changes what we consider “relevant” in the precision/recall trade-off. There is a massive difference between explicitly constraining the universe of potential results first to Wisconsin and then filtering to the term “data centers” versus searching the entire universe for the phrase “data centers in Wisconsin.” People rarely realize they are making this choice, hence the need for search providers to do both.

This is why we’ve built a search interface that isn’t just a blank text box, but rather one that offers configurable controls and filters. Configuration is a way of de-fuzzing results, but it is also a way to ensure that you get the most value out of the information we have in our catalog.

As our catalog (or “corpus”) has grown, the automation underlying our search capabilities has evolved in turn. We’ve added the ability to rapidly re-index interesting information — like when a new document is filed to a docket, we can reflect that change almost instantly rather than having to re-index the entire world of information we’ve already collected. We can now fully re-index all our document metadata in a matter of minutes, and re-index our entire catalog of searchable content in a couple hours.

This speed creates some counter-intuitions too. For instance, you might be able to use docket search to find a docket that was literally just added to a new PUC website, but you might not be able to find information accompanying that docket. That’s not because we don’t have it, but because the PUC website doesn’t have it yet. While that can be annoying, it’s fundamental to being good at monitoring and notifications — the core capabilities underpinning Halcyon Alerts. 

We’re making Halcyon Search better every day — while it is still early days and there are many more improvements forthcoming, today you can use it to find a needle in a haystack, narrow the universe, and build market intelligence. It’s also free. Try it today and let us know what you think sayhi@halcyon.io

Subscribe for more content like this; reach out with questions: sayhi@halcyon.io; follow us on LinkedIn and Twitter