Don’t let heuristics undermine your ML, learn to combine them
In today’s data-driven landscape, recommendation systems power everything from social media feeds to e-commerce. While it’s tempting to think that machine learning algorithms do all the heavy lifting, that’s only half the story. Real-world systems often rely on a mix of machine learning and heuristic rules — commonly referred to as product rules, business rules, or simply hacks — to generate the most relevant recommendations.
For example:
You can’t recommend tracks from the same artist too often;You should include content from subscriptions in the feed, but not overwhelm it;If a user has already disliked a certain category or author, the related content should be penalized or even filtered out;Explicit content can’t be recommended — except when appropriate.Photo by Cam Bradford on Unsplash
Rules come in two types: hard and soft. Hard rules act as filters, prohibiting certain documents from being recommended in specific contexts; failing to comply is considered a product bug. There’s nothing inherently wrong with these rules, but their number should be limited. Moreover, they should be applied as early as possible in the ranking process, either at the candidate generation stage or even during index construction. Soft rules, on the other hand, are more like guidelines: you can recommend such items, but preferably not too much (or the opposite, more is better). Having too many of these soft rules can make system debugging and development highly challenging.
Rules are technical debt.
I find that the quantity of such rules in a system often depends on the internal power dynamics within the team. Product managers usually find it convenient to express constraints through rules, while engineers typically dislike these hacks. In my previous team, I took pride in our ability to keep the number of such rules to a minimum.
Throughout my career, I’ve frequently encountered a recurring pattern. The engineering team struggles to train the system in a way that produces good recommendations (either overall or in specific aspects). The product team then resorts to what they know best — adding new rules. Such patches are justified when quick fixes are needed, but they’re hard to remove later. The system often remains in this patched state until a major refactoring takes place, much like regular technical debt.
The moral of the story — don’t skimp on hiring strong engineers 🙂
In an ideal system, there should be no such rules; all the fuzzy logic should be handled by an advanced enough model. I dream that someday we’ll reach this technological state (and I have a hypothesis on how to achieve it). However, for the time being, it’s not realistic. So, instead of completely forbidding these rules, I’ll talk about an approach that allows for some organization and limits the chaos.
A Structured Approach: The Reranking Framework
This framework allows the integration of machine learning models with product rule enforcement, helping to structure these rules while avoiding complete chaos. However, it’s flexible and not overly restrictive, so it can’t guarantee total order. In a way, it’s simply a language for describing rules. In my opinion, it’s quite convenient.
In this discussion, we’ll focus on the final stage of ranking, where there are not too many documents left — say, from a few dozen to a couple of hundred — and we want to compile the best list from them. What’s interesting about this stage is that we’re not just trying to assess each document in the current context as precisely as possible, but we’re also considering how these documents combine with one another. This is when listwise ranking comes into play (not to be confused with listwise learning for learning-to-rank, where only the loss function depends on all the documents in a query, not the ranking function). A typical application of this listwise approach is to enhance result diversity.
Here are the key principles of the approach.
The results are generated iteratively, starting from the first position and ending at the last. On each iteration, we select the most fitting document for the upcoming position. This is how the majority of reranking strategies, like the well-known DPP for diversification, work. For non-linear outputs, positions can be ranked by importance.On each iteration, we take all remaining documents and sort them by a value function. This could range from something simple like the output of a click probability model to something more complex: a combination of various model outputs (or multiple models) predicting different events, diversity components (such as similarity to previous documents), and manual boosts, etc. The value function can be recalculated on each iteration and can therefore depend on both the position and the documents already in the final output. It must be computationally efficient. Crafting the right value function is a rich topic in itself; the framework neither restricts nor simplifies this aspect.Product rules are expressed as follows: within a subset of positions X, the number of documents with property f should be either above or below a certain threshold C. Usually, X is a range of starting positions, such as 1 to 10 (the first page). Property f is best expressed as a threshold rule of some feature, i.e., [feature(doc) > threshold]. If needed, this format can be generalized to include non-binary properties.Rules have priority. If we can’t satisfy all rules, we discard the least important ones. To be more precise: if the highest-priority rule is achievable at a given position, it will definitely be enforced; otherwise, it won’t be. If the next-highest-priority rule is achievable under those conditions, it will be enforced; otherwise, we skip it. And so on. In other words, we select the lexicographically highest mask of fulfilled rules.
Here are a few examples of rules in this format:
At least half of the documents in the entire output should be subscriptions. However, if all documents from subscriptions have already been read, this rule becomes infeasible and will be discarded.The number of low-quality documents in the first 10 positions should not exceed 2.Between positions 10 and 20, there should be at least one document from a new category.
It’s worth noting that rules like “at least 5 documents with a certain property must be in the first 10 positions” can result in the first 5 positions being filled with documents lacking that property, followed by 5 with it. To make this more evenly distributed, you can add rules for intermediate ranges: at least 1 in the first 2 positions, at least 2 in the first 4, and so on.
Implementing this framework efficiently is a good challenge but entirely doable. Here’s a Python code sketch to illustrate how one could implement the described reranking framework. Keep in mind, this is not optimized for efficiency but should give a good starting point.
def rerank(documents, count, rules, scorer):
result = []
while len(result) < count and len(documents) > 0:
position = len(result)
candidates = documents
for rule in rules:
filtered = [doc for doc in candidates if rule(position, doc)]
if len(filtered) > 0:
candidates = filtered
next_doc = max(candidates, key=lambda doc: scorer(position, doc))
result.append(next_doc)
documents.remove(next_doc)
scorer.update(position, next_doc)
for rule in rules:
rule.update(position, next_doc)
return result
Lastly, enhancing the system’s debuggability and controllability is greatly facilitated by logging all executed and discarded rules.
As we’ve seen, a ‘rules-free’ recommendation system remains more of an ideal than a reality for now. But that doesn’t mean we’re stuck in chaos. A well-structured framework for managing rules can give you the organization you need without stifling the system’s potential.
From Hacks to Harmony: Structuring Product Rules in Recommendations was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
