Designing the Semantic Layer

At the heart of Tamarind lies its Semantic Engine. It is probably one of its strongest differentiators, and one of the reasons the product can scale without requiring every piece of content to be processed by an expensive AI model.

On its surface, Tamarind might seem just like another chat application, with the added feature of markdown-like editor (TipTap to be precise). But those are only the main mechanisms of data collection, as well as its first information surfacing locations. The job of the Semantic Engine is to turn the unstructured, messy information users generate into something Tamarind can understand, connect and retrieve later.

Content vectorization

Every time users write down something, whether it is on a page or in a conversation, a scheduled worker will fetch its contents and evaluate its semantic value.

Conversations behave quite differently from pages. Their nature is more dynamic and changing, a constant stream of data that might, or might not, be useful for the application to recollect. There is a whole layer of heuristics sitting in front of every sent message in charge of measuring their relevance and score. Once you start embedding things into vectors and consuming external APIs is when budget starts being eaten, thus the extensive usage of cheap heuristics on the early stages of the pipeline. This is the first applied filter.

The job of the Semantic Engine is to turn the unstructured, messy information users generate into something Tamarind can understand, connect and retrieve later.

Pages in contrast, are more static. A photograph of content that tends to have a higher intrinsic value, why would the user have spent the time writing it in the first place? The challenge here is the volume of tokens, rather than their throughput. They are much denser and richer in content than simple messages. Thus the need of chunking, that is, splitting the page into blocks of X tokens, while allowing a certain amount of hysteresis around that target size X. This hysteresis allows the engine to be flexible enough to fit full paragraphs of X±Y tokens into single chunks, instead of cutting a sentence in the middle.

The block size is a tricky business. On one hand you want to gather as much context as possible, so the underlying semantic value of the vector is representative enough of an idea or concept. On the other hand, the bigger the block is the broader its semantic representation will be, thus becoming inaccurate, risking giving false positives (or negatives) in future similarity comparisons.

Chunks and messages are both embedded using a relatively inexpensive model. These vectors are the bread and butter of the semantic engine. Once we start collecting these properly we can move into higher domains of knowledge: identifying the theme or topic of an entity.

Topic identification

Understanding what is the topic of a conversation, or a page, is essential for the late stages of the semantic engine. But once again, conversation themes have a different lifecycle than pages. Pages are relatively easy: a person writes down notes about certain task or ticket they are doing. There is usually one major theme in its thesis, and then multiple sub-themes orbiting the first, and often related. A couple of chunks might change here and there, but usually the main theme remains. And once the user is done writing their notes, the page and its semantic footprint remains stationary.

Once you start embedding things into vectors and consuming external APIs is when budget starts being eaten. One way to mitigate this behavior is to use cheap heuristics on the early stages of the pipeline.

Conversations are more interesting. Here we need to count for two or more participants rambling about different topics, sometimes in parallel. Topics are built once enough evidence is attained, but there might be other topics that were spoken about before, and that might still be relevant. Here we need to consider both recency and relevancy as the factors that determine whether a conversation "is about a topic or another".

In fact, a topic does not become a valuable semantic entity the first time a user writes about something new. Tamarind first needs enough semantically related content to establish that a meaningful theme is emerging from new messages.

In summary, the main difference between pages and conversation semantic identification is time. A page might be assigned certain theme and retain it, sometimes forever. Conversations on the other hand, might have certain topic as its prime theme of discussion for a while, and then change for a different one. We need to consider time-decay when we calculate the weight of each conversation topic, and set a half-life value for them, whcih can be a fixed value or variable, depending on the age of the conversation, amount of participants, message throughput, etc.

Surfacing insights

We finally arrive to the nicest part of the pipeline: surfacing the topics. This is the moment when we can return some value to our users, by classifying their assets by their topics or reminding them what were they talking about. Here is where Tamarind shines with features such as conversation suggestions, topic highlighting and deeplinks, topic minimaps or summarizations.

And it opens the door for future roadmap milestones, such as: knowledge graphs, reminders, cross-conversation topic discovery, etc.

These are the features that can keep Tamarind an evergreen product, and where I will invest most of the time after the MVP is delivered. The foundations are already being built. The next challenge is turning them into features that make Tamarind increasingly useful as its memory grows.