- The problem: PostgreSQL does not know where Thai words end
- The fix: let embeddings carry Thai
- Fusing with Reciprocal Rank Fusion
#The problem: PostgreSQL does not know where Thai words end
PostgreSQL's full-text search assumes whitespace separates words. Thai has no
such separator, so to_tsvector produces one long token instead of several.
SELECT to_tsvector('simple', 'การค้นหาแบบไฮบริด');
-- 'การค้นหาแบบไฮบริด':1
A reader searching for a word inside that string matches nothing.
#The fix: let embeddings carry Thai
Embedding models never tokenise on whitespace — they work on learned subword
units. So semantic search carries Thai queries, while keyword search remains the
better arm for exact English identifiers like pgvector or wire:stream, which
embeddings tend to blur together with their neighbours.
#Fusing with Reciprocal Rank Fusion
ts_rank and cosine similarity are not on comparable scales, so summing them
requires a weight that has to be refitted per corpus. RRF fuses on rank
instead, which sidesteps the problem entirely.
score(doc) = Σ 1 / (k + rank_i(doc)), k = 60