Qirava DMS docs

Search: full-text inverted index

Full-text search uses SEARCH <field> MATCH "<term>". When the field has a Search index, the engine serves the query from an inverted index: text is tokenized into lowercase alphanumeric words, and each token maps to the rows that contain it. A multi-word term is the AND (intersection) of its tokens' postings. Candidates are then verified with a case-insensitive substring check, and any WHERE filter is applied on top. This page covers tokenization, the inverted-index lookup, multi-token AND, and the exact-substring residual evaluation.

SEARCH syntax#

SEARCH <field> MATCH "<term>" is a clause on a SELECT and combines with WHERE, SORT, LIMIT, and projection. The term must be a string literal. Search can be combined with a WHERE filter (the filter is applied to the search candidates).

text

SELECT * FROM docs SEARCH body MATCH "fox" LIMIT 5

Tokenization#

Both the indexed text and the search term are tokenized the same way: split on any non-alphanumeric character, drop empties, lowercase each word, then sort and dedupe. So "the quick brown Fox" indexes the tokens brown, fox, quick, the, and a search for "Fox" lowercases to the token fox. Search is therefore case-insensitive and word-based.

Inverted index and multi-token AND#

A Search index is an inverted index: the BTreeMap maps each token to the positions of rows containing it (one entry per distinct token per row). A SEARCH ... MATCH "a b" term is tokenized to [a, b] and served as the AND (intersection) of the two tokens' postings lists — rows must contain every token. If the field has no search index, the query falls back to a scan.

  term "quick fox"  ──tokenize──▶  [fox, quick]
        postings[fox]   = {rowA, rowC}
        postings[quick] = {rowA}
        AND ─────────────────────▶  {rowA}
        residual substring check + WHERE ▶ final rows

Inverted-index search

Substring residual evaluation#

After the inverted index narrows the candidates, each candidate is verified with eval_search: the stored field (lowercased) must CONTAIN the search term (lowercased) as a substring. This residual keeps the result exact and consistent with the tokenizer's case-insensitivity. Any WHERE filter is also re-applied to the candidates.

Request

-- docs: 1 "the quick brown Fox"; 2 "lazy dog sleeps"; 3 "a fox and a hound"
SELECT * FROM docs SEARCH body MATCH "fox"
SELECT * FROM docs SEARCH body MATCH "dog"
SELECT * FROM docs SEARCH body MATCH "cat"

UPDATE docs SET body = "cat only" WHERE id = 2
SELECT * FROM docs SEARCH body MATCH "cat"

Response

// fox -> count 2 (docs 1 and 3) ; dog -> count 1 ; cat -> count 0
// after the UPDATE re-tokenizes doc 2:  cat -> count 1 ; dog -> count 0