Elasticsearch Analyzers for Russian and Uzbek Search

The short answer

In Uzbekistan the same user may type “non” (Uzbek Latin), “нон” (Uzbek Cyrillic) or “хлеб” (Russian), often with a typo. One analyzer cannot handle all of that. The working setup is:

• a Russian field with lowercasing, stop words, the Russian stemmer and search-time synonyms;
• a transliteration subfield where any Cyrillic is converted to Uzbek Latin and apostrophes are removed;
• a query that searches both, with fuzziness on the transliterated field.

Every rule is then checked with the _analyze API before it reaches production.

Step 1. Analyzers and mapping

PUT /products
{
  "settings": {
    "analysis": {
      "char_filter": {
        "strip_apostrophes": {
          "type": "pattern_replace",
          "pattern": "[‘’ʻʼ'`ъьЪЬ]",
          "replacement": ""
        },
        "ru_yo": { "type": "mapping", "mappings": ["ё => е", "Ё => Е"] },
        "cyr_to_lat": { "type": "mapping", "mappings_path": "analysis/uz_cyr_lat.txt" }
      },
      "filter": {
        "ru_stop": { "type": "stop", "stopwords": "_russian_" },
        "ru_stemmer": { "type": "stemmer", "language": "russian" },
        "product_synonyms": {
          "type": "synonym_graph",
          "synonyms": ["non, нон, хлеб, лепешка", "sharbat, шарбат, сок"]
        }
      },
      "analyzer": {
        "ru_index": {
          "tokenizer": "standard",
          "char_filter": ["ru_yo"],
          "filter": ["lowercase", "ru_stop", "ru_stemmer"]
        },
        "ru_search": {
          "tokenizer": "standard",
          "char_filter": ["ru_yo"],
          "filter": ["lowercase", "product_synonyms", "ru_stop", "ru_stemmer"]
        },
        "uz_translit": {
          "tokenizer": "standard",
          "char_filter": ["strip_apostrophes", "cyr_to_lat"],
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "ru_index",
        "search_analyzer": "ru_search",
        "fields": {
          "translit": { "type": "text", "analyzer": "uz_translit" }
        }
      }
    }
  }
}

The file config/analysis/uz_cyr_lat.txt on every node holds one rule per line, for both lower and upper case:

ў => o
Ў => O
қ => q
ғ => g
ҳ => h
ш => sh
ч => ch
ё => yo
ю => yu
я => ya
х => x
ц => ts
н => n
о => o

…and so on for the whole alphabet. Key decisions:

• Apostrophes are removed first, so “o‘zbek”, “o'zbek”, “ozbek” and “ўзбек” all become ozbek. Users rarely type the correct ‘ character.
• Synonyms only at search time. synonym_graph belongs in the search analyzer, and changing the list does not require reindexing documents.
• Synonyms go before the stemmer, otherwise “лепешка” is already stemmed and no longer matches the rule.

Step 2. Check everything with _analyze

POST /products/_analyze
{ "field": "name.translit", "text": "Нон Ўзбекча" }

Expected tokens: non, ozbekcha. Now the Russian side:

POST /products/_analyze
{ "analyzer": "ru_search", "text": "лепёшка" }

You should see the stemmed forms of “лепешка” and “хлеб” plus non and нон. If something is missing, the problem is the filter order or the ё normalization. Note that synonyms match exact word forms, so add the plural and other frequent forms to the rules where needed. Keep these calls as a small test suite and run it after every analyzer change.

Step 3. A query with typo tolerance

GET /products/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "name": { "query": "нонн" } } },
        { "match": { "name.translit": {
            "query": "нонн",
            "fuzziness": "AUTO",
            "prefix_length": 1
        } } }
      ]
    }
  }
}

• fuzziness: AUTO allows no edits for very short words, one edit for medium and two for longer words, so “nonn” and “nan” still reach “non”.
• prefix_length: 1 keeps the first letter exact: fewer false matches and faster queries.
• Fuzziness is applied to the transliterated field only. Mixing it with multi-word synonyms makes results harder to predict.

Uzbek suffixes

Elasticsearch has no built-in Uzbek stemmer, and Uzbek is agglutinative: “nonlar”, “nonni”, “nonning”. Options, from simple to complex:

• an edge_ngram subfield for prefix and autocomplete search;
• a pattern_replace filter that cuts a short list of frequent suffixes — carefully, so you do not damage word roots;
• a custom dictionary via stemmer_override for your top product names.

Common mistakes

• Using the built-in russian analyzer for everything: Latin-script queries never meet Cyrillic product names.
• Forgetting uppercase letters in the mapping file.
• Turning on fuzziness for SKUs and barcodes — use a keyword field with exact match for them.
• Changing index-time analyzers without a reindex: old documents keep old tokens.