Mapping custom locales in Elasticsearch to ensure IQ search indexing works

If your search engine indexing is not able to index you may need to map any custom languages to standard two-letter language keys. The IQ search is using a third party search engine, the Elasticsearch, so you need to make changes to your Elasticsearch instance. This is for the case where you do the mapping later, that is not at install time.

Before you begin

You need to know the following:
  • The names of the custom language values that need to be mapped to standard two-letter language keys. (For example Anglais should be mapped to the en key.)
  • the URL of your Elasticsearch instance
  • Some Elasticsearch engine parameters. Specifically, the names of the configuration index and the mapping or scheme. If you used the default bootstrap.json at installation time, these names are
    • udp-configuration for the index
    • locale-to-language for the mapping.

About this task

The Elasticsearch search engine indexes search strings by two-letter language codes (for example, fr for French). In SDL Knowledge Center, DITA content can denote its language the same way (for example, using en for English). But the DITA content can also use a five-character language-plus-locale string (for example, en-us for US English) or a custom string (for example, Anglais for English) to denote the content language.

By default, SDL Knowledge Center assumes that the first two characters of the DITA language value denote the correct Elasticsearch language in which to index. This approach works for strings such as en and en-us: both resolve to en, the right Elastisearch language. But this approach does not work for strings like Anglais, which resolves to the meaningless string An. In that last case, you must configure how your custom language values map to Elasticsearch languages.

You need to know how to post to the Index API and use the get API to fetch documents from the Elasticsearch engine, which is documented externally here:

Procedure

  1. Start by fetching the current locale mapping document in your Elasticsearch engine by making a GET request of the form:
    <url>/<index_name/<mapping>/?search
    where:
    • URL is the URL of the Elasticsearch engine your Content Delivery IQ index service is using
    • index_name is the name of the index containing the configuration for this engine
    • mapping is the name of the mapping (scheme) element.
    You get the existing locale mapping.
  2. Copy the document you've fetched with the locale mapping, and edit only the values, by adding any new mapping values for custom languages. Do not change or delete any of the keys in the mapping file because this could cause data corruption.
    In this example, you add Anglais to the list of values for the en key.
        {
          "id": "locale-map",
          "values": [
            {
              "key": "en",
              "values": [
                "en_GB",
                "en-GB"
                "Anglais"
              ]
            },
            {
              "key": "nl",
              "values": [
                "nl_NL",
                "nl-NL"
              ]
            }
          ]
        }
  3. Post the modified JSON document to your Elasticsearch engine, using the Index API .
  4. To ensure these changes are picked up you need to restart your IQ index service in Content Delivery.