Documentation Center

Mapping custom locales in Elasticsearch to ensure IQ search indexing works

If you find that your search engine indexing is not able to index, you may need to map any custom language identifiers to standard two-letter language keys. Because IQ search uses a third-party search engine, Elasticsearch, you need to make changes to your Elasticsearch instance. This is for the case where you do the mapping later, that is, not at install time.

Before you begin

Before creating your mapping, you need to know the following:
  • the names of the custom language values that need to be mapped to standard two-letter language keys. (For example Anglais should be mapped to the en key.)
  • the URL of your Elasticsearch instance
  • some Elasticsearch search engine parameters; specifically, the names of the indexes containing these mappings, and the name of the mapping. If you used the default bootstrap.json at installation time, these names are
    • udp-locale-to-language for the index
    • locale-map-11-1 for the mapping.

About this task

When the search engine indexes search strings, it uses two-letter language identifiers (say, fr for French) and five-character language-plus-locale identifiers (say, en-us for US English) to identify the language of the search string.

When a user publishes content from SDL Tridion Docs Content Manager, the content is tagged with a language code (one of the values listed for the Language list of values). Maybe that language code is one of the two-letter or five-character language identifiers used by Elasticsearch. But you may have also configured a custom language identifier in the Language list of values (for example, Anglais for English), and applied that custom identifier to some of your content. By default, Elasticsearch won't recognize such an identifier. This means that the content won't be indexed and can't be searched. To get around this issue, you must create a mapping from an Elasticsearch language identifier (say en-us) to one or more custom language identifiers (say Anglais and Inglese) .

If you use custom language identifier and therefore need to map them, configure these mappings at installation time if at all possible, before you actually start using the IQ indexing service. However, you may have to configure this mapping at a later time, and as long as you are careful about which elements you edit, you should be able to avoid data corruption. Specifically , do not change the key elements (representing the Elasticsearch language identifier); only change the mapped values (representing your custom language identifier or identifiers).

To change the mapping, you need to know how to use the Get API to fetch documents from the Elasticsearch search engine and how to post to the Index API, which is documented externally here:

Procedure

  1. Fetch the current locale mapping document in your Elasticsearch engine by making a GET request of the form:
    URL/INDEXNAME/MAPPINGNAME/?search
    where:
    • URL is the base URL of the Elasticsearch engine that your Content Delivery IQ index service is using
    • INDEXNAME is the name of the index containing the configuration for this engine (defaults to udp-locale-to-language)
    • MAPPINGNAME is the name of the mapping (scheme) element (defaults to locale-to-language).
    Your request fetches the existing locale mapping.
  2. Copy the document you've fetched with the locale mapping, and edit only the values sections, by adding any new mapping values for custom languages. Do not change or delete any of the keys in the mapping file, because this could cause data corruption.
    In this example, you add Anglais and Inglese to the list of values for the en-gb key.
        {
          "id": "locale-map",
          "values": [
            {
              "key": "en-gb",
              "indexLanguage": "english",
              "values": [
                "en_GB",
                "en-GB",
                "Anglais",
                "Inglese"
              ]
            },
            {
              "key": "nl-nl",
              "indexLanguage": "dutch",
              "values": [
                "nl_NL",
                "nl-NL"
              ]
            }
          ]
        }
  3. Commit your changes by posting the modified JSON document to your Elasticsearch engine, using the Index API .
  4. To ensure that these changes are picked up, restart your IQ index service in Content Delivery.