Documentation Center

Token

A token corresponds to a word, or short sequence of words and punctuation marks, in translatable text. When a translation memory (TM) is applied to source text, the TM analyzes the source into tokens. This analysis is independent of the analysis into segments. You can configure a TM to recognize text patterns as single tokens. For example, you can provide an expression that provides a definition of the patterns of text for dates. Such text patterns are called recognized tokens. You can provide rules for translating recognized tokens.