Package org.apache.lucene.analysis.icu.segmentation

# Class DefaultICUTokenizerConfig

java.lang.Object
org.apache.lucene.analysis.icu.segmentation.ICUTokenizerConfig
org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig

public class DefaultICUTokenizerConfig extends ICUTokenizerConfig
Default `ICUTokenizerConfig` that is generally applicable to many languages.

Generally tokenizes Unicode text according to UAX#29 (`BreakIterator.getWordInstance(ULocale.ROOT)`), but with the following tailorings:

- Thai, Lao, Myanmar, Khmer, and CJK text is broken into words with a dictionary.

WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `WORD_EMOJI`
  Token type for words that appear to be emoji sequences
  `static final String`
  `WORD_HANGUL`
  Token type for words containing Korean hangul
  `static final String`
  `WORD_HIRAGANA`
  Token type for words containing Japanese hiragana
  `static final String`
  `WORD_IDEO`
  Token type for words containing ideographic characters
  `static final String`
  `WORD_KATAKANA`
  Token type for words containing Japanese katakana
  `static final String`
  `WORD_LETTER`
  Token type for words that contain letters
  `static final String`
  `WORD_NUMBER`
  Token type for words that appear to be numbers

  ### Fields inherited from class org.apache.lucene.analysis.icu.segmentation.ICUTokenizerConfig

  `EMOJI_SEQUENCE_STATUS`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `DefaultICUTokenizerConfig(boolean cjkAsWords, boolean myanmarAsWords)`
  Creates a new config.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `boolean`
  `combineCJ()`
  true if Han, Hiragana, and Katakana scripts should all be returned as Japanese
  `com.ibm.icu.text.RuleBasedBreakIterator`
  `getBreakIterator(int script)`
  Return a breakiterator capable of processing a given script.
  `String`
  `getType(int script, int ruleStatus)`
  Return a token type value for a given script and BreakIterator rule status.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### WORD_IDEO

    public static final String WORD_IDEO
    Token type for words containing ideographic characters

  - ### WORD_HIRAGANA

    public static final String WORD_HIRAGANA
    Token type for words containing Japanese hiragana

  - ### WORD_KATAKANA

    public static final String WORD_KATAKANA
    Token type for words containing Japanese katakana

  - ### WORD_HANGUL

    public static final String WORD_HANGUL
    Token type for words containing Korean hangul

  - ### WORD_LETTER

    public static final String WORD_LETTER
    Token type for words that contain letters

  - ### WORD_NUMBER

    public static final String WORD_NUMBER
    Token type for words that appear to be numbers

  - ### WORD_EMOJI

    public static final String WORD_EMOJI
    Token type for words that appear to be emoji sequences

- ## Constructor Details

  - ### DefaultICUTokenizerConfig

    public DefaultICUTokenizerConfig(boolean cjkAsWords, boolean myanmarAsWords)
    Creates a new config. This object is lightweight, but the first time the class is referenced, breakiterators will be initialized.
    Parameters:
      `cjkAsWords` - true if cjk text should undergo dictionary-based segmentation, otherwise text will be segmented according to UAX#29 defaults. If this is true, all Han+Hiragana+Katakana words will be tagged as IDEOGRAPHIC.
      `myanmarAsWords` - true if Myanmar text should undergo dictionary-based segmentation, otherwise it will be tokenized as syllables.

- ## Method Details

  - ### combineCJ

    public boolean combineCJ()
    Description copied from class: `ICUTokenizerConfig`
    true if Han, Hiragana, and Katakana scripts should all be returned as Japanese
    Specified by:
      `combineCJ` in class `ICUTokenizerConfig`

  - ### getBreakIterator

    public com.ibm.icu.text.RuleBasedBreakIterator getBreakIterator(int script)
    Description copied from class: `ICUTokenizerConfig`
    Return a breakiterator capable of processing a given script.
    Specified by:
      `getBreakIterator` in class `ICUTokenizerConfig`

  - ### getType

    public String getType(int script, int ruleStatus)
    Description copied from class: `ICUTokenizerConfig`
    Return a token type value for a given script and BreakIterator rule status.
    Specified by:
      `getType` in class `ICUTokenizerConfig`