org.apache.lucene.analysis.icu.segmentation

## Class DefaultICUTokenizerConfig

- java.lang.Object
- - org.apache.lucene.analysis.icu.segmentation.ICUTokenizerConfig
  - - org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig

    public class DefaultICUTokenizerConfig
    extends ICUTokenizerConfig

Default `ICUTokenizerConfig` that is generally applicable to many languages.

Generally tokenizes Unicode text according to UAX#29 (`BreakIterator.getWordInstance(ULocale.ROOT)`), but with the following tailorings:

- Thai, Lao, Myanmar, Khmer, and CJK text is broken into words with a dictionary.

WARNING: This API is experimental and might change in incompatible ways in the next release.

### Field Summary

Fields
`static String` `WORD_EMOJI` Token type for words that appear to be emoji sequences
`static String` `WORD_HANGUL` Token type for words containing Korean hangul
`static String` `WORD_HIRAGANA` Token type for words containing Japanese hiragana
`static String` `WORD_IDEO` Token type for words containing ideographic characters
`static String` `WORD_KATAKANA` Token type for words containing Japanese katakana
`static String` `WORD_LETTER` Token type for words that contain letters
`static String` `WORD_NUMBER` Token type for words that appear to be numbers

### Fields inherited from class org.apache.lucene.analysis.icu.segmentation.ICUTokenizerConfig

`EMOJI_SEQUENCE_STATUS`

### Constructor Summary

Constructors
Constructor and Description
`DefaultICUTokenizerConfig(boolean cjkAsWords, boolean myanmarAsWords)` Creates a new config.

### Method Summary

All Methods Instance Methods Concrete Methods
`boolean` `combineCJ()` true if Han, Hiragana, and Katakana scripts should all be returned as Japanese
`com.ibm.icu.text.RuleBasedBreakIterator` `getBreakIterator(int script)` Return a breakiterator capable of processing a given script.
`String` `getType(int script, int ruleStatus)` Return a token type value for a given script and BreakIterator rule status.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### WORD_IDEO

    public static final String WORD_IDEO

Token type for words containing ideographic characters

#### WORD_HIRAGANA

    public static final String WORD_HIRAGANA

Token type for words containing Japanese hiragana

#### WORD_KATAKANA

    public static final String WORD_KATAKANA

Token type for words containing Japanese katakana

#### WORD_HANGUL

    public static final String WORD_HANGUL

Token type for words containing Korean hangul

#### WORD_LETTER

    public static final String WORD_LETTER

Token type for words that contain letters

#### WORD_NUMBER

    public static final String WORD_NUMBER

Token type for words that appear to be numbers

#### WORD_EMOJI

    public static final String WORD_EMOJI

Token type for words that appear to be emoji sequences

### Constructor Detail

#### DefaultICUTokenizerConfig

    public DefaultICUTokenizerConfig(boolean cjkAsWords,
                                     boolean myanmarAsWords)

Creates a new config. This object is lightweight, but the first time the class is referenced, breakiterators will be initialized.

Parameters:
  `cjkAsWords` - true if cjk text should undergo dictionary-based segmentation, otherwise text will be segmented according to UAX#29 defaults. If this is true, all Han+Hiragana+Katakana words will be tagged as IDEOGRAPHIC.
  `myanmarAsWords` - true if Myanmar text should undergo dictionary-based segmentation, otherwise it will be tokenized as syllables.

### Method Detail

#### combineCJ

    public boolean combineCJ()

Description copied from class: `ICUTokenizerConfig`
true if Han, Hiragana, and Katakana scripts should all be returned as Japanese

Specified by:
  `combineCJ` in class `ICUTokenizerConfig`

#### getBreakIterator

    public com.ibm.icu.text.RuleBasedBreakIterator getBreakIterator(int script)

Description copied from class: `ICUTokenizerConfig`
Return a breakiterator capable of processing a given script.

Specified by:
  `getBreakIterator` in class `ICUTokenizerConfig`

#### getType

    public String getType(int script,
                          int ruleStatus)

Description copied from class: `ICUTokenizerConfig`
Return a token type value for a given script and BreakIterator rule status.

Specified by:
  `getType` in class `ICUTokenizerConfig`