org.apache.lucene.analysis.cjk

## Class CJKBigramFilter

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.TokenFilter
      - - org.apache.lucene.analysis.cjk.CJKBigramFilter

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public final class CJKBigramFilter
      extends TokenFilter

  Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

  CJK types are set by these tokenizers, but you can also use `CJKBigramFilter(TokenStream, int)` to explicitly control which of the CJK scripts are turned into bigrams.

  By default, when a CJK character has no adjacent characters to form a bigram, it is output in unigram form. If you want to always output both unigrams and bigrams, set the `outputUnigrams` flag in `CJKBigramFilter(TokenStream, int, boolean)`. This can be used for a combined unigram+bigram approach.

  Unlike ICUTokenizer, StandardTokenizer does not split at script boundaries. Korean Hangul characters are treated the same as many other scripts' letters, and as a result, StandardTokenizer can produce tokens that mix Hangul and non-Hangul characters, e.g. "한국abc". Such mixed-script tokens are typed as `<ALPHANUM>` rather than `<HANGUL>`, and as a result, will not be converted to bigrams by CJKBigramFilter. In all cases, all non-CJK input is passed thru unmodified.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

      `AttributeSource.State`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `DOUBLE_TYPE` when we emit a bigram, it's then marked as this type
    `static int` | `HAN` bigram flag for Han Ideographs
    `static int` | `HANGUL` bigram flag for Hangul
    `static int` | `HIRAGANA` bigram flag for Hiragana
    `static int` | `KATAKANA` bigram flag for Katakana
    `static String` | `SINGLE_TYPE` when we emit a unigram, it's then marked as this type

    - ### Fields inherited from class org.apache.lucene.analysis.TokenFilter

      `input`

    - ### Fields inherited from class org.apache.lucene.analysis.TokenStream

      `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CJKBigramFilter(TokenStream in)` Calls `CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)`
    `CJKBigramFilter(TokenStream in, int flags)` Calls `CJKBigramFilter(in, flags, false)`
    `CJKBigramFilter(TokenStream in, int flags, boolean outputUnigrams)` Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `boolean` | `incrementToken()`
    `void` | `reset()`

    - ### Methods inherited from class org.apache.lucene.analysis.TokenFilter

      `close, end`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### HAN

          public static final int HAN

      bigram flag for Han Ideographs
      See Also:
        Constant Field Values

    - #### HIRAGANA

          public static final int HIRAGANA

      bigram flag for Hiragana
      See Also:
        Constant Field Values

    - #### KATAKANA

          public static final int KATAKANA

      bigram flag for Katakana
      See Also:
        Constant Field Values

    - #### HANGUL

          public static final int HANGUL

      bigram flag for Hangul
      See Also:
        Constant Field Values

    - #### DOUBLE_TYPE

          public static final String DOUBLE_TYPE

      when we emit a bigram, it's then marked as this type
      See Also:
        Constant Field Values

    - #### SINGLE_TYPE

          public static final String SINGLE_TYPE

      when we emit a unigram, it's then marked as this type
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### CJKBigramFilter

          public CJKBigramFilter(TokenStream in)

      Calls `CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)`

    - #### CJKBigramFilter

          public CJKBigramFilter(TokenStream in,
                                 int flags)

      Calls `CJKBigramFilter(in, flags, false)`

    - #### CJKBigramFilter

          public CJKBigramFilter(TokenStream in,
                                 int flags,
                                 boolean outputUnigrams)

      Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.
      Parameters:
        `flags` - OR'ed set from `HAN`, `HIRAGANA`, `KATAKANA`, `HANGUL`
        `outputUnigrams` - true if unigrams for the selected writing systems should also be output. when this is false, this is only done when there are no adjacent characters to form a bigram.

  - ### Method Detail

    - #### incrementToken

          public boolean incrementToken()
                                 throws IOException

      Specified by:
        `incrementToken` in class `TokenStream`
      Throws:
        `IOException`

    - #### reset

          public void reset()
                     throws IOException

      Overrides:
        `reset` in class `TokenFilter`
      Throws:
        `IOException`

Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.