org.apache.lucene.analysis.miscellaneous

## Class WordDelimiterGraphFilter

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.TokenFilter
      - - org.apache.lucene.analysis.miscellaneous.WordDelimiterGraphFilter

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public final class WordDelimiterGraphFilter
      extends TokenFilter

  Splits words into subwords and performs optional transformations on subword groups, producing a correct token graph so that e.g. `PhraseQuery` can work correctly when this filter is used in the search-time analyzer. Unlike the deprecated `WordDelimiterFilter`, this token filter produces a correct token graph as output. However, it cannot consume an input token graph correctly.

  Words are split into subwords with the following rules:

  - split on intra-word delimiters (by default, all non alpha-numeric characters): `"Wi-Fi"` → `"Wi", "Fi"`
  - split on case transitions: `"PowerShot"` → `"Power", "Shot"`
  - split on letter-number transitions: `"SD500"` → `"SD", "500"`
  - leading and trailing intra-word delimiters on each subword are ignored: `"//hello---there, 'dude'"` → `"hello", "there", "dude"`
  - trailing "'s" are removed for each subword: `"O'Neil's"` → `"O", "Neil"`
    - Note: this step isn't performed in a separate filter because of possible subword combinations.
  The combinations parameter affects how subwords are combined:
  - combinations="0" causes no subword combinations: `"PowerShot"` → `0:"Power", 1:"Shot"` (0 and 1 are the token positions)
  - combinations="1" means that in addition to the subwords, maximum runs of non-numeric subwords are catenated and produced at the same position of the last subword in the run:
    - `"PowerShot"` → `0:"Power", 1:"Shot" 1:"PowerShot"`
    - `"A's+B's&C's"` > `0:"A", 1:"B", 2:"C", 2:"ABC"`
    - `"Super-Duper-XL500-42-AutoCoder!"` → `0:"Super", 1:"Duper", 2:"XL", 2:"SuperDuperXL", 3:"500" 4:"42", 5:"Auto", 6:"Coder", 6:"AutoCoder"`
  One use for `WordDelimiterGraphFilter` is to help match words with different subword delimiters. For example, if the source text contained "wi-fi" one may want "wifi" "WiFi" "wi-fi" "wi+fi" queries to all match. One way of doing so is to specify combinations="1" in the analyzer used for indexing, and combinations="0" (the default) in the analyzer used for querying. Given that the current `StandardTokenizer` immediately removes many intra-word delimiters, it is recommended that this filter be used after a tokenizer that does not do this (such as `WhitespaceTokenizer`).

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

      `AttributeSource.State`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `CATENATE_ALL` Causes all subword parts to be catenated:
    `static int` | `CATENATE_NUMBERS` Causes maximum runs of number parts to be catenated:
    `static int` | `CATENATE_WORDS` Causes maximum runs of word parts to be catenated:
    `static int` | `GENERATE_NUMBER_PARTS` Causes number subwords to be generated:
    `static int` | `GENERATE_WORD_PARTS` Causes parts of words to be generated:
    `static int` | `PRESERVE_ORIGINAL` Causes original words are preserved and added to the subword list (Defaults to false)
    `static int` | `SPLIT_ON_CASE_CHANGE` Causes lowercase -> uppercase transition to start a new subword.
    `static int` | `SPLIT_ON_NUMERICS` If not set, causes numeric changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens).
    `static int` | `STEM_ENGLISH_POSSESSIVE` Causes trailing "'s" to be removed for each subword

    - ### Fields inherited from class org.apache.lucene.analysis.TokenFilter

      `input`

    - ### Fields inherited from class org.apache.lucene.analysis.TokenStream

      `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `WordDelimiterGraphFilter(TokenStream in, byte[] charTypeTable, int configurationFlags, CharArraySet protWords)` Creates a new WordDelimiterGraphFilter
    `WordDelimiterGraphFilter(TokenStream in, int configurationFlags, CharArraySet protWords)` Creates a new WordDelimiterGraphFilter using `WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE` as its charTypeTable

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static String` | `flagsToString(int flags)` Returns string representation of configuration flags
    `boolean` | `incrementToken()`
    `void` | `reset()`
    `String` | `toString()`

    - ### Methods inherited from class org.apache.lucene.analysis.TokenFilter

      `close, end`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### GENERATE_WORD_PARTS

          public static final int GENERATE_WORD_PARTS

      Causes parts of words to be generated:

      "PowerShot" => "Power" "Shot"

      See Also:
        Constant Field Values

    - #### GENERATE_NUMBER_PARTS

          public static final int GENERATE_NUMBER_PARTS

      Causes number subwords to be generated:

      "500-42" => "500" "42"

      See Also:
        Constant Field Values

    - #### CATENATE_WORDS

          public static final int CATENATE_WORDS

      Causes maximum runs of word parts to be catenated:

      "wi-fi" => "wifi"

      See Also:
        Constant Field Values

    - #### CATENATE_NUMBERS

          public static final int CATENATE_NUMBERS

      Causes maximum runs of number parts to be catenated:

      "500-42" => "50042"

      See Also:
        Constant Field Values

    - #### CATENATE_ALL

          public static final int CATENATE_ALL

      Causes all subword parts to be catenated:

      "wi-fi-4000" => "wifi4000"

      See Also:
        Constant Field Values

    - #### PRESERVE_ORIGINAL

          public static final int PRESERVE_ORIGINAL

      Causes original words are preserved and added to the subword list (Defaults to false)

      "500-42" => "500" "42" "500-42"

      See Also:
        Constant Field Values

    - #### SPLIT_ON_CASE_CHANGE

          public static final int SPLIT_ON_CASE_CHANGE

      Causes lowercase -> uppercase transition to start a new subword.
      See Also:
        Constant Field Values

    - #### SPLIT_ON_NUMERICS

          public static final int SPLIT_ON_NUMERICS

      If not set, causes numeric changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens).
      See Also:
        Constant Field Values

    - #### STEM_ENGLISH_POSSESSIVE

          public static final int STEM_ENGLISH_POSSESSIVE

      Causes trailing "'s" to be removed for each subword

      "O'Neil's" => "O", "Neil"

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### WordDelimiterGraphFilter

          public WordDelimiterGraphFilter(TokenStream in,
                                          byte[] charTypeTable,
                                          int configurationFlags,
                                          CharArraySet protWords)

      Creates a new WordDelimiterGraphFilter
      Parameters:
        `in` - TokenStream to be filtered
        `charTypeTable` - table containing character types
        `configurationFlags` - Flags configuring the filter
        `protWords` - If not null is the set of tokens to protect from being delimited

    - #### WordDelimiterGraphFilter

          public WordDelimiterGraphFilter(TokenStream in,
                                          int configurationFlags,
                                          CharArraySet protWords)

      Creates a new WordDelimiterGraphFilter using `WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE` as its charTypeTable
      Parameters:
        `in` - TokenStream to be filtered
        `configurationFlags` - Flags configuring the filter
        `protWords` - If not null is the set of tokens to protect from being delimited

  - ### Method Detail

    - #### incrementToken

          public boolean incrementToken()
                                 throws IOException

      Specified by:
        `incrementToken` in class `TokenStream`
      Throws:
        `IOException`

    - #### reset

          public void reset()
                     throws IOException

      Overrides:
        `reset` in class `TokenFilter`
      Throws:
        `IOException`

    - #### flagsToString

          public static String flagsToString(int flags)

      Returns string representation of configuration flags

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `AttributeSource`

Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.