Package org.apache.lucene.analysis.wikipedia

# Class WikipediaTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.wikipedia.WikipediaTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class WikipediaTokenizer extends Tokenizer
Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.

WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `ACRONYM_ID`
  `static final int`
  `ALPHANUM_ID`
  `static final int`
  `APOSTROPHE_ID`
  `static final String`
  `BOLD`
  `static final int`
  `BOLD_ID`
  `static final String`
  `BOLD_ITALICS`
  `static final int`
  `BOLD_ITALICS_ID`
  `static final int`
  `BOTH`
  Output the both the untokenized token and the splits
  `static final String`
  `CATEGORY`
  `static final int`
  `CATEGORY_ID`
  `static final String`
  `CITATION`
  `static final int`
  `CITATION_ID`
  `static final int`
  `CJ_ID`
  `static final int`
  `COMPANY_ID`
  `static final int`
  `EMAIL_ID`
  `static final String`
  `EXTERNAL_LINK`
  `static final int`
  `EXTERNAL_LINK_ID`
  `static final String`
  `EXTERNAL_LINK_URL`
  `static final int`
  `EXTERNAL_LINK_URL_ID`
  `static final String`
  `HEADING`
  `static final int`
  `HEADING_ID`
  `static final int`
  `HOST_ID`
  `static final String`
  `INTERNAL_LINK`
  `static final int`
  `INTERNAL_LINK_ID`
  `static final String`
  `ITALICS`
  `static final int`
  `ITALICS_ID`
  `static final int`
  `NUM_ID`
  `static final String`
  `SUB_HEADING`
  `static final int`
  `SUB_HEADING_ID`
  `static final String[]`
  `TOKEN_TYPES`
  String token types that correspond to token type int constants
  `static final int`
  `TOKENS_ONLY`
  Only output tokens
  `static final int`
  `UNTOKENIZED_ONLY`
  Only output untokenized tokens, which are tokens that would normally be split into several tokens
  `static final int`
  `UNTOKENIZED_TOKEN_FLAG`
  This flag is used to indicate that the produced "Token" would, if `TOKENS_ONLY` was used, produce multiple tokens.

  ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `WikipediaTokenizer()`
  Creates a new instance of the `WikipediaTokenizer`.
  `WikipediaTokenizer(int tokenOutput, Set<String> untokenizedTypes)`
  Creates a new instance of the `WikipediaTokenizer`.
  `WikipediaTokenizer(AttributeFactory factory, int tokenOutput, Set<String> untokenizedTypes)`
  Creates a new instance of the `WikipediaTokenizer`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `void`
  `end()`
  `final boolean`
  `incrementToken()`
  `void`
  `reset()`

  ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

  `correctOffset, setReader, setReaderTestPoint`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### INTERNAL_LINK

    public static final String INTERNAL_LINK
    See Also:
      - Constant Field Values

  - ### EXTERNAL_LINK

    public static final String EXTERNAL_LINK
    See Also:
      - Constant Field Values

  - ### EXTERNAL_LINK_URL

    public static final String EXTERNAL_LINK_URL
    See Also:
      - Constant Field Values

  - ### CITATION

    public static final String CITATION
    See Also:
      - Constant Field Values

  - ### CATEGORY

    public static final String CATEGORY
    See Also:
      - Constant Field Values

  - ### BOLD

    public static final String BOLD
    See Also:
      - Constant Field Values

  - ### ITALICS

    public static final String ITALICS
    See Also:
      - Constant Field Values

  - ### BOLD_ITALICS

    public static final String BOLD_ITALICS
    See Also:
      - Constant Field Values

  - ### HEADING

    public static final String HEADING
    See Also:
      - Constant Field Values

  - ### SUB_HEADING

    public static final String SUB_HEADING
    See Also:
      - Constant Field Values

  - ### ALPHANUM_ID

    public static final int ALPHANUM_ID
    See Also:
      - Constant Field Values

  - ### APOSTROPHE_ID

    public static final int APOSTROPHE_ID
    See Also:
      - Constant Field Values

  - ### ACRONYM_ID

    public static final int ACRONYM_ID
    See Also:
      - Constant Field Values

  - ### COMPANY_ID

    public static final int COMPANY_ID
    See Also:
      - Constant Field Values

  - ### EMAIL_ID

    public static final int EMAIL_ID
    See Also:
      - Constant Field Values

  - ### HOST_ID

    public static final int HOST_ID
    See Also:
      - Constant Field Values

  - ### NUM_ID

    public static final int NUM_ID
    See Also:
      - Constant Field Values

  - ### CJ_ID

    public static final int CJ_ID
    See Also:
      - Constant Field Values

  - ### INTERNAL_LINK_ID

    public static final int INTERNAL_LINK_ID
    See Also:
      - Constant Field Values

  - ### EXTERNAL_LINK_ID

    public static final int EXTERNAL_LINK_ID
    See Also:
      - Constant Field Values

  - ### CITATION_ID

    public static final int CITATION_ID
    See Also:
      - Constant Field Values

  - ### CATEGORY_ID

    public static final int CATEGORY_ID
    See Also:
      - Constant Field Values

  - ### BOLD_ID

    public static final int BOLD_ID
    See Also:
      - Constant Field Values

  - ### ITALICS_ID

    public static final int ITALICS_ID
    See Also:
      - Constant Field Values

  - ### BOLD_ITALICS_ID

    public static final int BOLD_ITALICS_ID
    See Also:
      - Constant Field Values

  - ### HEADING_ID

    public static final int HEADING_ID
    See Also:
      - Constant Field Values

  - ### SUB_HEADING_ID

    public static final int SUB_HEADING_ID
    See Also:
      - Constant Field Values

  - ### EXTERNAL_LINK_URL_ID

    public static final int EXTERNAL_LINK_URL_ID
    See Also:
      - Constant Field Values

  - ### TOKEN_TYPES

    public static final String[] TOKEN_TYPES
    String token types that correspond to token type int constants

  - ### TOKENS_ONLY

    public static final int TOKENS_ONLY
    Only output tokens
    See Also:
      - Constant Field Values

  - ### UNTOKENIZED_ONLY

    public static final int UNTOKENIZED_ONLY
    Only output untokenized tokens, which are tokens that would normally be split into several tokens
    See Also:
      - Constant Field Values

  - ### BOTH

    public static final int BOTH
    Output the both the untokenized token and the splits
    See Also:
      - Constant Field Values

  - ### UNTOKENIZED_TOKEN_FLAG

    public static final int UNTOKENIZED_TOKEN_FLAG
    This flag is used to indicate that the produced "Token" would, if `TOKENS_ONLY` was used, produce multiple tokens.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### WikipediaTokenizer

    public WikipediaTokenizer()
    Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a newly created JFlex scanner.

  - ### WikipediaTokenizer

    public WikipediaTokenizer(int tokenOutput, Set<String> untokenizedTypes)
    Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to the newly created JFlex scanner.
    Parameters:
      `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

  - ### WikipediaTokenizer

    public WikipediaTokenizer(AttributeFactory factory, int tokenOutput, Set<String> untokenizedTypes)
    Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to the newly created JFlex scanner. Uses the given `AttributeFactory`.
    Parameters:
      `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

- ## Method Details

  - ### incrementToken

    public final boolean incrementToken() throws IOException
    Specified by:
      `incrementToken` in class `TokenStream`
    Throws:
      `IOException`

  - ### close

    public void close() throws IOException
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Overrides:
      `close` in class `Tokenizer`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    Overrides:
      `reset` in class `Tokenizer`
    Throws:
      `IOException`

  - ### end

    public void end() throws IOException
    Overrides:
      `end` in class `TokenStream`
    Throws:
      `IOException`