# Lucene 8.4.0 analyzers-opennlp API

This module exposes functionality from Apache OpenNLP to Apache Lucene.

See: Description

Packages
org.apache.lucene.analysis.opennlp: Analysis components based on OpenNLP
org.apache.lucene.analysis.opennlp.tools: Tools to supply access to OpenNLP components.

This module exposes functionality from Apache OpenNLP to Apache Lucene. The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text.

For an introduction to Lucene's analysis API, see the `org.apache.lucene.analysis` package documentation.

The OpenNLP Tokenizer behavior is similar to the WhiteSpaceTokenizer but is smart about inter-word punctuation. The term stream looks very much like the way you parse words and punctuation while reading. The major difference between this tokenizer and most other tokenizers shipped with Lucene is that punctuation is tokenized. This is required for the following taggers to operate properly.

The OpenNLP taggers annotate terms using the `TypeAttribute`.

- `OpenNLPTokenizer` segments text into sentences or words. This Tokenizer uses the OpenNLP Sentence Detector and/or Tokenizer classes. When used together, the Tokenizer receives sentences and can do a better job.
- `OpenNLPFilter` tags words using one or more technologies: Part-of-Speech, Chunking, and Named Entity Recognition. These tags are assigned as token types. Note that only of these operations will tag

Since the `TypeAttribute` is not stored in the index, it is recommended that one of these filters is used following `OpenNLPFilter` to enable search against the assigned tags:

- `TypeAsPayloadFilter` copies the `TypeAttribute` value to the `PayloadAttribute`
- `TypeAsSynonymFilter` creates a cloned token at the same position as each tagged token, and copies the {{TypeAttribute}} value to the {{CharTermAttribute}}, optionally with a customized prefix (so that tags effectively occupy a different namespace from token text).