<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"[
]>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<chapter id="tools.namefind">

	<title>Name Finder</title>

	<section id="tools.namefind.recognition">
		<title>Named Entity Recognition</title>
		<para>
			The Name Finder can detect named entities and numbers in text. To be able to
			detect entities the Name Finder needs a model. The model is dependent on the
			language and entity type it was trained for. The OpenNLP projects offers a number
			of pre-trained name finder models which are trained on various freely available corpora.
			They can be downloaded at our model download page. To find names in raw text the text
			must be segmented into tokens and sentences. A detailed description is given in the
			sentence detector and tokenizer tutorial. It is important that the tokenization for
			the training data and the input text is identical.
		</para>
	
	<section id="tools.namefind.recognition.cmdline">
		<title>Name Finder Tool</title>
		<para>
			The easiest way to try out the Name Finder is the command line tool.
			The tool is only intended for demonstration and testing. Download the
			English
			person model and start the Name Finder Tool with this command:
			<screen>
				<![CDATA[
$ opennlp TokenNameFinder en-ner-person.bin]]>
			 </screen>
			 
			The name finder now reads a tokenized sentence per line from stdin, an empty
			line indicates a document boundary and resets the adaptive feature generators.
			Just copy this text to the terminal:
	
			<screen>
				<![CDATA[
Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr . Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC , was named
    a director of this British industrial conglomerate .]]>
			 </screen>
			 the name finder will now output the text with markup for person names:
			<screen>
				<![CDATA[
<START:person> Pierre Vinken <END> , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr . <START:person> Vinken <END> is chairman of Elsevier N.V. , the Dutch publishing group .
<START:person> Rudolph Agnew <END> , 55 years old and former chairman of Consolidated Gold Fields PLC ,
    was named a director of this British industrial conglomerate .]]>
			 </screen>
		</para>
	</section>
		<section id="tools.namefind.recognition.api">
		<title>Name Finder API</title>
		<para>
			To use the Name Finder in a production system it is strongly recommended to embed it
			directly into the application instead of using the command line interface.
			First the name finder model must be loaded into memory from disk or an other source.
			In the sample below it is loaded from disk.
			<programlisting language="java">
				<![CDATA[
try (InputStream modelIn = new FileInputStream("en-ner-person.bin")){
  TokenNameFinderModel model = new TokenNameFinderModel(modelIn);
}
]]>
			 </programlisting>
			 There is a number of reasons why the model loading can fail:
			 <itemizedlist>
				<listitem>
					<para>Issues with the underlying I/O</para>
				</listitem>
				<listitem>
					<para>The version of the model is not compatible with the OpenNLP version</para>
				</listitem>
				<listitem>
					<para>The model is loaded into the wrong component,
					for example a tokenizer model is loaded with TokenNameFinderModel class.</para>
				</listitem>
				<listitem>
					<para>The model content is not valid for some other reason</para>
				</listitem>
			</itemizedlist>
			After the model is loaded the NameFinderME can be instantiated.
			<programlisting language="java">
				<![CDATA[
NameFinderME nameFinder = new NameFinderME(model);]]>
			</programlisting>
			The initialization is now finished and the Name Finder can be used. The NameFinderME
			class is not thread safe, it must only be called from one thread. To use multiple threads
			multiple NameFinderME instances sharing the same model instance can be created.
			The input text should be segmented into documents, sentences and tokens.
			To perform entity detection an application calls the find method for every sentence in the
			document. After every document clearAdaptiveData must be called to clear the adaptive data in
			the feature generators. Not calling clearAdaptiveData can lead to a sharp drop in the detection
			rate after a few documents.
			The following code illustrates that:
			<programlisting language="java">
				<![CDATA[
for (String document[][] : documents) {

  for (String[] sentence : document) {
    Span nameSpans[] = nameFinder.find(sentence);
    // do something with the names
  }

  nameFinder.clearAdaptiveData()
}]]>
			 </programlisting>
			 the following snippet shows a call to find
			 <programlisting language="java">
				<![CDATA[
String sentence[] = new String[]{
    "Pierre",
    "Vinken",
    "is",
    "61",
    "years"
    "old",
    "."
    };

Span nameSpans[] = nameFinder.find(sentence);]]>
			</programlisting>
			The nameSpans arrays contains now exactly one Span which marks the name Pierre Vinken. 
			The elements between the begin and end offsets are the name tokens. In this case the begin 
			offset is 0 and the end offset is 2. The Span object also knows the type of the entity.
			In this case it is person (defined by the model). It can be retrieved with a call to Span.getType().
			Additionally to the statistical Name Finder, OpenNLP also offers a dictionary and a regular
			expression name finder implementation.
		</para>
			<section id="tools.namefind.api.onnx">
			<title>Using an ONNX Model</title>
				<para>
					Using an ONNX model is similar, except we will utilize the <code>NameFinderDL</code> class instead.
					You must provide the path to the model file and the vocabulary file to the name finder.
					(There is no need to load the model as an InputStream as in the previous example.) The name finder
					requires a tokenized list of strings as input. The output will be an array of spans.
					<programlisting language="java">
						<![CDATA[
File model = new File("/path/to/model.onnx");
File vocab = new File("/path/to/vocab.txt");
Map<Integer, String> categories = new HashMap<>();
String[] tokens = new String[]{"George", "Washington", "was", "president", "of", "the", "United", "States", "."};
NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, false, getIds2Labels());
Span[] spans = nameFinderDL.find(tokens);]]>
					</programlisting>
					For additional examples, refer to the <code>NameFinderDLEval</code> class.
				</para>
			</section>
	</section>
	</section>
	<section id="tools.namefind.training">
		<title>Name Finder Training</title>
		<para>
			The pre-trained models might not be available for a desired language, can not detect
			important entities or the performance is not good enough outside the news domain.
			These are the typical reason to do custom training of the name finder on a new corpus
			or on a corpus which is extended by private training data taken from the data which should be analyzed.
		</para>
		
		<section id="tools.namefind.training.tool">
		<title>Training Tool</title>
		<para>
			OpenNLP has a command line tool which is used to train the models available from the model
			download page on various corpora.
		</para>
		<para>
			Note that ONNX model support is not available through the command line tool. The models that can be trained
			using the tool are OpenNLP models. ONNX models are trained through deep learning frameworks and then
			utilized by OpenNLP.
		</para>
		<para>
			The data can be converted to the OpenNLP name finder training format. Which is one
            sentence per line. Some other formats are available as well.
			The sentence must be tokenized and contain spans which mark the entities. Documents are separated by
			empty lines which trigger the reset of the adaptive feature generators. A training file can contain
			multiple types. If the training file contains multiple types the created model will also be able to
			detect these multiple types.
		</para>
		<para>
			Sample sentence of the data:
			<screen>
				<![CDATA[
<START:person> Pierre Vinken <END> , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr . <START:person> Vinken <END> is chairman of Elsevier N.V. , the Dutch publishing group .]]>
			 </screen>
			 The training data should contain at least 15000 sentences to create a model which performs well.
			 Usage of the tool:
			<screen>
				<![CDATA[
$ opennlp TokenNameFinderTrainer
Usage: opennlp TokenNameFinderTrainer[.evalita|.ad|.conll03|.bionlp2004|.conll02|.muc6|.ontonotes|.brat] \
[-featuregen featuregenFile] [-nameTypes types] [-sequenceCodec codec] [-factory factoryName] \
[-resources resourcesDir] [-type typeOverride] [-params paramsFile] -lang language \
-model modelFile -data sampleData [-encoding charsetName]

Arguments description:
        -featuregen featuregenFile
                The feature generator descriptor file
        -nameTypes types
                name types to use for training
        -sequenceCodec codec
                sequence codec used to code name spans
        -factory factoryName
                A sub-class of TokenNameFinderFactory
        -resources resourcesDir
                The resources directory
        -type typeOverride
                Overrides the type parameter in the provided samples
        -params paramsFile
                training parameters file.
        -lang language
                language which is being processed.
        -model modelFile
                output model file.
        -data sampleData
                data to be used, usually a file name.
        -encoding charsetName
                encoding for reading and writing text, if absent the system default is used.]]>
			 </screen>
			 It is now assumed that the english person name finder model should be trained from a file
			 called en-ner-person.train which is encoded as UTF-8. The following command will train
			 the name finder and write the model to en-ner-person.bin:
			 <screen>
				<![CDATA[
$ opennlp TokenNameFinderTrainer -model en-ner-person.bin -lang en -data en-ner-person.train -encoding UTF-8]]>
			 </screen>
The example above will train models with a pre-defined feature set. It is also possible to use the -resources parameter to generate features based on external knowledge such as those based on word representation (clustering) features. The external resources must all be placed in a resource directory which is then passed as a parameter. If this option is used it is then required to pass, via the -featuregen parameter, a XML custom feature generator which includes some of the clustering features shipped with the TokenNameFinder. Currently three formats of clustering lexicons are accepted:
			<itemizedlist>
				<listitem>
					<para>Space separated two column file specifying the token and the cluster class as generated by toolkits such as <ulink url="https://code.google.com/p/word2vec/">word2vec</ulink>.</para>
				</listitem>
				<listitem>
					<para>Space separated three column file specifying the token, clustering class and weight as such as <ulink url="https://github.com/ninjin/clark_pos_induction">Clark's clusters</ulink>.</para>
				</listitem>
				<listitem>
					<para>Tab separated three column Brown clusters as generated by <ulink  url="https://github.com/percyliang/brown-cluster">
						Liang's toolkit</ulink>.</para>
				</listitem>
			</itemizedlist>
			 Additionally it is possible to specify the number of iterations,
			 the cutoff and to overwrite all types in the training data with a single type. Finally, the -sequenceCodec parameter allows to specify a BIO (Begin, Inside, Out) or BILOU (Begin, Inside, Last, Out, Unit) encoding to represent the Named Entities. An example of one such command would be as follows:
			 <screen>
			   <![CDATA[
$ opennlp TokenNameFinderTrainer -featuregen brown.xml -sequenceCodec BILOU -resources clusters/ \
-params PerceptronTrainerParams.txt -lang en -model ner-test.bin -data en-train.opennlp -encoding UTF-8]]>
			 </screen>
		</para>
		</section>
		<section id="tools.namefind.training.api">
		<title>Training API</title>
		<para>
			To train the name finder from within an application it is recommended to use the training
			API instead of the command line tool.
			Basically three steps are necessary to train it:
			<itemizedlist>
				<listitem>
					<para>The application must open a sample data stream</para>
				</listitem>
				<listitem>
					<para>Call the NameFinderME.train method</para>
				</listitem>
				<listitem>
					<para>Save the TokenNameFinderModel to a file</para>
				</listitem>
			</itemizedlist>
			The three steps are illustrated by the following sample code:
			<programlisting language="java">
				<![CDATA[
ObjectStream<String> lineStream =
		new PlainTextByLineStream(new MarkableFileInputStreamFactory(new File("en-ner-person.train")), StandardCharsets.UTF_8);

TokenNameFinderModel model;

try (ObjectStream<NameSample> sampleStream = new NameSampleDataStream(lineStream)) {
  model = NameFinderME.train("eng", "person", sampleStream, TrainingParameters.defaultParams(), nameFinderFactory);
}

try (ObjectStream modelOut = new BufferedOutputStream(new FileOutputStream(modelFile)){
  model.serialize(modelOut);
}]]>
			 </programlisting>
		</para>
		</section>

		<section id="tools.namefind.training.featuregen">
		<title>Custom Feature Generation</title>
			<para>
				OpenNLP defines a default feature generation which is used when no custom feature
				generation is specified. Users which want to experiment with the feature generation
				can provide a custom feature generator. Either via API or via an xml descriptor file.
			</para>
			<section id="tools.namefind.training.featuregen.api">
			<title>Feature Generation defined by API</title>
			<para>
				The custom generator must be used for training
				and for detecting the names. If the feature generation during training time and detection
				time is different the name finder might not be able to detect names.
				The following lines show how to construct a custom feature generator
				<programlisting language="java">
					<![CDATA[
AdaptiveFeatureGenerator featureGenerator = new CachedFeatureGenerator(
         new AdaptiveFeatureGenerator[]{
           new WindowFeatureGenerator(new TokenFeatureGenerator(), 2, 2),
           new WindowFeatureGenerator(new TokenClassFeatureGenerator(true), 2, 2),
           new OutcomePriorFeatureGenerator(),
           new PreviousMapFeatureGenerator(),
           new BigramNameFeatureGenerator(),
           new SentenceFeatureGenerator(true, false),
           new BrownTokenFeatureGenerator(BrownCluster dictResource)
           });]]>
				</programlisting>
				which is similar to the default feature generator but with a BrownTokenFeature added.
				The javadoc of the feature generator classes explain what the individual feature generators do.
				To write a custom feature generator please implement the AdaptiveFeatureGenerator interface or
				if it must not be adaptive extend the FeatureGeneratorAdapter.
				The train method which should be used is defined as
				<programlisting language="java">
					<![CDATA[
public static TokenNameFinderModel train(String languageCode, String type,
          ObjectStream<NameSample> samples, TrainingParameters trainParams,
          TokenNameFinderFactory factory) throws IOException]]>
				</programlisting>
				where the TokenNameFinderFactory allows to specify a custom feature generator.
				To detect names the model which was returned from the train method must be passed to the NameFinderME constructor.
				<programlisting language="java">
					<![CDATA[
new NameFinderME(model);]]>
				 </programlisting>
			</para>
			</section>
			<section id="tools.namefind.training.featuregen.xml">
			<title>Feature Generation defined by XML Descriptor</title>
			<para>
			OpenNLP can also use a xml descriptor file to configure the feature generation. The
            descriptor
			file is stored inside the model after training and the feature generators are configured
			correctly when the name finder is instantiated.

			The following sample shows a xml descriptor which contains the default feature generator plus several types of clustering features:
				<programlisting language="xml">
					<![CDATA[
<featureGenerators cache="true" name="nameFinder">
  <generator class="opennlp.tools.util.featuregen.WindowFeatureGeneratorFactory">
    <int name="prevLength">2</int>
    <int name="nextLength">2</int>
    <generator class="opennlp.tools.util.featuregen.TokenClassFeatureGeneratorFactory"/>
  </generator>
  <generator class="opennlp.tools.util.featuregen.WindowFeatureGeneratorFactory">
    <int name="prevLength">2</int>
    <int name="nextLength">2</int>
    <generator class="opennlp.tools.util.featuregen.TokenFeatureGeneratorFactory"/>
  </generator>
  <generator class="opennlp.tools.util.featuregen.DefinitionFeatureGeneratorFactory"/>
  <generator class="opennlp.tools.util.featuregen.PreviousMapFeatureGeneratorFactory"/>
  <generator class="opennlp.tools.util.featuregen.BigramNameFeatureGeneratorFactory"/>
  <generator class="opennlp.tools.util.featuregen.SentenceFeatureGeneratorFactory">
    <bool name="begin">true</bool>
    <bool name="end">false</bool>
  </generator>
  <generator class="opennlp.tools.util.featuregen.WindowFeatureGeneratorFactory">
    <int name="prevLength">2</int>
    <int name="nextLength">2</int>
    <generator class="opennlp.tools.util.featuregen.BrownClusterTokenClassFeatureGeneratorFactory">
      <str name="dict">brownCluster</str>
    </generator>
  </generator>
  <generator class="opennlp.tools.util.featuregen.BrownClusterTokenFeatureGeneratorFactory">
    <str name="dict">brownCluster</str>
  </generator>
  <generator class="opennlp.tools.util.featuregen.BrownClusterBigramFeatureGeneratorFactory">
    <str name="dict">brownCluster</str>
  </generator>
  <generator class="opennlp.tools.util.featuregen.WordClusterFeatureGeneratorFactory">
    <str name="dict">word2vec.cluster</str>
  </generator>
  <generator class="opennlp.tools.util.featuregen.WordClusterFeatureGeneratorFactory">
    <str name="dict">clark.cluster</str>
  </generator>
</featureGenerators>]]>
				 </programlisting>
		    The root element must be featureGenerators, each sub-element adds a feature generator to the configuration.
		    The sample xml contains additional feature generators with respect to the API defined above.
			</para>
			<para>
			The following table shows the supported feature generators (you must specify the Factory's FQDN):
			<table>
			  <title>Feature Generators</title>
			  <tgroup cols="2">
			    <colspec colname="c1"/>
			    <colspec colname="c2"/>
			    <thead>
			      <row>
				<entry>Feature Generator</entry>
				<entry>Parameters</entry>
			      </row>
			    </thead>
			    <tbody>
			      <row>
					<entry>CharacterNgramFeatureGeneratorFactory</entry>
					<entry><emphasis>min</emphasis> and <emphasis>max</emphasis> specify the length of the generated character ngrams</entry>
			      </row>
			      <row>
					<entry>DefinitionFeatureGeneratorFactory</entry>
					<entry>none</entry>
			      </row>
			      <row>
					<entry>DictionaryFeatureGeneratorFactory</entry>
					<entry><emphasis>dict</emphasis> is the key of the dictionary resource to use,
					       and <emphasis>prefix</emphasis> is a feature prefix string</entry>
			      </row>
			      <row>
					<entry>PreviousMapFeatureGeneratorFactory</entry>
					<entry>none</entry>
			      </row>
			      <row>
					<entry>SentenceFeatureGeneratorFactory</entry>
					<entry><emphasis>begin</emphasis> and <emphasis>end</emphasis> to generate begin or end features, both are optional and are boolean values</entry>
			      </row>
			      <row>
					<entry>TokenClassFeatureGeneratorFactory</entry>
					<entry>none</entry>
			      </row>
			      <row>
					<entry>TokenFeatureGeneratorFactory</entry>
					<entry>none</entry>
			      </row>
			      <row>
					<entry>BigramNameFeatureGeneratorFactory</entry>
					<entry>none</entry>
			      </row>
			      <row>
					<entry>TokenPatternFeatureGeneratorFactory</entry>
					<entry>none</entry>
			      </row>
						<row>
							<entry>POSTaggerNameFeatureGeneratorFactory</entry>
							<entry><emphasis>model</emphasis> is the file name of the POS Tagger model to use</entry>
						</row>
			      <row>
				<entry>WordClusterFeatureGeneratorFactory</entry>
				<entry><emphasis>dict</emphasis> is the key of the clustering resource to use</entry>
			      </row>
			      <row>
				<entry>BrownClusterTokenFeatureGeneratorFactory</entry>
				<entry><emphasis>dict</emphasis> is the key of the clustering resource to use</entry>
				</row>
				<row>
				<entry>BrownClusterTokenClassFeatureGeneratorFactory</entry>
				<entry><emphasis>dict</emphasis> is the key of the clustering resource to use</entry>
			      </row>
			      <row>
				<entry>BrownClusterBigramFeatureGeneratorFactory</entry>
				<entry><emphasis>dict</emphasis> is the key of the clustering resource to use</entry>
			      </row>
			      <row>
					<entry>WindowFeatureGeneratorFactory</entry>
					<entry><emphasis>prevLength</emphasis> and <emphasis>nextLength</emphasis> must be integers ans specify the window size</entry>
			      </row>
			    </tbody>
			  </tgroup>
			</table>
			Window feature generator can contain other generators.
			</para>
			</section>
		</section>
	</section>
	<section id="tools.namefind.eval">
		<title>Evaluation</title>
		<para>
		The built in evaluation can measure the named entity recognition performance of the name finder.
		The performance is either measured on a test dataset or via cross validation.
		</para>
		<section id="tools.namefind.eval.tool">
		<title>Evaluation Tool</title>
		<para>
		The following command shows how the tool can be run:
		<screen>
            <![CDATA[
$ opennlp TokenNameFinderEvaluator -model en-ner-person.bin -data en-ner-person.test -encoding UTF-8

Precision: 0.8005071889818507
Recall: 0.7450581122145297
F-Measure: 0.7717879983140168]]>
         </screen>
			 Note: The command line interface does not support cross evaluation in the current version.
		</para>
		</section>
		<section id="tools.namefind.eval.api">
		<title>Evaluation API</title>
		<para>
		The evaluation can be performed on a pre-trained model and a test dataset or via cross validation.
		In the first case the model must be loaded and a NameSample ObjectStream must be created (see code samples above),
		assuming these two objects exist the following code shows how to perform the evaluation:
			<programlisting language="java">
				<![CDATA[
TokenNameFinderEvaluator evaluator = new TokenNameFinderEvaluator(new NameFinderME(model));
evaluator.evaluate(sampleStream);

FMeasure result = evaluator.getFMeasure();

System.out.println(result.toString());]]>
			</programlisting>
			In the cross validation case all the training arguments must be
			provided (see the Training API section above).
			To perform cross validation the ObjectStream must be resettable.
			<programlisting language="java">
				<![CDATA[
InputStreamFactory dataIn = new MarkableFileInputStreamFactory(new File("en-ner-person.train"));
ObjectStream<NameSample> sampleStream = new NameSampleDataStream(
    new PlainTextByLineStream(dataIn, StandardCharsets.UTF_8));
TokenNameFinderCrossValidator evaluator = new TokenNameFinderCrossValidator("eng",
    null, TrainingParameters.defaultParams(), null, (TokenNameFinderEvaluationMonitor) null);
evaluator.evaluate(sampleStream, 10);

FMeasure result = evaluator.getFMeasure();

System.out.println(result.toString());]]>
			</programlisting>
		</para>
		</section>
	</section>
			<section id="tools.namefind.annotation_guides">
		<title>Named Entity Annotation Guidelines</title>
		<para>
			Annotation guidelines define what should be labeled as an entity. To build
			a private corpus it is important to know these guidelines and maybe write a
			custom one.
			Here is a list of publicly available annotation guidelines:
				<itemizedlist>
				<listitem>
					<para>
						<ulink  url="http://cs.nyu.edu/cs/faculty/grishman/NEtask20.book_1.html">
							MUC6
						</ulink>
					</para>
				</listitem>
				<listitem>
					<para>
						<ulink  url="http://acl.ldc.upenn.edu/muc7/ne_task.html">
							MUC7
						</ulink>
					</para>
				</listitem>
				<listitem>
					<para>
						<ulink  url="https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v6.6.pdf">
							ACE
						</ulink>
					</para>
				</listitem>
                <listitem>
                    <para>
                        <ulink  url="https://www.clips.uantwerpen.be/conll2002/ner/">
                            CONLL 2002
                        </ulink>
                    </para>
                </listitem>
				<listitem>
					<para>
						<ulink  url="https://www.clips.uantwerpen.be/conll2003/ner/">
							CONLL 2003
						</ulink>
					</para>
				</listitem>
			</itemizedlist>
		</para>
		</section>
</chapter>
