org.apache.nutch.urlfilter.ignoreexempt

Class ExemptionUrlFilter

java.lang.Object

org.apache.nutch.urlfilter.api.RegexURLFilterBase

org.apache.nutch.urlfilter.regex.RegexURLFilter

org.apache.nutch.urlfilter.ignoreexempt.ExemptionUrlFilter

All Implemented Interfaces:

Configurable, URLExemptionFilter, URLFilter, Pluggable

public class ExemptionUrlFilter
extends RegexURLFilter
implements URLExemptionFilter

This implementation of URLExemptionFilter uses regex configuration to check if URL is eligible for exemption from 'db.ignore.external'. When this filter is enabled, the external urls will be checked against configured sequence of regex rules.

The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be overridden using the property "db.ignore.external.exemptions.file" in ./conf/nutch-*.xml

The exemption rules are specified in plain text file where each line is a rule. The format is same same as `regex-urlfilter.txt`. Each non-comment, non-blank line contains a regular expression prefixed by '+' or '-'. The first matching pattern in the file determines whether a URL is exempted or ignored. If no pattern matches, the URL is ignored.

Since:

Feb 10, 2016

Version:

1

See Also:

URLExemptionFilter, RegexURLFilter

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE

Fields inherited from class org.apache.nutch.urlfilter.regex.RegexURLFilter

URLFILTER_REGEX_FILE, URLFILTER_REGEX_RULES

Fields inherited from class org.apache.nutch.urlfilter.api.RegexURLFilterBase

hasHostDomainRules

Fields inherited from interface org.apache.nutch.net.URLExemptionFilter

X_POINT_ID

Fields inherited from interface org.apache.nutch.net.URLFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
ExemptionUrlFilter()

Method Summary

Modifier and Type	Method and Description
boolean	filter(java.lang.String fromUrl, java.lang.String toUrl) Checks if toUrl is exempted when the ignore external is enabled
java.util.List<java.util.regex.Pattern>	getExemptions()
protected java.io.Reader	getRulesReader(Configuration conf) Gets reader for regex rules
static void	main(java.lang.String[] args)

Methods inherited from class org.apache.nutch.urlfilter.regex.RegexURLFilter

createRule, createRule

Methods inherited from class org.apache.nutch.urlfilter.api.RegexURLFilterBase

filter, getConf, main, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE

public static final java.lang.String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE

See Also:

Constant Field Values

Constructor Detail

ExemptionUrlFilter

public ExemptionUrlFilter()

Method Detail

getExemptions

public java.util.List<java.util.regex.Pattern> getExemptions()

filter

public boolean filter(java.lang.String fromUrl,
                      java.lang.String toUrl)

Description copied from interface: URLExemptionFilter

Checks if toUrl is exempted when the ignore external is enabled

Specified by:

filter in interface URLExemptionFilter

Parameters:

fromUrl - : the source url which generated the outlink

toUrl - : the destination url which needs to be checked for exemption

Returns:

true when toUrl is exempted from dbIgnore

getRulesReader

protected java.io.Reader getRulesReader(Configuration conf)
                                 throws java.io.IOException

Gets reader for regex rules

Overrides:

getRulesReader in class RegexURLFilter

Parameters:

conf - is the current configuration.

Returns:

the name of the resource containing the rules to use.

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)