org.apache.nutch.fetcher

## Class FetchItemQueues

- java.lang.Object
- - org.apache.nutch.fetcher.FetchItemQueues

-     public class FetchItemQueues
      extends Object

  A collection of queues that keeps track of the total number of items, and provides items eligible for fetching from any queue.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `DEFAULT_ID`
    `static String` | `QUEUE_MODE_DOMAIN`
    `static String` | `QUEUE_MODE_HOST`
    `static String` | `QUEUE_MODE_IP`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `FetchItemQueues(Configuration conf)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus` | `addFetchItem(FetchItem it)`
    `org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus` | `addFetchItem(Text url, CrawlDatum datum)`
    `int` | `checkExceptionThreshold(String queueid)` Increment the exception counter of a queue in case of an exception e.g.
    `protected static String` | `checkQueueMode(String queueMode)` Check whether queue mode is valid, fall-back to default mode if not.
    `int` | `checkTimelimit()`
    `void` | `dump()`
    `int` | `emptyQueues()`
    `void` | `finishFetchItem(FetchItem it)`
    `void` | `finishFetchItem(FetchItem it, boolean asap)`
    `FetchItem` | `getFetchItem()`
    `FetchItemQueue` | `getFetchItemQueue(String id)`
    `int` | `getQueueCount()`
    `int` | `getQueueCountMaxExceptions()`
    `int` | `getTotalSize()`
    `boolean` | `redirectIsQueuedRecently(Text redirUrl)`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_ID

          public static final String DEFAULT_ID

      See Also:
        Constant Field Values

    - #### QUEUE_MODE_HOST

          public static final String QUEUE_MODE_HOST

      See Also:
        Constant Field Values

    - #### QUEUE_MODE_DOMAIN

          public static final String QUEUE_MODE_DOMAIN

      See Also:
        Constant Field Values

    - #### QUEUE_MODE_IP

          public static final String QUEUE_MODE_IP

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### FetchItemQueues

          public FetchItemQueues(Configuration conf)

  - ### Method Detail

    - #### checkQueueMode

          protected static String checkQueueMode(String queueMode)

      Check whether queue mode is valid, fall-back to default mode if not.
      Parameters:
        `queueMode` - queue mode to check
      Returns:
        valid queue mode or default

    - #### getTotalSize

          public int getTotalSize()

    - #### getQueueCount

          public int getQueueCount()

    - #### getQueueCountMaxExceptions

          public int getQueueCountMaxExceptions()

    - #### addFetchItem

          public org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus addFetchItem(Text url,
                                                                                     CrawlDatum datum)

    - #### addFetchItem

          public org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus addFetchItem(FetchItem it)

    - #### finishFetchItem

          public void finishFetchItem(FetchItem it)

    - #### finishFetchItem

          public void finishFetchItem(FetchItem it,
                                      boolean asap)

    - #### getFetchItemQueue

          public FetchItemQueue getFetchItemQueue(String id)

    - #### getFetchItem

          public FetchItem getFetchItem()

    - #### checkTimelimit

          public int checkTimelimit()

    - #### emptyQueues

          public int emptyQueues()

    - #### checkExceptionThreshold

          public int checkExceptionThreshold(String queueid)

      Increment the exception counter of a queue in case of an exception e.g. timeout; when higher than a given threshold simply empty the queue.
      Parameters:
        `queueid` -
      Returns:
        number of purged items

    - #### redirectIsQueuedRecently

          public boolean redirectIsQueuedRecently(Text redirUrl)

      Parameters:
        `redirUrl` - redirect target
      Returns:
        true if redirects are deduplicated and redirUrl has been queued recently

    - #### dump

          public void dump()

Copyright © 2021 The Apache Software Foundation