org.apache.any23.plugin.crawler

## Class SiteCrawler

- java.lang.Object
- - org.apache.any23.plugin.crawler.SiteCrawler

-     public class SiteCrawler
      extends Object

  A basic site crawler to extract semantic content of small/medium size sites.
  Author:
    Michele Mostarda (mostarda@fbk.eu)

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `DEFAULT_NUM_OF_CRAWLERS` Default number of crawler instances.
    `static String` | `DEFAULT_PAGE_FILTER_RE`
    `static Class<? extends edu.uci.ics.crawler4j.crawler.WebCrawler>` | `DEFAULT_WEB_CRAWLER` Default crawler implementation.
    `Pattern` | `defaultFilters` Default filter applied to skip contents.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `SiteCrawler(File storageFolder)` Constructor.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `addListener(CrawlerListener listener)` Registers a `CrawlerListener` to this crawler.
    `int` | `getMaxDepth()`
    `int` | `getMaxPages()`
    `int` | `getNumOfCrawlers()`
    `int` | `getPolitenessDelay()`
    `Class<? extends edu.uci.ics.crawler4j.crawler.WebCrawler>` | `getWebCrawler()`
    `void` | `removeListener(CrawlerListener listener)` Deregisters a `CrawlerListener` from this crawler.
    `void` | `setMaxDepth(int maxDepth)` Sets the maximum depth.
    `void` | `setMaxPages(int maxPages)` Sets the maximum collected pages.
    `void` | `setNumOfCrawlers(int n)` Sets the number of crawler instances.
    `void` | `setPolitenessDelay(int millis)` Sets the politeness delay.
    `void` | `setWebCrawler(Class<? extends edu.uci.ics.crawler4j.crawler.WebCrawler> c)` Sets the actual crawler class.
    `void` | `start(URL seed, boolean wait)` Starts the crawler process with the `defaultFilters`.
    `void` | `start(URL seed, Pattern filters, boolean wait)` Starts the crawling process.
    `void` | `stop()` Interrupts the crawler process if started with `wait` flag == `false`.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_PAGE_FILTER_RE

          public static final String DEFAULT_PAGE_FILTER_RE

      See Also:
        Constant Field Values

    - #### DEFAULT_NUM_OF_CRAWLERS

          public static final int DEFAULT_NUM_OF_CRAWLERS

      Default number of crawler instances.
      See Also:
        Constant Field Values

    - #### DEFAULT_WEB_CRAWLER

          public static final Class<? extends edu.uci.ics.crawler4j.crawler.WebCrawler> DEFAULT_WEB_CRAWLER

      Default crawler implementation.

    - #### defaultFilters

          public final Pattern defaultFilters

      Default filter applied to skip contents.

  - ### Constructor Detail

    - #### SiteCrawler

          public SiteCrawler(File storageFolder)

      Constructor.
      Parameters:
        `storageFolder` - location used to store the temporary data structures used by the crawler.

  - ### Method Detail

    - #### getNumOfCrawlers

          public int getNumOfCrawlers()

      Returns:
        number of crawler instances.

    - #### setNumOfCrawlers

          public void setNumOfCrawlers(int n)

      Sets the number of crawler instances.
      Parameters:
        `n` - an integer >= 0.

    - #### getWebCrawler

          public Class<? extends edu.uci.ics.crawler4j.crawler.WebCrawler> getWebCrawler()

    - #### setWebCrawler

          public void setWebCrawler(Class<? extends edu.uci.ics.crawler4j.crawler.WebCrawler> c)

      Sets the actual crawler class.
      Parameters:
        `c` - a not `class`.

    - #### getMaxDepth

          public int getMaxDepth()

      Returns:
        the max allowed crawl depth, `-1` means no limit.

    - #### setMaxDepth

          public void setMaxDepth(int maxDepth)

      Sets the maximum depth.
      Parameters:
        `maxDepth` - maximum allowed depth. `-1` means no limit.

    - #### getMaxPages

          public int getMaxPages()

      Returns:
        max number of allowed pages.

    - #### setMaxPages

          public void setMaxPages(int maxPages)

      Sets the maximum collected pages.
      Parameters:
        `maxPages` - maximum allowed pages. `-1` means no limit.

    - #### getPolitenessDelay

          public int getPolitenessDelay()

      Returns:
        the politeness delay in milliseconds.

    - #### setPolitenessDelay

          public void setPolitenessDelay(int millis)

      Sets the politeness delay.
      Parameters:
        `millis` - delay in milliseconds.

    - #### addListener

          public void addListener(CrawlerListener listener)

      Registers a `CrawlerListener` to this crawler.
      Parameters:
        `listener` -

    - #### removeListener

          public void removeListener(CrawlerListener listener)

      Deregisters a `CrawlerListener` from this crawler.
      Parameters:
        `listener` -

    - #### start

          public void start(URL seed,
                            Pattern filters,
                            boolean wait)
                     throws Exception

      Starts the crawling process.
      Parameters:
        `seed` - the starting URL for the crawler process.
        `filters` - filters to be applied to the crawler process. Can be `null`.
        `wait` - if `true` the process will wait for the crawler termination.
      Throws:
        `Exception`

    - #### start

          public void start(URL seed,
                            boolean wait)
                     throws Exception

      Starts the crawler process with the `defaultFilters`.
      Parameters:
        `seed` - the starting URL for the crawler process.
        `wait` - if `true` the process will wait for the crawler termination.
      Throws:
        `Exception`

    - #### stop

          public void stop()

      Interrupts the crawler process if started with `wait` flag == `false`.

Copyright © 2010–2019 The Apache Software Foundation. All rights reserved.