org.apache.storm.hdfs.bolt

Class AbstractHdfsBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

org.apache.storm.hdfs.bolt.AbstractHdfsBolt

All Implemented Interfaces:

Serializable, IBolt, IComponent, IRichBolt

Direct Known Subclasses:

AvroGenericRecordBolt, HdfsBolt, SequenceFileBolt

public abstract class AbstractHdfsBolt
extends BaseRichBolt

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected OutputCollector	collector
protected String	configKey
protected FileNameFormat	fileNameFormat
protected Integer	fileRetryCount
protected org.apache.hadoop.fs.FileSystem	fs
protected String	fsUrl
protected org.apache.hadoop.conf.Configuration	hdfsConfig
protected long	offset
protected int	rotation
protected ArrayList<RotationAction>	rotationActions
protected FileRotationPolicy	rotationPolicy
protected Timer	rotationTimer
protected SyncPolicy	syncPolicy
protected Integer	tickTupleInterval
protected Object	writeLock

Constructor Summary

Constructors

Constructor and Description
AbstractHdfsBolt()

Method Summary

Modifier and Type	Method and Description
void	cleanup()
protected abstract void	closeOutputFile()
protected abstract org.apache.hadoop.fs.Path	createOutputFile()
void	declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer)
protected abstract void	doPrepare(Map conf, TopologyContext topologyContext, OutputCollector collector)
void	execute(Tuple tuple)
Map<String,Object>	getComponentConfiguration()
void	prepare(Map conf, TopologyContext topologyContext, OutputCollector collector) Marked as final to prevent override.
protected void	rotateOutputFile()
protected abstract void	syncTuples() Make the best effort to sync written data to the underlying file system.
protected abstract void	writeTuple(Tuple tuple) writes a tuple to the underlying filesystem but makes no guarantees about syncing data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rotationActions

protected ArrayList<RotationAction> rotationActions

collector

protected OutputCollector collector

fs

protected transient org.apache.hadoop.fs.FileSystem fs

syncPolicy

protected SyncPolicy syncPolicy

rotationPolicy

protected FileRotationPolicy rotationPolicy

fileNameFormat

protected FileNameFormat fileNameFormat

rotation

protected int rotation

fsUrl

protected String fsUrl

configKey

protected String configKey

writeLock

protected transient Object writeLock

rotationTimer

protected transient Timer rotationTimer

offset

protected long offset

fileRetryCount

protected Integer fileRetryCount

tickTupleInterval

protected Integer tickTupleInterval

hdfsConfig

protected transient org.apache.hadoop.conf.Configuration hdfsConfig

Constructor Detail

AbstractHdfsBolt

public AbstractHdfsBolt()

Method Detail

rotateOutputFile

protected void rotateOutputFile()
                         throws IOException

Throws:

IOException

prepare

public final void prepare(Map conf,
                          TopologyContext topologyContext,
                          OutputCollector collector)

Marked as final to prevent override. Subclasses should implement the doPrepare() method.

Parameters:

conf -

topologyContext -

collector -

execute

public final void execute(Tuple tuple)

getComponentConfiguration

public Map<String,Object> getComponentConfiguration()

Specified by:

getComponentConfiguration in interface IComponent

Overrides:

getComponentConfiguration in class BaseComponent

declareOutputFields

public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer)

cleanup

public void cleanup()

Specified by:

cleanup in interface IBolt

Overrides:

cleanup in class BaseRichBolt

writeTuple

protected abstract void writeTuple(Tuple tuple)
                            throws IOException

writes a tuple to the underlying filesystem but makes no guarantees about syncing data.

this.offset is also updated to reflect additional data written

Parameters:

tuple -

Throws:

IOException

syncTuples

protected abstract void syncTuples()
                            throws IOException

Make the best effort to sync written data to the underlying file system. Concrete classes should very clearly state the file state that sync guarantees. For example, HdfsBolt can make a much stronger guarantee than SequenceFileBolt.

Throws:

IOException

closeOutputFile

protected abstract void closeOutputFile()
                                 throws IOException

Throws:

IOException

createOutputFile

protected abstract org.apache.hadoop.fs.Path createOutputFile()
                                                       throws IOException

Throws:

IOException

doPrepare

protected abstract void doPrepare(Map conf,
                                  TopologyContext topologyContext,
                                  OutputCollector collector)
                           throws IOException

Throws:

IOException