com.amazonaws.services.elasticmapreduce.util

クラス StreamingStep

java.lang.Object

com.amazonaws.services.elasticmapreduce.util.StreamingStep

public class StreamingStep
extends java.lang.Object

Class that makes it easy to define Hadoop Streaming steps.

See also: Hadoop Streaming

 AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
 AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials);

 HadoopJarStepConfig config = new StreamingStep()
     .withInputs("s3://elasticmapreduce/samples/wordcount/input")
     .withOutput("s3://my-bucket/output/")
     .withMapper("s3://elasticmapreduce/samples/wordcount/wordSplitter.py")
     .withReducer("aggregate")
     .toHadoopJarStepConfig();

 StepConfig wordCount = new StepConfig()
     .withName("Word Count")
     .withActionOnFailure("TERMINATE_JOB_FLOW")
     .withHadoopJarStep(config);

 RunJobFlowRequest request = new RunJobFlowRequest()
     .withName("Word Count")
     .withSteps(wordCount)
     .withLogUri("s3://log-bucket/")
     .withInstances(new JobFlowInstancesConfig()
         .withEc2KeyName("keypairt")
         .withHadoopVersion("0.20")
         .withInstanceCount(5)
         .withKeepJobFlowAliveWhenNoSteps(true)
         .withMasterInstanceType("m1.small")
         .withSlaveInstanceType("m1.small"));

 RunJobFlowResult result = emr.runJobFlow(request);
 

コンストラクタの概要

コンストラクタ

コンストラクタと説明
StreamingStep() Creates a new default StreamingStep.

メソッドの概要

メソッド

修飾子とタイプ	メソッドと説明
java.util.Map<java.lang.String,java.lang.String>	getHadoopConfig() Get the Hadoop config overrides (-D values).
java.util.List<java.lang.String>	getInputs() Get list of step input paths.
java.lang.String	getMapper() Get the mapper.
java.lang.String	getOutput() Get output path.
java.lang.String	getReducer() Get the reducer
void	setHadoopConfig(java.util.Map<java.lang.String,java.lang.String> hadoopConfig) Set the Hadoop config overrides (-D values).
void	setInputs(java.util.Collection<java.lang.String> inputs) Set the list of step input paths.
void	setMapper(java.lang.String mapper) Set the mapper.
void	setOutput(java.lang.String output) Set the output path for this step.
void	setReducer(java.lang.String reducer) Set the reducer
HadoopJarStepConfig	toHadoopJarStepConfig() Creates the final HadoopJarStepConfig once you are done configuring the step.
StreamingStep	withHadoopConfig(java.lang.String key, java.lang.String value) Add a Hadoop config override (-D value).
StreamingStep	withInputs(java.lang.String... inputs) Add more input paths to this step.
StreamingStep	withMapper(java.lang.String mapper) Set the mapper
StreamingStep	withOutput(java.lang.String output) Set the output path for this step.
StreamingStep	withReducer(java.lang.String reducer) Set the reducer

クラスから継承されたメソッド java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

コンストラクタの詳細

StreamingStep

public StreamingStep()

Creates a new default StreamingStep.

メソッドの詳細

getInputs

public java.util.List<java.lang.String> getInputs()

Get list of step input paths.

戻り値:

List of step inputs

setInputs

public void setInputs(java.util.Collection<java.lang.String> inputs)

Set the list of step input paths.

パラメータ:

inputs - List of step inputs.

withInputs

public StreamingStep withInputs(java.lang.String... inputs)

Add more input paths to this step.

パラメータ:

inputs - A list of inputs to this step.

戻り値:

A reference to this updated object so that method calls can be chained together.

getOutput

public java.lang.String getOutput()

Get output path.

戻り値:

Output path.

setOutput

public void setOutput(java.lang.String output)

Set the output path for this step.

パラメータ:

output - Output path.

withOutput

public StreamingStep withOutput(java.lang.String output)

Set the output path for this step.

パラメータ:

output - Output path

戻り値:

A reference to this updated object so that method calls can be chained together.

getMapper

public java.lang.String getMapper()

Get the mapper.

戻り値:

Mapper.

setMapper

public void setMapper(java.lang.String mapper)

Set the mapper.

パラメータ:

mapper - Mapper

withMapper

public StreamingStep withMapper(java.lang.String mapper)

Set the mapper

パラメータ:

mapper - Mapper

戻り値:

A reference to this updated object so that method calls can be chained together.

getReducer

public java.lang.String getReducer()

Get the reducer

戻り値:

Reducer

setReducer

public void setReducer(java.lang.String reducer)

Set the reducer

パラメータ:

reducer - Reducer

withReducer

public StreamingStep withReducer(java.lang.String reducer)

Set the reducer

パラメータ:

reducer - Reducer

戻り値:

A reference to this updated object so that method calls can be chained together.

getHadoopConfig

public java.util.Map<java.lang.String,java.lang.String> getHadoopConfig()

Get the Hadoop config overrides (-D values).

戻り値:

Hadoop config.

setHadoopConfig

public void setHadoopConfig(java.util.Map<java.lang.String,java.lang.String> hadoopConfig)

Set the Hadoop config overrides (-D values).

パラメータ:

hadoopConfig - Hadoop config.

withHadoopConfig

public StreamingStep withHadoopConfig(java.lang.String key,
                             java.lang.String value)

Add a Hadoop config override (-D value).

パラメータ:

key - Hadoop configuration key.

value - Configuration value.

戻り値:

A reference to this updated object so that method calls can be chained together.

toHadoopJarStepConfig

public HadoopJarStepConfig toHadoopJarStepConfig()

Creates the final HadoopJarStepConfig once you are done configuring the step. You can use this as you would any other HadoopJarStepConfig.

戻り値:

HadoopJarStepConfig representing this streaming step.