DBILITY

hadoop job control 본문

bigdata/hadoop

hadoop job control

DBILITY 2017. 4. 16. 12:17
반응형
  1. 순차적으로 실행
    JobConf job1=new JobConf(getConf());
    ...
    JobClient.run(job1);
    
    JobConf job2=new JobConf(getConf());
    ...
    JobClient.run(job2);
    
    JobConf job3=new JobConf(getConf());
    ...
    JobClient.run(job3);
  2. 의존관계의 연결
    JobConf job1=new JobConf(getConf());
    ...
    JobConf job2=new JobConf(getConf());
    ...
    JobConf job3=new JobConf(getConf());
    ...
    JobConf job4=new JobConf(getConf());
    ...
    JobControl con=new JobControl("DependancyJob");
    org.apache.hadoop.mapred.jobcontrol.Job wrapperJob1 =new org.apache.hadoop.mapred.jobcontrol.Job(job1);		
    con.addJob(wrapperJob1);
    
    org.apache.hadoop.mapred.jobcontrol.Job wrapperJob2 =new org.apache.hadoop.mapred.jobcontrol.Job(job2);		
    con.addJob(wrapperJob2);
    
    org.apache.hadoop.mapred.jobcontrol.Job wrapperJob3 =new org.apache.hadoop.mapred.jobcontrol.Job(job3);		
    con.addJob(wrapperJob3);
    wrapperJob3.addDependingJob(wrapperJob1);
    wrapperJob3.addDependingJob(wrapperJob2);			
    
    org.apache.hadoop.mapred.jobcontrol.Job wrapperJob4 =new org.apache.hadoop.mapred.jobcontrol.Job(job4);		
    con.addJob(wrapperJob4);
    wrapperJob4.addDependingJob(wrapperJob3);
    
    con.run();
  3. 여러개의 Mapper와 한개의 Reducer
    JobConf tmConf = new JobConf(false);
    ChainMapper.addMapper(conf, MapperClass, LongWritable.class, Text.class, Text.class, IntWritable.class, true, tmConf);
    
    JobConf umConf = new JobConf(false);
    ChainMapper.addMapper(conf, MapperClass, Text.class, IntWritable.class, Text.class, IntWritable.class, true, umConf);
     
    JobConf wrConf = new JobConf(false);
    ChainReducer.setReducer(conf, ReducerClass, Text.class, IntWritable.class, Text.class, IntWritable.class, true, wrConf);
     
     /*JobClient client = new JobClient(conf);
     RunningJob job = client.submitJob(conf);
     client.monitorAndPrintJob(conf, job);*/
     
     RunningJob job = JobClient.runJob(conf);

 

반응형
Comments