DBILITY

hadoop 2.6.4 window 10 pseudo distribution mode 설치 본문

bigdata/hadoop

hadoop 2.6.4 window 10 pseudo distribution mode 설치

DBILITY 2017. 4. 1. 09:33
반응형
  1. hadoop 2.6.4 binary 다운로드
    http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz
  2. winutils 다운로드
    https://github.com/steveloughran/winutils 
    winutils-master.zip
    다운로드
  3. eclipse hadoop 2.x plugin 다운로드
    https://github.com/winghc/hadoop2x-eclipse-plugin
  4. eclipse > dorpins에 hadoop-eclipse-plugin-2.6.0.jar 복사
  5. HADOOP_HOME 환경변수 설정 ( E:/hadoop-2.6.4  )
  6. hadoop-2.6.4.tar.gz 압축해제 후 %HADOOP_HOME% 에 복사
  7. winutils-master.zip 압축해제 후 hadoop-2.6.4/bin 파일을 %HADOOP_HOME%/bin에 복사
  8. 디렉토리 생성
    %HADOOP_HOME%/data/namenode,
    %HADOOP_HOME%/data/datanode,
    %HADOOP_HOME%/temp,
    %HADOOP_HOME%/userlog
  9. %HADOOP_HOME%/etc/hadoop 환경설정
    core-site.xml
    <configuration>
        <property>
            <name>hadoop.tmp.dir</name>
            <value>\hadoop-2.6.4\temp\</value>
        </property>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://0.0.0.0:50071</value>
        </property>
    </configuration>
    hdfs-site.xml
    <configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>file:/hadoop-2.6.4/data/namenode</value>
            <final>true</final>
        </property>
        <property>
            <name>dfs.datanode.data.dir</name>
            <value>file:/hadoop-2.6.4/data/datanode</value>
            <final>true</final>
        </property>
    </configuration>
    mapred-site.xml
    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
        <property>
            <name>mapred.job.tracker</name>
            <value>0.0.0.0:9001</value>
        </property>
        <property>
            <name>mapreduce.application.classpath</name>
            <value>
                /hadoop-2.6.4/share/hadoop/mapreduce/*,
                /hadoop-2.6.4/share/hadoop/mapreduce/lib/*,
                /hadoop-2.6.4/share/hadoop/common/*,
                /hadoop-2.6.4/share/hadoop/common/lib/*,
                /hadoop-2.6.4/share/hadoop/yarn/*,
                /hadoop-2.6.4/share/hadoop/yarn/lib/*,
                /hadoop-2.6.4/share/hadoop/hdfs/*,
                /hadoop-2.6.4/share/hadoop/hdfs/lib/*
            </value>
        </property>
    </configuration>
    yarn-site.xml
    <configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
        <property>
            <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
            <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
            <name>yarn.nodemanager.log-dirs</name>
            <value>\hadoop-2.6.4\userlog</value>
            <final>true</final>
        </property>
        <property>
            <name>yarn.nodemanager.local-dirs</name>
            <value>\hadoop-2.6.4\temp\nm-localdir</value>
        </property>
        <property>
            <name>yarn.nodemanager.delete.debug-delay-sec</name>
            <value>600</value>
        </property>
        <property>
            <name>yarn.application.classpath</name>
            <value>
                /hadoop-2.6.4/,
                /hadoop-2.6.4/share/hadoop/common/*,
                /hadoop-2.6.4/share/hadoop/common/lib/*,
                /hadoop-2.6.4/share/hadoop/hdfs/*,
                /hadoop-2.6.4/share/hadoop/hdfs/lib/*,
                /hadoop-2.6.4/share/hadoop/mapreduce/*,
                /hadoop-2.6.4/share/hadoop/mapreduce/lib/*,
                /hadoop-2.6.4/share/hadoop/yarn/*,
                /hadoop-2.6.4/share/hadoop/yarn/lib/*
            </value>
        </property>
    </configuration>
    hadoop-env.cmd
    set JAVA_HOME=E:\hadoop-2.6.4\jre
    set HADOOP_HOME=E:\hadoop-2.6.4
    set PATH=%PATH%;%JAVA_HOME%\bin;E:\hadoop-2.6.4\bin;E:\hadoop-2.6.4\sbin;
  10. java 1.8.x jre %HADOOP_HOME% 복사
  11. 관리자모드 콘솔에서 %HADOOP_HOME%/bin이동 ,hdfs namenode -format
  12. %HADOOP_HOME%\sbin 이동 start-dfs.cmd , start-yarn.cmd 실행
  13. Eclipse Run On Hadoop 실행시
    Run Configuration → Classpath → User Entries → Advanced Options → Add External Folder → %HADOOP_HOME%/etc/hadoop
    job.setJar("jar full path");

※심볼릭 링크 관련 오류시 gpedit.msc → 컴퓨터구성 → Windows 설정 → 보안 설정 → 로컬 정책 → 사용자 권한 할당 → 심볼 링크 만들기 → 로그인유저 추가

반응형

'bigdata > hadoop' 카테고리의 다른 글

hadoop 2.7.5 compile  (0) 2018.04.11
hadoop job control  (0) 2017.04.16
hadoop 2.x winutils  (0) 2017.03.27
hadoop 2.6.x eclipse plugin  (0) 2017.03.23
hadoop secondary sort 4 ( 보조정렬 실습 )  (0) 2017.03.12
Comments