Notice
Recent Posts
Recent Comments
Link
일 | 월 | 화 | 수 | 목 | 금 | 토 |
---|---|---|---|---|---|---|
1 | 2 | 3 | 4 | 5 | 6 | 7 |
8 | 9 | 10 | 11 | 12 | 13 | 14 |
15 | 16 | 17 | 18 | 19 | 20 | 21 |
22 | 23 | 24 | 25 | 26 | 27 | 28 |
29 | 30 |
Tags
- Spring
- mapreduce
- Python
- R
- MSSQL
- 보조정렬
- table
- JavaScript
- GIT
- vaadin
- SPC
- Eclipse
- mybatis
- react
- Java
- Sqoop
- es6
- plugin
- Android
- IntelliJ
- Kotlin
- SSL
- xPlatform
- window
- tomcat
- SQL
- hadoop
- Express
- NPM
- 공정능력
Archives
- Today
- Total
DBILITY
hadoop 2.6.4 window 10 pseudo distribution mode 설치 본문
반응형
- hadoop 2.6.4 binary 다운로드
http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz - winutils 다운로드
https://github.com/steveloughran/winutilswinutils-master.zip다운로드 - eclipse hadoop 2.x plugin 다운로드
https://github.com/winghc/hadoop2x-eclipse-plugin - eclipse > dorpins에 hadoop-eclipse-plugin-2.6.0.jar 복사
- HADOOP_HOME 환경변수 설정 ( E:/hadoop-2.6.4 )
- hadoop-2.6.4.tar.gz 압축해제 후 %HADOOP_HOME% 에 복사
- winutils-master.zip 압축해제 후 hadoop-2.6.4/bin 파일을 %HADOOP_HOME%/bin에 복사
- 디렉토리 생성
%HADOOP_HOME%/data/namenode,
%HADOOP_HOME%/data/datanode,
%HADOOP_HOME%/temp,
%HADOOP_HOME%/userlog - %HADOOP_HOME%/etc/hadoop 환경설정
core-site.xml
hdfs-site.xml<configuration> <property> <name>hadoop.tmp.dir</name> <value>\hadoop-2.6.4\temp\</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://0.0.0.0:50071</value> </property> </configuration>
mapred-site.xml<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/hadoop-2.6.4/data/namenode</value> <final>true</final> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/hadoop-2.6.4/data/datanode</value> <final>true</final> </property> </configuration>
yarn-site.xml<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapred.job.tracker</name> <value>0.0.0.0:9001</value> </property> <property> <name>mapreduce.application.classpath</name> <value> /hadoop-2.6.4/share/hadoop/mapreduce/*, /hadoop-2.6.4/share/hadoop/mapreduce/lib/*, /hadoop-2.6.4/share/hadoop/common/*, /hadoop-2.6.4/share/hadoop/common/lib/*, /hadoop-2.6.4/share/hadoop/yarn/*, /hadoop-2.6.4/share/hadoop/yarn/lib/*, /hadoop-2.6.4/share/hadoop/hdfs/*, /hadoop-2.6.4/share/hadoop/hdfs/lib/* </value> </property> </configuration>
hadoop-env.cmd<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>\hadoop-2.6.4\userlog</value> <final>true</final> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>\hadoop-2.6.4\temp\nm-localdir</value> </property> <property> <name>yarn.nodemanager.delete.debug-delay-sec</name> <value>600</value> </property> <property> <name>yarn.application.classpath</name> <value> /hadoop-2.6.4/, /hadoop-2.6.4/share/hadoop/common/*, /hadoop-2.6.4/share/hadoop/common/lib/*, /hadoop-2.6.4/share/hadoop/hdfs/*, /hadoop-2.6.4/share/hadoop/hdfs/lib/*, /hadoop-2.6.4/share/hadoop/mapreduce/*, /hadoop-2.6.4/share/hadoop/mapreduce/lib/*, /hadoop-2.6.4/share/hadoop/yarn/*, /hadoop-2.6.4/share/hadoop/yarn/lib/* </value> </property> </configuration>
set JAVA_HOME=E:\hadoop-2.6.4\jre set HADOOP_HOME=E:\hadoop-2.6.4 set PATH=%PATH%;%JAVA_HOME%\bin;E:\hadoop-2.6.4\bin;E:\hadoop-2.6.4\sbin;
- java 1.8.x jre %HADOOP_HOME% 복사
- 관리자모드 콘솔에서 %HADOOP_HOME%/bin이동 ,hdfs namenode -format
- %HADOOP_HOME%\sbin 이동 start-dfs.cmd , start-yarn.cmd 실행
- Eclipse Run On Hadoop 실행시
Run Configuration → Classpath → User Entries → Advanced Options → Add External Folder → %HADOOP_HOME%/etc/hadoop
job.setJar("jar full path");
※심볼릭 링크 관련 오류시 gpedit.msc → 컴퓨터구성 → Windows 설정 → 보안 설정 → 로컬 정책 → 사용자 권한 할당 → 심볼 링크 만들기 → 로그인유저 추가
반응형
'bigdata > hadoop' 카테고리의 다른 글
hadoop 2.7.5 compile (0) | 2018.04.11 |
---|---|
hadoop job control (0) | 2017.04.16 |
hadoop 2.x winutils (0) | 2017.03.27 |
hadoop 2.6.x eclipse plugin (0) | 2017.03.23 |
hadoop secondary sort 4 ( 보조정렬 실습 ) (0) | 2017.03.12 |
Comments