apache tajo HA 설치

소스컴파일 후 공식사이트 매뉴얼에 따라 진행,

사내 또는 사설 mvn repository가 있다면 pom에 추가하면, 빌드시간이 좀 줄어든다.

5대의 서버에 각각 user는 tajo, datagroup에 추가하고,

ssh인증키로 접근 가능하게도 해야한다.

active tajoMaster를 kill하고 보니 backup tajoMaster가 active로 바뀌어 있었다.

죽였던 tajoMaster는 tajo-daemon.sh start master 시작됨.

derby를 사용하니 catalog공유가 안된다. mariadb를 설치한다.

설치 후 테스트 중

갑자기 Backup master가 active, Active master도 active 상태가 되었다.

일단 1) tajo.worker.resource.disks와 2) tajo.master.ha.monitor.interval 주석처리하니 괜찮다.

1)은 tajo.worker.resource.dfs-dir-aware와 관계가 있을까?

　　 tajo.worker.resource.dfs-dir-aware가 설정되면,tajo.worker.resource.disks는 무시된다고 함.

2)는 tajo.master.ha.monitor.interval은 초단위라 적혀 있는데,설마 milisecond로 적어야 하나?

　　 TajoConf.java 소스를 보니 예상대로 요녀석이 문제였음.5로 한게 문제..default로 해 놓으니 잘됨.

　　 시간이 너무 짧아 죽은거로 판단되었나 보다^^;

tajo.worker.resource.cpu-cores를 설정하지 않으면,실제 core수를 사용한다고도..

 

worker호스트들에 tail -f tajo-tajo-worker-hostname.log 상태에서 web-UI execute query 실행 결과 worker전체가 열심히 일을 함.

 

[root@big-master target]# cp -rf tajo-0.11.3 /bigdata
[root@big-master target]# ln -s /bigdata/tajo-0.11.3/ /tajo
[root@big-master target]# chown -R tajo.tajo /bigdata/tajo-0.11.3/
[root@big-master target]# vi etc/profile
export TAJO_HOME=/tajo
export PATH=$PATH:$TAJO_HOME/bin
:wq!
[root@big-master target]# rsync -az /etc/profile big-slave1:/etc/profile
[root@big-master target]# rsync -az /etc/profile big-slave2:/etc/profile
[root@big-master target]# rsync -az /etc/profile big-slave3:/etc/profile
[root@big-master target]# rsync -az /etc/profile big-slave4:/etc/profile

[root@big-master target]# su - tajo
[tajo@big-master ~]$ cd /tajo/conf/
[tajo@big-master conf]$ vi tajo-env.sh
export HADOOP_HOME=/hadoop
export JAVA_HOME=/jdk
export TAJO_CLASSPATH=/tajo/lib/mariadb-java-client-2.2.3.jar
export TAJO_MASTER_HEAPSIZE=1000
export TAJO_WORKER_HEAPSIZE=5000 # default 5G, 15G는 있어야 한다고..
export TAJO_QUERYMASTER_HEAPSIZE=1000
export TAJO_WORKER_OPTS='-XX:+UseParallelOldGC'
export TAJO_LOG_DIR=${TAJO_HOME}/logs
export TAJO_PID_DIR=${TAJO_HOME}/tajo/pids
:wq!

[tajo@big-master conf]# cp tajo-site.xml.template tajo-site.xml
[tajo@big-master conf]$ vi tajo-site.xml
<property>
  <name>tajo.rootdir</name>
  <value>hdfs://hadoop-cluster/tajo</value>
  <description>Base directory including system directories.</description>
</property>

<property>
  <name>tajo.master.umbilical-rpc.address</name>
  <value>big-master:26001</value>
  <description>TajoMaster binding address between master and workers.</description>
</property>

<property>
  <name>tajo.master.client-rpc.address</name>
  <value>big-master:26002</value>
  <description>TajoMaster binding address between master and clients.</description>
</property>

<property>
  <name>tajo.resource-tracker.rpc.address</name>
  <value>big-master:26003</value>
  <description>TajoMaster binding address between master and workers.</description>
</property>

<property>
  <name>tajo.catalog.client-rpc.address</name>
  <value>big-master:26005</value>
  <description>CatalogServer binding address between catalog server and workers.</description>
</property>

<property>
  <name>tajo.master.ha.enable</name>
  <value>true</value>
</property>

<!--<property>
  <name>tajo.master.ha.monitor.interval</name>
  <value>5</value>
</property>
-->

<!-- Worker Settings -->

<property>
  <name>tajo.worker.resource.cpu-cores</name>
  <value>2</value>
  <description>Number of CPU cores</description>
</property>

<property>
  <name>tajo.task.resource.min.memory-mb</name>
  <value>1000</value>
  <description>Minimum allocatable memory per task</description>
</property>

<!--<property>
  <name>tajo.worker.resource.disks</name>
  <value>2</value>
  <description>Available disk capacity (usually number of disks)</description>
</property>
-->
<property>
  <name>tajo.worker.resource.dfs-dir-aware</name>
  <value>true</value>
</property>

<property>
  <name>tajo.worker.tmpdir.locations</name>
  <value>/tmp/tajo-${user.name}/tmpdir</value>
  <description>A base for other temporary directories.</description>
</property>
:wq!

[tajo@big-master conf]$ vi workers
big-slave1
big-slave2
big-slave3
big-slave4
:wq!

#caltalog-store로 derby default임
#mariadb로 변경,사전에 mariadb설치 및 db,user가 생성되어야 함.
#tajo.catalog.jdbc.connection.id depreciated tajo.catalog.connection.id 변경
#tajo.catalog.jdbc.connection.password depreciated tajo.catalog.connection.password 변경
[tajo@big-master conf]$ cp catalog-site.xml.template catalog-site.xml
[tajo@big-master conf]$ vi catalog-site.xml
<property>
  <name>tajo.catalog.connection.id</name>
  <value>tajo</value>
</property>
<property>
  <name>tajo.catalog.connection.password</name>
  <value>tajo</value>
</property>

 <!-- MariaDB Catalog Store Driver -->

<property>
  <name>tajo.catalog.store.class</name>
  <value>org.apache.tajo.catalog.store.MariaDBStore</value>
</property>
<property>
  <name>tajo.catalog.uri</name>
  <value>jdbc:mariadb://big-slave2:3306/tajo?rewriteBatchedStatements=true</value>
</property>
:wq!

[tajo@big-master conf]$ wget -P ../lib https://downloads.mariadb.com/Connectors/java/connector-java-2.2.3/mariadb-java-client-2.2.3.jar

#HA 적용을 위해
[tajo@big-master conf]$ vi masters
big-master
big-slave1
:wq!

[hadoop@big-master conf]$ hdfs dfs -mkdir       /tajo
[hadoop@big-master conf]$ hdfs dfs -chmod g+w   /tajo

[tajo@big-master conf]$ rsync -avz /bigdata/tajo-0.11.3 big-slave1:/bigdata/tajo-0.11.3
[tajo@big-master conf]$ rsync -avz /bigdata/tajo-0.11.3 big-slave2:/bigdata/tajo-0.11.3
[tajo@big-master conf]$ rsync -avz /bigdata/tajo-0.11.3 big-slave3:/bigdata/tajo-0.11.3
[tajo@big-master conf]$ rsync -avz /bigdata/tajo-0.11.3 big-slave4:/bigdata/tajo-0.11.3
[tajo@big-master conf]$ exit
[root@big-master conf]# ssh big-slave1 "ln -s /bigdata/tajo-0.11.3/ /tajo"
[root@big-master conf]# ssh big-slave2 "ln -s /bigdata/tajo-0.11.3/ /tajo"
[root@big-master conf]# ssh big-slave3 "ln -s /bigdata/tajo-0.11.3/ /tajo"
[root@big-master conf]# ssh big-slave4 "ln -s /bigdata/tajo-0.11.3/ /tajo"

#backup master인 big-slave1의 tajo-site.xml에서 active master의 hostname을 backup master인 big-slave1로 변경
#hadoop resourcemanager HA구성시도 마찬가지로 standby는 yarn-site.xml hostname을 변경한다.
#hadoop HA구성시 hdfs zkfc -formatZK이 있다
[tajo@big-master bin]$ tajo haadmin -formatHA
Formatting finished successfully.

[tajo@big-master conf]$ start-tajo.sh
Starting TajoMasters on HA mode
big-master: starting master, logging to /tajo/bin/../logs/tajo-tajo-master-big-master.out
big-slave1: starting master, logging to /tajo/bin/../logs/tajo-tajo-master-big-slave1.out
big-slave4: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave4.out
big-slave3: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave3.out
big-slave2: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave2.out
big-slave1: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave1.out
Tajo master web UI: http://big-master:26080
Tajo Client Service: big-master:26002

[tajo@big-master conf]$ tsql

welcome to
   _____ ___  _____ ___
  /_  _/ _  |/_  _/   /
   / // /_| |_/ // / /
  /_//_/ /_/___/ \__/  0.11.3

Try \? for help.
default>
default> \l
information_schema
default
default> \c information_schema
You are now connected to database "information_schema" as user "tajo".
information_schema>
information_schema> \d
tablespace
databases
tables
columns
indexes
table_options
table_stats
partitions
cluster
session
information_schema> \q
bye!
[tajo@big-master conf]$ hdfs dfs -ls -R /tajo
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/system
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/system/ha
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/system/ha/active
-rw-r--r--   3 tajo supergroup        111 2018-04-20 11:35 /tajo/system/ha/active/192.168.100.180_26001
-rw-r--r--   3 tajo supergroup          0 2018-04-20 11:35 /tajo/system/ha/active/active.lock
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/system/ha/backup
-rw-r--r--   3 tajo supergroup        111 2018-04-20 11:35 /tajo/system/ha/backup/192.168.100.181_26001
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/system/resource
-rwxr-xr-x  20 tajo supergroup     112274 2018-04-20 11:35 /tajo/system/resource/system_conf.xml
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/warehouse
drwxr-xr-x   - tajo supergroup          0 2018-04-20 11:35 /tajo/warehouse/default

#catalog용 mariadb machine에서 접속확인
[tajo@big-master conf]$ ssh big-slave2
[tajo@big-slave2 ~]$ netstat -an | grep 3306
tcp6       0      0 :::3306                 :::*                    LISTEN
tcp6       0      0 192.168.100.182:3306    192.168.100.181:42968   ESTABLISHED
tcp6       0      0 192.168.100.182:3306    192.168.100.180:52164   ESTABLISHED
[tajo@big-slave2 ~]$ mysql -u tajo -p tajo
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.2.14-MariaDB Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [tajo]> show tables;
+-------------------+
| Tables_in_tajo    |
+-------------------+
| COLUMNS           |
| DATABASES_        |
| INDEXES           |
| META              |
| OPTIONS           |
| PARTITIONS        |
| PARTITION_KEYS    |
| PARTITION_METHODS |
| STATS             |
| TABLES            |
| TABLESPACES       |
+-------------------+
11 rows in set (0.00 sec)

MariaDB [tajo]>

[tajo@big-master conf]$ stop-tajo.sh
[tajo@big-master conf]$ exit
logout
[root@big-master target]#
[root@big-master target]# vi /tajo/bin/tajo-service.sh
#!/bin/bash

start() {
        source "/etc/profile"
        start-tajo.sh
}

stop() {
        source "/etc/profile"
        stop-tajo.sh
}

case $1 in
        start|stop) "$1" ;;
esac

exit 0
:wq!
[root@big-master target]# chown tajo.tajo /tajo/bin/tajo-service.sh
[root@big-master target]# chmod 755 /tajo/bin/tajo-service.sh
[root@big-master target]# vi /etc/systemd/system/tajo.service
[Unit]
Description = tajo server ( ver. 0.11.3 )
After = network.target hadoop.service
Requires = network.target

[Service]
Type=oneshot
User=tajo
Group=tajo
ExecStart = /tajo/bin/tajo-service.sh start
ExecStop = /tajo/bin/tajo-service.sh stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
:wq!

[root@big-master target]# chmod 644 /etc/systemd/system/tajo.service

[root@big-master target]# systemctl daemon-reload
[root@big-master target]# systemctl start tajo.service
[root@big-master target]# systemctl status tajo.service
● tajo.service - tajo server ( ver. 0.11.3 )
   Loaded: loaded (/etc/systemd/system/tajo.service; disabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-05-01 19:15:18 KST; 10s ago
  Process: 3876 ExecStart=/tajo/bin/tajo-service.sh start (code=exited, status=0/SUCCESS)
 Main PID: 3876 (code=exited, status=0/SUCCESS)

May 01 19:15:08 big-master tajo-service.sh[3876]: Starting TajoMasters on HA mode
May 01 19:15:10 big-master tajo-service.sh[3876]: big-slave1: starting master, logging to /tajo/bin/../logs/tajo-tajo-master-big-slave1.out
May 01 19:15:10 big-master tajo-service.sh[3876]: big-master: starting master, logging to /tajo/bin/../logs/tajo-tajo-master-big-master.out
May 01 19:15:11 big-master tajo-service.sh[3876]: big-slave4: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave4.out
May 01 19:15:11 big-master tajo-service.sh[3876]: big-slave2: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave2.out
May 01 19:15:11 big-master tajo-service.sh[3876]: big-slave3: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave3.out
May 01 19:15:11 big-master tajo-service.sh[3876]: big-slave1: starting worker, logging to /tajo/bin/../logs/tajo-tajo-worker-big-slave1.out
May 01 19:15:18 big-master tajo-service.sh[3876]: Tajo master web UI: http://big-master:26080
May 01 19:15:18 big-master tajo-service.sh[3876]: Tajo Client Service: big-master:26002
May 01 19:15:18 big-master systemd[1]: Started tajo server ( ver. 0.11.3 ).
[root@big-master target]# systemctl enable tajo.service
Created symlink from /etc/systemd/system/multi-user.target.wants/tajo.service to /etc/systemd/system/tajo.service.
[root@big-master target]# jps
2944 SqoopJettyServer
1603 JournalNode
2070 JobHistoryServer
4346 Jps
1002 QuorumPeerMain
1355 NameNode
1804 DFSZKFailoverController
1948 ResourceManager
4029 TajoMaster
[root@big-master target]#

WebUI active http://big-master:26080/

 

WebUI backup http://big-slave1:26080/

 

 

Cluster Service Configuration Defaults
→http://tajo.apache.org/docs/current/configuration/service_config_defaults.html