source table의 hit ratio를 높이기 위해 그룹핑연산sql을 실행. mapreduce.job.ubertask.enable=true 설정하고 해봐야겠다.
loaders 설정없이 map만으로 처리하는게 더 빠른다.물론, 파일병합을 따로 해야 하지만.
[sqoop2@big-master ~]$ sqoop2-shell
Setting conf dir: /sqoop/bin/../conf
Sqoop home directory: /sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> set server --url http://big-master:12000/sqoop
Server is set successfully
sqoop:000> show server -all
Server host: big-master
Server port: 12000
Server webapp: sqoop
sqoop:000> show connector
+------------------------+---------+------------------------------------------------------------+----------------------+
| Name | Version | Class | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
| kite-connector | 1.99.7 | org.apache.sqoop.connector.kite.KiteConnector | FROM/TO |
| oracle-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO |
| ftp-connector | 1.99.7 | org.apache.sqoop.connector.ftp.FtpConnector | TO |
| hdfs-connector | 1.99.7 | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
| kafka-connector | 1.99.7 | org.apache.sqoop.connector.kafka.KafkaConnector | TO |
| sftp-connector | 1.99.7 | org.apache.sqoop.connector.sftp.SftpConnector | TO |
+------------------------+---------+------------------------------------------------------------+----------------------+
sqoop:000> show link
+------+----------------+---------+
| Name | Connector Name | Enabled |
+------+----------------+---------+
+------+----------------+---------+
sqoop:000> show job
+----+------+----------------+--------------+---------+
| Id | Name | From Connector | To Connector | Enabled |
+----+------+----------------+--------------+---------+
+----+------+----------------+--------------+---------+
sqoop:000> create link -c generic-jdbc-connector
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: oracleLink
Database connection
Driver class: oracle.jdbc.OracleDriver
Error message: String must start with: jdbc:
Connection String: jdbc:oracle:thin:@xxx.xxx.xxx.xxx:1521/ora1
Username: dev
Password: ********
Fetch Size:
Connection Properties:
There are currently 0 values in the map:
entry#
SQL Dialect
Identifier enclose:
New link was successfully created with validation status OK and name oracleLink
sqoop:000> create link -c hdfs-connector
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: hdfsLink
HDFS cluster
URI: hdfs://hadoop-cluster
Conf directory: /hadoop/etc/hadoop
Additional configs::
There are currently 0 values in the map:
entry#
New link was successfully created with validation status OK and name hdfsLink
sqoop:000> create job -f oracleLink -t hdfsLink
Creating job for links with from name oracleLink and to name hdfsLink
Please fill following values to create new job object
Name: oracle2hdfs
Database source
Schema name: DEV
Table name: TB_SAMPLE_SOURCE
SQL statement:
Column names:
There are currently 0 values in the list:
element#
Partition column:
Partition column nullable:
Boundary query:
Incremental read
Check column:
Last value:
Target configuration
Override null value:
Null value:
File format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
2 : PARQUET_FILE
Choose: 0
Compression codec:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0
Custom codec:
Output directory: /oracle/tb_sample_source
Append mode:
Throttling resources
Extractors:
Loaders:
Classpath configuration
Extra mapper jars:
There are currently 0 values in the list:
element#
New job was successfully created with validation status OK and name oracle2hdfs
sqoop:000> show job
+----+-------------+-------------------------------------+---------------------------+---------+
| Id | Name | From Connector | To Connector | Enabled |
+----+-------------+-------------------------------------+---------------------------+---------+
| 6 | oracle2hdfs | oracleLink (generic-jdbc-connector) | hdfsLink (hdfs-connector) | true |
+----+-------------+-------------------------------------+---------------------------+---------+
sqoop:000> start job -n oracle2hdfs -s
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0025:No primary key - Please specify partition column.
sqoop:000> update job -n oracle2hdfs
Updating job with name oracle2hdfs
Please update job:
Name: oracle2hdfs
Database source
Schema name: DEV
Table name: TB_SAMPLE_SOURCE
SQL statement:
Column names:
There are currently 0 values in the list:
element#
Partition column: SEQ
Partition column nullable:
Boundary query:
Incremental read
Check column:
Last value:
Target configuration
Override null value:
Null value:
File format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
2 : PARQUET_FILE
Choose: 0
Compression codec:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0
Custom codec:
Output directory: /oracle/tb_sample_source
Append mode:
Throttling resources
Extractors:
Loaders:
Classpath configuration
Extra mapper jars:
There are currently 0 values in the list:
element#
Job was successfully updated with status OK
sqoop:000> start job -n oracle2hdfs -s
Submission details
Job Name: oracle2hdfs
Server URL: http://big-master:12000/sqoop/
Created by: sqoop2
Creation date: 2018-04-22 21:03:21 KST
Lastly updated by: sqoop2
External ID: job_1524396616145_0003
http://0.0.0.0:8089/proxy/application_1524396616145_0003/
2018-04-22 21:03:21 KST: BOOTING - Progress is not available
2018-04-22 21:03:36 KST: RUNNING - 0.00 %
2018-04-22 21:03:46 KST: RUNNING - 0.00 %
2018-04-22 21:03:56 KST: RUNNING - 0.00 %
2018-04-22 21:04:07 KST: RUNNING - 0.00 %
2018-04-22 21:04:17 KST: RUNNING - 0.00 %
2018-04-22 21:04:27 KST: RUNNING - 0.00 %
2018-04-22 21:04:37 KST: RUNNING - 5.00 %
2018-04-22 21:04:47 KST: FAILED
Exception: Job Failed with status:3
#mapreduce log확인
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate sqoop2
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy10.delete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:545)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2053)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714)
at org.apache.sqoop.connector.hdfs.HdfsToDestroyer$1.run(HdfsToDestroyer.java:69)
at org.apache.sqoop.connector.hdfs.HdfsToDestroyer$1.run(HdfsToDestroyer.java:52)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
at org.apache.sqoop.connector.hdfs.HdfsToDestroyer.destroy(HdfsToDestroyer.java:52)
... 12 more
2018-04-22 21:04:42,726 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1524396616145_0003Job Transitioned from FAIL_ABORT to FAILED
권한관련 오류가 났다. 2년전에랑 달라진 건 hadoop version뿐 설치시 별다르게 한것도 없는데...
