Notice
Recent Posts
Recent Comments
Link
일 | 월 | 화 | 수 | 목 | 금 | 토 |
---|---|---|---|---|---|---|
1 | 2 | 3 | 4 | |||
5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 | 13 | 14 | 15 | 16 | 17 | 18 |
19 | 20 | 21 | 22 | 23 | 24 | 25 |
26 | 27 | 28 | 29 | 30 | 31 |
Tags
- Eclipse
- es6
- Java
- xPlatform
- IntelliJ
- mapreduce
- Kotlin
- tomcat
- SPC
- Python
- table
- maVen
- GIT
- 보조정렬
- 정렬
- hadoop
- JavaScript
- Express
- 공정능력
- Sqoop
- react
- vaadin
- Spring
- R
- SQL
- window
- MSSQL
- NPM
- Android
- mybatis
Archives
- Today
- Total
DBILITY
독거 가능성 100% 노후에 라면값이라도 하게 센스를 발휘합시다!😅
Please click on the ad so that I can pay for ramen in my old age!
点击一下广告,让老后吃个泡面钱吧!
老後にラーメン代だけでもするように広告を一回クリックしてください。
flume kafka channel 테스트 본문
반응형
kafka를 flume channel로 이용해 보자.
먼저 channel을 만든다. 순서보장을 위해 parition은 1개만 선언하고, ha용 replication factor는 2로 실행
[kafka@big-slave4 ~]$ kafka-topics.sh \
--zookeeper big-master:2181,big-slave1:2181,big-slave2:2181/kafka-cluster \
--topic flume-channel --partitions 1 --replication-factor 2 --create
Created topic "flume-channel".
leader 확인
[kafka@big-slave4 ~]$ kafka-topics.sh \
--zookeeper big-master:2181,big-slave1:2181,big-slave2:2181/kafka-cluster \
--topic flume-channel --describe
Topic:flume-channel PartitionCount:1 ReplicationFactor:2 Configs:
Topic: flume-channel Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
[kafka@big-slave4 ~]$ kafka-console-consumer.sh \
--bootstrap-server big-slave2:9092,big-slave3:9092,big-slave4:9092 \
--topic flume-channel --from-beginning
^CProcessed a total of 0 messages
[kafka@big-slave4 ~]$ kafka-consumer-groups.sh \
--bootstrap-server big-slave2:9092,big-slave3:9092,big-slave4:9092 \
--list
Note: This will not show information about old Zookeeper-based consumers.
KMOffsetCache-big-master
console-consumer-94532
[kafka@big-slave4 ~]$ kafka-consumer-groups.sh \
--bootstrap-server big-slave2:9092,big-slave3:9092,big-slave4:9092 \
--group console-consumer-94532 --delete
Note: This will not show information about old Zookeeper-based consumers.
Deletion of requested consumer groups ('console-consumer-94532') was successful.
[kafka@big-slave4 ~]$ kafka-console-consumer.sh \
--bootstrap-server big-slave2:9092,big-slave3:9092,big-slave4:9092 \
--topic flume-channel \
--group flume-channel-consumers \
--from-beginning
client flume conf
#Source
agent.sources = randomGen
agent.sources.randomGen.type = com.dbility.bigdata.flume.source.RandomDataGenerator
agent.sources.randomGen.batchSize = 10
agent.sources.randomGen.channels = kafkaChannel
#Channel
agent.channels = kafkaChannel
agent.channels.kafkaChannel.type = org.apache.flume.channel.kafka.KafkaChannel
agent.channels.kafkaChannel.kafka.bootstrap.servers = big-slave2:9092,big-slave3:9092,big-slave4:9092
agent.channels.kafkaChannel.kafka.topic = flume-channel
agent.channels.kafkaChannel.kafka.consumer.group.id = flume-channe1-consumers
client agent 실행
E:\apache-flume-1.8.0-bin>bin\flume-ng agent -n agent -c conf -f conf\randomGen2.properties
E:\apache-flume-1.8.0-bin>set JAVA_HOME=E:\apache-flume-1.8.0-bin\jdk180\jre
E:\apache-flume-1.8.0-bin>set JAVA_OPTS=" -Xmx256M"
E:\apache-flume-1.8.0-bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File E:\apache-flume-1.8.0-bin\bin\flume-ng.ps1 agent -n agent -c conf -f conf\randomGen2.properties
WARN: Config directory not set. Defaulting to E:\apache-flume-1.8.0-bin\conf
Sourcing environment configuration script E:\apache-flume-1.8.0-bin\conf\flume-env.ps1
WARN: Did not find E:\apache-flume-1.8.0-bin\conf\flume-env.ps1
WARN: HADOOP_PREFIX or HADOOP_HOME not found
WARN: HADOOP_PREFIX not set. Unable to include Hadoop's classpath & java.library.path
WARN: HBASE_HOME not found
WARN: HIVE_HOME not found
Running FLUME agent :
class: org.apache.flume.node.Application
arguments: -n agent -f "E:\apache-flume-1.8.0-bin\conf\randomGen2.properties"
반응형
'bigdata > flume' 카테고리의 다른 글
flume hadoop 연동 테스트 소스 데이터 생성 (0) | 2018.05.03 |
---|---|
window에서 사용할 Flume Sources 테스트 (0) | 2018.04.25 |
flume-ng window batch (0) | 2018.04.24 |
flume-ng windows agent test (0) | 2018.04.23 |
Comments