flume-ng windows agent test

flume은 Source, Channel, Sink로 구성되며,

Source는 데이터를 수집, Channel은 sink로 넘기기전 저장하는 버퍼(?), Sink는 Channel의 데이터를 저장하는 역할이다.

Sink가 다른 Source를 향할 수 있다.최종적으론 stream을 처리하는 특정 포트(?)로 전송하거나 hdfs에 저장하는데 사용하겠다.

 

flume.apache.org에서 apache-flume-1.8.0-bin.tar.gz를 다운받아 윈도우에서 테스트 해봄

공식사이트 User Guide에 따라 그냥 테스트해 봄

 

window용 wget사용해서 다운로드 먼저 설치, 압축해제를 위해 7Zip path도 설정해야함.

물론 그냥 다운로드하고,압출프로그램에서 풀어도 된다.ㅎㅎ

netcat source도 테스트해보니 잘 된다.참고로 window telnet exit는 Ctrl+]이다.

E:\>wget http://apache.tt.co.kr/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz
--2018-04-23 19:30:39--  http://apache.tt.co.kr/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz
Resolving apache.tt.co.kr (apache.tt.co.kr)... 211.47.69.77
Connecting to apache.tt.co.kr (apache.tt.co.kr)|211.47.69.77|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 58688757 (56M) [application/x-gzip]
Saving to: 'apache-flume-1.8.0-bin.tar.gz'

apache-flume-1.8.0-bin.tar.gz 100%[=================================================>]  55.97M  14.6MB/s    in 20s

2018-04-23 19:30:59 (2.74 MB/s) - 'apache-flume-1.8.0-bin.tar.gz' saved [58688757/58688757]

E:\>7z x "apache-flume-1.8.0-bin.tar.gz" -so | 7z x -aoa -si -ttar

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21


Extracting archive:
--
Path =
Type = tar
Code Page = UTF-8

Everything is Ok

Folders: 137
Files: 1601
Size:       88956846
Compressed: 1410048

E:\>cd apache-flume-1.8.0-bin

E:\apache-flume-1.8.0-bin>

#flume-ng.cmd에 JAVA_HOME설정,Quotation으로 묶으니 안되네?
#powershell로 실행을 한다.xp client가 상당할텐데..별도 bat작성이 필요하겠다.
set JAVA_HOME=C:\Dev64\tools\jdk\jdk180\jre
E:\>cd apache-flume-1.8.0-bin\bin

E:\apache-flume-1.8.0-bin\bin>flume-ng help

E:\apache-flume-1.8.0-bin\bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File E:\apache-flume-1.8.0-bin\bin\flume-ng.ps1 help


Usage: flume-ng <command> [options]...

commands:
  help                  display this help text
  agent                 run a Flume agent
  avro-client           run an avro Flume client
  version               show Flume version info

global options:
  -conf  <conf>                           use configs in <conf> directory
  -classpath,-C  "value1;value2; .."    append to the classpath
  -property "name1=val;name2=val; .."   sets a JDK system property value
  -dryrun,-d                              do not actually start Flume (test)
  -plugins-path "dir1;dir2"             semi-colon-separated list of plugins.d directories. See the
                                          plugins.d section in the user guide for more details.
                                          Default: \/plugins.d

agent options:
  -conf-file,-f  <file>                specify a config file (required)
  -name,-n  <name>                     the name of this agent (required)

avro-client options:
  -rpcProps <file>           RPC client properties file with server connection params
  -host,-H  <host>           hostname to which events will be sent (required)
  -port,-p  <port>           port of the avro source (required)
  -dirname <dir>             directory to stream to avro source
  -filename  <file>          text file to stream to avro source [default: std input]
  -headerFile,-R <file>      headerFile containing headers as key/value pairs on each new line

  Either --rpcProps or both --host and --port must be specified.

Note that if <conf> directory is specified, then it is always included first
in the classpath.
E:\apache-flume-1.8.0-bin\bin>copy ..\conf\flume-conf.properties.template ..\conf\flume-conf.properties
        1개 파일이 복사되었습니다.
#E:\apache-flume-1.8.0-bin\conf\log4j.properties 수정
#flume.root.logger=INFO,LOGFILE,console
E:\apache-flume-1.8.0-bin\bin>cd ..
E:\apache-flume-1.8.0-bin>bin\flume-ng.cmd agent -c conf -f conf\flume-conf.properties -n agent

E:\apache-flume-1.8.0-bin>set JAVA_HOME=C:\Dev64\tools\jdk\jdk180\jre

E:\apache-flume-1.8.0-bin>powershell.exe -NoProfile -InputFormat none -ExecutionPolicy unrestricted -File E:\apache-flume-1.8.0-bin\bin\flume-ng.ps1 agent -c conf -f conf\flume-conf.properties -n agent

WARN: Config directory not set. Defaulting to E:\apache-flume-1.8.0-bin\conf
Sourcing environment configuration script E:\apache-flume-1.8.0-bin\conf\flume-env.ps1
WARN: Did not find E:\apache-flume-1.8.0-bin\conf\flume-env.ps1
WARN: HADOOP_PREFIX or HADOOP_HOME not found
WARN: HADOOP_PREFIX not set. Unable to include Hadoop's classpath & java.library.path
WARN: HBASE_HOME not found
WARN: HIVE_HOME not found

  Running FLUME agent :
    class: org.apache.flume.node.Application
    arguments: -n agent -f "E:\apache-flume-1.8.0-bin\conf\flume-conf.properties"

2018-04-24 09:32:55,812 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
2018-04-24 09:32:55,819 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:E:\apache-flume-1.8.0-bin\conf\flume-conf.properties
2018-04-24 09:32:55,825 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:loggerSink
2018-04-24 09:32:55,826 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:loggerSink
2018-04-24 09:32:55,826 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: loggerSink Agent: agent
2018-04-24 09:32:55,836 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent]
2018-04-24 09:32:55,836 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels
2018-04-24 09:32:55,844 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel memoryChannel type memory
2018-04-24 09:32:55,852 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel memoryChannel
2018-04-24 09:32:55,853 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source seqGenSrc, type seq
2018-04-24 09:32:55,861 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: loggerSink, type: logger
2018-04-24 09:32:55,866 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel memoryChannel connected to [seqGenSrc, loggerSink]
2018-04-24 09:32:55,875 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{seqGenSrc=PollableSourceRunner: { source:org.apache.flume.source.SequenceGeneratorSource{name:seqGenSrc,state:IDLE} counterGroup:{ name:null counters:{} } }} sinkRunners:{loggerSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@224b6078 counterGroup:{ name:null counters:{} } }} channels:{memoryChannel=org.apache.flume.channel.MemoryChannel{name: memoryChannel}} }
2018-04-24 09:32:55,889 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel memoryChannel
2018-04-24 09:32:55,950 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: memoryChannel: Successfully registered new MBean.
2018-04-24 09:32:55,951 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: memoryChannel started
2018-04-24 09:32:55,953 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink loggerSink
2018-04-24 09:32:55,954 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source seqGenSrc
2018-04-24 09:32:55,955 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.SequenceGeneratorSource.doStart(SequenceGeneratorSource.java:103)] Sequence generator source do starting
2018-04-24 09:32:55,956 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: seqGenSrc: Successfully registered new MBean.
2018-04-24 09:32:55,957 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: seqGenSrc started
2018-04-24 09:32:55,963 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 30                                              0 }
2018-04-24 09:32:55,964 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 31                                              1 }
2018-04-24 09:32:55,965 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 32                                              2 }
2018-04-24 09:32:55,965 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 33                                              3 }
2018-04-24 09:32:55,966 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 34                                              4 }
2018-04-24 09:32:55,966 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 35                                              5 }
2018-04-24 09:32:55,967 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 36                                              6 }
2018-04-24 09:32:55,967 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 37                                              7 }
2018-04-24 09:32:55,967 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 38                                              8 }
2018-04-24 09:32:55,968 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 39                                              9 }
2018-04-24 09:32:55,968 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 31 30                                           10 }