[Pinpoint] Pinot, Kafka 설치

---

[Pinpoint-HBase] HBase v2.5.10 설치하기

Pinpoint 3.0을 사용하려면 Pinot, Kafka 설치 전 HBase를 설치하는 매뉴얼은 위에 기록했다.

Pinpoint 3.0부터는 System Metrics를 수집하는 것이 하나의 특징을 보인다.

그렇다면 이러한 System Metrics를 수집해보는 것이 당연하다 느껴져서 이를 도전해보고자 했다.

System Metrics 수집은 Pinot와 Kafka를 필요로 하기에 설치를 진행했다.
(Pinot는 처음 들어보는…..)

Kafka v2.13 설치하기

wget https://dlcdn.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz
tar -xvzf kafka_2.13-3.7.1.tgz

위 경로에서 2.13 버전을 다운받아 압축을 풀어준 후에는 zookeeper.properties에서 포트를 변경하였다.

기본 포트는 2181이지만 HBase의 ZooKeeper와 겹치기에 2191로 변경하여 올렸다.
(HBase와 동일한 ZooKeeper를 연동할 수 있는지 테스트 해봤지만 잘 안됐다…)

vim ./kafka_2.13-3.7.1/config/server.properties
   # root directory for all kafka znodes.
   #zookeeper.connect=localhost:2181 
   zookeeper.connect=localhost:2191 


vim ./kafka_2.13-3.7.1/config/server.properties
   # the port at which the clients will connect
   # clientPort=2181
   clientPort=2191

./kafka_2.13-3.7.1/bin/zookeeper-server-start.sh ./kafka_2.13-3.7.1/config/server.properties
./kafka_2.13-3.7.1/bin/kafka-server-start.sh ./kafka_2.13-3.7.1/config/server.properties

위 처럼 명령어들을 실행하면 일단 Kafka자체는 성공이다.

./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic inspector-stat-agent-00 --bootstrap-server localhost:9092
./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic inspector-stat-app --bootstrap-server localhost:9092
./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic system-metric-data-type --bootstrap-server localhost:9092
./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic system-metric-tag --bootstrap-server localhost:9092
./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic system-metric-double --bootstrap-server localhost:9092
./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic url-stat --bootstrap-server localhost:9092
./kafka_2.13-3.7.1/bin/kafka-topics.sh --create --topic exception-trace --bootstrap-server localhost:9092

Pinot와 Kafka간 필요한 테이블 및 토픽들이 있는데 먼저 Kafaka의 토픽을 생성한다.

Pinot v1.1.0 설치하기

Pinot는 개인적으로 처음 들어보는 오픈소스인데 포도 품종 중에 피노 누아라는게 있는데 이를 따서 Pinot라고 하는건가 싶다.
(그래서 로고가 와인잔인가 싶기도….)

wget https://dlcdn.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz
tar -xvzf apache-pinot-1.1.0-bin.tar.gz

위처럼 파일을 받았으면 JAVA 옵션과 로그경로를 좀 바꿔보려고 한다.

vim ./apache-pinot-1.1.0-bin/bin/pinot-admin.sh
   if [ -z "$JAVA_OPTS" ] ; then
   #ALL_JAVA_OPTS="-Xms4G -Dlog4j2.configurationFile=conf/log4j2.xml -Dpinot.admin.system.exit=true"
   ALL_JAVA_OPTS="-Xms512m -Xmx1G -Dlog4j2.configurationFile=conf/log4j2.xml -Dpinot.admin.system.exit=true"
   else
   ALL_JAVA_OPTS=$JAVA_OPTS
   fi

초기 힙메모리 설정이 너무 높게 되어 있어서 낮추어줬고

vim ./apache-pinot-1.1.0-bin/conf/log4j2.xml
 <Properties>
          <Property name="LOG_ROOT">./apache-pinot-1.1.0-bin/logs</Property>
    <Property name="LOG_PATTERN">%d{yyyy/MM/dd HH:mm:ss.SSS} %p [%c{1}] [%t] %m%n</Property>
    <Property name="PINOT_COMPONENT">all</Property>
  </Properties>

로그 경로를 잡아주었다.

다만 예시일 뿐이지 실제 로그 경로 상대경로가 아닌 절대경로로 작성했다.

./apache-pinot-1.1.0-bin/bin/pinot-admin.sh StartController -zkAddress localhost:2191 -controllerPort 9000
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh StartBroker -zkAddress localhost:2191
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh StartServer -zkAddress localhost:2191

위 명령어들로 Pinot를 실행시킨다.

그 후에는 Pinot의 테이블들을 생성해줘야한다.

wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/inspector-module/inspector-collector/src/main/pinot/multi-table/multi-table.sh
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/inspector-module/inspector-collector/src/main/pinot/multi-table/template_OFFLINE.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/inspector-module/inspector-collector/src/main/pinot/multi-table/template_REALTIME.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/inspector-module/inspector-collector/src/main/pinot/multi-table/template_SCHEMA.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/inspector-module/inspector-collector/src/main/pinot/pinot-inspector-stat-application-realtime-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/inspector-module/inspector-collector/src/main/pinot/pinot-inspector-stat-application-schema.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-dataType-realtime-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-dataType-schema.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-double-offline-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-double-realtime-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-double-schema.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-tag-realtime-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/metric-module/metric/src/main/pinot/pinot-tag-schema.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/uristat/uristat-common/src/main/pinot/pinot-uriStat-offline-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/uristat/uristat-common/src/main/pinot/pinot-uriStat-realtime-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/uristat/uristat-common/src/main/pinot/pinot-uriStat-schema.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/exceptiontrace/exceptiontrace-common/src/main/pinot/pinot-exceptionTrace-offline-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/exceptiontrace/exceptiontrace-common/src/main/pinot/pinot-exceptionTrace-realtime-table.json
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/exceptiontrace/exceptiontrace-common/src/main/pinot/pinot-exceptionTrace-schema.json

다행히 데이터들은 제공을 하고 있어서 위 파일들을 내려받고 확인하여 Kafka broker 포트들과 Replication들을 확인한다.

./apache-pinot-1.1.0-bin/json/multi-table.sh 0 0 http://localhost:9000

Inspector 테이블을 생성하고

./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddSchema -schemaFile pinot-dataType-schema.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddSchema -schemaFile pinot-double-schema.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddSchema -schemaFile pinot-exceptionTrace-schema.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddSchema -schemaFile pinot-inspector-stat-application-schema.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddSchema -schemaFile pinot-tag-schema.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddSchema -schemaFile pinot-uriStat-schema.json -controllerHost localhost -controllerPort 9000 -exec

스키마를 생성하고

./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-dataType-realtime-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-double-offline-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-double-realtime-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-exceptionTrace-offline-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-exceptionTrace-realtime-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-inspector-stat-application-realtime-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-tag-realtime-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-uriStat-offline-table.json -controllerHost localhost -controllerPort 9000 -exec
./apache-pinot-1.1.0-bin/bin/pinot-admin.sh AddTable -tableConfigFile pinot-uriStat-realtime-table.json -controllerHost localhost -controllerPort 9000 -exec

테이블을 생성하고….
(문득 이쯤에서 이게 맞나 싶은 생각이 들게 되었다.)

curl -X POST -H "Content-Type: application/json" -d @pinot-dataType-realtime-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-double-offline-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-double-realtime-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-exceptionTrace-offline-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-exceptionTrace-realtime-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-inspector-stat-application-realtime-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-tag-realtime-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-uriStat-offline-table.json http://localhost:9000/tables
curl -X POST -H "Content-Type: application/json" -d @pinot-uriStat-realtime-table.json http://localhost:9000/tables

위 명령어로 추가가 안될경우 curl로 테이블 생성하는 방법이고, 이렇게 하면 Pinot과 Kafka의 설정은 끝이 났다……

다음으로는 Pinpoint의 Collector Starter, Web Starter를 설치해보겠다.
(Pinot를 사용할 예정이면 Starter를, Pinot를 사용하지 않는다면 그냥 Collector와 Web을 설치하면 된다.)