카프카 producer와 consumer 테스트

카프카 producer와 consumer 테스트

 

이미지 출처-https://www.tutorialspoint.com/apache_kafka/images/cluster_architecture.jpg

 

이미지 출처-https://www.researchgate.net/profile/Asad-Javed-9/publication/338516815/figure/fig2/AS:846484845125632@1578829122361/Example-of-Kafka-cluster-with-three-brokers.png

카프카 토픽 생성

• topic list

kafka-topics.sh --list --bootstrap-server localhost:9092

• topic create
  • replication-factor : 복제본 개수(2)
  • partitions : 파티션 개수(3)

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 2 --partitions 3 --topic helloworld

$ kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 2 --partitions 3 --topic helloworld
Created topic helloworld.

• topic list

kafka-topics.sh --list --bootstrap-server localhost:9092

$ kafka-topics.sh --list --bootstrap-server localhost:9092
helloworld

• topic delete

kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic helloworld

$ kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic helloworld

728x90

프로듀서(Producer), 컨슈머(Consumer) 실행하기

• 프로듀서 실행 / 메시지 Write

kafka-console-producer.sh --broker-list localhost:9092 --topic helloworld

• 컨슈머(Consumer) 실행 / 메시지 Read

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic helloworld

--from-beginning 옵션을 추가하면 토픽에 있는 모든 레코드를 확인 할수 있다.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic helloworld --from-beginning

• 컨슈머 그룹(Consumer Group) list

kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

$ kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
console-consumer-58810

• 컨슈머(Consumer) status and Offset 확인

kafka-consumer-groups.sh --describe --group console-consumer-58810 --bootstrap-server localhost:9092

$ kafka-consumer-groups.sh --describe --group console-consumer-58810 --bootstrap-server localhost:9092

GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
console-consumer-58810 helloworld      0          -               1               -               consumer-console-consumer-58810-1-034dcce6-8cc4-46d9-a09d-dfcf32469460 /192.168.0.103   consumer-console-consumer-58810-1

• 컨슈머 그룹(Consumer Group) delete

kafka-consumer-groups.sh --delete --group console-consumer-58810 --bootstrap-server localhost:9092

$ kafka-consumer-groups.sh --delete --group console-consumer-58810 --bootstrap-server localhost:9092
Deletion of requested consumer groups ('console-consumer-58810') was successful.

 

참고URL

- Kafka 문서 : https://kafka.apache.org/documentation/#introduction

- Kafka 시스템 구조 알아보기 : https://velog.io/@jwpark06/Kafka-%EC%8B%9C%EC%8A%A4%ED%85%9C-%EA%B5%AC%EC%A1%B0-%EC%95%8C%EC%95%84%EB%B3%B4%EA%B8%B0