2024-03-30 16:59:08.0|分类: kafka|浏览量: 399
Kafka 是一种流行的分布式消息队列系统。 它具有以下特点:
Kafka 的常见应用场景包括:
在使用 Kafka 时,需要注意以下几点:
1、先安装Zookeeper 2、下载安装包kafka_2.13-3.7.0.tgz 下载地址:https://mirrors.aliyun.com/apache/kafka/ 3、复制kafka_2.13-3.7.0.tgz到C:\me\studyhome\kafka 4、解压安装包kafka_2.13-3.7.0.tgz 5、进入到配置文件C:\me\studyhome\kafka\kafka_2.13-3.7.0\config 6、修改配置文件server.properties log.dirs=C:\\me\\studyhome\\kafka\\kafka_2.13-3.7.0\\kafka-logs zookeeper.connect=localhost:2181 7、进入安装目录C:\me\studyhome\kafka\kafka_2.13-3.7.0 .\bin\windows\kafka-server-start.bat .\config\server.properties 报错:输入行太长。 命令语法不正确。 原因:目录太深了:C:\me\studyhome\kafka\kafka_2.13-3.7.0 解决办法将安装目录移动到C判断,安装目录缩短:C:\kafka3.7.0 8、进入安装目录C:\kafka3.7.0,执行启动命令: C:\kafka3.7.0>.\bin\windows\kafka-server-start.bat .\config\server.properties [2024-03-30 16:56:05,657] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [2024-03-30 16:56:06,099] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [2024-03-30 16:56:06,203] INFO starting (kafka.server.KafkaServer) [2024-03-30 16:56:06,204] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer) [2024-03-30 16:56:06,223] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient) [2024-03-30 16:56:06,230] INFO Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,230] INFO Client environment:host.name=LAPTOP-78C0012V (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,230] INFO Client environment:java.version=1.8.0_231 (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,230] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,230] INFO Client environment:java.home=C:\Program Files\Java\jdk1.8.0_231\jre (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,230] INFO Client environment:java.class.path=.;C:\Program Files\Java\jdk1.8.0_231\lib\dt.jar;C:\Program Files\Java\jdk1.8.0_231\lib\tools.jar;C:\kafka3.7.0\libs\activation-1.1.1.jar;C:\kafka3.7.0\libs\aopalliance-repackaged-2.6.1.jar;C:\kafka3.7.0\libs\argparse4j-0.7.0.jar;C:\kafka3.7.0\libs\audience-annotations-0.12.0.jar;C:\kafka3.7.0\libs\caffeine-2...ns-2.2.8.jar;C:\kafka3.7.0\libs\trogdor-3.7.0.jar;C:\kafka3.7.0\libs\zookeeper-3.8.3.jar;C:\kafka3.7.0\libs\zookeeper-jute-3.8.3.jar;C:\kafka3.7.0\libs\zstd-jni-1.5.5-6.jar (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,232] INFO Client environment:java.library.path=C:\Program Files\Java\jdk1.8.0_231\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Java\jdk1.8.0_231\bin;C:\Program Files\Java\jdk1.8.0_231\jre\bin;C:\me\workhome\maven\apache-maven-3.6.2\bin;C:\me\workhome\mysql-5.7.26-winx64\bin;C:\Program Files\TortoiseSVN\bin;C:\Users\conca\AppData\Roaming\npm;C:\Program Files\nodejs;C:\Program Files\nodejs\node-global;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\PuTTY\;C:\Program Files (x86)\Tencent\微信web开发者工具\dll;C:\Program Files\nodejs\;C:\Program Files\微信web开发者工具\dll;C:\me\studyhome\flink\flink-1.12.7\bin;C:\Users\conca\AppData\Local\Programs\Python\Python311\Scripts\;C:\Users\conca\AppData\Local\Programs\Python\Python311\;C:\Users\conca\AppData\Local\Microsoft\WindowsApps;C:\Users\conca\AppData\Roaming\npm;C:\Users\conca\AppData\Local\Programs\EmEditor;. (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,233] INFO Client environment:java.io.tmpdir=C:\Users\conca\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,234] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,235] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,236] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,237] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,237] INFO Client environment:user.name=conca (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,245] INFO Client environment:user.home=C:\Users\conca (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,246] INFO Client environment:user.dir=C:\kafka3.7.0 (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,247] INFO Client environment:os.memory.free=1003MB (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,248] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,248] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,253] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@765d7657 (org.apache.zookeeper.ZooKeeper) [2024-03-30 16:56:06,263] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket) [2024-03-30 16:56:06,272] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn) [2024-03-30 16:56:06,275] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [2024-03-30 16:56:06,278] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn) [2024-03-30 16:56:06,279] INFO Socket connection established, initiating session, client: /127.0.0.1:50661, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn) [2024-03-30 16:56:06,295] INFO Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x10003ffe1230001, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn) [2024-03-30 16:56:06,299] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient) [2024-03-30 16:56:06,599] INFO Cluster ID = cMVm3bq8Rn2zW-3NKX406A (kafka.server.KafkaServer) [2024-03-30 16:56:06,657] INFO KafkaConfig values: advertised.listeners = null alter.config.policy.class.name = null alter.log.dirs.replication.quota.window.num = 11 .... zookeeper.ssl.truststore.password = null zookeeper.ssl.truststore.type = null (kafka.server.KafkaConfig) [2024-03-30 16:56:06,720] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2024-03-30 16:56:06,722] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2024-03-30 16:56:06,722] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2024-03-30 16:56:06,729] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2024-03-30 16:56:06,744] INFO [KafkaServer id=0] Rewriting C:\kafka3.7.0\kafka-logs\meta.properties (kafka.server.KafkaServer) [2024-03-30 16:56:06,795] INFO Loading logs from log dirs ArraySeq(C:\kafka3.7.0\kafka-logs) (kafka.log.LogManager) [2024-03-30 16:56:06,802] INFO No logs found to be loaded in C:\kafka3.7.0\kafka-logs (kafka.log.LogManager) [2024-03-30 16:56:06,814] INFO Loaded 0 logs in 19ms (kafka.log.LogManager) [2024-03-30 16:56:06,818] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2024-03-30 16:56:06,820] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2024-03-30 16:56:06,905] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner$CleanerThread) [2024-03-30 16:56:06,925] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread) [2024-03-30 16:56:06,948] INFO [MetadataCache brokerId=0] Updated cache from existing None to latest Features(version=3.7-IV4, finalizedFeatures={}, finalizedFeaturesEpoch=0). (kafka.server.metadata.ZkMetadataCache) [2024-03-30 16:56:06,980] INFO [zk-broker-0-to-controller-forwarding-channel-manager]: Starting (kafka.server.NodeToControllerRequestThread) [2024-03-30 16:56:07,411] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [2024-03-30 16:56:07,437] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer) [2024-03-30 16:56:07,445] INFO [zk-broker-0-to-controller-alter-partition-channel-manager]: Starting (kafka.server.NodeToControllerRequestThread) [2024-03-30 16:56:07,493] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,495] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,495] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,499] INFO [ExpirationReaper-0-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,500] INFO [ExpirationReaper-0-RemoteFetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,520] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler) [2024-03-30 16:56:07,521] INFO [AddPartitionsToTxnSenderThread-0]: Starting (kafka.server.AddPartitionsToTxnManager) [2024-03-30 16:56:07,587] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient) [2024-03-30 16:56:07,610] INFO Stat of the created znode at /brokers/ids/0 is: 137858,137858,1711788967604,1711788967604,1,0,0,72061991566639105,214,0,137858 (kafka.zk.KafkaZkClient) [2024-03-30 16:56:07,611] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://LAPTOP-78C0012V:9092, czxid (broker epoch): 137858 (kafka.zk.KafkaZkClient) [2024-03-30 16:56:07,682] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,709] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,710] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,733] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator) [2024-03-30 16:56:07,740] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator) [2024-03-30 16:56:07,769] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator) [2024-03-30 16:56:07,773] INFO [TxnMarkerSenderThread-0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager) [2024-03-30 16:56:07,773] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator) [2024-03-30 16:56:07,851] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2024-03-30 16:56:07,895] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread) [2024-03-30 16:56:07,906] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Enabling request processing. (kafka.network.SocketServer) [2024-03-30 16:56:07,910] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.DataPlaneAcceptor) [2024-03-30 16:56:07,921] INFO Kafka version: 3.7.0 (org.apache.kafka.common.utils.AppInfoParser) [2024-03-30 16:56:07,922] INFO Kafka commitId: 2ae524ed625438c5 (org.apache.kafka.common.utils.AppInfoParser) [2024-03-30 16:56:07,926] INFO Kafka startTimeMs: 1711788967916 (org.apache.kafka.common.utils.AppInfoParser) [2024-03-30 16:56:07,929] INFO [KafkaServer id=0] started (kafka.server.KafkaServer) [2024-03-30 16:56:08,391] INFO [zk-broker-0-to-controller-forwarding-channel-manager]: Recorded new controller, from now on will use node LAPTOP-78C0012V:9092 (id: 0 rack: null) (kafka.server.NodeToControllerRequestThread) [2024-03-30 16:56:08,420] INFO [zk-broker-0-to-controller-alter-partition-channel-manager]: Recorded new controller, from now on will use node LAPTOP-78C0012V:9092 (id: 0 rack: null) (kafka.server.NodeToControllerRequestThread) |