Windows Kafka ERROR Failed to clean up log for __consumer_offsets

小渣渣

回顾：

Kafka 配置 JMX 远程监控
https://www.itsvse.com/thread-9968-1-1.html

Kafka 配置参数详解
https://www.itsvse.com/thread-9965-1-1.html

Kafka 手动设置 offset 偏移量
https://www.itsvse.com/thread-9641-1-1.html

.NET/C# 消息队列之 Kafka 操作[附源码]
https://www.itsvse.com/thread-9619-1-1.html

Windows Kafka 错误如下：

[2021-07-06 09:05:54,809] ERROR Failed to clean up log for __consumer_offsets-42 in dir C:\tmp\kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: C:\tmp\kafka-logs\__consumer_offsets-42\00000000000000000000.timeindex.cleaned -> C:\tmp\kafka-logs\__consumer_offsets-42\00000000000000000000.timeindex.swap:


[2021-07-06 09:06:06,460] ERROR [Log partition=__consumer_offsets-42, dir=C:\tmp\kafka-logs] Could not find offset index file corresponding to log file C:\tmp\kafka-logs\__consumer_offsets-42\00000000000000000000.log, recovering segment and rebuilding index files... (kafka.log.Log)

[2021-07-06 09:06:10,607] ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '72057597827809280' does not match current session '72057597827809313' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2021-07-06 09:06:10,616] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
        at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1904)
        at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1842)
        at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1809)
        at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:96)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:308)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)

临时解决方案

停止 zookeeper 和 Kafka ，删除 C:\tmp\kafka-logs\__consumer_offsets-42\ 文件夹下面的所有文件。

小渣渣

kafka进程不定期挂掉。

1、ERROR Failed to clean up log for __consumer_offsets-30 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)

2、linux会定时清理/tmp目录下的文件，我的kafka日志文件目录正是放在了/tmp/kafka-logs目录下，导致被定时给清理掉了，所以kafka在尝试读取或追加日志时就会出错。
grep log.dirs /opt/kafka_2.12-2.3.0/config/server.properties
/tmp/kafka-logs
3、修改
log.dirs=/opt/kafka_2.12-2.3.0/kafka-logs/
4、重启kafka
5、添加kafka日志目录到清理白名单中
centos7下/tmp目录的清理由服务systemd负责，其相关配置文件在/usr/lib/tmpfiles.d目录下，我们修改配置文件tmp.conf，将kafka日志目录加进去，
#防止删除kafka日志文件
X /tmp/kafka-logs

centos6下/tmp目录的清理是通过tmpwatch来实现的，tmpwatch则依赖于cron的定时调度，调度文件为/etc/cron.daily/tmpwatch
#防止删除kafka日志文件
X /tmp/kafka-logs

小渣渣

【转】Kafka Windows 系统运行一段时间就挂掉解决方案
https://www.itsvse.com/thread-9984-1-1.html