Hi, I deployed strimzi kafka on my env using the operator but found that kafka is failed to start as zookeeper keeps restarting. After I did the following things just for troubleshooting:

  • Scale down the strimzi-kafka-operator pod to 0
  • Scale down kafka pods to 0
  • Remove the liveness and readiness probes for the zookeeper pods
  • Decrease the replica of zookeeper pods to 1
  • I can see there's only one zookeeper pod launched but still keeps restarting every 60s. When checking the pod logs, it seems everything looks good. The pod logs will stop at the last line, "Peer state changed: leading ... ", till pod restarted. Any idea why it happens?

    Here is the log details:

    # kubectl logs -n my-kafka -f my-zookeeper-0
    Detected Zookeeper ID 1
    Preparing truststore
    Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
    Certificate was added to keystore
    Preparing truststore is complete
    Looking for the right CA
    Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
    Preparing keystore for client and quorum listeners
    Preparing keystore for client and quorum listeners is complete
    Starting Zookeeper with configuration:
    # The directory where the snapshot is stored.
    dataDir=/var/lib/zookeeper/data
    # Other options
    4lw.commands.whitelist=*
    standaloneEnabled=false
    reconfigEnabled=true
    clientPort=12181
    clientPortAddress=127.0.0.1
    # TLS options
    serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
    ssl.clientAuth=need
    ssl.quorum.clientAuth=need
    secureClientPort=2181
    sslQuorum=true
    ssl.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
    ssl.trustStore.password=[hidden]
    ssl.trustStore.type=PKCS12
    ssl.quorum.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
    ssl.quorum.trustStore.password=[hidden]
    ssl.quorum.trustStore.type=PKCS12
    ssl.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
    ssl.keyStore.password=[hidden]
    ssl.keyStore.type=PKCS12
    ssl.quorum.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
    ssl.quorum.keyStore.password=[hidden]
    ssl.quorum.keyStore.type=PKCS12
    # Provided configuration
    tickTime=2000
    initLimit=5
    syncLimit=2
    autopurge.purgeInterval=1
    # Zookeeper nodes configuration
    server.1=my-zookeeper-0.my-zookeeper-nodes.my-kafka.svc:2888:3888:participant;127.0.0.1:12181
    + exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/zookeeper-server-start.sh /tmp/zookeeper.properties
    ... ...
    2022-09-15 13:55:25,898 INFO PrepRequestProcessor (sid:1) started, reconfigEnabled=true (org.apache.zookeeper.server.PrepRequestProcessor) [ProcessThread(sid:1 cport:-1):]
    2022-09-15 13:55:25,899 INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 13:55:25,899 INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 13:55:25,908 INFO Peer state changed: leading - broadcast (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
            

    @scholzj Thanks for your prompt reply! Which configuration and logs are you referring to? I thought for the zookeeper configuration, it has been included in the above logs, or you want to check the full logs of the zookeeper pod? I pasted the full logs as below. Hope that helps.

    Detected Zookeeper ID 1
    Preparing truststore
    Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
    Certificate was added to keystore
    Preparing truststore is complete
    Looking for the right CA
    Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
    Preparing keystore for client and quorum listeners
    Preparing keystore for client and quorum listeners is complete
    Starting Zookeeper with configuration:
    # The directory where the snapshot is stored.
    dataDir=/var/lib/zookeeper/data
    # Other options
    4lw.commands.whitelist=*
    standaloneEnabled=false
    reconfigEnabled=true
    clientPort=12181
    clientPortAddress=127.0.0.1
    # TLS options
    serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
    ssl.clientAuth=need
    ssl.quorum.clientAuth=need
    secureClientPort=2181
    sslQuorum=true
    ssl.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
    ssl.trustStore.password=[hidden]
    ssl.trustStore.type=PKCS12
    ssl.quorum.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
    ssl.quorum.trustStore.password=[hidden]
    ssl.quorum.trustStore.type=PKCS12
    ssl.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
    ssl.keyStore.password=[hidden]
    ssl.keyStore.type=PKCS12
    ssl.quorum.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
    ssl.quorum.keyStore.password=[hidden]
    ssl.quorum.keyStore.type=PKCS12
    # Provided configuration
    tickTime=20000
    initLimit=5
    syncLimit=2
    autopurge.purgeInterval=1
    # Zookeeper nodes configuration
    server.1=my-zookeeper-0.my-zookeeper-nodes.my-kafka.svc:2888:3888:participant;127.0.0.1:12181
    + exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/zookeeper-server-start.sh /tmp/zookeeper.properties
    2022-09-15 14:22:15,696 INFO Reading configuration from: /tmp/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
    2022-09-15 14:22:15,702 INFO clientPortAddress is 127.0.0.1:12181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
    2022-09-15 14:22:15,702 INFO secureClientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
    2022-09-15 14:22:15,705 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
    2022-09-15 14:22:15,705 INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
    2022-09-15 14:22:15,705 INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
    2022-09-15 14:22:15,713 WARN No server failure will be tolerated. You need at least 3 servers. (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
    2022-09-15 14:22:15,724 INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [main]
    2022-09-15 14:22:15,725 INFO autopurge.purgeInterval set to 1 (org.apache.zookeeper.server.DatadirCleanupManager) [main]
    2022-09-15 14:22:15,726 INFO Purge task started. (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask]
    2022-09-15 14:22:15,730 INFO Log4j 1.2 jmx support found and enabled. (org.apache.zookeeper.jmx.ManagedUtil) [main]
    2022-09-15 14:22:15,731 INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [PurgeTask]
    2022-09-15 14:22:15,740 INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream) [PurgeTask]
    2022-09-15 14:22:15,742 INFO Purge task completed. (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask]
    2022-09-15 14:22:15,747 INFO Starting quorum peer, myid=1 (org.apache.zookeeper.server.quorum.QuorumPeerMain) [main]
    2022-09-15 14:22:15,763 INFO ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@159f197 (org.apache.zookeeper.server.ServerMetrics) [main]
    2022-09-15 14:22:15,814 INFO zookeeper.client.portUnification=false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:15,814 INFO zookeeper.netty.advancedFlowControl.enabled = false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:15,814 INFO handshakeThrottlingEnabled = false, zookeeper.netty.server.outstandingHandshake.limit = -1 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:15,858 INFO Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [main]
    2022-09-15 14:22:15,859 WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory) [main]
    2022-09-15 14:22:15,859 INFO zookeeper.client.portUnification=false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:15,859 INFO zookeeper.netty.advancedFlowControl.enabled = false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:15,859 INFO handshakeThrottlingEnabled = false, zookeeper.netty.server.outstandingHandshake.limit = -1 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:15,860 INFO Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [main]
    2022-09-15 14:22:15,860 WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory) [main]
    2022-09-15 14:22:15,864 INFO zookeeper.quorumCnxnTimeoutMs=-1 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,881 INFO Logging initialized @638ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [main]
    2022-09-15 14:22:15,941 WARN o.e.j.s.ServletContextHandler@353352b6{/,null,STOPPED} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler) [main]
    2022-09-15 14:22:15,941 WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler) [main]
    2022-09-15 14:22:15,957 INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [main]
    2022-09-15 14:22:15,957 INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO tickTime set to 20000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO minSessionTimeout set to 40000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO maxSessionTimeout set to 400000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO initLimit set to 5 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO syncLimit set to 2 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,957 INFO connectToLearnerMasterLimit set to 0 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,966 INFO  (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,966 INFO   ______                  _                                           (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO  |___  /                 | |                                          (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO     / /    ___     ___   | | __   ___    ___   _ __     ___   _ __    (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO    / /    / _ \   / _ \  | |/ /  / _ \  / _ \ | '_ \   / _ \ | '__| (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO   / /__  | (_) | | (_) | |   <  |  __/ |  __/ | |_) | |  __/ | |     (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO  /_____|  \___/   \___/  |_|\_\  \___|  \___| | .__/   \___| |_| (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO                                               | |                      (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO                                               |_|                      (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,967 INFO  (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:host.name=my-zookeeper-0.my-zookeeper-nodes.my-kafka.svc.cluster.local (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.version=11.0.14 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.vendor=Red Hat, Inc. (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_5.x86_64 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.class.path=/opt/kafka/bin/../libs/accessors-smart-2.4.7.jar:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/automaton-1.11-8.jar:/opt/kafka/bin/../libs/checker-qual-3.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-3.1.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-3.1.0.jar:/opt/kafka/bin/../libs/connect-file-3.1.0.jar:/opt/kafka/bin/../libs/connect-json-3.1.0.jar:/opt/kafka/bin/../libs/connect-mirror-3.1.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-3.1.0.jar:/opt/kafka/bin/../libs/connect-runtime-3.1.0.jar:/opt/kafka/bin/../libs/connect-transforms-3.1.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.5.82.jar:/opt/kafka/bin/../libs/error_prone_annotations-2.3.4.jar:/opt/kafka/bin/../libs/failureaccess-1.0.1.jar:/opt/kafka/bin/../libs/generex-1.0.2.jar:/opt/kafka/bin/../libs/gson-2.8.6.jar:/opt/kafka/bin/../libs/guava-30.1-jre.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/j2objc-annotations-1.3.jar:/opt/kafka/bin/../libs/jackson-annotations-2.12.3.jar:/opt/kafka/bin/../libs/jackson-core-2.12.3.jar:/opt/kafka/bin/../libs/jackson-databind-2.12.3.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.12.3.jar:/opt/kafka/bin/../libs/jackson-dataformat-yaml-2.12.3.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.12.3.jar:/opt/kafka/bin/../libs/jackson-datatype-jsr310-2.12.3.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.12.3.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.12.3.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.12.3.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.12.3.jar:/opt/kafka/bin/../libs/jaeger-client-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-core-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-thrift-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-tracerresolver-1.6.0.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.34.jar:/opt/kafka/bin/../libs/jersey-common-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka/bin/../libs/jersey-server-2.34.jar:/opt/kafka/bin/../libs/jetty-client-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-http-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-io-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-security-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-server-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-util-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jline-3.12.1.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.16.1.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jose4j-0.7.8.jar:/opt/kafka/bin/../libs/json-path-2.6.0.jar:/opt/kafka/bin/../libs/json-smart-2.4.7.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka-agent-0.28.0.jar:/opt/kafka/bin/../libs/kafka-clients-3.1.0.jar:/opt/kafka/bin/../libs/kafka-env-var-config-provider-1.0.0.jar:/opt/kafka/bin/../libs/kafka-kubernetes-config-provider-1.0.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-3.1.0.jar:/opt/kafka/bin/../libs/kafka-metadata-3.1.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-plain-0.10.0.jar:/opt/kafka/bin/../libs/kafka-quotas-plugin-0.2.0.jar:/opt/kafka/bin/../libs/kafka-raft-3.1.0.jar:/opt/kafka/bin/../libs/kafka-server-common-3.1.0.jar:/opt/kafka/bin/../libs/kafka-shell-3.1.0.jar:/opt/kafka/bin/../libs/kafka-storage-3.1.0.jar:/opt/kafka/bin/../libs/kafka-storage-api-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-3.1.0.jar:/opt/kafka/bin/../libs/kafka-tools-3.1.0.jar:/opt/kafka/bin/../libs/kafka_2.13-3.1.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.4.10.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.4.0.jar:/opt/kafka/bin/../libs/kubernetes-client-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-apps-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-batch-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-certificates-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-common-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-core-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-extensions-5.12.0.jar:/opt/kafka/bin/../libs/libthrift-0.14.1.jar:/opt/kafka/bin/../libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/logging-interceptor-3.12.12.jar:/opt/kafka/bin/../libs/lz4-java-1.8.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.8.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/metrics-core-4.1.12.1.jar:/opt/kafka/bin/../libs/mirror-maker-2-extensions-1.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent-0.28.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.68.Final.jar:/opt/kafka/bin/../libs/nimbus-jose-jwt-9.10.jar:/opt/kafka/bin/../libs/okhttp-4.9.0.jar:/opt/kafka/bin/../libs/okio-2.8.0.jar:/opt/kafka/bin/../libs/opa-authorizer-1.4.0.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-kafka-client-0.1.15.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../libs/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-6.22.1.1.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.4.4.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-1.0.0.jar:/opt/kafka/bin/../libs/scala-library-2.13.6.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.3.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.6.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snakeyaml-1.27.jar:/opt/kafka/bin/../libs/snappy-java-1.1.8.4.jar:/opt/kafka/bin/../libs/tracing-agent-0.28.0.jar:/opt/kafka/bin/../libs/trogdor-3.1.0.jar:/opt/kafka/bin/../libs/zjsonpatch-0.3.0.jar:/opt/kafka/bin/../libs/zookeeper-3.6.3.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.6.3.jar:/opt/kafka/bin/../libs/zstd-jni-1.5.0-4.jar (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:os.version=4.18.0-305.49.1.el8_4.x86_64 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:user.name=1000650000 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:user.home=/ (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:user.dir=/opt/kafka (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:os.memory.free=239MB (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:os.memory.max=15468MB (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO Server environment:os.memory.total=254MB (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO zookeeper.skipACL=="yes", ACL checks will be skipped (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,968 INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,969 INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,969 INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,969 INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,969 INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer) [main]
    2022-09-15 14:22:15,972 INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory) [main]
    2022-09-15 14:22:15,972 INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory) [main]
    2022-09-15 14:22:15,973 INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [main]
    2022-09-15 14:22:15,973 INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase) [main]
    2022-09-15 14:22:15,981 INFO Using TLS encrypted quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,981 INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,981 INFO multiAddress.enabled set to false (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,981 INFO multiAddress.reachabilityCheckEnabled set to true (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,981 INFO multiAddress.reachabilityCheckTimeoutMs set to 1000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,981 INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,981 INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:15,982 INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap) [main]
    2022-09-15 14:22:15,984 INFO The digest value is empty in snapshot (org.apache.zookeeper.server.DataTree) [main]
    2022-09-15 14:22:15,987 INFO Snapshot loaded in 5 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase) [main]
    2022-09-15 14:22:15,988 INFO binding to port /127.0.0.1:12181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:16,047 INFO bound to port 12181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:16,047 INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:16,049 INFO bound to port 2181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
    2022-09-15 14:22:16,051 INFO jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.14+9-LTS (org.eclipse.jetty.server.Server) [main]
    2022-09-15 14:22:16,075 INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session) [main]
    2022-09-15 14:22:16,076 INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session) [main]
    2022-09-15 14:22:16,077 INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session) [main]
    2022-09-15 14:22:16,080 WARN ServletContext@o.e.j.s.ServletContextHandler@353352b6{/,null,STARTING} has uncovered http methods for path: /* (org.eclipse.jetty.security.SecurityHandler) [main]
    2022-09-15 14:22:16,089 INFO Started o.e.j.s.ServletContextHandler@353352b6{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
    2022-09-15 14:22:16,096 INFO Started ServerConnector@576d5deb{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector) [main]
    2022-09-15 14:22:16,096 INFO Started @853ms (org.eclipse.jetty.server.Server) [main]
    2022-09-15 14:22:16,096 INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer) [main]
    2022-09-15 14:22:16,096 INFO Using 40000ms as the quorum cnxn socket timeout (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
    2022-09-15 14:22:16,101 INFO Election port bind maximum retries is 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [main]
    2022-09-15 14:22:16,102 INFO zookeeper.fastleader.minNotificationInterval=200 (org.apache.zookeeper.server.quorum.FastLeaderElection) [main]
    2022-09-15 14:22:16,102 INFO zookeeper.fastleader.maxNotificationInterval=60000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [main]
    2022-09-15 14:22:16,105 INFO Creating TLS-only quorum server socket (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-my-zookeeper-0.my-zookeeper-nodes.my-kafka.svc/10.254.17.16:3888]
    2022-09-15 14:22:16,107 INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider) [main]
    2022-09-15 14:22:16,108 INFO 1 is accepting connections now, my election bind port: my-zookeeper-0.my-zookeeper-nodes.my-kafka.svc/10.254.17.16:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-my-zookeeper-0.my-zookeeper-nodes.my-kafka.svc/10.254.17.16:3888]
    2022-09-15 14:22:16,111 INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,112 INFO New election. My id = 1, proposed zxid=0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,120 INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x30, n.zxid:0x0, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
    2022-09-15 14:22:16,321 INFO Peer state changed: leading (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,322 INFO LEADING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,325 INFO zookeeper.leader.maxConcurrentSnapSyncs = 10 (org.apache.zookeeper.server.quorum.LearnerMaster) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,325 INFO zookeeper.leader.maxConcurrentDiffSyncs = 100 (org.apache.zookeeper.server.quorum.LearnerMaster) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,325 INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,325 INFO zookeeper.leader.ackLoggingFrequency = 1000 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,325 INFO zookeeper.leader.maxTimeToWaitForEpoch = -1ms (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,328 INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,328 INFO minSessionTimeout set to 40000 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,329 INFO maxSessionTimeout set to 400000 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,329 INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,330 INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,330 INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,330 INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,330 INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,331 INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,331 INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,331 INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,333 INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,333 INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,333 INFO Created server with tickTime 20000 minSessionTimeout 40000 maxSessionTimeout 400000 clientPortListenBacklog -1 datadir /var/lib/zookeeper/data/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,336 INFO LEADING - LEADER ELECTION TOOK - 224 MS (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,338 INFO Peer state changed: leading - discovery (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,338 INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,358 INFO Snapshot taken in 20 ms (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,394 INFO Peer state changed: leading - synchronization (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,394 INFO Have quorum of supporters, sids: [[1],[1]]; starting up and setting last processed zxid: 0x3100000000 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,403 INFO Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1 (org.apache.zookeeper.server.quorum.CommitProcessor) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,404 INFO Configuring CommitProcessor with 1 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,410 INFO PrepRequestProcessor (sid:1) started, reconfigEnabled=true (org.apache.zookeeper.server.PrepRequestProcessor) [ProcessThread(sid:1 cport:-1):]
    2022-09-15 14:22:16,411 INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,412 INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
    2022-09-15 14:22:16,422 INFO Peer state changed: leading - broadcast (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
            

    @scholzj Since I've stopped the operator (scale replica down to 0), just want to filter out noise and narrow down the issue, I thought the logs for the operator and the Kafka CR would not be needed in order to analyze this particular issue, i.e.: kafka failed to launch as its dependency, zookeeper, keeps restarting for some reason.

    But if you think that's needed, I can share them all here. Just that, once I relaunch the operator, everything that I made against the kafka and zookeeper pods manually will be reset, e.g.: the liveness and readiness probes will go back, the number of zookeeper and kafka pods will go back to 3. :-)

    FYI: Below is the Kafka CR. Actually, I see it was in Ready status, which looks good. Only the zookeeper pods keeps restarting, which is very interesting.

    apiVersion: kafka.strimzi.io/v1beta2
    kind: Kafka
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"Kafka","metadata":{"annotations":{},"name":"my","namespace":"my-kafka"},"spec":{"kafka":{"config":{"offsets.topic.replication.factor":3,"transaction.state.log.min.isr":3,"transaction.state.log.replication.factor":3},"listeners":[{"name":"plain","port":9092,"tls":false,"type":"internal"}],"replicas":3,"storage":{"type":"jbod","volumes":[{"deleteClaim":true,"id":0,"size":"50Gi","type":"persistent-claim"}]},"version":"3.1.0"},"zookeeper":{"replicas":3,"storage":{"deleteClaim":true,"size":"10Gi","type":"persistent-claim"}}}}
      creationTimestamp: "2022-08-22T06:32:56Z"
      generation: 1
      managedFields:
      - apiVersion: kafka.strimzi.io/v1beta2
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:annotations:
              .: {}
              f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:spec:
            .: {}
            f:kafka:
              .: {}
              f:config:
                .: {}
                f:offsets.topic.replication.factor: {}
                f:transaction.state.log.min.isr: {}
                f:transaction.state.log.replication.factor: {}
              f:listeners: {}
              f:replicas: {}
              f:storage:
                .: {}
                f:type: {}
                f:volumes: {}
              f:version: {}
            f:zookeeper:
              .: {}
              f:replicas: {}
              f:storage:
                .: {}
                f:deleteClaim: {}
                f:size: {}
                f:type: {}
        manager: kubectl-client-side-apply
        operation: Update
        time: "2022-08-22T06:32:56Z"
      - apiVersion: kafka.strimzi.io/v1beta2
        fieldsType: FieldsV1
        fieldsV1:
          f:status:
            .: {}
            f:clusterId: {}
            f:conditions: {}
            f:listeners: {}
            f:observedGeneration: {}
        manager: okhttp
        operation: Update
        time: "2022-09-15T04:44:01Z"
      name: my
      namespace: my-kafka
      resourceVersion: "26675903"
      uid: 98719e78-17a8-47f1-a631-7f224d9b9cb2
    spec:
      kafka:
        config:
          offsets.topic.replication.factor: 3
          transaction.state.log.min.isr: 3
          transaction.state.log.replication.factor: 3
        listeners:
        - name: plain
          port: 9092
          tls: false
          type: internal
        replicas: 3
        storage:
          type: jbod
          volumes:
          - deleteClaim: true
            id: 0
            size: 50Gi
            type: persistent-claim
        version: 3.1.0
      zookeeper:
        replicas: 3
        storage:
          deleteClaim: true
          size: 10Gi
          type: persistent-claim
    status:
      clusterId: Spfxxs89Q46wAC7ib_qsJg
      conditions:
      - lastTransitionTime: "2022-09-15T04:40:06.923442Z"
        message: default.replication.factor option is not configured. It defaults to 1
          which does not guarantee reliability and availability. You should configure
          this option in .spec.kafka.config.
        reason: KafkaDefaultReplicationFactor
        status: "True"
        type: Warning
      - lastTransitionTime: "2022-09-15T04:40:06.924515Z"
        message: min.insync.replicas option is not configured. It defaults to 1 which
          does not guarantee reliability and availability. You should configure this option
          in .spec.kafka.config.
        reason: KafkaMinInsyncReplicas
        status: "True"
        type: Warning
      - lastTransitionTime: "2022-09-15T04:44:01.903Z"
        status: "True"
        type: Ready
      listeners:
      - addresses:
        - host: my-kafka-bootstrap.my-kafka.svc
          port: 9092
        bootstrapServers: my-kafka-bootstrap.my-kafka.svc:9092
        name: plain
        type: plain
      observedGeneration: 1