Sign all brokers certificates with the generated CA Export the certificate from the keystore:

keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file

Sign it with the CA:

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}

Import both the certificate of the CA and the signed certificate into the keystore:

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert

keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed

Import CA to client truststore and broker/server truststore:

keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert

Add these line in the configuration server.properties: listeners=PLAINTEXT://localhost:9092, SSL://localhost:9192 ssl.client.auth=required ssl.keystore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.keystore.jks ssl.keystore.password=blablabla ssl.key.password=blablabla ssl.truststore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.truststore.jks ssl.truststore.password=blablabla security.inter.broker.protocol=SSL

The problem is that when I start kafka, then I get this error:

[2019-02-26 19:03:59,783] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2019-02-26 19:04:00,011] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9192) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-02-26 19:04:00,178] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9192) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-02-26 19:04:00,319] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9192) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
EDIT:
server.properties:
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://localhost:9092, SSL://localhost:9192
ssl.client.auth=required
ssl.keystore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.keystore.jks
ssl.keystore.password=onailime
ssl.key.password=onailime
ssl.truststore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.truststore.jks
ssl.truststore.password=onailime
security.inter.broker.protocol=SSL
# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/home/xrobot/kafka_2.12-2.1.0/data/kafka
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
zookeeper.properties:
# the directory where the snapshot is stored.
dataDir=/home/xrobot/kafka_2.12-2.1.0/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
probably your hostname and your certificate don't match. add this line to your server.properties file.
ssl.endpoint.identification.algorithm=
From Kafka version 2.0.0 onwards, hostname verification of servers is enabled by default for client connections as well as inter-broker connections. by adding this line, you assign an empty string for ssl.endpoint.identification.algorithm.
                There are different options to disable the hostname validation. You can choose to only disable it for a specific client. See details here cwiki.apache.org/confluence/display/KAFKA/…
– GC001
                Jun 24, 2021 at 3:49
                This worked for me, for my case hostname and certificate are different. This saved my day!
– Mohamed Niyaz
                Aug 7, 2022 at 9:27
                Worked for me too. This property SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG  was set to null before the upgrade. And post upgrade, it started failing as ConcurrentHashmap doesn't allow nulls as keys or values. Debug the code to find where exactly the Null Pointer is originating, and then this comment helped fix it. Thank you!
– shazwashere
                Aug 27, 2022 at 7:23
This problem can be caused by a connection problem or firewall setting.
You can test access with:

openssl s_client -debug -connect servername:port -tls1_2
Answer should be "Verify return code: 0 (ok)
If not you have no access.
This is an old thread but I can share some lessons learnt the hard way:
Authentication failure can happen for a number of reasons. It's necessary to understand what failed in the SSL handshake. A pcap with the SSL handshake messages will definitely help.
If this is is for a client connecting to a broker.
In server.properties you have:
ssl.client.auth=required
It should be
ssl.client.auth=none 
if the clients are not authenticating to the server. In the question there is no step described in which the clients are created their own key/certificate.
Also, just for testing purposes, in the client you can configure:
enable.ssl.certificate.verification=false
This property in false makes the client not validate the server's certificate with the CA. It's useful when the SSL hanshake error is due to server's certificate not validated.
                In my case I had also neglected to restart the correct service, which should be done after updating the JKS.
– DustWolf
                Jul 27 at 9:42
Kafka connect initially did not start in my case because of the ssl handshake error. I can sort it with setting the property ssl.endpoint.identification.algorithm=
Thereafter i got similar error when I create a source connector.
[2022-09-07 15:35:18,817] ERROR [Producer
clientId=connector-producer-SourceConnector-0] Connection to node -1
(kafkaserver:9092) failed authentication due to: SSL handshake failed
(org.apache.kafka.clients.NetworkClient) [2022-09-07 15:35:18,817]
WARN [Producer clientId=connector-producer-SourceConnector-0]
Bootstrap broker kafkaserver:9092 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
the solution for this adding identification algorithm parameter for both consumer and producer.
ssl.endpoint.identification.algorithm=
producer.ssl.endpoint.identification.algorithm=
consumer.ssl.endpoint.identification.algorithm=
Yes this could be happening because of domain name mismatch, but just to add if you have two-way authentication enabled
ssl.client.auth=required
you will have to disable the below flag in both places, broker(server.properties) as well as in you client(producer or consumer) configs
ssl.endpoint.identification.algorithm=""
If your client is java based you can do the following:
props.put(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "");
I see a lot of responses here trying to guess how to solve a SSL handshake problem that is not clear. The logs only show
failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
The problem is that we don't know the reason for SSL handshake failure.
I created a tool helps you determine what is the real error by looking at the network handshake exchange:
https://github.com/rodolk/dlltagent_tools
Then you can decide whether that was due to an expired certificate, not accepted self-signed certificate, unknown CA or any other of the many possible errors. And after hat you can choose one of all the helpful responses.
The tool is prepared for Kubernetes but it can also be used with docker without kubernetes.
In my case, after seeing similar errors, I had to add the host name to the listeners value in my server.properties.
Before:
listeners=SSL://:9093
After:
listeners=SSL://my-host-name:9093
This host name should be the same as the CN-value in your broker certificate. This is the same value you should have for the host.name property.
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.