java - Migrating from MySql: MariaDB server closing client connections unexpectedly

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
We in the process of migrating from MySql to MariaDB due to licensing/commercial usage reasons.
We have successfully replaced the MySql connector jar with MariaDB client jar (first change) and are now trying to replace MySql server with MariaDB server without changing the data files.
All our applications run perfectly for about 8-12 hours after which we see the following exception:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection
Caused by:
org.hibernate.exception.JDBCConnectionException: Cannot open connection
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
        at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
        at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
        at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
        at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:494)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:315)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:257)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
        at $Proxy4.getMessageCountByStatus(Unknown Source)
        at com.onmobile.cmfweb.monitoring.CmfMessagesMonitor.getMessageCounts(CmfMessagesMonitor.java:56)
        at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:270)
        at com.onmobile.cmfshare.MethodInvockingBean.invoke(MethodInvockingBean.java:28)
        at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:212)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:79)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4
        at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
        at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
        at org.mariadb.jdbc.Driver.connect(Driver.java:114)
        at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
        at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
        ... 21 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4
        at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:509)
        at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:669)
        at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:264)
        at org.mariadb.jdbc.Driver.connect(Driver.java:110)
        ... 28 more
Caused by: java.io.EOFException: unexpected end of stream, read 0bytes from 4
        at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:84)
        at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92)
        at org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77)
        at org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67)
        at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:467)
        ... 31 more
The exceptions disappear if MySql server is used again.
From what I have debugged so far MariaDB server is closing the client connections for some reason. I initially suspected of idle connections but we are using Hibernate configurations like 'testOnBorrow' so it shouldn't be the case.
Can anyone help us to find out the cause for this and help us fix the problem? Is there any specific configuration in MariaDB that I should be doing?
I've even run our applications by increasing the value of the param 'interactive_timeout' in the MariaDB server but it did not help.
Btw, we are using Spring-Hibernate and pooling our connections using Commons-pool jar.
Any kind of help will be deeply appreciated.
We encountered the same problem recently, due to a combination of incorrect parameters.
This error is caused by your web instance trying to use a connection that is no longer a valid one.
This can be resolved by making sure the following parameters are correct:
You have a validationQuery configured for your database connection. i.e. validationQuery="SELECT 1" in case of mariadb, in the server configuration.
You have the wait_timeout set to a reasonable value. 8 hours keepAlive seems a bit optimistic, we are now using wait_timeout=180
Make sure the validationInterval, which you can also set in the server configuration (server.xml in case of a tomcat setup), is set to a value that is lower than the wait_timeout value. I reduced the wait_timeout to 15s in one case, with the validationInterval value being higher than that, which caused the error to still appear at times. 
Now it is set to validationInterval=60, combined with the wait_timeout=180 which should catch any broken connections in time.
You can try to increase 'wait_timeout' instead of 'interactive_timeout'. 8-12 hours until error correlates quite well with the default value of wait_timeout which is 10 hours. JDBC driver does not set interactive client flag when connecting, so change to 'interactive_timeout' might have no effect. I also replied to the cross-post in Maria KB ;)
                Thanks once again. I'll post how the test goes after setting 'wait_timeout' by tomorrow or the day after.
– Vivek
                Sep 16, 2013 at 13:16
                Also experiencing the "unexpected end of stream, read 0bytes from 4" ... haven't been able to find a solution so far.
– 4nti
                Oct 16, 2015 at 15:09
                @4nti - there are a few extra things to try in the cross post mentioned - mariadb.com/kb/en/mariadb/… - but it's not clear whether they resolved the problem
– J Richard Snape
                Oct 17, 2015 at 23:24
According to the findings in this article you could get away with two options:
You should try turning innodb_file_per_table off
or...
Increasing the open file limits, this tells you the current value on yout machine $ ulimit -n, should you be running on high spec machines
I also had an error like this one Could not connect to localhost:3306: unexpected end of stream, read 0 bytes from 4
My architecture was an webapp of Tomcat that was inserting data into MariaDB.
I was using the connector version 1.5.1RC!
I switched to the 1.3.1 connector version and now runs fine!
I also faced this issue. The issue goes away with lowering the wait_timeout.
SHOW SESSION VARIABLES LIKE 'wait_timeout'; ->> would be 8 hours. which is huge.
SET session wait_timeout=300; ->> this should fix the issue.
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.