这个问题我也是无意间碰到的,之前一直是使用单机的ActiveMQ,所以也没这个问题,但是做集群时碰到这个问题,问题是这样子出现的:

首先,我准备了三台虚拟机,然后使用 Replicated LevelDB 的方式配置集群,配置如下:

        <persistenceAdapter>
            <!--<kahaDB directory="${activemq.data}/kahadb"/>-->
            <replicatedLevelDB
               directory="${activemq.data}/leveldb"
               replicas="3"
               bind="tcp://0.0.0.0:61619"
               zkAddress="192.168.209.133:2181,192.168.209.134:2181,192.168.209.135:2181"
               zkPath="/activemq"
               hostname="test3"
        </persistenceAdapter>

之后使用 ./activemq console 命令将三个虚拟机中的ActiveMQ成功启动。

接着我断开master节点,发现剩下两台slave节点会自动选举一个成为master节点,再把原来的master节点启动,就相当于一个新的slave节点加入了集群。

这样子,我以为集群就是好的了,然后写代码,发布消费,都是很正常的。然后问题就出现了。

我在把master节点关闭,本来期望着剩下的节点能自动选一个成为master节点,可结果发现本该成为master节点的服务抛出了异常,重启都无效,而且此时代码也访问不了,ActiveMQ的管理后台也访问不了,就相当于整个集群挂了!!!

报错大概是这样子的:

    java.io.IOException: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)
	at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)
	at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:667)
	at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)
	at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
	at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
	at org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:110)
	at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
	at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:230)
	at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
   Caused by: java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3472)
	at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3476)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
	at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:90)
	at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:78)
	at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:73)
	at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:1011)
	at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:952)
	at org.iq80.leveldb.impl.DbImpl.recoverLogFile(DbImpl.java:564)
	at org.iq80.leveldb.impl.DbImpl.<init>(DbImpl.java:209)
	at org.iq80.leveldb.impl.Iq80DBFactory.open(Iq80DBFactory.java:82)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_init$2.apply$mcV$sp(LevelDBClient.scala:687)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_init$2.apply(LevelDBClient.scala:667)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_init$2.apply(LevelDBClient.scala:667)
	at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549)
	... 11 more
   Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 63 more

然后花了半个小时的时间百度(Google进不去,可惜了),得到的解决方法主要有四种:

1、 移除lib目录中的pax-url-aether-1.5.2.jar包(对我情况无效,lib目录下没有这个包,园友们可以试试)

2、注释或者删除conf/activemq.xml中id="logQuery"的bean,它的class="io.fabric8.insight.log.log4j.Log4jLogQuery"( 对我情况无效,园友们可以试试

3、清除所有的数据,也就是删除每个节点的上面配置的 replicatedLevelDB 节点中 directory 属性指向的目录 ,比如我这里就是删除每个节点下的 data/leveldb 目录(证实有效)

4、换低版本的activemq试试(这个没试)

唯一能解决这个问题的办法是清除所有数据,这让我难以接受,如果哪天线上环境不小心把master节点停了,然道要清除所有数据才能重启?这种情况谁都无法接受。

只能自己想办法了,看异常信息,开头是:

  java.io.IOException: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)
	at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)
	at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:667)
	at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)
	at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
	at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
	at org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:110)
	at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
	at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:230)
	at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
     ....

这个大致信息可以理解为,本节点在作为master节点启动时出错了,在 IOExceptionSupport.create()方法想创建一个对象时抛出异常,异常信息是下面的:

  Caused by: java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 63 more

这个明显是说找不到类:com.google.common.util.concurrent.internal.InternalFutureFailureAccess,那好吧,我给你去找这个类。

于是接着百度,果然度娘最喜欢辜负了苦心人,没有!

这时脑子突然机灵了一下,为什么不去Maven上面找找呢?maven直通车: https://mvnrepository.com/

然后我搜索 com.google.common.util.concurrent.internal.InternalFutureFailureAccess 这个类,果然在上面找到一个包存在这个类:

进去看详情,好家伙,这个包就两个版本,还两三年没更新了,然后我选择1.0.1版本进去,下载jar包:

为避免不熟悉maven的朋友找不到,我将两个版本的jar包都下载再来了,因为担心每个版本依赖不一样,大家可以从百度网盘获取: https://pan.baidu.com/s/1T7sxBYuqqrPnyGWU4VJz9g (提取码: mtaj )

jar下载下来后,我将包放到每个节点ActiveMQ的lib目录下,然后重启每个节点, 问题完美解决!!!!

这个问题写这么多,是因为确实度娘上的资料太少了,相信以后还有很多园友会碰到这个问题了,特此记一下!

- - - - - - - - - - - - - - - - - - - 分割线- - - - - - - - - - - - - - - - - - -

另外,还有一个小问题,在使用过程中,发现ActiveMQ启动后,抛出下面的异常:An IOException was thrown (should never happen in this method).

  java.lang.RuntimeException: An IOException was thrown (should never happen in this method).
	at org.apache.activemq.leveldb.record.CollectionKey$Buffer.bean(CollectionKey.java:264)
	at org.apache.activemq.leveldb.record.CollectionKey$Buffer.getKey(CollectionKey.java:284)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1$$anonfun$apply$mcV$sp$4.apply(LevelDBClient.scala:757)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1$$anonfun$apply$mcV$sp$4.apply(LevelDBClient.scala:740)
	at scala.Option.map(Option.scala:146)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply$mcV$sp(LevelDBClient.scala:740)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:707)
	at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:707)
	at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549)
	at org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:706)
	at org.apache.activemq.leveldb.replicated.SlaveLevelDBStore$$anonfun$send_wal_ack$1.apply$mcV$sp(SlaveLevelDBStore.scala:185)
	at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
  Caused by: java.io.EOFException
	at org.fusesource.hawtbuf.proto.CodedInputStream.readRawByte(CodedInputStream.java:346)
	at org.fusesource.hawtbuf.proto.CodedInputStream.readRawVarint32(CodedInputStream.java:240)
	at org.fusesource.hawtbuf.proto.CodedInputStream.skipField(CodedInputStream.java:117)
	at org.apache.activemq.leveldb.record.CollectionKey$Bean.mergeUnframed(CollectionKey.java:172)
	at org.apache.activemq.leveldb.record.CollectionKey$Buffer.bean(CollectionKey.java:259)
	... 14 more

虽然有这个异常,但是ActiveMQ还是能正常启动正常使用,不过我还是有些担心,所以就查了一下,结果发现后台跑了两个ActiveMQ进程,杀掉再启动还是会再启动两个进程,没办法,只好试试重启一下机器,结果问题解决