Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Mongodb 4.2.8: Unable to add session into the cache because the number of active sessions is too high

Suddenly we started to experience following problem during connection to Mongo: {u'code': 261, u'ok': 0.0, u'$clusterTime': {u'clusterTime': Timestamp(1614532995, 3141), u'signature': {u'keyId': 0L, u'hash': Binary('\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0)}}, u'codeName': u'TooManyLogicalSessions', u'operationTime': Timestamp(1614532995, 3141), u'errmsg': u'Unable to add session into the cache because the number of active sessions is too high'} .

This happens for any connection driver we use:

And happens not for every query, but roughly for 30-40% of all queries.

Meanwhile, maxSession has default value (1000000) and I have following data from db status:

"logicalSessionRecordCache" : {
        "activeSessionsCount" : 2244,
        "sessionsCollectionJobCount" : 48430,
        "lastSessionsCollectionJobDurationMillis" : 0,
        "lastSessionsCollectionJobTimestamp" : ISODate("2021-02-28T16:56:03.438Z"),
        "lastSessionsCollectionJobEntriesRefreshed" : 0,
        "lastSessionsCollectionJobEntriesEnded" : 0,
        "lastSessionsCollectionJobCursorsClosed" : 0,
        "transactionReaperJobCount" : 49566,
        "lastTransactionReaperJobDurationMillis" : 1,
        "lastTransactionReaperJobTimestamp" : ISODate("2021-02-28T17:03:17.631Z"),
        "lastTransactionReaperJobEntriesCleanedUp" : 0,
        "sessionCatalogSize" : 33
Once the issue has been discovered, I checked periodically a number of sessions in config.system.sessions. It varied from 11k to 560k (most of the time it was between 80k and 350k), which seems to be quite high.
However, the problem remained disregard of the amount of sessions.
An error is sudden, we have the same load as before (I don't know the number of sessions we used to have before but we didn't add any new clients - we have about 3k connections.
There is no sharding, only a replica (one primary and one secondary).
I would really appreciate any advice on how to overcome this problem.
UPD: another thing that looks weird for me:
> db.system.sessions.count()
416068
> db.currentOp(true).inprog.length
how is it possible to have such a difference?
                I meant despite of number of seasons I experienced the issue. The application errors related to the problem appeared when Mongo had different number if sessions, from 11 000 to 500 000.
– Andrey Rusanov
                Mar 1, 2021 at 1:27
                I also did an investigation and it seems that normally I used to have up to 3000 of sessions before, however I can't really prove it (I experimented with same resources setup locally)
– Andrey Rusanov
                Mar 1, 2021 at 1:29
                It is unclear whether your question gives numbers for the same moment in time or three different moments. You need to take one particular instant, clearly document everything you know about it, and clearly state separately which of the observations also apply at other instants.
– D. SM
                Mar 1, 2021 at 2:15
Most likely you are going to need to do some debugging in your application to figure out where you are leaking sessions.
Update driver and server to most recent versions.
Identify where your application is using explicit sessions. Explicit sessions are those that you start via a start_session call. The driver also uses sessions automatically by itself, those are called implicit sessions.
Lacking evidence to the contrary, you have a session leak. Use https://docs.mongodb.com/manual/reference/command/killAllSessions/ to destroy all sessions, then graph the number of active sessions over time to see what your trend looks like.
Review your code and match every start_session call with how that session is ended (if any). If you do not use a scoped API like https://docs.mongodb.com/ruby-driver/master/tutorials/ruby-driver-sessions/#creating-a-session-from-a-mongo-client you need to CAREFULLY consider where each of the sessions is going to get destroyed.
Check your code for no timeout cursors. Those would probably hold session references (explicit or implicit).
Going by the information you provided in the question, my guess is your session state inspection isn't done properly so go over that again and make sure you are looking at the right things.
                Thank you, I am going to go through your points and look for any possible issue. The most confusing part is that application has been untouched for 5-6 months, so I started to blame Mongo :)
– Andrey Rusanov
                Mar 2, 2021 at 6:38
                An issue appeared once and hasn't came back yet; I also don't really see any explicit sessions atm in the code. However, I really appreciate your help and detailed answer
– Andrey Rusanov
                Mar 7, 2021 at 7:45
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.