Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I run over 500 000 uids to fetch their associated data via a promise (promiseUserData(uid)). For some reason the memory usage is not cleared from loop to loop.

Roughly 4mb of data is used per loop and is not cleared from memory.

I am ignorant as how to fix this. Promise pooling also results in this issue.

Help would be much appreciated.

Thank you

for (var i = 0; i <= numPasses; i++) {
    var subset = uids.splice(0, 1000);
    var userDataArray = [];
    var promises = [];
    subset.map((uid) => {
        promises.push(promiseUserData(uid).then((userData) => {
            if (userData) {
                userDataArray.push(userData);
    await Promise.all(promises).catch((err) => {
        console.log('Error in Promises.all: ', err);
                is not cleared from memory. -> how do you measure that? Which timespan are you talking about? How much memory is left? ( cause some GCs only stop the world when neccessary)
– Jonas Wilms
                Nov 29, 2017 at 14:00
                I measured this with both memwatch-next and process.memoryUsage().heapUsed.  This code is running on nodejs and I have also manually called the GC but that doesn't seem to help.
– James Barnes
                Nov 29, 2017 at 14:02
                What if you use let in the for loop instead of var? var is function scope and let is block scope, maybe using let will clear up the promises array, not sure why re assigning it would not clear up memory.
– HMR
                Nov 29, 2017 at 14:39
                @HMR I have updated my code to only use let and const but no such luck. Will try a different method of looping.
– James Barnes
                Nov 29, 2017 at 14:59
You are creating 1000 promises at once for every iteration.
I am not entirely sure what you are trying but I could imagine you are looking for something like Bluebird.map(). It is a handy way to process a lot promises with a set concurrency. Probably you are already using splice inside of that loop in order to process your promises in batches.
This approach requires you to install and import bluebird:
const userDataArray = [];
try {
  await Bluebird.map(uids, async (uid) => {
    const userData = await promiseUserData(uid);
    if (userData) { userDataArray.push(userData); }
  }, { concurrency: 100 })
} catch (err) {
  console.log('Error in Promises.all: ', err);
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.