Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:325

I have a PyTorch tensor called out_probs which is produced like this:

out_probs=F.softmax(out_dec[:,0],dim=0)
Also, the shape of out_probs is [128,20004]
out_probs is the result of a softmax operation and it's not supposed to contain any negative value so naturally the result of out_probs[out_probs<0 is going to be an empty tensor(actually I checked and it was empty)
But when I'm running
torch.multinomial(out_probs, 1)
I'm getting :
RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:325
That implies my tensor has a negative entry and I don't know why this is happening?
x[0] *= 1e100
out_probs = F.softmax(x, dim=0)
print('Negative values:', torch.sum(out_probs < 0).item())
y = torch.multinomial(out_probs, 1)
results in the following output
Negative values: 0
RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:298
It turns out this is getting triggered because out_probs contains nan entries.
print('nan values:', torch.sum(torch.isnan(out_probs)).item())
gives
nan values: 128
Which are caused by mathematical instabilities in softmax.
Strangely, when the values in out_probs are infinite you get the proper error message
RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry = infinity or NaN) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:302
This bug should probably be reported at https://github.com/pytorch/pytorch/issues if it hasn't been fixed in the most recent version.
By the way I'm using PyTorch 1.0.1.post2
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.