Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I'm taking a course on PyTorch . And I'm wondering why we need to separately tell to torch.utils.data.DataLoader output on what device it's running on. If the model is already on CUDA why doesn't it automatically change the inputs accordingly? This pattern seems funny to me:

model.to(device)
for ii, (inputs, labels) in enumerate(trainloader):
    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)
Is there a use case where I'd like to have model running on GPU but my inputs to be on CPU mode or vice versa? 
When you call model.to(device) (assuming device is a GPU) your model parameters will be moved to your GPU. Regarding to your comment: they are moved from CPU memory to GPU memory then.
By default newly created tensors are created on CPU, if not specified otherwise. So this applies also for your inputs and labels.
The problem here is that all operands of an operation need to be on the same device! If you leave out the to and use CPU tensors as input you will get an error message.
Here is an short example for illustration:
import torch
# device will be 'cuda' if a GPU is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# creating a CPU tensor
cpu_tensor = torch.rand(10)
# moving same tensor to GPU
gpu_tensor = cpu_tensor.to(device)
print(cpu_tensor, cpu_tensor.dtype, type(cpu_tensor), cpu_tensor.type())
print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())
print(cpu_tensor*gpu_tensor)
Output:
tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619]) torch.float32 <class 'torch.Tensor'> torch.FloatTensor
tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619], device='cuda:0') torch.float32 <class 'torch.Tensor'> torch.cuda.FloatTensor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ac794171c178> in <module>()
     12 print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())
---> 14 print(cpu_tensor*gpu_tensor)
RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'other'
                What does it mean they are on GPU - maybe you have an article link to share? =D  Also we have torch.no_grad why not have torch.to('CUDA') and all things will be put on GPU?
– Mihkel L.
                Dec 9, 2018 at 18:10
                @MihkelL. Meant is the ram (random access memory). So GPU has it's on RAM. So there is CPU RAM, if they "are on CPU" they are stored in CPU Memory, if they "are on GPU" they are stored in GPU RAM. I hope this makes sense to you! (Terms RAM and Memory are used as synonyms here)
– MBT
                Dec 9, 2018 at 18:12
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.