CUDA

CUDA is what lets PyTorch (and other frameworks) run computations on an NVIDIA GPU instead of the CPU — often 10-100x faster for the matrix math deep learning relies on.

Interview mindset:

For every operation: Is this running on the CPU or GPU? Do the tensors involved actually live on the same device?

Checking for GPU Availability

Always check if a GPU is actually available before assuming your code will use one — code should gracefully fall back to CPU.

Interview check:Know why hardcoding "cuda" without checking availability will crash on a CPU-only machine.

import torch
  
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Moving Tensors & Models to GPU

Both your model and your data need to be on the same device — a common source of runtime errors is mismatched devices.

Interview check:Know the exact error PyTorch throws when tensors are on different devices.

model = model.to(device)
  x = x.to(device)
  
  output = model(x)

GPU Memory

GPU memory (VRAM) is limited and separate from system RAM. Large batch sizes or models can run out of it.

Interview check:Know that reducing batch size is the most common fix for a CUDA out-of-memory error.

torch.cuda.memory_allocated()
  torch.cuda.empty_cache()

Mixed Precision Training

Uses lower-precision (16-bit) floats for parts of training to speed things up and use less memory, without meaningfully hurting accuracy.

Interview check:Know the basic tradeoff — faster and more memory-efficient, at the cost of some numerical precision.

scaler = torch.cuda.amp.GradScaler()
  
  with torch.cuda.amp.autocast():
      output = model(x)
      loss = loss_fn(output, y)

CUDA Interview Questions You Should Be Able to Answer

Why is training on a GPU faster than on a CPU?
What happens if your model and data are on different devices?
What causes a CUDA out-of-memory error, and how would you fix it?
What is mixed precision training, and what's the tradeoff?

The Standard You Want

You should be comfortable debugging a device-mismatch error immediately, and know what to try first when you hit an out-of-memory error mid-training.

A GPU only helps if your code is actually using it correctly.