/mnt/d/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

Errors

While some errrors are specifically designed for the fastai library, the general idea still holds true in raw Pytorch as well.

device_error[source]

device_error(e:Exception, a:str, b:str)

Verbose error for if a and b are on different devices Should be used when checking if a model is on the same device, or two tensors

The device error provides a much more readable error when a and b were on two different devices. An situation is below:

inp = torch.rand().cuda()
model = model.cpu()
try:
    _ = model(inp)
except Exception as e:
    device_error(e, 'Input type', 'Model weights')

And our new log:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-28-981e0ace9c38> in <module>()
      2     model(x)
      3 except Exception as e:
----> 4     device_error(e, 'Input type', 'Model weights')

10 frames
/usr/local/lib/python3.7/dist-packages/torch/tensor.py in __torch_function__(cls, func, types, args, kwargs)
    993 
    994         with _C.DisableTorchFunction():
--> 995             ret = func(*args, **kwargs)
    996             return _convert(ret, cls)
    997 

RuntimeError: Mismatch between weight types

Input type has type:         (torch.cuda.FloatTensor)
Model weights have type:     (torch.FloatTensor)

Both should be the same.

hook_fn[source]

hook_fn(m, i)

Simple hook fn to return the layer

class PreHook[source]

PreHook(m, hook_func, is_forward=True, detach=True, cpu=False, gather=False) :: Hook

Creates and registers a hook on m with hook_func as a forward pre_hook

class ForwardHooks[source]

ForwardHooks(ms, hook_func, is_forward=True, detach=True, cpu=False)

Create several forward-hooks on the modules in ms with hook_func

hook_outputs[source]

hook_outputs(modules, detach=True, cpu=False, grad=False)

Return Hooks that store activations of all modules in self.stored

By using forward hooks, we can locate our problem layers when they arrive rather than trying to figure out which one it is through a list of confusing errors.

For this tutorial and testing we'll purposefully write a broken model:

from torch import nn
m = nn.Sequential(
    nn.Conv2d(3,3,1),
    nn.ReLU(),
    nn.Linear(3,2)
)

layer_error[source]

layer_error(e:Exception, model, *inp)

Verbose error for when there is a size mismatch between some input and the model. model should be any torch model inp is the input that went to the model

layer_error can be used anywhere that you want to check that the inputs are right for some model.

Let's use our m model from earlier to show an example:

inp = torch.rand(5,2, 3)
try:
    m(inp)
except Exception as e:
    layer_error(e, m, inp)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-70-382624f1bf50> in <module>()
      4     m(inp)
      5 except Exception as e:
----> 6     layer_error(e, m, inp)

<ipython-input-69-9bc6231f04db> in layer_error(e, model, *inp)
     21         [h.remove() for h in hooks.hooks]
     22         e.args = [f'Size mismatch between input tensors and what the model expects\n{"-"*76}\nLayer: {i}, {layer}\nError: {args}']
---> 23         raise e

<ipython-input-70-382624f1bf50> in <module>()
      2 inp = torch.rand(5,2, 3)
      3 try:
----> 4     m(inp)
      5 except Exception as e:
      6     layer_error(e, m, inp)

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119 

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    421 
    422     def forward(self, input: Tensor) -> Tensor:
--> 423         return self._conv_forward(input, self.weight)
    424 
    425 class Conv3d(_ConvNd):

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--> 420                         self.padding, self.dilation, self.groups)
    421 
    422     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Size mismatch between input tensors and what the model expects
----------------------------------------------------------------------------
Layer: 2, Conv2d(3, 3, kernel_size=(1, 1), stride=(1, 1))
Error: Model expected 4-dimensional input for 4-dimensional weight [3, 3, 1, 1], but got 3-dimensional input of size [5, 2, 3] instead

This will also work with multi-input and multi-output models:

class DoubleInputModel(nn.Sequential):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(nn.Conv2d(3,3,1),
            nn.ReLU(),
            nn.Linear(3,2))
    def forward(self, a, b):
        return self.layers(a), self.layers(b)

model = DoubleInputModel()

inp = torch.rand(5,2, 3)
try:
    model(inp, inp)
except Exception as e:
    layer_error(e, model, inp, inp)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-76-85d761322577> in <module>()
      4     model(inp, inp)
      5 except Exception as e:
----> 6     layer_error(e, model, inp, inp)

<ipython-input-69-9bc6231f04db> in layer_error(e, model, *inp)
     21         [h.remove() for h in hooks.hooks]
     22         e.args = [f'Size mismatch between input tensors and what the model expects\n{"-"*76}\nLayer: {i}, {layer}\nError: {args}']
---> 23         raise e

<ipython-input-76-85d761322577> in <module>()
      2 inp = torch.rand(5,2, 3)
      3 try:
----> 4     model(inp, inp)
      5 except Exception as e:
      6     layer_error(e, model, inp, inp)

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-74-a8699275a055> in forward(self, a, b)
      6             nn.Linear(3,2))
      7     def forward(self, a, b):
----> 8         return self.layers(a), self.layers(b)

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119 

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    421 
    422     def forward(self, input: Tensor) -> Tensor:
--> 423         return self._conv_forward(input, self.weight)
    424 
    425 class Conv3d(_ConvNd):

/mnt/d/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--> 420                         self.padding, self.dilation, self.groups)
    421 
    422     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Size mismatch between input tensors and what the model expects
----------------------------------------------------------------------------
Layer: 2, Conv2d(3, 3, kernel_size=(1, 1), stride=(1, 1))
Error: Model expected 4-dimensional input for 4-dimensional weight [3, 3, 1, 1], but got 3-dimensional input of size [5, 2, 3] instead

Much more readable!