deprecation message for non-full backward hook

[Solosneros]

🐛 Bug

I am getting the following warning when using opacus on my system:
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py:1025: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "

To Reproduce

Steps to reproduce the behavior:
(sorry, I had problems running google colab)..

1. Execute the minimal example code given below:

import numpy as np
import opacus
import torch
import torch.nn as nn
from opacus.utils.uniform_sampler import UniformWithReplacementSampler
from torch.utils.data import DataLoader, TensorDataset, Dataset


DEVICE = "cuda:0"

BATCH_SZ = 8
FEATURE_DIM = 16
N_CLASSES = 2
DATASET_LEN = 50_000

class DummyDataset(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, i):
        x = torch.randn([FEATURE_DIM]).to(DEVICE) 
        y = 0
        return x, y

    def __len__(self):
        return DATASET_LEN

train_ds = DummyDataset()
train_loader = DataLoader(train_ds, BATCH_SZ, shuffle=False)

model = nn.Linear(FEATURE_DIM, N_CLASSES).to(DEVICE)
criterion = nn.CrossEntropyLoss()

privacy_engine = opacus.PrivacyEngine()
model, optimizer, train_loader = privacy_engine.make_private(
        module=model,
        optimizer=torch.optim.SGD(model.parameters(), 0.01),
        data_loader=train_loader,
        noise_multiplier=1,
        max_grad_norm=10,
    )

for epoch in range(2):
  for x, y in train_loader:
        x = x.to(DEVICE)
        y = y.to(DEVICE)
        optimizer.zero_grad()
        out = model(x)
        loss = criterion(out, y)
        loss.backward()
        optimizer.step()

Expected behavior

No error message.

Environment

PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-1026-oem-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.1.0
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.10.1
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==0.10.1
[pip3] torchvision==0.11.2
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_1
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-service 2.4.0 py39h7f8727e_0
[conda] mkl_fft 1.3.1 py39hd3c417c_0
[conda] mkl_random 1.2.2 py39h51133e4_0
[conda] mypy_extensions 0.4.3 py39h06a4308_0
[conda] numpy 1.20.3 py39hf144106_0
[conda] numpy-base 1.20.3 py39h74d4b33_0
[conda] numpydoc 1.1.0 pyhd3eb1b0_1
[conda] pytorch 1.10.1 py3.9_cuda10.2_cudnn7.6.5_0 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] pytorch-ranger 0.1.1 pyhd8ed1ab_0 conda-forge
[conda] torch-optimizer 0.3.0 pyhd8ed1ab_0 conda-forge
[conda] torchaudio 0.10.1 py39_cu102 pytorch
[conda] torchvision 0.11.2 py39_cu102 pytorc

[alexandresablayrolles]

Thanks for flagging this. We are currently discussing with the Pytorch team as the new proposed hooks are not ideal for our use-case.

[Solosneros]

Thanks for letting me know!
I noticed that the bug template on colab is for opacus >= 1.0.0. As this seems to be a small issue, I would open a PR for that. Or is that already taken care of?

[ffuuugor]

Whoops, thanks for pointing out the old colab in the bug report template - fixed now.

With full backward hooks, unfortunately, it's not as simple as just replacing the deprecated hooks with the new method - their behaviour not always matches.
One issue in particular is in-place operations - they are flat out forbidden with the new method, but are actively used in torchvision models (at least the last time I checked couple months ago). Abandoning support for torchvision models is a hard trade-off to make, so we're currently weighing our options (including #259 proposal).

I'll leave this issue open for tracking progress in the future, as we definitely plan to address this eventually

[mmsaki]

having this same issue

[alexandresablayrolles]

Thanks for flagging @mmsaki. Currently, it is a warning so you can safely ignore it. We are working on a solution for the next version of Pytorch.

[xiyuanyang45]

has this issue been solved now?

[xiyuanyang45]

i have the same problem

[karthikprasad]

Hi @dDCTRr, the warning for hooks still exist due to the reasons outlined by @ffuuugor in this comment; you can safely ignore it though (apologies for the annoyance).

That said, starting Opacus 1.2.0, we support functorch based per-sample gradient computation (no hooks, no warnings). To use this, simply set grad_sample_mode="functorch" in the call to make_private(). You can find more details about this in the release notes.
Note: as mentioned in the release notes, functorch support is still in beta mode, and it could be slower than hooks.

[weiiewwei]

i met the same error

[weiiewwei]

has this issue been solved now?

[xiyuanyang45]

is seems that the warning has no effect on code execution | | 杨熙元 | | ***@***.*** |

…

---- Replied Message ---- | From | ***@***.***> | | Date | 03/9/2023 12:32 | | To | ***@***.***> | | Cc | Xiyuan ***@***.***> , ***@***.***> | | Subject | Re: [pytorch/opacus] deprecation message for non-full backward hook (Issue #328) | has this issue been solved now? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[erik-buchholz]

Hi @dDCTRr, the warning for hooks still exist due to the reasons outlined by @ffuuugor in this comment; you can safely ignore it though (apologies for the annoyance).

That said, starting Opacus 1.2.0, we support functorch based per-sample gradient computation (no hooks, no warnings). To use this, simply set grad_sample_mode="functorch" in the call to make_private(). You can find more details about this in the release notes. Note: as mentioned in the release notes, functorch support is still in beta mode, and it could be slower than hooks.

Hi @karthikprasad,

I tried setting grad_sample_mode='functorch' or to ew`, but in both cases the warning is still shown. Can you give me any hint on why that is? And more importantly, can I still assume that it is safe to ignore the warning? I am using the module within a GAN, but below is a reduced example that already triggers the warning. I am using Opacus version 1.4.0 and Pytorch 2.0.0+cu118.

Thanks a lot for your help!

import os

import numpy as np
import torch
import torch.nn as nn
import torchvision.transforms as transforms
from opacus import PrivacyEngine
from torch.utils.data import DataLoader
from torchvision import datasets
import torchvision.transforms.functional as F

channels = 1
img_size = 28
img_shape = (channels, img_size, img_size)

def load_data(batch_size: int) -> DataLoader:
    # Configure data loader
    os.makedirs(f"tmp/data/mnist", exist_ok=True)
    dataloader = torch.utils.data.DataLoader(
        datasets.MNIST(
            f"tmp/data/mnist",
            train=True,
            download=True,
            transform=transforms.Compose(
                [transforms.Resize(img_size), transforms.ToTensor(), transforms.Normalize([0.5], [0.5])]
            ),
        ),
        batch_size=batch_size,
        shuffle=True,
    )
    return dataloader

class Generator(nn.Module):
    def __init__(self, dp: bool = False):
        super(Generator, self).__init__()
        self.model = nn.Linear(100, int(np.prod(img_shape)))

    def forward(self, z):
        img = self.model(z).tanh()
        img = img.view(img.size(0), *img_shape)
        return img


m = Generator(True)
m = m.to('cuda')
opt = torch.optim.Adam(m.parameters())
dataloader = load_data(128)
privacy_engine = PrivacyEngine(accountant='rdp')


m, opt, dataloader = privacy_engine.make_private_with_epsilon(
    module=m,
    optimizer=opt,
    data_loader=dataloader,
    epochs=10,
    target_epsilon=10,
    target_delta=1e-5,
    max_grad_norm=1.0,
    poisson_sampling=True,
    grad_sample_mode="functorch"  # ew
)

loss = torch.nn.BCELoss()
for epoch in range(1, 10 + 1):
    for i, (imgs, _) in enumerate(dataloader):
        valid = torch.ones((imgs.size(0), 1), device='cuda:0')
        fake = torch.zeros((imgs.size(0), 1), device='cuda:0')

        real_imgs = imgs.to('cuda')

        # -----------------
        #  Train Generator
        # -----------------
        m.train()
        opt.zero_grad()

        z = torch.randn(size=(imgs.shape[0], 100), device='cuda')

        # The following line triggers the warning:
        gen_imgs = m(z)
        # Quit here, as this is enough to reproduce the warning
        break
    break

The displayed warning:

[...]/lib/python3.10/site-packages/torch/nn/modules/module.py:1344: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "

[volmodaoist]

I meet the same Userwarning.