[Focal] iptables rules are not applied on boot

[emkll]

Description

Towards #4768

The firewall logic is not working reliably in Ubuntu Focal, and firewall rules are not correctly restored on boot. This appears to be a discrepancy between Vagrant boxes and a default Ubuntu 20.04 install.

Steps to Reproduce

In an Ubuntu Focal install (non-staging, and likely non-VM), run sudo iptables -L or sudo ip6tables -L

Expected Behavior

iptables and ip6tables rules should be correctly applied and enforced, with default DROP policy

Actual Behavior

iptables rules are set to overly permissive defaults:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
      

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
       

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

[conorsch]

If this is a discrepancy between Vagrant boxes (we use the Bento images: https://github.com/chef/bento/tree/3dd165cf422572ab4240c8f1ce9a7e40b7960457/packer_templates/ubuntu), then it should be easy to reproduce in Focal staging VMs on Qubes, since those are configured directly from an ISO.

I took a look at #5712 recently, and while I had questions, the iptables rules were definitely populated. @emkll suggested in standup today to force-reboot the VMs post-provisioning and check again, see if the firewall rules don't restore.

@kushaldas also requests a full list of all installed packages inside the hardware Focal instance to aid in debugging.

[emkll]

See the following files for installed packages on a default Focal install on hardware:

installed-app.txt
installed-mon.txt

[zenmonkeykstop]

Can't replicate this on Focal VMs (bento-20.04, 202008.16.0). iptables rules are present and survive a reboot. Gonna try a hardware install next and see if I can replicate.

[zenmonkeykstop]

VM output of apt list --installed attached for reference:

monvm.txt
appvm.txt

[kushaldas]

I can finally reproduce this. After I did a reboot of the app, I can not reach the server anymore :( Debugging.

[conorsch]

Was able to reproduce the issue in Qubes Focal VMs. Although I recalled that during initial review of #5712, the iptables tests were passing, I didn't actually document that in the PR comments.

[conorsch]

@kushaldas I see you've got a PR in #5780 to force the rules to persist on Focal post-reboot. Do you have a hypothesis for why the hardware installs behave differently from the VM installs? Looking through the bento configs, it's not obvious to me what the discrepancy is. I'll review the package lists above next.

[kushaldas]

@kushaldas I see you've got a PR in #5780 to force the rules to persist on Focal post-reboot. Do you have a hypothesis for why the hardware installs behave differently from the VM installs? Looking through the bento configs, it's not obvious to me what the discrepancy is. I'll review the package lists above next.

Nope, I could not identify that part.

[conorsch]

Here's a diff of the package lists, for comparison's take. The reference is the hardware install, the changes (+/-) are what's installed in the VM image. All packages included in both are omitted.

package list diff

--- clean/focal-hw-app-packages.txt
+++ clean/focal-vm-app-packages.txt
-bash-completion
+binutils
+binutils-common
+binutils-x86-64-linux-gnu
+cifs-utils
+cloud-guest-utils
-command-not-found
+cpp
+cpp-9
-cron-apt
-efibootmgr
-friendly-recovery
+gcc-9-base
-grub-efi-amd64
-grub-efi-amd64-bin
-grub-efi-amd64-signed
+grub-gfxpayload-lists
+grub-pc
+grub-pc-bin
+ifupdown
+keyutils
+language-pack-en
+language-pack-en-base
+language-pack-gnome-en
+language-pack-gnome-en-base
+libbinutils
+libc6-dev
+libc-dev-bin
+libcrypt-dev
+libctf0
+libctf-nobfd0
+libisl22
-liblockfile1
-liblockfile-bin
+libmpc3
+libncurses-dev
+libnfsidmap2
+libreadline-dev
+libssl-dev
+libtalloc2
+libtevent0
+libtirpc3
+libtirpc-common
+libwbclient0
-libx11-6
-libx11-data
-libxcb1
-libxext6
-libxmuu1
+linux-image-4.14.175-grsec-securedrop
-linux-image-5.4.88-grsec-securedrop
+linux-libc-dev
+linux-modules-5.4.0-42-generic
+make
-mokutil
+nfs-common
-popularity-contest
-postfix
-python3-serial
-python3-setuptools
+rpcbind
-shim
-shim-signed
+tasksel
+tasksel-data
-ubuntu-standard
+ubuntu-server
-xauth
+zlib1g-dev

There are several net-related packages like cifs-utils, cloud-guest-utils, ifupdown, and nfs-common, but I haven't found a succinct explanation yet. Note also the ubuntu-{standard,server} discrepancy, which causes much of the package differences.

[conorsch]

Simply installing ifupdown on Focal is sufficient to restore the iptables rules post-reboot. While there's an argument for migrating to iptables-persistent as in #5780, the more conservative approach is to force the known-good rules on every reboot, rather than restoring whatever rules were on the host at time of reboot. Thoughts, @emkll?