Interaction of libvirt and AppArmor

I'm teaching at the University of Applied Science Burgenland in Eisenstadt (Austria). We recently had a lab (which took place in the lab in Eisenstadt but students were working from home due to Covid reasons) where the task is to set everything up for virtualisation and then live-migrate a running virtual machine to another server using libvirt (we're using the command-line with virsh).

For just one group out of several – with identical initial Debian installations, migration failed with an error message. The migration command was:

virsh -c qemu+ssh://root@primary/system migrate --live --unsafe \
    debian-1 qemu+ssh://root@secondary/system

For the lab we're using NFS because setting up a more advanced filesystem would take too much time, that's why we're using the --unsafe option. The following error message resulted (error message broken to several lines, this was all in a single line):

error: internal error: Process exited prior to exec:
libvirt:  error : unable to set AppArmor profile
'libvirt-d22db7ca-50ca-43bd-b6da-1ccecf5a83e7' for '/usr/bin/kvm':
No such file or directory

It turned out that this group had managed to fill up the /var partition with logfiles but after cleanup this still did produce the same message. So the hunch here is that some files that AppArmor and/or libvirt create dynamically could not be created and that was the reason why this failed. It also turned out that some AppArmor files that were correctly installed on the first machine were missing on the second.

Trying to reinstall AppArmor and related files using apt-get with the --reinstall option did not work, the missing config files in /etc/apparmor.d were not re-created. So removing the packages with the purge command (which removes all config files) and then reinstalling everything fixed the installed AppArmor files and made the migration finally work. I have no idea which files were missing.

When googling for the error message above I found a debian bug-report Where one of the dynamically generated files in /etc/apparmor.d/libvirt was zero length. This, however was not the problem in our case but indicates that AppArmor isn't very good at checking errors when a filesystem is full. So there are probably other files that are dynamically generated that were the problem in our case.

The following sequence of deinstall and reinstall commands fixed the problem in our case, note that just removing files as in the debian bug-report did not fix the issue in our case:

dpkg --purge apparmor-utils apparmor-profiles
dpkg --purge apparmor
rm -rf /var/cache/apparmor
apt-get install apparmor apparmor-utils apparmor-profiles
dpkg --purge libvirt-daemon-system
apt-get install libvirt-daemon-system
systemctl restart libvirtd.service
systemctl restart virtlogd.service
systemctl restart virtlogd.socket

I'm not sure restarting the services is really necessary but there was another issue that libvirt could not connect to the virtlog socket and this was fixed by restarting the virtlog.{service,socket}.