Qube restars every 2 minutes

A VM seems to have gone rogue and I don’t understand why. It restarts every two minutes. It does not seem to be dependent on the NetVM nor the template on which it is based.

For obvious reasons I’ve given a pseudonym to the qube: example-qube. Bellow is the log from /var/log/qubes/vm-example-qube.log

2021-02-14 17:52:02,000 Starting example-qube
2021-02-14 17:52:21,103 Setting Qubes DB info for the VM
2021-02-14 17:52:21,103 Starting Qubes DB
2021-02-14 17:52:21,179 Activating the example-qube VM
2021-02-15 11:46:01,399 Starting example-qube
2021-02-15 11:46:11,214 Setting Qubes DB info for the VM
2021-02-15 11:46:11,214 Starting Qubes DB
2021-02-15 11:46:11,261 Activating the example-qube VM
2021-02-15 11:54:01,116 Starting example-qube
2021-02-15 11:54:05,063 Setting Qubes DB info for the VM
2021-02-15 11:54:05,064 Starting Qubes DB
2021-02-15 11:54:05,129 Activating the example-qube VM
2021-02-15 17:25:11,130 Attaching network
2021-02-15 17:25:53,748 Attaching network
2021-02-15 17:28:01,299 Starting example-qube
2021-02-15 17:28:20,501 Setting Qubes DB info for the VM
2021-02-15 17:28:20,505 Starting Qubes DB
2021-02-15 17:28:20,725 Activating the example-qube VM

Then I guess at this point i decided to see what would happen if I paused this qube while it was running. (the log is a continuation of the previous one)

2021-02-15 17:30:01,485 Starting example-qube
2021-02-15 17:30:21,923 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:32:01,354 Starting example-qube
2021-02-15 17:32:22,549 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:34:01,921 Starting example-qube
2021-02-15 17:34:19,751 Setting Qubes DB info for the VM
2021-02-15 17:34:19,752 Starting Qubes DB
2021-02-15 17:34:19,929 Activating the example-qube VM
2021-02-15 17:35:20,112 Start failed: Cannot connect to qrexec agent for 60 seconds, see /var/log/xen/console/guest-example-qube.log for details
2021-02-15 17:36:01,305 Starting example-qube
2021-02-15 17:36:25,566 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:38:01,771 Starting example-qube
2021-02-15 17:38:27,099 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:40:01,984 Starting example-qube
2021-02-15 17:40:27,069 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:42:01,123 Starting example-qube
2021-02-15 17:42:26,887 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:44:01,740 Starting example-qube
2021-02-15 17:44:26,772 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:46:01,883 Starting example-qube
2021-02-15 17:46:26,774 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:48:01,765 Starting example-qube
2021-02-15 17:48:26,916 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:50:01,895 Starting example-qube
2021-02-15 17:50:26,788 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:52:01,674 Starting example-qube
2021-02-15 17:52:26,613 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:54:01,479 Starting example-qube
2021-02-15 17:54:26,444 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:56:01,418 Starting example-qube
2021-02-15 17:56:27,144 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 17:58:01,826 Starting example-qube
2021-02-15 17:58:27,591 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:00:01,743 Starting example-qube
2021-02-15 18:00:27,663 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:02:01,346 Starting example-qube
2021-02-15 18:02:27,555 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:04:02,058 Starting example-qube
2021-02-15 18:04:28,552 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:06:01,468 Starting example-qube
2021-02-15 18:06:26,919 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:08:01,588 Starting example-qube
2021-02-15 18:08:28,341 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:10:01,678 Starting example-qube
2021-02-15 18:10:27,217 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:12:01,924 Starting example-qube
2021-02-15 18:12:28,260 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:14:01,901 Starting example-qube
2021-02-15 18:14:28,301 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:16:01,504 Starting example-qube
2021-02-15 18:16:28,010 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:18:01,961 Starting example-qube
2021-02-15 18:18:27,224 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:20:01,869 Starting example-qube
2021-02-15 18:20:28,051 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:22:01,267 Starting example-qube
2021-02-15 18:22:27,059 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:24:01,604 Starting example-qube
2021-02-15 18:24:27,732 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:26:01,447 Starting example-qube
2021-02-15 18:26:27,433 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:28:01,133 Starting example-qube
2021-02-15 18:28:27,300 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:30:01,556 Starting example-qube
2021-02-15 18:30:27,912 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:32:01,769 Starting example-qube
2021-02-15 18:32:28,043 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:34:01,583 Starting example-qube
2021-02-15 18:34:28,038 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:36:01,581 Starting example-qube
2021-02-15 18:36:27,237 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:38:01,973 Starting example-qube
2021-02-15 18:38:28,451 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:40:02,074 Starting example-qube
2021-02-15 18:40:28,041 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:42:01,599 Starting example-qube
2021-02-15 18:42:28,260 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:44:01,566 Starting example-qube
2021-02-15 18:44:27,908 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:46:01,288 Starting example-qube
2021-02-15 18:46:27,674 Start failed: internal error: libxenlight failed to create new domain 'example-qube'
2021-02-15 18:48:01,242 Starting example-qube
2021-02-15 18:48:05,353 Setting Qubes DB info for the VM
2021-02-15 18:48:05,353 Starting Qubes DB
2021-02-15 18:48:05,456 Activating the example-qube VM

At this point the computer was shut down and restarted some hours later. In this last section of the log, while it did not restart, I changed the netVM to none and swapped the template through:

  • debian-10
  • fedora-32
  • debian-10-minimal

So it does not seem to be dependent on netVM and templateVM as you can see by it continuing the series of reboots. I should also note that when it lives for longer (>2min) it’s because I forgot to shut it down immediately.

2021-02-16 03:52:01,941 Starting example-qube
2021-02-16 03:52:19,244 Setting Qubes DB info for the VM
2021-02-16 03:52:19,245 Starting Qubes DB
2021-02-16 03:52:19,385 Activating the example-qube VM
2021-02-16 03:54:01,537 Starting example-qube
2021-02-16 03:54:05,100 Setting Qubes DB info for the VM
2021-02-16 03:54:05,101 Starting Qubes DB
2021-02-16 03:54:05,210 Activating the example-qube VM
2021-02-16 03:56:01,703 Starting example-qube
2021-02-16 03:56:05,666 Setting Qubes DB info for the VM
2021-02-16 03:56:05,666 Starting Qubes DB
2021-02-16 03:56:05,733 Activating the example-qube VM
2021-02-16 04:16:01,826 Starting example-qube
2021-02-16 04:16:05,585 Setting Qubes DB info for the VM
2021-02-16 04:16:05,586 Starting Qubes DB
2021-02-16 04:16:05,677 Activating the example-qube VM
2021-02-16 04:18:01,291 Starting example-qube
2021-02-16 04:18:05,465 Setting Qubes DB info for the VM
2021-02-16 04:18:05,465 Starting Qubes DB
2021-02-16 04:18:05,568 Activating the example-qube VM

I am hopeful I can find here someone who can help me with this

Mistery solved. It was some chron job that was starting a VM. Had set it setup for backing up that VM’s contents a while ago but had renamed it since.

Recently I had renamed it back to the original name, so the cron job kicked in every 2 minutes.

An interesting (yet unsurprising) finding is that if you run

qvm-run --pass-io example-qubes "some command"

it actually unpauses the qube. Makes sense.