Samba shares cause long shutdowns

oli · January 25, 2023, 11:38am

Hi, we use several samba shares in our company to persistantly store data. We have qubes that automatically mount these shares when the qubes start up. But when its time to shut the qube down again then the systemd unit that is created for each mount, times out instead of shutting down gracefully. Here is an excerpt from console of one such qube:

[2023-01-25 12:16:28] [   27.170811] CIFS: Attempting to mount \\example.com\mountpoint
[2023-01-25 12:16:34] [   32.957077] systemd[1]: systemd-fsckd.service: Succeeded.
[2023-01-25 12:17:03]          Stopping .[0;1;39mSession c1 of user user.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Removed slice .[0;1;39msystem-modprobe.slice.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped target .[0;1;39mMulti-User System.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped target .[0;1;39mLogin Prompts.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped target .[0;1;39mTimers.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mPeriodic ext4 Onli…ata Check for All Filesystems.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDiscard unused blocks once a week.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDaily rotation of log files.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDaily exim4-base housekeeping.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDaily man-db regeneration.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mUpdate system time each 6h.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mPeriodically check for updates.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDaily Cleanup of Temporary Directories.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped target .[0;1;39mSystem Time Synchronized.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped target .[0;1;39mSystem Time Set.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Closed .[0;1;39mLVM2 poll daemon socket.[0m.
[2023-01-25 12:17:03]          Stopping .[0;1;39mAvailability of block devices.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mCUPS Scheduler.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mGetty on tty1.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mAuthorization Manager.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mQubes GUI Agent.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mQubes misc post-boot actions.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mQubes network uplink (eth0) setup.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mQubes remote exec agent.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mSystem Logging Service.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mRealtimeKit Scheduling Policy Service.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mSerial Getty on hvc0.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mDisk Manager.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mDaemon for power management.[0m...
[2023-01-25 12:17:03]          Stopping .[0;1;39mLSB: Start/stop secondary xen domains.[0m...
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mAuthorization Manager.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mSystem Logging Service.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mCUPS Scheduler.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mGetty on tty1.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mSerial Getty on hvc0.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mQubes remote exec agent.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDisk Manager.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mDaemon for power management.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mRealtimeKit Scheduling Policy Service.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mQubes misc post-boot actions.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mQubes network uplink (eth0) setup.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Removed slice .[0;1;39msystem-getty.slice.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Removed slice .[0;1;39msystem-qubes\x2dnetwork\x2duplink.slice.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Removed slice .[0;1;39msystem-serial\x2dgetty.slice.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mQubes base firewall settings.[0m.
[2023-01-25 12:17:03]          Stopping .[0;1;39mXen driver domain device daemon.[0m...
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mXen driver domain device daemon.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mAvailability of block devices.[0m.
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mLSB: Start/stop secondary xen domains.[0m.
[2023-01-25 12:17:03]          Stopping .[0;1;39mLSB: Xen daemons.[0m...
[2023-01-25 12:17:03] [.[0;32m  OK  .[0m] Stopped .[0;1;39mLSB: Xen daemons.[0m.
[2023-01-25 12:17:04] [   63.366123] systemd[1]: session-c1.scope: Succeeded.
[2023-01-25 12:17:04] [   63.366765] systemd[1]: Stopped Session c1 of user user.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mSession c1 of user user.[0m.
[2023-01-25 12:17:04] [   63.367267] systemd[1]: session-c1.scope: Consumed 2.974s CPU time.
[2023-01-25 12:17:04] [   63.367950] systemd[1]: Stopping User Login Management...
[2023-01-25 12:17:04]          Stopping .[0;1;39mUser Login Management.[0m...
[2023-01-25 12:17:04] [   63.369669] systemd[1]: Stopping User Manager for UID 1000...
[2023-01-25 12:17:04]          Stopping .[0;1;39mUser Manager for UID 1000.[0m...
[2023-01-25 12:17:04] [   63.375765] systemd[1]: systemd-logind.service: Succeeded.
[2023-01-25 12:17:04] [   63.376758] systemd[1]: Stopped User Login Management.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mUser Login Management.[0m.
[2023-01-25 12:17:04] [   63.409453] systemd[1]: qubes-gui-agent.service: Succeeded.
[2023-01-25 12:17:04] [   63.409839] systemd[1]: Stopped Qubes GUI Agent.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mQubes GUI Agent.[0m.
[2023-01-25 12:17:04] [   63.431534] systemd[1]: run-user-1000-doc.mount: Succeeded.
[2023-01-25 12:17:04] [   63.431699] systemd[1]: Unmounted /run/user/1000/doc.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Unmounted .[0;1;39m/run/user/1000/doc.[0m.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mUser Manager for UID 1000.[0m.
[2023-01-25 12:17:04]          Stopping .[0;1;39mUser Runtime Directory /run/user/1000.[0m...
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Unmounted .[0;1;39m/run/user/1000.[0m.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mUser Runtime Directory /run/user/1000.[0m.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Removed slice .[0;1;39mUser Slice of UID 1000.[0m.
[2023-01-25 12:17:04]          Stopping .[0;1;39mPermit User Sessions.[0m...
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mPermit User Sessions.[0m.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped target .[0;1;39mRemote File Systems.[0m.
[2023-01-25 12:17:04]          Unmounting .[0;1;39m/home/user/example.com/mountpoint.[0m...
[2023-01-25 12:17:04]          Unmounting .[0;1;39m/rw/home/user/example.com/mountpoint.[0m...
[2023-01-25 12:17:04]          Stopping .[0;1;39mQubes memory information reporter.[0m...
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Stopped .[0;1;39mQubes memory information reporter.[0m.
[2023-01-25 12:17:04] [.[0;32m  OK  .[0m] Unmounted .[0;1;39m/rw/home/user/example.com/mountpoint.[0m.
[2023-01-25 12:17:06] [.[0;1;31m*.[0m.[0;31m*    .[0m] A stop job is running for /home/user/example.com/mountpoint (3s / 1min 31s)
.[K[.[0;31m*.[0;1;31m*.[0m.[0;31m*   .[0m] A stop job is running for /home/user/example.com/mountpoint (3s / 1min 31s)
.[K[ .[0;31m*.[0;1;31m*.[0m.[0;31m*  .[0m] A stop job is running for /home/user/example.com/mountpoint (4s / 1min 31s)
.[K[  .[0;31m*.[0;1;31m*.[0m.[0;31m* .[0m] A stop job is running for /home/user/example.com/mountpoint (4s / 1min 31s)
.[K[   .[0;31m*.[0;1;31m*.[0m.[0;31m*.[0m] A stop job is running for /home/user/example.com/mountpoint (5s / 1min 31s)
.[K[    .[0;31m*.[0;1;31m*.[0m] A stop job is running for /home/user/example.com/mountpoint (5s / 1min 31s)

And this continues until it times out at 1 min 31 sec.

So the question is, how to fix this?

I have tried in different qubes, in both Debian and Fedora, using different kernels. Always the same result.
This did not happen in Fedora 35, just started when either switching to Qubes 4.1 (from 4.0) or switching from F35 to F36, don’t remember when exactly this started.

EDIT: Forgot to say that the mounts are created by running a script from /rw/config/rc.local and is not relient on systemd. The systemd .mount unit files created automatically by systemd and when checking the status of them I get the following

systemctl status home-user-example.com-mountpoint.mount
● home-user-example.com-mountpoint.mount - /home/user/example.com/mountpoint
     Loaded: loaded (/proc/self/mountinfo)
     Active: active (mounted) since Wed 2023-01-25 10:45:39 CET; 2h 10min ago
      Until: Wed 2023-01-25 10:45:39 CET; 2h 10min ago
      Where: /home/user/example.com/mountpoint
       What: //example.com/mountpoint

SteveC · January 25, 2023, 5:56pm

I have solved this.

But I am not on my qubes box and I can’t tell you right now what the solution is.

Please bear with me and I can get you the answer in 8-12 hours.

SteveC · January 26, 2023, 4:14am

OK, here it is:

Put a file like this in /etc/systemd/system. Name it something like my-samba-volume.mount (Obviously, change “my samba volume” to something more meaningful to you.) Read what you put into it, some of the items you will have to correct for your system.

[Unit]
Description=Mount My Samba Volume.
Doumentation=man systemd.unit man systemd.mount man systemd.special
After=remote-fs-pre.target
After=network.target
Wants=network.target
After=network-online.target
Wants=network-online.target
After=NetworkManager-wait-online.service
Wants=NetworkManager-wait-online.service
Conflicts=umount.target
Before=umount.target
DefaultDependencies=no

[Mount]
Where=/home/user/MySambaVolumeMountDirectory
What=[address of the samba mount]
Type=cifs [if you are using cifs]
Options=noauto,noatime,nodiratime,x-systemd.automount
TimeoutSec=5

[Install]
WantedBy=multi-user.target

This is an extremely frustrating thing. Apparently dismounting happens too late in the shutdown process if you don’t have this file, so the shutdown hangs because there’s still something mounted.

NOTE: My qube is debian, I have no idea whether this will work in Fedora. If not, perhaps all that is wrong is the directory name.

oli · January 26, 2023, 9:33am

Thanks @SteveC, this might solve it except that our qubes are volatile root so /etc/systemd is not persistent through reboots. I will try to put it in /usr/local/lib/systemd instead and see what happens. Can’t use ~.config since the mount needs to be run as root.

oli · January 26, 2023, 4:58pm

Hmm, yeah this wont work. Because once the .mount unit is activated it creates links inside /etc/systemd which is not persistent through reboots.
This needs to be fixed in another manner.

But thanks for the suggestion.

cayce · January 26, 2023, 5:22pm

Sure it is, so long as you are making changes within the template. Clone the one you’re currently leveraging, add your service there, switch AppVMs to use it & profit!

SteveC · January 26, 2023, 5:27pm

Cayce is correct.

My apologies for forgetting to mention the change needs to be done to the template. Whether you can just make the change to your current template or need to create a new one as Cayce suggests depends on the circumstances. (I already had a template dedicated to the software using the mount, so I added this file to that template.)

oli · January 27, 2023, 2:52pm

Thanks for your inputs @SteveC and @cayce but I found a very elegant solution.

The problem is that the service qubes-network-uplink@.service (when in an active qube it is called qubes-network-uplink@eth0.service usually) shuts down before the remote-fs.target. So editing /usr/lib/systemd/system/qubes-network-uplink@.service and adding a dependency there fixes it. I added the following dependency:

Before=remote-fs.target

SteveC · January 27, 2023, 5:26pm

I plan to give that a shot in my samba-using qubes to see if it works. Vastly cleaner than a big file that has to be different for different shares.

Thanks!!

SteveC · February 1, 2023, 4:54am

Forgot about this until now.

Your solve does work on my system. Thank you!!!

brendanhoar · February 1, 2023, 3:09pm

This has also been addressed in 4.1 testing repo.

B

SteveC · February 1, 2023, 11:28pm

…and I spoke too soon, it only works in one of three places it’s needed. I’m trying to figure out why. But I’ve always got my old method to fall back on.

Rudd-O · February 21, 2023, 11:08am

Add mount option _netdev to the shares, so that network is not deinitialized before unmounting the shares. If you set up the share mount via a systemd unit, add an After=network-online.target dependency to the unit, in addition to the Before=remote-fs.target dependency. That’ll keep the mount and unmount attempts bracketed between the network coming online and the remote file system target being completed.

Good luck!