R4.2 Time/Date/Clock Synchronization problem

Johnboy · January 18, 2024, 6:21am

Hi,

with R4.2 there’s often the problem that my AppVM’s or Templates have a starting date of
Wed Dec 6 01:00:00 CET 2023
(Fri Nov 10 12:26:00 AM UTC 2023 for whonix)

Some VM’s manage to update their time manually. (Firefox in a dispVM complains about invalid certificate|date/time in the future on forum.qubes-os.org but loads properly after a couple of seconds).
Some don’t (e.g. vault).
That’s also a problem for updating whonix, complaining about “invalid certificate chaining” deb.kicksecure.com while fedora-38 and dom0 update without clocksync problems.
I know about sudo qvm-sync-clock, but adding it to /rw/config/rc.local does not seem to be the proper solution.

sudo qubesctl --show-output --skip-dom0 --targets=“whonix-gateway-17” state.sls update.qubes-vm

Summary

whonix-gateway-17:

        ID: update
  Function: pkg.uptodate
    Result: False
   Comment: E: Failed to fetch tor+https://deb.debian.org/debian/dists/bookworm/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch tor+https://deb.debian.org/debian/dists/bookworm-updates/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch tor+https://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch tor+https://deb.debian.org/debian/dists/bookworm-backports/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch tor+https://fasttrack.debian.net/debian/dists/bookworm-fasttrack/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch tor+https://deb.kicksecure.com/dists/bookworm/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch tor+https://deb.whonix.org/dists/bookworm/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Failed to fetch https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses not yet valid certificate.  Could not handshake: Error in the certificate verification. [IP: 127.0.0.1 8082]
            E: Some index files failed to download. They have been ignored, or old ones used instead.
   Started: 00:25:56.415523
  Duration: 27328.863 ms
   Changes:

        ID: notify-updates
  Function: cmd.run
      Name: /usr/lib/qubes/upgrades-status-notify
    Result: False
   Comment: Command "/usr/lib/qubes/upgrades-status-notify" run
   Started: 00:26:23.750485
  Duration: 30311.043 ms
   Changes:   
            ----------
            pid:
                1489
            retcode:
                100
            stderr:
            stdout:

Summary for whonix-gateway-17

Succeeded: 0 (changed=1)
Failed: 2

Total states run: 2
Total run time: 57.640 s

Any ideas?

DVM · January 18, 2024, 9:14am

What is your Clock qube (check in Global Settings)? Can you open a terminal inside it and check the time (timedatectl)?

Johnboy · January 18, 2024, 9:42am

Clock qube: sys-net (i checked “disposable” during installation IIRC)

$ timedatectl
               Local time: Thu 2024-01-18 10:41:10 CET
           Universal time: Thu 2024-01-18 09:41:10 UTC
                 RTC time: Thu 2024-01-18 09:41:10
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Maybe of further interest: I do have Windows installed parallel. The time there is -1 hour (compared to real time/qubes) which I don’t change, and I disabled time synchronization there. It’s mainly for some gaming.

DVM · January 18, 2024, 9:50am

sys-net seems to have the correct date and time. Can you run the same command but in a qube that has the wrong time?

Since it’s a different system, it shouldn’t affect Qubes.

Johnboy · January 18, 2024, 11:44am

I somehow broke the bug with sudo qvm-sync-clock to use sudo apt-get in e.g. whonix-gateway-17.

If it reappears I’ll post again.

whonix-gateway-17 template:

timedatectl 
               Local time: Thu 2024-01-18 11:44:20 UTC
           Universal time: Thu 2024-01-18 11:44:20 UTC
                 RTC time: n/a
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: n/a
          RTC in local TZ: no

Johnboy · January 19, 2024, 6:47am

Well.
AppVM vault gives me:

timedatectl 
               Local time: Wed 2023-12-06 01:28:24 CET
           Universal time: Wed 2023-12-06 00:28:24 UTC
                 RTC time: n/a
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

Whonix updates don’t work again.
Template whonix-gateway-17:

timedatectl 
               Local time: Fri 2023-11-10 00:29:36 UTC
           Universal time: Fri 2023-11-10 00:29:36 UTC
                 RTC time: n/a
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: n/a
          RTC in local TZ: no

Someone call back pink panther

apparatus · January 19, 2024, 7:26am

Check the logs in your vault appvm:

sudo journalctl -b -u qubes-sync-time.service

joe.blough · January 19, 2024, 4:43pm

Check the time in your BIOS and update it if needed. That could be the problem. Mine was off by 8 hours and Qubes was locked to it.

joe.blough · January 24, 2024, 5:20pm

Also try this (remember to shut down the template after install):

Then: In Dom0 sudo qvm-sync-clock

github.com/QubesOS/qubes-issues

Clock does not synchronize

opened 10:36PM - 20 Apr 19 UTC

closed 04:21AM - 30 Nov 19 UTC

sergeyfrolov

T: bug R: duplicate C: other P: default

**Qubes OS version** Qubes release 4.0 (R4.0) **Affected component(s) or fun…ctionality** sys-net clock sync **Brief summary** Clock does not synchronize. `systemd-timesyncd.service` requires `/var/run/qubes-service/clocksync` to be present in `sys-net`, but it is absent, therefore service will not start. I don't think I had done anything specific or time-related to trigger this bug. **Expected behavior** Clock syncs **Actual behavior** Clock doesn't sync. Clock is incorrect both in dom0 and sys-net **Solutions you've tried** After creating `/var/run/qubes-service/clocksync`, restarting the services in sys-net, and running `qvm-sync-clock` in dom0, I got it working.

I’m not sure if you need to reboot during those steps.

Johnboy · January 28, 2024, 8:26am

Took some time, because sometimes it works, sometimes it doesn’t, then it works again …

qvm-run --pass-io --user root whonix-gateway-17 'journalctl -b -u qubes-sync-time.service'
Nov 10 00:25:52 host systemd[1]: qubes-sync-time.service - Update time from ClockVM was skipped because of an unmet condition check (ConditionPathExists=!/etc/sdwdate.d/qubes-sync-time-disabled-by-sdwdate.marker).

apparatus · January 28, 2024, 8:33am

Check logs in your vault AppVM, not in your Whonix TemplateVM.

Johnboy · January 28, 2024, 2:20pm

sudo journalctl -b -u qubes-sync-time.service
Jan 22 01:00:15 vault systemd[1]: Starting qubes-sync-time.service - Update time from ClockVM...
Jan 22 01:00:15 vault systemd[1]: qubes-sync-time.service: Deactivated successfully.
Jan 22 01:00:15 vault systemd[1]: Finished qubes-sync-time.service - Update time from ClockVM.

apparatus · January 28, 2024, 3:11pm

What’s the output of timedatectl in your clock qube (sys-net?)? Does it have correct time?
Check the output of timedatectl in your clock qube and logs in your vault AppVM at the same time.

Qball · January 28, 2024, 4:55pm

Get your GPS clock working and sync when neccessary and you will not have these things happening. This is a very old turd since 2016. If you want Qubes history search for Whonix boot clock randomization. The JR’s theory won the Voitila price for excelence in brainwashing. It does wonders for Kerberos and friends (OTP TOTP etc). The only fool is you for expecting the developers not to sell out. This is life!

SteveC · January 28, 2024, 5:41pm

THANK YOU.

That package was missing from my minimal sys-net-wifi’s template.

One of my systems was off by 8 minutes. The other one goes completely nuts and gains an hour or two per day.

I issued the manual sudo qvm-sync-clock you mentioned. Let’s see, down the road how well it keeps in sync.

Also, you have to restart sys-net (or in my case sys-net-wifi).

joe.blough · January 28, 2024, 8:51pm

You might have to sync the BIOS clock too with your local time.

sudo hwclock -l systohc

My BIOS clock was set to UTC. I’ve been messing with a few motherboards lately and don’t know what caused what, but it took me forever to figure out why the BIOS and OS clock were out of sync sometimes and not others. I thought syncing the OS clock would automatically fix the BIOS clock but that wasn’t the case.

Johnboy · January 30, 2024, 1:09am

sys-net (correct time):

$ timedatectl 
               Local time: Tue 2024-01-30 01:58:16 CET
           Universal time: Tue 2024-01-30 00:58:16 UTC
                 RTC time: Tue 2024-01-30 00:58:16
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

$ sudo journalctl -b -u qubes-sync-time.service
-- No entries --

vault (incorrect time):

$ timedatectl 
               Local time: Mon 2024-01-22 01:14:40 CET
           Universal time: Mon 2024-01-22 00:14:40 UTC
                 RTC time: n/a
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

$ sudo journalctl -b -u qubes-sync-time.service
Jan 22 01:00:22 vault systemd[1]: Starting qubes-sync-time.service - Update time from ClockVM...
Jan 22 01:00:18 vault systemd[1]: qubes-sync-time.service: Deactivated successfully.
Jan 22 01:00:18 vault systemd[1]: Finished qubes-sync-time.service - Update time from ClockVM.

apparatus · January 30, 2024, 1:56am

What’s the output of this command in vault?

qrexec-client-vm --filter-escape-chars-stderr '@default' qubes.GetDate+nanoseconds

Do you have any policy errors in dom0 journalctl?
What’s your sys-net TemplateVM?

joe.blough · January 31, 2024, 4:21am

What happens if you add this for the vault settings?
Settings > Services
add clocksync

There’s also this with an open ticket:

Johnboy · February 2, 2024, 2:14pm

in vault:

$ timedatectl 
               Local time: Fri 2024-02-02 02:18:09 CET
           Universal time: Fri 2024-02-02 01:18:09 UTC
                 RTC time: n/a
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

$ qrexec-client-vm --filter-escape-chars-stderr '@default' qubes.GetDate+nanoseconds
2024-02-02T14:09:49,439237555+00:00

It’s based on fedora-38-xfce, same as sys-net.

How do I check for policy errors in dom0?
dom0:

$ sudo journalctl -b|grep -i policy

Feb 02 02:10:47 localhost kernel: Policy zone: Normal
Feb 02 02:10:47 localhost kernel: iommu: DMA domain TLB invalidation policy: lazy mode 
Feb 02 02:10:47 localhost usbguard-daemon[420]: [1706836247.826] (W) PERMISSIONS CHECK ON POLICY FILE ARE TURNED OFF!
Feb 02 02:11:08 dom0 usbguard-daemon[2761]: [1706836268.885] (W) PERMISSIONS CHECK ON POLICY FILE ARE TURNED OFF!
Feb 02 02:11:09 dom0 polkitd[2838]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
Feb 02 02:11:09 dom0 systemd[1]: Started qubes-qrexec-policy-daemon.service - Qubes remote exec policy daemon.
Feb 02 02:11:09 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-qrexec-policy-daemon comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 02 02:11:09 dom0 qrexec-policy-daemon[2894]: warning: !compat-4.0 directive in file /etc/qubes/policy.d/35-compat.policy line 16 is transitional and will be deprecated
Feb 02 02:11:18 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.InputKeyboard+: sys-usb -> @adminvm: allowed to dom0
Feb 02 02:11:18 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.InputKeyboard+: sys-usb -> @adminvm: allowed to dom0
Feb 02 02:11:18 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.InputMouse+: sys-usb -> @adminvm: allowed to dom0
Feb 02 02:11:21 dom0 at-spi-bus-launcher[4423]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +15: Eavesdropping is deprecated and ignored
Feb 02 02:11:21 dom0 at-spi-bus-launcher[4423]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +17: Eavesdropping is deprecated and ignored
Feb 02 02:11:24 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: sys-usb -> @default: allowed to dom0
Feb 02 02:11:24 dom0 at-spi-bus-launcher[4559]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +15: Eavesdropping is deprecated and ignored
Feb 02 02:11:24 dom0 at-spi-bus-launcher[4559]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +17: Eavesdropping is deprecated and ignored
Feb 02 02:11:25 dom0 systemd[1]: Starting rtkit-daemon.service - RealtimeKit Scheduling Policy Service...
Feb 02 02:11:25 dom0 systemd[1]: Started rtkit-daemon.service - RealtimeKit Scheduling Policy Service.
Feb 02 02:11:25 dom0 polkitd[2838]: Registered Authentication Agent for unix-session:2 (system bus name :1.40 [/usr/libexec/xfce-polkit], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale de_DE.utf8)
Feb 02 02:11:31 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.WindowIconUpdater+: sys-usb -> @adminvm: allowed to dom0
Feb 02 02:11:37 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.WindowIconUpdater+: vault -> @adminvm: allowed to dom0
Feb 02 02:11:37 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.WindowIconUpdater+: sys-net -> @adminvm: allowed to dom0
Feb 02 02:11:41 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: vault -> @default: allowed to dom0
Feb 02 02:11:43 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.WindowIconUpdater+: sys-whonix -> @adminvm: allowed to dom0
Feb 02 02:11:47 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.WindowIconUpdater+: disp7630 -> @adminvm: allowed to dom0
Feb 02 02:11:52 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: disp7630 -> @default: allowed to dom0
Feb 02 02:13:18 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: vault -> @default: allowed to dom0
Feb 02 02:17:34 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: vault -> @default: allowed to dom0
Feb 02 02:17:37 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: vault -> @default: allowed to dom0
Feb 02 02:18:13 dom0 qrexec-policy-daemon[2894]: qrexec: qubes.GetDate+nanoseconds: vault -> @default: allowed to dom0

I don’t know if this matters:
On boot only sys-usb boots up. Only after I login, other qubes are loaded (sys-net, vault, …)

R4.2 Time/Date/Clock Synchronization problem

whonix-gateway-17:

Summary for whonix-gateway-17

Succeeded: 0 (changed=1) Failed: 2

Succeeded: 0 (changed=1)
Failed: 2