That’s exactly what I am doing. Thanks for the links too.
My follow up questions arise from the difference between what the docs (including warnings), @unman and you suggest as proper commands.
Running the command isn’t the same as updating. You have to read what is written. It’s not the command that is “dangerous” but what it does and what you are letting it to do.
I understand that running a command and updating can be different (in case the command would ask the user whether to continue) or the same. Being new to the process, I had no way to know which one was the case and as it is a sudo command, considering also the warning in the docs (which I have read), I thought it would be better to ask before doing anything without understanding what the result would be. I hope that clarifies that the reason for the follow up questions.
Considering the obvious need for extra “education” and the additional work one has to do for something as simple as knowing how much an update will download I think it is appropriate to have it as a feature.
apt update gets the latest information from the repositories. apt full-upgrade will start the process of downloading the updates,
but won’t actually do anything unless you say “Yes” when prompted. It
will tell you what packages would be updated and how much would be
downloaded.
I thought I had included both commands, but apparently not.
I’m fairly sure that this is already covered in a GitHub issue.
I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.
I have just noticed something which seems Qubes specific:
After updating all templates (there is currently no update for dom0) in the Debian template I ran apt dist-upgrade and it showed that linux-image-5.10.0-10-amd64 will be removed. I allowed that.
Does that mean that Qubes Update does not run apt dist-upgrade thus not allowing the cleanup this command allows? If yes, do you think this is an issue to be reported? If no, what is the intention?
*I understand this is somewhat off-topic but I thought it might be good to clarify that too, as it is directly related to updates.
You are right, Qubes Update doesn’t run dist-upgrade.
That’s fine. In most cases it’s better not to run that, especially
non-interactively.
dist-upgrade and full-upgrade are both capable of removing packages,
whereas upgrade will not. Removal is probably fine for old kernel packages
(probably), but can cause confusion otherwise.
I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.
So, you would like to know the size of a repo before you sync it?
That would rely on the repo server actually telling you, and most don’t.
I don’t know if you can say that any more. Nice try
Nice!
You learn by experience. Don’t be afraid to break things. The knowledge and skills you build by fixing them is second-to-none.
(don’t break any production machine, of course…)
You can never have enough education. You never stop learning. Knowledge is power
You basically told Debian to upgrade EVERY single package, even if it breaks stuff. Some packages rely on specific versions of software in order to work (Python is a notorious example of this. “this function is ‘old’, let’s get rid of it!”).
“Security reasons” basically means that the updates are obtained in another VM, and then passed through into the template via qrexec. The package managers in the Debian, Fedora, Arch, CentOS and Gentoo templates have been configured so that they will work if you open a terminal and type in their respective update command.
The only main difference is that whatever system updates you download will be cached inside your template instead of a disposable management VM (and I’m sure you know that everything in your template is visible in the AppVMs you base on that template).
This one is less about preventing nasty things from getting in, but more about limiting the amount of unique information they can get about where they got into, if anything does get in.
EDIT:
This is fairly important, too. Yes there are ways to get the same results via command line, but the GUI is more of a convenience tool.
Not at all. As far as your network traffic goes, you’ve basically done a repo sync, and that’s all any third party would be able to see.
That would likely require quite a lot of back-and-forth via qrexec between dom0 and the VMs. Security concerns aside, it would be a lot of hacking stock-standard distribution package managers, which would require a custom version of each template’s package manager. It would only really be worth the effort if it was accepted upstream, and I honestly don’t see that happening, unfortunately…
I understand where you’re coming from with zypper, pacman, dnf, apt, and emerge showing the size of each package, but any solution like that would need to coexist with vanilla package managers from the distributions, to minimise maintenance work.
But hey, it can’t hurt to come up with a design and suggest it
It shows the details of the qubesctl command, not the package manager inside the template.
Yes, what you’re suggesting is a button that would open a window showing something like:
I can explain this. In Qubes 4.0, a separate window didn’t open, and the output of the dnf command was shown in red in the dom0 terminal.
You’re right in the sense that qubes-dom0-update by default automatically syncs the repos and downloads any and all new packages without asking first. The confirmation for the user is only presented once they are downloaded and moved into dom0, so I’ll give you that one.
What currently happens: qubes-dom0-update → sync repos → download any and all updates → move to dom0 → show updates and sizes → user confirmation → install updates
What @qubist is suggesting: qubes-dom0-update → sync repos → show updates and sizes → user confirmation → download → move to dom0 → install updates
I can see what your concern is with this process. I have updated Qubes OS with mobile data in the past, and have used my entire month’s allowance in 10 minutes. Yes, I know that the simple answer would be to just find a better internet connection, but if that isn’t an option for some people for whatever reason, knowing whether you’re downloading a security update or a completely new version of Xen before you do it would actually be helpful
Salt scripts weren’t exactly designed to do things like that.
That doesn’t mean that extra grains couldn’t be created in the pillars to show this, though…
So just to clarify, what you want is:
The ability to be able to see the size of updates at a confirmation screen before any data packets are transferred
Fairly feasible and straightforward
The ability to see progress and log updates in the Qubes Updater GUI
Less straightforward and a little “hacky”, but just as feasible
The ability to see how much data will be sent/received before a repo sync
Impossible, unless you already know the size of the repo…
Am I right?
On a side-note, there is actually a Qubes OS Tumbleweed template out there “in the wild”, and I have been trying to get my hands on it.
I don’t know if you can say that any more. Nice try
Hehe Well, maybe not a 100% noob any more but still fairly new.
You learn by experience. Don’t be afraid to break things. The knowledge and skills you build by fixing them is second-to-none.
Sure. I have already reinstalled the whole Qubes OS 4 times (or were they 5?). Not because I broke it but out of desire to (re)make things the right way. I am trying to reduce the rate of reinstalling by refining my knowledge rather than by breaking and fixing more things.
You can never have enough education. You never stop learning. Knowledge is power
On a lifetime scale - sure. Still, in the context of this thread, I hope you would agree it should not be necessary to become an expert in order to do such simple things. So by “education” and “additional work” I meant all the time and energy spent by all the people taking part in this thread for outlining a solution to a very simple problem which is already solved by most (perhaps all) distros/OSs. In the case of Qubes OS the info is obviously already there (or quite easily retrievable) - it just needs to be shown properly.
You basically told Debian to upgrade EVERY single package, even if it breaks stuff.
IIUC dist-upgrade removes only dependencies which were but are no longer necessary. Why would that break stuff? For so many years openSUSE’s zypper up/dup has never broken anything for me. Are you saying that Debian’s dependency handling is fragile and unreliable?
“Security reasons” basically means […]
Thanks for explaining. The warning is already removed, as mentioned earlier, and now those 2 Salt formulae are suggested as a command line alternative to using the GUI Qubes Updater.
The only main difference is that whatever system updates you download will be cached inside your template […] if anything does get in.
Would using those 2 Salt formulae result in such caching (and all its implications which you and @unman explained) or is it identical to using the GUI updater?
Also, will using the 2 Salt formulae result in missing those
The GUI Update tool is sometimes used to make configuration changes in dom0
and in templates. If you don’t use it then you will miss these.
Does this mean that by using the 2 Salt formulae from the docs one will miss those changes?
This is fairly important, too. Yes there are ways to get the same results via command line, but the GUI is more of a convenience tool.
What are these ways? Where can I read more?
That would likely require quite a lot of back-and-forth via qrexec between dom0 and the VMs. Security concerns aside, it would be a lot of hacking stock-standard distribution package managers, which would require a custom version of each template’s package manager. It would only really be worth the effort if it was accepted upstream, and I honestly don’t see that happening, unfortunately…
Why a lot of back-and-forth and hacking? Is it not as simple as running qvm-run with the commands suggested here in the template, parsing its output and outputting it to the GUI? Even I can do it in a bash script (but not in GUI). What security concerns do you see in that?
You’re right in the sense that qubes-dom0-update by default automatically syncs the repos and downloads any and all new packages without asking first. The confirmation for the user is only presented once they are downloaded and moved into dom0, so I’ll give you that one.
What currently happens: qubes-dom0-update → sync repos → download any and all updates → move to dom0 → show updates and sizes → user confirmation → install updates
Just to note: Currently the “sync repos” step takes about 150 MiB of download and about 10 MiB of upload traffic. I don’t know why this is so huge but it is really not quite acceptable for one who is on a limited mobile traffic. You know that from experience I wonder if that can be optimized through e.g:
Proper usage of HTTP caching headers
Using “metadata for the metadata”: a simple text file on the repo can contain the total size of the “sync repos” data and a checksum. Then checking for updates would not require 150+ MiB but a comparison of the remote and local copies of that text file (assuming nothing is cached locally. If it is, this can be solved too). IOW this:
What @qubist is suggesting: qubes-dom0-update → sync repos → show updates and sizes → user confirmation → download → move to dom0 → install updates
Turns into:
qubes-dom0-update → “Is there an update at all?” → “Would you like to sync repos?” → Y → tell the user how much ‘sync repos’ would download and suggest to the user to sync repos for more details (e.g. when pressing the Details button) → sync repos → show updates and sizes → user confirmation → download → move to dom0 → install updates
Salt scripts weren’t exactly designed to do things like that.
I had never heard of Salt before using Qubes OS, so this is new to me. Where can I see the source code of those scripts? Maybe we can suggest a redesign based on the parsing mentioned above.
So just to clarify, what you want is:
The ability to be able to see the size of updates at a confirmation screen before any data packets are transferred
Fairly feasible and straightforward
Yes.
The ability to see progress and log updates in the Qubes Updater GUI
Less straightforward and a little “hacky”, but just as feasible
That would mean actual Details. Otherwise the Details button makes no sense before anything is updated. Either it should be removed (which is even worse) or it should function properly matching at each phase of the process.
The ability to see how much data will be sent/received before a repo sync
Impossible, unless you already know the size of the repo…
I wonder if there is a way to know the size of the repo, i.e. if repos contain such “metadata for the metadata”. Otherwise perhaps an HTTP head request (curl -I) could give the size of the ‘sync repo’ data before downloading it. Would that make it possible?
On a side-note, there is actually a Qubes OS Tumbleweed template out there “in the wild”, and I have been trying to get my hands on it.
IMHO, this makes sense, since the Linux kernel version cannot be upgraded with just apt-get upgrade, which means that, if the Qubes Update tool were to use only apt upgrade (and not apt dist-upgrade), then users who update solely via the Qubes Update tool would never get kernel version upgrades.
This is not quite accurate. The main reason has to do with the fact that Salt is capable of delivering security fixes that standard package manager updates cannot (see here for more).
I believe that’s what this issue is about:
The Salt commands are equivalent to using the GUI updater. The documentation states this: “Advanced users may wish to perform updates via the command-line interface. The recommended way to do this is by using the command-line equivalents of the Qubes Update tool. [Then goes on to list the Salt commands.]”
[Side note: I just noticed we still have a 4.0 section here, so I’ll remove that now.]
No, because they are equivalent to using the Qubes Update tool.
Simply by using the Salt commands that we have been discussing, since they are equivalent to using the Qubes Update tool.
That would be a question/complaint for the upstream projects (Fedora and Debian, I suppose). When it comes to this specific aspect of the situation, we are simply using the tools and infrastructure they’ve built.
The perfect example is the Qubes packages inside a template. They have specific dependency versions, and if the distribution repos contain newer versions of dependencies and you force an upgrade, the package manager will usually remove the Qubes packages.
(ahem…ARCHLINUX!!!…sorry, I had something in my throat )
Disclaimer: I love Arch, but my Arch templates break more than my Gentoo templates…
I think you’re putting words in my mouth, and I don’t see how this will benefit anyone.
apt is solid, and there’s a reason why it has stuck around. But like any piece of software, it is only as good as the information you feed it.
(and before you say that I’m insinuating that apt is somehow being fed bad information from any entity, I am saying nothing even remotely close to that).
Because dom0 needs to know what’s going on by receiving status updates from the VMs, hence the back-and-forth.
It may well could be. Communication between process inside a single VM is the same as on any computer, and you know that. Any communication that goes between VMs (and/or dom0) is only done via the qrexec protocol.
As long as there is a communication channel that cannot be (easily) exploited/hijacked, then it’s potentially viable option.
Yeah, you definitely can’t say that you’re “new to Qubes” anymore after that comment
A VM should talk to dom0 as little as possible, and only when absolutely necessarily, in a nutshell. If you open up an extra channel for anything to get out of a VM and into dom0, you potentially increase dom0’s attack surface.
Unfortunately, as awesome as the distribution package managers are, you cannot assume that they are fully “trusted”, so if you were to give them a direct route into dom0, well, I’m sure someone could potentially find a way to exploit that…
Having said this, this wouldn’t be the only way to achieve what you’re suggesting.
It doesn’t seem quite focused on info about traffic usage although there is a mention about bandwidth. I wonder if it means that this is already considered and planned and if it would be appropriate to either suggest it explicitly via a separate issue or within the one you linked to. What do you say?
Although there is a proper solution, if somewhat unwieldy, you can
easily turn off automatic checking. Open Qubes Global Settings. Look at
the “Updater checks” section.
After your message I disabled all checks. Strangely, today I received a notification telling me that fedora-36 and whonix-gw-16 updates are available.
Is that a bug? Or is the proper solution you mentioned different from the easy one?
FWIW this happened while I was running qubes-dom0-update in a dom0 terminal (I was just willing to check if there are any dom0 updates and their size, as we discussed)
If you disabled all checks, that would be a bug. Except that change will not apply to running qubes,
so depending on circumstances, and what was running when you disabled
the checks, it is probably not a bug.
I did disable all checks on the day you told me that this is the way.
The only running qubes were the sys-* ones (firewall, net, usb, whonix). The system was just booted and nothing was started explicitly.
Should I report that?
Another thing I noticed when updating:
fedora-36 template said (dnf update) there would be 439M to be downloaded.
debian template said (apt update && apt dist-upgrade) there would be 128 MiB to be downloaded.
There were no updates for dom0.
However, after running the Salt formula which updates templates only, more than 800 MiB were downloaded and about 60 MiB were uploaded. So, something else generated traffic too, although I didn’t do anything on the machine whatsoever. I tried to add to 439+128 the 149 MiB downloaded by the repo sync of dom0 (i.e. assuming that some “check for updates” might have been running in the background ignoring my settings) but the result was quite below 800 MiB. I also considered the repo sync of fedora and debian but those were negligibly small in size. What might that difference be due to?
