Programming approaches to qubesadmin.tools (qvm-ls, ...) improvements

General Discussion
1

TL;DR: qvm-ls tweaks

I wanted these options:

  • Unix like Wildcard support for VM selection. e.g. qvm-ls fedora*, qvm-ls *template, …
    Since using grep with Regex is not always convenient (and is considerably slower).
  • A comprehensive pref output format for qvm-ls
  • Filtering of VMs based on their class or label
  • Filtering of VMs based on internal or servicevm features key.

So I implemented them here.

Table of Content

  • Background to my qubesadmin.tools tweaks
  • qvm-ls tweaks.
  • Historic technical decisions, Shared libraries, Easter eggs.
  • Other parts of qubes-core-admin-client repository.

Background

I felt that the good old qvm-ls(1) is no more sufficient for my needs and decided to tweak it a little bit. It is not always convenient to pipe qvm-ls output through various coreutils (grep, awk, sed, …). This is how I found this gem of Qubes repositories.

Historic technical decisions, Shared libraries, Easter eggs.

The upstream repository has very interesting educational and sometimes fun parts. If you have ever opened qvm-ls (or most other qvm admin tools) in a text editor, you should have recognized that strangely they are almost empty Python scripts, only importing the main code as a library. Actual code is stored in qubesadmin.tools library of qubes-core-admin-client repository as individual .py files. There is a historic commit in setup.py doing this to just save 100 milliseconds since Python libraries are compiled on import rather than interpreted entry points. Considering many Qubes user are still on 2nd Gen Intel CPUs, it is justifiable. Otherwise if you copy the qvm_ls.py to your ~/bin directory, rename it to anything, make it executable and add a #!/usr/bin/python3 line on top of the file, it will work exactly like the original qvm-ls. And you can add most of your Tweaks directly inside it. But that would be ugly. Since they are already libraries, I easily imported them and derived new classes with modifications only to routines I needed. There is already a nice hint by Mark inviting adding new formats (and actually a rather simple disabled perf format). I needed maxmem which was Todo in original file so I implemented it. Adding of filtering options is straight forward with Python argparse and some simple code.

Wildcard support was not as easy as VMNAME input is handled via QubesArgumentParser, VmNameAction & VmNameGroup classes of shared part of qubesadmin.tools library. On the positive side, if you implement them there, you can use them for all other qubesadmin.tools which support it (qvm-start, qvm-pause, qvm-remove`, …). And it runs much faster because of fewer API calls.

time qvm-ls-tt fedora*

takes just 618 milliseconds on this machine whereas

time qvm-ls | grep -e '^\(fedora\|NAME\).*'

takes 2268 milliseconds.

The Easter egg is these nice quotes inside the spinner.py inserted by Marek & Wojtek Porczyk. It made my day. It made me write my own spinner with braille alphabet.

3 Likes
2

This is really awesome ! I already read your other post but this is exactly something I’ve been thinking about : I want to do some filtering with qvm-devices and your repo give me a good idea of how to proceed. Thanks !

1 Like
3

Added --basedon to be able to filter output list to VMs based on specific template. Also --connects-with to be able to filter results to VMs using specific NetVM. Sorting based on prefered column is almost done. Sorting VMs based on memory usage is essential for myself.

2 Likes
4

I’m somewhat puzzled because you dont say why using grep is not always
convenient and the timings you report are very far from my
experience.

I use a basic shell script that gives me most of what you’ve
implemented here, including wildcard support, sort by memory, filtering
by NETVM, outputting by network tree, etc etc.
The speed on this x220 is around what you report for your qvm-ls-tt - I
find it hard to believe your machine is slower than the x220

I use the existing features of qvm-ls, including field selection, filtering by running,
tree output, piped to grep, awk or sort.
eg qvm-ls --running -O NAME,MEMORY |grep -v MEM|sort -h -k2 -r

I dont want to dissuade you from your efforts, but it doesnt seem to fit
any need to/for me.

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.
5

Interesting. The machine is an Elitebook 820 G1 (HCL), Are you benchmarking in dom0 or within an AdminVM where RPC policies come into play?

There are other things that I am going to implement which are not possible via the current qubesadmin tools. Such as ANSI color of labels in terminal. Last access time when the VM was powered on. USB Power-off and some other ideas. Or maybe they are possible?