Bees and brtfs deduplication

I tried to build qubes to have it build qubes 4.2 and integrate bees spec file of opensuse tumbleweed but was unsuccessful just building qubes from qubes-builder last month.

It was failing at get sources step with issues with fetchrd commit from a submodule… Anyone here was successful recently?

I guess i could try only to build fedora-37 template and bees there. Poking here in advance to augment my chances of success!

Thanks

1 Like

Houla that was a long ride to arrive to this point (still needing help though, not building yet with no debug information to chew on).

I finally was able to use qusal to deploy qubes-builderv2 (Qubes Salt Beginner's Guide - #42 by Insurgo)

  • Note: Following BOOTSTRAPPING.md is required to deploy dom0 requirements in proper order. The order of installing dependencies is not that clear for salt beginners and things fail if dom0 requirements aren’t filled (will try to open issues or PR soon enough there)

Once qubes-builderv2 is deployed (and confirmed working), I tried to follow guidelines I inferred from other builderv2 migrated from builderv1 to the point of having the following, which doesn’t build anything as of now. @marmarek @fepitre if you have any insights, that would be awesome.

qubes-builderv2’s builder.conf:

git:
  baseurl: https://github.com
  prefix: QubesOS/qubes-
  branch: release4.2
  maintainers:
    # marmarek
    - '0064428F455451B3EBE78A7F063938BA42CFA724'
    # simon
    - '274E12AB03F2FE293765FC06DA0434BC706E1FCF'

backend-vmm: xen
debug: true
verbose: true
qubes-release: r4.2

distributions:
  - host-fc37

artifacts-dir: /home/user/src/qubes-builderv2/artifacts

components:
  - builder-rpm:
      branch: main
      packages: False
  - bees:
      branch: main
      packages: False
      url: https://github.com/tlaurion/qubes-bees
      maintainers:
        - '0ACCB2B664EE17E054B05E0B4A38DA8BEB9C8396'

executor:
  type: qubes
  options:
    dispvm: "dom0"

stages:
  - fetch
  - prep
  - build
  - post
  - verify
  - sign:
      executor:
        type: local
  - publish:
      executor:
        type: local
  - upload:
      executor:
        type: local

I deployed my public key (which signed commits and tags: keys.openpgp.org) under qubesbuilder/plugins/fetch/keys/ which is recognized properly. It fetches, verifies tag and puts content of git repo correctly under bees directory, but it seems i’m missing something under .qubesbuilder (Makefile.builder not required anymore, I inferred where only that file is required + bees.spec) since bees tarball is not put in place as can be seen later on.

The builder only does the following (putting git repo content at the right place but not telling anything else):

[user@qubes-builder ~/src/qubes-builderv2(main)]
(130)$ ./qb --verbose -c bees package all
Running stage: fetch
13:27:23,198 [fetch] bees: source already fetched. Updating.
13:27:53,774 [executor:qubes:disp8379] copy-in (cmd): /usr/lib/qubes/qrexec-client-vm -- disp8379 qubesbuilder.FileCopyIn+-2Fbuilder-2Fbees /usr/lib/qubes/qfile-agent /home/user/src/qubes-builderv2/artifacts/sources/bees
13:27:55,040 [executor:qubes:disp8379] copy-in (cmd): /usr/lib/qubes/qrexec-client-vm -- disp8379 qubesbuilder.FileCopyIn+-2Fbuilder-2Fplugins-2Ffetch /usr/lib/qubes/qfile-agent /home/user/src/qubes-builderv2/qubesbuilder/plugins/fetch
13:28:00,838 [executor:qubes:disp8379] Executing '/usr/bin/qvm-run-vm -- disp8379 env -- VERBOSE=1 DEBUG=1 BACKEND_VMM=xen bash -c 'cd /builder && /builder/plugins/fetch/scripts/get-and-verify-source.py https://github.com/tlaurion/qubes-bees /builder/bees /builder/keyring /builder/plugins/fetch/keys --git-branch main --minimum-distinct-maintainers 1 --maintainer 0ACCB2B664EE17E054B05E0B4A38DA8BEB9C8396''.
13:28:02,241 [executor:qubes:disp8379] output: --> Verifying tags...
13:28:02,241 [executor:qubes:disp8379] output: ---> Good tag 596a650b16a908562e210638a03eb90fec7a759c.
13:28:02,241 [executor:qubes:disp8379] output: Enough distinct tag signatures. Found 1, mandatory minimum is 1.
13:28:02,241 [executor:qubes:disp8379] output: --> Merging...
13:28:02,266 [executor:qubes:disp8379] copy-out (cmd): /usr/lib/qubes/qrexec-client-vm disp8379 qubesbuilder.FileCopyOut+-2Fbuilder-2Fbees /usr/bin/qfile-unpacker 1000 /home/user/src/qubes-builderv2/artifacts/sources
Running stage: prep
Running stage: build
Running stage: post
Running stage: verify
Running stage: sign
Running stage: publish
Running stage: upload
13:28:12,339 [upload] host-fedora-37.x86_64: No remote location defined. Skipping.
[user@qubes-builder ~/src/qubes-builderv2(main)]

What am I missing under .qubesbuilder? It seems that the bees.spec is not considered, nor the downloading of prerequisites also specified into .quebesbuilder file above. Only content of the git repo is properly deployed, as can be seen here:

(130)$ ls -al /home/user/src/qubes-builderv2/artifacts/sources/bees/
total 72K
drwxr-xr-x 4 user user 4.0K Feb  9 13:05 ./
drwxr-xr-x 8 user user 4.0K Feb  9 13:28 ../
drwxr-xr-x 8 user user 4.0K Feb  9 13:28 .git/
drwxr-xr-x 2 user user 4.0K Feb  9 10:01 rpm_spec/
-rw-r--r-- 1 user user 4.2K Feb  9 10:01 bees.changes
-rw-r--r-- 1 user user  35K Feb  9 10:01 LICENSE
-rw-r--r-- 1 user user  250 Feb  9 13:05 .qubesbuilder
-rw-r--r-- 1 user user  102 Feb  9 10:01 README.md
-rw-r--r-- 1 user user   65 Feb  9 10:01 v0.10.tar.gz.sha256
[user@qubes-builder ~/src/qubes-builderv2(main)]

At the time of writing, here is the content of my .qubesbuilder under git repo:

$ cat /home/user/src/qubes-builderv2/artifacts/sources/bees/.qubesbuilder
host:
  rpm:
    build:
    - rpm_spec/bees.spec
source:
  modules:
  - gcc-c++
  - libbtrfs-devel
  - libuuid-devel
  - make
  - autotools
  files:
  - url: https://github.com/Zygo/bees/archive/refs/tags/v0.10.tar.gz
    sha256: v0.10.tar.gz.sha256

The repo (changing since not working) is at GitHub - tlaurion/qubes-bees: Best effort BRTFS offline deduplication, based on OpenSuse Tumbleweed rpm spec inclusion
Any advice welcome: i’m not understanding what i’m missing from linux example at GitHub - QubesOS/qubes-builderv2: Next generation of Qubes OS builder

Was told over matrix that @VERSION@ placeholders under bees.spec.in file needed to be placed and version file dropped under root of repo, which I did.

user@heads-tests-deb12-nix:~/qubes-bees$ ls -al *
-rw-r--r-- 1 user user   4239 Feb  8 16:44 bees.changes
-rw-r--r-- 1 user user  35149 Feb  8 16:34 LICENSE
-rw-r--r-- 1 user user    102 Feb  8 16:34 README.md
-rw-r--r-- 1 user user 177509 Feb  9 10:15 v0.10.tar.gz
-rw-r--r-- 1 user user     65 Feb  9 01:30 v0.10.tar.gz.sha256
-rw-r--r-- 1 user user      5 Feb  9 15:13 version

rpm_spec:
total 12
drwxr-xr-x 2 user user 4096 Feb  9 16:56 .
drwxr-xr-x 4 user user 4096 Feb  9 16:56 ..
-rw-r--r-- 1 user user 2390 Feb  9 15:42 bees.spec.in
user@heads-tests-deb12-nix:~/qubes-bees$ cat rpm_spec/bees.spec.in 
#
# spec file for package bees
#
# Copyright (c) 2023 SUSE LLC
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.

# Please submit bugfixes or comments via https://bugs.opensuse.org/
#


Name:           qubes-bees
Version:	@VERSION@
Release:	1%{?dist}

Summary:        Best-Effort Extent-Same, a btrfs deduplication agent
License:        GPL-3.0-only
Group:          System/Filesystems
URL:            https://github.com/Zygo/bees
Source:         https://github.com/Zygo/bees/archive/refs/tags/v%{version}.tar.gz
BuildRequires:  gcc-c++
BuildRequires:  libbtrfs-devel
BuildRequires:  libuuid-devel
BuildRequires:  make

%description
bees is a block-oriented userspace deduplication agent designed for large btrfs
filesystems. It is an offline dedupe combined with an incremental data scan
capability to minimize time data spends on disk from write to dedupe.

Hilights:

* Space-efficient hash table and matching algorithms - can use as little as 1
  GB hash table per 10 TB unique data (0.1GB/TB)
* Daemon incrementally dedupes new data using btrfs tree search
* Works with btrfs compression - dedupe any combination of compressed and uncompressed files
* Persistent hash table for rapid restart after shutdown
* Whole-filesystem dedupe - including snapshots
* Constant hash table size - no increased RAM usage if data set becomes larger
* Works on live data - no scheduled downtime required
* Automatic self-throttling based on system load

%prep
%autosetup -p1

%build
cat >localconf <<-EOF
	SYSTEMD_SYSTEM_UNIT_DIR=%{_unitdir}
	LIBEXEC_PREFIX=%{_bindir}
	LIB_PREFIX=%{_libdir}
	PREFIX=%{_prefix}
	LIBDIR=%{_libdir}
	DEFAULT_MAKE_TARGET=all
EOF

%make_build BEES_VERSION=%{version}

%install
%make_install

%files
%license COPYING
%doc README.md
%{_bindir}/bees
%{_sbindir}/beesd
%{_unitdir}/beesd@.service
%dir %{_sysconfdir}/bees
%{_sysconfdir}/bees/beesd.conf.sample

%changelog

user@heads-tests-deb12-nix:~/qubes-bees$ cat .qubesbuilder
host:
  rpm:
    build:
    - rpm_spec/bees.spec
source:
  modules:
  - gcc-c++
  - libbtrfs-devel
  - libuuid-devel
  - make
  - autotools
  files:
  - url: https://github.com/Zygo/bees/archive/refs/tags/v0.10.tar.gz
    sha256: v0.10.tar.gz.sha256

This is why nothing is happening. This option is telling qubes-builderv2 that there is nothing to build here. This mostly for builder-* components that are used as plugin.

Also, I don’t see any git submodule so this is failing. According to the README: Declare submodules to be included inside source preparation

I made some progress but still not there yet.

Reminder: my attempts (will get commit log clean when successful) is at GitHub - tlaurion/qubes-bees: Best effort BRTFS offline deduplication, based on OpenSuse Tumbleweed rpm spec inclusion

To replicate:

Then have the following builder.yml:

(1)$ cat builder.yml
git:
  baseurl: https://github.com
  prefix: QubesOS/qubes-
  branch: release4.2
  maintainers:
    # marmarek
    - '0064428F455451B3EBE78A7F063938BA42CFA724'
    # simon
    - '274E12AB03F2FE293765FC06DA0434BC706E1FCF'

backend-vmm: xen
debug: true
verbose: true
qubes-release: r4.2

distributions:
  - host-fc37

artifacts-dir: /home/user/src/qubes-builderv2/artifacts

components:
  - builder-rpm:
      branch: main
      packages: False
  - bees:
      branch: main
      url: https://github.com/tlaurion/qubes-bees
      maintainers:
        - '0ACCB2B664EE17E054B05E0B4A38DA8BEB9C8396'

executor:
  type: qubes
  options:
    dispvm: "dom0"

stages:
  - fetch
  - prep
  - build
  - post
  - verify
  - sign:
      executor:
        type: local
  - publish:
      executor:
        type: local
user@heads-tests-deb12-nix:~/qubes-bees$ cat .qubesbuilder
host:
  rpm:
    build:
    - rpm_spec/bees.spec
source:
  create-archive: true
  files:
  - url: https://github.com/Zygo/bees/archive/refs/tags/v0.10.tar.gz
    sha256: v0.10.tar.gz.sha256
user@heads-tests-deb12-nix:~/qubes-bees$ cat rpm_spec/bees.spec.in 
#
# spec file for package bees
#
# Copyright (c) 2023 SUSE LLC
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.

# Please submit bugfixes or comments via https://bugs.opensuse.org/
#


Name:           bees
Version:	@VERSION@
Release:        1%{?dist}

Summary:        Best-Effort Extent-Same, a btrfs deduplication agent
License:        GPL-3.0-only
Group:          System/Filesystems
URL:            https://github.com/Zygo/bees
Source:         https://github.com/Zygo/bees/archive/refs/tags/v%{version}.tar.gz
BuildRequires:  gcc-c++
BuildRequires:  util-linux-core
BuildRequires:  systemd-devel
BuildRequires:  make
Requires:       btrfs-progs
Requires:       systemd
Requires:       util-linux

# This removes errors from the build process, but it's not a good practice (fc37 related)
%define CXXFLAGS %{?CXXFLAGS} -Wno-error=restrict

%description
bees is a block-oriented userspace deduplication agent designed for large btrfs
filesystems. It is an offline dedupe combined with an incremental data scan
capability to minimize time data spends on disk from write to dedupe.

%prep
%autosetup

%build
cat >localconf <<-EOF
	SYSTEMD_SYSTEM_UNIT_DIR=%{_unitdir}
	LIBEXEC_PREFIX=%{_bindir}
	LIB_PREFIX=%{_libdir}
	PREFIX=%{_prefix}
	LIBDIR=%{_libdir}
	DEFAULT_MAKE_TARGET=all
EOF

%make_build BEES_VERSION=%{version}

%install
%make_install

%files
%license COPYING
%doc README.md
%{_bindir}/bees
%{_sbindir}/beesd
%{_unitdir}/beesd@.service
%dir %{_sysconfdir}/bees
%{_sysconfdir}/bees/beesd.conf.sample

%changelog
* Mon Feb 12 2024 Thierry Laurion <insurgo@riseup.net> - 0.10-1
- Initial package for fedora 37

I do not find a practical way to get -Wno-error=restrict passed down, which results in the following:

[user@qubes-builder ~/src/qubes-builderv2(main)]
(130)$ ./qb --verbose -c bees package all
[…]
13:43:22,970 [executor:qubes:disp4657] output: DEBUG: g++ -Wall -Wextra -Werror -O3 -I…/include -D_FILE_OFFSET_BITS=64 -std=c++11 -Wold-style-cast -Wno-missing-field-initializers -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -o ntoa.o -c ntoa.cc
13:43:22,970 [executor:qubes:disp4657] output: DEBUG: make[1]: Leaving directory ‘/builddir/build/BUILD/bees-0.10/lib’
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: In file included from /usr/include/c++/12/string:40,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: from …/include/crucible/ntoa.h:4,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: from ntoa.cc:1:
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: In function ‘std::char_traits::copy(char*, char const*, unsigned long)’,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: inlined from ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_S_copy(char*, char const*, unsigned long)’ at /usr/include/c++/12/bits/basic_string.h:431:21,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: inlined from ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_S_copy(char*, char const*, unsigned long)’ at /usr/include/c++/12/bits/basic_string.h:426:7,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: inlined from ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)’ at /usr/include/c++/12/bits/basic_string.tcc:532:22,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: inlined from ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::assign(char const*)’ at /usr/include/c++/12/bits/basic_string.h:1655:19,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: inlined from ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::operator=(char const*)’ at /usr/include/c++/12/bits/basic_string.h:823:28,
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: inlined from ‘crucible::bits_ntoa[abi:cxx11](unsigned long long, crucible::bits_ntoa_table const*)’ at ntoa.cc:31:10:
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: /usr/include/c++/12/bits/char_traits.h:435:56: error: ‘memcpy’ accessing 9223372036854775810 or more bytes at offsets [2, 9223372036854775807] and 1 may overlap up to 9223372036854775813 bytes at offset -3 [-Werror=restrict]
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: 435 | return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: | ^
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: cc1plus: all warnings being treated as errors
13:43:22,971 [executor:qubes:disp4657] output: DEBUG: make[1]: *** [Makefile:39: ntoa.o] Error 1
13:43:22,972 [executor:qubes:disp4657] output: DEBUG: make[1]: *** Waiting for unfinished jobs…

Full log:
qubes-bees_builerror.log (348.1 KB)
@fepitre @marmarek further advices?

@moderators : where should this post be moved to get more eyes? General?

Still no luck. Any help welcome