Addition of granular bcachefs formatting and multi-disk support #961

ProjectInitiative · 2025-02-08T15:49:15Z

These changes provide advanced usage of bcachefs formatting options. Because bcachefs provides
a one-stop-shop for wiping, partitioning, formatting and mounting with unique features for
each step, it has been broken out into it's own type if users wish to go beyond a single
disk setup.

examples/bcachefs.nix has been updated to showcase this advanced usage and some of the
available options.

Because all three processes take place via the type object, requiring "disks" attribute set
is not longer a hard requirement if a bcachefs type is provided. A check has been added to
account for that use case.

I will note that I have not gone through the official testing process called out in the contributing guidelines quite yet. I will looking into this soon, just wanted to get this PR out there for visibility :)

I have tested provisioning a VM with a 3 disk pool with this branch using nixos-anywhere, and also running the disko tool manually on another, already provisioned VM.

ProjectInitiative · 2025-02-09T00:30:39Z

I am planning on re-working this to follow the mdadm and zfs arch with mult-drive support, it is proving to be a good challenge.

Mic92 · 2025-02-12T14:06:01Z

Looking forward to it.

ProjectInitiative · 2025-02-12T18:42:13Z

@Lassulus You mentioned wanting to take a look at the current state. This is very rough, but this is what I currently have.

I added in some markers in the individual _create scripts (BCACHEFS POSITION and BCACHEFS_MEMBER POSITION) within bcachefs.nix and bcachefs_member.nix to track the order in which they get collated. When I load it into repl and look at the top level _create function, I see them out of order (member should come first). I am running into an order of ops issue (I think) where I am not properly tracking that disks tagged with bcachefs_member need to be processed first.

 nixosConfigurations.testmachine.config.disko.devices._create
"set -efux\n\ndisko_devices_dir=$(mktemp -d)\ntrap 'rm -rf \"$disko_devices_dir\"' EXIT\nmkdir -p \"$disko_devices_dir\"\n\n( # bcachefs pool1   /mnt/pool #\n  declare -a formatOptions=('--compression=zstd')\n  declare -a mountOptions=(verbose degraded)\n  mountpoint=/mnt/pool\n  name=pool1\n  type=bcachefs\n  echo \"Creating bcachefs device: $device\" >&2\n  \n  echo BCACHEFS POSITION\n
...
...
...

Curious your thoughts. Thanks

ProjectInitiative · 2025-02-23T00:38:37Z

Linking the following for visibility:
koverstreet/bcachefs-tools#308
systemd/systemd#8234

ProjectInitiative · 2025-02-24T02:21:11Z

Some additional pitfalls discovered while testing with real hardware:

Using nixos-anywhere fails because when installing on systems with kernels < 6.13, UUID installations fail. A "nofail" mount would need to be added to all installs which seems hacky and not suitable for people running older kernels.
Encryption won't work out of the box as I have not found (yet) a good way to implement the interactive passphrase required at setup even though mounting options allow for key-files and passphrase files. Not sure why initial provisioning does not support this.

Successful deployments are possible, but require additional "helpers" external to this project. I am not sure this is a feasible implementation at this time. Might be good as an experimental feature with a write up on what is needed to supplement within a user's nixos config.

This addition attempts to integrate with the current arch of supporting partitioning. examples/bcachefs-multi-disk.nix has been added to showcase this advanced usage and some of the available options. Current limitations and disko changes to allow for compatibility: * lib/types/gpt.nix was updated to generate a deterministic UUID based on the partition name if one is not provided. Throughout testing, partition labels were not creating correctly. * This should not have an effect on other, other than changing the default behavior to favor UUID over partition labels * Testing mostly succeeds. Multi-disk root setups are not working due to some bcachefs and systemd issues that are currently open. These issues are linked under _config of lib/types/bcachefs.nix * Because of this, the fileSystems nix config is not working on reboots due to the FS UUID presenting as available even if not all disks are initialized yet. A systemd-mount alternative has been provided. * The final reboot test might need to be omitted. * Bcachefs works the best with the latest (or kernel 6.13 or new) boot.kernelPackages = config.pkgs.linuxPackages_latest; should be added to the _config section for systems, but pkgs is not provided in this module.

Lassulus · 2025-03-12T20:40:00Z

lib/types/bcachefs.nix

+            udevadm settle
+          fi
+
+          ${lib.optionalString (config.content != null) config.content._create}


not sure if this makes sense? I don't think we can have other filesystems on top of bcachefs?

I think ${lib.optionalString (config.content != null) config.content._create} is a leftover artifact from the example I modeled this from, but you are correct, we cannot have any other filesystems on top of bcachefs. I'll remove this if this ever becomes viable.

Lassulus · 2025-03-12T20:46:23Z

lib/types/bcachefs.nix

+            if echo "$output" | grep -iq "no such device"; then
+                echo "Notice: bcachefs mount failed with 'No such device'. This is expected on kernels < 6.13."
+                echo "Current kernel version: $(uname -r)"
+                echo "The mount will succeed when you boot into your final system with a newer kernel."
+            else


wouldn't it be better to fail here instead of silently skipping? since we will end up with a wrong mounted filesystems afterwards

Yes, that is the downfall of the current implementation. I wrote a little more in the comments of this PR. It is essentially dead, because of some weird upstream bugs in bcachefs and systemd that prevent multi-disk UUIDs from mounting on boot, so does and will fail in almost every test I have completed using the disko testing framework. I worked around this by ignoring this error during provisioning and adding a custom systemd mount on my servers. This issue makes this PR DOA for the time being :(

Lassulus · 2025-03-12T20:48:38Z

lib/default.nix

+    /**
+      Generate a random UUID using uuidgen
+
+      mkUUID :: AttrSet -> str
+    **/
+    mkUUID =
+      { pkgs
+      , lib ? pkgs.lib
+      }: 
+      lib.removeSuffix "\n" (builtins.readFile (pkgs.runCommand "gen-uuid" {} ''
+        ${pkgs.util-linux}/bin/uuidgen -r > $out
+      ''));
+


I guess this code is not needed?

I guess not if I would be changing the default behavior to just hash the name. Good catch

Lassulus · 2025-03-12T20:50:02Z

lib/types/gpt.nix

-                description = ''
-                  The UUID (also known as GUID) of the partition. Note that this is distinct from the UUID of the filesystem.
+                type = lib.types.nullOr (lib.types.strMatching "[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}");
+                default = let 


why do we need to change the default?

Consistency reasons. Bcachefs does not support mount by label, and managing a multi-disk string such as /dev/sda1:/dev/sdX... between steps seemed unmaintainable. Hashing the old determining factor such as the partition names into a deterministic, pre-defined UUID allows bcachefs to set the UUID before formatting. Otherwise if a user left it blank, there would be no easy way to determine which devices are maintained across reboots. Unlike zpool or mdadm which create /dev/devices predictably, bcachefs does provide that functionality

Silverdev2482 · 2025-03-21T20:57:19Z

lib/types/bcachefs.nix

+# form with disko
+# https://github.com/koverstreet/bcachefs-tools/issues/308
+# https://github.com/systemd/systemd/issues/8234#issuecomment-1868238750
+##############################################################################


NixOS has some special work around for this, though I am not sure what exactly it is. I do have a working multi-device bcachefs rootfs server that has been running for months, though currently I am struggling to get a working bootloader on an arch install for a friend.

If you are able to point me to that work around please post it here. I also worked around this by having an external systemd mount for my non-root pools with a no-fail attribute and retry counter. However due to this weird boot-dependency issue this PR is blocked as it can't pass the VM tests which test root and reboots.

I'll try to figure out what it is, probably something with the initramfs, though I don't know much about thoses, but it probably will help with the Arch Install. My config is here: https://github.com/Silverdev2482/Router-Server bcachefs systems just works out of the box, I also can get you any other info about that server, though I am currently on my phone.

Well I got it working on the arch install with systemd-boot, I still get some mounting errors on boot, but they appear to be mostly ignorable, besides the delay waiting for it to pass. I think what did it was removing fsck from some config file for the initramfs.

I'm not too good with VMs, but I could learn, what VM tests are failing? maybe if I try to get the VM's to boot I could understand why it doesn't work. Though I'm not familiar with disko, just manual formatting, but I would like to help.

One of the problems is found here
When disko does it's provisioning script, it tries to mount the FS to verify it works. However the kernel used in the testing VM is not the most up-to-date (This would also fail for nixos-anywhere provisioning if it doesn't have the most up-to-date kernel either). For the bcachefs to be mounted by UUID, I believe kernel 6.13 or newer is required. Otherwise it just fails with "no device found". The other test that fails is when the VM does reboot, it isn't mounted via the systemd mount due to the issues linked at the top of this PR. Systemd is not handling multi-disk FS mounting very well at the moment.

In theory, we could skip these tests, but I am opposed because it would require users to explicitly provide their own post boot mounting strategy, and that is not maintainable, reproducible or easily debuggable. Furthermore, it means a multi-disk setup can't be used as a boot FS (at least at this present time)

See the testing guide for more on how the disko test VM works. It is derived from a Nixos testing framework

I seem to be a bit in over my head, but I haven't given up hope, Right now I'm just trying to spin up the VM, and then I will try to set up a multi device root VM.

ProjectInitiative force-pushed the master branch 2 times, most recently from b2b7b6d to 9dabcb4 Compare February 8, 2025 16:14

ProjectInitiative force-pushed the master branch from e2a5d83 to 32ac0ae Compare February 12, 2025 18:33

This was referenced Feb 13, 2025

Fix: Device dependencies not sorting correctly #968

Merged

fix: device dependencies not sorting correctly redux #971

Closed

ProjectInitiative marked this pull request as draft February 14, 2025 14:36

ProjectInitiative force-pushed the master branch 5 times, most recently from 26c7841 to 4d78db4 Compare February 23, 2025 00:05

ProjectInitiative force-pushed the master branch from 4d78db4 to 5a9144a Compare February 24, 2025 02:38

Mic92 requested a review from Lassulus February 24, 2025 03:28

ProjectInitiative force-pushed the master branch from 5a9144a to 1a351bf Compare February 25, 2025 16:49

ProjectInitiative force-pushed the master branch from 1a351bf to 418dbf9 Compare February 25, 2025 16:56

ProjectInitiative added 2 commits March 11, 2025 02:34

make error message more broad

8afea58

fix: add hack to allow failures

185cd86

Lassulus reviewed Mar 12, 2025

View reviewed changes

Silverdev2482 reviewed Mar 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addition of granular bcachefs formatting and multi-disk support #961

Addition of granular bcachefs formatting and multi-disk support #961

ProjectInitiative commented Feb 8, 2025

ProjectInitiative commented Feb 9, 2025

Mic92 commented Feb 12, 2025

ProjectInitiative commented Feb 12, 2025

ProjectInitiative commented Feb 23, 2025 •

edited

Loading

ProjectInitiative commented Feb 24, 2025

Lassulus Mar 12, 2025

ProjectInitiative Mar 12, 2025

Lassulus Mar 12, 2025

ProjectInitiative Mar 12, 2025

Lassulus Mar 12, 2025

ProjectInitiative Mar 12, 2025

Lassulus Mar 12, 2025

ProjectInitiative Mar 12, 2025

Silverdev2482 Mar 21, 2025

ProjectInitiative Mar 21, 2025

Silverdev2482 Mar 21, 2025

Silverdev2482 Mar 24, 2025

Silverdev2482 Mar 24, 2025

ProjectInitiative Mar 25, 2025

Silverdev2482 Mar 27, 2025

Addition of granular bcachefs formatting and multi-disk support #961

Are you sure you want to change the base?

Addition of granular bcachefs formatting and multi-disk support #961

Conversation

ProjectInitiative commented Feb 8, 2025

ProjectInitiative commented Feb 9, 2025

Mic92 commented Feb 12, 2025

ProjectInitiative commented Feb 12, 2025

ProjectInitiative commented Feb 23, 2025 • edited Loading

ProjectInitiative commented Feb 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ProjectInitiative commented Feb 23, 2025 •

edited

Loading