Guix-p2p substitutes PoC [LLM Slopware]

Guix
1

Hi friends,

skip to next heading to get straight to technicals. this isn’t an AI post btw.

I currently have a one month subscription to one of a the BigLLMs. Seeing our community cult leader doing all this vibe coding and being a 1000x dev made me have fear of missing out and also fear of being left behind.

I’ve been guiding the LLM for a few weeks, learning the ropes and trying to get it to make stuff that I want but don’t have time to do.

One huge pain in my computing life is how awful Guix substitute servers can be. It can take a really long time to reconfigure your system or install a large piece of software when the servers are running slow or simply lacking a substitute binary. This also always happens at the most inconvenient time. I think distributed p2p substitutes network could help here.

There was an attempt a while ago that didn’t really take off, which is disappointing. I would have rather had something to base off of and then take it to the finish line with the LLM.

So…I entered “Plan” mode and began telling it everything that I want. It asked me detailed questions about the implementation and started grinding.

guix-p2p

The idea is kind of simple, use a distributed hash table, similar to Bittorrent, to store NAR hashes that are seeded by peers. NAR metadata are still looked up against the real substitute servers so that there is still level of trust1.

guix-p2p runs as a system service and installs a guix extension that shadows the built in substitute command, so all commands that call that for querying and downloading a substitute binary talk to guix-p2p instead.

Right now there is no one on the network and seeding. I’m just playing around with guix containers talking to each other and seeding packages to each other.

I could imagine this useful for large packages that take a long time to build, like kernels and firefox. A lot of the time I eagerly guix pull and find that the substitute servers haven’t built a lot of the large binaries yet, so I have to build a kernel myself, which is not fun…

If anyone is interested, maybe we can organize some time to fire up the daemon and start seeding and downloading from one another for fun. Though, I completely understand if the fact that this is LLM developed is an absolute “no” for you.

1 Eventually I want to work towards peer attestation so substitute servers are optional.

Enough rambling, here’s an obligatory screenshot of the dashboard (demo mode):

image
image3646×1883 366 KB

2

There are a lot of high-brow computer science & networking science problems here. It’s interesting, but idk much about it.

For medium-to-large deployments, would it make more sense

  1. To dedicate the guix-p2p to network-local substitute servers? then configure guix clients to prioritize that substitute server’s DNS name?
  2. Or have most servers/hosts participating in p2p activity?

The latter is a more resilient network, but there are several networking issues, which should be split out according to concerns like whether it’s a large or small deployment.

  • The network needs to find blocks to exchange. There are potential cache coherency issues. If the guix-p2p network can prioritize connections to large nodes with tons of storage, cache coherency issues are less of a problem.
  • The network also needs to anticipate demand for specific nars that it shares. I’ve heard of guix/data-service which no doubt plays a part in managing load & storage for existing substitute servers. idk much about it. But, I think the network is healthier & more efficient when answering the question “does anyone know someone who has this nar” involves less nodes. Again, that depends on whether it’s a real-world deployment or casual guix users who want more reliable subsitute servers.

There are a few other networking issues that would pop up. In any case, you’re probably more aware of all that than me. It’s interesting though.

3

These are really good points. It seems hard to simulate for me and I don’t have the resources to start spinning up a bunch of large deployment p2p nodes.

It’s probably the case that having a local substitute/build server is the most effective way to use Guix. The only issue I’m really trying to solve right now is large, bleeding edge builds that public substitute servers didn’t finish building (or are oddly slow to serve bytes).

I doubt that I do!

4

yeh servers certainly are expensive. homelab is all i can afford.

i’m currently migrating my firewall logic to VyOS from OPNSense, then converting that OPNSense so I have a pair of VyOS routers in HA. from there, i’m hoping to serve Guix to servers via TFTP and eventually a more secure PXE method. I don’t plan on using PXE much, but i want to test Guix builds on ARM devices with it. Building derivations for bootloader images & kernels would help me learn a lot

The goal has always kinda been to have a build farm. Substitutes would be part of that. IDK whether i would want to use Guix for VMs or as VM host. I probably don’t want to run everything on Guix. Kubernetes seems mostly out of reach (esp. if i want nice things like Longhorn for storage backup).

Anyways here’s some more thoughts, but it was hard to avoid tangents. I should probably read more about these questions instead of writing them up (here or in my notes elsewhere lol). A project like this should be prepared up-front for years of time spent. It’s hard to get right and sometimes it’s easier to restart fresh & reconsolidate.

Bandwidth Availability for Substitutes

Having enough p2p peers online would probably help prevent some Guix servers reaching their bandwidth saturation. Depending on the IP Routes taken to the substitute server’s point(s) of presence, nearby router hops could become congested.

IDK how the Guix substitute servers load balanced or where they’re geographically located. How the Guix substitute servers are load balanced though. I found this data flow diagram for bordeaux.guix.gnu.org that also links to the Bordeaux build farm roadmap codeberg.

In the grand scheme of things, a p2p transfer method could really speed up transfers, since the traffic can flow in small chunks across many routes. Managing the metadata that determines what /gnu/store elements to make most available is difficult when the network operators are distributed/decentralized.

It would be interesting to experiment with this. Outside of that, I would probably run a local caching substitute server. I occasionally use guix copy for small transfers.

Reproducibility

I have more confidence in Guix’s reproducibility claims than with Nix. The lack of flakes and fairly limited set of channels limits makes it easier to verify (whereas with Nix, I think it’s difficult to prove that most use-cases are actually reproduced). The kinds of “knobs” that Guix channels give you to correct for “partial irreducibility” of the full derivation are also easier to manage. it’s just easier to identify what went wrong and where

If the network works and if there’s some auto-magical way to match p2p storage/bandwidth supply onto p2p derivation demand without requiring much human intervention 80% of the time … then it would gradually add to benefits of reproducibility. Substitute servers are usually very fast for me, but this won’t scale forever.

Swarm Block Protocol

I didn’t spend a ton of time reading, but in docs/swarm-protocol.md, what does this mean?

These are mathematically different constructs. BTv2 cannot be aligned with Guix’s content-addressing scheme without creating a parallel namespace.

Why 256kB for the protocol’s byte size? I’m convinced the selection of constants for other similar protocols was some kind of a black magic. e.g. How 1500 bytes became the MTU of the internet

Substitutes for Package Variants

This GSoC project experimented with Parameterized Packages for GNU Guix. A possible later extrapolation of this would be to create a system of URIs that help identify a limited set of package variants, where the URI structure then helps to manage substitutes for variants.

I think the “Parameterized Packages” above is a different take on “Content Addressing.” It’s a different type of address/uri that could match a store item from metadata… or maybe a range of store items, though that wouldn’t be reproducible.

IDK whether these problems are really solvable in a practical sense. I think networks/servers where these variants are used will probably just manage their own substitute caching.

The whole idea of managing store items & metadata is pretty interesting to me. For nix there are the nix store subcommands, including nix store diff-closures and nix store make-content-addressed. I’m unsure of whether both types of nar signature that doc mentions have an equivalent in Guix.

strace -e file guix hash -rx . shows all files presented as inputs to the hashing function, i think.

5

I love the idea and think I could easily provide some bandwidth to now that I have a symmetrical fiber connection.

I expect, that a lot of problems are already solved but Guile and Guix lack the implementation and integration of the required protocols. After all P2P and distributed ledgers aren’t new things.

6

To me it’s pretty obvious that this is a copy of Gentoo’s use-flags. When I scrolled to the bottom of the article… “I have also used Gentoo for some time in the past, which motivated me to try and bring something similar to USE flags to Guix.”

7

Wow, this throws me back about 25 years.