I actually find that nixpkgs being a monorepo makes it even better. The code is surprisingly easy to navigate and learn if you've worked in large codebases before. The scaling issues are good problems to have, and git has gotten significantly better at handling large repos than it was a decade ago, when Facebook opted for Mercurial because git couldn't scale to their needs. If anything, it's GitHub issues and PRs that are probably showing its cracks.
It did slowly sneak in over time I guess. In my last year of my master's eventually the faculty was forced to stop hosting its own intranet and mailing lists and migrate everything to the "cloud" (Microsoft 365 and Blackboard).
I have a copy at home of all the old wiki content and the old cs.uu.nl website. The university themselves didn't even think they should archive it so I archived it myself.
I hope there's other people with copies too. My archive isn't complete
To be clear, there is no bug here: derivers are simply not uniquely determined in the presence of fixed-output derivations, which is by design. That's even more true with CA derivations.
CA derivations also introduce the opposite situation, namely that the same derivation can produce different output paths for different users (if the build is not bitwise reproducible).
It's both, multiple derivations can produce the same (content-addressed) store object, and the derivations may not be reproducible and produce different (content-addressed) store objects each time.
The reality of executing arbitrary programs on non-deterministic computers is, unfortunately, N:M!
(Cue deterministic WASM derivations or something.)
The deriver field in Nix has always been a misfeature. It was intended to provide traceability back to the Nix expression used to create the derivation, but it doesn't actually do that (since that wasn't really possible in the pre-flakes world, without hermetic evaluation). So instead it just causes a lot of confusion when the deriver recorded in the binary cache doesn't match the local evaluation result, due to fixed-output derivations changing.
In the future, Nix will hopefully gain proper provenance tracking that will tell you exactly where a store path came from: https://github.com/NixOS/nix/pull/11749
I think Eelco has in mind a separate thing that would still be a store object field. But IMO we should not do that since derives are unique, and we should instead use the "build trace" instead, which properly handles that.
As Martin Schwaighofer has discussed, it is fine and in fact good for build traces entries to have arbitrary meta data, so the "claims" being cryptographically signed are more precise. (This is good for auditing, and if something looks suspicious, having full accountability.)
So on that grounds, if eelco would like to include some "this came from this flake" information as informal metadata. (formally the key must still the resolved derivation.) That is fine with me.
---
As I linked in my other reply, see my fast-growing https://github.com/NixOS/nix/pull/14408 docs PR where I try to formally nail all this stuff down for the first time.
I mentioned another alternative to adding flake-specific metadata to data structures that are transferred over the network, as part of the signed traces or otherwise, in a comment on that PR Eelco linked.
It's keeping flake-specific data locally, to guarantee that it matches how the user ended up with the data, not how the builder produced it. I think otherwise from the user POV such data could again look misleading.
Good point. It is misleading if different flakes end up producing the same derivation, and we don't want to resign our build trace entry to account for that (which would amplify reads into writes). Separate indirection for this eval->store layer accounting sounds good.
It refers to the new CLI, i.e. the "nix" command. The old CLI is the nix-* commands (nix-env, nix-store, etc.). The new CLI and flakes are still marked experimental, but the plan is to stabilize them and call that "3.0". So it won't be very different from the current 2.12 - you just won't need to enable some experimental features anymore to get the new CLI/flakes.
Because of politics (see RFC 49, 134, 136), I don't see this actually happening that soon.
There is agreement in principle to carve out a store only Nix so the no-flakes crowd can build their own UI on it which is held by them as a pre-requisite to any stabilisation of flakes, along with some bad blood of it being announced by blog post. But it's slow because the project maintainer and chief flakes advocate doesn't want to adjust the project just so an alternative frontend specifically to avoid their view of an easier Nix should go ahead.
People are still working somewhat constructively towards this in 134, but because their goals feel fundamentally opposed, it seems likely to go on a while.
The Nix team, including the Nix author and chief developer of flakes (upstream in this thread, so be kind all), is also working right now on enabling the nix3 UI for non-flakes setups as well: https://discourse.nixos.org/t/2023-01-16-nix-team-meeting-mi...
Consensus is hard but I'm glad to see people coming together on this. Really looking forward to seeing the new Nix command and flakes move forward (independently)
Funny i don't want a UI, i just want better type systems so some random field is easily known. I know it's difficult, but strictly from a user experience - that's all i want lol.
I'm really optimistic about Nickel, which is a new DSL based on Nix (in terms of inspiration) and designed for Nixpkgs-like use, but informed by the designs of other DSLs in the same space (e.g., Starlark, CUE, JSonnet).
One of the things it adds over Nix's design is types (it grew out of early efforts to add gradual typing to Nix). It has a gradual typing system with 'contracts' for dealing with untyped code.
I have spent some time doing exercises and examples in it, first in Nix code and then translating it to Nickel. It feels very Nix-y, in the best way, and I think in the end it will be very nice to use.
Definitely check it out if you have thoughts about Nixlang and how it could be better! Development is very active and the maintainers seem quite appreciative of testing and feedback.
I've been patiently waiting the nickel<->nixpkgs integration. Several hurdles they have to solve first though, iirc, but yea Nickel looks great.
Really that's almost all of my issues with Nix[(pkgs|os)]. Nickel hopefully solves that quite well.. we just need it to work with Nix first, which is a big hurdle on its own.
It's because Nix wants to install into /nix. Once upon a time doing "sudo mkdir /nix" wasn't a problem, but recent macOS releases have made that very hard.
Nix could switch to an alternate location on macOS (e.g. /opt/nix) but that has a lot of downsides for interoperability with other systems.
On Linux, users can put their Nix store in their home directory or other places and at run-time Nix remaps the directory using user namespaces. Unfortunately this isn't workable on macOS: the kernel doesn't support the features we need.
Using /nix and a separate group and daemon means the store can be read-only and be protected from modification in several ways. This is pretty helpful, as a lot of tools try very hard to write "next to" where they are installed -- corrupting the Nix store.
I sort of wonder if it would be more palatable if the Nix installer was a bit less in your face about what's going on? This would be similar to how Docker's works.
It would be less palatable when I found out. The group is fine. Why the daemon when other package managers use sudo is unclear. Even Homebrew moved to /opt.
Other package managers are okay with requiring sudo because they install stuff globally. Nix doesn't have that restriction, you can use it for local stuff, temporary shell environments, etc. So you need non-admin users to be able to use it too, and even admin users need to be able to use it without using sudo. For example, when entering a nix shell, you don't want the shell to run as root. Or when using direnv. Or just when using it as part of your build system.
Nix is a package manager, yes, but it's more than that, it's a generalized build system.
Nix isn't the only package manager for home directories or source packages. Working in a home directory doesn't require sudo or a daemon. sudo doesn't mean run everything as root.
> So you need non-admin users to be able to use it too
The build daemon and the user are used for privilege separation. The separation goes both ways. Users can't write directly to /nix/store and Nix can't write outside of /nix/store during build.
If anything, it's there to make things less invasive. It's nothing like the Docker daemon, which is a proxy for root.
Additionally, the daemon doesn't do anything unless users request that a package be built.
Other package managers have privilege separation without daemons. I don't need non admin users to run it. And I could configure sudo to let them if I did.
This is inherent to having binary packages. Binaries compiled to look for libraries in one location cannot just be copied to a system with a different layout. Nix also can't just use the OS conventions, since part of the point of Nix is that it does not use the global system state and only its own isolated, reproducible world. So then they had to make a decision and decided to just use /nix everywhere which worked fine for some years. And now MacOS has changed, choosing something else would invalidate all historical binary packages.
It's not hardcoded, but you end can't take advantage of the binary caches if you change the directory. A company could certainly create their own binary cache and distribute that to it's users.
Not entirely true, there are many ways in which you can use a custom location and still take advantage of the binary cache. You can do it with chroot, file system namespaces, bind mounts and so on. There's also a nice user friendly tool that does exactly this [1].
It’s not actually hardcoded, but you would have to compile everything from scratch as the hash would depend on that as well, throwing out the whole binary cache.
I'm sorry, but how exactly is this "amazing"? That you can find some bits to append to a number such that it becomes prime is rather obvious, given that there are an infinite number of primes and (probabilistic) primality tests are readily available. As other have pointed out, this is no more interesting than the fact that adding a "1" bit yields an odd number.
And what's the point? Yes, illegal information can be encoded as bits, those bits interpreted as numbers, and then you can apply transformations to those numbers. So what?
The point is that certain large prime numbers (of certain forms) are curated and published in a catalogue because they are notable (in and of themselves). The process you dismiss as trivial allowed Carmody to encode the illegal program as such a prime, and hence have it independently published in said catalogue, where it belongs entirely independently of whether it happens to turn into the illegal program when run through gunzip.
> That you can find some bits to append to a number such that it becomes prime is rather obvious, given that there are an infinite number of primes
That isn't obvious to me because I learned a long time ago about the strange nature of infinity. The infinite space of primes has infinite gaping holes that may very well swallow all numbers of a given form.
But I posted the question to the Math Stack Exchange, and it seems to be correct:
In NixOS (the Linux distribution based on Nix [1]) we use the functional approach for managing all the "static" parts of the system, i.e. packages, but also most configuration files in /etc - these get build by Nix functions in much the same way as packages. This is good since it means that (say) a rollback of the system will cause it to revert to a consistent combination of packages and configuration files. (So you don't get an Apache httpd.conf that refers to some feature that doesn't exist in the Apache you just rolled back to.)
However, when it comes to stateful stuff such as user data or databases, the funtional approach indeed doesn't really apply very well. NixOS manages state in an imperative manner in the same way as every other distribution. (For instance, NixOS uses Upstart jobs to create/update state.) That means that a rollback of your system won't rollback (say) your PostgreSQL database, unless somebody made the Upstart job of PostgreSQL do that. So when it comes to state, NixOS isn't better than other distributions, but it's not worse either ;-)
Thanks for the response. I agree that some functionality currently handled by package managers is hairy, even for other package managers. To some extent, I don't think this functionality should be managed by package managers. It would hurt usability because the user would need to set up configuration files, hook the package into the init/upstart system, and fix breakages across updates. On the other hand, I've never been a fan of running 'sudo apt-get install apache2' on a development machine and exposing a public port and running an application I don't always need every time I boot up.
The problem with the nix package manager that I see is it doesn't integrate into a hierarchy of any kind thus no partitioning of data. Also the nix language is very obtuse and domain specific(which is a good thing as well as bad). I plan on creating a meta package manager that is functional but allows output of arbitrary formats such as rpm, deb, etc. I also think integration is a separate but just as important aspect and hope to use mruby, rspec and augeas(or something else as it seems the parsing format is yet another NIH wheel).
Feel free to try it out to see that it does, in fact, do this. No special magic is needed (most of the time): for instance, for dynamic library dependencies you can just set the RPATH at link time to indicate precisely in which directory an application should look for its dependencies.
No, it won't work. This is my area of expertise and I can unequivocally state that it is not possible to solve this problem on a unix system. The linker infrastructure does not exist, and here's why:
RPATH (or better RUNPATH) has intractable conflicts in library dependencies. For example, this cannot work for any application which uses dlopen() and packages its loadable modules independently of the invoking binary. This is a problem for very high profile software such as:
* Apache, and just about any other application server with a plugin interface
* Perl, Python, Ruby, Java, and just about every other interpreted/VM oriented language
Furthermore, there exists other intractable conflicts in complex library dependency interactions. For example:
bin/B needs SONAME=D.so.1 which needs both SONAME=C.so.1 (v2, incompatible) and SONAME=B.so.1
The dependency relationship of every shared object in this chain are not static, and you cannot modify the invoking binaries (where RPATH must be set) every time an intermediate shared object changes its linking requirements. You also cannot guarantee ordering when a single node depends on multiple libraries -- the order of evaluation is non-deterministic. You might say this is a corner case, but you would be surprised when you try at scale. This is not a corner case.
Seriously, I understand everything about this. Can't be done. You feel free to try it out and see that it doesn't.
Sure, eventually you run into scaling issues, but that's a first world problem.