What do we want to make a standard?

tom commented

2025-12-19 23:45:02 +00:00

Owner

It's almost certainly wise to not have a wild sprawl of things in the core infra. However, there's trade-offs between "we always use the same distro" and "we always use the recommended one for the software we want to run".

To keep infra costs down, it feels like we don't want loads of VMs. We almost certainly either want containers or a base os that does everything.

It's almost certainly wise to not have a wild sprawl of things in the core infra. However, there's trade-offs between "we always use the same distro" and "we always use the recommended one for the software we want to run". To keep infra costs down, it feels like we don't want loads of VMs. We almost certainly either want containers or a base os that does everything.

tom changed title from ~~How much do we want to standardise things?~~ to What do we want to make a standard?

2025-12-19 23:48:17 +00:00

tom added this to the High-level approach project

2025-12-19 23:48:37 +00:00

tc424 commented

2025-12-22 22:09:44 +00:00

Owner

Despite having just said nix is probably "too mad", it might not actually be bonkers for the base OS - it happily runs docker, podman, libvirt, nspawn, incus, etcdeclaratively, and deploy-rs can do 'auto rollback on deploy failure' which is nice for remote machines. There's even also Proxmox on NixOS, though I haven't tried it.

VMs/system containers on top of that could then be whatever makes most sense for the app/service being run, or the person/team administering it?

Despite having just said nix is probably "too mad", it might not actually be bonkers for the base OS - it happily runs docker, podman, libvirt, nspawn, incus, etcdeclaratively, and deploy-rs can do 'auto rollback on deploy failure' which is nice for remote machines. There's even also [Proxmox on NixOS](https://github.com/SaumonNet/proxmox-nixos), though I haven't tried it. VMs/system containers on top of that could then be whatever makes most sense for the app/service being run, or the person/team administering it?

tc424 commented

2025-12-22 22:10:29 +00:00

Owner

Oh, IncusOS is now also a thing..

Oh, [IncusOS](https://linuxcontainers.org/incus-os/introduction/) is now also a thing..

tom commented

2025-12-23 00:17:55 +00:00

Author

Owner

TBH, I don't know enough to have a view. There's a lot to be said for nixos, and "being mad" shouldn't count against it as a standalone argument.
My argument against it as a standard pretty much boils down to this: freeIPA is a non-mad "all things identity" service that I think we should run. Ignoring any debate about if that is a good choice, were we to run it, it runs best on Red Hat. I don't want to be in a place where we make our lives hard for reasons of purity - it's got to make sense on merit.

Perhaps we should have a "system" vs. "appliance" distinction?

TBH, I don't know enough to have a view. There's a lot to be said for nixos, and "being mad" shouldn't count against it as a standalone argument. My argument against it as a standard pretty much boils down to this: freeIPA is a non-mad "all things identity" service that I think we should run. Ignoring any debate about if that is a good choice, were we to run it, it runs best on Red Hat. I don't want to be in a place where we make our lives hard for reasons of purity - it's got to make sense on merit. Perhaps we should have a "system" vs. "appliance" distinction?

tc424 commented

2025-12-27 17:08:47 +00:00

Owner

Copying some bits from IRC: https://github.com/freeipa/freeipa-container exists, and mentions podman as well as docker (running podman on a top-level host feels like a much less cursed idea that docker, due to the latter's propensity to fuck around with network setup.)

Both proxmox and incus seem to have decent terraform providers:

Copying some bits from IRC: https://github.com/freeipa/freeipa-container exists, and mentions podman as well as docker (running podman on a top-level host feels like a much less cursed idea that docker, due to the latter's propensity to fuck around with network setup.) Both proxmox and incus seem to have decent terraform providers: - https://registry.terraform.io/providers/bpg/proxmox/latest/docs - https://registry.terraform.io/providers/lxc/incus/latest/docs

tom commented

2025-12-28 10:39:51 +00:00

Author

Owner

I'm leaning a bit towards Incus - Proxmox is pretty great, but running a bunch of tiny containers is probably more like what we mostly need to do.

Rows
Columns

What do we want to make a standard? #2