Reduce the bus factor for infrastructure components

[p2004a]

Currently we have infrastructure components that are managed by a single developer, and more importantly that ownership is attached to a single developer. The ownership of such infra needs to be transferred to the BAR entity, with multiple users having admin access to the components

The first step will be to audit all of the components and identify the gaps.

[p2004a]

It should be pretty much solved by now, but let's keep it open because comprehensive audit wasn't really finished

[baccenfutter]

Have you heard of / ever used SSHCA? Every standard oppenssh-server has a builtin feature where connecting ssh-keys need to present a certificate signed by your SSH CA. All your users need to get their SSH pubkey signed by your CA. The certificate also contains the user's principals (i.e roles) with which you can narrow down who is allowed to connect to individual servers.

It goes something like this:

TrustedUserCAKeys /etc/ssh/sshca.pub
RevokedKeys /etc/ssh/revoked_keys
AuthorizedPrincipalsFile /etc/ssh/principals/%u

Match User admin
    PasswordAuthentication no
    AuthorizedKeysFile none

From now on the admin user is required to present a pubkey with a cert signed by your sshca.key in order to connect. The sig comes with an expiry date of commonly 1y after which the user's pubkey needs to be resigned.

Adding existing certs to /etc/ssh/revoked_keys immediately revokes them.

SSH root-access can be entirely prohibited. The admin user gets password-less sudo and each server uses sudo-replay to record every single sudo command and/or interactive shell.

Both, maintaining an SSHCA VM and automatically configuring all VMs/servers to enforce this SSHCA, as well as sudo-replay can easily be accomplished with Ansible (IaC).

[p2004a]

Yes, I'm quite familiar with certificate based auth/authz.

First, this issue is not really about SSH access, SSH access is actually a minor concern. This is about ownership of accounts that own the infrastructure. Think: GCP Organization Admin, Cloudflare account admin, Domain owner, Account owner with VM provider, GitHub org admin. So, the top level owners of underlying infrastructure components.

And back to the SSH CA: the number of people with access to servers is very little (2-3) and ironically not everyone that has this access is actually very proficient with using those tools at the moment, the benefit of setting up CA for this purpose is simply not worth it, and there are simpler alternatives (e.g. using GitHub user keys based automation) if we want to.