If you are at this page, then something has gone horribly wrong. Primary backups have failed, The main datacenter is inaccessible and the encrypted flash drives are missing or unavailable. This is the last ditch attempt to regain access to backups and passwords before giving up and starting from scratch.
If you have no idea what this site is, then it isn't for you. This is a set of instructions to be used by the people who wrote them to recover from an absolute shitstorm, how to access the "break glass" servers to recover crucial information in the event that some kind of catastrophic failure has occurred.
Stop panicing. Follow these instructions step by step. Do not overthink anything.
Follow these instructions. Don't make the situation worse.
First things first. Gain access to the backup server. This server is open to the Internet and only allows key-based authentication. Fail2ban, etc are in place. The private keys are stored on encrypted flash drives at the office, house and bank safety deposit box. If you can't access the encrypted flash drives then it's time to utilize the "break glass" NFC implant.
Are you glad you got this thing embedded in your hand yet? It might have seemed dumb at the time, but if you're reading these instructions then it's probably going to pay off.
If you have a laptop, use it. If not, I guess shit got really bad and you're using the smartphone to administrate with SSH. I'm sorry, this is going to suck. To gain SSH access on a smartphone:
Congrats, you have managed to login to the backup server. Let's mount the backup copy of the Vault password database. In the event, that the backup folder is not mounted (it should be, but who the fuck knows what happened) it will need to be mounted manually. You need the password. Remember it? I didn't think so, it's saved on the encrypted flash drives or on the NFC implant. To access the NFC implant password:
Now mount the drive:
Now that you have the backup drive mounted, you can restore the Hashicorp vault service to regain access to passwords/keys/etc.
In theory, a systemd service definition should have been pre-setup on this server, but it's me so who knows. Here is what a service definition can look like:
[Unit] Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/ Requires=network-online.target After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60 StartLimitBurst=3 [Service] User=vault Group=vault ProtectSystem=full ProtectHome=read-only PrivateTmp=yes PrivateDevices=yes SecureBits=keep-caps AmbientCapabilities=CAP_IPC_LOCK CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK NoNewPrivileges=yes ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID KillMode=process KillSignal=SIGINT Restart=on-failure RestartSec=5 TimeoutStopSec=30 StartLimitInterval=60 StartLimitBurst=3 LimitNOFILE=65536 LimitMEMLOCK=infinity [Install] WantedBy=multi-user.target
And the Vault configuration file:
ui = true
#mlock = true
#disable_mlock = true
storage "file" {
path = "/opt/vault/data"
}
# HTTPS listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/vault/tls/tls.crt"
tls_key_file = "/opt/vault/tls/tls.key"
}
Once the config file is placed at /etc/vault.d/vault.hcl, you can proceed to set it up.
With any luck, you now have restored access to the Vault database and can start accessing credentials to login to shit and rebuild. Utilize the backup directory to restore git repositories, VMs and data as needed. Good luck.