Well that's a hot mess.

If you are at this page, then something has gone horribly wrong. Primary backups have failed, The main datacenter is inaccessible and the encrypted flash drives are missing or unavailable. This is the last ditch attempt to regain access to backups and passwords before giving up and starting from scratch.

What is this site?

If you have no idea what this site is, then it isn't for you. This is a set of instructions to be used by the people who wrote them to recover from an absolute shitstorm, how to access the "break glass" servers to recover crucial information in the event that some kind of catastrophic failure has occurred.


Don't Panic

Stop panicing. Follow these instructions step by step. Do not overthink anything.

Follow these instructions. Don't make the situation worse.

Gain access to the backup server

First things first. Gain access to the backup server. This server is open to the Internet and only allows key-based authentication. Fail2ban, etc are in place. The private keys are stored on encrypted flash drives at the office, house and bank safety deposit box. If you can't access the encrypted flash drives then it's time to utilize the "break glass" NFC implant.

Using the Implant (if flash drives are unavilable)

Are you glad you got this thing embedded in your hand yet? It might have seemed dumb at the time, but if you're reading these instructions then it's probably going to pay off.

If you have a laptop, use it. If not, I guess shit got really bad and you're using the smartphone to administrate with SSH. I'm sorry, this is going to suck. To gain SSH access on a smartphone:

Mounting the backup of Vault

Congrats, you have managed to login to the backup server. Let's mount the backup copy of the Vault password database. In the event, that the backup folder is not mounted (it should be, but who the fuck knows what happened) it will need to be mounted manually. You need the password. Remember it? I didn't think so, it's saved on the encrypted flash drives or on the NFC implant. To access the NFC implant password:

Now mount the drive:

Starting Hashicorp Vault

Now that you have the backup drive mounted, you can restore the Hashicorp vault service to regain access to passwords/keys/etc.

In theory, a systemd service definition should have been pre-setup on this server, but it's me so who knows. Here is what a service definition can look like:

Description="HashiCorp Vault - A tool for managing secrets"

CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID


And the Vault configuration file:

ui = true
#mlock = true
#disable_mlock = true
storage "file" {
  path = "/opt/vault/data"

# HTTPS listener
listener "tcp" {
  address       = ""
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"

Once the config file is placed at /etc/vault.d/vault.hcl, you can proceed to set it up.

With any luck, you now have restored access to the Vault database and can start accessing credentials to login to shit and rebuild. Utilize the backup directory to restore git repositories, VMs and data as needed. Good luck.

Helpful files on the backup server