In one of my favourite movies Gerard Butler utters the line “lessons not learned in blood are soon forgotten”. That is certainly how I felt last week while attempting to regain access to my cloud accounts.
Ok, Ok – Maybe a movie about a jail-bound revenge obsessed serial killer isn’t the best analogy for a nerd playing god in a cloud lab.
Whenever playing with a cloud service that manages provisioning & activations for your cloud users, it’s important to be alert and understand what actions you are taking. How was I reminded of this? I broke my own setup!
While deleting a bunch of users in a fit of rage at an inconvenient Lab setup last week, I was neither alert, nor understanding. Have a look at the below screenshots and have a guess at what happened.
Yep – Thats right. james@xellolabs.com. Deprovisioned. Deleted. Gone.
The importance of this account? It was my Google Cloud Super admin.
Here is the part where you say – It’s alright James, use your emergency account. Your rainy day account. The forgot my password account. The one you keep spare for when you get hit by a bus.
Bummer for me – I didn’t configure one.
Ok, fair enough, There’s an account recovery process right?
The problem? Account recovery doesn’t work for a disabled account.
Next. Maybe I can re-enable the account using Okta? Remember that fit of rage I was talking about? I also deleted the Okta Application.
Yep. I’m a bonehead.
Ok, next option? Google support. Surely they should be able to help me!
Case: #21249639
Subject: Okta Deleted/Disabled my organisation Super Administrator
Evening Google,
I was testing some Okta configuration for google cloud configuration late at night. Unfortunately I deleted an okta account that matched the GCP account, and it disabled/deleted the corresponding account: james@xellolabs.com which is the super administrator for my GCP test organisation. I am no longer able to log into the google cloud console or admin.google.com to configure IAM or any extra users.
I can provide proof of ownership for the DNS zone xellolabs.com
I can also provide information as to how the organisation is configured, likely how it is billed (Can't remember the exact credit card)
Apologies for this boneheaded error, I will setup an emergency account in future.
Cheers,
JamesYou can see where this is going right?
Google Response 1 (Paraphrased for length)
Dear James,
That certainly is boneheaded of you. Please refer to our support article here for admin password reset. https://support.google.com/a/answer/33561?hl=en
Thanks
Google Support. No thank-you google – I need some actual help here.
Morning Google,
Unfortunately this does not solve my problem. As I mentioned in the original ticket, Okta has disabled the Super Administrator for my google organisation. I cannot log in to admin.google.com to create new accounts or modify users. I can also not recover the account as it is disabled.
When I log into the google console, I receive the following username:
fe0b6545633f422e8501f62c8d3ca20aJames@xellolabs.com
I believe it should be just james@xellolabs.com
If you could assist with activating the account I can log in, as I definitely have working credentials
Cheers,
JamesGoogle Response 2 (Paraphrased for length)
Dear James,
I understand you have a problem. It looks like you are having an account ownership dispute. Unfortunately google cannot adjudicate these disputes, please reach out to the super admin for access. Should you like, I can contact the super admin and ask them to contact you.
If none of this works, we can completely delete your organisation and you build your lab again.
Thanks,
Google Support.FFS Guys! It’s me! I swear!
At this point I requested that google contact my super admin. After all, this lab had a population of 3. Me, Myself and I. Surely they would flick a message to one of my many emails?
Outlook? Gmail? Work? Hotmail? Custom Email?
Zilch. Nada. Nothing. All in all a frustrating experience with Google support. Not something I would recommend on the best of days.
Finally I realised that I had configured a service account for Azure-> GCP integration. I had credentials for this thankfully. I could check who was in my google org & who I was supposed to be contacting!
One quick login to admin.google.com:
Even better news? This was the account I needed! In a past life I had been a lazy bastard and not configured the minimum RBAC!
Account Recovered!
Ok so what did we learn today?
- Configure a break glass account. Store it offline & test it regularly.
- Don’t rely on support to recover your account. Contextual information like a service account names will be overlooked & you may get the runaround.
- Don’t add your break glass accounts to automated deprovisioning flows. You are asking for trouble.
- Document the level of access you grant to service accounts. You never know when this information might come in handy!
- Have a tested BCDR strategy, using off cloud backup or IaC to restore.
Most importantly, I learnt my lesson in blood. Don’t be a bonehead and hopefully you won’t run into this problem yourself!
James Westall 