Gitlab DELETING Production Databases | Prime Reacts

Recorded live on twitch, GET IN

GO SUB TO HIM

MY MAIN YT CHANNEL: Has well edited engineering videos

Discord

[jrhager]

I was watching this in realtime cause I had a gitlab account. They fixed it on-stream. People wanted him to be fired, and the lead helped him and REFUSED to punish him saying 'we all make mistakes' and they fixed it, implemented a post mortem, and they got it fixed. :)

[ChungusTheLarge]

Legend has it: On that day, a site-reliability engineer was born

[zacbackas]

i do devops and this video stressed me out the entire time

[liquidcode]

"software engineers hate him... find out this one simple trick a dev used to fix all bugs permanently"

[blackfrog]

the amount of anxiety i felt while watching the original video the first time was insane
i felt so bad for the guy and imagining my self in that position 🤣

[dexterantonio]

All of the in progress Toy Story 2 got deleted with a rouge rm -rf. The backups failed. The only reason that movie came out was because someone was working from home and had the stuff sync to a remote server

[THEMithrandir]

For production servers we actually alias "rm", "mv" and all installed other tools that delete/rename files so that they ask for confirmation and print the user, host and affected files.

[simonced]

Ok, I'll come clean on my rm -rf ~, but this is really weird so get ready.
I work on windows with vim, and I wanted to create a file. I did :e ~/folder/filename
The thing is for some reason, (maybe because I used the wrong / instead of \...) a folder ~ was created in my current working directory (project I was working on)
I opened a terminal (powershell) and typed "ls" to see that yes, a stupid folder "~" was in my project, then I typed "rm -rf ~"...
Then hell let lose...
I hit CTRL-C maybe harder than GitLab engeneer 1, because it was taking some time, I didn't expect that...
I realized that ~ even in windows powershell, is your current user folder. (since when!?)
Basically, I had lost all my dot files, and surprisingly, none of the other files.
My bossed helped me to restore my files from a backup of that day, and I was back up and running in 45 min.

20 years of experience in IT, windows and Linux, and that happened...

[DemiImp]

ALWAYS opt to move/rename and not remove/delete. Deleting is one of the most dangerous things you can do.

[maxdignitas]

It's not the fact that the engineer deleted the database in prod. It's the fact that the company's backup procedures cost them a full 24 hours.

[Eagledelta]

I interviewed with Gitlab back in 2018, after this event, and remember one of the interviewers telling me that they were GTFO Azure and moving the GCP because of issues they had in the past with them

[Temet]

I find the outcome is not that bad taking into account this chain of disasters!

[teodor-valentinmaxim]

I feel for the guy. I've been working in the field for a year now, and I had a mishap, a happy little accident if I would say so, where I've accidently deleted something more than I was supposed to from a table in prod db. So I've spend at least 8 hours, after my shift, learning how to use backups to restore the table etc. It was a learning experience as well, but now I'm so paranoid, that for every statement I have to do on prodbuction db, I check thrice.

[efkastner]

16:41 Etsy has/had a yearly award for the best outage. It’s the “three-armed sweater award” because the 500 page is someone knitting an extra arm in to a sweater. It was highly coveted with an actual physical trophy (I think one year it was an actual 3 armed sweater)

[lix]

Honestly huge respect for GitLab!
I think they've handled this problem pretty responsibly, which you sadly can't always expect from a big corporation.

[adambickford]

When I review my own PR I'm taking off my 'pride of authorship' goggles and putting on my 'well ackshually' code review hat everyone else gets.

And people think it's a joke when editors let you change the entire theme based on the db connection. Its only funny until its tragic.

[Tobarja]

Did Prime ever realize that "they never accidentally deleted a production database again" was sarcasm?

[jorios]

I need a compilation of this kind of anecdotes as "Production Friday Horror Stories"

[yp]

I remember, at my previous job, I felt the same way when they gave me credentials for the prod database and it had all the grant and admin access. I didn’t use it once, instead asked them to create a new readonly user and give it to developers.

[desuburinga]

I had a less intense rm -rf experience.

I was pretty green back then learning Python with Ubuntu and had multiple versions of Python installed. It got pretty annoying so one day I decided to do some clean up. Unbeknownst to me, I happened to delete the Python version that Ubuntu was using and lost the GUI interface + a bunch of functionality and only had access to CLI. Panic ensues and frantic googling began. Fun times.