That Sinking Feeling (The #HugOps Song)

Written and performed by Forrest Brazeal (x3)

Lyrics:
VERSE 1
My day was going great;
Just pushed a code update;
But then the pagers started humming:
Oops, did I just delete
Half the production fleet?
That sinking feeling's coming, from deep within my plumbing

PRECHORUS
Now my life is flashin',
Hope my boss will show compassion
And I really, really need someone to say...

CHORUS
Hey hey hey, it's gonna be okay.
You didn't just set fire to your resume.
This happens to the best,
Try not to get too stressed,
It'll be an awesome story someday.

VERSE 2

I tweaked a small config,
Turns out that it was big
And now my app has been beheaded;
When I do something wrong,
I fear I don't belong,
How can the world forget it, we're trending now on Reddit.

PRECHORUS 2
Oh, mistakes will find you
But you've got a team behind you
So fix the process, yes, but don't dismay:

CHORUS 2
Hey hey hey, it's gonna be okay.
We'll do a full postmortem some other day.
It was a swing and miss,
We're gonna learn from this
And we'll all be better engineers, like,

BREAK
Look, here's what you need to know when something gets destroyed
If there's negligence or malice then you shouldn't be employed
But if a human is assumin' then the problem is the system;
It's gut-wrenching, butt-clenching, but you work with 'em.
It can happen to a junior, happen to a senior.
Just because you caused a little pause doesn't mean you're
Incompetent, it means you're doing work with real effect.
It's a scar of battle, baby. That'll earn you some respect.

PRECHORUS 3
And we all have been there
Made a slip or two or ten there;
So we try to do a blameless RCA...

CHORUS 3
Hey hey hey, it's gonna be okay.
You will still be here when this blows away.
We've all screwed something up,
So welcome to the club.
We would love to hear your story someday.

[ChrisHeerschap]

That one thumb down is the person for whom this hit just that little bit too close to home.

[soupglasses]

THE testing in prod song, keep your Juniors safe y'all. ❤

[michaelhoffmann]

I want to laugh, but I'm under the desk in a fetal position, sobbing while trying to tell myself it will be OK....

[maynardewm]

Thinking about just how awesome software engineering culture is. Imagine this happening in any other field where you make a mistake that takes everything down. Traditional engineering, journalism, business, finance, healthcare. You would be fired for a mistake like that, possibly sued or even worse. We just go “eh, it’s not your fault, it’s a process issue. Have you really joined the team until you’ve brought down prod?”

So many thing about SWE that no other field gets to enjoy. Open source, blameless RCA, Stack Overflow, automating your job is encouraged, can be fine fully remote. We are lucky

[TylerDistad]

I found this when it was shared around the AWS ops metrics meeting today.

[frosttt]

Thanks for the video, it's super fun.

Regarding the lyrics, I did my best, please forgive any mistakes, here you go:

My day was going great just push the code update but then the pager started humming.
Oops did I just delete, half the production fleet, that sinking feeling is coming, from deep within my plumbing.
Now my life is flashing, hope my boss will show compassion and I really, really need someone to say:

Hey, hey, it's gonna be ok. You didn't just set fire to your resumé, this happens to the best, try not to get too stressed.

I tweaked this small config, turns out that it was big, and now my app has been beheaded. Wouoo...
When I do something wrong, I fear I don't belong. How can the world forget it, we're trending now on reddit.
All mistakes will find you but you have a team behind you also fix the process yes, but don't dismay.

Hey, hey, it's gonna be ok. We'll do a full postmortem some other day. It was a swing a miss but we will learn from this and we'll all be better engineers.

Like look here's what you need to know when something is destroyed. If there's negligence or malice then you shouldn't be employed but if the human is (assuming?) and the problem is the system ...

it can happen to a junior or to a senior just because you caused a little pause doesn't mean you're incompetent that mean you do and work really effective, is a scar of battle baby that'll earn you some respect.

And we all have been there made a slip or two or ten there so we try to do a blameless RCA (root cause analysis) heeeeyyyy

Hey, Hey, it's gonna be ok. You will still be here when this blows away, we all screwed something up, so welcome to the club, we will love to hear your story some day

Hey, Hey, it's gonna be ok. You will still be here when this blows away, we all screwed something up, so welcome to the club, we will love to hear your story some day

[UrsReupke]

Between this and your other tech songs, Forrest, would you kindly team up with Jonathan Coulton and bring us DevOps: The Musical?

[monalizende]

Someone shared it on my office group and now I'm listening it on loop .. creativity at its best 😂😂😂

[chrispachulski]

I've had this on repeat for 8 hours and shared it with my entire team, cheers for such a catchy and fun song :)

[dan.morenus]

Farking awesome! Linked from our internal outages newsletter. :)

[hubertusbruch]

Can't get this song out of my head for one week now. Great song! And so relatable.

[skunkworks_ca]

Well this is firmly embedded in my head for the day now.

[aaroncaito]

Forrest, you killed it, again. I look forward to seeing what you produce next, I'm sure it'll raise the bar, again.

[mateusfreira]

I cannot belive this has only 36k views, this is a masterpiece! #respect amazing work.

[TheOnceAndFutureDoug]

I need this on Spotify. I have a playlist for when I need a sonic hug and this would go to the top.

[sacha]

I'm sending this to the new hire next time they get worried about breaking something.

[WillyChuck]

Definitely resonates with Google SRE...

[PublicHealthGeek]

For the Facebook engineer who made the world a little less toxic for a day. #facebookdown #serverdown

[kylegashler]

This is awesome. Shared around New Relic today, you got lots of love!

[patrickisboard]

Github is down on a Friday, I'd like to send this to the engineer who done it. It's alright buddy, we've all done it at some point.