InfluxDB teaches customers an important lesson

👉 Equipment used:

👉 Affiliate:

👉 Leave a tip for us via cryptocurrency if we've helped you out:
› Bitcoin: 1EaEv8DBeFfg6fE6BimEmvEFbYLkhpcvhj
› Bitcoin Cash: qzwtptwa8h0wjjawr5fsm0ku8kf40amgqgm6lx4jxh
› Dash: XwQpZuvMvU44JT7C7Uh6xHvkSadzJw9fMN
› Dogecoin: DKetsoCvwa2hF29ssgUA4Wz4hxT4kj3KLU
› Ethereum: 0x6f6870feb48f08388ee345cf0261e2f03d2fa310
› Ethereum classic: 0x671bfd61ba87edf6365c97cea33d66ba73645510
› Litecoin: LWnbTTAjojZQt68ihFJFgQq3cYHUsTcyd7
› Verge: DFumZ5sMhi3JktLQpsTVtV9xUt3zKDrcZV
› Zcash: t1Ko3FkphQYoQroQc8k2DVk4WKMAbmNR8PH
› Zcoin: a8QdvArHmdRYe1MjiqtP6jDNe6Z4JgnRKZ

[UNgineering]

Here's how this works in many large companies: a dev tells the manager "we need to get InfluxDB" the manager requests a subscription from the accounting department, then passes the login info to the dev. Since the accounting dept signed up, they will receive all the newsletters from influxdb, all of which will be ignored. The dev will never know about this discontinuation.

[countfrackula]

As a sysadmin, what they did violates all best-practice polices for shutting down server(s). When shutting any server down, you should power it off and leave it off for at least a week at absolute bare minimum (preferably at least a month) to see if someone complains. If no one complains, then the server is likely no longer needed. Abruptly shutting down a server without retaining any backups even with warning is an absolute no-no. In my opinion, everyone involved should sue them for breach of contract.

[todaresq]

My university did something similar. They sent out an e-mail to alumni about a change in needing 2FA to log in to email provided by Google. The 2FA was in the subject line, and the main body of the email. A small part at the bottom of the email, after the 2FA instructions, and some random BS, was that they would also be ending Google Drive and Docs access for alumni, and we should migrate our data off. That was missed by almost all alumni. Grad and doctoral students lost years of research and papers… because they did not see what amounted to a PS in a long email about 2FA.

[TomBortels]

The "scream test" someone mentioned is very important. You never delete immediately. Take the service down, and give people a chance to freak out about it, *in addition* to ample warning of the event. There are people who just aren't paying attention, but WILL respond when they have an outage. Once you have a week or two or a month of service down without anyone complaining - you're clear to finish the decommission.

[jemsterr]

But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes, ” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.

[curtmcd]

I worked for a real storage company. About 10 years ago, a mistake in which drive to reformat resulted in a customer losing data. The loss was considered an existential catastrophe for the company and the terrible event still reverberates today. A bidirectional multiple-confirmation requirement was immediately added to the system to prevent it from happening again.

[zozodjr]

The stupidest thing is that these platforms will send 100 emails for advertising but 1 for actually something a user would care about. Even if you are keeping an eye out, you have to dig through spam for a warning

[rabidpb]

Should have noticed the warning signs when Influx suddenly went closed-source without notice a couple of years ago. Our company pivoted away sharpish when that happened.

[XeonProductions]

As someone who works in tech, this is why you do a scream test before completely decommissioning something.

[watsonwrote]

I cannot believe they wiped their customers' data without calling them! What a horrible company!

[robr]

I think people in charge tend to prioritize short-term benefits of outsourcing a lot of IT to "the cloud" b/c it looks great to the higher-ups on the balance sheet. When everything goes wrong they blame the lower-level people regardless if they wanted to outsource or not, while the people making the decisions usually face zero consequences. Our system incentivizes this kind of behavior.

[bretthutchinson]

Having been part of many decommissioning exercises in the past, the entire engineering team not understanding the risk of what they were doing is almost unheard of. Email and website notifications are not nearly enough. You have to actually shut down the service to see who wasn't paying attention, and then make it temporarily available again to give customers time to migrate (you will almost ALWAYS have to do this for a service of any reasonable size). Even after that, you're taking a huge risk by not keeping offline/cold backups almost indefinitely, or at least several years until the physical media's natural end of life (for absolute worse-case disaster recovery). This is not just for the customer, this is in the interest of the company both in terms of legal liability and brand reputation.

[togethermutiny]

Just saw the mugs linked above the comments! Smart move. I love your clever, hilarious slogans! Bravo again, Louis!

[xjasonx]

I'm at a loss... how could a business that does data handling ever let this happen? Let alone ever get off the ground and become solvent?

[PyroNine]

I'm with you. Even when it's for non payment or when the customer actually cancels their account, I put their data offline but retrievable for a while. Sometimes it just sits there for a few months and I delete it at my leisure, in other cases we get frantic calls from people who didn't understand that the data would be gone. When that happens you can feel the waves of relief over the phone when I tell them I can put the data back up for them.

[ahaveland]

JFC. That is a totally appalling disregard of data, which is sacred and often irreplaceable.
Influx, should now experience a massive outflux of customers, and their directors prevented from running so much as a lemonade stand for egregious incompetence.
Lesson also - NEVER keep all your data in one place, replicate it and back it up.

[Some_Some_Some_Some]

This kind of deprecation should be done in multiple slices.
They should AT LEAST contact those who are still using the database close to the deadline, and they should set the databases read-only a few weeks prior.

Even with not so sensitive services, the way we usually handle this where I work is:

1 - Send out an email warning with a deadline
2 - Send a reminder close to the deadline to those who haven't migrated yet
3 - On that day, review users still using it and contact them directly
4 - Give a new deadline to those users and prevent writing any new data - an exception can be made if some customers need more time to migrate for a specific amount of time
5 - A few days before that deadline: final review of the traffic - send a last warning
6 - Stop the service for a while and keep the data, allowing you to turn it back online easily if anything goes wrong.
7 - Cleanup (delete everything)

[VeljkoMirkovic]

I worked at a hosting company and we were going to be migrating to a newer generation of servers where we couldn't reassign the same IP. It took us around 7 months to migrate because we took our job seriously and for the 5 months we tried to reach out to people with every single communications channel, sometimes including social media of employees we got on LinkedIn or through phones on their company website.

We even put up a backup server and left the deprecated server for a bit longer to get in touch with everyone, which was a few 100 clients.

[LonSeidman]

You are spot on! It's not hard to pick up the phone and call the customer!

[wettuga]

I hope they lose all their clients after this "maneuver", because they definitely made many clients lose money. That's also a lesson to the more careless costumers: always backup locally and remotely!