An Update Borked My Linux System. How Did I Fix It?

This morning, I decided to update my Linux workstation. In the middle of the updates, something went horribly wrong. Either I hit something I shouldn't have on the keyboard, or my window manager crashed on its own. Either way, the window manager killed itself in the middle of an update which included a kernel update...

WANT TO SUPPORT THE CHANNEL?

DONATE CRYPTO:
💰 Bitcoin: 1Mp6ebz5bNcjNFW7XWHVht36SkiLoxPKoX
🐶 Dogecoin: D5fpRD1JRoBFPDXSBocRTp8W9uKzfwLFAu
📕 LBC: bMfA2c3zmcLxPCpyPcrykLvMhZ7A5mQuhJ

DT ON THE WEB:

FREE AND OPEN SOURCE SOFTWARE THAT I USE:

Your support is very much appreciated. Thanks, guys!

[peterbrown]

It happens so infrequently (thankfully) that you forget how to fix it.

[GadgeteerZA]

Yes first rule of a totally borked bootup is "Don't Panic!". This morning I had both my primary data and backup data drives for my home server both lose their partitions - the TestDisk utility did an amazingly easy job of recreating the partitions (after a cup of coffee).

[ozmosyd]

For those new to Linux.
Another, knowledge share gem.
Thank you chap!

[imtiredtoday]

This is a great way to rescue a Linux install.
Also I heard that you have multiple USB drives with ISO's on them. Something you can try to consolidate them is Ventoy, it's a bootloader for running multiple ISO's from a USB disk.
For example I have a Ubuntu (server and desktop), fedora, rocky, Debian, arch, Garuda, endeavour OS and windows on the same USB drive

[NofarahTech]

hey Derek, something you forgot to mention is that this solution works only for legacy bios setup, if you have uefi you must also mount the EFI partition. And not to mention, inside the usb drive you don't need sudo privileges

[TheXipherZero]

This is why I always recommend keeping a current latest kernel and an LTS kernel on Arch. If something happens with a latest kernel update, either something like what happened here or compatibility issues with some changes on the new kernel, you can simply boot the LTS kernel and reinstall or roll back the latest without having to boot a live environment.

[NebulaHatesWatchdog]

Gentoo users can do this without an update 😎

[ahmetsert]

You can upgrade your system in virtual console in order to avoid system failures because there is a slew of reasons for the graphical environment to crash especially during a major update/upgrade. Hope it helps.

[srivathsansudarsanan]

Thank God you're talking about this. I had this problem where Pacman crashed in the middle of installation and it deleted my vmlinuz image. Fortunately I knew how to counter this and had a spare bootable usb with me.

[OctaviusPelagius]

Since I installed openSUSE Tumbleween in one of my machines I've tasted the divinity of the BTRFS file system paired with Snapper in order to have snapshots created for me and available in a GRUB menu so I don't have to rely on USB pendrives that can be misplaced or easily corrupted. I think that heavenly match can be installed in any distro. Arch would benefit tremendously from this.

[joejohnston]

Great instructional video and very important information. Thank you, DT!

[davititchanturia]

interesting tip:
- create new partition
- just copy live cd iso file content inside this partition
- if something brokes choose to boot from this partition.
wuala you always have live cd with you

[W__W]

If you cannot load after installing new kernel, just reboot and press Esc many times, while PC is loading, and then, in the grub menu, you can select previous kernel.

[andrewwigglesworth]

Debian GNU/Linux ... you'd just reboot into the previous Kernel. I'm not sure, though, how you'd get this particular error on a Debian system anyway.

[DylanMatthewTurner]

This is one thing that's great about Linux. You can fix problems.
Can't tell you how many times I had Windows break back when I still used it and have the only solution be to reinstall from scratch (reset wouldn't even work!).
On Linux, usually just gotta chroot or look for a solution online. Someone has probably solved your problem before.
No need to reinstall from scratch like on Windumb

[martin.]

I’m using a custom GRML for this - with ZFS support and the keyboard layout set to Dvorak.

You can work around a lot of these boot issues from the boot loader command prompt and at least get your system to boot, but it is a lot more tedious than the process you described. And the boot loader command prompt also typically only supports QWERTY.

[kdemetter]

1:08 Oof, that brings back some deeply repressed trauma for me

[xraptorx]

I had a similar issue once. But i solved it a bit different. I started a liveUSB, chrooted and just installed the previous Kernel before the update, which was archived on my machine. Pacman -S was not an option since the new kernel was not compatible with my network drivers of my Wireless devices, which i rely on since the router is on the other side of the building.

I had to do it once with my arch system and later with my Debian. Though the Debian case was a bit more difficult to fix. I borked my machine by accidentally overwriting my GLIBC with a newly, freshly compiled GLIBC that i needed to recompile software i developed for my arch system. (I forgot that the folder in which i compiled the GLIBC to was already used by my system native GLIBC)
At that point i didn't reinstall the old GLIBC but just used a snapshot iso i had from that system since that can be less trouble as fiddling with that library.

But for most Kernel issues, if a Arch update trashes your system with a new kernel and you have no internet connection, a previously installed Kernel package should do the trick.

[octagear]

Just another little niche tip i had to figure out the hard way: When you need to fix the Linux install on an old or exotic peace of hardware, some BIOSs or UEFIs only have drivers for USB2.0 - meaning it won't let you boot off of a newer drive. I have a micro-sd to usb adapter which uses usb2.0 which helped me out ( just a little fyi sidenote here ^^ )

[bwillan]

Thanks for the great video. I am in a similar problem with manjaro. I had the kernel not install properly on an update a few months ago. I was able to rescue it with a manjaro usb stick and using my timeshift backup, but I've been afraid to update the system since. I am rather behind in the updates to my system.