I'm Going Back to Fedora - VLOG

Today I talk about my problems with troubleshooting Arch and why I'm going back to Fedora.
👇 PULL IT DOWN FOR THE GOOD STUFF 👇

===== Follow us 🐧🐧 ======

==== Special Thanks to Our Patrons! ====

#ramble #vlog #thelinuxcast

[TheLinuxCast]

Giving a like to the video really helps the channel. Grattitude to those of you who took the time. 🙏

[HeathenHacks]

You don't always have to fix things on your own.

Bug/Issue submissions exist to let the devs know about your issue.

I mean, if you'd just hop to the next distro without atleast submitting a bug report, they (the devs) wouldn't know about that problem.

In my case, tho, if an application is misbehaving, I first check what packages have been recently updated and then check if the application that's misbehaving depends on any of them.

If a dependency of the application has been updated recently, I downgrade it, then if the misbehaving application starts to work again (or even if it didn't do anything), I submit a bug report.

That way, the devs would know about the issue and perhaps people who are having the same issue would see it as well.

[yramagicman]

Ah... Debugging. That's my job and I have a love hate relationship with it. Here's how I would approach this issue:

1. Logs. The logs are often helpful, but sometimes obtuse and opaque. When the logs aren't immediately helpful it's time to move on. You did mention that there are some possible errors that could be causing it, and the core dump is a good indication that you found the problem.
2. Observation. What is going on when your system crashes? Can you reproduce the crash in an observable way? I half wonder, because it only happens when you walk away, if it's some interaction between qtile and your screensaver/screen locker. xscreensaver has a test mechanism that allows you to preview a selected screensaver, this could be useful for debugging.
3. If you can't reproduce the crash, start eliminating things one at a time. In your case, I would remove things from my bar until I was left with just the time and workspace listing. I believe you mentioned `pcmi` or some similar thing. I'd start by removing everything that relies on that functionality. Keep eliminating things until the crash stops.
4. If you eliminate everything on one window manager and still have issues, switch to another one and see if the crash persists. I'm guessing i3 wouldn't crash. The other solid option here would be to use a Wayland compositor like Sway to get Xorg out of the picture.
5. Assuming you don't have to switch window managers as part of your process of elimination, the next step is to add things back one at a time and try to cause the crash. Once you're crashing again, remove the last item you added. Leave that item off and continue adding things back to your setup. If you continue having a stable system, you found the root cause. Otherwise, repeat the add/remove process until you have everything added that doesn't cause a crash.

This process takes time, especially with these pesky bugs that don't happen when you're sitting there. I would plan on taking a couple weeks to debug this. The other complication on arch is updates. It's entirely possible something happened with qtile to cause things to go sideways. If you're trying to debug this for real, I would ignore updates until it was solved, then update and see if qtile has pushed a new version in the intervening time. If they have, it's likely the update would contain a fix. If it doesn't, it's time to file a bug report.

[the_linux_legend]

This is a problem with file descriptors. Basically it seems like qtile is opening up way too many files at the same time and therefore cannot access anymore file descriptors. If your system is out of file descriptors than xorg doesnt have any either therefore core dump and crash. I know this because of the OSError too many open files /proc/meminfo, this is always a sign of reaching file descriptor limit on a process. The limit is set per process, per user and is low by default.

[lunsj]

Don't feel bad about hopping. We all do it. I hopped to Fedora 38 when it came out and I'm really happy with it. Rock solid. Fedora's plasma is very good.

[swartze]

For what it's worth, at 4:28 in your video, there is a "Too many open files" error 3 lines up frrom your cursor. This will prevent new files from opening which can cause the other errors you're seeing. I can't be sure from a few random screen shots, but any OSError is worth looking into and open file issues can often be corrected by changing ulimit with out causing other issues.

[Joel]

What I've done in the past is to have two different distos installed on separate disks. Sometimes you can fix things but sometimes you don't have the time to investigate and need to wait for devs to implement a fix. If you have problems with one distro you can just temporarily use the other.

And sometimes using multiple distros or distro hopping can reveal useful information about what your issue is. For example, on Fedora I was experiencing random reboots, so I switch to pop os. I experienced the same thing on pop os which led me to realize it was likely a hardware issue.

[GCoda]

after 7 years of arch (same install) i switched to fedora year ago and pretty happy for now

[Winnetou]

I commend you Matt for trying to solve it. Indeed, there's no guarantee you'll succeed, but in most cases you can, or at least learn a bit about the issue or something complementary. So this not working this time, I hope, won't kill the future attempts to check what the problem is. Also, while on this, only go as far as you're confortable. It might be 5 minutes one day or 10 hours on another. Don't push yourself too hard, because, like I said, there's no guarantee that your frustration will have a positive impact. You might still spend 10 hours and find and solve nothing.

More on this particular video, regarding logs, always start with the oldest, from a bit before the problem happened. Because it can often be a single simple thing that broke or ... happened, and then all hell to break loose, but the "hell break loose" part to be an effect!! not the cause. So always go as far back as you can and search from there for weird things. This way you might not even reach things like that python stack trace. Investigating on the "effect" rarely (though at times it can) gives any meaningful info about the root cause. Like in your case, it seems that a lot of programs started crashing or having problems: xorg, pulseaudio, network manager and so on. I don't think it's a problem with them, but they have the same cause (or they are chained and one having a problem causes a problem in the next one and so on). Someone mentioned that it problably was "too many opened files" which was seen in the qtile logs at 4:20 (nice). Frankly from how it manifested, it might be that indeed. Not being able to create a file means that a lot of programs cannot function properly, or at all, anymore.

[schemage]

I think that it is a fallacy to believe that the only solution to a problem is spending forever fixing that problem. It makes no sense to rebuild a car from scratch (in any situation) but if its a choice between installing my distro or spending days fixing up all those dot files that I messed up, the former is prefered. Call it laziness.

[MarkusHobelsberger]

I think you were on the right track there with the qtile bar. I'd have gone for disabling the widgets on the bar one by one and see if the the problem goes away. The thing with these intermittent problems in software development (which is a part of my daily job) is that you very often have to spend time to find a problem over a long period of time. If you are not willing to commit to that and rather reinstall, that's a fair decision, too - especially on a machine you need to use productively.

One thing I miss on Linux is a proper classification of the log entries, so a user can immediately see which are just messages and which are the actual errors. Journalctl tries to do that by highlighting some errors in red or yellow, but often enough the logs are cluttered with harmless messages anyway, so you can't got by these in practice.

I actually got into the habit to have one terminal always open with journalctl -f or tail -f /var/log/syslog so I can look at it multiple times a day and "learn" which messages are to be expected and harmless. Once an unfamiliar entry pops up, I can recognize it immediately. One popular example are the constant messaging of IO errors on sr0 (the non-existant CD drive, a long known bug in wine) when using applications in wine. But when a real IO error let's say on sda1 was to pop up, I'd have to be alerted.

[Omonarc]

You should give Silverblue a try (or some other immutable variant of Fedora.) It's been incredibly stable to the point where it's almost boring! 😆 There are some annoying things, but once you start figuring out how to work with the OS being immutable, it's actually quite nice, especially for desktop use.

[twigoftheforest]

I used to run Arch all the time... then one day I got tired of troubleshooting and went to Fedora 21. Been running it ever since. When pipewire came out, it was the best and worst thing. The sound quality was much better and bluetooth worked much better. But there was random bugs where the mic quality would go or it would mute audio. For that reason I always stay on the (n - 1) Fedora release now. Contrary to popular belief, a version is supported for 12 months even though a new version comes out ever 6 months. I miss the "I know everything on my system" feel Arch gave me, but I feel like Fedora allows me to be lazy and I'm content with that lol

[AFSninja]

A video on a minimal fedora install and setup a system could be cool if you havnt done it already.

[thomas_brewer]

I used Pop!_OS, then switched to Arch (failed install of openssl which required me to boot from USB because I couldn't decrypt my disk), then switched back to Pop_OS! (had some issues with my monitor), tried to switch back to Arch (update resulted in no kernel found), then I switched to Fedora. A lot of issues I had with Pop!_OS and Arch just aren't an issue on Fedora. Such a breath of fresh air. Funny enough back in the day Fedora was my Linux distro of choice for desktop and Centos for server in the early 2000's then I switched to Mac for desktop and Unbuntu for all server stuff. I should have never left. Home sweet home.

[simplicity]

Good video, found you randomly in recommended.
I am a brand new Linux User. Less than one week in with typical milquetoast Ubuntu setup, Intermediate Tech Literacy, only started programming 3 months ago.

Within four days:
Installing a separate python environment -> installed the wrong one and wrong dir -> "sudo apt-get purge python3" lmao -> obliviously *removed the default package* -> reboot -> forgot my whoami and can't boot to desktop -> remembered and fixed in 20 mins (thanks to Google)

Changed NVIDIA Driver in bash -> "... purge nvidia*" -> followed default re-install procedure -> reboot -> Monitors colour space are in: 0: RGB, 1: -R-GB, 2: G-RB, "nvidia-smi" is broken, ubuntu-drivers install is fine -> back to default driver -> still broken

Can still boot and move all my files before reinstall.
Fail Fast; Fail Often

I have been told "learn programming" etc... Being High-Functioning ASD & ADHD, reading the logs and problem solving outside my comfort zone is the definition of fun for me, submitting a question to a forum is a last resort.

At-least I have the incentive to go back to Windows 11, completely re-organise my file structure, re-do the partitions, fresh install all OS, use Ubuntu as a back-up, roadmap to another Distro which is does justice to my decision of abandoning "OS on-rails" and migrate from there.

My only regret is not experiencing sooner, when my plasticity was higher.

[kk-linux-rocks]

Matt. A suggestion. You should have ONE PC that's your production PC that just runs. You only apply the security updates as they are posted, but nothing else. Find a non-rolling distro and just use it for that. That's what so many of us (including me) have done. My main video rendering PC is LMDE 5. It just works each and every time and I only apply updates once or twice per month with it. You can use your other PC for evaluating and distro-hopping.

[SudeeptoDutta]

Would having btrfs as filesystem helped here ? Having pacman hooks to create snapshots before & after system upgrade, we can rollback to previous snapshots just to check if the issue is with any package updates.

Let me know your thoughts.

[alexstone]

2:11 i can understand that, i have been looking at linux through developer eyes and probably always will as i think entering one command is easier and can be automated instead of going through 3 menus and pressing options and remembering what i set to what using screenshots or whatever

I am thinking of going to some old distro and just starting to improve the documentation, report issues maybe fix some bugs, im currently looking at almalinux, and yes i know its old but with distrobox and flatpak it does not matter and you can run very recent KDE on it using CRB which i think is a great idea

[wyfyj]

Knew you'd go right back to Fedora. At least you are a fan boy of both. Just like my two lol