Arch Linux

written by Ilya Ilembitov

Original article is in the Arch Forum in “Topics going nowhere” section. Obviously Arch mods felt the need to trash this article as this subject is often brought up there and do not end receiving very useful replies. I thought it was a good read and asked “ilembitov if I could post his article here. So here is what he said , his exact words:

Hi, all. Reading a lot of Linux forums (not only this one) I can see the major problems of all Arch users: Arch Linux is bleeding edge, but it also has the unpredictable updates. Which is why you can see posts like “I haven’t updated my system in a while, today I ran the update routine, and basically, nothing works”. Which is not to mention less dramatic, but more frequent complaints like “I updated the package foo and I got myself into troubles”. Besides, I believe that the introduction of the LTS kernel is actually a good sign that there are also users who need Arch to be stable.

Ok, I started thinking about what should a user do, so that things like that wouldn’t happen. Here is my short howto.

1. Basically, as soon as you have freshly installed your Arch system and all the desired apps, copy the list of installed packages to the IgnorePkg section of /etc/pacman.conf
This way, when a new package comes out, it won’t be automatically updated.
2. If some package doesn’t work for you as expected (and you know that it’s due to an upstream bug or maintainer’s mistake), delete it from that section and update it until the problem gets solved – then get it back to IgnorePkg
3. You should only make transition to a next major release of anything when it makes sense. Say, the new version of the kernel has a better support for your hardware, or runs faster. Or a program you use gets a new feature and you do need that feature. If there are no changes for you, why bother? But even when you do need to upgrade, do not update the package as soon as it leaves /testing! There is also another reason to upgrade a package – when it’s required (as a dependency) to upgrade a package you need.
4. A package in Arch repos has the upstream version as well as the internal version that comes from maintainers. So, after the package has hit the repos, you better wait until the internal version goes higher: that means, people have bumped into problems with upgrading it (even though it’s not /testing already), maintainers have fixed it, you are less likely to get surprises. Even better if you also wait until some bugfix releases to appear upstream as well.
5. As soon as you make the transition and get to the state when everything goes okay, get the package back to IgnorePkg and work. Until next major update you need! (then roll back to the step 3).

I guess, that sounds reasonable. But why does one have to do that? Here is the thing.

As far as I understand, this is how the major update of the package gets to Arch:
1. It hits the /testing
2. People who actually use testing (and the maintainer, if he has chosen to support this package in the repos because he actually uses it on a regular basis and needs it to work) upgrade, bump into troubles. Troubles from upstream, but also troubles due to a maintainer’s mistake. People write bug reports. It it’s the maintainer’s fault, he fixes that. People try that again, write bug reports, etc.
3. The cycle in number 2. goes on until there are no bug reports for some time. Then the package goes to /extra or /core
4. People who do not use /testing, update to this new version. They bump into troubles, write bug reports and so on and so on.

So basically, it may sound reasonable. But still one actually can’t just upgrade without worry while using the current (non /testing) repos. So, there must be a problem. Which is?..

It’s obvious. Arch is already a bleeding edge distro. So, who is going to use the testing repo of the bleeding edge distro? I’m afraid, not many. A lot less than people who don’t. That means, that there is less probability someone would notice a bug while the package is in /testing. Or, possibly, there could also be no one to do that! So, /testing doesn’t do the job it’s expected to do, or just doesn’t solve the problem completely.

This is why problems happen. My post was caused by a discussion on a Russian Linux-related forum: a user has complained that (among other troubles) after an upgrade to GNOME 2.28 he got issues with Nautilus. And he was asking for an advice, whether or not he should switch to Debian testing. In the neighbor thread Debian Testing users were discussing the following: why the hell didn’t the Debian maintainers update Nautilus to 2.28 in testing while the other GNOME packages were upgraded? I’m sure I don’t need to point out the irony here.

OK, one could get a foil cap and apply my solution to the problem. Which is basically: wait until other users bump into troubles, those will get sorted out – then you make an upgrade. This is time-consuming, but also more flexible. Perhaps, this is the part of the Arch way. Like, Slackware users don’t want somebody else to define dependencies for them, so Arch users don’t want somebody to do Q&A for them, thus restricting their freedom.

So, we don’t need to change anything and everybody just gets along? Not really. Let’s just imagine everybody applies my approach. And it’s pretty reasonable, isn’t it? If you aren’t getting any benefit from an upgrade – why bother? If you do need to upgrade – why risk, those who feel like experimenting or just aren’t wise enough will do the dirty job. The point is, that when everybody applies this approach, there are no “risk takers” or “unreasonable ones”. And this is where the whole thing gets stuck. OK, actually I don’t really believe all Archers are going to do that. But I guess this is a good illustration why current model isn’t perfect.

There is a reason why it works in Debian unstable->testing chain (which basically lives like a rolling release system when there is a lot of time until the next stable Debian release). Because Debian userbase is large. There are lots of people who use testing and sid. There are other distros that are based on testing and sid (not to mention Ubuntu – there is also Sidux, for example). Arch just doesn’t have that. And so, it doesn’t work out for him.

My point is not about switching to traditional release cycle (I don’t think anyone would seriously consider it). Arch is partially fine because it doesn’t have releases. But there is also another thing. Arch has lots of things other distros don’t have. Pacman, ABS, AUR, configuration being done via the /etc/rc.conf, etc. What if a user wants to have all these and not have troubles with upgrades? Does he have to stick with the paranoid IgnorePkg approach to get a stable and predictable system?

Of course, we can’t just develop a universal solution against maintainers’ mistakes or upstream bugs. People do mistakes (I really didn’t mean to offend the maintainers here), and apps can’t be bug-free. But we should develop a system to detect those issues before they get to the end user machine. Perhaps, we should track the statistics of previous troubles caused by the package and consider it to define the time the package should spend in /testing after the last solved bug report before it gets out if no show-stoppers occur. Okay, may be that will do Arch less bleeding edge, but it will also cause more people to try installing versions from testing (and, consequently, lessen the time needed for it to get to /extra or /core). Perhaps, the package should also automatically roll back to the previous stable version if the recent upgrade was reported to have a serious issue (with the upgrade getting back to /testing). There is /var/cache for that, but this still could make upgrades more reliable.

Sorry for this pretty wordy article. But I do believe that the issue is pretty important. Perhaps, I don’t have the right picture (for example, may be the testing repo is more complicated than that). I would like those heroes who managed to get through this sea of words  to share some ideas.

And Arch Mods and Users then emphasized on the fact how important it is for Arch being a community driven distro to depend on it’s users to run the [testing] repository and file bug reports. My question: Who gets to enjoy the stable product then? Clearly, Arch is not about stability. Bleeding edge and stability can never go hand in hand. Atleast Arch devs are not hiding this fact and stating it bluntly, but yes there are some people who go to other forums *cough ubuntuforum cough* and whine about Arch being unstable and it broke their system.

B [Arch forum Mod] wrote:

I like how users feel like a victim when this thing (a community-driven distro) only thrives when both users and devs alike provide input and feedback. And most Arch users only complain ‘OMFG ITS BORKEN’ when it actually breaks, but never a) run testing or b) report bugs. Maybe you’ll see them saying ‘oooh distro X has shiny  new feature Y. When is it in Arch!?88’.

Then what do you expect?

Even Ubuntu users seem to be more active on that front. Seriously. Also, if you can’t fix what breaks during upgrades, then Arch is not for you. Period. Either you got what it takes or you don’t. I see a *lot* of people here that seem to be running Arch for a while yet don’t have a clue of the very basics. There are distros that cater to that kind of public.

Later part of ilembitov’s reply:

Actually many people here have probably gotten the wrong idea of why I described the IgnorePkg approach. It’s all about rethorics – I guess that it’s good that many people have found this technique bad in some way. Basically, I believe, this was the ultimate summary of all advices that people give somebody when he says that Arch is too unstable for him, yet he likes being bleeding edge. In such discussions people usually either mention the /var/cache/ approach for rolling back or the IgnorePkg thing (which are actually equivalent). I have just thought of an ultimate solution on this basis. It just takes the whole thing to an extreme. And actually, such a solution roughly imitates what maintainers in larger, non-rolling distros do: this is pretty much the way Debian (or Ubuntu) people make their stable release (or stable + backports repo, for that matter). And my point was to (while still taking things to an extreme) show why this thing is by design wrong – the same way all actually popular approaches taking the similar direction are wrong. And certainly I did not recommend this HowTo. I believe this solution to actually solve the problem of one user completely yet creating a lot more problems in the long run for Arch Linux. Moreover, I have tried hard to show that currently, a rolling release-based distro pretty much involves techniques that are common in different projects – and thus, inappropriate.

So my point was – we should redesign the process (and this is why I wanted to provoke a brainstorm sort of thing). People who started protecting maintainers have a clear misconcept of what I was talking about. I did not intend to blame the maintainers. That’s unethical and anyways, it doesn’t relate to the topic. I believe that, ArchLinux, being a non-mainstream distro will always have less maintainers and less users (thus, less bug reports) than needed. The thing is, how to redesign the process so that the lack of resources will be less noticeable. Design the fail-proof mechanisms in the process when things have already gotten nasty – and design the mechanisms and procedures so that things will be less nasty. That may involve encouraging the use of /testing (I was actually surprised by somebody mentioning that basically it’s a bad idea to use /testing only partially. I always thought I would be fine to install an app from testing provided I have satisfied all the dependencies with required versions. And this is also a problem.) That may involve redefining the policy that the maintainers should follow when the packages that got out of /testing have revealed show-stoppers. This is what I was talking about.

Oh, and more thing. People who believe that Arch is okay to have these kind of issues and that users should be ready to deal with them or just leave – they actually mistake an advanced user-oriented product for low-quality and poorly-designed product. It’s about enjoying the flexibility and other virtues of Arch and not enjoying its flaws because you are proud you “have survived”. After all, software, even being fun, still has one major point – it should work.

All I gotta say is that, brace yourself for any package breakage and system instability if you use Arch and don’t complain about it. Don’t feel like a victim, you have been warned! It’s not a bug, it’s a feature. 😛