Maintainers Matter: The case against upstream packaging

The case against upstream packaging

Original Maintainers Matter: The case against upstream packaging Editable

version 1 of 1

Expand comments.

Disclaimers, et al. These opinions are not at all related to Arch Linux, though maintaining Arch packages has certainly colored my opinions. I’ve been a volunteer maintainer for years now, because I feel it helps people. I have a bias towards the value that maintainers and maintaining packages provide.

While most people are taking about the new future of Universal Linux Packages from a technical standpoint, the social perspective isn’t getting much attention. Frankly any technical issue can be solved by a better "universal package" but the social changes can’t be improved by emailing a patch. So let’s examine the relationship between upstream developers, distribution maintainers and end users. With a focus on how we got here and where we might go next.

Normally I really like the word ‘upstream’. But it has a lot of connotations and doesn’t suit the tone of this essay. To define it, upstream is generally used as a collective pronoun to mean "the group of people who gives us software". They are people who fix bugs, add features and occasionally make mistakes. They have the best interests of their users and their software at heart. They are good people who always mean well and take pride in their work. Upstream is not a single group, but spread among everyone who writes software.

"Upstream", however seems to be a mostly-FOSS concept, largely a side effect of the selective pressures that distributions provide. Since this essay goes into what a weakening of distributions would look like I’m going to use the original, older, uglier, more agnostic term "Independent Software Vendor" or ISV.

Why say ISV? Why does this word matter? "Upstream" is used both collectively for FOSS as a whole and singularly when talking about a single project. This re-use of the word is a small flaw as it suggests all software developers are fundamentally the same. This is not the case: some have very different values. Being "upstream" also has an implicit social contract attached to it. The "I", for independent, is to remind us that this is not always the case. And the "V" is for vendor, a reminder that sometimes these are businesses who consider profitability to overrule all other considerations.

Obviously this does a disservice to those who volunteer their time and intelligence without compensation, and do consider software quality and user happiness to be important. For the purposes of this essay I’m painting everyone with the "ISV" brush, even those pillars of the community who deserve better. But we are heading towards a future where the nice guys don’t win.

Let us start with the (recent) past that kicked off the whole idea of "ISVs can package stuff" and "maybe we don’t need maintainers".

The Cargo Cult of App

The promise: Phone app stores came out of nowhere and suddenly had millions of developers appear out of the woodwork overnight! If we make our distribution more like phone app stores then we get millions of application developers!

The reality: App Stores have been a nearly unmitigated disaster for users. Supposedly one of Linux’s failings is that there is too much pointless choice, too many K and G versions of things and it divides developer efforts. Why have so many window managers and text editors? App Stores have the same problem. With the traditional FOSS model, there are a hundred different programs and each program is missing a different feature. In the App Store, there are a hundred different programs doing the same thing but each screws the user over in a slightly different way. Spying? Ads? Battery sucking rookie mistakes? Battery sucking bitcoin mining botfarm ? Take your pick.

The promise: But Linux is different! No one does those horrible things on Linux! And no one ever will!

The reality: But is it different? Somehow we’ve gotten this idea that everyone who writes software for Linux does it for the warm fuzzies. No Linux ISV tries to take advantage of people, and none of them ever had. Linux has been around for 20 years. Everyone has been ethical and trustworthy the whole time. Clearly they have proven themselves and will continue to be ethical and trustworthy. Alternatively, Linux is somehow immune to this darker side of human nature.

Maintainers are the Difference

Linux-related news might suggest maintainers are useless. Either they are bikeshedding on mailing lists or closing bugs as WONTFIX, or introducing security vulnerabilities in software they don’t understand, or generally being a BOFH. Of course this is wrong – we simply want to use the best operating system possible. So what does a maintainer actually do?

Mostly a maintainer is a packager: they take software from an ISV and repackage it for a distribution’s packaging system. And here is where some people feel is a huge waste. Why package the same software a hundred times for a hundred different distributions? (There is good reason for this "waste", but let me finish describing everything else a maintainer does.)

More fundamentally, the maintainer is the primary line of defence and interaction between users and developers. Maintainers shield developers from uninformed users, allowing the devs to write software with less support overhead. Non-bugs are caught and filtered out. Low-quality bugs reported to the distribution’s tracker often becomes a good bug when the maintainer reports it upstream. Without distros the ISV would have to carry the full weight of support. Here’s an example:

<NewUser> I've never gotten OpenSCAD to work on my computer, so I don't know anything particular to it. <@teepee> NewUser: we could help with that (hopefully) :) <NewUser> I'm gonna upgrade my system software when Slackware proper is RTMed. We'll see then. <@teepee> ahh, cool, slackware is still around * teepee used that like 15 years ago <NewUser> Not a lot. I'm also installing Arch. <@teepee> ahh, with arch, that's easy, you just complain to keenerd if it's not working <keenerd> teepee: Spot on :-) <@teepee> :) <@teepee> it's always good to know there's someone who cares 

Maintainers also shield users from developers, offering a layer of quality control. At the very least, all the software in a distribution should play friendly with any other piece of software in the repositories. Maintainers are responsible for that, as well as choosing a subset of the (subjectively) best software FOSS has to offer. Maintainers will disable features that they feel act in bad faith. Maintainers’ greatest power is the ability to outright say "This is not good enough for our users" and consequently punish an ISV by either patching out the offensive part or in extreme cases removing the software from the repositories. ISVs know this and so don’t act out. After 20 years of enforced good behavior this has lead to the idea of ISVs as "the benevolent upstream developer." This is why Linux doesn’t have spyware, doesn’t come with browser toolbars, doesn’t bundle limited trials, doesn’t nag you to purchase and doesn’t pummel you with advertising.

It is simply not possible for abusive software to exist in Linux today. No maintainer would willingly release it. If a maintainer ever got a reputation for releasing abusive software they would be stripped of their privileges. Someone who cares steps up and becomes a maintainer instead. The system is aggressively self correcting. It has been like this for twenty years, so much to the point that few Linux ISVs even consider being unethical. And Linux users have a very hard time imagining that any ISV could ever act in bad faith.

Maintainers Provide Consistency

For the sake of argument, let us presume that all ISVs are ethical and hold the interests of the community as their utmost ideal. Would this utopia still require maintainers to provide quality control? Yes, because no one can agree on what "quality" is beyond "does it build and run". Any good distribution is not about the file extension on the package – it is about the decisions the software implicitly makes. A distribution is a world view provided by the maintainers and shared by the users. In other words, maintainers don’t just ship packages. They provide a certain flavor of consistent and preferred opinions to a group of users. (As an aside, this is why users of different distros often don’t see eye-to-eye.)

Could ISVs possibly make everyone happy? They can’t. It is impossible. What happens when two groups of users expect different things from the same software? To give an example, imagine an ultralight distribution like Puppy. They want the software as slim as possible. Recompile as i386, it’ll save 5%. Chop out the built-in-help, we’ll use the website. Get rid of the autosave-every-0.1-seconds, it’ll wear out the flash drives. Now imagine an Ubuntu user. They want every bell and whistle enabled. In fact, could you add integration for our new notification and desktop search services? And then imagine the paranoid LFS user. The don’t trust any binary package and insist on getting full sources, reviewing them and building them From Scratch.

What is an ISV to do? They can try to package half a dozen different flavors of their software. Now a hypothetical cross-distro Universal Packaging Format is not saving them any effort because they need to make (and test, and support) a different build target for each demographic of users. Or the ISV can ignore some users, making a subset of them unhappy.

If people are unhappy enough they will repackage the software in a more suitable manner. Now the users have unwittingly become maintainers. The moment someone starts improving 3rd party software, they become a maintainer.

And if a maintainer shares their packages, they’ll attract like-minded users who appreciate the work as well as like-minded maintainers who can share the workload. And thus a distribution is born. It can’t be fought. A monoculture will breed diversity.


To clarify a few points often heard:

  • The promise: Sandboxing makes you immune to bad ISVs.
  • The reality: It protects you from some things and not others. App Stores have proven that ISVs will try to get away with as much abusive behavior as possible. Sandboxing protects against many threats, but does not suddenly make people ethical. Distributions with human maintainers are more likely to make mistakes than an OS-enforced sandbox, but human maintainers are also more likely to sniff out the dumb stunts that ISVs try to pull. No human maintainer is ever going to willingly package a flashlight app which phones home. From source anyway.
  • The promise: By cutting out the maintainer middle-man, security fixes can be pushed out to users faster.
  • The reality: Often maintainers are faster, particularly with anything using shared libraries. A distro can update a shared library once and all software is now safe, without needing to rebuild every piece of software using it. Most of the sandboxes do not share libraries and so you are waiting on the slowest ISV to rebuild their software. If they ever do. ISVs are notorious for using ancient libraries with extensive out-of-tree patches and never migrating the patches forward.
  • The promise: A universal package means that ISVs don’t need be aware of every distro and its unique quirks.
  • The reality: The don’t need to and never have. Put out the sources with a makefile. The maintainers can handle the rest. It’s what we do. Heck, the makefile is optional. We can write one. (But please merge it when we send a pull request!)
  • The promise: App Stores are a re-spin of the traditional distribution repository. One central location, easy search, easy install, automatic updates. Its practically the same thing.
  • The reality: They are still sorely lacking. No app store has the slightest concept of dependencies. They often aren’t even dogfooded for the operating system’s software. And of course anything that passes an automated scan can get into an app store. Distributions provide extensive curation that you can’t get without an army of volunteers.
  • The promise: Make an App Store and a million devs will appear to fill it with software.
  • The reality: Smartphones had a large user base before they had any apps. Get the users first and the devs will follow. Thinking an app store will bring in more developers is cargo cult logic at its finest. It will however bring in a wave of opportunistic software that was previously barred from Linux. There are entire frameworks designs to wrap an Android or iOS application with malware, integrating ads or analytics with a single click. That simply wouldn’t make it past a maintainer.
  • The promise: A common packaging effort will improve the quality of software on Linux, because packaging for every distro is too hard.
  • The reality: Handwriting is too hard. I have created a solution, the "keyboard". Now people will no longer write dumb things because the hard part is solved. Making packaging "easy" will not improve the quality. Just like how legible writing has no relation on writing intelligently, the mechanics of packaging has no relation on the quality of software. Write good software, release sources, and every distro will package it for you.
  • The promise: ISVs releasing packages means less duplicated work between distros.
  • The reality: Distros exist because users can’t agree on how exactly we want software. A universal package will not be good enough for some percentage of them. The actual act of packaging is the fastest part. The slowest part (fixing bugs that upstream hasn’t) is shared between distributions because all of our patches are public.
  • The promise: Many embarrassing bugs will be avoided, because ISVs are smarter and more experienced than maintainers.
  • The reality: Oh boy the stories I could tell about the things I’ve seen. But out of courtesy, I won’t name names. Needless to say they are as dumb and human as we maintainers are. If you want, think of maintainers as outside consultants who will find the craziest thing an organization is doing. Except we don’t charge $200/hour for what amounts to best-practice common sense.
  • The promise: Software directly from the ISV has a complete chain of trust and is better for users.
  • The reality: This is only true if the ISV won’t try to sneak something in. If you don’t trust your maintainers, don’t get rid of them. Verify their work with your own builds and examine the chain of trust the distribution provides. Distros are usually way more open about the chain of trust than ISVs are.

Yes, all of this will happen.

Shuttleworth has explicitly stated that the whole point of their new universal packages are to take control away from the maintainers/users and give it to the ISV. To quote:

"With snappy Ubuntu, Owncloud can publish exactly what they want you to use as a snappy package, and can update that for you directly, in a safe transactional manner with full support for rolling back. I think upstream developers are going to love being in complete control of their app on snappy Ubuntu Core."

Not only will they love it, some are probably already cackling with glee at the prospects of "complete control".

Alternative Solutions

No, I’m not just going to carp about how change is bad. Here are some other ways to achieve the same goals.

Let’s say your distro wants to attract developers. Suse’s method has been to set up a comprehensive build service . Now the ISV doesn’t have to maintain a distro-specific toolchain, the distro will do that for them. And if it builds on the build server, the distro promises that it’ll run across their operating system. Of course these services are not cheap. In exchange for hosting the service, the distribution retains their ability to curate. If an ISV acts in bad faith and refuses to meet the distro’s standards, then the distro can de-list their packages. Suddenly the ISV needs to spin up a tool chain and hope that people will google them instead of using the integrated search and proceed to "sideload" their software.

Let’s say you don’t have the infrastructure for a giant build server, but you still want it to be easy for devs to get software into the distro. First and foremost, your distro should be easy to contact. Put in big letters on the home page something like "Want your software in the official repo? Send an email to this address and our maintainers will look at your proposal and get back to you soon."

Let’s say you’re an ISV and you want your software in a distro. You could learn how to make and release packages, but it is much less effort to cooperate with the distro and find a maintainer who will champion your software. And keep in mind this is a purely social solution to a supposedly technical problem. Supporting a dozen distros doesn’t require learning a dozen build systems, just making a dozen friends at different distros. Admittedly if you’re an ISV who does proprietary software you are going to need some really good friends.

But let’s presume you are an ISV who writes proprietary software. In this case, please don’t release binaries. Give the maintainers source code and they’ll do everything for you. You might think it is impossible to be proprietary and release sources, but you would be wrong. Microsoft has some brilliant lawyers who have drafted licenses which let you keep proprietary levels of control.

post_install() { echo $message; }

Maintainers, distributions and all the variety of "wasted duplicated effort" is the only reason the system has worked. Maintainers are the ambassadors between users and developers. Together we have formed a beautiful symbiosis with FOSS programmers because we could freely tweak, build and distribute their software. We will always disagree, do something about it, attract users, attract volunteers and formalize these groups as distributions. Putting packaging in the hands of those who write software (instead of those who use it) will inevitably lead to a proliferation of bad actors, as has been demonstrated in every other major operating system lacking strong community curation.

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Maintainers Matter: The case against upstream packaging

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址