pieterh wrote on 16 Nov 2015 15:14
I spoke at DomCode in November 2015 (excellent conference, small and beautiful city!) explaining my top rules for building open source communities. One person asked me later to explain why I recommend to merge quickly, without waiting for Continuous Integration testing to finish, and without review of the code. I'm going to call this strategy Optimistic Merging or OM. Here's the reasoning behind OM.
Standard practice (Pessimistic Merging, or PM) is to wait until CI is done, then do a code review, then test the patch on a branch, and then provide feedback to the author. The author can then fix the patch and the test/review cycle starts again. At this stage the maintainer can (and often does) make value judgments such as "I don't like how you do this" or "this doesn't fit with our project vision."
In the worst case, patches can wait for weeks, or months, to be accepted. Or they are never accepted. Or, they are rejected with various excuses and argumentation.
PM is how most projects work, and I believe most projects get it wrong. Let me start by listing the problems PM creates:
- It tells new contributors, "guilty until proven innocent," which is a negative message that creates negative emotions. Contributors who feel unwelcome will always look for alternatives. Driving away contributors is bad. Making slow, quiet enemies is worse.
- It gives maintainers power over new contributors, which many maintainers abuse. This abuse can be subconscious. Yet it is widespread. Maintainers inherently strive to remain important in their project. If they can keep out potential competitors by delaying and blocking their patches, they will.
- It opens the door to discrimination. One can argue, a project belongs to its maintainers, so they can choose who they want to work with. My response is: projects that are not aggressively inclusive will die, and deserve to die.
- It slows down the learning cycle. Innovation demands rapid experiment-failure-success cycles. Someone identifies a problem or inefficiency in a product. Someone proposes a fix. The fix is tested and works or fails. We have learned something new. The faster this cycle happens, the faster and more accurately the project can move.
- It gives outsiders the chance to troll the project. It is a simple as raising an objection to a new patch. "I don't like this code." Discussions over details can use up much more effort than writing code. It is far cheaper to attack a patch than to make one. These economics favor the trolls and punish the honest contributors.
- It puts the burden of work on individual contributors, which is ironic and sad for open source. We want to work together yet we're told to fix our work alone.
Now let's see how this works when we use Optimistic Merge. To start with, understand that not all patches nor all contributors are the same. We see at least four main cases in our open source projects:
- Good contributors who know the rules and write excellent, perfect patches.
- Good contributors who make mistakes, and who write useful yet broken patches.
- Mediocre contributors who make patches that no-one notices or cares about.
- Trollish contributors who ignore the rules, and who write toxic patches.
PM assumes all patches are toxic until proven good (4). Whereas in reality most patches tend to be useful, and worth improving (2).
Let's see how each scenario works, with PM and OM:
- PM: depending on unspecified, arbitrary criteria, patch may be merged rapidly or slowly. At least sometimes, a good contributor will be left with bad feelings. OM: good contributors feel happy and appreciated, and continue to provide excellent patches until they are done using the project.
- PM: contributor retreats, fixes patch, comes back somewhat humiliated. OM: second contributor joins in to help first fix their patch. We get a short, happy patch party. New contributor now has a coach and friend in the project.
- PM: we get a flamewar and everyone wonders why the community is so hostile. OM: the mediocre contributor is largely ignored. If patch needs fixing, it'll happen rapidly. Contributor loses interest and eventually the patch is reverted.
- PM: we get a flamewar which troll wins by sheer force of argument. Community explodes in fight-or-flee emotions. Bad patches get pushed through. OM: existing contributor immediately reverts the patch. There is no discussion. Troll may try again, and eventually may be banned. Toxic patches remain in git history forever.
In each case, OM has a better outcome than PM.
In the majority case (patches that need further work), Optimistic Merge creates the conditions for mentoring and coaching. And indeed this is what we see in ZeroMQ projects, and is one of the reasons they are such fun to work on.
References:
- ZeroMQ RFC 22, C4.1: the Collective Code Construction Contract.
My top tips were, for what it's worth:
- People before code: build the right community and it will build the right code.
- Progress before consensus: look for processes that work without upfront consensus (except on rules).
- Problems before solutions: use a problem-driven process.
- Contracts before internals: use contracts to test behavior, not inspection of internals.
- Rules before hope: write down your development process or use C4.1.
- Merit before power: treat everyone fairly and equitably.
- Market before product: aim for a market of competing and interoperating projects, not a single product.
Comments
I am always grateful for your guidance via emails and blog articles.
Since most of the humanity are tied to corporate jobs, designing a better corporate model will certainly help.
The principle of optimistic merge seems to be to enforce trust.
In corporations, to enforce trust usually means to enforce open allocation. However, I think there's more to enforcing trust in corporations.
How would you enforce trust in corporations in addition to declaring open allocation as a set of rules if you are designing a new corporate architecture?
Note : It would be better if people are notified of replies via emails or whatnot. After a while, I forget to revisit this page. I don't trust my memory.
You normally get email notifications for comments to a page here you've commented on, edited, or have Watched. Maybe they end up in your spam…
I've no recommendations for enforcing trust in corporations except to use the old technique of competition in a free market. Smaller independent businesses that compete will always build better products than large monopolists.
Portfolio
I think there's a 5th category that you've missed, that of an ostensibly good contributor who appears as 1. or 2. on your list but only does so to build trust.
If "Mallory" then puts in a patch to CurveZMQ that mails all private keys to a server they control and does so in a way that is obfuscated so all tests continue to pass, it may get missed for a long time with an optimistic merge strategy.
However unlikely this is, my point is that the reason optimistic merging is not done is the fear that nasty hidden behaviours are added to the project, and the belief that the maintainers have enough intelligence and experience to catch such behaviour before it enters the project.
I do get your point though that if "Mallory" did try this, it would be permanently part of the project's history and after discovery it would provide a useful example of "Here's when someone tried to break our security model, here's how long it took us to revert it"
I'd like to know your thoughts on dealing with hidden malice of this type instead of overt intrapersonal malice.
Yes, this a threat that I'm aware of, and it has always been a concern.
It affects all open source, indeed all software development. I believe closed software is more at risk of backdoors than open source, and this is I think what we've seen as an industry. It's relatively simple for Mallory to get vulnerabilities into a closed project, and extremely hard to find them afterwards.
So the economics of open source, with thousands of people able to look at the code and spot anomalies, works against Mallory.
Let's assume Mallory is patient and deceitful and acts like a valid contributor long enough to get control over a project, and then slowly builds in his/her backdoors. Then careful code review won't help you. Mallory simply has to gain enough trust to become a maintainer, which is a matter of how, not if.
What does help is:
Maliciousness often hides as incompetence, so any practice that catches accidental vulnerabilities will make it harder to be malicious. E.g. in our C code we use a strict allocation model that gets rid of buffer overflows. Any code that doesn't respect this stands out, and gets fixed.
The best defense against Mallory is size and diversity of the community. So in this respect, OM should deliver more secure software.
Portfolio
Thanks for such a clear answer. I agree.
So what you're trying to do with OM is optimize for the common case (people want to contribute) while requiring small, clear problem solving patches that restrict the scope of malice and/or incompetence (rare).
When you moved the Github repo of zeromq, my builds suddenly failed.
We have to learn from the previous mistakes of a "human in the middle", such as people who can have modify content.
IPFS to the rescue: https://ipfs.io/
Tarballs, no GIT!
I'm sorry if we caused stress when we renamed the git repository some years ago (is this what you're referring to). Yet we've always provided tarballs via zeromq.org, which most packagers use.
Indeed every version ever released is on http://download.zeromq.org/ so your builds should never break.
Yet, getting the source (or a tarball) from github is going to be more secure than getting it from anywhere else.
Portfolio
You're describing something similar to how Wikipedia pages work, and that is admitted by all to be one of the more newbie-hostile sites on the internet.
Consider that you are creating the conditions for something like this, which is all too common on Wikipedia:
- new contributors invariably commit without a full understanding of what they are doing
- those who appoint themselves protectors of the code will revert the new contributor's work *instantly* if it doesn't meet any one of their capricious objections, often because of unconscious biases. The reverter will often feel no obligation to explain why. If they do, they will at best refer the contributor to a list of policies
- (side effect: the list of policies will expand infinitely until it is an effective way to repulse any outsider)
- new contributors feel slapped in the face and never come back
I can't edit my post, so an addendum: on Wikipedia, people set up automatic notifications for this sort of thing, so they can quickly revert. It's even easier to do that on Github. So you're not only setting yourself up for random contributions, you're setting yourself up for random vigilantes to revert every new contribution.
Long story short there isn't any particular set of *technical* practices that is going to make people treat contributors well.
What you need there are cultural practices. For instance, some of the Wikipedias like the Russian Wikipedia, will randomly assign an experienced editor to help with a new contribution. That creates a learning community more like what you are after.
I've contributed often to Wikipedia, as a newbie, and sometimes experienced this yet not always. It depends on the page. Some pages are hostile, like some open source projects.
If a project is hostile, it makes no difference what rules you offer: those in power will be aggressive and offensive. The solution then is to fork the project and be a better owner. This is hard on Wikipedia because you can only have one page for a topic. For open source it's trivial.
I think the rules we develop (like C4, the umbrella contract we use) are more to help the majority of decent people work together than to educate abusive people into behaving.
So in practice what you describe happens, yes, yet not because of OM. A precondition for successful OM is decent owners who want their projects to grow.
Which is what we see in practice: many ZeroMQ projects practice this, and it works nicely, delivers stable growth and happiness. (And we do not see patch wars.)
Portfolio