The Digital Revolution Roadshow: Vilnius
On 9th December 2013 I'll be giving the opening keynote at Build Stuff '13, in Vilnius, Lithuania. I'll also be giving a technical talk in the conference itself, so this is a rare occasion to come and heckle me not once, but twice in a day. This is an awesome conference, with over 50 speakers in three days. If you bring a copy of any of my books, I'll sign them for you. Here is the abstract for my keynote…
The New TIPC Transport in ZeroMQ
ZeroMQ now supports TIPC, the cluster IPC protocol developed by Ericsson. This will be part of the next major 4.1 release. In this article I'll interview Ericsson's Erik Hugne, the engineer who added this transport to ZeroMQ.
Securing ZeroMQ: Soul of a New Certificate
While wrapping ZeroMQ's new security API up in the high-level C binding, I accidentally a certificate format. With all respect to X.509 and existing PKI standards, the reason I built CURVE for ZeroMQ was to get simple and foolproof security. Using a complex legacy certificate format would spoil the soup. I've no idea what the right format is, but to start, I'm going to try to collect requirements.
Using ZeroMQ Security (part 2)
In the previous article I gave an overview of how and why ZeroMQ's security layers work. In this article I'll develop a simple secure application, step by step. We'll use a simple example of a server PUSH socket sending "Hello" to a client PULL socket. We'll work through the ZeroMQ NULL, PLAIN, and CURVE security mechanisms, up to full authentication. The examples are in C but the principles apply to all languages.
Using ZeroMQ Security (part 1)
In this series of articles I'll explain how to use the new ZeroMQ security layers in your applications. The example we'll make is a chat application that provides unbreakable strong security. In this first article, I'll explain more about ZeroMQ's security technology, how it works, and why we chose it. (Read part 2.)
The Castle and the City
One of the topics for my upcoming book "Culture and Empire" is how to build online communities. I'm going to argue that there are two general layouts for a large organization — the Castle and the City — and compare these. Is your project a Castle, or a City?
Charlie and the X-ray Factory
Last week I had the pleasure of visiting not one major European physics research facility, but two. My first stop was at ESRF in Grenoble, then I went to visit CERN in Geneva. Both these organizations are moving to use ZeroMQ for their control systems. In this article I'll explain what that means.
Securing ZeroMQ: Circus Time
Thanks to the quiet but persistent work of Martin Hurton, the master branch of libzmq, the ZeroMQ core library, now "does security". In this last article in the mini-series, "Securing ZeroMQ", I'll explain what we built, and why, and how this can work for your ZeroMQ applications.
The Gender Gap in Coding
My daughter, nine, comes with me to tech conferences like FOSDEM, in Brussels. She's been using a Linux PC since she was two, self-taught and self-motivating. I'd like to teach her to code, and perhaps make a profession in the software industry. But the chances she'll succeed at that are getting lower each year. I think there's a reason for the so-called "gender gap" but it's not any of the usual explanations.
10 Tips for an Awesome Open Source Project
In this article, ten top tips for making your open source project awesome! We've collected these tips by paying scientists to study the most awesome open source projects in the world. Whether you're starting a new open source project, or open sourcing your company's kind of old codebase, these tips will help you. Enjoy and retweet!
A Web Server in 30 Lines of C
There are developers who never built distributed systems. Then there are developers who have done this before, using a messaging system. Then there are those who've done it the hard way, using BSD sockets. Privately, I call these the Good, the Bad, and the Ugly, because of how they respond when they meet ØMQ and read the Guide. Today, "Getting Started with ZeroMQ for Uglies". With a "Hello, World" web server in —40— 30 lines of C, just for fun.
Licenses for Protocols
A few years back, when we set-up the Digital Standards Organization, one of our goals was to create a simpler framework for protocol development. At the time, licensing a protocol spec as open source was unheard of, and offensive to many people. Today it's just radical. In this article, I'll look at why and how to license a protocol spec.
Doing Stuff You're Bad at is Good For You
One of the tricks I use to not burn out on a project is to work as hard on learning new things as I do on doing what I already know. Last December I bought a piano and started teaching myself to play. The results… well, my daughter likes them and that's good enough for me.
Securing ZeroMQ: draft ZMTP v3.0 Protocol
In the previous article in this series, "Securing ZeroMQ", I showed a proof-of-concept for CurveZMQ. Now we're moving that into the ZeroMQ protocol, ZMTP. It's not a small change. You can't just sprinkle security over a protocol like chocolate chips onto pancakes. It means a new protocol, and this gives us a chance to address other problems with ZMTP. The result is ZMTP v3.0. In this article I'll explain ZMTP v3.0. If you never read the 1.0 or 2.0 spec, don't worry, I'll cover those briefly too.
The PUB-SUB Hacks: Census
In the first article on PUB-SUB hacks, we looked at the Meerkat pattern. In this second article I'll walk through a pattern—Census—that's loosely inspired by the "Surveyor" socket type from the defunct Crossroads fork of ZeroMQ. It's a fairly simple pattern. The code for this article is in a GitHub gist.
The PUB-SUB Hacks: Meerkat
While the ØMQ pub-sub pattern is well-known for being a one-way pattern, it does in fact support a two-way flow of information. SUB sockets (or more accurately, XSUB sockets) can send messages to XPUB sockets. This week, I'll post a series of short articles that cover different patterns—Meerkat, Census, and Repeater—we can build using this functionality. As usual, I'll demonstrate each pattern with code you can download and run.
Securing ZeroMQ: CurveZMQ protocol and implementation
This week, we flesh out the basics of our CurveCP-derived protocol, with an implementation in C that I'm calling CurveZMQ. In this article I'll explain what this simple but powerful security protocol looks like. The code here will already work over 0MQ sockets but our next stage is to move this into libzmq itself, for all sockets over tcp:// and ipc://. So stay tuned!
Securing ZeroMQ: the Sodium Library
Marc Falzon (@falzm) pointed me to libsodium, aka Sodium, a repackaging of the NaCl library. The Sodium README says that it's, "tested on a variety of compilers and operating systems, including Windows, iOS and Android," and "tests and benchmarks will be performed at run-time, so that the same binary package can still run everywhere." This fixes the biggest problem with NaCl, which is that it optimizes at compile-time, so you can't ship it in binaries. Today I tried Sodium, and here are the results of my tests.
Securing ZeroMQ: CurveCP and NaCl
One of the biggest user requests for ØMQ is a good security layer. Mainstream options like TLS/SSL are complex, slow and designed for web browsing, not high-speed messaging. In this article I'll present CurveCP, one of the most exciting security developments in recent years. It's part of the NaCl networking and cryptography library and looks perfectly suited to ØMQ. To demonstrate this, I've made a "Hello World" proof of concept that shows an authenticated, confidential connection from one client to one server, over ØMQ.
Making Music on Linux
I've always loved music, and have always been a terrible musician. Recently I found myself playing a cheap Casio keyboard and graduated to a rather nicer Kawai electronic piano. The Kawai has midi outputs, so as an experiment I've been recording some of my rambling piano compositions. I'll explain how I got these into a nice digital format using just a dead cat and a length of garden string.
Solving the Discovery Problem
One long-standing question for 0MQ developers is how to discover services on the network. A lot of people have built answers to this, such as ZeroConf and UPnP, or even DNS, but they tend to be over-complex and unfriendly to application developers. In this article I'll explain zbeacon, which is a new module in the CZMQ binding for 0MQ that does service discovery.
Patents Considered Evil
The concept of "intellectual property" dates from the French revolution, and patents in their modern form were solidified in the 1850s. At that time, Europe debated the pros and cons of patents exhaustively. The Economist newspaper was founded as an anti-patent, pro-free trade paper. Most familiar arguments for and against patents date to this era. The patent system was ended in several countries (Switzerland, Netherlands, Germany). However, free-trade economists lost power after a serious economic crisis, and the patent system came back into force by the 1870s. The patent system remained largely unquestioned until the 21st century, when it started to seriously affect the software industry.
Code Connected Volume 1
The title of this book for programmers, "Code Connected", is a homage to Steve McConnell's "Code Complete", which was easily one of the best books on software development ever written. Code Connected focuses on the basic skill of producing working code, in minimal form, to answer a wide diversity of problems. All the problems are related one way or another to distributed applications, and the answers all, in one way or another, use ZeroMQ.
Measuring Cost Gravity
The other day I bought a little black and white laser printer for my office. It cost about 50 Euro, and it prints very nicely, and rapidly. The first comparable consumer lasers came from HP in 1985 and cost about $4,000. They were huge, and slow. I wondered, could we use these two data points to compute the "cost gravity" of laser printers?
Burning Down the House: AMQP revisited
When the Wikipedia page for a contentious topic shows no real argument, you know someone with time and money has something to hide. And AMQP is certainly contentious. In my experience, and because I'm observant, rather than sensitive, the AMQP/1.0 authors are simple bullies who believe might makes right. But one learns to fight back against bullies, because, karma. I'm going to coin a nickname for AMQP/1.0, which is the "Burning Down The House" release.
How to Make Money from Open Source
There are, it has been said, two ways to make really large-scale software. Option One is to throw massive amounts of money and problems at empires of smart people, and hope that what emerges is not yet another career killer. If you're very lucky, and are building on lots of experience, and have kept your teams solid, and are not aiming for technical brilliance, and are furthermore incredibly lucky, it works.
The End of Stable Releases?
The C4 process we adopted some time back for ZeroMQ (except our dependency on Jira), CZMQ, and some other projects, looks like it's working as planned. But the drama of science lies in the extremes. If we really can reduce change latency to almost zero using C4, how does this affect how we deliver stable releases?
The Myth of Intelligent Design
The dominant theory of design is that you take smart, creative people and money, and produce amazing products. The smarter the people, the better the results. I'm going to claim that theory is bogus and based on a quasi-religious model of the "inventor" and "invention" as a function of individual minds. As an alternative I'll present the Theory of Heuristic Innovation, which states roughly that we do not invent solutions, we discover them, and that discovery process can be highly automated.
Git Branches Considered Harmful
One of git's great features is how easy it makes branches. Almost all git projects use branches, and the selection of the "best" branching strategy is like a rite of passage for an open source project. Vincent Driessen's git-flow is maybe the best known. It has 'base' branches (master, develop), 'feature' branches, 'release' branches, 'hotfix' branches, and 'support' branches. Many teams have adopted git-flow, which even has git extensions to support it. However, in this article I'll argue that public git branches are harmful, based on experience and evidence, and propose a branch-free approach, based on forks.
Good, Cheap, and Fast - PC3
The Pedantic Code Construction Contract (PC3) is an evolution of the GitHub Fork + Pull Model, and the ZeroMQ C4 process, aimed at providing an optimal collaboration model for commercial software projects. PC3 helps an organization build consistently good software, cheaply, and rapidly.
The Lazy Perfectionist and other Patterns
In this article I'm presenting a series of patterns for success in software engineering. These patterns aim to capture the essence of what divides glorious success from tragic failure. They were described as "religious maniacal dogma" by a manager, and "anything else would be fucking insane" by a colleague, in a single day. For me, they are science, the results of decades of trial by error. Treat the Lazy Perfectionist and others as tools to use, sharpen, and throw away if something better comes along.
Social Engineering 101
Social architecture is the discipline of designing and building large-scale, successful on-line communities. An underlying toolkit is the more focused skill of social engineering, or making friends and influencing people. It's often confused with social hacking but is quite different. In this article I'll explain the basics. As a case study I'll tell the story of how I talked myself into seat 2A in first class on United UA 973, Brussels to Chicago.
Diversity is Joy
If you're into community building, you will inevitably bump against individuals who cost you sleepless nights, who draw you into endless email threads debating the most inane details, and who generally turn what should be a pleasant win-win experience into a war of words. In this short article I'll explain how to deal with such people.
The Economics of Evil
In 2008 I'd been president of the FFII for two years and we worked hard to get people involved in the fight against software patents. We developed the rule of thumb that positive campaigns don't work. Every campaign needs a bad guy. So as I moved back to building open source communities, I wondered how this rule could translate to different kinds of community.
Testing Considered Evil
Here's a provocation: the more you test software, the worse it will be. To understand why, I need to explain how we (as a profession or industry) actually make good software. Very few people understand this, and we use techniques like unit tests for support, not illumination.
Credit-based Flow Control
Using network buffers - such as 0MQ's queues - has a disadvantage when you are sending data to many readers. Your writer will block when the buffer is full. To avoid this, you can use non-blocking writes, and poll for sockets that are 'ready for writing'. But here's an alternative that dispenses with high-water marks and blocking writes. It's called "credit-based flow control".
How to Recognise and Prevent Burnout
Any organisation or community that relies on pro-bono efforts from its members runs the risk of burnout. In this article I'll explain what causes burnout, how to recognise it, how to prevent it, and (if it happens) how to treat it. Disclaimer: I'm not a psychiatrist and this article is based on my own experiences of working in pro-bono contexts for the last 20 years, including free software projects, and NGOs such as the FFII.
MOPED - a Message-Oriented Pattern for Elastic Design
People learning to build distributed systems with 0MQ for the first time (there's a protocol joke in there somewhere) can make some awful mistakes. One cannot just convert design approaches like "let's make a structure of classes" into "let's make a structure of tasks". So, here's a simple design pattern, that I call MOPED, for building distributed systems. Surprisingly, MOPED works, and is fun.
NOM-1 grammar and notes
Time to get formal. Even the simplest contracts have to be robust against our charming but ineradicable stupidity. Perhaps the simplest formal language for an unprotocol is ABNF. So here is a first draft grammar for NOM-1, with notes. This page will morph into a real unprotocol draft spec.
First steps to NOM-1
NOM-1 is a lightweight messaging unprotocol for #zeromq over UDP. The name stands for "NOM-oriented messaging". There's a good reason for the cuteness. If I gave it a confidence-inspiring name like "Message-oriented Datagram Protocol", people might use it without further thought. The silly name will make life and public acceptance hard for NOM-1. So it's going to have work really hard to succeed.
How to Grow a Community
There are a few ways people define open source or free software. One is by the license: "yes, we can see the code". Two, is by process: "yes, anyone can contribute". Three is by community: "yes, it's built by all of us". For me, only the last measure counts, and steps one and two are milestones. But they're not sufficient, as many failed projects show. I'll present fifteen measures that I've extracted from years of trying (and sometimes managing) to build self-steering, sustainable communities. You tell me if I'm on the right track here or not.
Cheap and Nasty, two essential patterns
Unprotocol enlightenment comes one step at a time. While it can be tempting to try to use generic solutions, it's often wiser to divide problems into classes, each with an optimal solution. Let's look at Cheap and Nasty, two essential patterns for unprotocol design.
Clouds and Unprotocols
The future seems clear. Mobile applications, clouds, and unprotocols. My previous gig was CEO of Wikidot, which runs this site. It's a cloud. Cheap, scalable, instantly available to anyone who needs it. My next gig will be portable applications. Always available, smart, geolocated, social. Put the two together and you have something magic, as Google knows.
Using a RESTful Transport Layer
Couple of us are developing a new mobile concept, which I'll write about later somewhere else. The little thing talks to a web service, and for this we're using RestTL, an unprotocol that "specifies standard rules for representing resources, and standard mechanisms for working with them in a RESTful fashion over a plain HTTP client-server network."
Finally, the ØMQ wire level protocol!
ZMTP/1.0 defines the wire level protocol used by ØMQ (versions 2.x) over TCP. It has a framing layer, a connection layer, and a content layer. It's a neat little unprotocol that is free to remix, lightweight, and easy to understand.
Keywords for Unprotocols
Up to now, software protocols have been designed with levels of formality that tend to exclude facile participation from the fat belly of developers. tl;dr: protocols in suits bore the pants off most of us. We propose the concept of the "unprotocol", which is a lightweight spec that's easy to read, fast to implement, and obvious to get right.
Elegant Little Pieces
Unprotocols are elegant little pieces that we remix to create order and sanity in the swirling chaos that is the software industry. There are, IMO, three essential properties of an unprotocol. One, it's immune from capture. Two, it's lightweight. Three, it uses natural semantics. Said @leastfixedpoint: It's a mystery to me why AMQP isn't yet specified in little elegant pieces like this: http://rfc.zeromq.org/spec:13.