pieterh wrote on 14 Nov 2014 11:16

The ZeroMQ community started many years ago as a group of low-latency fanatics who focused their "make it cheaper to build distributed systems" mission on a single library, libzmq. The mission hasn't changed, yet the focus is wider than ever. In this article I will talk about a young project, zproject, which makes it cheaper to build other projects that use ZeroMQ.

Pause, Rewind, Play

In 1999 I wrote an article for Dr. Dobb's Journal on "Template Based Code Generation". The PDF whitepaper says, "Template-based code generation is a powerful way to reduce the cost and effort of writing code. It's a technique that's underused by most developers, because there is little literature on this subject, and there are few decent code generation tools."

In 1999 I'd already been writing and using template based code generators for 15 years. Jonathan Schultz and I worked for about ten years to refine these techniques, ending in 2005 or so with GSL v4, which we used heavily to build the OpenAMQ messaging system (the first, and the fastest, AMQP server and client).

OpenAMQ taught us a few things, not all happy lessons. One lesson was especially harsh: meta hurts. In 2005 we abandoned our entire technology stack, code generators and all, believing the world was not ready for it, and never would be. When we started building ZeroMQ, it was in C++ (ugh) and by hand, with zero code generation (oh god why?!). We chucked GSL into hibernation and I stopped coding for a while, it was too painful to step back so far into the past.

It seemed, for years, that no-one understood the power and beauty of code generation. It was too different from the industry standard perception of how we make and value code. Code was, and mostly still is, seen as an asset rather than a liability. The truth is that in software design, as in any domain, all that matters are our models. Good models are simple, abstract, accurate, and tethered to reality. Bad models are fuzzy, complex, and based on falsehoods.

Here is a bad model: "Freedom is about having more flavors of ice-cream." Here is a good model: "Freedom is about doing interesting things with other people." The difference should be immediately obvious. If you're confused here, you need to review your priorities in life. A bad model confuses and traps us. A good model empowers us.

Code generation isn't about replacing the hapless coder with a robot. It's about teaching that coder to build models instead of code, to think in terms of simple, clean, accurate abstractions. That is, to learn to develop simple domain specific languages, with the tools to design and test and implement them cheaply and rapidly.

I used to call this "Model Oriented Programming", before shutting down our code generation machines and focusing instead on social architecture, aka "messing with people on a large scale, for their own benefit of course, cause otherwise that'd be Machiavellian and anti-social and even oh noes very profitable."

Well, code generation seems to be in fashion. Yay! Finally. GSL — the original and the best — found a new home on GitHub.

I'm happy to say that a slice of the ZeroMQ community has embraced model oriented code generation with a passion. About seven months ago I announced zproto, a tool for generating ZeroMQ servers, and clients.

While I like zproto and use it every day, the more interesting tool, because it solves far more horrid problems, is zproject.

Both zproto and zproject are reboots of old tools from the OpenAMQ toolkit (ASL and Boom, if you care, which you don't, as we both know :)

Since I've already talked about zproto, let me explain zproject.

The Problem with Makefiles

Once, long ago, a smart young person invented Make. He (for even in those days the technical edge of our field was very gender biased) had a chance to save the world. Or at least make it a much better place. Instead, he declared (according to the friend of a friend), "I realize the Makefile syntax is horrid, yet I already have seven users, so changing it is impossible!"

The rest is history. Every system has its own way of compiling and linking code. It's such fun to write a new project makefile specification that companies make one every year or two. Microsoft — for once I'm not joking when I mention their name — has different formats for Visual Studio 2008, 2010, 2012, and 2013.

Worse, the abstraction is largely useless. Sure, compiling and linking is one thing we want to do. However we also want to generate documentation. We want to build test frameworks, and run them each time the code changes. We want to target arbitrary platforms, today and in the future. We want to wrap our precious C APIs in other languages. We want to do endless things with our project, and above all, escape the limits and conditions set by our tool vendors.

This isn't hard to do. It just demands a higher level of abstraction.

Here is another good model:

<project
    name = "zyre"
    description = "an open-source framework for proximity-based P2P apps"
    script = "zproject.gsl"
    >
    <include filename = "license.xml" />
    <version major = "1" minor = "1" patch = "0" />
    <use project = "czmq" />
    <class name = "zyre" />
    <class name = "zyre_event" />
    <class name = "zre_msg" />
    <class name = "zyre_peer" private = "1" />
    <class name = "zyre_group" private = "1" />
    <class name = "zyre_node" private = "1" />
    <model name = "zre_msg" />
</project>

And here is a bad model for the same knowledge:

AC_PREREQ(2.61)
AC_INIT([zyre],[m4_esyscmd([./version.sh zyre])],[zeromq-dev@lists.zeromq.org])
AC_CONFIG_AUX_DIR(config)
AC_CONFIG_MACRO_DIR(config)
AC_CONFIG_HEADERS([src/platform.h])
AM_INIT_AUTOMAKE([subdir-objects tar-ustar foreign])
m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
PV_MAJOR=`echo $PACKAGE_VERSION | cut -d . -f 1`
PV_MINOR=`echo $PACKAGE_VERSION | cut -d . -f 2`
PV_PATCH=`echo $PACKAGE_VERSION | cut -d . -f 3`
AC_DEFINE_UNQUOTED([PACKAGE_VERSION_MAJOR],[$PV_MAJOR],
    [ZYRE major version])
AC_DEFINE_UNQUOTED([PACKAGE_VERSION_MINOR],[$PV_MINOR],
    [ZYRE minor version])
AC_DEFINE_UNQUOTED([PACKAGE_VERSION_PATCH],[$PV_PATCH],
    [ZYRE patchlevel])
AC_SUBST(PACKAGE_VERSION)
LTVER="2:0:1"
AC_SUBST(LTVER)
ZYRE_ORIG_CFLAGS="${CFLAGS:-none}"
AC_PROG_CC
AC_PROG_CC_C99
AM_PROG_CC_C_O
AC_LIBTOOL_WIN32_DLL
AC_PROG_LIBTOOL
AC_PROG_SED
AC_PROG_AWK
PKG_PROG_PKG_CONFIG
...

Which goes on forever, or 262 lines, whichever comes first. Learning autoconf is about as much fun as falling off your kick bike and smashing your face on cheap concrete pavement. Actually, having done both, I prefer the latter. Not that the alternatives are better. CMake, you say? Yeah, CMake is like smashing your face on high quality pavement. You can admire the stonework while the blood pours down your face.

Here's the equivalent CMakeLists.txt (again I truncate to reduce your empathic pain):

cmake_minimum_required(VERSION 2.8)
project(zyre)
enable_language(C)
enable_testing()
set(SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
set(BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
foreach(which MAJOR MINOR PATCH)
    file(STRINGS "${SOURCE_DIR}/include/zyre.h" ZYRE_blah blah
    string(REGEX MATCH "#define ZYRE_VERSION_${which} ([0-9_]+) blah
    if (NOT ZYRE_REGEX_MATCH)
        message(FATAL_ERROR "failed to parse ZYRE_VERSION_ blah blah
    endif()
    set(ZYRE_${which}_VERSION ${CMAKE_MATCH_1})
endforeach(which)
set(ZYRE_VERSION ${ZYRE_MAJOR_VERSION}.${ZYRE_MINOR_VERSION}. blah
include(CheckIncludeFile)
CHECK_INCLUDE_FILE("linux/wireless.h" HAVE_LINUX_WIRELESS_H)
CHECK_INCLUDE_FILE("net/if_media.h" HAVE_NET_IF_MEDIA_H)
include(CheckFunctionExists)
CHECK_FUNCTION_EXISTS("getifaddrs" HAVE_GETIFADDRS)
CHECK_FUNCTION_EXISTS("freeifaddrs" HAVE_FREEIFADDRS)
include(CheckIncludeFiles)
check_include_files("sys/socket.h;net/if.h" HAVE_NET_IF_H)
if (NOT HAVE_NET_IF_H)
    CHECK_INCLUDE_FILE("net/if.h" HAVE_NET_IF_H)
endif()
file(WRITE ${BINARY_DIR}/platform.h.in "
#cmakedefine HAVE_LINUX_WIRELESS_H
#cmakedefine HAVE_NET_IF_H
#cmakedefine HAVE_NET_IF_MEDIA_H
#cmakedefine HAVE_GETIFADDRS
#cmakedefine HAVE_FREEIFADDRS
")
configure_file(${BINARY_DIR}/platform.h.in ${BINARY_DIR}/platform.h)

Sorry, I removed some blank lines that were loitering around to pretend that is a "language" instead of a slowly rotting heap of fish heads and unwanted celery sticks. Did I see some bits of Perl 3.x in there? I'm sure some people write really pretty CMake scripts. Meh.

I've expressed my abundant affection for autoconf and CMake, which leaves about ten other random make systems to mock. Let me summarize my experience: all make systems are rubbish, and some are worse than others.

There's a book I want to write, on how to use C properly. By "properly" I mean as we do in CZMQ and Zyre and our other C projects. Actors, classes, neat abstractions for concurrency, and APIs, and useful packages of functionality, that are obvious and trouble-free and accurate, and properly anchored to reality. Perhaps this will be "Code Connected Volume 2" or perhaps a book on "Scalable C".

Anyhow, before zproject, I would have to spend a full chapter (don't mock it, one chapter is equal to fifteen Imperial weekends) simply explaining how to write a Makefile, or autoconf voodoo. What use are elegantly crafted examples if your readers cannot build and run them?

With zproject, we are starting to develop that abstraction and test it with real cases. zproject takes a project model, and turns it into makefiles and project files for MSVC (four flavors), Android, MinGW, QT, autoconf, Travis CI, and CMake. It generates a full project skeleton (in C) and maintains this for you as you add and remove classes in your project. It calls code generators, and builds man pages. It even builds bindings for your C libraries, in QML, and soon Ruby and other languages.

And we started zproject less than a month ago, on 20 October 2014. Kevin Sapper, Joe Eli McIlvain and Phillip E. Mienk have done most of the heavy work. It started out as part of CZMQ, then grew an independent life.

Using zproject

To use zproject, fork and clone the GitHub repo, then read the README. You will need to build and install GSL v4. You don't need to learn GSL to use zproject, though if you want to contribute, you do.

zproject is a community project, like most ZeroMQ projects, built using the C4.1 process, and licensed under MPL v2. It solves the Makefile problem really well. It is unashamedly for C, and more pointedly, for that modern C dialect we call CLASS. CLASS is the Minecraft of C: fun, easy, playful, mind-opening, and social.