Hacking the Edges

pieterhpieterh wrote on 23 Sep 2014 17:34


On October 30 and 31, Marvin Arnold and myself will be organizing a hackathon in Brussels, with the goal: produce a prototype for a fully decentralized Internet. The idea came after Marvin, the guy behind Unplugged and I were talking about ways to kick off something together.

A while back I proposed building an "edgenet" built on ZeroMQ, Zyre, and other layers. We did a fundraiser that collected four times the amount I asked for (a token $1,000). Now we've been working hard on pieces of the puzzle (it is a large puzzle), and it feels like a good time to kick things up a notch.

I'm running, every month or so, courses in Brussels to teach our techniques of contract-driven distributed system development on top of ZeroMQ. So the plan is to take the first of these events and turn that into a hackathon for edgenet, which is a great example of a real distributed system that we could build on top of ZeroMQ.

We have two days, and most of the weekend that follows. There will be food, beer, WiFi, and if I can get my act together, t-shirts for everyone who takes part. This is not going to be a ZeroMQ tutorial. If you come hoping to get kind words and a gentle introduction into how to make a simple application, be warned! We're going to be working fast, focussing on writing and implementing RFCs, building code on GitHub, and interacting with others who can't be present.

Our goal is to get an end-to-end prototype running on Android. That is, a real app that can do real stuff. If we can't get the necessary Android SDK skills, our fallback plan is to run the network on Linux.

A new decentralized Internet is a vast challenge, so I'm going to cut down the problem to its absolute minimum. We need to solve three functional problems (comprising something like seventeen point four gazzilion technical problems):

1. How to exchange information (text and photos) anonymously between a disconnected group of people, on an arbitrarily large range of topics.
2. How to create authentic identities that can collect credit (be it bitcoin or karma), so that people have an economic incentive to participate.
3. How to send information directly from one such identity to another, across any distance, in full security.

This is kind of, "How do I rebuild Reddit when there is no Internet?"

The best answer I have at this stage is something like NNTP, modernised to run over ZeroMQ. If you know what that means, very good, come to the head of the class.

You can maybe see why I put the three problems in that order. We know that a global decentralized Internet will depend on billions of nodes to participate and forward data between each other. Only we have this mass of participants can we start doing person-to-person messaging. There are no existing global mesh networks. We do have to make one, as a first step.

If this is sounding insane yet weirdly compelling to you, please give yourself a pat on the back. You can use either hand.

The edgenet proposal — which though I like, I'm not attached to — is to create opportunistic "cells" around WiFi hotspots, and then treat these as dial-up links in the old NNTP model. As nodes see nodes, they exchange stuff. There's never any single consistent state. Every node has its view of the world. Perhaps there are nodes with larger memories. Perhaps not. Perhaps there are nodes who don't need to be anonymous, and who can happily route stuff directly from A to B. We'll see.

What tools do we already have? Let me list a few:

  • Marvin has pieces of a BLE (Bluetooth low energy) stack. This could be e.g. a future ZeroMQ transport (remember that ZeroMQ is basically a standardized queuing/routing API over arbitrary transports).
  • We have ZeroMQ, in C++, Java (JeroMQ), .NET (NetMQ), Erlang (ezmq), and C (libzmtp). These all talk the same ZMTP protocol. Only the C++ stack currently does encryption. Trevor Bernard has plans to add this to JeroMQ.
  • We have various discovery tools in CZMQ (the high level C binding for ZeroMQ), including UDP beaconing and zgossip. CZMQ is also wrapped in lots of languages such as Go and Ruby and Python and even C++ (though the smartest C++ developers are those who switched back to C, IMHO).
  • We have a protocol toolkit called zproto that generates protocol stacks in C and Java and Go, with minimal effort. This is like protobufs mixed with ASN.1, without the pain. These days we build every serious ZeroMQ protocol using zproto.
  • We have Zyre, an opportunistic clustering layer that uses CZMQ's discovery to find peers, and then connect them into an IRC-like group chat model. Zyre was always intended as the core of an edgenet cell, Since I began building it in 2012 (in C), it's become quite popular, wrapped in several languages, and also reimplemented in Java (jyre), Python (pyre), and others.

OK. What we do not have, and really need, is people with the skills to wrassle the Android SDK and to build simple and nice Android UIs. I once spent a few months of my life doing that, and it's not my sweet spot. I just don't have the patience for Java. Most of these projects have Android builds, yet when I plug in a phone and run adb, my heart sinks every time. Maybe it's just the post-traumatic effect of too much kimchi and soju.

So in Brussels, on 30 and 31 October (and on 1 and 2 November if needed) we will mash this together with new protocols and new stacks, and aim to get anonymous cat photos floating around a group of phones, tablets, and laptops.

Who is this for?

  • You already know ZeroMQ and you want to push your skill level up several notches.
  • You are passionate about a decentralized Internet and you have some experience with the problem.
  • You are into Android and have no fear of adb, the SDK, and perhaps even kimchi.
  • You know a thing or two about NNTP and can tell me why I'm totally wrong to think of using it.

Ideally, you are OK with C since most of the core work will happen in that language. We use a nice Python-like C style called CLASS.

You will have to know how git and github work, to be able to join in. We'll try to set-up a projector with an IRC channel, so that the outside world can chat to us, follow what we're doing, and join in via GitHub pull requests, as per our usual way of working together.

Instead of making this a paid workshop as was planned, it's going to be free for the first N people who register and who get past the strict entrance requirements. This means emailing me, telling me you loved my book (any book, it's all the same), and then tell me why you have to be there, with whatever arguments you want to add. If you get a seat, and you can't come, you will be free to pass this to someone else. N is variable. At some stage the free seats will be gone.

Alternatively, if you want to come as a paid participant, that is also great. The price will be EUR 1,200 per person for the two days, and the weekend if wanted. Meals will NOT be included in this event. (Mainly, we won't be taking breaks.) iMatix will collect such payments, and issue invoices.

Participants' names will be private though all the code we make will be public, so GitHub throw-aways are totally OK. All code we make will be MPLv2 licensed.

Questions? Comments? Let me know.


Add a New Comment
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License