Modern Software Development or Slow as Molasses

I’ve been looking at that issue for quite a while, and always tried to play nice and be quiet about this, but quite recently I’ve experienced the discrepancies that I am going to describe quite intimately, so I feel I can no longer hold back.

But lets start with a little introduction and motivation for the topic at hand. I am going to be talking about Software-Developers mostly, but some of the general ideas are probably relevant to all kinds of crafts and handiwork.

The tools we use are shaping the way we work. They also shape the results of our work, the shape how we divide our time between different tasks, they shape how we estimate the effort needed to finish a project. And they shape what kind of projects we approach, and what the size of the projects we approach is.

They shape how we think about our work, and about the world.

So I’ve been pondering the idea of getting into Android App Development for a while, and this week I put my plans into action and attended an Android Developer Meet, and got started with the first basic Android tutorials. I tried to approach this whole endeavor with an open mind, fully expecting that the first few steps might be a bit rocky, and that it can be frustrating when you are on unfamiliar ground as an experienced developer.

Never the less, with the helpful support from someone at the Meet, I managed to stumble my first few steps into the foreign fields of Android development.

Now as a bit of a back story, to say the least, I’ve been developing Software for quite a while, especially on the frontend and graphical user interface side. I started with Delphi and Pascal, moved on to C++ and I’ve used many of the different user interface libraries that are available for C++, like wxWidgets and Qt. Over the years though, I started to prefer a rather obscure and not so well known toolkit for my personal projects. It is called FLTK, or Fast and Light Tool-Kit.

Moving more into professional Development in the recent years though, I am currently working more often with the Qt toolkit.

But never the less, the next day after this Android Development Meet, I decided to give the good old FLTK another spin, just to see how it compares to what I learned about Android the other day.

I had to tinker a bit with the environment on my laptop to get started with the FLTK library again, but I managed to get it all going.

I typed up a little “Hello World” style basic program, compiled it, started it, fired up the included FLTK Form Designer “fluid” and added a few buttons and controls to my program, when I suddenly came to realize the horrors that I experienced the other day.

Compiling and starting a basic program based on FLTK on my laptop happens in an instant, compiling the project happens under 1 second, it is barely noticeable. Starting the program on my laptop, the window opens almost as soon as I press the enter key after I entered the name of the binary in the console.

The FLTK Designer is fast, it starts in an instant, you create a form, add elements, etc. at no point at all is there any waiting involved.

And slowly it dawned me, I tried to fend the thought off, but I could not help myself. I tried so hard to stay open minded about android, to not be the grumpy old man that says “In the olden days everything was better”.

I tried to come up with several excuses and defences, of course Android has way more capabilities, it has to support different devices, resolutions, layouts, it has a more complex rendering pipeline, using accelerated graphics for its display, and supporting transitions and animations, etc.

But how could I kid myself, watching the IDE spinning its wheels over some gradle script, looking at the massive boilerplate that was generated for even the most simple project, waiting for the designer to switch between the text- and the graphical view taking more than a few seconds. Waiting what felt like ages for the emulator to start and show me the results of my feeble efforts. And even if I did not need to restart the emulator for changes to become apparent, just having the updated app restarted took way longer than what I looked at in the case of my simple FLTK project.

There is just one dominant feeling inside of me, and that is, how can anyone accept these kind of tools, why is there no outrage about this massive timesink, how can there be such a large number of Android developers, and nobody dares to speak up for some sanity in the development tools.

And then I start to really think, maybe that is why so many Android Apps are the way they are, because the developers are stuck in molasses. There is no room in Android development to quickly whip up a crappy little experiment, and bring it live, there is not much room, for radical and quick experimentation, everything needs to be well thought out, planned, and then implemented.

It is a set of tools, that more or less forces the developer to adhere to a corporate style slow moving waterfall model of development, because quite simply, no other style of development is possible.

Even for my little Hello World style Android app, I started to fear the moment of having to switch to the designer view in Android Studio, which would take long enough to put me out of my flow, and anger my patience.

That is probably why paper sketching is so popular in App development, and why there is a proliferation of “Mockup” Tools, and Designer Tools, specialised towards App development, because apparently, it is not possible to “think” visually and sketch in the Android Studio Designer. It is only possible to realize a design that is already thought through.

Funny enough, I sometimes use an old and discontinued UI Designer called wxDesigner, to sketch and plan my user interfaces, before I implement them in the framework required for the job.

But it goes further than this, the tools we use change the way we think, and I think being confined to limited tools like the Android tools confine the way we think about ourselves as Programmers. And now I am no longer talking solely about the slowness, but also about the complexity of the environment, that is the number of concepts that need to be understood to implement even the most basic application.

One such source of complexity is for example the changes between the various platform versions of the android system, and the way that the platform implementers choose to expose the developers to those changes. To say the least, references to changes between android versions are all over the place in the documentation and everywhere. In my opinion this is unnecessary noise, and noise takes attention and attention is limited.

In fact I think this leads to a point where it is more or less necessary to be a full time Android developer to write Apps. It is not likely or easily possible, to be, lets say a scientist for example, and to develop a little Android App on the side to control some lab equipment, or to support a little experiment.

And I think that is bad, it is a choice that the developers of the platform are making. It is a choice that takes power away from developers, because it binds attention that could be used elsewhere. Of course you could argue that it gives developers job security, because making the system too complex to understand for outsiders, makes them hard to replace, but well, software should be used to make the world more interesting, and not necessarily more boring.

I feel that these kinds of tools give the developer the feeling of being a cog in the machine, you are suddenly “just” an implementer, implementing other peoples ideas, realizing designs that other people thought out, you can no longer think about problems to solve, but rather all your creative energy is bound in the problem of handling the platform. It becomes a purpose of its own.

And it goes the other way around too, suddenly if you have an idea or problem, you can no longer solve it by implementing an app yourself, when you are a “non” programmer. You need a programmer, because it is not possible to write a simple app yourself, while still keeping focused on your actual field of expertise.

Anyways to wrap up my little rant, I think doing simple things should be easy, and writing a simple app should be simple. And shortcomings should be complained about and should be fixed. Can’t possible be an acceptable situation that one of the most popular platforms in the world is a pain in the butt to develop for.

Great Crisis inspires Great Leadership

Have you ever participated in a great crisis? Probably the answer is no. Most of the people in control of the Western World as we know it, have never seen a great crisis. Or as Tyler Durden would say:

We have no Great War. No Great Depression. Our Great War’s a spiritual war… our Great Depression is our lives.

We and everyone else around is enjoys a comfortable sedentary lifestyle. Even if there is individual crisis and the associated trauma, it has been a long time that there has been a collective crisis. And why is this a challenge for us, and for the western civilization? Because great crisis is a good teacher, it requires and therefore develops great leadership.

Have you been to the Moon recently? No? And neither was anyone else. The last manned Moon landing was in 1972! About ten years before I was even born. At that time a publicly accessible Internet wasn’t even available yet. Still, this incredible feat of engineering was accomplished.

What was driving those people at those times, how did they get the capable leadership and vision to pull through such an unimaginable project? One of the driving forces was the cold war, the result of World War II, which inspired a sense of competition between superpowers. But World War II also allowed produced a vast population of veterans, pilots and officers. People that were very capable precisely because the had gone through such a great crisis. They had a sense of what was possible through conducted cooperation, and they learned to release their knowledge and energy for the good of humanity, and made the moon landing possible.

And now Civilization is at a crossroad again, we are slowly losing that practical knowledge, the world lacks people and understanding of the forces that drives it. Beneath the surface is a vast network of machines, structures and organizations, that are for a large part on autopilot operated be the ants and bees on its levers and buttons, but without any oversight or insight.

I like to compare this to the scene in Matrix 2, where Neo goes to see Councilor Harmann in the underbelly of the secret underground city. The look at large machinery, that keeps the city running, and the Councilor laments how he does not understand what exactly it is that they do, but that he knows they keep the system up.

So now we have this challenge, on the one hand, there is the desire to avoid such a gruesome experience like a worldwar again, but there is on the other hand a lack of human beings, that carry the experience of overcoming such a crisis. So how do you learn what can only be learned through crisis, and at the same time work to avoid such a crisis?

Well, that is a good question, but it is a question for another time, as I have to leave know.

But maybe, I will feel inspired to write a follow up, or a part 2.

-Richard

C++ Challenge: SimpleServer API

This week I’ve prepared a little challenge in C++ programming. The example is not extrodinarily complex, but not completely simple either. The purpose of this example is, to challenge myself a little to get to know boost::asio networking, and to contemplate about how simple APIs for prototyping could look.

So the challenge is to create a class, that allows prototyping simple network servers, by implementing a convenient base class. The interfaces and base classes are already layed out, including an example that send the reply “Hello” for any incoming connection.

The specification is a bit open, and is not so much concerned with corner cases. So using the simplest solution possible is probably the right thing to do.

Ok, here we go for the boilerplate: read on, I will add a few explanations below.

#include <string>
#include <functional>
#include <iostream>

class IConnection
{
public:
    typedef std::function<void(const std::string& str)> send_function;
    virtual void receive(const std::string& str) = 0;
    virtual ~IConnection() {}
    IConnection(send_function send_) :
        send(send_) {}
protected:
    send_function send;
};

class IConnectionFactory
{
public:
    virtual IConnection* create(IConnection::send_function send_) = 0;
    virtual ~IConnectionFactory() {}
};

class HelloConnection : public IConnection
{
    public:
        HelloConnection(IConnection::send_function send_) :
            IConnection(send_) {}
        void receive(const std::string &str) {
            std::cout << "Received: " << str << std::endl;
            std::cout << "Sending Reply: \"Hello\"" << std::endl;
            send("Hello");
        }
};

class HelloConnectionFactory : public IConnectionFactory
{
    public:
        IConnection* create(IConnection::send_function send_) {
            return new HelloConnection(send_);
        }
};

class SimpleServer // <-- IMPLEMENT THIS CLASS!
{
    public:
        SimpleServer(int portno, IConnectionFactory *connection_factory) {}
        void run() {}
};

int main(int argc, const char *argv[])
{
    SimpleServer server(1234, new HelloConnectionFactory);
    server.run();
    return 0;
}

So, it goes like this: IConnection is the virtual interface for implementing a connection. That is, for every new connection, a new instance of an IConnection implementation is spawned, that is bound to that connection. When the connection is closed, the instance is deleted.

The IConnection receives a function pointer as argument, so that it can send replies. You can see how this is supposed to work in the example: “HelloConnection”. That example simple prints the received string on stdout, and sends “Hello” back.

SimpleServer is the class that you need to implement, here you can have all the bookkeeping, and you are probably going to need some internal connection class, that handles  each socket created, etc.

Anyways, I will probably be implementing the SimpleServer myself this week, so I might be able to post a follow up with a solution.

But anyways, this task is simple enough, that a veteran C++ programmer should be able to do it. And it is definitely intended as a practice to work one ones programming skill, so go ahead and try it.

Post your solution in the comments, or send them via email: richard.spindler@gmail.com

If you send via mail, I will add your solution to a follow-up post. 🙂

Have fun,
Cheers
-Richard

Managing C++ libs with CMake ExternalProject and Git

Writing modular software, you might sooner or later come into the situation, that you have some functionality, that you want to share between different projects. Let’s say, to have built a couple of convenience classes and function that you want to reuse.

There are different ways to skin the cat, one of them might be using a dedicated C++ dependency manager like Biicode. Or a general package manager for your platform of choice, like apt-get on Debian and Ubuntu, or Homebrew on OSX.

Usually you will start with putting your module or library into a git repositry, if you haven’t done that already. For Code that I want publish publicly, I usually choose my github account.

Now for creating a homebrew package, or a debian package, you either need to create some project description, a bit of boilerplate, and need some platform like launchpad to publish packages. Or in the case of biicode, you need an additional account at another platform, and you have to install another tool.

However, there is another approach, that uses software that you probably already have, and that is CMake and Git.

CMake has a functionality called ExternalProject_add, which can use, to automatically check out a git repository, and build that as part of the normal build process.

Of course, the same functionality would be possible with git submodules, but honestly, I always found this part of git rather cumbersome, and unnecessarily complex, for what I want to do.

So, I’ve created two small examples, that show how to create a small library, and link to it through CMake ExternalProject_add:

The first one is the library: cmake_simple_lib

And the second one is the project that uses the library: cmake_simple_demo

I hope you find that useful.

Cheers
-Richard

The decline of the hierarchical Organization in Popular Culture

Being a fan of the popular Space Saga “Star Trek”, and especially of the “Next Generation” Enterprise with Captain Jean Luc Piccard, I cannot help but notice the stark discrepancy in tendency to currently popular Television Series.
Star Trek and the Star Fleet is largely based around a huge hierarchical military organization. The chain of command is clearly defined, the captain receives orders from Star Fleet, the first officer respects orders by the captain, and everyone else respects orders from the first officer and the captain and so on. But it does not stop there, there are further roles within the organization, that supplement the strict hierarchical order. There is the doctor and the counselor, both of which are below the captain, in the chain of command, but never the less can override the will of the captain as experts, should they decide that the captain is incapable of fulfilling his obligations because of a medical or emotional condition.

In a way, the structure is laid out almost like a game of chess, where different roles have different advantages and disadvantages that can override each other in complex scenarios, making strategic games possible, and interesting.
I regard Star Trek as a historical artifact, that allows us today to take a peak into a system of cultural thinking an expectations that has changed not so long ago. A TV-Series is in a way a reflection of the society that created it, and a reflection of the society that were its primary consumers.

And in comparing the series to todays popular television shows, I cannot help but notice quite a stark contrast. So I keep asking myself, what has changed. What has changed in the perception of society in the perception of writers and producers, as well as consumers that made such a change inevitable.

Of course, there are series that contain plots that are concerned with hierarchical structure, and military organizations, Game of Thrones being the most prominent one that comes to my mind. But the topic of this series is never to indulge in the intricacies of a hierarchical organization, but rather to wreck havoc in every possible way of backstabbing and killing leaders and moving hierarchies around.

This is in stark contrast to the Star Trek series, where the concept of the structure of Star Fleet is never questioned as such. It is accepted as a de facto institution. The game is about how to work within the confines of this structure, and seldom and only in far and in between edge cases to question Star Fleet structures as such. So that does happen, but it is the exception, rather than the rule.

And if a series is not about wrecking havoc and creating intense drama and violence, it goes of into the other end of the spectrum.

Take all of the popular sitcoms, like Friends, How I met your mother, Big Bang Theory, The office and probably some more. These Series feel more like a kindergarten of childish innocence. The characters trapped in a Hell of Douglas Copelandian Nihilism. Nothing is serious, everything is wrapped in layers of self-depreciating sarcasm and cynicism. The Brave new World, everything is fun and games, and nothing is serious or important. The topic is no longer about creating great things, and exploring unknown territories, but rather it is about cuddling up in a warm and safe nest, and exchange pleasantries with ones fellow inmates. Every screwup can be shrugged of with a joke, and every tragedy with an outbreak of over acting dramatization, often about not much more than spilled milk.

Contrast this with seriousness of uncomfortable conversations between the leading officers of the Enterprise. Where they spend time pondering how their next move will be perceived, and whether it is moral and ethical to proceed in a certain path. Whether it is a viable tradeoff to insult the feelings of an adversary, or even if stall or a bluff is an appropriate response. None of those Strategic meetings are taking place in modern TV.

Is it that people have changed, that desires and goals have changed, or maybe it is our perception. The idea that we are no longer seeing us as serious and important in the things we do. That in fact the seriousness in which Jean Luc Piccard and his mates acted were overacting devoid of any resemblance in the real world, where all is actually fun and games.

Well, that is probably for everyone to decide for him- or herself.

Group Ritual Design

What makes a good climbing group? In a way it is the process of working together as a group, that actually creates the group, or that contributes to group learning.

But the same is true for any good work group. People work together on a challenge and through the challenge they learn how to work together.

Designing these kind of experiences is usually difficult. It is easier and can happen in situations that are arranged in a play. Like for example a climbing garden, there is a predetermined Course, with constructed obstacles, that are clearly designed to be challenging, but not too difficult or dangerous. There is a guide, and there are safety lines installed, that are controlled and solid.

The experience of going through that obstacle course is similar of a challenging group experience, but because it is prepared and limited in time, it is safe for the participants.

Developing an unlimited challenge is more difficult, because first of all it needs to be decided whether it will take place within the professional part of the participants time, or within his leisure time. If it takes part in his professional time, some argument of economic viability needs to be constructed. And for the leisure time project, the activity needs to be at least attractive and stimulating enough, to be considered of value to the participants.

What makes the process attractive could be rewards in the form of respect and validation within the social group.

What is needed to get at least some group coherence is a form of interaction. Those interactions could happen during regular meetings, while practicing together, a play or an act. The interactions could also happen through phone or email, when the topic is more of an abstract matter.

There is also the question of how much time should be spend in interaction. For some kind of sports on an amateur level, one or two hours each day can be enough. In the workplace 8 hours is considered adequate, however, not all of this time is usually spent in interaction.

What is also important, is the have the right intensity and variation of interaction. When the goal of the group interaction is to learn or improve a skill, then intense practice is needed, but also eventually alone time for the participants, where they can reflect upon their progress, and then again collective gatherings of reflection might be helpful as well.

The meetings and gatherings are conducted is a matter of ritual, and ritual can either be accidental and informal, or structured and preplanned. Both are valid approaches, and both can fail or work in different ways.

Usually when it is a physical meeting, it happens in a room, and there is a date and time, when the gathering happens.

The room already provides a setting, there might be chairs, suggesting the meeting will be conducted sitting, the chairs might be arranged in a circle, or facing a conductor in the front. This already sets a mood. Or the chairs me be stacked in a corner, suggesting that the participants themselves should choose an arrangement. There might be a carpet and sitting-cushions, inviting the participants to take a seat on the floor.

There are very different modes of acquiring status within a group, in some groups the publishing of a written paper is an adequate form of expression. The paper has to conform to a certain writing style, and it has to fit into a template, that is set up for the occasion.

In other groups the mode of expression means taking part in a competition, pitching one member of a sports-team against a member of another team in a duel.

And all those group interactions and modes of operation are codified into some set of ritual, and as the popularity of groups or rituals spreads, rituals get copied and adopted. Like the Gene-Pool, they become part of the Meme-Pool.

Popular Memes multiplying, mutating, and unsuccessful ones disappearing.

How the Fossil DVCS solves the privacy problem of cloud based applications

If you do not know what a DVCS (Distributed Version Control System) is, just think of Git, the popular source code management system used by the Linux kernel.

But it is quite different from Git, and seems to solve a couple of problems, that always left me a bit unsatisfied when dealing with Git. In fact, it seems to give me the same nice and warm feeling I got from using the Darcs DVCS back in the days.

I will start out with a couple of issues I had when using git that I experienced recently.

1) For many developers today, when they say they are using Git, they are in fact using github.com a service that has Git in its core, but provides a number of additional services. A nice webinterface, a ticket tracker, a wiki and probably more that I am not yet aware of. However, while the product and concept is well executed, it is still a hosted solution, and while you are using it, you are dependent on an external company to provide this service.

2) I’ve been trying to replicate a similar service that github.com provides on my own machines, and experimented with software like indefero and phabricator, but those solutions leave quite a lot to be desired. InDefero for example provides both a wiki and a ticketing system, but now I have the following problems: I have to provide a MySQL Database for the wiki content, and the ticket system, that needs to be maintained, updated and backuped. Which is kind of a headache, especially considering the nice solution that Fossil is providing.

3) There are various variants of git based wikis that I’ve tried, but installing them and keeping them running is almost always a nontrivial issue.

4) Overall, using git without relying on 3rd parties to manage various aspects of the deployment, is almost impossible, or creates quite a lot of busy-work. I only quite realised that when I came across the ingenious solution that Fossil came up with.

So lets see what Fossil does that makes it so nice. It is quite simple:

Fossil stores Tickets and Wiki data within the repository. And not only that, it comes with an integrated web interface, to conveniently use this ticket system and wiki. And said web interface can be easily made available on a public website.

These choices enable a couple of convenient side effects. Whenever you have a checkout you automatically can easily use wiki and ticket system offline, and once you push to the shared repository everything is update again. Great for using it on the road without internet access. Also, makes keeping a backup of the complete project way easier.

Makes for a much more resilient and independent tool, way better suited for the lone hacker and his small crew. Hosting is trivial, you can setup a private server, or use any kind of virtual server hosting.

In fact, I like that concept so much, I am already thinking about what kind of webapps could be “unclouded” with such an approach as well. Using a distributed versioned database that keeps a complete local backup and only needs to resync periodically to the master host.

Maybe it would be possible to extract some of the Fossil infrastructure and bake it into a general web framework. Or build something similar for different kinds of webapplications.

Have fun,
Cheers
-Richard

A new way of thinking and risk assessment

If you haven’t started reading the ribbonfarm blog I strongly urge you to consider reading it, because I find it to be on of the most pleasurable blogs right now. Venkats writing is very precise, and he manages to explain even very difficult concepts clearly and easily understandable.

But he is not the only lonely writer to be working on these ideas. There seems to be quite a trend of going into systems thinking and finding ways to incorporate risk and probability into every day decision making.

"The Black Swan" by Nassim Nicholas Taleb started that discussion in popular literature, followed by "Antifragile", by the same author.

But also "The World Until Yesterday" by Jared Diamond devotes a chapter to discuss the topic of risk assesment. And then most recently, Scott Adams published his book: "How to Fail at Almost Everything and Still Win Big", which goes on to discuss the idea of systems thinking on a more practical level.

And to be frank, I feel that I can apply this kind of thinking at a lot of situations. It is as if everywhere I look systems thinking could improve something or indeed does improve things for me already.

Have fun,
Richard