Software Engineering: Workflow Pipeline

Building Software is difficult, it helps to have the best tools available though. One such Tool to have is an integrated development pipeline for your workflow though. Especially if you are planning to work as a team.
And since there are not really many “pre-made” tools, for this, you are probably going to need to create your own. Of course, depending on what kind of software you are working on, you will also need to choose the components your pipeline is based on.
Also, creating such a pipeline can be a bit of a chicken egg problem, because you might not to start working until you have the setup ready, but you are not going to know what setup you need, unless you start working on your problem. It might be easier though, if you’ve already done a few similar projects, and you already know what you need.
Choosing a tools for a pipeline might also depend on your budget, there are commercial tools available that integrate into certain development environments. But as I like to go “raw” in most of my development efforts, I am going to present you mostly with open source software, that is free to use, even for commercial projects.
I might not be a big fan of “over engineered” tools, but I am a big fan of mendaciously planned workflows and processes. And by process I mean everything that happens from the line of code entering the editor, to the click of a satisfied customer into the gui, or whatever it is that has been delivered. There needs to be a way from my line of code to get under the customers mouse cursor, that is working as smoothly and as reliably as a Swiss clockwork.
Many of those tools, the average software engineer or programmer probably is already familiar with, but I believe that is also of value to be aware that these tools and workflows, don’t just appear magically. They are conceived, built and it takes great care and determination to get them set up just right.
Also, I believe that especially “non technical” managers, if there is such a thing, need to have some idea about what is going on in software development. Startup founders that are building software based businesses also come to my mind here.
But lets get to the beef, the 3 basic building blocks of the set are, the source code repository, the build server, and eventually the test environment. The source code repository is where all the work of the team gets together. Here the integration happens, here the source code is backed up, should one of the developer workstations crash, and lose all its data. Here all the changes are recorded, here is where older states of the software could be restored.
The source code repository is also the key hole, through which the build server connects to the developers. Depending on what your development philosophy is, everything that you need to build the software should be in the repository. So that the build server can get it to build the software. When a developer wants to have something in the built, he needs to put it into the repo, so that the build server can get it from there. There should be no shortcuts, no tampering with the build server. No pressing buttons on the build server, no adjusting files, etc. once the process is in place.
The build server itself, should have a standard installation of the development environment you are using. However, in a way that a build can happen fully automated. If that means a few scripts here and there are needed, then be it so. Whatever your final product is, the build-server should be able to create it autonomously. The product could be a finished windows installer, for example. So then the build server creates that installer from the information he gets out of the repository.
The last role of the build server should then be to push the finished product into some kind of test-environment, be it whatever it is for your product. It could be a virtual machine if you have some rich client application, it could be a webserver, if your product is web based, or it could be some device, like tablet, smartphone, etc. if this is what you do. Could be in emulator of some kind too. Or an app-store, if you are publishing to your users directly from your workflow.
The point of this being is that whoever is testing your product, especially if the testers are not the same people as your developers, and they should not be, need to “sit” as close to development as possible. Without actually needing to disturb them constantly of course.
Source Code Repository
As for the source code repository, I do prefer git these days, although probably any distributed revision control system will do. With distributed meaning, that working with the repository does not require a permanent connection to the repository server, allowing the developers to work offline, which is generally convenient because local operations are faster, they can work offline, on the train, the plane, etc… also there is no single point of failure, such as the server being offline for maintenance, etc.
However, a word of warning, git has a ton of features, and you ain’t gonna need all of them. So don’t get too comfortable applying clever tricks. In certain developer cultures there is a tendency to advocate the use of branching for example. Others tend to shy away of using them. I am firmly in this camp. Of course, it all may boil down to a matter of judgement, and situation, bit in general, don’t use it. That is branching allows to keep several streams of development in parallel, which I believes fragments the development efforts unnecessarily, and I use other techniques to introduce “larger” changes, mostly on the sourcecode level, rather than the repository level.
In my process, the role of the source code repository is simple and limited:
  1. Bring developers code together
  2. Record history
  3. Feed to build server
There are some features that are convenient, like for example using a rebase pull to keep the history clean and avoid superfluous merge-markers, etc. But I again, I like to not get to clever about that.
As for branching in the source code repository, I mentioned using other techniques, one of them being modularising the code and have it communicate via fixed interfaces. This should allow it, if applied reasonable to implement even larger enhancements, in small increments, without confusing the other developers, as you can work mostly in your own newly created module.
That means however, that the core of the application has to be already relatively stable. So only few devs can work on that, but that is probably the nature of software development, and having branching in your version control system ain’t gonna change that anyways.
Build Server
As for the build server, I’d recommend to use the open source build server Jenkins, these days, but in a way it is just the scaffold for your actual build tools. It is java based, but can be used for any kind of environment, I use it for C++ for example. Eventually it will just trigger some scripts, to do what ever it is you have to do for a build of your software. If your process is already well developed, and you can do all your build and delivery steps via a script, you can already hook it into Jenkins, add a trigger based on commits to your repository, and have a go.
There are different philosophies about how to go about your build. Some people like to have everything from compilers to libraries, etc. in their repository, and basically keep all the build tools in version control too. I for one prefer a rather “vanilla” environment, based on a standard Linux distribution, with all dependencies out of the package system, with as little customization as possible. This makes it more convenient to what I do. If however you have specific requirements on your gcc version, a cross compiler, etc. because you do work for obscure embedded platforms, then the other philosophy might do you better.
Depending on your process, and the infrastructure you have, the build server might be a good place to run your unit tests, code metrics, static analyzers, etc. Jenkins has all kind of capabilities to record and archive your results. You might also be able to do a setup that pushes releases based on tags in your repository. That includes possible uploads to your homepage, an appstore, posting updates to shareware software news sites, beta areas of your user community, etc. If you are in anyways like me, you want to get updates out to your users as fast as possible, and this is your secret weapon, to spread as quickly as you can.
That is I am mostly talking about apps, if you are doing web development, well, then you probably need to adjust that recommendations to your specific needs.
If however you are in a more corporate context, you might not just push software into the wild, but rather into a test sandbox first. So this is what we do next.
Test Environment
This shouldn’t be to difficult, so, whatever kind if device it is you run on, set up an installation that is as close to vanilla as reasonable, and make sure, either the build server pushes regularly to it, or the device can fetch updates in a reasonable time frame. Preferable again, installation of your test environment should be automated and not require much tinkering.
The take-away here being that as I said, you want to have a tight feedback loop between testers/users and developers. And that feedback loop should be automated, because if it is not and setting up test devices requires developer intervention, then it is doomed to distract your development crew, and it is doomed to deteriorate. That is, bit rot will set in, people will complain, developers need to fix it, so then they cannot do their actual job, and things will deteriorate further, a deadly spiral.
However, if you have a large user community, you can also push updates to selected beta testers, and have them report back for example. The details of which is beyond this article though.