YOW 2011 - loose thoughts

Talks I attended at this year conference were focused on Agile, functional programming and low level performance. It turned out to be quite an interesting mix.

Marry Poppendieck talked about ways of supporting continuous design. One of the advices was to create an environment that facilities fair fights. To achieve this you need to hire good, diverse people. Their diversity let them solve complex problems because they can approach them from multiple angles. Once you have a team like that leave them alone and simply keep supporting them.

I haven’t learnt much new at Agile Architecture workshop delivered by Rebecca Wirfs-Brock but it was good to hear that I’m not some kind of weirdo that demands impossible :).

Simon Peyton Jones delivered 2 amazing and full of passion sessions about functional programming. I’ve done a bit of this type of programming before but I was surprised by Haskell type system and its ability to ensure lack of side effects at the compile time. On top of that on my way back to Sydney I had a good chat with Tony Morris who told me about a project where he used functional programming to create a composable data access layer. Composablity and strict control over side effects is enough to push Haskell to the top of my list of languages to play with :).

Inspired by Coderetreat Brisbane organized by Mark Ryall I’ve decided to use Conway’s Game of Life as a problem that I will try to solve every time I learn a new language. It’s worked for CoffeeScript and I hope it will work Haskell.

The end of the conference was filled by performance tips from .NET (Joel Pobar and Patrick Cooney) and Java (Martin Thompson) land. Both sessions emphasized that computation is cheap and memory access is slow. By simply keeping all required data in L1/L2 CPU cache you can cut the execution time by half. Obviously this was presented using a micro benchmark but still it is something that it’s worth keeping in mind.

Functional languages rely on immutable data structures to isolate processing and in this way control side effects. Unfortunately this means that a lot of memory is pushed around and this has a significant influence on the overall performance. Again, it’s all about trade-offs :).

Martin talked about “mechanical sympathy” which is boils down to having at least basic understanding of hardware your software runs on. Without it your software not only doesn’t take full advantage of the underlying hardware but often works against it which has severe impact on the overall performance. It’s one of those advices that most of us won’t use on daily basis as most our infrastructure is in the Cloud but it’s good to keep it in mind.  

We’ve been told multiple times that “free lunch is over” and CPUs are not getting any faster. Martin proved that this is not correct. It’s true that we are not getting more MHz, we are actually getting less MHz than we used to but CPUs get faster because every year engineers design better chips. The message Martin is trying to send is that we should prove/disprove assumptions based on evidence and not beliefs.

All in all it was another good conference and I will try to attend it next year.

Healthy team

I’ve had recently a few conversations about what constitutes a great software development team and I found myself using phrase ”healthy team” a lot. It’s a rather vague expression so the goal of this blog post is to explain what I mean by it.

People

A healthy team consists of people that share the same vision (the end goal of the project) and the same basic principles (high care factor, working as a team, not leaving broken build overnight, etc.). Even one person that doesn’t share them can cause a fair amount of frustration for the rest of the team and frustration is what needs to be avoided at any price as there is nothing that kills productivity faster.

Skills

In ideal world every team would have only senior members that are highly skilled and experienced. In real life this never happens and to be honest I don’t see any problem with having junior members on a team as long as they want to learn,  are capable of learning quick and there are not too many of them. If I had to pick I would say 1 junior for every 3 to 5 senior members is more or less the right ratio.

Software development is complex and it’s unreasonable to expect software engineers to be experts in every possible technical area. That’s why I believe every team needs to have at least one expert for each significant horizontal/cross cutting concern. Such a person would ensure consistency and correctness across all vertical features. One person can drive multiple horizontals. Sample horizontals: CSS, deployment, messaging, etc..

At the same time all team members, including experts, should work on vertical end to end features and ask experts for help when they struggle with a certain horizontal concern. In this way knowledge related to horizontals flows from experts to the rest of the team. On top of that every team member can influence a horizontal with experience he’s gained working on vertical features.

This approach takes initially more time than splitting the team by skills and let people work in isolation on what they know best. But there are a few very valuable advantages of this approach in the long run. Team is more resilient to departures because knowledge is shared and the quality of the final product is higher because the product is created based on feedback from both horizontals and verticals. Obviously, there are certain specialized areas of software engineering which might not work with this approach but it should be just fine for most of us.

Leadership

I’ve worked on a few projects where there was no team leader and this always lead to the same problem. Different team members had different visions of what we were trying to achieve and how we were supposed to achieve it. I know this might be a slippery slope and be against some of Agile principles but at this stage of my career I believe that every team should have a team leader to make sure everybody is going in the same direction and that the team avoids basic/fatal mistakes.

A good team leader is experienced, trusts other team members, treats them as partners, listens to them and always is more than happy to answer their questions but at the same time should be able to make the final call when it’s needed. And no, I’m not trying to promote absolute dictatorship as way of leading teams :).

Process

A healthy team is a team that is as independent as possible and is not hindered by the process it needs to follow. On top of that the team members should be able to use tools that make them most productive and they should feel that they are in the driver’s seat.

Size

I believe that a healthy team is a small team. Ideally 5 to 7 people but no more than 10. More than that and communication becomes challenging which slows down knowledge sharing and increases frustration.

Update: I've been asked a few times if 5-7 is the initial size of the team. The short answer is no. Ideally 2-3 people form the initial team and then bring more people when they are needed and team is ready to accommodate them. I didn't write 1-3 on purpose because I think that it's always good to have someone that you can bounce your ideas off.

 

The order of paragraphs matters but all of them deal with important issues and I wouldn’t ignore any of them lightly.

The blog post has been updated on 24/01/2012 with comments about the initial size of the team.

Tags: Software

Good process helps, bad process destroys

I hear very often that all a successful project needs is a great team. This is based on an assumption that no matter how bad the rest of the project is a good team always finds its way to successful ending. This is correct only when it comes to areas that the team has control over. Very often process is not one of them and a bad process causes great teams to become mediocre as they can’t operate in their usual way.

Reliable end to end tests with Selenium WebDriver

These are the slides and code from the talk I gave last Tuesday at the Sydney Alt.Net. If 2 links is to much of clicking for you then you can grab both slides and code from here as a zip archive.

Virtual desktops via Virtual Machines

I’ve got recently a very decent laptop (Lenovo W520, 16GB RAM) and one of the fastest SSDs on the market(Vertex 3 240GB). The machine is flying and basically does whatever I ask it to do in no time :).
I’ve always wanted to keep client projects, my personal projects and the rest (Outlook, pictures, etc) fully isolated. The main reason being the fact that every project requires some specific setup that I don’t want to keep forever and re-installing the whole machine after each piece of work doesn’t appeal to me. On top of that I don’t want to limit myself and don’t play with the latest technology just because it might break something on my system and I won’t be able to work.
So I’ve created 2 VMs. One for my personal stuff(VS, Ruby, Erlang, but no Outlook) and one for my current client. The idea is that I will create a separate VM for each new client. Another advantage of this approach is that if something happens to my laptop I can copy the VM to a different machine and be up and running within minutes. I backup all VMs twice a week to an external disk using CrashPlan. A nice side effect of using CrashPlan is that I get versioning so in case of emergency I don’t have to go all the way back to the initial state of the VM.
The VMs are obviously not as fast as the host with IO being the main bottleneck but they are fast enough so I can work comfortably. I can even run 2 VMs at the same time and they both start within seconds and are responsive enough so I can have fully functional VS open in each of them.
Have a look at screenshots from CrystalDiskMark to see the IO performance drop.

I started the whole experiment a week ago so we will see how it goes. Drop me a line if you know how to squeeze more performance out of VirtualBox.

Automated database deployments to AppHarbor

AppHarbor is trying to provide .NET developers with similar goodness that Heroku offers to their Ruby counterparts. At the moment deployment to AppHarbor has at least 6 distinct stages:

• Code gets pushed to AppHarbor
• Code gets compiled
• Unit tests get run
• Config transformations get applied
• Configuration Variables get applied
• Code gets actually deployed to a web server

At least one obvious stage is missing which is the database deployment. In .NET world there is nothing that would be even close to Ruby on Rails migrations but we can do something about that.
There is a small library called DbUp that simply makes sure that a given set of SQL scripts gets executed only once. This the ”up” part of RoR migrations. Now the question is when the database upgrade should be performed. There are basically 2 ways. Either before the new code gets deployed or after. I prefer the former option because as long as the database changes are non-breaking the app is fully functional at all times. If the database upgrade fails then the new version of the application won’t be deployed.
If the upgrade is done after the new version of the application is deployed then it needs to be done at the startup which prevents users from using the application for some period of time. What is more if the database upgrade fails then the new version of the application needs to be rolled back. In most cases the new code won’t work with the old database schema.
At the moment AppHarbor doesn’t have an explicit step when database upgrade can be performed. So to work around this problem I created a simple MsBuild target called UpgradeDatabaseInProduction that gets run when the application is successfully compiled. All the target does is extract production connection string from Web.Release.config and pass it to DbUp console app that performs the upgrade.

<target Name="UpgradeDatabaseInProduction"
        AfterTargets="MvcBuildViews"
        Condition="'$(Configuration)'=='Release'">
  <calltarget Targets="UpgradeDatabase" />
</Target>

<target Name="UpgradeDatabase">
  <xmlpeek XmlInputPath="$(MSBuildThisFileDirectory)Web.$(Configuration).config"
           Query="/configuration/connectionStrings/add[@name='database']/@connectionString">
    <output TaskParameter="Result" ItemName="ConnectionString" />
  </XmlPeek>
  <exec Command="$(MSBuildThisFileDirectory)..\libs\dbconsole-custom\DbUp.Console.exe -cs "@(ConnectionString)" -d $(MSBuildThisFileDirectory)SqlScripts" />
  
  <warning Text="Database upgraded" />
</Target>

This works only because AppHarbor build servers can talk  to AppHarbor database servers. To be honest it’s more of a hack than a proper solution :) but at least I can deploy the whole app (code + database changes) with single git push which is simply priceless.