Joel Tosi of VersionOne wrote a good article on testing myths.

He raises a good example for the age old excuse that “I don’t have time to write unit tests”.   The example provided was some code that processed a data feed/push with transformation.  In this type of scenario, it can take a substantial amount of time to complete a full test run.  If the job takes an hour or so to run, that time can add up as bugs are found and the code is troubleshooted and retested.  It would be far preferable to invest much of this time up front in creating and running automated unit tests which can simulate troublesome data issues and problems in a matter of seconds or minutes, in a more controlled and granular manner (than processing a full data feed, or even smaller simulated data feeds).

The fact that these unit tests can be fully automated in a Continuous Integration build as either unit tests or more robust integration tests, is even better.  This way, the unit test is generated once, and will be run every time a build is done (probably at least once a day, if not multiple times per day).

If developers are not spending time writing automated unit tests, it very well may cost more time through the course of the project.

I will be one of the first people to admit that I have a limited exposure to agile methodologies, including Scrum, as I feel I have so much more to learn and experience.  Since we started implementing Scrum a few years ago, and some other agile practices such as Continuous Integration and Test Driven Development, I have become more and more passionate around “agile”.  It just seems to make sense. Trust me, I am nowhere near the agile implementation mastery that I would like to be.  There is still much to be learned…

Team First

It can be disheartening when I see posts online from Scrum teams that absolutely hated the experience and want to go back to Waterfall.  How can that be?  Agreed, Scrum can be a difficult pill to swallow at first when one is very used to the ebb and flow of the waterfall, but I feel the core principles of the Agile Manifesto and the core principles of Scrum should make sense and eventually impassion the team. So why does this not happen?

I believe one of the main reasons this might be happening is because some of us are not doing a good job as Scrum Masters.  We need to do a better job selling agile to the teams (and implementing, but that is for another post).  Selling does not mean shoving down their throats, blinding following the Scrum rules.  It means put yourself in their shoes and see the change from their eyes.  More meetings?  Possibly.  More transparency?  Of course.  More productivity?  Let’s hope so.  Less risk? Definitely.  These are not bad things for the team, but they may not see it that way initially.  It could take months, or longer, for them to “get it”.  Unfortunately, some of them may never “get it”, or may never want to.

Once we understand the changes that are occurring, not just in process, but also in culture, we can do a better job communicating to the team.  Each team member may see things differently, and we should acknowledge that fact and see if we can find a common avenue to help address issues or concerns across the team.

These types of discussions and transition plans could be well beyond the skill level and experience of a new Scrum Master, so in that case a professional agile or Scrum coach may be necessary.

We must not forget that a large part of Scrum and agile is team empowerment, and valuing people over process.  If the team is resisting, then the process probably isn’t working, and may need adjustment.  Use common sense.  I strongly feel that if the Scrum Master protects and empowers the team to the best of their ability, success in a better process in the end should be inevitable.  It may not be Scrum by-the-book, but I think that is perfectly fine.

Selling Leadership

If you are doing Scrum, or another agile process/methodology, hopefully leadership has bought in and is fully committed to going agile.  Perhaps they have not.  Even if they have stated that they are committed, they may not fully understand what it means to go agile.  Helping to form cross-functional empowered teams, providing dedicated Product Owners, and embracing and leading the agile cultural change, are all things that leadership should be doing.

If leadership does not appear to be fully on board, perhaps the benefits of agile are not clear?  Perhaps we need to do a better job selling agile to leadership.  Agile best practices are in their best interest as well.  Motivated, well-equipped, cross-functional teams working on the highest value items to the business at all times?  Reduce waste in process; increase productivity; increase transparency and thus reduce delivery risk.  Who would not want that?  Continuous Integration is an easy sale, in my opinion.  There are no real negatives that could possibly come close to outweighing the automation and reduced waste and risk that would ensue.  Test Driven Development is possibly more difficult to sell, but I am convinced it can be done with supporting arguments and information.

Everything can, and should, be started in moderation.  That also helps reduce the risk of spending time on processes or procedures that may ultimately fail.  It also reduces the risk that team members will be caught up focusing on areas that are not providing the highest value to the business.

With leadership support, agile teams can do so much more.  It is our job to help leadership see the benefits of going agile, and to try to obtain that support.

Selling the Client

I think much of the same arguments above apply to communicating the benefits of agile to clients or our users.  Shorter feedback loops, more transparency, adaptation to change; I see all of these as positives from a client perspective.  There is definitely a lot of adjustment that will probably have to be made (doing small iterations means delivering smaller chucks of features to the client at a time, which is possibly quite different than what they are used to), but there is a reason for this (see adaption to change and shorter feedback loops).

So if a Scrum or agile transition is not going as well as planned, or preparations are being made for a future transition, keep in mind that we must all be good stewards of agile.  We must be able to sell agile, to a certain extent.

After that, it should continue to sell itself.

In my opinion, any organization that produces software, regardless if the software is to be sold or used internally, should be using Continuous Integration (CI) as a best practice to help improve the quality of their software, increase productivity and reduce risk.

This post does not aim to re-define CI, as there are plenty of other sources online that can give you the run-down, but it does aim to share my thoughts as to why it is so important.  Let me summarize CI at a high-level, and then break down the positive impact as I see it.

What Is Continuous Integration?

Good question, and an excellent place to start.  Continuous Integration, or “CI” for short, in the simplest form is a process which fully automates the compilation of your projects latest source code from source control.  There are some very important points in that sentence, so let me highlight them:

• fully automated
• compilation of latest source code
• from source control

These points are very important, but often seem to be missed at some point or another.  If developers are running build scripts manually (either from their machines, or on a server somewhere), that doesn’t qualify as CI.  If they are still doing builds manually (not even running scripts, ouch) from their machines, then keep reading….

The “fully automated” and “from source control” parts actually abstract out a few important concepts that I don’t want to skip over.  The concept is that developers should all be checking in their code as frequently as possible and practical for the project.  It is important that their code compiles against the latest source, but their entire feature or problem set they are working on may not have to be fully baked out.  The act of checking-in the code, will trigger a CI build.  This is where the “fully automated” part comes into play.  A proper CI server or tool will automatically check source control for any new changes.  When it detects that somebody has checked-in a change, it will get the latest code and force a build (compile) of all the latest source.  If the compile works, that’s great!  If the compile breaks for whatever reason, it should notify the developer who broke the build with their change, and optionally, other team members as well.

That’s the bare-bones CI process in a nutshell.  However, even that basic process can save a team from some major headaches down the line, especially for a geographically dispersed or outsourced team.

Improve Quality

A bare-bones CI process that simply compiles the latest source code may not seem to directly impact the quality of your software, but it can have some nice benefits that will indirectly improve your quality.  Since CI is fully automated, it will lay the foundation for a repeatable and consistent build process which will immediately notify the team where there are build issues, and what those issues are.  Having this type of stability in your build process will help improve the quality of your software indirectly.

In terms of having a direct impact on your software quality, a good CI process will lay the foundation for rolling in additional automated tasks into a daily or nightly build, such as running unit tests, static code analysis, security testing, building installers, deployment tasks and more.  Basically, any task that can be automated, should.  This will give your entire build process a much needed overhaul in terms of automation and consistency.  Having testing and analysis built into the build process is a really good way to start improving on the software quality.  These tools will not improve the quality themselves, but they will increase the transparency of potential issues to the project team, and help them locate and focus in on potential issues more proactively.

Improve Productivity

Using an automated build process should have a rather obvious direct correlation to team productivity.  If it is fully automated, then that means they are spending less time doing builds.  Correct!

Let me also highlight some other potential issues that can have dramatic impacts on team productivity.  Let me start with some questions.  Do you have a geographically disperse team?  How often do your developers check in their code?  Do they get the latest code and ensure their proposed changes will compile before they check those changes in?  Do they double-check that any new dependencies they are adding are also properly checked-in and working?

Any time a developer checks in their code without properly checking if their changes will negatively impact others on the team (i.e. they “broke” the build), there is the potential for some major downtime or troubleshooting while team members figure out why their build broke and how to best fix the problem.

When teams are spread out across the globe, or at least different timezones, this can become a bigger problem.  I have seen cases where a team member in the US checked in their latest code on a Friday night, shortly before leaving work.  On Sunday, team members in India needed the latest code to continue working, so they checked out the code and it didn’t compile!  The team member who “broke” the build had forgotten to ensure that all their files, including new dependencies, were checked-in and working before leaving the office.  Had this team had a proper CI build in place at that time, the developer (and team in our case) would have been notified of the broken build within minutes of having checked-in the code.  With any luck, the notification would have arrived before the team member left the office.

Now keep in mind that CI does not promote blindly checking in your code without performing other best practices (like ensuring the proposed code changes compile against the latest code, unit tests pass, etc).  However, a good CI process will help cover the gap for when these types of issues, or mistakes, do happen.

In the cases I mention above, a proper CI process will help improve productivity through a few means.  Since CI is automated, it only needs to be written once (and probably tweaked over time), which helps reduce the time team members are performing low-value tasks, such as building, running unit tests and analyzing the code.  In a few cases, CI can improve team productivity in the, hopefully rare, cases where a broken build would have cost the team hours of troubleshooting and fixing just to get back up and running.

Reduce Risk

Hopefully a few trends have become apparent above.  An automated and easily reproducible build process can help reduce risk, just in the fact that it can help eliminate potential human errors during a build process.  Having the team notified immediately as soon as a broken build is detected can also reduce delivery risk by helping to keep the team productive and the project on-track.  Building in additional steps into a nightly build, like automated unit testing, security testing, code analysis, documentation generation, installer generation, even deployment tasks (the list can go on), can help further reduce risk of potential undetected issues or human errors in each of those steps.

Summary

The more time that the team can focus on producing good, quality code, and spend less time manually running low-value tasks such as some of the above, means the team is able to produce value to the business.  And that is very important.

Best Practices

Here are some of the best practice that I encourage others to follow based on my experiences:

• Use source control and ensure the team is proactively checking-in code as frequently as possible
• Find a build server product that you like, and most importantly integrates with your source control product.  It should be easy to use and to configure/maintain.  There are quite a few CI servers out there, many of which are open-source and free – but just ensure you will be able to understand and easily configure the tool, as it will quickly become your best friend or worst enemy depending on how easy it is to use and the features it supports.  I do not claim to have a broad exposure to many CI servers, but I will post a future review on some build server tools I have used in the hopes that it will help somebody in their review.
• Keep different build configurations for each project:
  • CI Build Configuration (mandatory) – Keep it lean and mean.  Configure the build server get the latest code and compile full project on every check-in to source control.  Possibly run critical unit tests as long as they are fast.  Remember, the goal with this configuration is to notify the team of a broken build as soon as possible.  If possible, have the build server clean out the local source directory and dependencies before getting the latest code from source control.  This should help ensure a clean build every time.
  • Nightly Build Configuration (recommended) – Have this build on a regular schedule for the project.  This should perform in much the same manner as the CI Build Configuration (clean build), but you have the luxury of time now.  The primary objective is not to notify the team of a broken build as soon as possible, but to perform a full build and additional quality checks along the way.  Even deploying to a development or test environment may make sense depending on the project and the team’s needs.  Build servers will often let a team member manually trigger a build by clicking on a button or a link.  This means even if this build configuration is scheduled to occur in the middle of the night, the team can still use it to do a full build, run tests, scan the code and deploy, during the middle of the day if needed.
  • Weekly Build Configuration (optional) – This configuration was based on our needs around full static code analysis.  Full static code analysis can take a really long time, depending on the produce used, what it is scanning for, and the volume of source code it needs to analyze.  It can also put a heavy load on the build server, effectively locking it up for a lengthy period of time.  For us, it simply did not make sense to scan the code everyday, as it didn’t offer that much value to the team at this point.  Once a week seemed to give us the results we wanted, and did not task the build servers during the busy week days (when the build servers should be focused more on CI builds).
• Try to keep all project dependencies within a relative path under the project in source control.  In other words, try to refrain from referencing dependencies in the GAC (.Net) or on the local machine that the build server may not be able to easily retrieve from source control.  Build servers will typically support build agents, and may federate the build on any agent that is available.  Having dependencies required to be pre-installed, or pre-setup on each build agent in a real pain to maintain, and also complicates project deployment.  It is my opinion that having all dependencies stored in source control with the project is the best way to go.  This way the build server doesn’t require any special setup or steps to compile.  It simply gets the latest code (and dependencies) from source control and triggers the build.  This also allows the build agent to potentially purge the build directory and all dependencies before it gets the latest code – which can help ensure a clean build each and every time.  This also limits the potential impact of one project’s dependency from impacting another project, assuming they are sharing the same build agent (common to share build agents in our organization).
• Look for ways to further improve your build and deployment process.  Automate everything over time.
• Suggestion:  If cost is an issue, and virtual machines (VMs) are available in your organization, then use hosted VMs for the build server and build agents.  I find that in almost all cases, a properly configured VM will have no difficulty running CI builds quickly.  Full static code analysis can take much longer if the VM does not have sufficient resources.  VMs can be easier to setup, and/or clone, and can reduce cost across a medium or large deployment of build agents.  If you are using VMs, ensure you have enough disk space allocated for frequently building your selected projects.  In my case, having a dedicated logical drive for builds, at least 20-30 GB may be sufficient for a number of projects (again, it depends on the size of the projects, the logs and artifacts that get generated for each build, how often the CI builds will get triggered, how long you want to keep the build logs).  In our case, static code analysis consumes much more RAM, CPU and disk on the VMs, than just a regular CI build.  This is part of the reason for us doing it on the weekends during idle time.