Software Environments: Separation and Configuration is Key

I just left a meeting with several folks from the server group, the web services group, and my applications group.  The discussion was around environments and promotion of software through these environments. Let me tell you, there are some very different ideas out there regarding this process.

I thought I would let you know how I prefer to make this work so that there is the best opportunity for a successful installation. Now these concepts are not specific to internal software development where the customer is part of the same organization as the group developing the software. These concepts are the same for shrink wrap or cloud based software.  I have worked with all of these types of software development projects and have been successful utilizing these concepts.

The first, and arguably one of the hardest to implement, is the concept of a separation of environments. The separation of environments from step to step should encompass all pieces of a software system wherever possible. This means that each environment should have, where applicable, a web services server, an application server, a database instance (server if possible) to name a few. 

At a minimum a team should have three such environments; a development environment, a test environment, and a production environment. Each containing a distinct and atomic system that requires gates to be passed in order to achieve entry.

These gates should include automated unit tests and code analysis. The code deployed in each environment should have been proven in the previous environment. The automated code that does the actual deployment to each environment should be exactly the same in each environment so you are always executing the same thing from environment to environment and not creating unnecessary variables.  This allows you to have the experience of many deployments prior to the one crucial deployment to production, thus mitigating and alleviating most risks of deployment.

One of the single most important aspects of achieving separation of environments is Configuration Management. Configuration Management is the management of all pieces of a software product’s lifecycle from initial coding through deployment. Configuration Management as a discipline in software development is getting more and more respect these days, for which I am very happy. This is a very challenging aspect to any software development project.  A properly maintained and automated configuration management system is a must for any organization that puts any product into production ands wants to do it as efficiently as possible.

It is the details of each system where a configuration management engineer earns his money. Each will be different based on the product, but the concepts are the same.  One of my favorite books Software Configuration Management Patterns deals with some of these concepts and how source control management is an integral part of the process.

No matter what system you are working on, try to keep your environments separated and remember: automate, automate, automate.

Till next time…

Technorati Tags: Configuration Management,SCM,release planning

ASP.Net – Code Behind and the Times

In 2001 Microsoft gave us ASP.Net.  The successor to what we now call “Classic ASP”. This brought the full power of the Visual Studio IDE to web development. Allowing us to write “Code Behind” in our favorite language, either Visual Basic or C#, for WebForms development inside Visual Studio.

This was a significant improvement to Classic ASP and to the developers ability to quickly produce web applications based on Microsoft Technology.  Up till then we had to use Visual Interdev. And was that a mess or what?

While I was an engineer at FM Global back in 2001, I was on the team that adopted .Net while it was still in beta.  (I know, pretty progressive for an insurance company, huh.) We had Microsoft consultants on site what seemed to be 24/7. We were learning a completely new way to code in a completely new IDE. Visual Studio .Net was simply amazing to all of us.  We were all Visual Basic developers then and being able to code using VB in this new environment was fantastic.

A couple of years later we were building an extranet application for our clients to be able to access their insurance information on the web.  We had been programming in .Net for a while but we were still VB programmers.  We were writing procedural code in an Object Oriented world.  There were some Code Behind methods that were several hundred lines of code long and full of spaghetti.

This is when we started learning about how VB can be a fully object oriented language and about utilizing Agile development techniques to help with the quality of our code.  It all started to come together. Our designs improved because we started to design our code to be testable. Our time to build and release methods were being revamped so we could be more efficient.  We were using code generation tools. It was a time of great learning.  We made our share of mistakes but all-in-all it was good.

Why do I reflect on such times?  I was reminded about these times because I am currently working with a group that is in a very similar situation.  Folks who are mostly COBOL programmers learning ASP.Net for the first time or are early in their object oriented programming learning and none of them have any Agile exposure at all.  I am sharing my experiences with them to help them grasp some of these concepts.

While doing a code review, one finds an asp button with an OnClick event that has 300+ lines of code in it, one is reminded of these times.  I am coaching my team in the craftsmanship of Agile Software Development, Object Oriented Programming and ASP.Net. So it behooves me to try to remember where I was and remind myself that there was a time when I wrote 300+ lines of code in a single method.  Well, maybe not that much but still…

Till next time…

Technorati Tags: Visual Studio,Agile,Object Oriend Programming,Microsoft,ASP.Net

Interesting CCNet behavior

We have a utility that generates our build scripts when it is passed a directory path as an argument.  The utility finds all the .sln files in the directory and its sub directories.  Then it interrogates the sln files to find out what the project dependencies' are and then generates a build script.  We use this utility in all of our CCNet projects.  CCNet calls the utility and then the next CCNet task calls the resultant batch script.

The engineers wanted to use this utility for their daily local builds so I created a single batch file that calls the utility based on a known Build location that contains all our sln files and then calls the batch files that were just generated.  Unfortunately, if one of the batch files failed it would exit out of the entire script with the exit code not allowing the engineer’s build to continue.  Now this is expected behavior given that none of the generated scripts’ EXIT statements have the /B argument. So I added it.

Everything seemed to be fine when I ran my first tests.  These were positive tests to make sure existing green CCNet projects would stay green.  But then on my negative tests something unexpected came to light.  My red build returned green.  I wanted to validate this was the case so I removed the /B from the batch script and ran the test again. The build turned red.

I am not sure why this is the case.  I looked through the CCNet doc but could not find the answer.  I am not sure if this is a CCNet issue or a CMD.exe issue (CCNet uses this when shelling out to batch files.)

My solution was to add an argument to our script building utility. The utility, with the new argument present, will generate the scripts with the /B on the EXIT command.  If the argument is not present then it does not include the /B.  I then added the new argument to the batch script for the engineers and now we are all happy. 

Well sort of. I still want to know why CCNet behaves this way. I created a thread on the ccnet-user google group discussion board.  I’ll update you on any responses i get.

till next time…

New Feature in CCNet 1.5

So I was looking at the CCNet documentation this morning because I needed to refresh my mind regarding some syntax.  As I was searching I found the Parallel Task feature.  This feature allows you to run several tasks at the same time as the name might indicate.  Well this is all well and good and I will use the Parallel Task feature.  But what was even more exciting to me (I get excited easily) was the Sequential Task feature.  I know, I know.  You say CCNet has always had sequential tasks.  In fact, they are built sequentially automatically.  Yes this is true.

But what I could never do is have CCNet move on to the next task if the first task failed.  Now I can.  See, the Sequential Task has an attribute associated with it called continueOnFailure.  This is a boolean attribute that if set to “true” allows the set of tasks inside the sequence to build even if the previous one failed.  I love it!

I love it because I am using a ccnet applet tool which I have written about in a previous post that allows me to forcebuild a CruiseControl.Net project from the command line.  This comes in very handy.  Well, I use this in all of our builds here at Emergent.  But sometimes because of the nature of our re-usable config files I want to force a build that may not exist yet on some build machines, say, if I were just testing it on one machine. So I put these forcebuild steps inside this sequential task with the continueOnFailure set to true and I am golden. 

I haven’t put this into practice yet because I just found out about it.  But, I’ll let you know if it doesn’t work.  I’m thinking it will.

Till next time…

The Evolution of a Build System Part 2

In my last post I talked about how Continuous Integration was where I started when faced with rebuilding an entire build process from scratch.  I talked about CruiseControl.Net and some of the challenges I faced at a high level.  Today I am going to talk about taking the Continuous Integration system and moving to a nightly Full Build system.

I began here:

Nightly Build and Test

• Only built portion of code base
• Built from the Tip
• VC80 and VC90 would build on alternate nights
• All platforms were built and tested serially
• Packaging was done as a separate process only when deemed necessary
• Close to 24 hour turn around

This is where we are now:

• Builds all solutions and all configurations of entire mainline
• Builds from latest Green CI Change-list #
• Builds all platforms and compilers in same night
• Build takes advantage of multi-core hardware and RAID drives by building concurrently where possible
• Build and Integrated Tests timing is down to 14hrs total
• Have deployable ISOs for all platforms prior to 10am daily

I built a couple of tools along the way and made use of my staff (thanks Scott and David) and our IT infrastructure and staff.  Being the IT Manager helped considerably.  The IT staff (thanks Jean and Luke) was extremely helpful in automating the process and continues to play a part. 

Here are some next steps:

• Automate the deployment to a QA environment
• Automate tests against the QA deployment
• Continue to bring down the build time

All of this was made possible because we were able to reproduce the build environment and standardize on a process.

Till next time…

The Evolution of a Build System Part 1

A couple of weeks ago I began setting up a new set of build servers for our west coast office.  I’ve pretty much got it setup so it is almost a push button operation.  Not quite there with all the DCC tools yet but almost.  We typically stand up one machine with the OS and run a single installation script which installs Visual Studio 2005, 2008 and the SPs.  It also installs Perforce, CruiseControl.Net and the latest console SDKs automatically. There are other minor tools the script installs as well.  I began to think about how far we had come. It wasn’t always like that…

This is a far cry from where we were when I started working at Emergent about a year and a half ago. Then there was a Twiki page that listed all of the various items required for each build machine. Your fingers ached after scrolling through the instructions.  Add this set of environment variables, make sure this path was added to the system path, install this tool at this path.  It was a very manually intensive process to setup a build machine.  Since the first thing I was challenged with was to streamline the build process, I had to have a reproducible build environment.

So, a combination of organic build scripts and manually setup machines made for a very unstable, and in my eyes, unreliable build system.  I had to first address the reproducibility of a build machine.  But where to start?  I knew we would be finishing a release soon.  Then I think it was Gamebryo 2.5. So I would start preparing for the next release cycle.

The build was taking close to 24 hours.  That was just too long.  Don’t get me wrong.  It was doing a lot of stuff.  It was building all console versions (Win32, PS3, Xbox360 and Wii) of the product and every solution configuration for each.  Then it was executing a rigorous automated testing suite.  So I chose to let that be for now.

Continuous Integration was where I started.  I had experience implementing CruiseControl.Net before so it seemed natural that I mentally moved to it.  My budget was not large so I had to make choices.  I reviewed ElectricCloud and AntHillPro. Both would have done what I wanted but where did I want to spend the little money I had?  I decided to forgo the expensive build systems and spend my money else where. 

I started small. I took a piece of the technology, the base framework, and created a CCNet project on my own box just to prove that I could do it with the C++ code. Then I incorporated each compiler, VS2005 using devenv, VS2008 using MSBuild, VSI from Sony for the PS3, and GCC and make for the Wii builds.  It took a while but the POC was working in a week.  I had some issues with MSBuild and C++ which I outline in a previous post so I decided to move to devenv for all the Win32 builds.  I will spare you the details further.

As my team was piecing the CI process and hardware together, we slowly started putting together a script that evolved through the process for build machine setup. I made sure my team knew I wanted there to be as little manual intervention to a new build setup as possible.  We re-factored existing scripts, combined them into a series of standard named configuration scripts in a well-known-location.  The team slowly began to change their way of thinking.  My mantra was as soon as you have to do something twice, evaluate and automate.  I tried to lead by example.  I engineered a lot of the initial setup and brought the team up to speed.

Continuous Integration as a culture was fairly new to Emergent product engineers.  Yes, they understood the concept and new it would help them be more efficient.  They understood the bringing feedback closer to the time of development was crucial to limiting the time it took to fix the issue.  They understood all of the reasons. But, it took a while for them to actually catch on to the whole “Red means dead” culture.  Finally, after a few months of CI in its full operation,  The engineers were on board and even worked with each other to promote the whole methodology. After all, I believe, these engineers to be the smartest people on the planet.

I was, indeed, pleased to see the fruits of my team’s labors playing out and helping the engineering team as a whole be more efficient.

In Part 2 I’ll talk about the transfiguration of a nightly build system “pieced together with duct tape”, to a streamline, reliable, reproducible build system and process.

Till next time…