Implementations - When Things Go Bump In The Night!

Sometimes bad things happen.  The question - how fast are you able to respond and put things back on track?

Technical teams need to plan and test their ability to rollback changes that are negatively impacting users.  This needs to be a formal part of your process - sometimes that means falling forward, fixing the broken functionality in production; sometimes that means falling back and reverting to a known state where the application was functioning properly.

Let's be honest - none of us are perfect all of the time.  Occasionally, we are going to miss something and move code into the production environment that ends up breaking things and making life miserable for the folks actually using the application.  

What have you done to prepare for this moment?

1. Does your implementation plan include steps to proactively monitor the system as changes are moved into production?
2. Does your implementation plan allow time for you to look at patterns of application activity to validate that you are not seeing patterns that would indicate that something is broken within the application?
3. Does your implementation plan include validation steps to ensure that the application is acting accordingly and that the new/changed functionality is working?
4. Does your implementation plan identify risk points and fallback strategies?
5. Have you tested your implementation plan?
6. Have you tested your fallback plan?

Every environment is different - but we mostly face the same types of challenges.  Breaking things that users expect to work is not what we want to do, but occasionally, it happens!

Yes, planning your implementations takes time.  Yes, planning validation steps and inserting them into the overall implementation plan takes additional time.  Yes, testing your implementation plan chews up more time.   And, yes, testing your fallback plan takes even more time.  That all being true, failing to do any of this means that your operating by the seat of your pants and your recovery time when things go bad will hurt you worse.

1. When your application fails - the reputation with your customers takes a hit.
2. The longer the outage increases the risk that your customers will walk away from you and begin to use a competing application?
3. Will an outage impact your revenue stream?

Now, all that said, sometimes even with all of the best planning, you're going to get caught.  It's bad enough when it's a web based application/service where you can make immediate changes or perform and immediate rollback.  However, what if it's software that you're sending out to customers - that they install on their systems?  How do you think Microsoft feels when they send out one of their regular patches only to find out that it causes the blue screen of death.  This can affect millions of computers around the globe.

Not to hit Microsoft too hard - but, in April they released a patch to Windows 8.1 that was required before any other updates could be applied.  This impacted retail customers as well as corporate customers and was not fixed until early May.  I'm sure that they ran their standard quality processes against the code prior to the initial release in April - but something obviously got by the testers, and it wasn't a minor issue.  It took time for Microsoft to hit the reset button, but in the meantime their customers were in an uproar.  

I'm going to walk softly here because as a developer it could happen to me or the teams that I lead.  I'm not trying to single out Microsoft - anyone that writes software has had to recover from a misstep when loading code into production or sending out a new release.  What I'm advocating for is that you implement solid processes that minimize the risk to your customers, your team and your organization.

What are you doing to mitigate the risk of an implementation/software release going bad?

Tags: Development; Programming; Programming Languages; Change; Decisions; Decision Making; Project Management; SDLC; Lifecycle;

For more information on David L. Collison: LinkedIn Profile

Languages, Languages, Languages ...

I speak regularly with High School and College level students promoting careers in technology - specifically careers with Applications Development, Quality Assurance and Project Management.  I enjoy interacting with the students and telling them what they may experience as they transition from the role of a student into the professional workforce.  Along the way, I also ask students to think about interning - either with my organization or some other organization in the area.  This is a fantastic opportunity for students to get real-world experience, become visible to an employer, and begin to understand some of the concepts taught in the classroom as they apply them to tasks in the workplace.

Many times, students ask me why we use the languages that we use and why we don't take advantage of the latest scripting or language fad.

My answer - why should I?  They seem to be stumped by my response.

Well, really, why should I?  Why should my team be disrupted on a regular basis just to use the latest language fad?  Why should I continually experience a productivity loss as team members shift between languages?  Why should I take the risk of bugs being introduced into production systems because the developers are not experienced with the language/tools that they are using?  Why should I risk customers being subjected to performance issues because the developers haven't figured out how to optimize the programs they're creating?  Why should I make the on-boarding experience for new developers more difficult?

Look, there are a handful of languages that we've committed to within our organization.  Four languages are used to generate all of our production level systems.  A couple of additional scripting languages have been used to create internal tools that assist us in production setup, as well as assisting us in ripping through logs to find specific information when we are researching issues in our environment.

Programming languages are programming languages - if you're a good programmer, you can pickup any language.  If you're a good developer, you understand that once you understand the various constructs of a language - learning a new language simply means translating the specific syntax of the language you want to learn to the languages you already understand.  It's translating how to use an if statement, a loop, a case statement, etc.  

Let's say it take you a month to become familiar with the new language - assume that your productivity drops by 50% and for 6 months after your initial learning period, you're running at only 75% productivity.  That means that over a 7 month period - you're productivity has cost the organization 8 weeks of development time.  Now multiply that across the entire team.  Assume their are 10 members on the team - that's a total of 80 weeks of lost development effort.  Assuming this same 10 member team and an average of 2 weeks vacation time for each team member - you have a total of 500 weeks that you can allocate to development efforts.  The end result is that you've stripped the team of 16% of the overall capacity of the team for the year.

Now, the numbers discussed above don't even take into account the impacts down the line as we move through test, implementation and post-implementation support.  How much more time will the Development Team need to create and execute unit tests?  How much time will the Development Team need to perform integration and regression testing?  How much time will the Development Team need to perform load testing?  How much extra time will the Quality Assurance Team need to test the application?  What additional issues will be experienced during the implementation of the code into production?  Can you quantify the number of additional issues that will be identified post-implementation that will need to be fixed and how much additional time it will take to address those issues?
 
The impact on productivity across the organization is significant and material - that is real time that prevents projects from being initiated or completed.  That's time that allows your competitors to either close the gap or get further ahead.

I don't mind taking on a new language when there is a need, or when there is a specific purpose.  Over the last couple of years, we've actually brought in specific tools that have allowed us to move in to the mobile market.  At this point, a select number of developers within the overall team have been exposed to these new tools.  Over time additional developers will be brought in to the loop.  Here, there is a specific need - our tools at the time did not lend themselves to creating cutting edge mobile apps.

Take a step back and look at the decisions being made within your organization.  There's usually a reason that organizations aren't moving as quickly from one language to another as developers would like.  To the developers that would like to get an organization to change - don't just go in and say that you need to be able to use a new language.  Document what you feel the benefits are, and just as importantly recognize the impact to the organization and craft your discussion points to address the costs that the organization will incur.


Tags: Development; Programming; Programming Languages; Change; Decisions; Decision Making;

For more information on David L. Collison: LinkedIn Profile