Sometimes bad things happen. The question - how fast are you able to respond and put things back on track?
Technical teams need to plan and test their ability to rollback changes that are negatively impacting users. This needs to be a formal part of your process - sometimes that means falling forward, fixing the broken functionality in production; sometimes that means falling back and reverting to a known state where the application was functioning properly.
Let's be honest - none of us are perfect all of the time. Occasionally, we are going to miss something and move code into the production environment that ends up breaking things and making life miserable for the folks actually using the application.
What have you done to prepare for this moment?
- Does your implementation plan include steps to proactively monitor the system as changes are moved into production?
- Does your implementation plan allow time for you to look at patterns of application activity to validate that you are not seeing patterns that would indicate that something is broken within the application?
- Does your implementation plan include validation steps to ensure that the application is acting accordingly and that the new/changed functionality is working?
- Does your implementation plan identify risk points and fallback strategies?
- Have you tested your implementation plan?
- Have you tested your fallback plan?
Every environment is different - but we mostly face the same types of challenges. Breaking things that users expect to work is not what we want to do, but occasionally, it happens!
Yes, planning your implementations takes time. Yes, planning validation steps and inserting them into the overall implementation plan takes additional time. Yes, testing your implementation plan chews up more time. And, yes, testing your fallback plan takes even more time. That all being true, failing to do any of this means that your operating by the seat of your pants and your recovery time when things go bad will hurt you worse.
- When your application fails - the reputation with your customers takes a hit.
- The longer the outage increases the risk that your customers will walk away from you and begin to use a competing application?
- Will an outage impact your revenue stream?
Now, all that said, sometimes even with all of the best planning, you're going to get caught. It's bad enough when it's a web based application/service where you can make immediate changes or perform and immediate rollback. However, what if it's software that you're sending out to customers - that they install on their systems? How do you think Microsoft feels when they send out one of their regular patches only to find out that it causes the blue screen of death. This can affect millions of computers around the globe.
Not to hit Microsoft too hard - but, in April they released a patch to Windows 8.1 that was required before any other updates could be applied. This impacted retail customers as well as corporate customers and was not fixed until early May. I'm sure that they ran their standard quality processes against the code prior to the initial release in April - but something obviously got by the testers, and it wasn't a minor issue. It took time for Microsoft to hit the reset button, but in the meantime their customers were in an uproar.
I'm going to walk softly here because as a developer it could happen to me or the teams that I lead. I'm not trying to single out Microsoft - anyone that writes software has had to recover from a misstep when loading code into production or sending out a new release. What I'm advocating for is that you implement solid processes that minimize the risk to your customers, your team and your organization.
What are you doing to mitigate the risk of an implementation/software release going bad?
Tags: Development; Programming; Programming Languages; Change; Decisions; Decision Making; Project Management; SDLC; Lifecycle;
For more information on David L. Collison: LinkedIn Profile
I have been distracted with several items over the last few months and apologize for my absence. That said, let's dig into our discussion and see if we can do some sharing.
Sharing my thoughts on Microsoft ...
My early career was built on Microsoft. I graduated high school in the early 80's just as the IBM PC started to be sold. Yep, aging myself. Anyhow, after learning BASIC and Fortran on a DEC 1170 - I migrated to programming on an IBM PC using BASIC and then learning PASCAL and Assembler. Then I learned dBase III and Clipper - learning to program against databases. All on a PC with floppy drives - yep, no hard drives at that time, just dual floppies where you needed to swap disks to manage all of the data. None of this would have been possible without the OS that Microsoft built and licensed to IBM.
Then in marched the IBM compatibles - Compaq the most successful in the early days.
Not only was I programming, but I also ended up selling IBM and IBM compatible equipment. The company I worked for let me lug around a Compaq portable while I worked for them. I had access to Microsoft compilers for Basic and Assembler, and used Borlands Pascal. The company I worked for also had me teaching programming courses.
Along the way we transitioned to Windows and I learned how to program using MS Visual Basic. This became my bread and butter for quite some time - allowing 2 buddies and I to eventually open our own consulting firm. We built a healthy practice, hiring additional developers and working for several local and national companies.
I was able to enter the field of software development right as Microsoft was transitioning the industry. Moving us all from monolithic computers operating in the back room to a vision of a computer on everyone's desk. I knew the first time that I saw the IBM PC - that it was my future. I didn't look back - I wanted nothing to do with the mainframe class machines, I wanted to figure out how these small (by the standards at that time) boxes could be used in business. And oh how the world turns as I know manage development activity on mainframe class machines and Unix/Linux clusters.
For a long time - Microsoft owned the development community. They created tools that allowed us to transform the industry, thereby further fueling their growth.
Then Microsoft lost it's way - it became more focused on protecting it's cash cow and locking developers into their environment. Companies attempted to displace Microsoft and turn us onto their platform, but most failed. Then along came the browser and smartphones - to say Microsoft was not ready would be an understatement. They nearly lost the browser wars and only woke up at the last minute to use their monopoly power of the OS to win the browser war. The smartphone is a different story. Windows Phone is an also ran and I'm not sure Microsoft will ever have the chance to be a driving platform within the phone space. Oh, they'll stay in, but in my opinion, they won't truly compete with Google or Apple. They missed the boat.
On the desktop Win8 was a colossal fail - yes, they can brag about the numbers sold, but most companies are using the license terms to downgrade to Win7. I personally hate Win8 - it's a mindless combination of OS modes that make no sense on the desktop. Win8 may be pretty on a tablet and phone, but on the desktop it only irritates. Where I used to regularly recommend Windows equipped laptops and desktops to friends and family, I've now moved to pushing Apple's ecosystem. Many people only need an iPad, and for those that need more, the MacBook Air, MacBook Pro, iMacs and Mac mini's are the choice.
And this is a stunning development - ask my family and friends, for a long, long time I viewed Apple skeptically and always thought they were too expensive. Well, you may pay more for a Mac, but if that allows me to walk away from the problems that I experience in the Windows environment and mess that they call Win8, it's worth the money.
Where the Apple ecosystem just works, the Microsoft ecosystem confuses - mindlessly flipping between metro mode and desktop mode, sometimes for no apparent reason.
Of all the expert articles written about this farce of an OS and what Microsoft needs to do to fix it, I vote for Windows Red as documented by InfoWorld.
Can they come back, who knows? They'll never go away, they're too big and have too much money in the bank. They would really need to screw up to disappear, not saying it can't happen, but the chances are remote. They have become the new IBM - too big to fail.
All that taken into context, the best move they could have made was to ditch Balmer and replace him with Satya Nadella. I don't think Balmer would ever allow Office on the iPad, and while the package introduced by Nadella is handicapped, it's at least a start. Microsoft appears to be making some steps to stop the ship from sinking - I wish them well, but really wish they'd kill the FrankenOS on the desktop.
I know this has nothing to do with Project Management or specifically with software development, but I thought I needed to do a mind dump on this topic.
Tags: IBM; Microsoft; Windows 7; Windows 8; Win7; Win8; Change;
If you'd like more information on my background: LinkedIn Profile
