Don’t Be Afraid … Change!

Over the last several years, my teams have made significant changes to our project management, development, testing and implementation processes and paradigms. We’ve had some bumps along the way and in no way do I consider this evolution over. I will continue to challenge my team members to find a better way to build the mouse trap that drives our software development lifecycle.

Let’s look at some of the successes the teams have had introducing change into the organization:

1. Our project management process used to be split into separate and distinct teams: 1) project management for business and operational teams; 2) project management for technology teams. This has been consolidated into one project management team managing all project related activity across the organization. This has streamlined activity and improved communication across all teams involved in the process.
2. Our lifecycle was revamped as we consolidated the project management organization. During the revamp – we created RACI documentation that clearly outlined 1 owner for each key task/milestone within the overall project activity, along with those that would act in the roles of responsible, consultative and informed parties. This has provided clarity to who owns each activity and what the key tasks are that are included within every project.
3. Recently we made additional changes to the lifecycle – completely eliminating 1 phase and adjusting the tasks and milestones associated with both our scoping and design phases. This has allowed us to streamline the process and give clarity to when project activity actually starts within the organization.
4. We have challenged all of our team members to review the overall standard tasks/milestones within any given project, and challenge the project manager and sponsor to eliminate tasks and standard artifacts when it is justified and where there is minimal risk associated with utilizing an altered process.  This decision then is documented as part of the project and communicated to the team to ensure that down the road, people can review the effort and understand why certain documents or tasks might not have been done. This should allow us to be more nimble and find the best path for each project to take thru the organization.
5. A couple of years ago, it took up to 3 weeks to complete regression testing associated with key functionality within our application silos. This was a manual effort that typically was handled by multiple individuals. The teams have worked hard to automate the regression tests associated with these applications. Today, the execution of these tests can be done in less than 17 hrs. The labor savings moving forward will more than pay for this effort. This also allows us to provide quicker feedback to the development teams and ensures that our regression testing is more consistent across all projects.
6. Recently, the teams also completed the resurrection of standardized test data to be used across our test and development environments. This will be further enhanced as our automated tests are enhanced to create/modify/cleanse and reset data needed for a given test. This will reduce the time spent creating test data for each project and will ensure downstream processes/applications have the data that is needed to execute their regression tests.
7. Over the last several years, our implementation plans have seen significant improvement. Our business demands that our systems be available 24x7x365. When you want to buy a tank of gas at 3AM or 3PM, you want the transaction to go through so that you can continue on your way. You would not find it acceptable for us to deny the transaction because we are updating our systems.  Our implementation processes have been templated and include all implementation activity, across all application and infrastructure silos, commands, scripts, validation and fallback steps in the order they will be executed.  Recently, our external auditors informed me that our implementation plans are among the best that they have ever reviewed. This has hardened our implementation activity – reducing the overall time needed to implement changes into our production environment, standardizing the validation activity during and after implementation activity has occurred and most importantly having fallback steps at the ready when any validation activity shows that there may be an issue.
8. Our development teams have changed their processes to include code reviews of all changes prior to those changes being introduced into our production environment. This is hardening our code base and reducing the risk of making changes within our production environment.
9. Our development teams have changed their processes to ensure that unit tests are created for all code being touched within a project.  These unit tests are then kept in a repository and used in future projects. This, again, has hardened our applications and reduced the overall number of bugs slipping through the process into our quality assurance testing or into our production environment.
10. Our development teams are now doing nightly builds – with reports going back to each team on build failures, unit test execution failures, code coverage and other statistics. This is giving immediate feedback to our development team and ensures that issues are addressed within 24 hours.

These are just some of the changes that have been implemented by the teams over the last several years. And we’re not done! Moving forward, we will move the regression testing into our nightly build process. We will be evaluating further changes to our development lifecycle to reduce the amount of time between idea generation and product introduction into our production environment. We will continue to improve upon our implantation planning. We will introduce key metrics across the development lifecycle. And where it makes sense, we will introduce Agile techniques into the development process.

I talk to college and high school students on a regular basis to discuss what it means to be in the technology field (specifically software development). The one constant that I tell students is that being in this field means you will constantly have to learn. They will learn new languages, they will drive change into the organization, they will experience multiple different development paradigms. All of this is good – it means the organization is alive, is learning and is adapting to its competition. 

Don’t settle for how it works today or with someone telling you, “well, that’s the way we’ve done it for decades!” Your job, no matter what role you play in the organization, is to find a better way, review your suggestion with those that can help implement, trial those changes and then roll them out across the team and the organization. Change is good – don’t settle for the status quo! 

If you'd like more information on my background: LinkedIn Profile

Test Driven Development - Think about it!

Over the last few years I've been moving my teams towards Test Driven Development.  If you're not there, you need to be looking at this paradigm.  In essence, the following is what you are shooting for:

1. Write your test.
2. Run your test - the test should fail because the code hasn't been written yet.
3. Create the bare minimum of code that will allow the test to pass.
4. Continue to refactor the code until you are satisfied - is it simple, have you removed any duplication?
5. Repeat - accumulating additional tests until full functionality matches the agreed upon requirements and design criteria.

I've been on this push for a couple of different reasons: 1) The QA Team is spending an inordinate amount of time performing regression tests that should be in the build process; 2) The QA Teams and Development Teams should spend most of their time on new feature/new function testing - not regression testing; 3)  Within the old paradigm that we were following, by the time the development team member became aware of a defect in the code, they had long since moved on to other parts of the code and it was not fresh on the mind.

I've recently spoken with several groups of students - some high school and some at the collegiate level - and the one common theme that I have been impressing on all of these groups is that they need to think about testing first.  That means while doing the design work and prior to any code being written, they should understand how they will test the code and then execute against the plan.

In a separate session. when meeting with some of the instructors - another individual recommended that the instructors should have a base set of tests created that the students will need to execute against their assignments.  The students could do this at any time to get immediate feedback.  If an instructor were to pursue this paradigm within their classrooms, I would encourage them to create a portion of the tests, but force the students to create their own set of tests.

Unless our teams are working on a completely new system - there should exist somewhere in the organization a base set of tests that cover the current functionality.  It may not be extensive, but it's there somewhere.  That's the baseline!  No build of the system should happen without improvements to that baseline with your development team increasing the code coverage with additional unit tests that are then included in the set of regression tests the next time that code is touched.

Let's face it, immediately after the code has been written and the developers focus shifts on to the next task, their ability to maintain the code begins to drift.  I'm not saying they can't maintain it, but that they will need to spend time re familiarizing themselves with the code and then planning how to make the change.  Additionally, you can't guarantee that the next time the code needs to be touched it will be touched by the same developer.  By having these tests complete and in the pool of regression tests, you set an expectation with the developer that they can't claim they are complete with the code until all regression tests are complete and they can prove that they have tested against the new feature/functionality of the application.

Really, when it comes down to it, it's about accountability within the development team.  They are just as responsible for the quality of the code as are the people working the front end of the project - discovery and requirements, design - and those working the back end of the project quality assurance and the user representatives.

If you'd like more information on my background: LinkedIn Profile

Project Managers: Tactical ways to Push Testing

So, in my previous post, I talked about some of the high level things that you, as the Project Manager, need to be thinking about, as it relates to testing, when managing your projects.  In this post, I'm going to step down in to the tactical.  Specific things that you can begin to push within your projects and my recommendations for who should shoulder the responsibility.  These recommendations are based off of things that I've seen work in my current environment as well as things that I've seen work in various other organizations.

Now, before we dive in, a word of caution.  You need to evaluate what I'm saying and throw it up against the reality of the environment within which you work.  Some of what I say will work, some of what I say would probably spin heads within your organization.  Take what works for you and throw out the rest.  We're currently reevaluating some of the test strategies that are used within my teams and six months from now, I may have a different take on the subject.  My biggest concern is ensuring that we put enough thought in to the process to allow less experienced team members to be successful when working with unfamiliar subsystems or product features.

And, with that said, it's time to dig into the details and talk about things that I've seen work - note for those readers willing, I'd love to get feedback from others on what they are seeing work/not work in their environments.

First up, what are your end users (or the team members representing the end users) doing to drive the quality initiative?  These are the folks that are creating the user stories or requirements for the products that you're building.  Are they being held accountable at the start of the process to define what success means?  Are they telling you up front what they will do when performing user acceptance testing?  If not, it's time to sit down and have a conversation.  Why do they feel comfortable telling you what to build if they can't tell you how they will know it is right?  Specifically, what are the conditions and scenarios that they will use to validate that the functionality delivered is acting properly and producing the correct results?

1. What set of information needs to exist within the system to allow them to create the scenarios that they will use to test the requested functionality?  What specific elements of data will be used in the testing - both from a positive and negative test perspective?
2. What business logic is being used to assess whether the outcome is valid?  What will change within current processes?
3. Ultimately, what does success mean to them?  How will they know that what you've delivered works or doesn't work?

Now, ask yourself, if these do not exist prior to the design and development of the solution, how will the team know that the proposed solution will actually solve the problem?  This function must be driven by the end users (or those team members representing the end users).  They can collaborate with others on the team, including business analysts, technical team members and dedicated testers.  But, ownership of the acceptance test plan must fall on the shoulders of the end users.

One of the things that I like to do once the users have defined their needs and the design phase is underway, is to have the technical subject matter experts (SME's) begin to identify high level integration test objectives.  This is built at the same time that the technical design is being worked and is a very brief document outlining the gotchas that the development teams need to be aware of when developing and exercising the new systems/features.  This is intentionally kept very high level - usually just descriptions of each item that will need to be fleshed out in later stages.

1. Identify touch points between various sub-systems that will need to be tested.
2. Identify data replication strategies that will be used and that will need to be tested.
3. Identify data transformation strategies that will be used and that will need to be tested.
4. Identify services that will be relied on that will need to be tested.

This acts as a roadmap for the entire technical team.  It should be a roadmap used to identify data validation and functional validation.  It can be used by those that may be unfamiliar with specific subsystems to gain a further understanding of the interrelationships between the various subsystems and external/internal services.  The IT SME's own this high level integration test plan and pass it along to others in the process.

As we dive down in to the detail design - several things begin to happen.  First, the development team begins to flesh out the high level integration test strategy and it becomes the Integration Test Plan.  This document details the specific actions that will be used to test the touch points previously identified and should include the data and functional validation - data should be traceable throughout the system with the end points being validated.  This includes identifying the specific data elements that will be used to perform the tests as well as the expected positive/negative results.  While driven by individual development team members, the plan will be reviewed and validated by the Technical Leads.

Additionally, the individual development team members begin to identify their individual Unit Test strategies.  Each functional unit of code being created or modified needs to be tested to ensure that it is performing to spec prior to moving in to integration testing.  I believe in the paradigm that the tests should actually be created prior to the code being written, but I've seen development teams be successful using several different paradigms - so I won't harp on it.  The individual developers own this and are responsible for ensuring that they develop clean code.

Once the developers have been able to execute their individual Unit Tests, then they work together and move in to pre-Integration Test planning.  This time is spent ensuring that the necessary data has been identified and created/setup, that the configuration changes needed to the system and supporting sub-systems have been applied, that the network configuration changes needed to open ports to talk to internal/external services have been applied, and that all of this has been documented for production.  This particular block of activity is owned by the Development Team and the Technical Leads.  Infrastructure Teams may support the activity, but the Development Team and Technical Leads should ensure the configuration is complete and documented for the Implementation Plan.

Now we step in to Integration Testing - executing the plan that was documented earlier.  It is inevitable that something will have been missed - but that is what this stage is for, to find the bumps before you move your changes into the quality assurance (QA) and production environment.  Here you will find the missing data elements or calls to services/api's that work differently than documented.  You will uncover the fact that replication strategy for data you rely on wasn't documented properly and the underlying data you were expecting to see doesn't exist.  All sorts of unknowns will surface and you'll spend time playing whack-a-mole to clean it all up to a point where you believe it is functional.  The Development Team and the Tech Leads own this piece of the puzzle and work to ensure that all of the individual pieces come together and work as a whole. 

Now, let's take a few minutes to talk about Regression Testing.  I'm actually in the midst of transitioning our teams form the old style of moving code in to QA and then running a full regression test of the code along with new function/feature testing to a model where the code is built nightly and then the suite of regression tests is run when the build has finished.  This has been a tectonic shift for the teams in that the developers are getting daily feedback on code that has been checked in to the repository and we actually are building up a set of statistics that show us the overall state of the code.  I can now walk in on a daily basis and see the various package builds, number of tests being executed and the success/failure rate - along with a whole suite of statistics on coverage and severity of the failures.  Benefit number one - immediate feedback to the development team while the code is still fresh in their minds.  If we need to change something due to a failing test, the developer doesn't need to spend time remembering what changes were made.  Benefit number two - reduction of effort by our QA Team Members to get code through the testing process - their focus is on new feature/function testing, not regression testing.  Benefit number three - the developers have more buy-in on the testing process and begin to have a better understanding of the underlying data.  The Development Team is responsible for the Regression Testing and must review the results with QA to validate that the code is ready to move in to QA Testing.

QA Testing is focused on the new features/functionality to ensure that they are operating within defined constraints.  Here is where I want to hit the system hard with positive and negative testing associated with the new features/functionality being introduced within the overall system.  What happens where certain pieces of data are missing from messages exchanged between subsystems?  What happens when a user forgets to input data?  What happens when a web service that we are relying on isn't available?  What happens when the system thinks everything processed fine?  I will also want to take a look at load testing on some system to ensure that we are staying within expected SLA's.  This testing is the responsibility of the QA Team with input from the Development Team, Tech Leads and support from the Infrastructure Teams.

As  part of the work that is being done during QA Testing - we should also work with those team members representing the end users to execute the User Acceptance Tests (UATs).  This should tie all the way back to the beginning stages of the project where the business teams defined the success criteria of the effort and began building the definition of their UATs.  During the entire project, these high level UATs should have been fleshed out and reviewed to ensure that they align with the final requirements of the project.  QA and the User Teams should jointly execute and review the results of the associated UATs and prepare to sign off and give final authorization to move the code in to production.

As a Project Manager, you need to assign tasks to cover these activities within your overall project plan.  More importantly, you need to be following up with the responsible parties at various stages of the lifecycle to ensure that they are not being ignored.  Your organization may hold different team members responsible for the specific activities outlined.  They may also skip some of the items outlined above.  However, you need to ensure you are covering enough of the pieces to confirm that the system being built will satisfy the requirements originally identified at the beginning of the overall effort.

Well, we've reached the end of the discussion today.  I hope that you've found some of what I've discussed useful and that you're able to take pieces back in to your organization.  Feel free to share and let me know your thoughts on what has been presented and let me know what works in your organization.

Tags: SDLC, Software Development Lifecycle,  Project Lifecycle, Project, Manager, Development, Paradigm, Security, Secure Development Lifecycle, Communication, QA, Quality Assurance, Testing

For more information on David L. Collison: LinkedIn Profile

Project Managers, Bake Testing in to all of your Projects!

Well, this week I'm going to jump back in on the topic of quality - more importantly the need to build it in from the start.  This is a topic that many teams struggle with - how much testing is enough?  When do you call it good enough to move code in to production and celebrate success?

While I can't give you a definitive answer, it is important for you to look at each project and assess the risk of failure.  More importantly, you need to look at the various functional areas and ask yourself - what if there is an error, what will the impact be?  How will the application be used?  What is the audience?  What if this error is impacting one user?  What if this error is impacting multiple users?  Can downstream processes within the application - or dependent applications - continue to operate?

One of the questions that I've been asked by various developers - why are you making me do all this testing?  Isn't that the job of QA (Quality Assurance)?  Well, I hate to burst your bubble, but everyone is responsible for the quality of the end product.  Depending on the structure of the overall team involved in the project, the level of effort of the developers dials up or down.  If the organization has a strong QA Team, the developers may only be responsible for unit testing and some level of integration testing.  In other organizations, development may be responsible for everything up to and including user acceptance testing.  In web facing projects, business representatives may be integrated in to the Agile team and test directly with the developers - with the developers making changes on the spot.

I'm not going to try and convince you to pick one methodology or paradigm over another.  Each organization is unique and your organization has built the test structure that is being used for some set of reasons.  What I am going to do is explain some of the things that need to be done along the way - I can tell you who within the project I think is responsible, but this probably won't match up with your organization.  (I'll let you in on a little secret - it probably won't even match up with the way that my teams handle testing.  We're reworking how and who performs testing, so some of this is still up in the air as I write this blog entry.)

Whew, now that we've laid the ground work, let's dig in to the details and see if we can make some progress. Regardless of what methodology that you follow to develop software - the quality equation must be dealt with at the beginning of the project and must touch through all of the parts of your lifecycle to ensure that you're providing the highest quality software in to the production environment.  This is just as true in an Agile environment as it is within a waterfall environment.

So, let's touch base on some of the key things that need to be addressed by the overall test plan associated with your development effort:

1. Acceptance Criteria - when the team thinks it is done, what will be used to judge whether the software can move in to the production environment?
2. Integration Test Strategy - what are the application touch points that need to be addressed?  This includes system to system touch points - messaging, it includes intra-application - application sub-systems - that need to be tested, if your replicating data - it should include testing the replication mechanism, and it should also include testing of the security mechanisms used within the application.
3. Regression Test Strategy - is it necessary to perform a full regression test against the application, or is it valid to consider a limited regression test based on the functionality being touched by the development team?
4. Test Data - what data can be used to ensure that you can execute all of the tests needed to validate the functionality of the system?  This should include both positive and negative testing of the application.
5. Recovery Test Strategy - depending on the system you are building, you may have include functionality to allow the system to recover from various types of failure points - if so, you need to ensure that you test the recovery functionality?
6. Security Test Strategy - depending on your application, you may or may not need to include elements to secure the data being captured and/or manipulated by the system.  If this is the case, you need to ensure that you are testing the security elements to ensure that you are not allowing access to sensitive and secure data to individuals that do not have the proper authority.

On top of all that, each individual developer should be responsible for developing and executing unit tests against the code that they are developing.

All in all, there is a quite a bit of activity that surrounds the overall test strategy associated with the development process.  You may not formally address each of the individual items identified above, but in some way, these will get addressed throughout your project.  Either by accident or by design, someone should be answering the above questions before you put your code in to production.

As the Project Manager, it is to your benefit to ensure that these questions are being asked early.  This allows for the entire team to proactively address the test strategy and test requirements.  What good is it for development to be finalized if the development team doesn't understand how they will perform integration testing, or if the team doesn't understand how they will validate the acceptance criteria?

Next time, I'll dig in to some of the tactics that I've used - both within the current Teams that I work with and within previous organizations where I've worked.

In the meantime, feel free to chip in to the conversation and talk to me about the strategies that you use within your teams to manage your test strategies.  What are the components you feel are important and who within your teams has responsibility?  I'm not hear to judge what anyone does, but by sharing, we all might learn something new.

Tags: SDLC, Software Development Lifecycle,  Project Lifecycle, Project, Manager, Development, Security, Secure Development Lifecycle, Communication, Testing, QA

For more information on David L. Collison: LinkedIn Profile