Project Managers: Tactical ways to Push Testing

So, in my previous post, I talked about some of the high level things that you, as the Project Manager, need to be thinking about, as it relates to testing, when managing your projects.  In this post, I'm going to step down in to the tactical.  Specific things that you can begin to push within your projects and my recommendations for who should shoulder the responsibility.  These recommendations are based off of things that I've seen work in my current environment as well as things that I've seen work in various other organizations.

Now, before we dive in, a word of caution.  You need to evaluate what I'm saying and throw it up against the reality of the environment within which you work.  Some of what I say will work, some of what I say would probably spin heads within your organization.  Take what works for you and throw out the rest.  We're currently reevaluating some of the test strategies that are used within my teams and six months from now, I may have a different take on the subject.  My biggest concern is ensuring that we put enough thought in to the process to allow less experienced team members to be successful when working with unfamiliar subsystems or product features.

And, with that said, it's time to dig into the details and talk about things that I've seen work - note for those readers willing, I'd love to get feedback from others on what they are seeing work/not work in their environments.

First up, what are your end users (or the team members representing the end users) doing to drive the quality initiative?  These are the folks that are creating the user stories or requirements for the products that you're building.  Are they being held accountable at the start of the process to define what success means?  Are they telling you up front what they will do when performing user acceptance testing?  If not, it's time to sit down and have a conversation.  Why do they feel comfortable telling you what to build if they can't tell you how they will know it is right?  Specifically, what are the conditions and scenarios that they will use to validate that the functionality delivered is acting properly and producing the correct results?

1. What set of information needs to exist within the system to allow them to create the scenarios that they will use to test the requested functionality?  What specific elements of data will be used in the testing - both from a positive and negative test perspective?
2. What business logic is being used to assess whether the outcome is valid?  What will change within current processes?
3. Ultimately, what does success mean to them?  How will they know that what you've delivered works or doesn't work?

Now, ask yourself, if these do not exist prior to the design and development of the solution, how will the team know that the proposed solution will actually solve the problem?  This function must be driven by the end users (or those team members representing the end users).  They can collaborate with others on the team, including business analysts, technical team members and dedicated testers.  But, ownership of the acceptance test plan must fall on the shoulders of the end users.

One of the things that I like to do once the users have defined their needs and the design phase is underway, is to have the technical subject matter experts (SME's) begin to identify high level integration test objectives.  This is built at the same time that the technical design is being worked and is a very brief document outlining the gotchas that the development teams need to be aware of when developing and exercising the new systems/features.  This is intentionally kept very high level - usually just descriptions of each item that will need to be fleshed out in later stages.

1. Identify touch points between various sub-systems that will need to be tested.
2. Identify data replication strategies that will be used and that will need to be tested.
3. Identify data transformation strategies that will be used and that will need to be tested.
4. Identify services that will be relied on that will need to be tested.

This acts as a roadmap for the entire technical team.  It should be a roadmap used to identify data validation and functional validation.  It can be used by those that may be unfamiliar with specific subsystems to gain a further understanding of the interrelationships between the various subsystems and external/internal services.  The IT SME's own this high level integration test plan and pass it along to others in the process.

As we dive down in to the detail design - several things begin to happen.  First, the development team begins to flesh out the high level integration test strategy and it becomes the Integration Test Plan.  This document details the specific actions that will be used to test the touch points previously identified and should include the data and functional validation - data should be traceable throughout the system with the end points being validated.  This includes identifying the specific data elements that will be used to perform the tests as well as the expected positive/negative results.  While driven by individual development team members, the plan will be reviewed and validated by the Technical Leads.

Additionally, the individual development team members begin to identify their individual Unit Test strategies.  Each functional unit of code being created or modified needs to be tested to ensure that it is performing to spec prior to moving in to integration testing.  I believe in the paradigm that the tests should actually be created prior to the code being written, but I've seen development teams be successful using several different paradigms - so I won't harp on it.  The individual developers own this and are responsible for ensuring that they develop clean code.

Once the developers have been able to execute their individual Unit Tests, then they work together and move in to pre-Integration Test planning.  This time is spent ensuring that the necessary data has been identified and created/setup, that the configuration changes needed to the system and supporting sub-systems have been applied, that the network configuration changes needed to open ports to talk to internal/external services have been applied, and that all of this has been documented for production.  This particular block of activity is owned by the Development Team and the Technical Leads.  Infrastructure Teams may support the activity, but the Development Team and Technical Leads should ensure the configuration is complete and documented for the Implementation Plan.

Now we step in to Integration Testing - executing the plan that was documented earlier.  It is inevitable that something will have been missed - but that is what this stage is for, to find the bumps before you move your changes into the quality assurance (QA) and production environment.  Here you will find the missing data elements or calls to services/api's that work differently than documented.  You will uncover the fact that replication strategy for data you rely on wasn't documented properly and the underlying data you were expecting to see doesn't exist.  All sorts of unknowns will surface and you'll spend time playing whack-a-mole to clean it all up to a point where you believe it is functional.  The Development Team and the Tech Leads own this piece of the puzzle and work to ensure that all of the individual pieces come together and work as a whole. 

Now, let's take a few minutes to talk about Regression Testing.  I'm actually in the midst of transitioning our teams form the old style of moving code in to QA and then running a full regression test of the code along with new function/feature testing to a model where the code is built nightly and then the suite of regression tests is run when the build has finished.  This has been a tectonic shift for the teams in that the developers are getting daily feedback on code that has been checked in to the repository and we actually are building up a set of statistics that show us the overall state of the code.  I can now walk in on a daily basis and see the various package builds, number of tests being executed and the success/failure rate - along with a whole suite of statistics on coverage and severity of the failures.  Benefit number one - immediate feedback to the development team while the code is still fresh in their minds.  If we need to change something due to a failing test, the developer doesn't need to spend time remembering what changes were made.  Benefit number two - reduction of effort by our QA Team Members to get code through the testing process - their focus is on new feature/function testing, not regression testing.  Benefit number three - the developers have more buy-in on the testing process and begin to have a better understanding of the underlying data.  The Development Team is responsible for the Regression Testing and must review the results with QA to validate that the code is ready to move in to QA Testing.

QA Testing is focused on the new features/functionality to ensure that they are operating within defined constraints.  Here is where I want to hit the system hard with positive and negative testing associated with the new features/functionality being introduced within the overall system.  What happens where certain pieces of data are missing from messages exchanged between subsystems?  What happens when a user forgets to input data?  What happens when a web service that we are relying on isn't available?  What happens when the system thinks everything processed fine?  I will also want to take a look at load testing on some system to ensure that we are staying within expected SLA's.  This testing is the responsibility of the QA Team with input from the Development Team, Tech Leads and support from the Infrastructure Teams.

As  part of the work that is being done during QA Testing - we should also work with those team members representing the end users to execute the User Acceptance Tests (UATs).  This should tie all the way back to the beginning stages of the project where the business teams defined the success criteria of the effort and began building the definition of their UATs.  During the entire project, these high level UATs should have been fleshed out and reviewed to ensure that they align with the final requirements of the project.  QA and the User Teams should jointly execute and review the results of the associated UATs and prepare to sign off and give final authorization to move the code in to production.

As a Project Manager, you need to assign tasks to cover these activities within your overall project plan.  More importantly, you need to be following up with the responsible parties at various stages of the lifecycle to ensure that they are not being ignored.  Your organization may hold different team members responsible for the specific activities outlined.  They may also skip some of the items outlined above.  However, you need to ensure you are covering enough of the pieces to confirm that the system being built will satisfy the requirements originally identified at the beginning of the overall effort.

Well, we've reached the end of the discussion today.  I hope that you've found some of what I've discussed useful and that you're able to take pieces back in to your organization.  Feel free to share and let me know your thoughts on what has been presented and let me know what works in your organization.

Tags: SDLC, Software Development Lifecycle,  Project Lifecycle, Project, Manager, Development, Paradigm, Security, Secure Development Lifecycle, Communication, QA, Quality Assurance, Testing

For more information on David L. Collison: LinkedIn Profile

All it takes is a little communication

Back in action again with another posting.  My activity on this blog slowed down over the summer months as I focused on work and relaxing - probably a little more on the relaxing side.  Now it's time to  get back in the groove and start sharing again.  For those of you that have been reading this blog for a while, I hope you enjoy the new contributions!  And for those of you seeing this for the first time welcome - I'm attempting to share my thoughts and opinions on the software development process and the roles that people play.  My hope is that everyone reading through these articles can take away something that helps them, and that through the messages that I receive, that I learn something along the way.

In many of my past entries I've discussed various issues that can cause a project to go off the rails.  This could be anything from a minor irritation within the overall lifecycle, or it could be more significant causing a reevaluation of the overall project.  Obviously, I'd much rather deal with the former vs the latter, but as with anything in real life, we don't always get to choose the trials that we face.  And, sometimes we learn the most as we fight through the fire.

One of the things that I'd like to tackle in this particular discussion is the need for true open and honest communications between all members of the team.  Many times, an issue that surfaces within the overall project is the culmination of the effect of someone not getting out of their seat and talking with a team member.  Ok, getting out of your seat may not be the right terminology in today's connected and distributed world.  Team members have many options to connect with other team members - email, instant messaging, video conferencing, texting and - wait for it - good old fashioned face to face discussions.

Let's be honest with each other - how many times have we all been irritated and wanted to pull our hair out because the response we get from someone when it hit the fan was, "Well, that's what I was told to do.", or "That's what the design said to do."  In most instances I then ask the team member, "But you knew that wouldn't work, right?"  And in most instances I'll hear them admit that they knew the design was wrong or that the item in question had been overlooked or that they didn't feel that they could discuss it with someone else on the team.

To be clear, this is an example and the point of this post is not to lay blame for all issues at the feet of the requirements or design teams.  In many instances, the people doing the requirements and/or design work don't necessarily understand all of the fine details of how the system is plugged together. Understanding the fine technical underpinnings of the system is the responsibility of the technical teams.

If someone, anyone, on the project team identifies that something is missing, incomplete or just not right, it is their responsibility to get it corrected.  By whatever means are necessary.  It is not proper to sit on the issue and knowingly let your teammates head down a false path, or to let the project stall.  If you're truly on the team, you're invested in making the team, the project and the company successful in creating solutions that can be used by your customers. The key phrase, "you're invested"!  If you're invested in the idea, then that means you're taking it personally to make sure that it's right and that the end result exceeds expectations.

How difficult is it to poke your head up from what it is your doing to have a conversation?  Not every issue requires a meeting with 10 people in the room looking for various options and contingencies.  Take a stab at solving the issue on your own with a few key people.  Review your thoughts with key individuals on your team and then make it happen - get it corrected and resolve the issue before it has time to fester and become a potential roadblock to the project.  If you can't get to the bottom of it and find a reasonable solution, then work with the Project Manager to get a larger team of individuals to together to work through the issue.  Oh, and by the way, make sure you're keeping the Project Manager in the loop on what you've found - that doesn't mean they need to take control of the situation, it just means you're keeping them informed.  In fact you can tell them upfront: I don't need you to get involved.  I'm working through some options right now with a couple of people, if we can't find a solution by the end of the day, then I'll need you to step in and assist with coordinating a larger part of the team to help find a solution.

As I've said in past postings, people are messy.  We all make mistakes from time to time.  Maybe when putting the design document together I forget that a certain data element needs to be replicated to another application in the enterprise.  By forgetting to include this in the overall design, I have inadvertently introduced a situation that may impact the companies ability to bill for certain services or maybe the information won't show properly to the customer service representative answering customer calls.  In either case, the mistake was small, but the impact to the customer is real and measurable.  

If down the line, one of the software engineers notices that I overlooked documenting how the data element needs to be replicated between systems - it isn't the end of the world.  A quick call to confirm that "not replicating" the data wasn't an intentional decision and in fact was a mistake - allows the project to proceed.  Documents can quickly be updated and the change communicated to the rest of the team.  However, if the software engineer sees the mistake, doesn't say a word and we get all the way in to production without replicating the data, we have negatively impacted our customer by potentially billing improperly for services or by delaying assistance when they call in for help.

Look, one of the reasons we are all in the jobs that we are in, is to solve problems.  Sometimes that means sitting at your desk and doing what's in the job description.  However, in today's world, the job description is just the start of what you really do in your job.  You've also been hired to be part of a team!  It means removing the small roadblocks before they become big roadblocks.  It means uncovering and resolving those inconsistencies before they can be seen by the customer.  It means finding ways to work with others to provide the unexpected.  It means noticing that your teammate is stuck and helping find a way around the obstacle.  It means thinking through something that you believe is wrong - validating the assumption, and then working with the team to provide a solution before the issue has a chance to delay the project, increase costs or call in to question the need to finish the project.

Today, we are hyper-connected.  Take advantage of the tools that you have to break down barriers and talk with those around you to solve problems.  I don't want you to think that your job is to just satisfy the "requirements" or "design documents".  I want you to challenge those things that you think are wrong or that negatively impact the customer and work with others on the team to identify solutions that allow the project to move forward and meet or beat the timeline, reduce the overall effort and provide a superior solution.

Tags: SDLC, Software Development Lifecycle,  Project Lifecycle, Project, Manager, Development, Paradigm, Security, Secure Development Lifecycle, Communication

For more information on David L. Collison: LinkedIn Profile
 

Back to Security - Ignore and You'll Pay

Def Con 21 kicks off this week, so I thought it would be an appropriate time to swing our discussions back to the topic of security.  In some instances, repetition of a theme can become stale. Security is one topic where you must always pay attention and never, ever drop your guard.

Leading up to Def Con 21, white hat researchers have announced that they have figured out how to hack into both the Ford and Toyota automotive systems.  They have announced that they will release a 100 page report detailing their efforts.  While they will not release the specifics of how they achieved the takeover, now that they've reported it can be done, you can bet that others will follow suit.  According to reports, the researchers were able to force a Prius into suddenly slamming on the brakes while the car was moving 80 mph, they were also able to force the steering wheel to jerk and force the engine to race unexpectedly.  On the Ford Escape system, they were able to disengage the breaks on the vehicle while it was moving at low speeds.

Additionally, academic researchers have also figured out how the anti-theft systems of several automotive makers can be bypassed.  Volkswagen, Porsches, Audis, Bentleys, and Lamborghinis are all subject to the hack.  The researchers claim that they have figured out multiple ways to bypass the immobilizer mechanism by reverse engineering the algorithm.  Volkswagen actually preemptively went to the courts and received an injunction preventing the authors from publishing the details of their research.  While the judge has prevented the information from being published, now that it is known that it is possible to hack the algorithm, others will undoubtedly follow and perform their own research.

Earlier this year, a tool called "DropSmack" was created by a security consultant that used DropBox to remotely take over a PC.  The consultant was able to add macros to a Word document on DropBox and then use spearfishing techniques to get an executive at a company that he was working for to open the said document.  Once the document was open, the consultant was then able to take over the PC.

How many of you work in organizations that either knowingly or unknowingly have PC's that are connected to DropBox?

As altruistic as we would like to be, we live in a society where some people are intent on gaining access to systems that they don't own for various reasons.  Sometimes they just want the thrill of being able to gain access to something that they shouldn't have access to; sometimes they want to take everything you've got.  If you're not paying attention and doing the right things to secure your network, secure the applications you use and secure the applications you create internally, you're making it easy for the bad guys.  The latest report from Verizon - the 2012 Verizon Data Breach Investigations Report - identifies that 78% of all attacks last year, were targets of opportunity.

As you are designing software, the primary question you need to ask yourself is, how will someone attempt to use this to do something that they shouldn't be doing?

Security needs to be a significant focus of the design, build and test phases of the lifecycle.  Additionally, regular testing needs to occur once your applications move in to the production environment to ensure that the system is not vulnerable to the latest vulnerabilities.

I've heard some organizations make the claim that they can't afford to take the time to worry about this, that they are small and nobody is going to notice their web site.  Well, I hate to be the one to burst your bubble, but if your network is accessible via the internet - and who's isn't these days - then you're at risk.  If you don't take the time and dollars necessary to secure your web site and put the technology in place to protect your network, someone is going to find you.  They may not necessarily be looking for you, but they will find you.  They have built tools that move through the internet going from one system to the next looking for vulnerabilities.

If you haven't patched the software you've purchased to run your systems, and kept those patches up to date - then they will find it.  If you haven't patched the open source software you are using, and kept those patches up to date - they will find the opening.  If you haven't prevented cross-site scripting from impacting your web site, if you haven't prevented SQL injection - then they will find it.  There are easy things you can do to protect yourself and not all of it costs money:

1. Keep your systems patched to the most recent releases - whether purchased or open source.
2. Install firewalls in your network and keep the firewall patched to prevent unauthorized programs from accessing your network.
3. Institute and enforce policies for users to change their passwords regularly and to use strong passwords.
4. Install and use anti-virus/anti-malware/anti-spyware protection.
5. Don't open emails from people you don't know?
6. Institute whitelisting of applications and internet sites that can be used and accessed from within your organization.

From a development standpoint - if you're not familiar with OWASP, then you need to hit their web site and immediately begin to pay attention to the vulnerabilities that they report.  This site identifies the most common vulnerabilities impacting web sites.  They explain the vulnerability and recommend what you can do to prevent these types of attacks on your network.  I'm not saying that this is the be all and end all of security, but this site at least points you in the right direction.

You also need to ensure that your developers take the time necessary to educate themselves on secure development techniques.  This isn't an overnight miracle that will suddenly make you invulnerable, it is a process that will take time, but will slowly and surely improve the security stance of your applications.  They are books, websites and seminars dedicated to this topic.  Pick the one you can afford and get moving - today!

Tags: SDLC, Software Development Lifecycle,  Project Lifecycle, Project, Manager, Development, Paradigm, Security, Secure Development Lifecycle

For more information on David L. Collison: LinkedIn Profile

Tools and the SDLC - Let's Make a Change!

How do you transition from one tool to another or make the decision to change the type of development paradigm that your teams are using?  I'm not going to pretend it's easy, nor am I going to pretend that I'm an expert at it.  But, I have been through several of these across different organizations and the path is similar.  What I'd like to do is share some of the insight that I've gained throughout my career and ask that others provide feedback on what they've seen and experienced.

I've seen organizations shift their development processes back and forth between very informal processes, to formal processes, to something in the middle.  Sometimes adding/changing/removing processes or tools, sometimes swapping out the entire paradigm by which the team operates.   All of this is being done in the name of improving time to market, increasing quality and/or improving the productivity of the team members.  Admirable goals, no doubt.  In fact, consultant organizations have popped up over the years that specialize in taking teams to specific development paradigms and switching the entire tool sets used across the entire lifecycle.  This is not something that is done lightly.  There is usually a loss of productivity through the transition period and an outlay in real dollars as the organization trains entire teams on the new paradigm.  Market opportunities are impacted as teams shift to the new paradigm and the potential to fall behind the competition is real.

Software Engineering is both an art and a science.  The science part is the syntax - this is straight forward and understood and can be taught.  In fact we have excellent resources - both formal and informal - that allow anyone interested in getting in to software development to learn.  On-line tutorials, books, free on-line courses, paid on-line courses, high school courses, college courses, and seminars.  Along the way, at any level, prospective and experienced developers can get assistance via dedicated web forums, videos, and sites that hold sample code.  If you need help learning how to code a particular language or a specific function, you can find help on the web. I still remember my amazement back in the late 80's when I posted a development question up on a early portal and received an answer within a matter of hours from someone half-way around the world.

Project Management is also both an art and a science.  The science part is managing the defined process and set of artifacts.  Knowing when certain activities need to be completed.  Understanding when a team member or portion of the team needs to complete specific artifacts.  Managing the Risks and Issues.  Understanding when to communicate to the team, individual team members, across and up through the organization.

The art of Software Engineering and Project Management is the individual.  Put two engineers in the same room to solve the same problem and you will invariably get two different answers.  Assign two different Project Managers to the same project and they will drive results differently - one may be more of a driver - managing the process by proof; the other more of a expressive - using relationships to primarily drive the process.  Making software development work is not always taking preexisting components and placing them together.  It is not like building a prefabricated home.  It is knowledgeable people understanding how to finesse the tools and the processes to move the ball forward.

Development Paradigms and the Development Lifecycle (SDLC) are the sauce that brings order to the development process.  Attempting to provide the pieces needed at the appropriate times to allow all parties to be satisfied.  Moving from no development process or paradigm to a structured paradigm can be difficult - trust me, I've been there, done that.  However, moving from one structured paradigm to a different paradigm is often worse.  When there was no process to follow, getting someone to create a specific artifact or to follow a specific process is an effort, but can be done.  Getting someone to change the way they create an artifact or to stop creating the artifact altogether is tough.  All of a sudden, fifty people will show up to explain why they need the artifact the way that it is today, or that by not producing the artifact, you are going to break 100 different things across the organization.

Fundamentally, I guess I've always asked for feedback on the processes and tools that are used within the organizations that I'm leading.  Yes, I want to bake in what we are doing, but I also want feedback, and if I can be convinced of the value, I'll work with the teams to make the changes and to drive overall improvement.  Here are the ground rules that I ask people to consider before recommending changes to the processes, tools and the overall development paradigm that we use:

1. How will this impact the customer?  Everything that we do must be in the context of our customers.  What's the worst case scenario if we suddenly transition from the way we currently do something to the new way being proposed?  What are the perceived benefits of making the change?
2. What does this do to our quality?  Yes, this is related to the first item, but stands alone on the overall merits.  Does the change allow us to fundamentally alter the quality equation?  Can it reduce the efforts needed to validate existing quality?  Does it provide additional quality measurements that are not used or possible today?  Does it expose us from a quality perspective and unintentionally add risk to our systems?
3. What does this gain us against our competition?  Does it allow us to leapfrog with some feature set or capability?  Does it alter the balance and allow us to take advantage of competitive weaknesses?
4. How does this impact the flow of data through the organization - does it reduce the quality of data, does it reduce the number of places data is stored, does it reduce the number of times data is entered through various systems?
5. Can the change provide a significant reduction in the time it takes to move through the overall lifecycle and improve time from concept thru delivery?
6. Does it impact our security stance?  Does it improve or decrease our ability to secure our intellectual property (IP) and the data held within our systems?

These are not conversations based on the concept of - hey, I found this great new tool on the internet and played with it at home, so I want to start using it at work!  The dialog must cover the above points and we must also take in to consideration the impact across the team.

I can't afford to change tools, languages, processes or artifacts on every project that runs through the organization.  That said, I'm always interested in having a discussion on ways we can make positive changes that can reduce our internal efforts, improve the security stance of the organization, drive better quality within the products we build, and reduce the overall time to market.

As it sits today, I have people trying out different processes within the way that we run our project meetings.  I have people making recommendations on new tools that we can use within the lifecycle.  Team members are also making recommendations on changes to some of the artifacts that we use.  Will all of these changes make it in to production - probably not.  Will some - I hope so.  Over the last six years we've made significant strides that have allowed the organization to dramatically increase the number of projects being worked as well as the quality of products being delivered.  Now we must change to meet the future needs of the organization.  We must continue to innovate and deliver solutions that make a difference.

When and how do you allow changes to be made to your overall lifecycle, the tools that you use and the artifacts that are generated?

Tags: Project Management, SDLC, Project, Management, Project Manager, Software Development Lifecycle,  Project Lifecycle, Project, Manager, Development, Paradigm

For more information on David L. Collison: LinkedIn Profile