It's NOT about the Tech or the Process - it's about the Customer!

This week I have the pleasure of participating in the annual SHAZAM Forum!  It is an event that my organization holds each year to keep our Financial Institutions informed on key trends within the industry.  Over the next few days I will have the opportunity to interact with hundreds of our customers.  To listen to their stories, to understand their needs and to enjoy some time together.

Let's be honest, as much as I care about the tools that we use and the processes that we have in place.  Our customers really could care less.  They care about that fact that the systems we provide will function - that they can open a new account when someone walks into one of their branches, that they can process a loan when one of their commercial customers needs a new piece of equipment, that their account holders can login to their internet banking site and transfer money between accounts, that their account holders can use their debit card when purchasing groceries at their local grocery store and that when I detect some fraudster starting to steal their customers money - that it is stopped.

They really don't care that I use Java or C++, they could care less if I used Agile or Waterfall project management paradigms.  They don't think about the various testing methodologies in place to prevent bugs from impacting their ability to perform their work.

Put simply - our customers want and deserve to be delighted when using the systems and services that my organization provides.  They don't want to know the details of why or how we made decisions and built the services/systems, they just want to be able to use them to conduct business with their account holders.

I look forward to our annual Forum because it's my opportunity to get to know the folks that choose to do business with the organization that I work for; I get to hear what is important to them; I get to listen to the frustrations that they have using the systems that my teams build; I get to listen to the concerns that they have about the market that they serve.

Let's be honest, on a daily basis, most individuals within an IT organization have very few opportunities to interact with the customers of the systems that they build.  I'm intentionally not counting the internal customers.  Internal customers are important, but they do not provide a true reflection of what our Financial Institutions, their employees and their account holders are challenged with on a daily basis.

It's important to occasionally step away from the technology and immerse yourself in the world that your customers deal with every day.  To understand their interactions with the systems and services that you build so that you can understand what needs to change in future iterations.

In other words - it's not all about me, it really is all about the customer!

For those of you that will be attending our Forum - I look forward to seeing you again this year or meeting you if it's your first time!

If you'd like more information on my background: LinkedIn Profile

Test Driven Development - Think about it!

Over the last few years I've been moving my teams towards Test Driven Development.  If you're not there, you need to be looking at this paradigm.  In essence, the following is what you are shooting for:

1. Write your test.
2. Run your test - the test should fail because the code hasn't been written yet.
3. Create the bare minimum of code that will allow the test to pass.
4. Continue to refactor the code until you are satisfied - is it simple, have you removed any duplication?
5. Repeat - accumulating additional tests until full functionality matches the agreed upon requirements and design criteria.

I've been on this push for a couple of different reasons: 1) The QA Team is spending an inordinate amount of time performing regression tests that should be in the build process; 2) The QA Teams and Development Teams should spend most of their time on new feature/new function testing - not regression testing; 3)  Within the old paradigm that we were following, by the time the development team member became aware of a defect in the code, they had long since moved on to other parts of the code and it was not fresh on the mind.

I've recently spoken with several groups of students - some high school and some at the collegiate level - and the one common theme that I have been impressing on all of these groups is that they need to think about testing first.  That means while doing the design work and prior to any code being written, they should understand how they will test the code and then execute against the plan.

In a separate session. when meeting with some of the instructors - another individual recommended that the instructors should have a base set of tests created that the students will need to execute against their assignments.  The students could do this at any time to get immediate feedback.  If an instructor were to pursue this paradigm within their classrooms, I would encourage them to create a portion of the tests, but force the students to create their own set of tests.

Unless our teams are working on a completely new system - there should exist somewhere in the organization a base set of tests that cover the current functionality.  It may not be extensive, but it's there somewhere.  That's the baseline!  No build of the system should happen without improvements to that baseline with your development team increasing the code coverage with additional unit tests that are then included in the set of regression tests the next time that code is touched.

Let's face it, immediately after the code has been written and the developers focus shifts on to the next task, their ability to maintain the code begins to drift.  I'm not saying they can't maintain it, but that they will need to spend time re familiarizing themselves with the code and then planning how to make the change.  Additionally, you can't guarantee that the next time the code needs to be touched it will be touched by the same developer.  By having these tests complete and in the pool of regression tests, you set an expectation with the developer that they can't claim they are complete with the code until all regression tests are complete and they can prove that they have tested against the new feature/functionality of the application.

Really, when it comes down to it, it's about accountability within the development team.  They are just as responsible for the quality of the code as are the people working the front end of the project - discovery and requirements, design - and those working the back end of the project quality assurance and the user representatives.

If you'd like more information on my background: LinkedIn Profile

Metrics - How I Use Today

Metrics are a funny thing - what works for one team may not work for another team.  In practice, I have always found that I need to adjust the metrics that I track based on the organization.  I find it humorous when some vendor walks in the door and tells me that if I just revamp my process to use their custom SDLC solution and manage by their key indicators that all of my problems will be solved.  Dream on!  If I’ve learned one thing, it is the fact that I must be sensitive to the unique culture of the organization and create a process and metrics that work within the overall organization.

So, that said, I’m going to walk through some of the metrics and processes that my teams currently utilize and how these get used to strengthen the overall lifecycle.  You may find some of this useful, or you may decide that the whole thing is nonsense.  That’s up to you - what I can say is it works for the teams that I manage and is trusted by the management team to give insight into our overall portfolio.  This is still somewhat high level, but it will give you the general flavor of the things that I consider important.

One of the first things that I did walking in the door was to have the entire development team begin tracking what time was being spent on what activity.  There were multiple reasons for initiating this type of tracking:

1. It allowed each individual developer and their manager to identify where they were spending their time.  It also initiated conversations on why people were spending time on certain activities.  Over a period of several months, we collectively managed to increase time spent on corporate approved projects from 35% of the overall time to 65% of the overall time.  We continue to look at this metric to ensure that people are spending time on approved project activity.
2. It allows me to understand how many hours people are working and to look at when I need to increase our headcount.  Over the last six years, I have used this data, along with other metrics information, to initiate the discussion on headcount increases and win those discussions.

Time tracking allows me to identify several metric points that provide insight into our overall lifecycle, as well as down to individual resource management:

1. Overall time spent on a given project across all resources.  How did it compare to our baseline estimates?
2. Time spent on activities within the project vs the baseline projections.
3. Are our estimates accurate?
4. Did we identify development activity that was missed during the design stages?
5. Individual time analysis - approved project activity vs production maintenance vs defect management vs administrative overhead.
6. Team analysis - approved project activity vs production maintenance vs defect management vs administrative overhead.

This has proven so successful within my team, that the entire IT Department now tracks their time and all individuals within the Business Teams track their time against project activity.

While time trackings helps at an individual project level and at a resource level - it is responsive and does not allow you to project out over a period of time.  For that you need a view into your entire portfolio.  To that end, we have a Portfolio Plan that is reviewed on a weekly basis by myself and my peers across the organization.  We do this to identify bottlenecks in the process, find ways to breakdown barriers and push the projects forward.  Here are the key metrics that we review:

1. Each project within our Portfolio Plan is broken into 6 distinct phases - Discovery, Requirements, Technical Design, Development, Testing and Launch.
2. Each phase has two key dates - Start and Finish.
3. Each phase has resource allocations assigned for each team.
4. At the end of each phase we adjust our timelines and allocations for the following phases based on what we’ve learned to date within the project.
5. Each project in the Portfolio Plan is tied to the associated “real” project housed in our MS-Project Server.
6. Each “real” project has standard milestones that identify the initiation of each phase and the completion of each phase.
7. Each day - routines are run that allow us to provide analysis of the Portfolio Plan vs what is happening within the “real” projects.
8. Periodically, I go in to relevel the entire Portfolio Plan to ensure that we are not over-allocating any one team and that we have enough work moving through the pipeline.

Why do we do this?  Simple, it helps us control the activity in the organization today and allows us to review the allocations of the various teams and ensure that we are levelling project activity over a period of time.  Most importantly, it forces us to prioritize what is moving through the pipeline.

1. I can tell when activity within any phase is going to be completed ahead of schedule, on schedule or is behind schedule.
2. I can use the above information to tell me when I need to reallocate resources so that I can hit internal target dates for critical projects.
3. I can use this information during budgeting to review with Senior Management and discuss our pipeline to see if they are ok with the volume of activity we are managing or if we need to increase our pipeline.
4. I can use this information to hold individuals and teams accountable.
5. I can project out when we will be trying to force through too much activity within the overall pipeline and use that information to initiate conversations about the priority of project activity.
6. I can use this information in conjunction with the time tracking data to identify when additional resources might be useful.

So, removing ourselves from the project level and going down to where the rubber meets the road, let’s look at some of the metrics that we are tracking within the development team as it relates to coding:

1. We have begun implementing nightly builds/nightly regression tests across all of the development teams.  Some teams are further along in the process than others, but all are moving towards the same overall goals.
2. Jenkins is used as the tool that automates all builds for all packages across all environments.
3. Jenkins is also used to gather results from the build process and the regression test process.
4. Jenkins is used to distribute emails to team members and management members identifying the results of the build/regression test process.  Developers are then responsible for going in and fixing the issues that have been identified.
5. Sonar is a tool that allows us to view the results of the build/test processes in a web application - broken out by each application.
1. Line Coverage - how much of our code has been validated with test cases
2. Unit Test Success - how many of our tests completed successfully, how many failed.
3. Violation of Rules Compliance - how many instances there are where code is not conforming to coding rules - both those identified as industry best practices and those that we have defined internally
4. Severity off the violation.
5. Code Reviews - I can tell if the code has gone through our review process.
6. Other metrics are provided - but I don’t consider them as useful as those identified above.

All of the above gives me insight into the overall confidence of the package before we move to production.  I can review this anytime I want and see our progress in knocking down the number of bugs being found by our standard unit and regression tests.  I can see if we are improving the overall amount of code that is covered by our tests.  I can see if we are reducing the overall number of violations that we are finding within any given application.

Moving in to the formal test cycle, I then begin to track a different set of metrics:

1. How many defects were found by QA that were not identified by the Development Team through unit, integration or regression testing?
2. How many new package builds occur before QA has a clean package ready for production?

Each week we notify the entire management team of changes impacting the 6 phases of our project lifecycle.  Those projects where dates have shifted for any given phase are identified along with explanations - a miss by the team, extended due to resource constraints, extended because we missed something, reduced because it was easier than we anticipated.  Whatever the reason, it is noted and everyone has visibility.

On a bi-monthly basis we meet with Senior Management to discuss the overall portfolio.  The slide in timelines - positive and negative - that we are seeing across all projects.  Projects impacted by levelling activities.  Reviewing the priority of projects so that we understand where to focus time when the pipeline is packed full.  Understanding shifts in strategy that we need to accommodate within the overall pipeline.

At the end of our projects, we provide summary information to our Senior Management Team that compare the total number of hours spent within the various teams vs the baseline estimates.  We report on the total $ expenses vs the baselined estimates.

Again, this is somewhat high level, but it gives you a flavor for some of the things that I consider useful as my teams move projects through the lifecycle.  Most of this was accomplished using tools we already had in place - but that we weren’t taking full advantage of:

• MS-Project
• MS-Project Server
• PWA
• Reporting Analytics
• Jenkins (Build / Test Automation)
• Sonar (Consolidated Build Reporting)

Yes, there are other metrics that we track and discuss.  This has all been done with an eye on improving individual and team productivity, reducing the number of errors moving through development and into QA (ultimately into production), holding individuals and teams accountable and ensuring that we are prioritizing activity within the organization to align with the overall strategy communicated from Senior Management.  I’m focused on those metrics that can provide significant results for the organization.  I’m not interested in adding metrics to produce pretty charts.

None of these metrics are full-proof and each must be taken at face value.  Sometimes the metrics lie and that’s where it is the job of us as individuals to interpret and validate the results.

Tags: SDLC, Software Development, Metrics, Lifecycle, Software, Application Development, Project Office

If you'd like more information on my background: LinkedIn Profile