Useless Retests

Testers have been recently tasked with testing the fixes that application development has created. They are finding that most of the bugs are still broken. I found this disturbing. So I took a look at a couple bugs that had been assigned to me. One of them had not been fixed yet. No wonder it still did not work. Another had been fixed. However that fix was not released to the test team.

This conundrum points to some kind of disconnect. I would be unhappy if I were on the test team now. They are wasting their time due to some incompetence. Somewhere in our process there is a severe lack of coordination. This responsibility falls on the shoulders of the team leads.

In theory, our process should prevent such nonsense from happening. One of the problems on our project is that we seem to have very little process. Not a lot of it is written down. If it is written down, nobody seems to know where it is. Or the stuff that is written down is obsolete, incomplete, or downright wrong.

Our bug tracking tool is supposed to help with this mess. You would think that you would change the state of a defect to “ready to test” once it has actually be fixed and delivered to test. I got the feeling that the team leads are just telling the test team to retest everything. A lot of the senior guys are saying we need to get this process under control.

For now I am staying out of this mess. I have found that on this project, you usually get the privilege of fixing problems if you bring them up. When the pain level gets high enough, I will step in. The last time I did this, I ended up serving as team lead for a couple years. That is a thankless and difficult job indeed. So for now I am just writing this rant to all those who read my blog. Enjoy.

Mozilla Bug Reporting

I read an entry on LiveJournal by Zwol entitled “On the Unfriendliness of Bug Reporting”. Zwol wanted to input a new bug he found in Firefox. However there were a lot of confusing points in the bug submission process. You have to decide which component to file the bug under. It is not intuitively obvious which option to choose. Some large categories already have a lot of bugs under them. There are other strange choices on the bug submission screen that have users giving up on the process.

Another part of the problem is the sheer number of fields on the bug report screen. Sure you want to add all the information you have. Zwol recommends a special developer mode for the bug reporting screen. That way the devs can see all the fields they need, and the average user can just enter the simple information to get the bug submission done.

The Mozilla team replied in the comments section of the blog on LiveJournal. They said they are working on the problem. A group of students have been assigned this task. That sounds good given the fact that they are an open source project. Perhaps they were already thinking about improvements to the bug submission screen.

On my project, you also have to go through a lot of hoopla to submit a bug report. New testers have a lot of trouble getting this right. The result is that bugs get lost of channeled to the wrong developers. Our bug submission screen is not only large, it has many screens to boot. This does have some advantages. When you know what you are doing, you can add a lot of important information about a bug. You can attach screen shots of the problem. And you can really specify a lot of characteristics of the bug.

I would think that a product made available to the general public at large should have a much simpler bug submission page. Being an open source project, I bet they roll their own open bug tracking system. And they probably developed their own bug submission screens. Who is the developer going to write those screens for? Unless care is exercised, it is probably not geared toward the casual home user who finds a problem. Maybe it needs to be.

Customer Trouble Tickets

A trouble ticket tracking system may have a short term goal of keeping tabs on software defects. However this is usually a means to another end. You don't track problems for the heck of it. You do it to ensure the users of your product can do their jobs. The Software Maintenance blog has a story about and developer and his trouble tickets.

This developer is staying proactive in making sure customer problems are being solved. Too many times a problem falls between the cracks. It takes an active effort to make sure all problems are prioritized and worked. A bug tracking system can help with this. But you need a process built around it to ensure the best results. You also need some level of diligence in the users of the tracking system.

Microsoft Excel

Normally we have an orderly and well defined process for tracking software defects. If the customer detects the problem, they enter the problem in their proprietary bug tracking tool. Then the help desk team enters this information into ClearQuest for us. A team lead assigns the ClearQuest defect reports to developers. Developers fix the problems. ClearQuest gets updated. Testers and then the customer gets the fix. All during this process, the Help Desk monitors progress by checking ClearQuest.

Currently there is a lot of test activity going on. Our internal testers are trying to complete a smoke test of our application suite. At the same time, our customer is performing a functionality test. The new change is that the customer is not entering problems they find in their bug tracking tool. Our internal test team is supposed to determine when the customer finds a bug, enter the information in ClearQuest, and forward the information to the dev team. I guess our customer does not want to incur the overhead of their own tracking tool for their testing.

I do not know how the deviations in process started. But we seem to be getting a lot of bug lists from our internal test team via Excel spreadsheet. They also provide a summary of smoke and functional test results to a broad audience. The spreadsheet appears to be going to all developers. And it is causing a lot of confusion. For now I am ignoring these spreadsheets and email because I am working on another task for development. But the other developers are going bonkers. Excel by itself is not a bad product. But using it to manage bug lists can be tricky. The new process seems to be adding to the pandemonium.

Home grown bug tracking schemes can work. But a real bug tracking system is better. Yes there is some overhead involved. However you do not have to worry about what version of the spreadsheet you are looking at. You also can avoid getting a bunch of e-mails with Microsoft Excel spreadsheets.

Relative Priority

Our customer has a system to track all problems that their users report. One property of trouble tickets is their priority. This priority ranges from 1 to 5 (1 being highest) based on a strict set of criteria. We are nearing the end of our software maintenance contract on our project. Most of the problems that get reported these days are priority 4.

Recently our customer headquarters group has been thinking about raising the priority of some problems which are crucial for their success. Their goal is to get the trouble tickets fixed quickly. However this will only lead to trouble. High priority tickets require a lot of status reporting overhead. So upping the priority might actually delay the resolution of certain problems.

I think the root cause of the problem is that there is no way for the customer to indicate the relative priority of problems they report. Our process is to work the issues in priority order. And if two or more tickets have the same priority, we work on the oldest ones first. Another part of the problem is that our software development team is shrinking in size due to the end of the contract. Us remaining developers are doing the work of two people these days.

My response to the customer concerns was to have the developers drop whatever they were working on to concentrate on the few high priority issues. It is my sincere hope that we can knock out these problems and leave the customer in good shape. This will also make my life easier as we try to close out this maintenance project.

Bug Conference Call

My team maintains a suite of applications that are used by over 600 users. With this many users, there are bound to be many trouble tickets submitted. We are normally pretty good at resolving these problems, since maintaining this software is why we get paid. Our customer loses money every time our software does not work correctly. So you can imagine the urgency to fixing bugs quickly. Currently we have conference calls twice a week with the customer to review each and every bug.

Back in the old days, when we had foolishly released buggy software into production, we had many open trouble tickets at a time. That put pressure on the group in our customer organization that oversaw our maintenance. Unfortunately that meant the pressure was applied to our team even more. Every day I would have to spend a good deal of time in my customer's office getting grilled on the progress of every trouble ticket. To this day I regret that upper management in my company did not protect me from such torture. But I digress.

Our customer currently drives the review of the open trouble tickets twice a week. We do not meet physically. However everybody dials in to an AT&T teleconference. I think that many people are actually on the line when we hold these conferences. But mostly the host of this call controls the conference. There is usually one main representative from the users of our software. And there are leads from all the development teams associated with the system. Since we are on top of most of the current problems, these conference calls have been going quite smoothly lately. So well in fact, that I am planning on taking a nice vacation soon. See you later.

Bug Summary

Twice a week we review the list of outstanding trouble tickets with our customers. Our help desk assists with getting the team prepared for this meeting. So the help desk produces a daily list of all open trouble tickets. This list has a minimal set of information like the priority of the problem, which team it is assigned to, which developer is working the problem, etc.

Our customer performs a similar roll up of trouble ticket information into their own spreadsheet format. Their summary is more detailed. When you print it out, it is hard to read the text. So you have to zoom in to read anything.

I have found these aids somewhat beneficial when assessing where we are with all known issues in our applications. However the list is just a start. I have found that you need to use the list to determine which issues you need to do some homework on. If you are not familiar with the problems, and you don't know where we are in the resolution process, having the list by itself is useless.

Somehow I wish we could get a list that is more dynamic. I suspect our defect tracking system can do this. We just have to set it up. Might require generation of a custom report. But the advantage would be that, every time you ran this report, you would have the most up to date information. This could be worth doing.

Trouble Ticket Assignment

Our last defect tracking system was very flexible. You might say it was too flexible. Anybody was allowed to assign a trouble ticket to anybody else. This can in handy often. For example, if I met another team lead, and we both decided a ticket should be transferred to a member of their team, I could just assign the trouble ticket directly to the individual. Now I do not think this flexibility was a property of the bug tracking software we used. I suspect it was due to the way the software was configured by us.

The current defect tracking system we use is more restricted. Only those with team lead authority can assign trouble tickets. This causes pain sometimes. Suppose my team lead is out sick today. Then I need to hunt down somebody with the access if I want a trouble ticket assigned. This is true even if I need the ticket assigned to myself. The result is that the system is often inefficient. And that results in less trouble ticket resolution productivity. Plus it gets developers frustrated.

I wonder how other systems are set up. Perhaps it depends on the size of the project. I imagine that a small team does not need as many restrictions as a larger one. The funny thing is that our project is actually winding down. So we do not have a lot of developers left. But we still have trouble ticket assignment on lock down. It might be time to change the rules in our system. For now I have gained the ability to assign trouble tickets myself. This is because our team lead quit, and I got drafted to take over his duties.

Authority

In our old defect tracking system, pretty much anybody could assign any trouble ticket to anybody. I do not think this was a property of the bug tracking software we were using. It was most likely due to the way the system was configured. This flexibility was actually nice to have. If I went to another team lead, and they told me that somebody on their team could take over a trouble ticket, I could just assign the ticket to that team member.



The current defect tracking system we use is more formal. Only team leads and managers can assign trouble tickets. This has some benefits in that there is some control to the madness. But it has its drawbacks. When my team lead is out sick, I have to hunt down somebody who has the access to assign tickets. This is a waste of time that leads to inefficiency. I imagine we could change the rules in our system to give us more freedom.



I wonder what the norm is. Perhaps on smaller projects, things are not on lock down. The funny thing is that our project is winding down and we do not have that many folks left. This is one of the reasons I had to get the authority to assign tickets. Our team leader left the company, and somebody had to take over.

The Big Picture

I have been working with my client for a very long time. They have many sites all over the United States. And their sites are big (a lot of workers). All trouble tickets are controlled by a centralized system. This system controls prioritization of problems. It also supports assignment of problems to divisions in the organization, as well as to individuals. There is automatic e-mail notification if you set it up. And there are alerts that fire when problems are not fixed in required times. This is just an overview of this complex trouble ticket system.

Our specific project is required to keep updating the status of tickets that are assigned to us. Normally we only let our help desk personnel actually interface with the system. The rest of the staff on our project uses a separate, smaller problem tracking system. The tool of choice for us is IBM Rational Clearquest. In fact, our customer requires we use this system since they have a license for it. The basic flow is that trouble tickets are created in our system by the help desk, transcribed from the customers trouble ticket system. We update the tickets in our own system. And the help desk transcribes the status back to the customer system.

The dual system has a number of benefits. Status updates can be controlled and sanitized for customer viewing. We have a much smaller data set in our local trouble ticket system. And we are also at liberty to modify our own system as we see fit. There is some overhead and delay associated with keeping the two systems in sync. I plan to describe more of our setup, and also other setups I have encountered in the past.

The Hunt for Bugs

I work in the world of software maintenance. Even though maintenance involves a lot of tasks, the most important is resolving software bugs. A key tool for this activity is a good bug tracking system. I am starting a blog to discuss this aspect of software maintenance.

The advantage I have, like some other developers, is that I have worked on a lot of projects in my career. And each of them tracked bugs in different ways. I intend to review the mechanics of these techniques to sort out the good from the bad. In the end it might boil down to people issues. But I believe the bug tracking systems make a big difference too.

At my current position we have migrated from using PVCS Tracker to IBM Rational Clearquest. So why don't I start with the use of Clearquest, and the processes that go along with it to track problem resolution.