I have been using ClearQuest since around 1997, and ClearCase for longer. I’ve worked on schemas in many banks and other large companies, around the world. I co-authored the tools on this site. I’ve delivered training at various levels around the world. I’m certainly not the last word on ClearQuest, but I think I know it pretty well.
To be frank I’m not sure how ClearQuest came to have such a large installed base. The decision makers tend to be IT managers, not process or release engineers, and heaven forfend that any one that’s actually going to use it should have any input in to the decision. These IT managers hear the right words, “full audit trail”, “configurable process”, and most importantly “integrates with ClearCase”. They hear: web client, native client, Rich Client Platform-based client, and they don’t think – “Jeez, how are we going to support and maintain all those clients and deal with their myriad differences and quirks”, they think, “ooh, 3 clients, that should cover all the bases”. No one got fired for buying Microsoft, and people rarely get fired for buying IBM, so they go with the herd and blithely hand over several hundred grand.
A few years ago I gave a ClearQuest admin training course to a financial exchange. I was explaining some quirks in the Designer that they were struggling with, and they looked at me and said “why did we buy this?”. I didn’t even sell it to them! I thought: you fucking idiots, you’re asking me why you bought this? You’ve just coughed up a quarter of a million pounds for ClearQuest licenses and you haven’t tested that it works! Where is the due diligence? And I find this pretty typical. ClearQuest is chosen using a stock list of “requirements” – usability and maintainability do not figure in them, or the managers are too apathetic and credulous to test it themselves.
I didn’t set out to compare ClearQuest with anything else, although my next post might be called “What’s right with JIRA”. I have been dealing with Atlassian and Perforce recently, and what is refreshing is that they give you one, honest price for their software, with a volume discount. When buying from IBM you have to “negotiate” a discount, have lunch with them and so on. Possibly it’s the old-fashioned way of doing business, but I certainly prefer honest, upfront pricing. In practice IBM tend to give everyone at least a 25% discount from what I’ve seen, which probably gives naive IT managers a warm fuzzy feeling that they’ve got this great deal. They don’t consider the asking price of ClearQuest: USD 4,810 plus 1,720 for a MultiSite licence. Let’s get this in perspective here – I’ve been on sites that have 7000 ClearQuest and ClearQuest MultSite licences. Let’s say they get a 25% discount, that’s 34 MILLION UNITED STATES DOLLARS A YEAR. For some clunky, ugly, grey blob, that’s a throwback to the early 90s. No doubt they will be on some enterprise licence plan, but you better believe, they are paying in the millions of dollars for ClearQuest. Per year.
Contrast that with JIRA which costs USD 4,800, plus half that in maintenance in subsequent years. That’s not per user, that’s for one instance which can host as many projects and users as you like. And you don’t need more than one instance, because it has real functioning issue and project level security. Unlike ClearQuest.
What went wrong
It wasn’t always like this – in the beginning, ClearQuest (created by some long-forgotten company that sold it to Rational) was a fairly innovative (for the early 90s) tool having good support for custom processes. If you want to know what ClearQuest version 1 looked like, just fire up ClearQuest version 7, and viola. It’s pretty much the same. Still many of the same bugs. However, back then, the API was pretty complete, it did mostly what it said on the tin. There wasn’t much competition around then. However this is now, and ClearQuest has stagnated. It’s been overtaken by everyone from the commercial products, eg JIRA and TeamTrack, to open source freebies like TRAC. Yet people still pay money hand over fist for it.
I don’t blame the developers, they seemed a fairly nice bunch. And there are only a few of them. Although the demise has been long and steady, the acquisition of IBM seems to have accelerated it, perhaps this is because of the absolute and utter confusion over the ClearQuest roadmap.
In the beginning there was only a win32 native app, written with Microsoft’s MFC, which was later ported to Unix using MainWin. Then came CQWeb, an ASP application. Most users at the company I worked at at the time thought this had been written by an intern as a summer project. It was that bad. Then came an RCP-based client, available as a separate download. A few years ago, I was told that all new features were going in to the RCP client. About the same time a new improved CQWeb was released, a J2EE app that was a vast improvement. In ClearQuest 7, the RCP client is shipped as well as the native client, and appears to be the default. What I have heard from people who went to last year’s RUC is that the focus now is on the next version of CQWeb, all other clients and ClearQuest MultiSite are superceded and will be end of life’d. In truth though, nothing is going to be end of lifed whilst the sheep are handing over their hard-earned.
And then, about 18 months ago, came Jazz. There is a huge overlap between Jazz and ClearQuest. If you were a high-up manager at IBM, where would you be putting your development resources? The interesting stuff will be in Jazz, ClearQuest will be left to wither on the vine, as it should. At RUC 2006 some of the developers gave the first demo of Jazz. It was more like a pop concert than a software expo, with people taking pictures of the screen with their phones. The demo was excellent, extremely fluid, the product looked great. Then they said, “we will show you how it integrates with ClearQuest”. They opened up a record, and instead of this nice form where you can use rich text, and link records with hyperlinks and so on, after a few seconds this big ugly grey clearquest form hove into view. Who is gonna cripple Jazz by integrating it with ClearQuest and CC? No doubt the developers were forced to provide the integration so as not to scare off the customers of the cash cow.
Anyway, all well and good, but what are the people that support ClearQuest supposed to make of this? Do they merrily support new clients as they are shipped? Not simple at all – all of the 3 clients have functional differences, for instance in the RCP client you cannot drill-down through charts, or create reports. Labels have different wrapping behaviour in each client, so if you use labels to give help on forms, you need to synchronise your upgrade with schema releases. But how is the corporate support dept supposed to support all these clients? When someone asks a question, the support guy has ask them what client they’re using, and oh err, in fact can you go back to the native client and do whatever it was you were trying to do there.
Not to mention that the ClearQuest installer is now about 1.5 Gb uncompressed because of all this gubbins. Listen, there is no magic in ClearQuest, it’s just a bug-tracker. Whereas there is some magic in say the MVFS – I accept there is some clever code in there, but ClearQuest is just a bug-tracker, and not a very good one at that. Is a 1.5 Gb installer acceptable to you?
Have you called Rational support recently? In the UK at any rate, it’s a heart-breaking experience. You will speak to someone who has probably done an internal ClearQuest training course, but they will never have used it in a real organisation, so they will never have experienced any of the real-world problems people come up against all the time, or the problems of scale.
In the mid 90s I was doing a migration from a custom defect tracker to ClearQuest, of about ten thousand records. There was a fairly severe bug whereby the importer would strip all line breaks from long text fields, such as the description. I called support. A couple of days later I got the response. I wish I had kept the original email, because I still dine out on this story. The response was roughly: “Open ClearQuest. Edit the issue. Put the cursor where you want a new line. Press enter. Repeat for all records”.
And no, this wasn’t a misunderstanding. This is what happens when you have zero real-world experience of the tool that you are supporting. And the support is no better in 2008.
The other favourite tactic of IBM Rational support is to stall you by making you gather information that can’t be in any way relevant, or by trying to get you to test without your virus checker running. Or the ClearCase people will say, “sorry, Samba is not supported, or this or that clustering system is not supported”, even though everyone uses them and you are reporting what is patently a client problem.
What they do not do is make any attempt to reproduce the problem themselves. If you sent them a schema dump file I doubt they would know what to do with it.
What is deeply deeply demoralising is their lack of follow-up or tracking of your problem tickets. I have reported a problem from Big Bank A, and they say, “ok great, I’ll log a problem ticket”. I then go to Big Bank B, who are being equally hit by the defect, so, thinking that the weight of numbers might bump up the priority, I report the problem from Big Bank B. And they tell me “you are the first person that has reported that“. I used to spend a long time writing good quality bug reports with exact steps to reproduce, and they basically just binned them. How’s that for a kick in the teeth. IBM acknowledged privately that they have a massive backlog of customer issues and they are trying to do something about it now, but it’s not acceptable. People need to vote with their feet, in my opinion IBM Rational are not deserving of any loyalty at all.
Look, they’re not all bad, if you can get your problem escalated there are good people both in the Netherlands and in Lexington, but not enough to go round. And why should you always have to escalate to get anywhere?
If you call or mail Perforce support, you will be answered by someone who is well-trained, and probably has experience in the real world. They will be able to string a sentence together, in English. You give them your client and server version, then they will go away and reproduce the problem, without constant prodding. They will work on the problem until you have a work-around, and ensure you are happy with it. If it is a bug they will produce a meaningful bug report for the development team. Calling Perforce support is a pleasant surprise, but really, it should always be like that.
Atlassian goes one further, and are completely open about all their bugs and feature requests. They get huge involvement from their community of users, who suggest workarounds and ways to improve the product. I’m sure they deflect masses of their support calls by having an open instance of their defect tracker.
How bad can it be…
So what is wrong with ClearQuest? I don’t know where to start. There are big problems and small problems, and they seem to stem from:
- the complete lack of strategic direction for ClearQuest
- the fact that IBM add “features” so they can get more customers, at the expense of fixing existing problems
- fear of turning people off by making an upgrade hard, eg by bumping the underlying schema (the metaschema)
Let’s take database performance. ClearQuest supports 3 or 4 “industrial” databases. It talks to them via ODBC. But rather than using anything so new-fangled as stored procs, or prepared statements, it sends fresh batches of SQL over the network to the database, which has to parse each query afresh.
I had a problem where every time you changed a record in the results set, there would be a 3 second delay. I turned on SQL tracing at the database level, to see that for each record you select, it does a query on the main defect table, and joins it with a table corresponding to every reference field, and every reference field that has! The parse and compile time was well over a second. A Defect had a reference to Project, which had tons of referenced records on it, mostly configuration information. By using a dummy Project record with no references, I worked around it.
The argument served up by Rational is normally that they can’t use anything so esoteric as prepared statements because their usage will vary too much from database vendor to vendor. But this is bollocks – JIRA supports a whole range of databases and uses prepared statements. Have they done all this coding and testing themselves? Of course not, they’ve been sensible and used an industry-standard database abstraction layer which creates all the necessary prepared statements on application startup. The result is that jira is blisteringly fast, and supports many databases, including free ones.
OK, so we’ve established that performance is bad, you will need a fast LAN connection to make the native client usable. But the decision not to use prepared statements begat the need for a web interface for people over the WAN, so bang, we’re got our multiple interfaces problem. Now, if you’re unfortunate enough to have invested in ClearCase UCM, you cannot use the web interface, as the ClearQuest bit is embedded in the ClearCase client. You need to have native access. You only have one alternative, go and spend more money on ClearQuest MultiSite.
ClearQuest MultiSite just does not work. On synchronisation, you will get all sorts of weird and wonderful assertion failures, about mulocks or “failed num_rows != 1″, or master_uuids, or your bunch of 4096 IDs being burnt up, and so on. The developers have not bothered to trap errors and display a sensible error message, all you get is assertion failures with a message that only makes sense to the developers that wrote it. And if anyone knows where I can go to see their beaten and bloody bodies I’d be grateful.
Even when it works it doesn’t work. By which I mean, let’s say you manage to get it set up, it’s syncing up without intervention every twenty minutes. This is where your problems begin. A developer in Tokyo looks in ClearQuest, and sees a new bug. He has some information on the bug, so he think’s he’ll add a comment. But oh dear, he can’t, the record is mastered in New York. So he mails a New York developer and asks him to change the mastership to Tokyo. But they’re in different time zones, they’re 15 hours apart, so Japanese developer has to wait until the next day to modify the record. By which time he’s completely fucking forgotten what he was going to say.
Yes, there are hacks available here. You can use a record script alias to pop up the record in CQWeb at the mastering site. But oops, that won’t work if you’re already using CQWeb, and probably won’t work on the RCP client. And why should the developer have to deal with two different interfaces, just to add a single little comment? I’ve even implemented a request for mastership scheme, which took two sync cycles to get mastership over, but these are just hacks, nasty grotesque hacks.
The point is the tool breaks the user’s flow of thought, it forces them to do non-intuitive things just to update a record. The tool, the limitations of the tool in fact, defines the process. The lunatic is in complete control of the asylum.
And you need to pay them extra money to use ClearQuest MultiSite.
Out of the Box Schemas
For a tool that is continually touted as being “Enterprise”, it is distinctly un-enterprisey. I’m not sure what Enterprise means, but let’s start with an ability scale to large number of users. In my opinion, ClearQuest does not successfully scale above around 50 users.
Take a look at the query editor, if you want to filter by user, it will show you all users in your database:
There is no ability to filter by group or project role membership, without doing a bunch of joins. Have you tried to use the dialog above? It’s the most counter-intuitive thing you’ve ever seen. Click on a name and it just highlights it, you realise you need to click on a checkbox so you do that, it highlights a name and selects it. You try to uncheck a name and it unchecks all the highlighted ones.
And what in the name of all that’s holy is this dialog about?
You don’t select an item and click Add, you need to check it. Forget everything you ever thought you knew about Windows applications, you are now a ClearQuest admin.
Take a look at what is laughably called the Enterprise schema. You would think that there would be some way to break a Project down in to parts, whether you call them Components or Subsystems or whatever. And that you could associate these components with a project. You have to code that functionality yourself, and time and again I’ve seen the lists of components for projects actually hard-coded in the schema. Every change to the components requires a schema upgrade. And by the way, when you get up to six or seven hundred schema versions you will have serious performance problems when checking out and upgrading.
The “Enterprise” schema lacks support for something as basic as Roles. That is, I am a developer in project A, a manager in project B, and have no permissions in project C. The IBM training would you have you set up groups called developers, managers and so on, but clearly this doesn’t support the scenario above. Even if you use one group per project, the problem is only security admins can change this – then they have permissions over all groups. If you want to scale to greater than 50 users you will need to code your own role-based schema. Shouldn’t this be part of the “Enterprise” schema? Or distributed as a package so that you can upgrade when they fix the bugs? And don’t even get me started on the package mechanism.
Now, we’ve implemented standard schemas that has these enterprise features, and you can buy it if you want to. Great for us, bad for companies that need to hire people to implement it for them, who then leave. All of the features that you would expect must be written by you. And IBM tell you about the problems with the designer, they say, “look, it’s not an IDE”. But you’re forced to use it as one.
I was talking about the roadmap earlier, as far as reporting is concerned, it’s even more of a labyrinth. First Crystal Reports was bundled with ClearQuest. Then (following some contractual dispute I imagine) it was completely removed, but you could add the runtime engine back in. Then a version of the runtime is distributed, but it’s different from the previous version, and not forwards or backwards compatible, so reports written with 8.5 only work with that version of the runtime, and the same with 10. Either way, should you actually want to write a report, you need to buy Crystal and install it. And let me tell you, Crystal is a dog of a product. Even to print a record, you will need to buy Crystal Reports.
And when it is all working, your reports can only be based on a single query, and that must not be a raw SQL query, so only simple ANDs and ORs please, no relative dates, or field X <> field Y, you know, stuff that you might be used to if you’ve ever used a DBMS after around 1970.
Reporting is essential for what is a database with a bit of business process and a front-end glued on, yet the tools are completely inadequate. You can run a query and see the results in a tabular format, or run a chart, or several charts, but all in separate windows.
Compare with jira dashboards. All the information for a project, in one place. If a picture is worth a thousand words, that’s gotta be worth about a million.
I think IBM are well aware of the problems with Crystal. A couple of years ago they were bigging up Eclipse BIRT, which is a perfectly inoffensive and free reporting tool. But then they didn’t follow it up by integrating it with ClearQuest in any way. All they really need, is a new workspace item which would be a bookmark (“favourite” in Microsoft language), and embed a browser. You click the workspace item and the report opens, they could use parameters as well and pass them url-encoded to the target. The problem at the moment with using anything that is not Crystal is users need to go off somewhere else to look for their reports, meaning a disconnected experience.
Much as I dislike Joel Spolsky, the pompous tyrant, he has written some interesting articles on software usability that I can’t find right now. The point is that badly designed software makes you feel sad when you try to use it:
In fact if you’re extremely lucky, your unusable design will do nothing more than make people sad. They’ll try to accomplish things, and either fail, or struggle, and for very real reasons this will literally make them unhappy.
When I use ClearQuest I feel fucking suicidal. Open up a query. At the top of the screen is a button that says “Run Query”. That little underlined R means you press Alt-r to activate the button. It doesn’t work. It’s never worked. How hard is that as a developer to screw that functionality up? Press F2 when a workspace item is selected. You should be able to rename it. You can’t. You have to unlearn alot of this standard windows crap that has built up in your brain when you use ClearQuest.
What else… you can’t reorder columns in your result set, you have to delete them all then add them back in. If you have a listbox, it retains order but there is no way to order your items apart from deleting and adding back.
There’s no MessageBox function that you can use in the native client AND in CQWeb. That means, if you want to deny someone a particular action, all you can do is pop-up the unhelpful message “Permission denied for Action Close for user bob“. When you want to say “Permission denied to Close this release until all linked approval records signed off“.
There’s no revert event to hook into, so if you create linked records as part of the initialisation, you can’t delete them, they’re just stuck as orphans. I suspect a revert hook would require a metaschema change, so don’t hold your breath on that one.
No support for float fields – so you can’t enter 4.5. So people use a String field, which means the database can’t do arithmetic operations on this field without casting it. Oh, you can’t cast it unless you do raw SQL queries… and then you can’t use that query as the basis for a report.
What makes me sick is that IBM won’t implement something until they need it. And then they probably won’t document or support its usage. Say you have a field Affected Releases, which is a reference list to a Release type. You’re familiar with the parent child control that will let you add a Release record say. Wouldn’t it be nice if you could make that control display only released releases, and for your Target Releases field, display only unreleased versions. But by default it will display ALL releases for ALL projects. Yes you can navigate down to a query and use that, but you have to know that this query is there. Many people have reported this as a problem. However nothing is done about it. Possibly there is no one looking at the tickets who says “all of these are symptoms of the same problem”. But anyway…
If you use CQTM, I can’t remember what type it is, but you’ll associate something like a test configuration with an iteration. It’s the same control, but when you do this, lo and behold, it only shows the relevant test configs for that iteration. I asked someone from IBM if that trick was available for general use, and it’s not. They will only implement something when it suits their needs, screw the customer.
Likewise embedding COM components in the ClearQuest GUI. You need to extend a form to support your business requirements, for instance you need a treeview, or anythng that is not in the standard palette. You can do it, but there is bugger all documentation, and it’s not going to work on the RCP or web clients. The only reason you can do it is because IBM needed to for the control that displays the changeset.
You want to pop up a new form in ClearQuest, for instance to implement your Clone record function? Then you need the CLEARQUEST.APPLICATION2 COM interface. But it’s not documented, and the official advice is not to use it as it may change or be removed at any moment.
I could go on and on about the myriad defects and shortcomings of ClearQuest, but really, this is already far too long and I bored even myself with that last section. At one stage I cared about improving ClearQuest, so I spent energy on giving them feedback. Now I hope it just curls up and dies quietly.
If you’re using ClearQuest you should take a look at the competition, if only to satisfy yourself that ClearQuest is the best out there for your money. Once you have chosen something else, I wouldn’t be too concerned about the complexity of a migration, in truth it’s quite straightforward. Once you get round to migrating away from ClearCase, the defect-tracker migration is easy.
Don’t expect too much thanks though – in the topsy-turvy world of corporate politics, the person that owns the $10 million contract with IBM gets the plaudits and recognition, not the person that saves that figure, and oh, increases developer productivity on the side. Your reward will just be the thanks you get from the users, and they will be hugely grateful.