Know Thy User, For He is Not Thee

It’s easy to say but hard to do. First, it’s hard to find the right users to talk to. When you do find them and start asking, they'll often remember wrongly, or not at all, or can’t describe what they do remember. “I was just kinda using it and stuff, and it went all funny on me, like.” Or they won’t want to seem dumb by admitting that they had trouble with your app.  Interviewers commonly poison the results by asking leading questions, like “Wouldn’t it be really super cool if you could do [this or that]?”And even if you can recruit representative users (which you can’t), and get the real truth out of them (ditto);  the time of a skilled interviewer is extremely expensive so you can’t interview many users. There's no way to get statistically valid information to find out what your users are doing.

Microsoft has tried to solve this problem by instrumenting the user interface, using what they call the Customer Experience Improvement Program. In IE 7, Office 2007, and Visual Studio 2008, they ask users to opt in for data collection with the following dialog box, which the user sees at installation time:

It's an enormously powerful idea, to instrument the user interface to find out what users actually are doing. You can get data from huge numbers of users at very low cost, and not worry about their memories either. Getting them to check the box can be tricky, but apart from that it works really well. The major snag is that Microsoft won't make this code available to us, despite my repeated calls over the last year or so.

So I wrote one, and you can download it here. The block diagram looks like this:

On the server side, you'll find various queries for fetching and displaying information.

Now that you have the code, you really need someone to help you use it. There are subtleties here – which events are log worthy and which are not? The logger comes set up with some defaults, but every application is different. The Undo command is particularly problematic here.  And how do you get users to sign up for it? Microsoft reports that some of its products have as low as a 10% signup rate – actually not a bad sample if it were random, but their polite request doesn’t make it. I know how to solve that problem.

And once you’ve gotten your data, you need someone to help you make sense of it. What queries should you make into your data set? How should you segment the user population without collecting their personally identifiable data? What does it actually mean when this command gets used twice as often as that one? How do you track the source of command (did the user click print from the dialog box, or the toolbar button, or the keyboard shortcut?)

Call me at 978 356 6377, or use the contact information at the left, and I'll show you how to make this application really sing.