How To Properly Review an Operating System

Impetus

I read too many OS reviews: Win7, OS X, Ubuntu, Mandriva, Fedora, PCBSD, etc. The list is practically endless, and since I like operating systems, I keep reading. Most of the reviews are really quite poor, however -- the reviews end up being almost useless because the method is so screwed up and the critiques so opinionated that the review is only valid for the reviewer. The comments on these reviews are generally flames disagreeing with the author's viewpoint.

I'm going to highlight some of the major mistakes made and recommend how to make your next review bulletproof.

1. Geek Cred vs. Open Mind

Probably the most common type of review right now has a title like "I dropped Windows and used Linux for a week." The reviewer is almost always a Windows power-user who writes for a semi-major tech blog. He has a lot of Windows knowledge and normally writes as an expert on his subject. He's used to knowing what he's talking about and rarely has a task he can't complete.

When we step into a new arena, we need to leave our pride behind. Hubris doesn't help us review something we are unfamiliar with. An expert car mechanic needs to take his self-confidence down a notch when he starts training on jet engines for airplanes, wouldn't you agree? Reviewers need to realize that they are experts no longer and not to take the frustration out on an operating system just because it doesn't work exactly the way the one they've been learning for years does.

This holds true whether the transition is Windows XP to Ubuntu 8.10, Solaris to Windows Vista, or Windows XP to Windows 7. There is, of course, some room for the "Differences from $MyOldOS" section, but the majority of the review should be impartial.

2. Clicky Clicky vs. Research

Perhaps because of this power-user mentality, reviewers also seem to think that they should be able to understand everything about the system, even deep architectural details, without doing a lick of research on the new OS. Clicking around to try to find a solution may work when you know what you're doing, but it took you years to get to that point on your old OS. Clicking in new surroundings is foolhardy. Nothing works that easily. Anything with any complexity needs a manual, even your MP3 player. Read that manual before you start your review, eh?

Go on-line and read an overview and "Get started" guide from the producer. Buy a book and spend two hours reading before you dive in. Do something. Don't, however, do the computer equivalent of walking into a foreign country without having learned any of the customs or language and expect everything to be laid at your feet. You're just being a bad OS tourist.

3. Hardware Compatibility

"I booted only to find that my wireless headset and my pro sound board didn't work!" Did you look at the hardware compatibility lists before you started?

Make sure your hardware is supported. Sure, you can comment in your review about how short the compatibility list is, but review the operating system with supported components so that it works the way it was intended to. Don't piece together a Hackintosh then complain that this OS X frenzy is all BS because it didn't work for you. Borrow hardware from a friend if you need to.

4. Surface vs. Depth

The installation is not the operating system. You do not need to spend five of your ten pages detailing the installation unless that is the process you are reviewing. You do not need to spend the other five on how you failed to get WoW to work.

Instead, have a list of common activities or tasks you want to evaluate. Go on Facebook. Watch a movie. Listen to some music. Use a popular (but supported) portable music player. Print to a (supported) printer. Try to re-frame your normal publishing work-flow into the new operating system. Install an application. The list could be endless. Make the review useful for more than just yourself by including common tasks for your readers. If your readers are all DJs, those tasks will probably be completely different than mine. Make the target audience clear so that Digg boys don't flame you in the comments.

5. Realistic Expectations

Take a look at how the producer describes the OS and what niche it's supposed to fill. Is the system a dedicated firewall, like IPCop or Smoothwall? If so, judge it according to its firewall ability and configurability, not its desktop appearance. Is it touted as an audio / video production system the way Ubuntu Studio is? Then expect it to perform those tasks well but be lacking in some other areas.

Unless the OS is claimed (by the maker, not fans) to be a drop-in replacement for your old OS, don't judge it on that criteria.

Related articles by Zemanta

• Ubuntu's Shuttleworth praises Windows 7, welcomes fight (theregister.co.uk)