Linux Today - Benchmarketing 101 -- Or FUD with Numbers

Herring vs. Windows NT

Benchmarks

Herring

Windows NT

Swimming Speed:

5 mph

0.1 mph

Cost:

$1.35

$500

TCO (year):

$492.75 *

$6,180 **

Windows NT represents poor value proposition
What Consumers Want	Herring	Windows NT
Nutrition	YES	NO
Ease of Preparation	YES	NO
Pleasing Texture	YES	NO
Taste	YES	NO
Goes with crackers	YES	NO
Preferred by penguins	YES	NO
Easy to use	YES	???

Part Deaux: HUH?!

Statistics unrelated to use: The statistics gathered above are totally unrelated to the use of either Windows NT or herring as a computing platform.

Ignoring superior competitors: A shark is a faster swimmer than a herring, but we conveniently left it out of our chart.

Irrelevant features: The features gathered are totally irrelevant to the use of herring or Windows NT as a computing platform.

Leaving out statistics: We deliberately left out any statistics that might make herring look bad as a computing platform. For example, the herring's brain is less powerful at number-crunching than any known microprocessor, thus making herring a poor choice for mathematical programming. We don't mention that, just as Microsoft never mentions that NT 4.0 is ill-suited for use as a corporate firewall when compared to FreeBSD or Linux.

Conclusions not following from the facts: The conclusion reached (that herring was preferable to Windows NT as a computing platform) was not supported by any of the facts mentioned. What we are trying to do here is snow you with facts, and then slip in a conclusion and hope (FUD distraction method #1) that the fact that we've sandwiched the false statement between true facts will distract you enough so that you don't spot that it is false.

Part Tres: Conclusions

Secondly, you should examine how meaningful the benchmark is. If the benchmark says that solution (a) does 'x' operations per second, solution (b) does 'x+1000' operations per second, but only need to do 'x/10' operations per second, then the fact that one solution can do more operations per second than the other is meaningless. Both will fill your needs, and you should choose which one is most cost-effective for your situation. Why buy a Mercedes when all you need is a Honda Civic?

For example, if you're trying to set up a low-cost web server for your small business to feed a T1 line, a $499 eMachine 333 running Linux will saturate that T1 just as assuredly as a $35,000 quad-processor Dell with four network cards and Windows NT. This is a case where a benchmark (the Dell is MUCH faster than the eMachine) is irrelevant.

Finally: Be aware. Benchmarketing is nothing new. IBM was a past master of benchmarketing back in the 70's, for example (they invented 'TPS', Transactions Per Second, to quantify how their mainframes were superior to the competition). Whenever you see benchmarks in the press, there is a 90% certainty that you are seeing benchmarketing in action. Be careful out there.

Eric Lee Green is the networking and systems guru for Enhanced Software Technologies Inc., "The Bru Guys". He deeply regrets the use of a hotmail.com EMAIL address but hastily explains that he's had it since before Hotmail was sold to Microsoft, and further excuses it by saying that he'd prefer SPAM to burden Microsoft rather than his employer.