You Can't Blind Test Cars

[Ghrankenstein]

You Can’t Blind-Test Cars

Well You Could, But it Wouldn’t Be Good…

An Editorial by Ghrankenstein


Before I started writing automobile reviews, I wrote reviews on wines, beers, and coffees. I was in the beverage industry then, and just as I am privileged to have the opportunity to drive many wonderful vehicles that I otherwise couldn’t even touch, I got to try wines and spirits that few people ever got to buy. Those were expressions of pure pleasure, and translating the romance of the experience into words was great fun, to the point where it became part of the experience, just as it is today.

Just as we have Edmunds, Automobile, and Road and Track, we had Wine Spectator, Malt Advocate, and Robert Parker. We saw alternative stabs at beverage reviews from non-industry sources, just as we frequently see in the automotive industry. The forums on which I used to participate (back in the USENET days) frequently rung with many of the same enthusiasms and frustrations that we encounter today in our automotive discussions.

This article is about bias, why it’s necessary, and why it isn’t a bad thing. Bias is a regular topic, and often an accusation, on automotive forums including ours.

The difference between beverage tasting and driving is that you can taste beverages blindly, without prejudice. You can’t blind-test cars. That’s why bias exists, and that’s why reviewers who claim to be unbiased can never be completely so.

Two Schools of Thought

I see the landscape of automotive publication as a continuum in terms of review content and audience. For the purpose of this article, I’m rating “left” and “right” in terms of left-brain and right-brain function, rather than left-wing and right-wing politics. The left side of the continuum centers on the factual, the quantitative, and the tangible. The right side focuses on the subjective, the qualitative, and the intangible. Neither is more important than the other in the overall picture, though it can become a war when hard-earned dollars and ultimately economies come into question. I think that most of us know where we grade, and I appreciate the fact that our differences in these relative priorities have resulted in so much cheerful disagreement.

JDPower, Consumer Reports, and TrueDelta, among others, represent the left.

Lexus rules this roost. Honda, Toyota, Cadillac, and Hyundai are players. Buick, Saturn, and Chevy are trying. Seven-seaters are hot. Hybrids are the Holy Grail.

The left celebrates its lack of bias, and why not? Numbers don’t lie. We know which cars are the fastest, and if one car is faster than the other. We know which cars rate the highest in gas mileage, and we know which sticks the best to a circular skidpad. We know which cars have the fewest problems, which score the highest in safety tests, and which have the most features for the dollar. We think we can tell real wood from fake wood, but the amount of real wood is even more important. We know which vehicle has the most cubic feet, the highest tow rating, the most cup-holders, and we measure NVH in terms of interior decibels.

In the electronic age, when sales are made and lost based on spreadsheet comparisons long before showrooms are entered, automakers are competing. The horsepower war is alive and well. Hot-button features such as DVD Navigation and keyless operation have quickly infiltrated high-budget segments. Numbers of camshafts, valves, and gear ratios have become value statements. The things that can be compared across the board are the things people research.

The left needs to recognize its biases.

While the left markets its lack of bias, there is no quantitative formula for a good car. Road and Track even went so far as to develop a “fun to drive” quotient based on numerous performance and price statistics. More objectively, quantitatively similar cars can be vastly different when subjective qualities are taken into consideration. Furthermore, people prejudge vehicles based on their expectations. Consumer Reports posts “projected reliability,” while performance-oriented sources routinely speculate on horsepower and 0-60 times.

Many supposedly tangible factors are tangible. NVH can represent many things, some of them desirable, such as an engine timbre that reflects power. Handling is rated on various “excellent” to “poor” scales. So is some kind of “overall” rating. Materials are largely the same, but their assembly, their appearance, and their “quality,” are largely based on personal preference, and unblind prejudice. People find comfort in the familiar. The same principle applies to ergonomics. It’s the most clinical way to evaluate a vehicle short of a pap smear.

While a relative lack of bias is marketable, so is sales, and that means business. Business means subscription sales, web traffic, and the generation of sales leads. That means the client’s bias must also be taken into consideration. When clients are biased toward a specific product, and sales leads are worth millions, generating that business is a highly influential factor. The left needs to acknowledge the bias of its clients if it is to remain truly unbiased.

The left ultimately needs to recognize that vehicles are more than just than value statements, they are emotional statements. Of pure factual sources, TrueDelta.com has been unique in offering an editorial on the intangible value of a great vehicle. When vehicles can’t be blind-tested, the left needs to recognize this in evaluating itself, and certainly its vehicles.

Automobile, Car and Driver, and yours truly, among others, represent the right.

BMW and the Chevrolet Corvette are king here. Mazda, Nissan, and Audi are players. Saab and Pontiac are trying. Station wagons and minivans are cold. Two-seaters are the ultimate.

The right celebrates driving, and the intangible qualities of vehicles, and why not? Great vehicles aren’t made up of statistics, and so-called imperfections are frequently what separate great vehicles from the merely competent ones. When push comes to shove, they recommend the vehicles that inspire the senses. What’s fastest, quickest, and stickiest are important as base criteria; but feedback, involvement and emotivist personality are more important than tangible value factors in these opinions. One can be so convinced of the beauty and driving characteristics of a vehicle that less can equal more, if the rewards are greater.

This category doesn’t want to compromise, and it starts with groundplan. A crossover by any other name is a station wagon, and even that can be sporty in comparison to a minivan. Meanwhile, a true combi can offer performance and handling that won’t be matched by any crossover. None of these utility-oriented products can match the style and driving pleasure of a good sedan, coupe, or two-seater. No automatic can match a good stick, but really good automatics can come close.

Quantitative discrepancies are tolerable. Despite their advertising, some brands rate lower in crash safety scores. Powerbands might register impractical, but fun. In the real world a slower car in the quarter-mile, or less sticky on the skidpad, might be more rewarding in canyon run. A straight-line dragster might loaf in the corners, while a toad could handle curves with aplomb. Such great vehicles are often Spartan in terms of features, and yet retain their desirability. What do leather seats and a Navigation screen really contribute to the joy of driving?

The right needs to recognize its biases.

Nobody wants to own a piece of crap, and no vehicle is fun when it’s sitting in the shop. Few people who use their vehicles as daily drivers want to give up amenities, especially when the mundane appliance parked alongside has more. Few daily drivers want their vehicles to give up safety, and even fewer can afford to have automotive toys.

Just as important, few drivers can agree upon which intangible qualities are the most desirable. Some prefer a visceral, high-feedback experience even to the point of tail-swapping danger. Others love delicate subtlety and the driving intimacy of a familiar nuance. Each driver, each reviewer, and each owner, should emphasize that their impressions are opinion and not fact. Bias from expectation is just as relevant, as brand idiosyncrasies can be irregularly modified, and justified by subjective preferences. Few vehicles drive better than others to everyone, and non-enthusiasts often seek intangible qualities that are completely foreign to enthusiasts.

The right ultimately needs to recognize that not all vehicles are designed for them, and that they serve more needs and tastes than their own. Favorite qualities can be found in more vehicles than just the favorites.

Outside the Spectrum

Forbes. Playboy. Paul and Anita Leinert. Geraldo. Where do they dream this stuff up?

A short entry, this is the most vague barometer of the general public in existence: A handful of points within the enormous database of consumers, who happen to have access to the media. These nitwits are embraced by the automotive industry because they have already decided their opinions, based on the biases that remain unknown or unacknowledged to them, are predictable. Just as torque can be amplified, so can celebrity or “expert” endorsements.

Do they matter? For the time being, yes, as long as their flavor of the month status matters. By the time they stop mattering, that crown will have been passed on. The American Idol panel will eventually be called upon for the car tastes, and they’ll be just as useless and yet just as important because someone else thinks they’ll benefit.

These critics need to have the humility to realize that they’re only being exploited for their status. Since they don’t have the humility to stay out of politics, they won’t stay out of anything else where their opinions are valued but unmerited. This is the bias that frustrates me the most. They’re just furthering their own careers beyond the stuff I really liked (i.e. Jennifer Aniston in Leprechaun) anyway. I’m normally a fan of necessary evils, but not in this case.

We should recognize this pattern in ourselves, and when we’re blindly adhering; we’re above this. Think of the brands you “hate” the most, and remember the vehicle among them that you’d still love to drive; that’s your bias.

A Personal Conclusion

I’m the first to admit that I’m biased. As an industry employee, I have a vested interest in the success of the product. I also chose that product based on my passion, and against the industry trend of the time. Because of my employment, I drive a disproportionate number of my products over my competitors, though I frequently drive them in comparison.

I don’t have much of a choice in terms of review style, since I don’t have access to any kind of instrumentation beyond my giant green butt. I’m fortunate, in that I’d already developed a method to my madness, and a joy of interpreting the ethereal into something that is understandable at best, appreciable at worst. With the automotive landscape already offering a wealth of opinions, which I have taken as valid but opinions nonetheless, I’ve tried to offer something that consumers can use, readers can appreciate, and some information available from nowhere else. It’s all with my own clear bias, nothing more and nothing less, with ample room left for disagreement and debate.

On our forum, I’ve grown to appreciate the bias. I’ve written some harsh stuff, and some kind stuff, and all of it has been evaluated critically, and debated, much to my gratitude. We have a passionate community, and that means bias. We have an educated community, and that means bias. We have members, and enthusiastic contributors that range from the left (brained) to the right, whose opinions have made me think and rethink. I’m sure that our masters of discussion would say the same, and I can only hope that our casual readers and contributors would agree.

There is one thing we can agree on: we love a great vehicle. The motivations vary infinitely, and the personal biases are congruent. That great vehicle represents everything from the needs and budget to the dreams and aspirations of every member on GMI, and we’re inevitably going to disagree.

Regardless of the wine at the tasting, and regardless of how many clods of mucus were floating in that spit bucket, my spitting technique never failed to reward me with any less than a dollop of wine-spit right back in the face. Since I think about the cosmic implications of such things, I realized that it was never about me, and always the wine. No matter what, it was always funny.

This is the way that it should be. I’m glad we’re passionate about our vehicles. Let bias, humility, and ultimately great vehicles reign.

Sincerely,

Insane W. Ghrankenstein

Anyone who read this before I edited it (only one word): Hope you enjoyed it!

[MichiganBuick]

Well said And, yes I read the whole thing.

[commodorecoops]

Originally Posted by Ghrankenstein

Great vehicles aren’t made up of statistics, and so-called imperfections are frequently what separate great vehicles from the merely competent ones. When push comes to shove, they recommend the vehicles that inspire the senses. What’s fastest, quickest, and stickiest are important as base criteria; but feedback, involvement and emotivist personality are more important than tangible value factors in these opinions.

If only everyone thought like that...

And yes, when you do blind test cars, the result is bad. Half the world's population ends up in Toyotas...

[bone]

Agree with everything in your excellent editorial with the exception of the ,"Right's", opinion of stationwagons. They perhaps are cold to Minivans and only luke-warm to Crossovers,however,IMO,they love stationwagons,real stationwagons. Remember Car and Driver's Boss Wagons? All of the "right" media love the BMW,Mercedes,Subaru,and Volvo wagons. It should come as no surprise that North American RWD wagons from the '50s to '80s are highly thought after.

[unkillsam]

Forbes. Playboy. Paul and Anita Leinert. Geraldo. Where do they dream this stuff up?


If only it was possible to have a blind auto test. I would love to see what these people would have to say about domestic cars, GM in particular, if they didn't know who manufactured the car.

With the GMT-900's, Lamda's, and new Epsilon's, if these people had taken a blind test last year before they debuted, and were led to belive that the vehicles were either Honda's or Toyota's, there whould have been no "Well it's the best so far from GM, but X, Y, and Z are still wrong with it"

IMHO they would be sipping the import cool-aid and what not.

[SobeSVT]

Originally Posted by Ghrankenstein

JDPower, Consumer Reports, and TrueDelta, among others, represent the left.

Lexus rules this roost. Honda, Toyota, Cadillac, and Hyundai are players. Buick, Saturn, and Chevy are trying. Seven-seaters are hot. Hybrids are the Holy Grail.

Automobile, Car and Driver, and yours truly, among others, represent the right.

BMW and the Chevrolet Corvette are king here. Mazda, Nissan, and Audi are players. Saab (????) and Pontiac are trying. Station wagons and minivans are cold. Two-seaters are the ultimate.

WARNING: Ghrankenstein's left side of his brain had nothing to do with the creation of this article.

Now seriously, I agree very much with most of the artcle. I would have to say though that the right is MUCH more powerful and influential in the opinions of car aficionados. Who in GMI does really enjoy reading Consumer Report or KBB? Who doesn't enjoy reading a good Car Magazine?

The problem is though, that sometimes these Car Magazines *cough*Car*cough*and*cough*Driver* pose as beacons of reason and disguise their bias as objectivity. That is not right and is very dishonest, especially since a lot of people that are not car fans use them as the last word of what is good and bad in the car world and can't tell bias from true.

[banzai79]

I don't really have a problem with car mags having biases towards certain makes and models, I do too, so I can't blame them. The problem as I see it is that I also don't want to constantly read about how some guy really loves a certain kind of car that I'm just not into. Therefore, I no longer subscribe to most car mags, I know their opinions already, they don't change, and it's not very good journalism.

The other problem is that you are supposing the 'left brain' people don't have biases, and collect representative data during their 'testing.' Some people, like CR in particular, in no way conduct a scientific test, yet it is portrayed as such.

[tgagneguam]

Ghrankenstein, a wonderfully thoughtful, refreshingly well written piece. Thanks for the enjoyable Sunday morning read.

Incidentally, everyone carries their own past experiences, be it consciously or otherwise, into their new experiences. We are all biased as a result.

[eaton53]

I read that the 1st Aurora was blind tested and fared extremely well when people didn't know it was an Oldsmobile.

[ragingfish]

Great article!

The only car mag I ever read is autoweek. I don't particularly care for the down and dirty stats, but instead prefer just a general idea of what's happening in the industry. They do road test vehicles as well, and from the moderate number of reviews i've read, they do not seem to have a blatant bias in either direction (unlike CR who reeks of bias IMHO). I remember they did an article on solstice vs. miata, and in the end they admitted that it was a very hard competition, but IIRC (it's been a while), they gave the solstice the win.

[megan]

You will never change some peoples opinions, no matter what. Some think any Japanese car is superior to any domestic car, no matter what ratings. Blind testing would be great, but peoples bias would come back after they find out the true manufacturer.

[Martino_z]

I was going to quote the entire article in case anyone didn't read the whole thing but I won't because I know you did. This isn't commentry. This should be the bible for anyone considering buying a car new or old. We all need to consider the bias we have, but also remember the satisfaction that comes from that bias. I know I do when I'm driving my V8 holden.

Cheers Ghranko

[DeliveryDave]

Great artical as always. I don't subscribe to any but if i'm interested I'll go to a borders, barnes and noble, library what ever the case and read several at no cost! I was at a local M.T. show and for me I was most impresses with Dodge - why? Not because of the product but because of perception. I have read many articles on how bad dodge is but I was impressed, thought the outside was ugly but the roomy inside.

Biased can be good, but for the consumer information is king and for me it is important to get both sides of the bias.

Thanks as always, great read.

[Hurricane]

Very interesting article!

[GotAWD]

i gotta go shave and im only at the first half of your editorial. thank god you dont write up more stuff here dude. ill be back to finish the rest. good work though grank.

k so i just finished the article. good read and i need to shave again. anyways grank good read and i hope your employer treats you well in your job. you bring light to those that cant or dont want to get educated about the automotive world.

lates

i think he may have gone out and done some wine testeing between breaths