"Seven serious problems with 'Consumer Reports'"

[BurnOut]

Source: www.truedelta.com

1. "Serious problems"

Consumer Reports' ratings are based on the number of "serious problems" reported by its members. I have searched in vain through their annual auto issues for a definition of what counts as a serious problem.

In contrast, TrueDelta will report measures like "times in the shop" and "days in the shop." These mean what they seem to mean. If a vehicle is in the shop for something other than routine maintenance, that's serious enough.

2. Relative ratings

Consumer Reports rates each model relative to the average vehicle. As a result, the absolute number of problems a vehicle will experience remains unclear. Does an "above average" vehicle "never break?" Is a "below average" vehicle "always in the shop?"

In the absence of hard numbers, people tend to assume that the best vehicles are better than they are and that the worst vehicles are worse than they are. I recently had a vigorous discussion with the owner of a Japanese SUV. As proof of his vehicle's superior reliability, he noted that it had been the highest rated brand in Consumer Reports' 2005 auto issue. This rating was based on 2004 vehicles, which were less (usually much less) than a year old at the time. His brand's cars had had eight "serious problems" per hundred vehicles. While this was less than half the eighteen problems per hundred domestic brand vehicles, the absolute difference was just one-tenth of a serious problem per car. Another implication: few (if any) vehicles are likely to have even one serious problem this early in their lives.

This did not--and does not--strike me as anything to get wound up over. The real problem: very few people who glance through the magazine think about the absolute numbers behind the relative ratings.

In contrast, TrueDelta will report absolute ratings in a form least likely to lead to misinterpretation.

3. Ranges

Consumer Reports' rates models on a five-point scale from "much worse than average" to "much better than average" using their well-known red and black dots. More than half of domestic brand vehicle models earn an "average" rating, while many Hondas and Toyotas earn an "above average" rating. (With the average getting ever better, "much better than average" ratings have been becoming increasingly rare.)

"Average" means within twenty percent of the average, so 80 to 120 on an index with 100 being average. "Better than average" ranges from 121 to 140. So if one vehicle is "average" and another is "better than average," then the difference between them can range anywhere from a single point--totally insignificant--to 60 points--very significant. The red and black dots appear simple to understand, but they conceal far more than the convey. As a result, many readers of the magazine understand far less than they think they do.

In contrast, TrueDelta will clearly report the absolute differences between vehicles. For example, analysis of the data might find that one vehicle over the first five years of ownership will take 2.3 extra trips to the shop, for a total of 3.6 extra days.

4. Only averages

The reliability of all vehicles has been steadily improving. Currently, even the average eight-year-old domestic brand model is reported (on page 17 of the 2005 auto issue) to have fewer than one-and-a-half "serious problems" per year. Yet most people would not buy such a car because they fear it will have "lots of problems."

While perceptions are undoubtedly distorted by Consumer Reports' emphasis on relative ratings, another factor is likely involved: people are afraid of getting a lemon, an unusually troublesome car or truck. Even if the average is the same for two models, the chances of getting a lemon could be far higher for one than the other. People might fear that even as the average rate of problems for domestic vehicles comes down the odds of getting a lemon remain uncomfortably high.

Based on Consumer Reports' reported results there's no way to know one way or the other, as they only report averages. To my knowledge, they have never discussed the odds of getting an unusually good or bad example of a particular model.

In contrast, TrueDelta will report the odds of getting a lemon and the odds of getting a perfect car (in addition to reporting the average number of trips to the shop and days in the shop).

(article continued at: http://www.truedelta.com/pieces/shortcomings.php)

Comment: Seems to be the marketing stuff for a CR competitor, but I think that some valid points are made.

[griswold44]

Look up GMI member mkaresh.

[SSrevolution]

I'm glad someone finally points these things out.

[cadillacpimpin]

i read it, its true they are using out dated data collection techniques (though im no statistician, so what do i know)

[GM_Fanatic]

mkaresh will be happy to see someone posting a link to his site! I'm happy for him, his website is maturing quite rapidly.

[61695]

Who reads this rag?

[mjd1001]

I like Consumer reports for what it is....It is a tool to use when researching products, like cars. I am NOT going to make my car buying decision based only on what they say, but I do like reading what they have to say, as there may be things they point out that I would have never considered before.

I know CR doesn't have a lot of supporters on GM boards, but remember, it is a source of information. If they say something you dont' agree with, then ignore it. But the common bashing of CR that occurs on this board simply because they don't rate 'our favorite GM car' higher than the competition is not going to change the minds of import buyers.

[61BelAir]

quick question for anyone here. Who did the report some approximately 10 years back, where they rigged a Old Chevy pickup with outboard gas tanks to blow up upon collision. The reason I ask is I overheard some people talking about consumer reports and their vehicle evaluation and it got me to thinking about these reports and how biased they may or may not be.

[cfch3399]

Unfortunately the vast majority of Ameicans in the market for a car.
Despite the pretension of Americans of loving freedom and wanting to be different than anyone else, they sure do things alike. You wouldn't want to buy a car that people don't say is any good, would you? So how do you find out what others are saying about cars, what source is there that tells you if a car is any good? CR. And off they go, in droves, to bookstores and libraries around the country to be told by CR that they have to buy a Toyodugh or Licksus or Honda.

Originally Posted by 61695

Who reads this rag?

[cfch3399]

Dateline NBC. And they are still on the air. You would think that journalistic intergrity demand them to shut down their corrupt operation.

Originally Posted by 61BelAir

quick question for anyone here. Who did the report some approximately 10 years back, where they rigged a Old Chevy pickup with outboard gas tanks to blow up upon collision. The reason I ask is I overheard some people talking about consumer reports and their vehicle evaluation and it got me to thinking about these reports and how biased they may or may not be.

[jzEllis]

Originally Posted by mjd1001

I like Consumer reports for what it is....It is a tool to use when researching products, like cars. I am NOT going to make my car buying decision based only on what they say, but I do like reading what they have to say, as there may be things they point out that I would have never considered before.

I know CR doesn't have a lot of supporters on GM boards, but remember, it is a source of information. If they say something you dont' agree with, then ignore it. But the common bashing of CR that occurs on this board simply because they don't rate 'our favorite GM car' higher than the competition is not going to change the minds of import buyers.

good point. it's just unfortunate that those who know bumpkiss about cars use it as their bible in the area of things automotive.

[camaro_freak]

Originally Posted by cfch3399

Dateline NBC. And they are still on the air. You would think that journalistic intergrity demand them to shut down their corrupt operation.

unfortuately integrity has nothing to do with news reporting anymore and the media is never held accountable for their actions ... blech

[uujjj]

These 7 points about the CR survey are all legitimate criticisms. In the 1980s when the serious problems per car ratio was high, some of these weren't such a big deal. In particular, #2 and #3 weren't problems. These days, though, 80% of cars are very reliable and just a few are not, and trying to distinguish between two very reliable cars based on reliability is splitting hairs and does the consumer no favors.

[uujjj]

Originally Posted by cfch3399

Unfortunately the vast majority of Ameicans in the market for a car.
Despite the pretension of Americans of loving freedom and wanting to be different than anyone else, they sure do things alike. You wouldn't want to buy a car that people don't say is any good, would you? So how do you find out what others are saying about cars, what source is there that tells you if a car is any good? CR. And off they go, in droves, to bookstores and libraries around the country to be told by CR that they have to buy a Toyodugh or Licksus or Honda.

What you are advocating isn't any better. Consumers do need a source of basic information about the cars they are considering. Consumer Reports provides an absolutely essential service. It just doesn't provide it very well.

wanting information about reliability != hating freedom, doofus

[Geotpf]

Originally Posted by 61BelAir

quick question for anyone here. Who did the report some approximately 10 years back, where they rigged a Old Chevy pickup with outboard gas tanks to blow up upon collision. The reason I ask is I overheard some people talking about consumer reports and their vehicle evaluation and it got me to thinking about these reports and how biased they may or may not be.

This was done by Dateline, a NBC TV program which has absolutely no connection to Consumer Reports.