Consumerists: The New Car-Salespeople

[mkaresh]

Yesterday was a stressful day for reasons well beyond this editorial. After a night to sleep on it and take another look, it is even clearer to me that while some of your points are off--specifically your understanding of normality and skewness--some are constructive. I do not really think of it as an "attack."

My main problem remains that I had no idea it was coming. I continue to find it odd that you never even hinted that such a piece was in the works, or asked me any questions about my methodology.

I added the best-worst graphs at the suggestion of people who, like yourself, said I needed something to enable people to quickly comprehend the results. They are a solution to one of your criticisms. Clearly not a solution you like, but one that a number of people have emailed me to say they like.

Though the graphs are there, the repair rate numbers are also there. So I think you go too far in saying the graphs makes me just like everyone else.

Confidence intervals cannot go on the main page for the simple reason that over 90 percent of people would not be able to understand them, and it's not possible to teach intro to statistics when posting results.

I am trying to sort out how to provide additional detail for those who want it, while not overloading those who don't. Going to a second page with the confidence intervals is the start of this. You'll also find many articles in the blog--don't know if you've read thoroughly there--under the subject of "research methods."

[goblue]

Originally Posted by mkaresh

I'm not saying that driving style and road conditions don't have an impact. I'm just saying that the impact is smaller than you imply. FWIW, alignments and brake jobs are excluded from my analysis because they are two repairs that tend to be heavily affected by these extraneous variables. Brake jobs clearly have a significant impact on some others' results.

The brands that you'd expect to benefit from driving style are those driven by the oldest drivers: Buick, Lincoln, Mercury. Do you really want to argue that the standing of these brands is due to driving style? Or does this variable only apply when it suits you?

It's simply false that a sample size in the thousands is necessary for accurate results, at least not with regard to what I'm measuring and how precisely I'm trying to measure it. If you wanted to know the rate of A/C compressor failure within one percentage point then, yes, you should probably have a sample size in the thousands. But I'm not doing that.

Would I like to report more detailed information and include more variables? Absolutely, and I plan to. But that's a goal that must be worked toward. I am very consciously working within the limits of my methods and sample sizes.

In general, my reviews track with others out there. And there's no big conspiracy where reviewers all get together and decide how to review a car. As I said before, I have not consistently praised or panned the products of any manufacturer.

CR's survey is more susceptible to their reviews than mine is. They let people decide if a repair is serious enough to report. If they give a car a positive review, and people tend to like it partly as a result, then problems could well seem less serious.

My survey has people report every repair that is not in a list of wear items. It doesn't provide that sort of wiggle room.

Still, the effect will generally be in one direction: under-reporting of problems that occur. If the cars simply don't have problems, then there's no impact. People aren't making up a significant number of problems.

I've been recommending the Ford Freestyle for a couple of years now. Has this affected survey responses? In my results, the 2005 has a moderately high repair rate, while the 2006 has a low repair rate. My review didn't change from one model year to the next. I don't think people drive the 2006s less aggressively or over smoother roads, either.

What I do think happened is that Ford really got it's act together on this car after the first model year.

I wasn't saying that there was only one definition of bias. You'll never like anything where GM doesn't look great, and that is the simplest form of bias.

In general, people with strong biases tend to perceive me as biased. It doesn't matter what their personal bias is. I'm been called biased by fans of import brands as well as fans of domestic brands. Strongly biased people want all of the results to come out their way. And reality is rarely if ever that clean.

Personally, I could care less what your results are showing, I've keyed off the claims that you are substantially better than CR (probably) and JD (probably not) I've bashed GM vehicles repeatidly and substantially over the years. In regards to Buick - thats precisely why I think Buick shows up so high, and Pontiac is lower - even though in the past these vehicles came out of the same plants with the same platforms, powertrains, and much of the same equipment. I've never believed Buick is as high as it's showed. So, yes - I would argue that Buick is high because of driving style. And you know what, that might imply why Toyota is up there too - to a small extent, ie its not the only reason. Up until recently, the only way to get cheap hp was a domestic, and 80-85% take rates on the 4 cyl Accord and Camrys bolsters this claim compared to the domestics. High hp and acceleration is important to drivers who use it - in general. I'm not going to go into specifics, but one of the cars I bought new and currently own is not a GM or GM aligned product. I see your reviews falling into the same traps as some others, bashing a GM vehicle for something, and overlooking it on a toyota - for example.

Next up is maintenence, when Toyota and Honda started developing their reputations in the US, their routine maintenence requirements and rates were higher, this has been largely reported. When you have that reputation, problems may be overlooked. Now, factor in guilt (by some) over the foreign purchase, and again, problems may be minimized - no one wants to feel bad about their purchase - when the purchase was specifically motivated by reliability - I've seen this one first hand, 3 times now with three different people. Statistically, that means I'm just very observant AND lucky, or its happening to others.

I have serious concerns with all this in regards to your advertising on car enthusiast sites, and again - when you are reviewing vehicles and putting reliability info in the reviews. When you are using a randomized method and advertising across a broad medium, using multiple formats, then I will begin to look at your data.

What I do feel is that buying a car from a reputable, main line manufacturer based on reliability is silly. How you drive it and maintain it will have a much higher impact than anything. Why, because the catastrophic things happen to all manufacturers, recently in surprisingly similar rates, and the rest is more a function of maintenence and driving style.

[regfootball]

Mike, I absolutley believe in what you are trying to do with truedelta. Especially the part about quantifying the level of difference.

Gotta pretty up the web site though. It keeps getting better, but. At least its not a nightmare to navigate like edmunds is. if i want to read user reviews on edmunds, i have to go through about 8 pages to get where i want, and pack a lunch.

[Carver]

Originally Posted by asim

It looks like a couple people just picked up their statistic books. Ghrank, you use (p<0.10)? Thats nice, I usually determine it at 0.05. Anyhow I am glad people like you Ghrank and Mkaresh are questioning these traditional surveys that the big companies have. They don't report on their ways of finding their information, which I believe is imperative. The people should know how these results were founded.

Also, thanks Mkaresh for giving a confidence interval, it really puts things in perspective. =]

Both of you are somewhat lenient. In most of the social sciences .05 is the largest significance level tolerated. We usually want to see .01 and .001 and even lower.

[wthomp01]

Wow! My head is swimming after all of that discussion about TrueDelta ratings etc. Frankly, I find it fascinating. Through all of this I've been trying to piece together the methodology TrueDelta uses and so far I've got a murky picture. The reason I'm suspect of TrueDelta's findings is because of where the respondents are seemingly recruited from. I question whether the demographics of the drivers are representative of the drivers in the US. Another huge problem here is that in order to create reliability ratings where you'd be able to say something like "for all cars the average reliability rating is X and that model Y deviates from average X by Z amount" is problematic for TrueDelta. The reason is that in order to get the proper average reliability rating your sample would have to be representative of the vehicles on the road. In other words, too many GM vehicles in the sample may skew the average up or down depending on how their vehicles performed. I'm not sure if TrueDelta reports its numbers in this way but I don't believe they've got proper representation to make that type of statement definitively.

Reliability research when it relates to parts and services is NEVER entirely only about the vehicle. It is about the vehicle, the driver, the location, the miles driven annually and a whole host of considerations. Karesh stated that driving style is of little importance when evaluating reliability because it has a low impact on the ratings. That may be true as a literal term by itself (as in a hard driver vs. a soft driver) but in combination with the other factors I mentioned which, to a degree, are inherent in driving style (as a term to describe the difference in the way consumers drive; not just hard and soft; but rather an all encompasing term more indicitive of lifestyle). Driving style, the way I've termed it, absolutely has something to do with failure rates of hard parts (and all sorts of other parts) and some of those hard parts can cause the vehicle to be inoperable. Can we distinguish between a hard part failing because of driving style versus a poor quality part? No...but this would be a good reason to have a sample that is both representative of the drivers in the US and the vehicles on the road.

I've also saw that on AskPatty.com in May 2007 Karesh stated the following: "I'm not sure what it means just yet, but I've just learned that women comprise only about 6 percent of my vehicle reliability research panel at truedelta.com." If this doesn't scream bias I don't know what does. Frankly, for a guy who credentials himself to the hilt in his marketing of TrueDelta this is either very revealing of his grasp of sampling skills OR it is patronizing to the female readers of AskPatty.com. If 94% of the TrueDelta panel are males recruited from automotive enthusiast sites this basically means that TrueDelta's reliability ratings are only representative of male car enthusiasts that frequent automotive websites. We can talk about stats until we're blue in the face but if the data collection method and sampling isn't strong and representative in the first place any further discussion is mute. It follows the old addage of 'garbage in; garbage out'.

I do not think any market researcher worth their salt would look at the numbers TrueDelta is reporting, whether in line with CR or JDP or any other source, and put them in any type of report to give to management. At best, they'd be able to say this is primarily representative to male drivers of these models (and they may not be able to say that given the sources for recruiting panelists). Any top line numbers, like a comparison of Asian vs. Domestic vs. European makes or GM vs. Ford vs. Toyota would be meaningless and errant since the entire vehicle population isn't represented in the TrueDelta sample. This is also an important consumer reader-beware issue and exactly why the original poster of this thread is correct in asking for disclosure of the sample and methodology. Just like with CR and JDP, the average consumer is going to hear TrueDelta beating its drum and, not being forthright with its methodololgy and sample, all the promotion of quality and innovation of reliability ratings certainly clouds the issue and easily hides what's really behind the numbers. In the TrueDelta case, the consumer reading the ratings needs to beware.

BTW, this isn't an ad for CR or JDP. I'm not advocating anyone here. I simply don't believe that the way to get ahead is by stepping on people to get there. TrueDelta's product should stand on its own merits, sample, methodology, accuracy, etc. Remember: Beware of the man who cloaks himself in other people's weaknesses for he too has something to hide.