Introduction
Members of The Lenox Wine Club (LWC) are “veterans” – all have been drinking wine for 30+ years. In the last 23 months, the Club conducted 11 blind tastings and they shattered many pre-conceived notion. For example, the first nine found that 3-liter box wines costing $4 per 750 ML bottle equivalent were preferred to wines bottles costing $80+. Wine economists would say “nothing new there, we have been making that point for some time.[1]”
In light of these findings, the 10th tasting was different. Tasters were simply asked if they could distinguish between two very different wine types from the Piedmont region of Italy:
- Powerful tannic wines: Barolos and Barbarescos;
- Light drinking reds: Dolcettos and Barberas.
When asked to guess which two wines were the Barbaresco and Barolo, nine people chose the Barbera while only five chose the Barbaresco and only three tasters correctly selected the Barolo. And more people chose the Dolcetto as the powerful tannic wine than the Barolo!
The inability to distinguish Barolos from Dolcettos was a shock and led to the 11th and ultimate tasting event: trying to tell the difference between red and white wines.
Below, we review the results from earlier tastings and proceed to what happened at the “ultimate” tasting.
Earlier Tastings
As indicated above, the economic literature on blind tastings has documented there is no correlation between wine prices and what people like. In our tastings, a 3-liter box was included in 8 tastings: it got the top score in 6 tastings, a 2nd twice and 3rd once. And these results came against wines costing as much as $85. Highlights are presented in Table 1.
Table 1. – Box and Most Expensive Wines Scores
The expensive wines in these tasting were selected with care. For example, the Clos des Mouches Beaune (2009) was chosen because it had won the white wine tasting at the American Association of Wine Economists’ annual meeting at Princeton. It also received a 92 rating from Wine Spectator (WS). The Neyers received a 90 WS rating. The Greywacke, the Castello, and the Maximin Grunhaus all received 92 WS ratings. The Heartland received a 93 WS rating with the Alta Vista getting a 94 rating. The other two expensive wines had good reputations but were admittedly compromises resulting from Massachusetts laws that the US Supreme Court has ruled are in violation of the US Constitution.[2]
The performance of the box wines was certainly surprising. But the poor performance of the expensive, highly rated wines was even more of a shocker: four last places and two next-to-lasts!
The performance of the box wines was certainly surprising. But the poor performance of the expensive, highly rated wines was even more of a shocker: four last places and two next-to-lasts!
Special Features of the Tasting
Several wine tasting experts made convinced me to add a couple of special features to our tastings so we did.
a) Test for Taster Efficacy
Robert Hodgson is a West Coast vintner who was troubled by the different ratings his wines received from “judges”. So he developed a method to rate them.[3] It involves including a second glass of one or more of the wines being judged to the tasting. And the spread on how the two glasses of the same wines were rated gives an indication of the taster’s ability to distinguish between wines. So we adopted a modified “Hodgson” test: a second glass of one of the wines being tasted. We then looked at scores given by the tasters who had very low “Hodgson spreads”. They tended to prefer the boxes by an even greater extent than all the tasters.
b) Rank or Score?
Rankings and scores can differ. And there are legitimate arguments for rating wines by scores or rankings. This issue was discussed at some length at the American Association of Wine Economists annual meeting in Stellenbosch two summers back. [4] Dom Cicchetti make the case for scoring wines: it allows you to register your intensity of like or dislike. That sounds quite reasonable. But Neal Hulkower argues for rankings because scoring allows somewhat arbitrary factors to affect the outcomes. For example, the votes of a judge with a higher average scores will have a greater impact on the outcome than judges with lower average scores. Of course, it does not matter if ranking and scoring give the same results. And they often do. But in our 9th tasting, the ranks and scores did differ.
The Ultimate Tasting – Red vs. White
Differing taste preferences and the inability to distinguish between wines led to the 11th tasting: The question: Can members tell the difference between reds and whites? And one additional question: can members tell the difference between heavy and light? Four wines were selected for the tasting:
- A Light White – Pinot Grigio;
- A Heavy White – Chardonnay;
- A Light Red – Pinot Noir;
- A Heavy Red – Malbec.
Bota Box wines were used for each. As Table 1 shows, the Bota Box Chardonnay, Pinot Grigio, and Malbec had all won earlier tastings.
Since color was an issue in this tasting, all the drinkers had to be blindfolded. To make it easier for them, the wines were served in glasses without stems. In addition, there were two rounds of tasting. In the first, each taster had a helper who assisted them in picking up the different glasses and recording their choices. In the second round, the tasters and helpers changed places. The results are presented in Table 2. For both color and weight, a “1” means a correct choice.
An 8 in the Overall Total column on the far right means the taster guessed both color and weight correctly. And there were 3 out of 8 or 37.5% who did. Is that impressive? Keep in mind that by chance, a taster has a 50% chance of guessing the color correctly and a 50% chance of guessing the weight correctly. That means that the taster has a 25% chance of guessing both correctly without any tasting knowledge! How much did these chance guesses influence the outcome? From talking with the tasters ex-post, the answer is a lot. Tasters were amazed at how unsure of they were of their judgments.
Blank cells in the table mean wrong guesses and there were 8 misses on color and 10 on weight. 3 tasters thought Pinot Noir was white while 4 thought Pinot Grigio was a heavy wine. 7 tasters got the Pinot Grigio color right. And all but one got the Malbec weight right.
Table 2. – Results of The Lenox Wine Club’s 11th Tasting
Another way to look at this: with 8 tasters and 4 wines, there are 32 potential points for right guesses for both color and weight. That means 75% of the color guesses were right and 69% of the weight guesses were correct. Put slightly differently, there were 8 incorrect color guesses and 10 incorrect weight guesses.
The wines are listed left to right as they were served. And most LWC members started their tastings on the left. Sequencing can affect judgments. And it is generally believed the first wine tasted will appear more “powerful” than if the sequence was different.
Wine Rankings
Tasters were also asked to rank wines and the results are presented in Table 3 (low number means higher ranking). The Pinot Grigio got the highest overall ranking.
Table 3. – Wine Rankings
But as has been the case at the major tastings with “experienced tasters” and in our earlier tastings, there was very little agreement among Club members on which wines taste best. The Kendall W is generally accepted as the best measure of concordance in rank and it was only .0045 denoting no significant agreement. My explanation for the lack of agreement among tasters: nearly all wines today are “pretty good”; so how people rate wines in blind tastings depends on different individual taste preferences.
Is Such a Tasting Meaningful?
One could claim this blindfolded tasting was inappropriate because the visual aspect is an integral part of the enjoyment of wine. And I would agree. I happen to like heavy reds, and part of that is because I like their deep, red color. And for the same token, I do not like light whites – they look too “watery”. But still….
Conclusions
I learned a lot from the 11 tastings. They confirmed for me that all wines are now “pretty good” so there is no reason to pay more than $10 for a bottle of wine. So I buy 3-liter boxes for home consumption. And since I don’t like the image of drinking wine out of a box, I serve it in a decanter. When I go to a restaurant for dinner, I always ask for their least-expensive red.
So much for analysis. We drink wine for enjoyment. And fortunately, I will still get excited when a friend invites me to taste a “special” bottle of wine.
[1] Robin Goldstein, Johan Almenberg, Anna Dreber, John W. Emerson, Alexis Herschkowitsch, and Jacob Katz, “Do More Expensive Wines Taste Better? Evidence from a Large Sample of Blind Tastings”, Journal of Wine Economics, v. 3, no. 1 and Sébastien Lecocq and Michael Visser, “What Determines Wine Prices: Objective vs. Sensory Characteristics”, Journal of Wine Economics, vol. 1, no. 1. [2] As a Massachusetts citizen, I am not allowed to buy wine directly from out-of-state but only from a retailer located in the State. Retailers (including restaurants) must purchase from licensed distributers who have unexceptional selections. The distributors control the State legislature so the illegal laws remain on the books. More on this can be found here. [3] Robert T. Hodgson, “How expert are “expert” wine judges?” Journal of Wine Economics. [4] Domenic Cicchetti, “Blind Tasting of South African Wines: A Tale of Two Methodologies”, working paper no. 164, Journal of Wine Economics.