A new documentary called "America the Beautiful" is currently being screened at film festivals across the country. The film examines the effects of pop culture and the fashion industry on American ideas of beauty.
The idea for the documentary began when filmmaker Darryl Roberts conducted an informal sidewalk survey in Chicago. He asked 200 women if they felt attractive; only two said yes. In an interview about the film in the "New York Sun," Roberts said, "It's not all that complicated. I started doing the math, and if 198 women are saying 'no,' that means 99 percent of women feel unattractive." Based on that shocking finding, Roberts set about making a film to explore why almost all women feel unattractive.
Roberts surely has good intentions, but there's only one problem: his data is wrong.
In fact, studies find that about 90 percent of women say they are happy with how they look, and most consider themselves to be more attractive than average. In 1998 "USA Weekend" surveyed over a quarter of a million teens, mostly girls. Ninety-three percent reported feeling good about themselves. In 2000, the British Medical Association issued a report on eating disorders and the media that concluded, "The majority of young women (88 percent) say they are of average or above average self-confidence." And Harvard's Dr. Nancy Etcoff found in her 2004 report "The Real Truth About Beauty" (based on a survey of 3,200 women) that the majority of women described their body weight as "just right," and 88 percent said they were average or above average in attractiveness.
So what happened? Why did Roberts's poll find exactly the opposite? The answer lies in the difference between valid and flawed polling.
Bad Polls
On the surface, it seems very easy to find out what people think about a certain topic: You just ask them, right? Wrong. Welcome to the science of polling. There's a reason why polling companies are paid millions of dollars to get an accurate picture of what people think, what beliefs they hold, which products they buy and why, how they are likely to vote, and so on. Unless you are professional, it is very difficult to get meaningful poll results. The answer you get to a given question depends largely on who is asked the question, under what circumstances, and even how the question is asked.
How Polls Can Go Wrong
For one thing, there's the problem of what psychologists call demand characteristic. It basically means that people often tell researchers or questioners what they think he or she wants to hear. So, for example, before Roberts approached his 200 female subjects he introduced himself and explained what he was doing. If he said something like, "I'm making a documentary about how the fashion industry exploits women who don't feel good about themselves," the women would understand that when he asks them, "Do you feel attractive?" the answer he's looking for is probably No.
Aside from that, how many of us, men or women, are likely to tell someone who asks us as we're walking down the street that we feel attractive at that particular moment? Just because a woman might not tell a male stranger that she's attractive as she's headed home from work toward the subway doesn't mean she might not feel attractive a few hours later as she get dressed to go to dinner. If Roberts asked women on the street in front of a Weight Watchers clinic, he might get a different answer than women leaving a gym or beauty salon. Roberts's amateur street poll was badly flawed, and led him to make a film trying to explain his mistaken results.
Holocaust Poll
In one of the most infamous examples of flawed polling, a 1992 poll conducted by the Roper organization for the American Jewish Committee found that 1 in 5 Americans doubted that the Holocaust occurred. How could 22 percent of Americans report being Holocaust deniers? The answer became clear when the original question was re-examined: "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?" This awkwardly-phrased question contains a confusing double-negative which led many to report the opposite of what they believed. Embarrassed Roper officials apologized, and later polls, asking clear, unambiguous questions, found that only about 2 percent of Americans doubt the Holocaust.
Polls and surveys can provide important information about the public's beliefs. But to be valid, they must be based on sound methodologies, and news consumers should always look for information about the sample size, representativeness of the population, whether the participants were random or self-selected, and so on. When done correctly, polling is complicated.
Benjamin Radford is managing editor of the Skeptical Inquirer science magazine. He wrote about the media and pop culture in his book" Media Mythmakers: How Journalists, Activists, and Advertisers Mislead Us." His books, films, and other projects can be found on his website.
