'

Why I (More or Less) Trust the Nielsen Ratings

Categories: '

Written By

February 24th, 2008

We write about ratings all the time and sometimes the ratings for shows that we love aren't so good.   I love 30 Rock, you love Jericho or Moonlight (good news Moonlight fans, it  has the best ratings of all 3!).

Inevitably, we wind up getting comments and e-mail about how flawed, unreliable and untrustworthy the Nielsen counting is. 

I understand it, but after careful consideration and a little bit of research I think people who complain about Nielsen probably are going to need to get over it.

Over time, there will be better data.  Both by Nielsen and others.  More and more data will be available via set-top boxes, but in round numbers, only about 50% of the homes currently have any kind of set-top box. A  significant portion of the viewing population would be excluded by exclusive reliance on set-top box data. 

I don't think Nielsen's approach is perfect, or even close to perfect.  Not perfect by any measure, just like you and just like me.  But you know what?  Just like you and just like me, though we aren't perfect, it's good enough.  The TV networks say it's good enough, and the people buying the advertising time say it's good enough.   

That's right: the people who sell the product that pays for everything and the people who buy the product that pays for everything say it's good enough.  

Bill G. will vouch that sometimes I can be a little crazy.  If you need more references, that's easy for me to compile.  I wanted some anecdotal data so I randomly stopped people in public and pelted them with the following questions:

  • 1. Ever heard of the Super Bowl?
  • 2. What's the Super Bowl about?
  • 3. Ever heard of the Academy Awards/Oscars?
  • 4. What are the Academy Awards/Oscars About?
  • 5. Ever heard of the TV show Jericho?
  • 6. What's Jericho about?
  • 7. Ever heard of the TV show Moonlight?
  • 8. What's Moonlight About?
  • 9. Ever heard of American Idol?
  • 10. What's American Idol about?

I compiled the results.  If  you don't like my results, sorrry.  I don't like that I can't reel in Giselle Bundchen, but I got over it.   Or hopefully will someday soon.

There was 100% awareness of both the Super Bowl and the Academy Awards.  And if I accept  variants of "some sports thingy" as a correct answer, 100% of the people polled knew what the Super Bowl was.  This was not true in the case of the Oscars where although everyone had heard of it, only 88% could correctly state it was an awards show for the movie industry.

14% of the respondents said they'd heard of Jericho, but only 43% of that 14% could actually explain what Jericho was about.

Only 6% said they'd ever heard of Moonlight, of those 67% were able to correctly describe what the show is about.

But when it came to American Idol, 100% had heard of it, and if I accept "some talent show thingy" as an answer, 100% knew what the show was about.

If you want to complain about how networks decide on programming, timeslots, time and money spent on promotion - that's all fair and probably even productive for the networks if they're checking in.  

If you want to question the values of the United States of America based on what programs attract the most viewers, that's a values discussion that doesn't really have anything to do with Nielsen.  We're fine with people talking about that here, but I find that sort of discussion more enjoyable if you involve beer.

My survey was fairly random.  I didn't show up at the advance Jericho screening at Wondercon and poll the people there or Jericho would have fared dramatically better. That would have been a completely inaccurate sampling, too.  

As imperfect as Nielsen's sampling may be, with any and all  flaws, I still trust it more than my own random sampling.  But even if I didn't, it wouldn't mater.  It's good enough for the advertisers and the broadcast networks.  Until any of that changes, I bet on inertia.  Nothing will change, including the complaints about Nielsen when it comes to low-rated shows and shows on the bubble.

 
  • http://tvbythenumbers.com Robert Seidman

    Seymour, I’m 85% sure I knew that. By the way, just announced by Nielsen: a $3.5 million study to track “all-day media usage”. They’re actually going to follow people around!

  • Saving$$$

    Did you know with an S-video cord everything you download can be viewed on your tv. I got rid of cable a long time ago. I save over $1000/yr because of it. I spend about $6-8 a week for the shows I like. It is superior quality and I can watch the video file over and over. I will never go back to conventional viewing.

  • Seymour

    That will be a very useful study. I know I spent over 3 hours watching “Better know a…” on the Colbert Report online last weekend. I stopped because I got tired of the same 10 second commercials.

  • http://tvbythenumbers.com Robert Seidman

    Peter, we're pretty well aligned. While almost all of my guy friends watched Shaquille O'Neal's debut with the Phoenix Suns when they played the Lakers last Wednesday (Kobe vs. Shaq drama!) ESPN had 3.6 million viewers and another 450K or so watched it on the local FSN in Arizona. So roughly 4 million people or about 1/7th the number who watched American Idol that night.

    My 15 friends did not have that much impact! For me personally, Kobe vs. Shaq is way better drama than Ryan Seacrest vs. Simon, but my personal preferences do not mirror the broader population. After 45 years though, I'm starting to get used to it. ;)

  • http://tvbythenumbers.com Robert Seidman

    Peter, we're pretty well aligned. While almost all of my guy friends watched Shaquille O'Neal's debut with the Phoenix Suns when they played the Lakers last Wednesday (Kobe vs. Shaq drama!) ESPN had 3.6 million viewers and another 450K or so watched it on the local FSN in Arizona. So roughly 4 million people or about 1/7th the number who watched American Idol that night.

    My 15 friends did not have that much impact! For me personally, Kobe vs. Shaq is way better drama than Ryan Seacrest vs. Simon, but my personal preferences do not mirror the broader population. After 45 years though, I'm starting to get used to it. ;)

  • Holly

    I can see legitimate arguments about the size of Nieslen's sample or the method of choosing the sample, but there is nothing exceptionally inaccurate or outdated about using a sample. That's how social research is done. Actually, that's how most science is done. If a needle biopsy doesn't show any cancerous cells, you assume the growth isn't cancerous. When reports say that the US has a 99% literacy rate, it's not because they actually check whether every single person in the US could read. Both are based on sampling.

    Argue that they need a larger sample (I think they may be adding more people to the sample in the next couple of years, but I'm not sure). Argue that they need to reevaluate how they do their sampling (Do they contact people by phone? If so, are cell phones included? Does their sample include people from more rural areas or is it just major cities? Are cities like NYC and LA over-represented?) Argue about how viewing is counted (How is a viewer counted if they flip between two shows? Should Live+Same Day be changed to Live+Next Day? Should DVR viewing be included since most DVR viewers don't watch commercials? etc.).

    Claiming that the Nielsen's are invalid because every single TV viewer is not personally counted shows complete ignorance of how market research is conducted. The increased precision of the data would not outweigh the added expense and man power necessary to include every viewer.

  • Holly

    I can see legitimate arguments about the size of Nieslen's sample or the method of choosing the sample, but there is nothing exceptionally inaccurate or outdated about using a sample. That's how social research is done. Actually, that's how most science is done. If a needle biopsy doesn't show any cancerous cells, you assume the growth isn't cancerous. When reports say that the US has a 99% literacy rate, it's not because they actually check whether every single person in the US could read. Both are based on sampling.

    Argue that they need a larger sample (I think they may be adding more people to the sample in the next couple of years, but I'm not sure). Argue that they need to reevaluate how they do their sampling (Do they contact people by phone? If so, are cell phones included? Does their sample include people from more rural areas or is it just major cities? Are cities like NYC and LA over-represented?) Argue about how viewing is counted (How is a viewer counted if they flip between two shows? Should Live+Same Day be changed to Live+Next Day? Should DVR viewing be included since most DVR viewers don't watch commercials? etc.).

    Claiming that the Nielsen's are invalid because every single TV viewer is not personally counted shows complete ignorance of how market research is conducted. The increased precision of the data would not outweigh the added expense and man power necessary to include every viewer.

  • http://peter.jemts.com Peter

    Ok first off, that junk post is hilarious. Next, I think it’s important not to ask about House or Lost (these have also jumped above TV itself) and use NCIS. Personally, I love NCIS, but I don’t know a single other person that watches it, or even has heard of it and knows what NCIS stands for. No one uses “NCIS” in pop culture the way “CSI” is used (which comes up in movies like Superbad).

    Everyone should really stop complaining about bad sampling data. You have two choices, either you can get a little clicker in which you can tell “Big Brother” what you’re watching all day. Or they can bug and monitor every piece of your life constantly. I’m pretty sure people don’t like the latter, and the former… well people always lie, so I don’t think we can trust that data.

    For all I know every person in Texas turns on CSI and every senior turns on CBS at 8pm and falls asleep so every crime drama on CBS gets big ratings while they snooze. No, Nielson isn’t perfectly accurate, but there are over 300 million people in the US, and your 15 friends that all started watching Jericho (I’m one of those friends FYI) may not have the same impact you thought.

  • http://tvbythenumbers.com Robert Seidman

    Peter, we’re pretty well aligned. While almost all of my guy friends watched Shaquille O’Neal’s debut with the Phoenix Suns when they played the Lakers last Wednesday (Kobe vs. Shaq drama!) ESPN had 3.6 million viewers and another 450K or so watched it on the local FSN in Arizona. So roughly 4 million people or about 1/7th the number who watched American Idol that night.

    My 15 friends did not have that much impact! For me personally, Kobe vs. Shaq is way better drama than Ryan Seacrest vs. Simon, but my personal preferences do not mirror the broader population. After 45 years though, I’m starting to get used to it. ;)

  • Holly

    I can see legitimate arguments about the size of Nieslen’s sample or the method of choosing the sample, but there is nothing exceptionally inaccurate or outdated about using a sample. That’s how social research is done. Actually, that’s how most science is done. If a needle biopsy doesn’t show any cancerous cells, you assume the growth isn’t cancerous. When reports say that the US has a 99% literacy rate, it’s not because they actually check whether every single person in the US could read. Both are based on sampling.

    Argue that they need a larger sample (I think they may be adding more people to the sample in the next couple of years, but I’m not sure). Argue that they need to reevaluate how they do their sampling (Do they contact people by phone? If so, are cell phones included? Does their sample include people from more rural areas or is it just major cities? Are cities like NYC and LA over-represented?) Argue about how viewing is counted (How is a viewer counted if they flip between two shows? Should Live+Same Day be changed to Live+Next Day? Should DVR viewing be included since most DVR viewers don’t watch commercials? etc.).

    Claiming that the Nielsen’s are invalid because every single TV viewer is not personally counted shows complete ignorance of how market research is conducted. The increased precision of the data would not outweigh the added expense and man power necessary to include every viewer.

  • http://jerichomonster.blogspot.com/2008/01/bringing-guests-to-jerichon.html Jane

    Even advertisers are beginning to pull away from Nielsen. As for Jericho, people are comparing last year's viewers to this year's when viewers are down all around. As CBS found out last year Nielsen said there weren't enough viewers yet CBS found us watching online.Look at TiVo,Amazon Unboxed, iTunes, etc. There we are.

  • http://jerichomonster.blogspot.com/2008/01/bringing-guests-to-jerichon.html Jane

    Even advertisers are beginning to pull away from Nielsen. As for Jericho, people are comparing last year's viewers to this year's when viewers are down all around. As CBS found out last year Nielsen said there weren't enough viewers yet CBS found us watching online.Look at TiVo,Amazon Unboxed, iTunes, etc. There we are.

  • http://jerichomonster.blogspot.com/2008/01/bringing-guests-to-jerichon.html Jane

    Even advertisers are beginning to pull away from Nielsen. As for Jericho, people are comparing last year’s viewers to this year’s when viewers are down all around. As CBS found out last year Nielsen said there weren’t enough viewers yet CBS found us watching online.Look at TiVo,Amazon Unboxed, iTunes, etc. There we are.

  • http://www.copywriteink.blogspot.com Richard Becker

    Robert,

    Sure, you can trust less than 2 percent of the viewing audience. That is a valid personal decision. The networks do not, with Fox one of the least trusting of all.

    As for the survey you conducted, you're comparing apples and oranges. The Super Bowl and Oscars are institutions that have been around longer than they were televised. American Idol also has considerable participant engagement that transcends television.

    However, if we were to consider your survey valid, 14 percent would represent 15.5 million households, which is more than double what Nielsen says it is. So, in effect, you just proved the plight of Jericho. More people watch than Nielsen counts.

    Best,
    Rich

  • http://www.copywriteink.blogspot.com Richard Becker

    Robert,

    Sure, you can trust less than 2 percent of the viewing audience. That is a valid personal decision. The networks do not, with Fox one of the least trusting of all.

    As for the survey you conducted, you're comparing apples and oranges. The Super Bowl and Oscars are institutions that have been around longer than they were televised. American Idol also has considerable participant engagement that transcends television.

    However, if we were to consider your survey valid, 14 percent would represent 15.5 million households, which is more than double what Nielsen says it is. So, in effect, you just proved the plight of Jericho. More people watch than Nielsen counts.

    Best,
    Rich

  • http://tvbythenumbers.com Bill Gorman

    Richard,

    The Academy Awards may predate television, but the Super Bowl does not, it has been televised every year.

  • http://tvbythenumbers.com Bill Gorman

    Richard,

    The Academy Awards may predate television, but the Super Bowl does not, it has been televised every year.

  • http://www.copywriteink.blogspot.com Richard Becker

    Robert,

    Sure, you can trust less than 2 percent of the viewing audience. That is a valid personal decision. The networks do not, with Fox one of the least trusting of all.

    As for the survey you conducted, you’re comparing apples and oranges. The Super Bowl and Oscars are institutions that have been around longer than they were televised. American Idol also has considerable participant engagement that transcends television.

    However, if we were to consider your survey valid, 14 percent would represent 15.5 million households, which is more than double what Nielsen says it is. So, in effect, you just proved the plight of Jericho. More people watch than Nielsen counts.

    Best,
    Rich

  • http://tvbythenumbers.com Robert Seidman

    My survey was not intended to be valid or compare any particular show to any other particular show.

    none of the results struck me as particularly anomalous. Awareness and viewers correlate, but they are not equal. I know what AI is and what it's about. I don't watch it. Same for Big Brother 9. Again, while generally higher awarness correlates to the higher ratings, you can't say awareness = viewers and I wasn't at all implying that. I was inferring that if Jericho had higer awareness, it would probably have more viewers.

    I think the chances are good that the only thing my survey proved was…nothing. :)

  • http://tvbythenumbers.com Robert Seidman

    My survey was not intended to be valid or compare any particular show to any other particular show.

    none of the results struck me as particularly anomalous. Awareness and viewers correlate, but they are not equal. I know what AI is and what it's about. I don't watch it. Same for Big Brother 9. Again, while generally higher awarness correlates to the higher ratings, you can't say awareness = viewers and I wasn't at all implying that. I was inferring that if Jericho had higer awareness, it would probably have more viewers.

    I think the chances are good that the only thing my survey proved was…nothing. :)

© 2014 Tribune Digital Ventures