TV ratings in the age of digital TV

Who watches the watchers?

By Mark Slutsky | Ars Technica

This year has seen two major developments in the TV market: 3D and the Web. TV makers are betting that consumers will flock to stores this holiday season to upgrade their plain old 2D and Web-less panels with models that will let them bring the Internet into their living rooms without requiring them to add another box to their entertainment center. In this four-part series on the Future of TV, Ars takes an in-depth look at the major transition that TV is currently undergoing.
 
We take a look at the past, present, and future of TV ratings. How do networks, advertisers, and agencies measure audience sizes in a world of DVRs and BitTorrent? Will DVRing kill your favorite show? 

 
TV used to be so simple. Everyone had the same basic equipment (the only real qualifying factors being whether your set was color or black-and-white, and the size of the screen), and choice of programming was limited to whatever was on one of the three big networks at the exact time you were sitting on the couch, at least for those in the United States.

Boy, have things changed.

Now what we call TV includes everything from the old-school, over-the-air broadcasts, to cable programming, to video-on-demand, to the spiraling variety of video content available online. "Television" as a descriptor has become amorphous, its meaning constantly changing depending on its context. And what's changing even more drastically is the way we watch it; we have more options than ever to watch programming whenever and wherever we want.

That's terrific for consumers, at least on the surface level. But what does it mean for the business of TV—and the future of the programs we love? As common as pay TV is these days, most broadcasting is supported largely by advertising. With viewership fragmented between on-air broadcasts, video-on-demand, Hulu, Netflix, iTunes, and—importantly—illegal downloading, is it possible for networks and producers to get accurate data about who exactly is watching their shows? Are shows with niche appeal losing out because their numbers aren't being counted accurately? We decided to take a look at how the business of TV ratings is changing in the digital age.

The 5000-Channel Universe

Before we get into the sometimes-tricky business of ratings, let's take a look at the fragmented state of television audiences, and how we got here.

In television's real heyday in the 1950s and 1960s, practically all viewers were limited to what they could pick up on their "rabbit ear" antennas, which for the most part meant their local ABC, CBS, or NBC affiliates. It's hard to overstate how incredibly concentrated audiences were at that time. The Beatles' first appearance, on February 9, 1964, was watched in about 22 million households. Compared to a modern-day, big-ticket TV broadcast like American Idol's season nine finale, watched by 16 million households, that doesn't seem like such a big deal, even when you remember that the population of the US was about two-thirds of what it is today. But look at the percentages: 14.2 percent of households watched Idol, while an amazing 60 percent had their TVs tuned to the Fab Four in 1964, a figure that's simply unimaginable today.

Things began to get a little more complex with the widespread adoption of cable, which really took off in the '70s and '80s with the popularity of stations like TBS (Ted Turner's famous "superstation"), CNN and HBO. Widespread adoption of cable, along with both legal and grey-market satellite TV, opened up the "500-channel universe," a term coined by TCI executive John Malone, whose aggressive tactics (Wired called him the "Darth Vader of the Infobahn" in 1994) helped bring hundreds of new channels into American's homes.

The Internet would provide the next comprehensive, disruptive change in the way we watch TV, but it's worth mentioning two other developments that just predated the Internet video explosion by a hair, and which happened about the same time.

But how are Nielsen, and by extension, broadcasters and advertisers, keeping track of what you watch on your computer or download from torrent sites?

The first would be the DVD home video format, born in 1995 but adopted by consumers around the turn of the century. How did DVDs change the way we watch TV? Well, in the long and painful VHS era, people rented a lot of tapes, and they occasionally used them to record TV shows, but the market for actually purchasing movies on VHS was relatively small. In fact, movies were often "priced to rent"—sold for up to $100 for a single movie, and meant to be purchased by video rental stores. That changed with DVD, which were priced to sell to consumers—a smart bet, as for various reasons (size, quality, special features) people were willing to purchase DVDs in droves.

The culture of the DVD had an intriguing and unintentional side-effect on television. With some notable exceptions (like soap operas), TV prior to the DVD was largely single-episode-based—you'd have your rare season-ender cliffhanger or two-parter, but plot arcs were largely stuffed into individual episodes. (Watch an episode of Star Trek: The Next Generation some time and marvel at just how much story they could fit into 42 minutes.) Whether it caused it or not, the mass DVD purchasing phenomenon dovetailed perfectly with the shift towards long-arc stories—plots lasting multiple episodes or even whole seasons—on shows like The Sopranos and 24. DVDs seemed perfect for that kind of entertainment: who hasn't lost whole weekends to bingeing on seasons of Lost or The Wire? As viewers showed they were willing to shell out for whole seasons at a time, programmers surely took note.

The second major shift in the way we watch TV was the introduction of the DVR, or digital video recorder (also called the PVR—the "P" stands for "personal"). Made popular by TiVo, which released its first consumer device in 1999, the DVR allows viewers to do what's known as "timeshifting" in the industry—fancy terminology for "watching it later." You could already timeshift with VHS tapes, but that involved the world's least favorite task and butt of many a punchline, VCR programming. TiVo made it easy to keep track of your favorite shows automatically—even recording them while you watched something else. The technology became so popular, in fact, that TiVo itself became a victim of its own success, and fell by the wayside as cable companies rushed to make their own DVRs. From of a high of 4.418 million subscribers in July, 2006, TiVo reported only 2.272 million in their October 2010 earnings letter.

One of the other real advantages to consumers—but not advertisers or broadcasters—was the ability to fast-forward through commercials. In fact, one TiVo competitor, ReplayTV, offered the ability to skip them entirely, but was stopped by lawsuits from the major networks and removed the feature. But that ability to avoid commercials—one of the big selling points of much (but not all) of the TV you can watch on the Internet—is still an issue today, and as we'll see below, is very much taken account of by the TV ratings people. That's also true of VOD, or video-on-demand, which allows customers to watch stuff whenever without having to even set it up in advance (although Nielsen claims VOD numbers are small enough to be negligible).

The most recent—and most unpredictable—change in TV-watching is, of course, the boom in Internet video. Everything prior to Internet still involved sitting on the couch and looking at the same piece of tech; but now you can watch stuff via the Internet on your laptop, your phone, your tablet, or even… your television. Officially sanctioned stuff plays on sites like Hulu or network websites, albeit without the regular commercials, though some broadcasters are switching to a format called "TV Everywhere" which streams everything—including ads—exactly as it would appear on your TV screen (in your local market) to your computer. Show clips are uploaded to YouTube, with and without official permission. There are Web-only shows and downloadable video podcasts—do we even call that stuff "TV"? And finally, there's the elephant in the room—illegal or questionably legal streaming and downloading, the ultimate in convenience for those who don't want to pay. There are more ways to consume TV content today than ever before. So who's keeping track of it all?

The ratings game

In the middle of the first-season run of his show Louie on FX, comedian, writer, and star Louis CK tweeted (@louisck) "If you want LOUIE (on FX Tues. 11pm) to have a 2nd season, don't DVR, don't HULU. Watch it when it's on." The assumption was, of course, that only "live" TV viewing counts to broadcasters and that Internet viewership can't support a show. Was he right? Well, the answer is a little unclear.

Television ratings as we know them are synonymous with one company—Nielsen, which created the famous "Nielsen ratings" that measure television show's viewership. For broadcasters—and the advertisers who fund them—this is crucial data, determining the desirability, and thus price, of commercial airtime.

Nielsen's most famous methodology is the "diary," in which members of selected households record their viewing habits. Frankly, it's not the most reliable-sounding method, for a variety of reasons; you might forget to keep track of the shows you're watching, you might make mistakes, you might even deliberately keep false data if you'd rather have the ratings guys think you're watching PBS NewsHour than Keeping Up with the Kardashians.

Happily, while the diary system is still used in smaller markets, the national rating system has become much more sophisticated in recent decades. The company still picks a sample of Americans (about 20,000 households in total, totaling around 50,000 people)—not a random group, mind you, but one carefully selected to represent the country's demographics.

Every time a member in a Nielsen household sits in front of their TV, they're instructed to "check in" with the meter the company installs on the set. The meter logs what's being watched—and importantly for advertisers, who's watching it—by keeping track of Nielsen audio codes, silent to human ears, that are encoded in pretty much all programming and repeated every 2.6 seconds. Then, every morning at 3am, that data is sent back to their facility and processed, with the fresh ratings spit out to their clients. The crucial info is the "C3 rating," which measures how much any specific piece of programming has been watched in the three days from when it airs.

The code system allows for extremely granular data collection. For example, if you record a show on your DVR and play it the next day, it will still show up on the C3 rating, even if you didn't watch it "live." And, most importantly to advertisers, every commercial has its own unique code, which allows Nielsen to track whether you're fast-forwarding ads or not (according to the company, DVR users still watch about 45% of the commercials).

And DVR use keeps growing. "DVR penetration is growing dramatically," says Matt O'Grady, Nielsen's Executive Vice President of Media Product Leadership. "The biggest trend we see in TV is timeshifted viewing. In 2006, 6 percent of the TV-viewing country, which is pretty much everybody that's got cable, had a DVR. In 2010, we're almost up to 40 percent."

So that's how it works for the TV in your living room. But how are Nielsen, and by extension, broadcasters and advertisers, keeping track of what you watch on your computer or download from torrent sites? Well, in April Nielsen plans to being tracking what they call "extended screen," services like TV Everywhere. But here's where it gets a little tricky: services like Hulu or Xfinity (and of course, pirated shows) won't be counted in those numbers because they don't show the same commercials as the regular broadcasts. "You can't make up different rules for different screens, unless you want to treat that screen completely independently," O'Grady explains. "So the extended screen definition is, if you want credit for traditional TV—what we call 'commercial credit' or C3 credit—and you want that for your online contribution, then your telecast online has to match what you showed on TV."

"Hulu, Xfinity or NBC.com, whatever the site is, may have lower commercial loads," he continues. "We are committed to measuring that as well, but that moves away from currency, as we say. It moves away from the ability to monetize it just as TV is monetized—but there's great value in it."

It does raise some important questions though—for many, watching TV online is a way to get away from the restrictions (and ads) of the cable model, and TV Everywhere seems like a way to just transplant the current infrastructure online. (It's already getting flack from groups like Free Press, who claim it's anti-competitive.) Whether it will take hold or not among consumers is still in question. If broadcasters make their decisions based on regular and extended screen numbers only, they might not be working with representative data.

So was Louis CK right? Yes and no. If you DVRed episodes of Louie, you probably weren't doing much harm to his show's chances. Whether watching it on Hulu had impact either way is debatable—those numbers won't go to the advertisers that are responsible for most of FX's income, but the broadcaster will be seeing some revenue there. (Either way, someone must have explained that to him, because the tweet was deleted some time later.)

But, it should be added, according to Nielsen's studies (and they don't seem to really have a horse in this race either way), the overwhelming majority of viewing is still done on TVs. "There's a very interesting phenomenon happening in the marketplace, with online video garnering so much attention—and rightfully so, a lot of concern about how people continue to monetize their content on different screens," O'Grady says. "But the majority of the viewing today is still on the principal first screen. When you look at our numbers, for TV vs. online viewing vs. mobile viewing, it's predominantly TV. That makes sense; the 'best available screen' is what we call it, meaning the best viewing experience." Specifically, Nielsen's data shows American viewers each spending on average about 158 hours a month (sounds like a lot, doesn't it?) in front of the TV, with about three to four hours each for online and mobile.

The piracy factor

What Nielsen doesn't—and probably can't—measure is the number of people watching TV via "unofficial" channels online, such as bootleg streaming video sites or p2p file-sharing. "We could probably engineer that, but we're not in the business of doing that. We don't want to be the police," O'Grady says.

And, he adds, "What Nielsen takes very seriously—which you have to in this business—is representative samples. It would be very hard to get a representative sample of people who are readily admitting themselves as viewers of pirated content! [laughs] So I don't know how we'd do that, but you never know how the world's going to change."

One person who does have their eye on those often-murky waters is Ernesto Van Der Sar, the pseudonymous blogger behind torrentfreak.com, whose weekly "Most Pirated" Top 10 lists are reprinted by industry journal The Hollywood Reporter. Van Der Sar's weekly lists mostly focus on films; he used to follow TV on that schedule but now only does round-ups only annually, because, as he says "there were generally only a few small changes from week to week."

Nonetheless he does keep track, albeit with a methodology a little cruder than that of Nielsen's—though given his resources and the data he's working with, that's by necessity. "Public BitTorrent trackers report the number of downloads," he explains. "What we do is poll all of the public trackers we can find every day and collect all the data in a huge database. We then use filters to group similar titles and extract the most downloaded titles at the end of the week. For some trackers that do not report the actual downloads, we use a combination of downloaders and the file size to accurately estimate the number of downloads."

In his year-end roundups (such as this one), Van Der Sar then compares the downloading numbers to Nielsen's. The results vary from show to show; some attract more downloaders than viewers, some quite the opposite. Torrentfreak.com's pick for top-downloaded show last year, Heroes, for example, was downloaded 6,580,000 times, while Nielsen reported 5,900,000 US viewers. Number 10 on the list, True Blood, saw 1,600,000 downloads for its estimated 12,400,000 US watchers.

It does imply a sort of shadow audience; if those numbers are accurate, Heroes was only slightly behind True Blood, despite the latter show's numbers being almost double. But that's not the whole story—it's not accurate to assume these are all US-based viewers who would otherwise be watching Heroes on cable.

"This is a difficult question, because the effect can go both ways," Van Der Sar says. "Piracy might hurt the ratings of a show marginally because people do not watch the episode through official channels, but it is doubtful that these downloaders would have seen the show at all if it weren't for piracy."

And, importantly, he adds, "Most of the downloaders come from outside the US, so these have no impact on US ratings. On the flipside, one could argue that BitTorrent has actually helped TV-shows to build a stronger, broader, and more involved fanbase. People can catch up with a missed episode quite easily, in high-quality video whenever and wherever they want. The rise of unauthorized downloading of TV-shows is a signal that customers want something that is not available through other channels. Availability and convenience seems to be the key issue why people turn to BitTorrent." In other words, the very factors that are driving DVRs and time shifting are underlying piracy as well.

According to Van Der Sar, TV piracy is not necessarily an industry-crippler the way p2p mp3 sharing was for the music industry, but possibly an opportunity in disguise. "I don't think TV networks should be afraid of BitTorrent or piracy in general," he says. "But they shouldn't ignore it either. Piracy is a market signal, an opportunity. If interpreted correctly, TV-networks may hugely benefit from piracy by selling their shows to regions where the demand is highest. On the other hand, if they want piracy to decrease they only have to make their content available in user-friendly format. Hulu already decreased TV piracy in the US significantly, but there's still a lot of work to do, especially outside the US."

Whether that takes the form of region-specific viewing options that duplicate the TV experience completely, like TV Everywhere, or more à la carte options like Hulu, Netflix, iTunes/AppleTV and Xfinity remains to be seen. What's clear though, is that while the audience may be fragmenting, they're after the same thing—the shows they love, in a format that's easy and convenient to watch. That's simple enough, but keeping track of TV viewers' often-fickle desires is more complicated than ever before. And as viewing methods venture more towards the unconventional, broadcasters' methods of tracking their audiences will have to follow.