Stadium Tech Report: With advanced wireless network and app, Baylor brings ‘NFL Experience’ to McLane Stadium

McLane Stadium, Baylor University. Credit all photos: Baylor University

McLane Stadium, Baylor University. (click on any photo for larger image) Credit all photos: Baylor University

Just a few years ago, the Baylor University football program wasn’t a topic of national conversation. But now after a Heisman trophy, a Big 12 championship and perennial top rankings, Baylor is doing its best to stay at the front of the college football pack — and that effort extends to its new stadium, where Baylor has put in place a wireless network and a feature-filled app designed to bring an “NFL experience” to the Waco, Texas campus.

Now in its first season at the brand-new McLane Stadium, Baylor is already delivering an in-stadium fan technology experience that, like the team itself, ranks highly in the nation. Thanks to a Wi-Fi deployment from Extreme Networks, a DAS from AT&T and a new stadium app from sports-app leader YinzCam, Baylor is able to bring high-quality wireless connectivity to all parts of the 45,140-seat facility, along with advanced app features like live and on-demand streaming action video, as well as seating and parking maps for the new facility.

Like the recently opened Levi’s Stadium in Santa Clara, Calif., Baylor had an advantage with McLane Stadium in being able to make technology part of the original design, instead of having to retrofit it in later. “It’s an amazing opportunity to have a new stadium and be able to plan for technology from the bottom up,” said Pattie Orr, Baylor’s vice president for information technology and Dean of university libraries, in a recent phone interview. “It sure is nice to have technology in mind from the beginning.”

The house that RG3 built

McLane Stadium - Opening Game Day vs SMU

McLane Stadium – Opening Game Day vs SMU

But just like the Baylor team, the plan for the new stadium and its technology underpinnings had to come together quickly. Even late in the 2011 season, when then-Baylor quarterback Robert Griffin III was just starting to turn heads with his on-field heroics, the idea of building a new football facility on campus hadn’t yet been formally approved. In 2011, Baylor still played games in Floyd Casey Stadium, a 50,000-seat facility that opened in 1950, located about four miles from campus.

And then, RG3 happened. As many people associated with Baylor will tell you, when the Bears and Griffin quickly vaulted into the national consciousness — especially after a dramatic RG3-led win over Oklahoma and his subsequent winning of the Heisman trophy — the push for a new stadium quickly gathered steam. (For more background, read this excellent history of the stadium’s origin from the Waco Tribune-Herald.)

“Two years ago we still weren’t sure the stadium was coming,” said Bob Hartland, associate vice president for IT infrastructure, who also participated in the phone interview. “Then there was the Heisman trophy, and everything started becoming a reality.”

After the university gave its formal approval in July of 2012, planning for the $266-million facility could begin — with Orr and Hartland’s tech team having to employ a bit of crystal-ball thinking.

“We knew we needed to deliver for mobile devices,” said Hartland. “The hard thing was trying to predict what was going to happen 2 years out [when the stadium would open].”

Pattie Orr, VP of IT for Baylor

Pattie Orr, VP of IT for Baylor

Bringing an ‘NFL experience’ to Waco

And even though Baylor is private and smaller than its Big 12 conference competitors, the IT team made no small plans. “We wanted an NFL experience,” Orr said. To her, that meant an interactive mobile app that delivered live video to each and every seat.

“The best thing we could do was be forward looking,” said Orr. “What we pictured was, ‘could we have it in our hands?’ In the stadiums of the past, fans loved the big screens, and they still do. But there’s nothing like having it right in the palm of your hand.”

Orr said the Baylor IT team visited some existing stadiums with advanced networks, like AT&T Stadium and Gillette Stadium, as part of a technology vetting process. Eventually the Baylor IT department whittled the Wi-Fi selection down to three different approaches — one that included under-the-seat antennas, one that proposed an under-the-concrete solution, and one that relied mainly on overhead APs. That final one, from Wi-Fi provider Extreme Networks, became the winning bid, in part because the Baylor team liked its less-intrusive technology.

If you look closely under the overhangs, you can see Wi-Fi APs

If you look closely under the overhangs, you can see Wi-Fi APs

“Overhead [APs] are just less intrusive, operationally,” said Hartland, noting the need to drill holes in concrete and do special cleaning or weather-hardening for under-the-seat APs. If you look at McLane, you can see multiple overhang areas around the entire seating bowl, which facilitates overhead AP placements. According to news reports, the Extreme Wi-Fi deployment has 330 APs.

Baylor’s Orr also liked the Extreme Purview Wi-Fi analytics software, which provides detailed views of network usage.

“Analytics provide what you need to know,” Orr said. “If you’re in the dark on the fan experience, and don’t know which apps are being used, how can you tune it or make it better?”

On the DAS side, Baylor went with AT&T as the neutral host, though AT&T already has signed up main competitor Verizon Wireless as a client, meaning that the two largest providers of cellular service have enhanced coverage at McLane Stadium through the AT&T DAS, which reportedly has 486 antennas.

“Our goal was a high-density solution, for both cellular and Wi-Fi,” Orr said.

Solving for the standing-on-the-seat problem

Wi-Fi "coach" helps out at McLane Stadium.

Wi-Fi “coach” helps out at McLane Stadium.

While the network has been an early success — Orr said Baylor is already seeing Wi-Fi take rates as high as 33 percent of all attendees at games so far this season — there have also been a few interesting fixes that have been necessary, including re-tuning Wi-Fi APs to get around the interference quirk of students standing on their seats.

Call it technology meeting tradition, with tradition winning: A Baylor tradition to have underclass students standing for the whole game turned into students standing on top of seats at their new section in McLane Stadium — a shift that led to unexpected interference with the original Wi-Fi antenna placements. (One of the quirks of Wi-Fi networks is that the water inside human bodies is a very effective blocker of Wi-Fi signals.)

“We had not anticipated the students standing on seats, and that extra 20 inches really made a difference,” Hartland said. According to another story in the local paper, large band instruments also blocked Wi-Fi signals. Hartland said that since the original problems the IT team and Extreme have developed work-arounds and new antenna placements to fix the issue.

“It’s pretty fantastic that our students are so excited,” said Orr of the standing-interference issue. “You don’t see things like that much at the NFL level.”

Live video and app ‘coaches’

On the app side, Baylor went with YinzCam, a company with numerous stadium apps under its belt for all the top U.S. professional leagues. YinzCam, like Extreme, is also a partner with the NFL, giving YinzCam an edge in winning NFL stadium deployments.

Like other stadium apps, the Baylor In-Game app from YinzCam features multiple camera-angle choices for replays and live streaming video, as well as a host of stats and other team information. Important to Baylor and its new stadium are maps that help direct fans to parking areas, as well as to specialty concession stands in a facility that is new to everyone this season.

Using the app at McLane Stadium

Using the app at McLane Stadium

“We have some well-known smoked onion rings [at the stadium] and the app can help fans find which stands are selling them and how to get there,” Orr said. The parking feature on the app, she said, can send text directions to fans. Also special to Baylor is a “brick finder,” an app that lets fans who participated in a stadium fundraiser find where the brick with their name on it is.

One more NFL-like feature with a collegiate twist is Baylor’s embrace of the Extreme “Wi-Fi coaches” program, which has network-knowledgeable staff members walking around stadiums in highly visible gear offering hands-on help with connectivity and stadium app use. While Extreme has used the coaches program at pro venues like New England and Philadelphia, at Baylor Orr took advantage of in-house “talent,” using students in the MIS program as roaming “coaches,” giving them some real-world experience at network troubleshooting and customer service.

“We put them [the student coaches] in bright vests and have them stationed near concession stands, to offer a friendly face,” Orr said. “They’re terrific, and they give us real-time feedback.”

Orr said Baylor also has a journalism department student intern leading the technology team’s social media effort, which encourages fans to tweet out problems or questions they might have.

“With my gray hair I’m not too good on social media, but one thing I learned is that we need to embrace it,” said Orr. Hartland said that YinzCam reps told Baylor they “just need to get out there” on social media to support the app, and he reports pleasant surprises when the IT team tweets back.

“On social media, [fans] don’t expect to be contacted,” Hartland said. “They really appreciate it when we get back to them.”

Levi’s Stadium Wi-Fi update: Usage down from record, but still strong — 2.4 TB for Eagles game

Fans take pictures of opening-day kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

Fans take pictures of opening-day kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

With three NFL games and now one collegiate contest under its belt, the Wi-Fi network at Levi’s Stadium is still handling big loads of data traffic, though not quite at the Super Bowl-beating level of 3.3 Terabytes recorded at the Niners’ home opener.

According to statistics provided by Niners vice president for technology Dan Williams, the Levi’s Wi-Fi network carried 2.4 TB of data during the Sept. 28 game against the Philadelphia Eagles, and another 2 TB during the Oct. 5 contest with the Kansas City Chiefs, both sellouts with reported attendance of 70,799 fans. And at an Oct. 24 college game between Cal and Oregon with 55,575 fans in attendance, the network carried 1.5 TB of traffic.

As you might be able to guess from the bandwidth numbers, the Sept. 28 game also had more users on the network, with 22,942 unique users, compared to 21,133 at the Oct. 5 game. For Cal-Oregon the unique user count was 13,508. At the Niners’ regular-season home opener on Sept. 14, there were more than 30,000 fans using the Wi-Fi network, with a peak of 19,000 simultaneous connections. The peak numbers for the later dates were 15,500 for the Sept. 28 game, 14,500 for the Oct. 5 game, and 8,400 for the college game.

In-seat and express food orders stay strong

Screen grab from Levi's Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

Screen grab from Levi’s Stadium app showing in-seat food delivery option.

Food and beverage orders via the Levi’s Stadium app appear to be holding between the 1,500-2,000 order level for both the express pickup option as well as the in-seat delivery, with both features available to every seat in the house. According to Williams, for the Eagles game there were 1,144 express pick-up orders and 1,652 in-seat deliveries, while at the Kansas City game there were 1,162 express and 1,712 in-seat orders. At the Cal-Oregon game, where many fans were probably not as aware of the Levi’s app functionality, there were 551 express orders and 308 in-seat deliveries.

The video-replay feature of the Levi’s app also continues to attract solid interest, with 1,297 unique users watching 5,089 replays at the Eagles game and 1,111 unique users watching 4,986 replays at the Kansas City game. At the college game 234 fans used the replay feature, watching 1,059 replays.

One more interesting stat provided by Williams — the Apple iPhone is the overwhelming favorite device for connecting to the Levi’s Stadium Wi-Fi, with 62 percent share for the Sept. 28 game, and 60 percent share for both the Oct. 5 and Oct. 24 games. Android devices represented 24 percent (Sept. 28), 26 percent (Oct. 5) and 25 percent of all devices, while Apple iPads accounted for 2 percent, 3 percent and 1 percent of devices for the respective Sept. 28, Oct. 5 and Oct. 24 games.

DAS usage also remains strong, but not tops in NFL

On the DAS side of the Levi’s Stadium network we have some stats from AT&T to share (but none from any other carriers). According to AT&T, for the Oct. 5 game AT&T customers used 549 GB of data, which was only the fourth-highest AT&T DAS total for that weekend. Dallas (827 GB), San Diego (716 GB) and New Orleans (598 GB) all had higher AT&T DAS traffic totals for that weekend’s games. (Remember, these are results for stadiums with AT&T DAS networks only, not for all stadiums.) For the Sept. 14 game, the Levi’s Stadium AT&T DAS recorded 673 GB of traffic, according to AT&T.

Reports: Free Wi-Fi now available at Kansas City’s Kauffman Stadium for World Series

Screen Shot 2014-10-22 at 2.38.07 PMAccording to reports from Kansas City, the previously Wi-Fi-less Kauffman Stadium now has a working free Wi-Fi network for fans, apparently installed recently by Major League Baseball’s advanced media group. Though we haven’t yet received any official confirmation about the network’s launch from either the Royals organization or from MLBAM, two separate sources confirmed that free Wi-Fi is available in the stadium, and apparently has been for a few weeks now.

In our reporting about MLB Wi-Fi deployments for our Q2 Stadium Tech Report, we found that Kauffman Stadium was one of 10 MLB parks not yet offering Wi-Fi for fans in the seats. But since this spring MLBAM has publicly said that it plans to help install Wi-Fi in the remaining parks that need it, including helping to pay for the deployment if necessary. One possible reason for the speedy deployment at Kauffman could be the joint marketing deal between Apple, MasterCard and MLBAM to promote the use of the new Apple Pay service, which is being offered at both Kauffman Stadium and AT&T Park during the World Series. San Francisco’s AT&T Park already has Wi-Fi, a service the Giants have offered fans since 2004.

Though a thorough search of the Kansas City Royals’ team website shows no mention of Wi-Fi services available at Kauffman stadium, one Twitter user has reported finding a free “speedy” Wi-Fi service at recent playoff games there, and another visitor to the park for yesterday’s World Series opener also reported a live Wi-Fi service available to fans. Another tipster has told us that the network uses Cisco Wi-Fi gear, but again, nothing has yet been officially confirmed.

Anyone lucky enough to have a ticket to tonight’s Game 2, send us a Wi-Fi speedtest if you can…

Stadium Tech Report: DAS Group Professionals makes a name for itself with Levi’s Stadium DAS

DAS antennas hanging from a Levi's Stadium overhang. Credit, all Levi's photos: Paul Kapsutka, MSR

DAS antennas hanging from a Levi’s Stadium overhang. Credit, all Levi’s photos: Paul Kapsutka, MSR

If you look around at the walls, ceilings and overhangs at Levi’s Stadium, it’s hard to miss the small square boxes with the off-white color and a “DGP” logo in one corner. While the wires hanging out the back of each box make it an easy guess that the equipment has something to do with wireless networks, even many industry insiders may not know the company behind the boxes and the three-letter acronym.

Meet DAS Group Professionals, the Bay Area firm in charge of deploying a distributed antenna system (DAS) to make sure your cell phone gets a good signal at San Francisco 49ers games and any other event inside the 68,500-seat Levi’s Stadium. And while you might not be familiar with DGP, rest assured the company is extremely familiar with cellular deployments for large public venues, having installed similar DAS networks for airports, casinos and hotels, and even for San Francisco’s BART train system. Of course, most of that work was done when the company was called Forza Telecom, before changing its name to DGP earlier this year, another reason why “DGP” may not have rung any bells.

“It may appear like we just fell out of the sky, but we’ve actually built quite a few [DAS] systems,” said Steve Dutto, president of DGP, in a recent phone interview. “We’ve got years of experience.”

DAS antennas above a food stand

DAS antennas above a food stand

DAS: The Rodney Dangerfield of stadium connectivity

One thing that keeps firms like DGP in the shadows is the relative obscurity of DAS itself. While most people generally understand how cell phones work — you turn on your phone, and it connects to an antenna somewhere on a tower or rooftop — in crowded public facilities like stadiums, traditional cellular networks with towers several miles apart can’t handle the concentrated capacity. To provide connectivity for areas with large crowds, the latest tactic is to deploy a DAS, a network of lots of smaller antennas. Originally deployed in places like office buildings, hotels and convention centers, DAS is rapidly gaining favor in stadiums and arenas, helping to alleviate the “no signal” problem that has cropped up in many venues the past few years.

And while stadium Wi-Fi gets lots of headlines whenever it gets deployed — probably thanks again to the widespread understanding of how Wi-Fi works — there are already far more DAS deployments in stadiums than Wi-Fi, mainly because cellular carriers will pay almost all the associated costs of a DAS buildout to make sure their customers get a signal. According to our most recent 2014 State of the Stadium survey, 71.4 percent of our respondents said they had a full DAS at their facility, while only 35.7 percent had fan-facing Wi-Fi.

Steve Dutto, president, DGP

Steve Dutto, president, DGP

How does a DAS work? Usually, either a major cellular carrier or a third-party “neutral” host like DGP will build out the antenna infrastructure, which includes many small antennas and then cables to bring the connections back to a wiring room or data center. There, cellular carriers install their own cell-tower back-end gear to authenticate customers and to provide a connection to the company networks and the Internet. Since it’s in the cellular carriers’ interest to keep their customers connected (and using billable minutes and data), carriers will often pay the full cost of a DAS infrastructure by building and running it themselves. In the case like Levi’s, where DGP is the “neutral host,” DGP builds the infrastructure and then charges cellular carriers to use it.

Such deals are rarely publicized, and Dutto would not comment on how much each carrier was being charged to use the Levi’s DAS — though industry gossip has the figure somewhere around $5 million per carrier per year. And just like fight club, for many deployments the first rule of DAS is that you don’t talk about DAS, because no cellular carrier ever wants to admit that its network might need help. So just like Rodney Dangerfield, DAS often doesn’t get a lot of public respect. But at Levi’s Stadium and many other sports and entertainment venues, DAS is a booming business that would be sorely missed if it wasn’t there.

The ‘dream and the nightmare’ of building the Levi’s DAS

Now that DGP’s 700-plus antenna DAS deployment is up and running at Levi’s, Dutto can breathe a small bit easier. While the network is good business for the company and an obvious prominent calling card for the future, the aggressive deployment timeframe probably isn’t something Dutto is eager to repeat.

“Levi’s Stadium was both a dream and a nightmare [for DGP],” Dutto said, due in part both to the Niners’ aggressive performance expectations and the rapid buildout schedule. Of course, DGP was somewhat used to working quickly with the Niners — when the company put a DAS in Candlestick Park back in 2012 to solve that stadium’s legendary lack of connectivity, Dutto said it was deployed “in about 90 days.”

The success of the Candlestick deployment, Dutto said, led to the Niners offering the Levi’s DAS gig to DGP. With it, however, came the need to match the team’s out-front statements about how the stadium was going to be the best ever in terms of wireless connectivity. And with many people not knowing or not bothering to switch their phones to Wi-Fi, the Levi’s DAS, like most stadium DAS deployments, would probably handle most of the wireless connections.

Door sign for head end equipment room at Levi's Stadium (there are many of these)

Door sign for head end equipment room at Levi’s Stadium (there are many of these)

“We knew that it needed to be significantly better than anywhere else, right at the launch,” Dutto said. While the late addition of an early August soccer game at Levi’s pushed deployment schedules ahead even faster, Dutto said in the end it helped DGP overall.

“I wasn’t a big fan of getting ready for that date [the Aug. 2 soccer game was Levi’s first event] but it was a blessing in disguise,” Dutto said. “The trouble with a network is that you can’t really test it until everyone shows up. We got some good data from that event.”

After the “daily and nightly” discussions with the Niners’ tech team about antenna placements and other matters, the DAS network performed well when it mattered, during the Niners’ two preseason games on Aug. 17 and 24. According to a traffic report from the stadium tech team, the DAS network carried a combined 1.02 terabytes of wireless traffic for the two preseason games, which is on par with activity seen at big events in the past, like Super Bowls. According to Dutto DGP’s internal tests showed that few, if any, calls were dropped or didn’t connect.

“We’re at 98 percent [network performance] already, without [the network] being fully optimized,” Dutto said.

RF challenges and too many iPhone 4 customers

While many of the stadium’s Wi-Fi antennas are well hidden — including the ones in boxes under seats — the DAS antennas are a bit more prominent, especially if you are near where the first overhang comes close to the seats.

“Unfortunately, with Levi’s exposed steel-beam construction, if you want the DAS to work, you’re going to see it,” Dutto said. “It’s hard to be stealthy in there.”

And in many cases there isn’t just one but instead two DGP antennas side by side, which reflects the company’s decision to actually build two parallel DAS systems to better accomodate more wireless carriers. According to Dutto, AT&T and T-Mobile are based on one system, while Verizon Wireless, Sprint and public safety communications are handled by the other one. Currently, all the carriers are live on the DAS except for Sprint, which is still in the process of installing its back-end equipment.

DAS antenna in "Faithful Mile" area

DAS antenna in “Faithful Mile” area

One of the biggest challenges for DGP, Dutto said, comes from outside the stadium, and not inside. Like other open-air stadiums located in city cores, Levi’s Stadium faces significant interference from cellular antennas on nearby office building rooftops, as well as from the Santa Clara Convention Center right across the street.

“Less than a half mile from Levi’s you can see seven different rooftop cell sites,” said Dutto, who said the flat, open terrain around the stadium increases the ability for those signals to interfere with the stadium DAS deployment. Target Field in Minneapolis had some similar problems with cell antennas on nearby office buildings.

“We’ve done a lot of work with the carriers to adjust their macro networks around the stadium,” Dutto said. “We’ll do more of that as we go, and expect it to get better.”

And while Levi’s Wi-Fi network has shown itself to be incredibly robust, Dutto said that cellular connectivity over the DAS might be even faster than Wi-Fi in many instances, especially if fans have later-model phones with 4G LTE.

Even though Dutto said DGP’s testing recorded download speeds of up to 200 Mbps — and 65 Mbps sustained — he acknowledged that many Levi’s patrons might never see those kinds of numbers unless they snap up some of the new iPhone 6 models introduced by Apple this week. According to network stats collected by DGP during the preseason games, a lot of fans may be ready for an upgrade.

“There’s a lot more iPhone 4 users out there than we thought,” Dutto said.

CBS to stream entire 2014 SEC football schedule online and via app

If you’re one of those college football fans who simply can’t get enough SEC, CBS has you covered this season even if you’re not near a TV set. Starting with this weekend’s Georgia vs. South Carolina game, CBS said it will stream its entire schedule of SEC football games live online for free, and also through its Android and iOS apps. The online version of the broadcast, dubbed SEC Live, will also feature extras like an “all-22” overhead camera angle, and a special online postgame show.

The CBS slate of 16 SEC games is about one game a weekend, with a couple doubleheaders thrown in as well as the SEC championship game on Dec. 6. While the online streaming is not only convenient for those who want to watch games while away from their couch, it in many ways offers more features than the regular broadcast, with the multiple camera angles and other goodies like Twitter stream integration, player stats, highlights and polls. (Let’s see if CBS can fix the problem that plagues other online efforts, namely Twitter feeds that get ahead of the live video stream.)

The all-22 camera is one that adds a lot to football watching, especially for true football geeks who want to see plays unfold with coach-like access. ESPN had an all-22 camera among its options during the most recent BCS championship megacast. The inclusion of the multiple camera angles by CBS is good news, a signal that broadcasters are responding to fans’ wishes and using technology to answer the need. The entire package — especially the free nature of it, not requiring any cable contract qualification for access — is just another example of the savvy digital chops at CBS.

According to CBS, the SEC Live package each week will also include “a pregame show, a halftime show, and an original 5th Quarter postgame featuring analysis from CBS Sports’ lead college football analyst Gary Danielson.” We here at MSR are big fans of the postgame show with Danielson, which is sometimes conducted in a motorhome with a Skype-like feel. Good stuff.

Verizon, NFL fumble opening-day live video for NFL Mobile app — for the 2nd year in a row

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

It’s not a streak you want to continue, especially if you are the NFL, Verizon Wireless, or one of the frustrated customers who weren’t able to see live video using Verizon’s NFL Mobile application during Sunday’s season-opening games. But just like last year, when activation servers for a new version of the app got overwhelmed, issues arose again on Sunday, making live video — one of the premium features of NFL Mobile that makes the app and service so attractive — unavailable for most of the day, including the large slate of early games.

Verizon executives who spoke with MSR Monday declined to comment on the NFL Mobile issues, instead referring all questions to their partner, the NFL media team. Verizon wireless support was also apparently offline on Sunday, leaving the task of responding to the many irate messages and tweets to the small but valiant NFL Mobile Support team.

Alex Riethmiller, vice president of communications for NFL Media, said the issue once again was due to high traffic overwhelming the authentication servers, which (putting it simply) identify whether or not a customer is a Verizon subscriber, and whether or not that customer has access to the premium content. Some of the additional traffic may have been due to two changes Verizon made in NFL Mobile subscriptions from last season; one was to give free premium access (which lets you see live action) to all customers on a More Everything data plan, and the other was to charge all of those More Everything viewers another $1.99 a month if they wanted to also watch the NFL’s RedZone channel, one of the more attractive features of the app (since RedZone gives you lots of live action, even on games that might otherwise be blacked or not viewable for other reasons). Verizon customers who are not More Everything customers must pay $5 a month for access to the NFL Mobile premium content.

Theoretically, you should be able to order RedZone from this "settings" page of the app. But it didn't work when we tried to click the button.

Theoretically, you should be able to order RedZone from this “settings” page of the app. But it didn’t work when we tried to click the button.

For many NFL Mobile customers, yours truly included, any live video at all was not available for long stretches of Sunday, and after a brief 2-minute “preview” period the RedZone content was also unavailable, and the app was not allowing it to be purchased. As of Monday afternoon, the MSR official testing lab still couldn’t get the RedZone box to work — clicking it kept taking us to a page explaining the charges but not offering a way to add it to our bill.

“NFL Mobile requires a number of backend systems to ensure only Verizon customers can get live NFL video,” said Riethmiller, in a prepared statement. “We experienced issues with one of the systems that validates customers, and it took longer than anticipated to resolve because of tremendous demand. We are confident we have addressed the issue going forward.”

Kristin Rooney, Verizon Wireless director of sponsorships and branded entertainment, would not comment on the NFL Mobile outages in a phone interview Monday morning. However, Rooney was happy to talk about the new features in NFL Mobile this season, including the ability to watch local games live, even if they are out of the traditional prime-time window NFL Mobile has access to. In addition to the local games, NFL Mobile premium customers are supposed to be able to watch live games on Sunday nights, Monday nights and Thursday nights, and also have access to RedZone broadcasts during the day. That is, when the servers are working.

BUY... MORE... SERVERS

BUY… MORE… SERVERS

Another unclear option is what happens to RedZone access when NFL Mobile customers visit NFL stadiums — while Verizon blocks access to RedZone at some stadiums, at others it allows it, usually through a partnership with a team’s stadium app. Rooney, however, did not have a list of stadiums where NFL Mobile customers could watch RedZone broadcasts during a game; most famously, the service was blocked at last year’s Super Bowl in MetLife Stadium.

Rooney also declined to say just how many customers Verizon has for its NFL Mobile app, though the Google Play store says the app has between 10 and 50 million installs. However, since a non-premium version of the app is available to customers with contracts from other carriers, it’s unclear how many “premium” members were overwhelming Verizon’s servers. It’s worth noting that Verizon is no stranger to large numbers of single-day activations, as its systems support more than 100 million wireless subscribers, nation-wide.

In a related note, NFL Mobile wasn’t the only streaming product having issues Sunday. According to a report at TVPredictions.com, the DirecTV streaming service for NFL games also suffered outages Sunday, also for the second straight year.