Verizon goes under concrete to bolster NRG Stadium DAS for Super Bowl LI

Nodes on wheels, or NOWs, provide extra coverage for Verizon Wireless in Houston for Super Bowl LI. Credit: Verizon Wireless

Nodes on wheels, or NOWs, provide extra coverage for Verizon Wireless in Houston for Super Bowl LI. Credit: Verizon Wireless

In a slight twist from its strategy for last year’s Super Bowl, Verizon Wireless has installed DAS antennas underneath the concrete flooring of lower-tier seats at Houston’s NRG Stadium, to provide extra bandwidth for the expected high wireless data usage at Super Bowl LI.

Last year at Levi’s Stadium in Santa Clara, Calif., Verizon beefed up its distributed antenna system (DAS) with under-seat antennas it designed specifically for use in stadiums. The idea of mounting antennas under seats, a growing trend in the stadium Wi-Fi world, is gaining traction as another method of bringing signals closer to fans, especially in places (like lower bowl seats) where there are no overhangs or other places to mount gear.

And while Verizon has been preparing for Sunday’s big game at NRG Stadium for years, that didn’t stop the company from “continually tweaking” its network preparations, according to Leo Perreault, executive director of network operations for Verizon’s South Central market, a region that stretches from west of Florida to Arizona, including Houston. In a phone interview this week, Perreault said that Verizon installed the under-concrete antennas during the middle of the 2016 football season, giving the company “some good experience” with the deployment ahead of Sunday’s game.

Under concrete = easier install and maintenance

A view inside the head end room that runs Verizon's NRG Stadium DAS. Credit: Verizon Wireless

A view inside the head end room that runs Verizon’s NRG Stadium DAS. Credit: Verizon Wireless

It might not be well known outside of wireless networking circles, but signals will travel through concrete; many early stadium Wi-Fi designs (and some current ones, including a new network installed at the Pepsi Center in Denver) use antennas mounted under concrete floors, pointing up. Though fixed under-seat antennas can provide better coverage, Perreault said the ease of deployment made putting the additional DAS antennas underneath the floor a better option in Houston.

“This way [under the concrete] is non-intrusive,” Perrault said, noting that the devices are also not affected by stadium power-washing units. The decision may have been influenced by the fact that NRG Stadium’s new Wi-Fi network had a big issue with moisture in under-seat AP placements, forcing a mid-season rip and replace for all the under-seat Wi-Fi APs.

Even though antennas under concrete are not as powerful, Perrault said Verizon is “very pleased with the performance. It’s a good compromise.”

Biggest stadium DAS?

Between the game being the Super Bowl and it being in Texas, there’s no shortage of hyperbole surrounding the game and all its attendant facets, including the network technology. But when Perreault claims that the DAS Verizon has installed for NRG Stadium “might be the largest we have anywhere,” that might be true since it also serves adjacent properties including the NRG Convention Center, the NRG Arena and an outdoor DAS in the surrounding spaces. In addition to Houston Texans games, NRG Park (which includes the stadium) is also host to the Houston Livestock Show and Rodeo, which humbly bills itself as the “world’s largest livestock show and richest regular-season rodeo.”

An alien spaceship, or a temporary cell tower from Verizon? You choose.

An alien spaceship, or a temporary cell tower from Verizon? You choose.

Inside NRG Stadium, Perreault said the new Verizon DAS (built before the 2015 season) has more than 900 antennas. As neutral host, Verizon will also provide access to AT&T and T-Mobile on its network; Sprint, which built a previous DAS at NRG, will continue to run on that system.

Outside the stadium and around Houston, Verizon has done the usual big-event preparations, with lots of permanent and temporary macro network improvements, and portable units like COWs (cells on wheels) and smaller NOWs (nodes on wheels). You can review all the Verizon preparations in a company blog post.

As previously reported in MSR, Verizon also helped foot part of the bill for the new NRG Stadium Wi-Fi network, a deal that will give Verizon a reserved claim to 40 percent of the Wi-Fi network’s capacity, according to Perreault.

Whether or not Super Bowl LI breaks the wireless data consumption records set at last year’s game remains to be seen, but Perreault said there doesn’t seem to be any slowing down yet of the perpetual growth in wireless data use at stadiums, especially at big events like the Super Bowl.

“Fans just seem to find ways to consume whatever additional bandwidth you provide,” he said.

NFL builds its own Super Bowl app, with no concessions delivery and fewer replays

Screen shot of map function on NFL Super Bowl LI app.

Screen shot of map function on NFL Super Bowl LI app.

The NFL has built its own Super Bowl mobile app, breaking with a recent history of using stadium-app specialists like VenueNext and YinzCam to develop specific apps for Super Sunday.

Also unlike recent years, the NFL’s Super Bowl app will not feature instant replays or have any kind of food or drink delivery services. Instead, there appears to be a big focus on promoting Super Bowl events (especially those for this weekend) and for helping out of town tourists find their way to Super Bowl events and to the game itself.

Curiously, an interview about the app with the NFL’s CIO claimed that this year’s app will also be the first to include the ability for fans at the game to watch Super Bowl commercials. The story also claims without any attribution that “In the past, commercials weren’t on the app in order to avoid using too much bandwidth in the stadium.” However, at the most recent Super Bowls, including the past two, stadium bandwidth has been more than sufficient enough to stream plenty of video. And in fact, both of the last two Super Bowl apps have included the ability for fans at the game to see Super Bowl commercials.

Last year’s app, developed by VenueNext for the Levi’s Stadium hosting of Super Bowl 50, definitely showed Super Bowl commercials, part of what the San Francisco 49ers network team said was a record-breaking day of app-based video watching. The Super Bowl 49 app, built by YinzCam, also included Super Bowl commercials according to this NFL video and according to our previous reporting.

Fewer replays, no food or beverage delivery service

Screen shot of transportation info links from Super Bowl LI app.

Screen shot of transportation info links from Super Bowl LI app.

And even though NFL CIO Michelle McKenna-Doyle is quoted in the previous story about the new app as saying “You won’t feel like you’re using two separate apps as fans have in the past,” the Super Bowl LI app contains a link to download the separate NFL Mobile app, which is apparently where Super Bowl highlights and replays will live. There was no confirmation from the NFL or Verizon about whether or not fans in the stands would be able to watch the live broadcast of the game via NFL Mobile. Fans not at the game will be able to use NFL Mobile to watch the game on cellular devices; fans can also stream the game from the FoxSports website, for PCs or tablet devices.

This year’s app will also not include any way for fans to use the app to order food or beverage delivery to their seats; last year’s app did have the ability to order in-seat delivery of beverages or to place an order for food and beverage express pickup, a service used for 3,284 orders. NRG Stadium, however, does not offer full-stadium in-seat ordering like Levi’s Stadium does; the stadium does have serving staff with wireless devices providing in-seat ordering services for club sections, which will likely be in use at the Super Bowl as well.

NRG Stadium hit 4.11 TB on Wi-Fi for Texans-Raiders playoff game

NRG Stadium. Credit: Houston Texans Instagram

NRG Stadium. Credit: Houston Texans Instagram

In what became the last live tune-up before the Super Bowl, NRG Stadium in Houston saw fans use 4.11 terabytes of data on the venue’s new Wi-Fi network during Houston’s 27-14 playoff victory over Oakland on Jan. 7, according to the Texans.

Jeff Schmitz, vice president of information technology for the Texans, said that NRG Stadium also saw approximately 35,000 unique users on the network at the playoff game, a 48 percent take rate against the total attendance of 71,790. The peak concurrent user number for the game was almost 24,000 users, with all numbers setting season highs for the network that debuted at the NFL season start, according to Schmitz.

“The playoff game was definitely the biggest” network-traffic day for the Texans, said Schmitz in a phone interview. During the talk Schmitz clarified that the network went through a serious up-and-down stretch during the middle of the season, due to under-seat Wi-Fi AP enclosures that didn’t completely seal out moisture.

Under seat APs visible down seating row. Credit: 5 Bars

Under seat APs visible down seating row. Credit: 5 Bars

While MSR had previously reported on the issue, Schmitz clarified that the full replacement of the APs took place in late October, meaning that the network staff had to basically re-tune and adjust the network in the later months of the season.

But the 4.11 TB mark and the large number of connected users for the playoff game was a sign that the network was in fine shape for Sunday’s Super Bowl LI, where if history offers any guidance there might be another new record set for single-day Wi-Fi usage.

The number to beat is the 10.1 TB mark from Super Bowl 50 at Levi’s Stadium last February, part of a 26 TB wireless day (with 15.9 TB recorded on cellular and DAS networks).

Wi-Fi coaches staying for Super Bowl

Schmitz also said that the Extreme Networks “Wi-Fi coaches” would be on hand for Super Sunday, helping fans figure out how to connect to the Extreme-based network and its 1,250 APs.

“We thought we’d only keep the coaches (who roam the stadium helping fans with network issues) for half a season, but with the switch [in APs] we ended up having them there for every game,” Schmitz said.

Wi-Fi gear visible above concourse kiosk. Credit: 5 Bars

Wi-Fi gear visible above concourse kiosk. Credit: 5 Bars

Having the Wi-Fi coaches at the Super Bowl also makes sense since many fans at that game will likely be visiting NRG Stadium for the first time, as opposed to Texans season ticket holders.

Though NRG Stadium won’t have to contend with temporary structures like those built on the concourses at Levi’s Stadium last year, Schmitz said there is some extra network work ahead to make sure the auxiliary press box area has “beefed up” Wi-Fi as well as wired connections for media use.

Another thing missing from last year’s Super Bowl is the ability for fans in any seat to order beverage delivery via the game-day app. Though details of services for the Super Bowl app have yet to be fully announced, Schmitz said the only in-seat delivery would be for premium club-level seats, which in addition to app-based ordering will have live humans who can take orders from fans for in-seat delivery, something Texans fans have available during regular-season games.

“That service [fans ordering from servers] gets used the most,” Schmitz said.

Indiana Pacers upgrade Wi-Fi at Bankers Life Fieldhouse

Bankers Life Fieldhouse, home of the Indiana Pacers. Credit all photos: Frank McGrath/Indiana Pacers

Bankers Life Fieldhouse, home of the Indiana Pacers. Credit all photos: Frank McGrath/Indiana Pacers

Whenever you undertake a Wi-Fi retrofit project, one thing is for certain: You can always expect surprises along the way.

For the Indiana Pacers, the biggest surprise in their recent renovation of the Wi-Fi network at Bankers Life Fieldhouse was finding out that their venue already had holes drilled in the concrete under the seats, greatly simplifying (and reducing the cost) of the mainly under-seat deployment that just went live in December.

The new 400-plus AP network, using gear from Ruckus, replaces one of the NBA’s first in-stadium Wi-Fi networks, one built and run by SignalShare using gear from Xirrus. With SignalShare now in bankruptcy and facing legal charges of fraudulent behavior, the Pacers went a different route for their new network, which is part of a plan to bring more digital-based fan services to the 17-year-old venue in downtown Indianapolis, which seats roughly 18,000 for basketball games.

According to Kevin Naylor, vice president of information technology, Pacers Sports and Entertainment, that plan got an unexpected (and welcome) boost when the Pacers’ IT team looked and found pre-drilled holes underneath many of the seats, covered up with temporary aluminum plates. With Ruckus able to use the pre-drilled holes for its under-seat Wi-Fi design, the Pacers were able to save “hundreds of thousands of dollars” in deployment costs, Naylor said.

A new digital plan for fans

Editor’s note: This profile is from our latest STADIUM TECH REPORT, the ONLY in-depth publication created specifically for the stadium technology professional and the stadium technology marketplace. Read about the Sacramento Kings’ new Golden 1 Center and the new Wi-Fi network for the Super Bowl in our report, which is available now for FREE DOWNLOAD from our site!

Leading the venue’s new digital direction is Ed Frederici, chief technology officer, Pacers Sports and Entertainment, who joined the organization in the fall of 2015, after spending almost 6 years as the CTO of ExactTarget, a marketing automation provider that was aquired by Salesforce in 2013.

Though he came into the job “relatively ignorant of sports,” Frederici said he saw “a really interesting problem to solve” revolving around the ongoing evolution of the live-event fan, and who the new attendee was. With a plan to help drive the fan engagement through technology, Frederici, Naylor and the Pacers’ organization began a thorough assessment of Wi-Fi gear providers as part of their plan to bring a new network to Bankers Life Fieldhouse, replacing one that didn’t stand up to current use patterns.

“The old network tapped out when it got to about 3,000 [concurrent] users,” Frederici said.

Pacers director of IT Kevin Naylor shows off a new under-seat Wi-Fi AP

Pacers director of IT Kevin Naylor shows off a new under-seat Wi-Fi AP

According to Frederici, the Pacers looked at “all the major providers” of Wi-Fi gear, testing implementations live by putting gear into mobile merchandise-selling stands in use on the stadium concourses. The final decision, Frederici said, came down to a battle between Ruckus and Xirrus, with Ruckus the final winner.

Under seat the best option

According to Bart Giordano, vice president for business development and strategic partnerships, for Brocade’s Ruckus business unit, going under-seat with Wi-Fi seems to be the direction large public venues are all headed in.

“It [under seat deployment] is sort of standard now,” said Giordano. “You really need to have users close to the APs, and it’s hard to achieve that with overhead.”

With just over 430 APs in the new network, Frederici was worrying about the drilling costs — until it turned out that most of the drilling had already been done, apparently as part of the arena’s original electrical configuration.

“Seventeen years ago, cables were much thinner, and it looks like [the holes] were cored for electrical,” Frederici said. “But it worked out fabulously.”

And like several other venues have done recently, the Pacers have decided to scrap support for fan-facing services on the 2.4 GHz spectrum, which makes administration of the fan Wi-Fi network easier and cheaper. The team will still keep some 2.4 GHz connections for back of house use.

With 2.4 GHz, Naylor said, “the noise level just got really bad in the lower bowl. It’s much easier to go to [only] 5 GHz. Every phone made now has 5 GHz.” For the older phones, Naylor said, the arena’s neutral-host DAS run by ExteNet Systems can provide connectivity, with AT&T and Verizon Wireless already on the system with plans to add more carriers in 2017.

While the Pacers currently have a basic YinzCam-based game-day app, Frederici is looking forward to more services in the future, including the possibility of having amenities like live parking and traffic information available via the app, as well as blue-dot wayfinding to the seat. For this year, the Pacers have already added concession and restroom wait time alerts to the app, the first step in a planned process of greater digital engagement.

“We want to own the experience from your driveway to the stadium, then back home,” Frederici said. Part of the new network deal includes analytics software services from Ruckus partner Purple, which helps teams mine data from fan interaction with the Wi-Fi network.

“We’re excited to see what kind of data we can pull from them [Purple],” Naylor said.

Patriots’ new Wi-Fi network delivers during playoffs; AFC Champ game hits 5.11 TB mark

Gillette Stadium before the Sept. 11 game vs. the Miami Dolphins. Credit: Steve Milne, AP, via Patriots.com

Gillette Stadium before the Sept. 11 game vs. the Miami Dolphins. Credit: Steve Milne, AP, via Patriots.com

The new Wi-Fi network installed at the New England Patriots’ Gillette Stadium this season got a good workout during the postseason, with more than half the fans present logging on during the Patriots’ two playoff home games. According to the team, the network saw 5.11 terabytes of data used during Sunday’s game, one of the top Wi-Fi marks we’ve ever reported.

Sunday’s 36-17 New England victory over the Pittsburgh Steelers in the AFC Championship game saw the biggest Wi-Fi traffic numbers, with 38,436 unique users connecting to the network at some point during game day — a 57 percent take rate based on the reported sellout crowd attendance number of 66,829. Original stats provided by Fred Kirsch, publisher & vice president of content at Kraft Sports Productions (and overseer of all things technology at the stadium) showed a total tonnage of 3.70 terabytes for Wi-Fi traffic Sunday, with a peak concurrent client number of 30,824 and a peak throughput total of 3.14 Gbps.

That total was reset to 5.11 TB when Kirsch said additional traffic from “postgame activities” (likely the trophy presentations) was added in. The 5.11 TB mark puts Sunday’s game into the MSR unofficial “Top 5” list of single-day Wi-Fi traffic marks, supplanting the first College Football Playoff championship game, which saw 4.93 TB of Wi-Fi used at AT&T Stadium on Jan. 12, 2015. (We are still waiting for Wi-Fi figures from this year’s CFP champs game, so our list may change again soon!)

The Extreme Networks-based network saw almost a similar stress during the Pats’ 34-16 victory over the Houston Texans on Jan. 14, according to figures from Kirsch. Even though cold, rainy conditions persisted at both games, for the Texans game the Gillette network saw 2.97 TB of total traffic, with 35,536 unique connections — a 53 percent take rate. Peak concurrent connections on Jan. 14 were 28,620, while peak bandwidth use was 2.76 Gbps.

THE NEW TOP 5 FOR WI-FI

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB
2. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB
3. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB
4. Alabama vs. Texas A&M, Kyle Field, College Station, Texas, Oct. 17, 2015: Wi-Fi: 5.7 TB
5. Pittsburgh Steelers vs. New England Patriots, AFC Championship Game, Gillette Stadium, Foxborough, Mass., Jan. 22, 2017: Wi-Fi: 5.11 TB

Avaya files for Chapter 11 bankruptcy

mls14Networking company Avaya, which has made an aggressive move over the past couple years into the sports stadium networking space, including purchasing naming rights and technology contracts to the San Jose Earthquakes soccer stadium, has filed for chapter 11 bankruptcy protection, according to the company website.

No word yet on how the “restructuring” will affect any of Avaya’s stadium tech deals, which include Wi-Fi networks at the Pepsi Center in Denver, and at the Montreal Canadiens’ Bell Centre, along with Avaya Stadium in San Jose. More as we learn more.