Arizona State upgrades DAS, Wi-Fi at Sun Devil Stadium

Sun Devil Stadium at Arizona State. Credit all photos: ASU

Sun Devil Stadium at Arizona State. Credit all photos: ASU

When Arizona State University started renovating Sun Devil Stadium three years ago, the project wasn’t so much a simple wireless refresh as it was a total reset of what technology, sports and academia could co-create.

In addition to expanded Wi-Fi and DAS for the stadium (a venue that includes classrooms, meeting rooms and retail outlets), ASU activated a virtual beacon trial. The university also joined up with Intel to explore how Internet of Things devices might yield better environmental information about the bowl, including acoustic data, Jay Steed, assistant vice president of IT operations, told Mobile Sports Report.

The university’s IT department understood that a richer fan experience for football and other events would require a robust network. Steed and his colleagues visited other venues like Levi’s Stadium, AT&T Stadium, Stanford and Texas A&M to get a better handle on different approaches to networking, applications and services.

Regardless, some sort of refresh was overdue. Wedged between two buttes in the southeastern Phoenix suburb of Tempe, the 71,000-seat Sun Devil Stadium was completed in 1958 and needed infrastructure and technology updates. Wi-Fi access was limited to point-of-sale systems and stadium suites; fans generally relied on a DAS network.

Time for an upgrade

Editor’s note: This profile is from our latest STADIUM TECH REPORT, the ONLY in-depth publication created specifically for the stadium technology professional and the stadium technology marketplace. Read about the Sacramento Kings’ new Golden 1 Center and the new Wi-Fi network for the Super Bowl in our report, which is available now for FREE DOWNLOAD from our site!

“The stadium needed a lot of facelifting, not just from a technology perspective but also for the fan experience, like ADA compliance and overall comfort,” Steed said. “We didn’t just want to rebuild a venue for six football games a year, but extend its availability to 365 days and make it a cornerstone and anchor for the whole campus.”

The 'Inferno' student section got a priority for better connectivity.

The ‘Inferno’ student section got a priority for better connectivity.

The reset included tearing out the lower bowl to “punch some new holes” — new entry points to the stadium — and to add conduits and cabling for the new 10-gigabit fiber backbone for the stadium. The network can be upgraded as needed to 40- and even 100-gigabit pipes, according to Steed.

“We wanted to make sure it could support fans’ [connectivity needs] and all the facility’s operations with regard to video and StadiumVision, learning and education, and Pac-12 needs as well,” he said.

The overall stadium renovation was budgeted at $268 million; the technology upgrades will total about $8 million.

The university added 250 new DAS antennas. The vendor-neutral network includes AT&T, Verizon, Sprint and T-Mobile, which share 21 DAS sectors to keep cell service humming inside the stadium.

On the Wi-Fi side, ASU opted for Cisco’s access points. The networking vendor was already entrenched across the 642-acre campus; Steed and the IT department prefer the simplicity of a single-vendor network. Cisco helped with the hardware and RF engineering for Sun Devil Stadium. CenturyLink offered guidance on the networking and fiber pieces of the project, while Hunt-Sundt, a joint venture, was the contractor for most of the physical construction.

Wireless service for ‘The Inferno’

When the renovation is officially completed later in 2017 (most of the network is already live), there will be 1,100 APs in and around Sun Devil Stadium. The student sections, also known as The Inferno, get more APs and bandwidth since historical data has shown students to be the biggest bandwidth consumers in the stadium. Consequently, the ratio in the student sections is one AP to every 50 users; the rest of the bowl’s APs each handle about 75 users on average, Steed said.

Breakaway look at an under-seat AP

Breakaway look at an under-seat AP

ASU’s new Wi-Fi network was engineered to deliver 1.5 Mbps upstream and 3 Mbps downstream, but Steed said so far users are getting better performance – 8 Mbps up and 12 Mbps down. “We’re getting about 25 percent saturation,” he added. “Many users put their phones away during the games, but we see spikes at halftime and during commercial breaks.” Regardless, ASU continually monitors Wi-Fi and DAS usage and adjusts bandwidth as needed.

Another big challenge is the desert climate – temperatures regularly soar into triple digits. With about 450 under-seat APs in the bowl, Steed and his team had to make sure the enclosures could withstand heat and didn’t obstruct the walkways. “We’ll see how well the electronics do, baking at 120 degrees six months out of the year,” he laughed.

ASU is also working with Intel, using the stadium’s APs as part of an Internet of Things trial. As Steed described it, IoT sensors work alongside stadium APs to measure temperature, noise, vibration and other environmental data. “We also look at lighting control and water distribution and flow,” he said.

Concourses also got expanded Wi-Fi and DAS coverage.

Concourses also got expanded Wi-Fi and DAS coverage.

Automating the control of environmental functions like heating, cooling, power usage and facilities management will help the university toward its goal of being carbon-neutral by 2025, Steed added. The trials are designed so that the technology can be expanded across the university, possibly for campus transportation kiosks or student concierge services. IoT devices could give students and visitors information about adjacent buildings or landmarks around campus.

Separate but related, the university is also testing cloud-based, Bluetooth low energy (BLE) technology from Mist Systems. These “virtual beacons” use sensors attached to an AP to flag information or a point of interest for students or stadium visitors. “The virtualized beacon technology helps us understand where people are walking around and what they’re looking at in the stadium and elsewhere around campus,” Steed said.

They’re currently being tested in some of Sun Devil Stadium’s suites; Steed foresees expanding that to the student union to help guide people to meeting rooms, retail facilities or food vendors, for example.

Steed credited excellent communication and collaboration among the university’s athletic and IT departments and other players in the upgrade equation. “Our athletic director, Ray Anderson, brought the CIO and me into his office and really worked together with us,” he explained. “The biggest piece of our success was knowing that the AD supported our recommendations and brought us in as valued advisors.”

Update: Super Bowl LI breaks 37 TB wireless mark

NRG Stadium during Super Bowl LI. Credit: AP / Morry Gash/ Patriots.com

NRG Stadium during Super Bowl LI. Credit: AP / Morry Gash/ Patriots.com

It’s official now, and without any doubt Super Bowl LI broke the single-day wireless data use mark, with at least 37.6 terabytes used.

The official stats for Wi-Fi at NRG Stadium are finally in, with a mark of 11.8 TB, which is a bit more than the 10.1 TB recorded at last year’s Super Bowl at Levi’s Stadium, the previous top mark. The official stats were reported Thursday by Wi-Fi gear provider Extreme Networks, which posted them on the company website.

New DAS records even without any T-Mobile stats

On the cellular side Verizon Wireless, AT&T and Sprint all set new records, with Verizon reporting 11 TB of use and AT&T reporting 9.8 TB, while Sprint (which ran on its own DAS at NRG Stadium) hit 5 TB. At last year’s Super Bowl Verizon (7 TB) and AT&T (5.2 TB) had set their respective previous high-water marks, while Sprint had reported 1.6 TB at Levi’s Stadium. Even without numbers from T-Mobile the current DAS count is 25.8 TB, much higher than the 15.9 TB cellular total from Super Bowl 50.

(Unfortunately, T-Mobile right now is refusing to provide a total data number — a spokesperson who didn’t want to be quoted claimed on a phone call that the total data number was “not relevant,” and that T-Mobile would not provide a final number. However, we did see a blog post from the company claiming it passed its 2.1 TB total from last year by halftime, so at the very least we could probably accurately add at least another 2.2 TB to the overall DAS total. So we may see a combined total of all cellular and Wi-Fi nearing 40 TB before it’s all counted up, approved or not.)

One of our close friends in the business was at the game, and was kind enough to send us a bunch of Wi-Fi speedtests from NRG Stadium (go check our Twitter timeline at @paulkaps to see the tests linked).

What was interesting was watching the speeds go down when “spike” events occurred, like touchdowns and the end of Lady Gaga’s halftime show. The incredible comeback by the New England Patriots to claim a 34-28 overtime victory kept the network busy through the night, and after the game as well during the awards ceremony.

Tom Brady with the Lombardi Trophy. Credit: AP / Patriots.com

Tom Brady with the Lombardi Trophy. Credit: AP / Patriots.com

New record for take rate

According to Extreme, fans at NRG Stadium also set new high-water marks for unique connections to the network as well as for peak concurrent connections. At Super Bowl LI Extreme said it saw 35,430 fans connect to the network, a 49 percent take rate with the attendance of 71,795. Last year at Super Bowl 50 at Levi’s Stadium a total of 27,316 fans connected to the network out of 71,088 attending, a 38 percent take rate.

On the peak concurrent-connection side, Super Bowl LI set a new mark with 27,191 fans connected at one time, according to Extreme. At the Super Bowl 50, the top concurrent-connected mark was 20,300.

Extreme also released some social-media statistics, claiming that 1.7 TB of the Wi-Fi total was social media traffic. Leading the way in order of most users to fewer were Facebook, Instagram, Snapchat and Twitter. Interestingly, Snapchat consumed almost as much data as Facebook, according to pie graphs in the Extreme infographic, which did not provide any actual numbers for those totals. Extreme also did not report what is typically the highest use of bandwidth in any stadium situation, that being Apple iOS updates and Google Gmail activity.

The NFL, which had its own game-day application for Super Bowl LI, has not released any statistics about app use.

Congrats to all the carriers, integrator 5 Bars and Wi-Fi gear supplier Extreme Networks.

THE NEW TOP 6 FOR WI-FI

1. Super Bowl 51, NRG Stadium, Houston, Feb. 5, 2017: Wi-Fi: 11.8 TB
2. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB
3. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB
4. Super Bowl 49, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB
5. Alabama vs. Texas A&M, Kyle Field, College Station, Texas, Oct. 17, 2015: Wi-Fi: 5.7 TB
6. Pittsburgh Steelers vs. New England Patriots, AFC Championship Game, Gillette Stadium, Foxborough, Mass., Jan. 22, 2017: Wi-Fi: 5.11 TB

THE NEW TOP 4 FOR TOTAL USAGE

1. Super Bowl 51, NRG Stadium, Houston, Feb. 5, 2017: Wi-Fi: 11.8; DAS: 25.8 TB**; Total: 37.6 TB
2. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB; DAS: 15.9 TB; Total: 26 TB
3. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB; DAS: 6.56 TB**; Total: 12.79 TB**
4. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB; DAS: 1.9 TB*; Total: 8.6 TB*

* = AT&T DAS stats only
** = AT&T, Verizon Wireless and Sprint DAS stats only

Verizon goes under concrete to bolster NRG Stadium DAS for Super Bowl LI

Nodes on wheels, or NOWs, provide extra coverage for Verizon Wireless in Houston for Super Bowl LI. Credit: Verizon Wireless

Nodes on wheels, or NOWs, provide extra coverage for Verizon Wireless in Houston for Super Bowl LI. Credit: Verizon Wireless

In a slight twist from its strategy for last year’s Super Bowl, Verizon Wireless has installed DAS antennas underneath the concrete flooring of lower-tier seats at Houston’s NRG Stadium, to provide extra bandwidth for the expected high wireless data usage at Super Bowl LI.

Last year at Levi’s Stadium in Santa Clara, Calif., Verizon beefed up its distributed antenna system (DAS) with under-seat antennas it designed specifically for use in stadiums. The idea of mounting antennas under seats, a growing trend in the stadium Wi-Fi world, is gaining traction as another method of bringing signals closer to fans, especially in places (like lower bowl seats) where there are no overhangs or other places to mount gear.

And while Verizon has been preparing for Sunday’s big game at NRG Stadium for years, that didn’t stop the company from “continually tweaking” its network preparations, according to Leo Perreault, executive director of network operations for Verizon’s South Central market, a region that stretches from west of Florida to Arizona, including Houston. In a phone interview this week, Perreault said that Verizon installed the under-concrete antennas during the middle of the 2016 football season, giving the company “some good experience” with the deployment ahead of Sunday’s game.

Under concrete = easier install and maintenance

A view inside the head end room that runs Verizon's NRG Stadium DAS. Credit: Verizon Wireless

A view inside the head end room that runs Verizon’s NRG Stadium DAS. Credit: Verizon Wireless

It might not be well known outside of wireless networking circles, but signals will travel through concrete; many early stadium Wi-Fi designs (and some current ones, including a new network installed at the Pepsi Center in Denver) use antennas mounted under concrete floors, pointing up. Though fixed under-seat antennas can provide better coverage, Perreault said the ease of deployment made putting the additional DAS antennas underneath the floor a better option in Houston.

“This way [under the concrete] is non-intrusive,” Perrault said, noting that the devices are also not affected by stadium power-washing units. The decision may have been influenced by the fact that NRG Stadium’s new Wi-Fi network had a big issue with moisture in under-seat AP placements, forcing a mid-season rip and replace for all the under-seat Wi-Fi APs.

Even though antennas under concrete are not as powerful, Perrault said Verizon is “very pleased with the performance. It’s a good compromise.”

Biggest stadium DAS?

Between the game being the Super Bowl and it being in Texas, there’s no shortage of hyperbole surrounding the game and all its attendant facets, including the network technology. But when Perreault claims that the DAS Verizon has installed for NRG Stadium “might be the largest we have anywhere,” that might be true since it also serves adjacent properties including the NRG Convention Center, the NRG Arena and an outdoor DAS in the surrounding spaces. In addition to Houston Texans games, NRG Park (which includes the stadium) is also host to the Houston Livestock Show and Rodeo, which humbly bills itself as the “world’s largest livestock show and richest regular-season rodeo.”

An alien spaceship, or a temporary cell tower from Verizon? You choose.

An alien spaceship, or a temporary cell tower from Verizon? You choose.

Inside NRG Stadium, Perreault said the new Verizon DAS (built before the 2015 season) has more than 900 antennas. As neutral host, Verizon will also provide access to AT&T and T-Mobile on its network; Sprint, which built a previous DAS at NRG, will continue to run on that system.

Outside the stadium and around Houston, Verizon has done the usual big-event preparations, with lots of permanent and temporary macro network improvements, and portable units like COWs (cells on wheels) and smaller NOWs (nodes on wheels). You can review all the Verizon preparations in a company blog post.

As previously reported in MSR, Verizon also helped foot part of the bill for the new NRG Stadium Wi-Fi network, a deal that will give Verizon a reserved claim to 40 percent of the Wi-Fi network’s capacity, according to Perreault.

Whether or not Super Bowl LI breaks the wireless data consumption records set at last year’s game remains to be seen, but Perreault said there doesn’t seem to be any slowing down yet of the perpetual growth in wireless data use at stadiums, especially at big events like the Super Bowl.

“Fans just seem to find ways to consume whatever additional bandwidth you provide,” he said.

NFL builds its own Super Bowl app, with no concessions delivery and fewer replays

Screen shot of map function on NFL Super Bowl LI app.

Screen shot of map function on NFL Super Bowl LI app.

The NFL has built its own Super Bowl mobile app, breaking with a recent history of using stadium-app specialists like VenueNext and YinzCam to develop specific apps for Super Sunday.

Also unlike recent years, the NFL’s Super Bowl app will not feature instant replays or have any kind of food or drink delivery services. Instead, there appears to be a big focus on promoting Super Bowl events (especially those for this weekend) and for helping out of town tourists find their way to Super Bowl events and to the game itself.

Curiously, an interview about the app with the NFL’s CIO claimed that this year’s app will also be the first to include the ability for fans at the game to watch Super Bowl commercials. The story also claims without any attribution that “In the past, commercials weren’t on the app in order to avoid using too much bandwidth in the stadium.” However, at the most recent Super Bowls, including the past two, stadium bandwidth has been more than sufficient enough to stream plenty of video. And in fact, both of the last two Super Bowl apps have included the ability for fans at the game to see Super Bowl commercials.

Last year’s app, developed by VenueNext for the Levi’s Stadium hosting of Super Bowl 50, definitely showed Super Bowl commercials, part of what the San Francisco 49ers network team said was a record-breaking day of app-based video watching. The Super Bowl 49 app, built by YinzCam, also included Super Bowl commercials according to this NFL video and according to our previous reporting.

Fewer replays, no food or beverage delivery service

Screen shot of transportation info links from Super Bowl LI app.

Screen shot of transportation info links from Super Bowl LI app.

And even though NFL CIO Michelle McKenna-Doyle is quoted in the previous story about the new app as saying “You won’t feel like you’re using two separate apps as fans have in the past,” the Super Bowl LI app contains a link to download the separate NFL Mobile app, which is apparently where Super Bowl highlights and replays will live. There was no confirmation from the NFL or Verizon about whether or not fans in the stands would be able to watch the live broadcast of the game via NFL Mobile. Fans not at the game will be able to use NFL Mobile to watch the game on cellular devices; fans can also stream the game from the FoxSports website, for PCs or tablet devices.

This year’s app will also not include any way for fans to use the app to order food or beverage delivery to their seats; last year’s app did have the ability to order in-seat delivery of beverages or to place an order for food and beverage express pickup, a service used for 3,284 orders. NRG Stadium, however, does not offer full-stadium in-seat ordering like Levi’s Stadium does; the stadium does have serving staff with wireless devices providing in-seat ordering services for club sections, which will likely be in use at the Super Bowl as well.

Indiana Pacers upgrade Wi-Fi at Bankers Life Fieldhouse

Bankers Life Fieldhouse, home of the Indiana Pacers. Credit all photos: Frank McGrath/Indiana Pacers

Bankers Life Fieldhouse, home of the Indiana Pacers. Credit all photos: Frank McGrath/Indiana Pacers

Whenever you undertake a Wi-Fi retrofit project, one thing is for certain: You can always expect surprises along the way.

For the Indiana Pacers, the biggest surprise in their recent renovation of the Wi-Fi network at Bankers Life Fieldhouse was finding out that their venue already had holes drilled in the concrete under the seats, greatly simplifying (and reducing the cost) of the mainly under-seat deployment that just went live in December.

The new 400-plus AP network, using gear from Ruckus, replaces one of the NBA’s first in-stadium Wi-Fi networks, one built and run by SignalShare using gear from Xirrus. With SignalShare now in bankruptcy and facing legal charges of fraudulent behavior, the Pacers went a different route for their new network, which is part of a plan to bring more digital-based fan services to the 17-year-old venue in downtown Indianapolis, which seats roughly 18,000 for basketball games.

According to Kevin Naylor, vice president of information technology, Pacers Sports and Entertainment, that plan got an unexpected (and welcome) boost when the Pacers’ IT team looked and found pre-drilled holes underneath many of the seats, covered up with temporary aluminum plates. With Ruckus able to use the pre-drilled holes for its under-seat Wi-Fi design, the Pacers were able to save “hundreds of thousands of dollars” in deployment costs, Naylor said.

A new digital plan for fans

Editor’s note: This profile is from our latest STADIUM TECH REPORT, the ONLY in-depth publication created specifically for the stadium technology professional and the stadium technology marketplace. Read about the Sacramento Kings’ new Golden 1 Center and the new Wi-Fi network for the Super Bowl in our report, which is available now for FREE DOWNLOAD from our site!

Leading the venue’s new digital direction is Ed Frederici, chief technology officer, Pacers Sports and Entertainment, who joined the organization in the fall of 2015, after spending almost 6 years as the CTO of ExactTarget, a marketing automation provider that was aquired by Salesforce in 2013.

Though he came into the job “relatively ignorant of sports,” Frederici said he saw “a really interesting problem to solve” revolving around the ongoing evolution of the live-event fan, and who the new attendee was. With a plan to help drive the fan engagement through technology, Frederici, Naylor and the Pacers’ organization began a thorough assessment of Wi-Fi gear providers as part of their plan to bring a new network to Bankers Life Fieldhouse, replacing one that didn’t stand up to current use patterns.

“The old network tapped out when it got to about 3,000 [concurrent] users,” Frederici said.

Pacers director of IT Kevin Naylor shows off a new under-seat Wi-Fi AP

Pacers director of IT Kevin Naylor shows off a new under-seat Wi-Fi AP

According to Frederici, the Pacers looked at “all the major providers” of Wi-Fi gear, testing implementations live by putting gear into mobile merchandise-selling stands in use on the stadium concourses. The final decision, Frederici said, came down to a battle between Ruckus and Xirrus, with Ruckus the final winner.

Under seat the best option

According to Bart Giordano, vice president for business development and strategic partnerships, for Brocade’s Ruckus business unit, going under-seat with Wi-Fi seems to be the direction large public venues are all headed in.

“It [under seat deployment] is sort of standard now,” said Giordano. “You really need to have users close to the APs, and it’s hard to achieve that with overhead.”

With just over 430 APs in the new network, Frederici was worrying about the drilling costs — until it turned out that most of the drilling had already been done, apparently as part of the arena’s original electrical configuration.

“Seventeen years ago, cables were much thinner, and it looks like [the holes] were cored for electrical,” Frederici said. “But it worked out fabulously.”

And like several other venues have done recently, the Pacers have decided to scrap support for fan-facing services on the 2.4 GHz spectrum, which makes administration of the fan Wi-Fi network easier and cheaper. The team will still keep some 2.4 GHz connections for back of house use.

With 2.4 GHz, Naylor said, “the noise level just got really bad in the lower bowl. It’s much easier to go to [only] 5 GHz. Every phone made now has 5 GHz.” For the older phones, Naylor said, the arena’s neutral-host DAS run by ExteNet Systems can provide connectivity, with AT&T and Verizon Wireless already on the system with plans to add more carriers in 2017.

While the Pacers currently have a basic YinzCam-based game-day app, Frederici is looking forward to more services in the future, including the possibility of having amenities like live parking and traffic information available via the app, as well as blue-dot wayfinding to the seat. For this year, the Pacers have already added concession and restroom wait time alerts to the app, the first step in a planned process of greater digital engagement.

“We want to own the experience from your driveway to the stadium, then back home,” Frederici said. Part of the new network deal includes analytics software services from Ruckus partner Purple, which helps teams mine data from fan interaction with the Wi-Fi network.

“We’re excited to see what kind of data we can pull from them [Purple],” Naylor said.

AT&T: 23+ TB of cellular data used on D.C. Mall last weekend

AT&T's new "drum" antennas at Coachella. Photos: AT&T.

AT&T’s new “drum” antennas at Coachella. Similar antennas were used in Washington, D.C, this past weekend. Photos: AT&T.

The two big events on the National Mall in Washington, D.C., last weekend produced big wireless-data usage numbers, with AT&T reporting it saw 23.3 terabytes of cellular data used on the temporary and permanent networks it used to serve the National Mall.

To handle the traffic from huge public crowds for both Friday’s Presidential inauguration ceremonies and for Saturday’s Women’s March on Washington, AT&T deployed seven “super COWs” or cell towers on wheels to the area, joining a local Distributed Antenna System (DAS) network as well as local macro sites.

According to AT&T, it saw 4.3 TB of traffic on its National Mall networks on Thursday, followed by 8.8 TB on Friday, 8.3 TB on Saturday and 1.9 TB on Sunday.

Verizon Wireless, meanwhile, provided statistics for traffic on its National Mall-area networks for Friday’s events, but would not provide specific totals for Saturday’s march. According to Verizon, its customers used “approximately 7 terabytes of data” Friday, with two-thirds of that activity being used on social media applications and websites, with Facebook and SnapChat being the top two apps used by Verizon customers. According to Verizon, data sent from customer devices — meaning network uploads — was four times the typical daily average.

Verizon did not reply to emails and voice messages asking why it would not provide data totals from Saturday’s event in D.C.