Stadium Tech Report: THE COLLEGE FOOTBALL ISSUE looks at university Wi-Fi deployments

collegethumbIf there was a college football playoff for stadium wireless network deployments, which four teams would be in? Electing myself to the committee, I think my top picks would be the same venues we’re profiling in our latest Stadium Tech Report – Baylor, Nebraska, Stanford and Texas A&M. All four are pursuing high-end networks to support a better fan experience, leading the way for what may turn out to be the largest “vertical” market in the stadium networking field – sporting venues at institutions of higher learning.

To be sure, network deployments at major universities in the U.S. are still at the earliest stages — in our reporting for our latest long-form report, we found that at two of the top conferences, the SEC and the Pac-12, only four schools total (two in each conference) had fan-facing Wi-Fi, with only one more planned to come online next year. Why is the collegiate market so far behind the pro market when it comes to network deployment? There are several main reasons, but mostly it comes down to money and mindset, with a lack of either keeping schools on the sidelines.

Leaders look for NFL-type experiences

But at our “playoff” schools, it’s clear that with some ready budget and a clear perspective, college stadiums don’t need to take a back seat to anyone, pro stadiums included. The networks, apps and infrastructure deployed for this season at Baylor’s McLane Stadium and Nebraska’s Memorial Stadium are among the tops anywhere in sports, and the all-fiber infrastructure being put in place at Texas A&M should make that school’s Kyle Field among the most-connected if all work gets completed on time for next football season. Read in-depth profiles on these schools’ deployments, along with team-by-team capsule technology descriptions and an exclusive interview with Mississippi State athletic director Scott Stricklin in our latest report, available for free download from our site.

We’d like to take a second here to thank our sponsors, without whom we wouldn’t be able to offer these comprehensive reports to you free of charge. For our fourth-quarter report our sponsors include Crown Castle, SOLiD, Extreme Networks, Aruba Networks, TE Connectivity, and Corning.

Stadium Tech Report: Arizona Cardinals get stadium ready for Super Bowl with Wi-Fi upgrade

University of Phoenix Stadium. Credit all photos: Arizona Cardinals.

University of Phoenix Stadium. Credit all photos: Arizona Cardinals. (click on photos for larger image)

As they get set to host their second Super Bowl this February, the IT team at the University of Phoenix Stadium in Glendale, Ariz., knows now what they didn’t know then: The big game requires a big wireless network. Bigger than you think.

“It’s funny to look back now on that first Super Bowl,” said Mark Feller, vice president of information technology for the Arizona Cardinals, speaking of Roman numeral game XLII, held in the barely 2-year-old facility on Feb. 3, 2008. With a couple Fiesta Bowls and one BCS championship game (2007) under his belt in a facility that opened with Wi-Fi and DAS, Feller said he and his team “thought we had a good handle” on what kind of network was required for a Super Bowl crowd.

The NFL, he said, begged to differ. Those college games might have been big, but the Super Bowl was bigger.

“We had the Fiesta Bowl that year at night, and when the game was over there were people from the NFL there wanting to know when they could set up,” said Feller in a recent phone interview. “This year, we’re much better prepared. We know what the water temperature is this time.”

Rip and replace, with more and better gear

Wi-Fi railing antennas

Wi-Fi railing antennas

For Super Bowl XLIX, scheduled to take place in Glendale on Feb. 1, 2015, Feller and his team have not just tuned up their network — they have done a full rip and replace of the Wi-Fi system, installing new Cisco gear from back end to front, in order to support a wireless game-day demand that is historically second to none. Integrator CDW has led the Wi-Fi effort and Daktronics and ProSound did the installation of new video screens, and neutral host Crown Castle has overseen a revamp of the DAS system, again with more antennas added to bolster coverage. In all, there has been more than $8 million in wireless improvements before this year, Feller said, as well as another $10 million for two new video boards that are each three times larger than what was there before.

“The last three or four years there have been things we knew we needed to improve [before the Super Bowl],” Feller said. After extensive work with the NFL’s technical team — this time well before the Fiesta Bowl — Feller oversaw a “top to bottom” refurbishment that included replacing core Cisco networking gear with newer gear, and new and more Wi-Fi access points that now total somewhere north of the 750 mark, with some more to be added before the big game. The new network, which was in place for the start of the current NFL season, has undergone testing by CDW at each home game, Feller said. CDW also plans to expand the network outside the stadium before the Super Bowl, in part to handle the extra events that take place not just on game day but in the days leading up to the game.

“The plan is to install more [coverage] outside, in the plaza areas,” Feller said.

When it opened in 2006, the $455 million University of Phoenix Stadium was one of the first with full-bowl Wi-Fi, using Cisco gear from the inside out. “Cisco was in here before they called it [their solution] ‘connected stadium’,” Feller said. From core network switches to firewalls to edge switches, this year there is all new Cisco gear in the venue, as well as new 3700 series APs, with panel antennas and antennas in handrails.

“Handrail [antennas] are sometimes a bit of a challenge, because you need to drill through concrete that’s 40 feet up in the air, behind another ceiling,” said Feller, describing one particular design challenge. Another one was mounting antennas on drop rods from the catwalks below the stadium’s retractable roof, to serve the upper-area seating. There are also some new Wi-Fi APs on the front row of the seating bowl, pointing up into the crowd.

“It was a fun project,” Feller said.

Stadium with roof open

Stadium with roof open

All on board for the DAS

The upgrade for the stadium’s DAS, led by Crown Castle, was just finished a few weeks ago, Feller said, and included more coverage outside the stadium as well, with antennas placed on light poles and on the stadium’s shell.

“Crown Castle did a great job of managing the carriers” on what is a 48-sector DAS, Feller said. “It [the upgrade] really required a lot of creative thinking from their engineers.”

Since the stadium was originally designed with wireless in mind, Feller and his team didn’t need to build new head end room for the DAS upgrades. “But I wouldn’t say we have plenty of space left,” he added. “We’ve got a lot of new equipment.”

Though all the major carriers are expected to be on the DAS by the big game, league partner Verizon Wireless should have some special projects up its sleeve for the big game, including another demonstration of its LTE Broadcast technology, which optimizes things like live video over LTE cellular links.

New Cardinals app a preview of Super Bowl version?

The Cardinals also had a new version of the game-day team app for this season, built by stadium-app leader YinzCam. According to Feller the new app supports three different live video feeds, as well as instant replays.

Wi-Fi antenna on railing

Wi-Fi antenna on railing

“It’s really cool to have that ability to watch things like a touchdown pass at the end of the game,” Feller said. And while no details have yet been revealed, in an interview with NFL CIO Michelle McKenna-Doyle earlier this year MSR learned that the league and YinzCam are working on a Super Bowl app with its own new bells and whistles. (Stay tuned for more info on the Super Bowl app.)

In addition to two more regular-season home games in December, the University of Phoenix Stadium will have at least a couple more dry runs to help test the network, during the Dec. 31 Fiesta Bowl and during the NFL’s Pro Bowl, scheduled for Jan. 25. And though the Cardnials lost to the Atlanta Falcons Sunday, at 9-3 they are still tied with the Green Bay Packers for the best record in the NFC, something that has the Phoenix faithful optimistic about the postseason.

“We’re going to get some more test runs, on New Year’s Eve and during the Pro Bowl,” Feller said. “And maybe some home playoff games as well!”

(more photos below)

Wi-Fi antenna in roof rafters

Wi-Fi antenna in roof rafters

More antennas in rafters

More antennas in rafters

Wi-Fin antenna under overhang

Wi-Fi antenna under overhang

Stadium Tech Report — NFL stadium technology reports — NFC East

Editor’s note: The following team-by-team capsule reports of NFL stadium technology deployments are an excerpt from our most recent Stadium Tech Report, THE FOOTBALL ISSUE. To get all the capsules in one place as well as our featured reports, interviews and analysis, download your free copy of the full report today.

NFC EAST

Reporting by Chris Gallo

Dallas Cowboys
AT&T Stadium
Seating Capacity: 105,121
Wi-Fi – Yes, 1,525 access points
DAS – Yes, 1,374 antennas
Beaconing – No

AT&T Fan Experience board with single message

AT&T Fan Experience board with single message

Everything is bigger in Texas – including connectivity. AT&T Stadium features 1,525 Wi-Fi access points and 1,374 DAS antennas. That’s enough cellular capacity to service the small suburb of McKinney, Texas. While watching Tony Romo target Dez Bryant, Cowboys fans will notice a new 130-foot LED display along the east platform. The AT&T Fan Experience board features 40 robotic panels that work in combination with other stadium displays to entertain fans all game along. And don’t forget the big TV hanging in the middle of the place!

New York Giants
MetLife Stadium
Seating Capacity: 82,500
Wi-Fi – Yes, 850 access points
DAS – Yes, over 600 antennas
Beaconing – No

The Super Bowl champions just three years ago, it appears their bunk mates’ play is rubbing off on the Giants. Like the cross-town Jets, the G-Men missed the playoffs for the second straight season last year. If it’s any consolation, the Giants still share one sports top venues in MetLife Stadium. AT&T and Verizon gave enough love and attention the stadium last year as host of Super Bowl XLVII. Over 600 DAS antennas, 850 Wi-Fi access points, Cisco StadiumVision with over 2,100 HD TVs around the concourse – that’s a recipe for a good time at a game, at least from a connectivity standpoint.

Philadelphia Eagles
Lincoln Financial Field
Seating Capacity: 69,176
Wi-Fi – Yes, 600+ access points
DAS-Yes
Beaconing – No

Lincoln Financial Field is one of the “greenest” stadiums in all of professional sports. And that’s not just because of the Eagles’ colors. All (100 percent) of the Eagles operations are powered by the sun and the wind. The stadium’s connectivity is something fans can get behind too. With over 600 Wi-Fi access points and a group of Extreme Networks’ “Wi-Fi coaches”, the Eagles are doing everything they can to make sure fans are connected at games.

Washington Redskins
FedEx Field
Seating Capacity: 85,000
Wi-Fi – Limited / club level only
DAS-Yes
Beaconing – No

A new coach, name controversy, and a $27 million dollar renovation underway at FedEx Field. It’s been an offseason full of distractions in the nation’s capital. The good news: The team just signed a deal with supplier Huawei to put Wi-Fi in the stadium, first in the club seats and then (hopefully) next year in the full seating bowl.

FedEx Field gets suite-level Wi-Fi from Huawei

FedEx Field, home of the NFL’s Washington, D.C. franchise, is getting Wi-Fi service in its 330 suites from Chinese telecommunications gear provider Huawei, according to an announcement from the team and the company.

Rob Verkon, senior solution sales and marketing manager for Huawei Enterprise USA, said in a recent phone interview that the Wi-Fi network is currently being installed with a goal of making it available “for the final games [of the season] in December.” Washington hosts the St. Louis Rams on Dec. 7, and then has two more scheduled December regular-season home games, Dec. 20 vs. the Philadelphia Eagles and Dec. 28 against the Dallas Cowboys.

The deal is the first major stadium Wi-Fi win in the U.S. for Huawei, which claims to also have “networking and infrastructure” deployments in major stadiums in Europe. A major competitor to large U.S. networking firms like Cisco, Huawei has been at the center of controversy in recent years, including being tabbed as a security threat by U.S. government officials, and later as a reported target for N.S.A. surveillance.

According to the press release, Huawei has entered into a multi-year sponsorship deal as the “official technology partner” for the Washington team. Though Verkon would not reveal the specifics of the network deployment costs, he did confirm that the company will receive some payment in the form of advertising, signage and other sponsorship identification.

Verkon said Huawei will be deploying 170 802.11ac access points to cover the 330 suites in the 85,000-seat FedEx Field. Though there is no mention of further deployment arrangements in the press release, Verkon said that “the eventual project” will include bringing Wi-Fi to the full seating bowl for next season.

Currently, the Wi-Fi situation at FedEx is a bit of a black hole; according to Verkon there is some existing Wi-Fi service in the stadium, and the team’s home page even lists Wi-Fi in its stadium guide (with the message “Fans in the seating bowl may access the free wireless network. Please see signs around the stadium for for login information.”). But in our research for NFL Wi-Fi services for our most recent Stadium Tech Report, MSR could not confirm that Wi-Fi services were available for the entire seating area.

Stadium Tech Report: Nebraska adds Cisco-powered Wi-Fi, IPTV to Memorial Stadium

Memorial Stadium, University of Nebraska. Credit all photos: University of Nebraska.

Memorial Stadium, University of Nebraska. Credit all photos: University of Nebraska.

The state of Nebraska might not have a professional football team, but University of Nebraska fans now have a pro-style wireless experience at home games thanks to a high-density Wi-Fi network and IPTV features recently installed by the school at the 85,000-plus seat Memorial Stadium.

Combined with a new mobile app that delivers multiple live video streams and replay options, the loyal Cornhusker fans — who’ve come to Lincoln, Neb., and filled Memorial Stadium to the brim every game day since the 1960s — now have a wireless fan in-game experience among the best anywhere, collegiate or pro. Installed for use this season by CDW, the new network features Cisco Wi-Fi gear and Cisco’s StadiumVision and StadiumVision Mobile systems, as well as a new app developed by NeuLion and additional video-streaming capabilities from EVS.

“We have the most loyal fans in the country,” said Kelly Mosier, director of digital communications for the University of Nebraska athletic department. Part of putting in a high-definition wireless network, Mosier said, was to meet growing fan connectivity needs and to “stay ahead of the curve” in stadium experiences. “We wanted to reward the fans for being so loyal, and give them a new experience to brag about,” Mosier said.

A look at video options in the Husker app

A look at video options in the Husker app (click picture for bigger view)

One of the first collegiate programs to install big TV screens in its stadium, Nebraska now has a Wi-Fi network with more than 800 access points, both inside Memorial’s seating bowl as well as outside, at entrance and parking areas. In addition to synchronized IPTV broadcasts on stadium flat-panel screens, Nebraska fans also have access to a wide range of live video and video replays of game-day action, thanks to the new game-day app.

According to Dan Floyd, director of information technology for Nebraska athletics, the new network is already a hit, with a peak of 25,000 simultaneous users during the second game this fall. And even though Floyd and Mosier are aware of some problem areas, as Floyd said there has already been “a lot of positive feedback” from the technology upgrades.

Better sound, better connectivity

The network deployment was part of a recent $12.3 million upgrade to the football facility, which has been the home of the Huskers since 1923. Included in the total spend was also a revamp of the public-address and sound systems, but according to Mosier “the biggest chunk” of the spending went to cover the Wi-Fi deployment.

The need for better connectivity became apparent a few years back, Mosier said, and it set in motion a “very long process” of a couple years in length in which Nebraska officials looked at other existing stadium deployments and technology choices before making their decisions. One of the easier picks was selecting Cisco as the main Wi-Fi gear supplier, since Mosier said that most of the rest of the school’s campus was already wired with Cisco networking gear. And a recent deployment of Cisco stadium Wi-Fi at Pinnacle Bank Arena in downtown Lincoln, Neb., where the Husker basketball team plays, helped further push the football decision toward using Cisco.

“We wanted to provide a seamless experience for fans between multiple venues, and to play well with the rest of the campus,” Mosier said. “It was just a no-brainer to use a Cisco [Wi-Fi] product.”

Also in Cisco’s favor was its StadiumVision digital display system and its StadiumVision Mobile product, which both bring advanced IPTV features to static stadium screens as well as to mobile apps. StadiumVision allows for synchronization of programming across a wide array of networked displays, while StadiumVision Mobile supports several live broadcast channels that can be used to provide live content to mobile apps. The Brooklyn Nets use StadiumVision Mobile to bring live action channels to fans who visit the Barclays Center for games.

“We wanted to not just provide the networking infrastructure, but on top of that provide something extra,” Mosier said. “Our fans are pretty savvy, and they are looking for things beyond what the casual fan might be looking for. StadiumVision and StadiumVision Mobile makes sense for our fan base. The ‘extra screen’ approach really lets them control their own video experience.”

Some additional replay options in the Husker app

Some additional replay options in the Husker app

On the Huskers’ game day app, Mosier said, fans at the start of the season could choose between three StadiumVision Mobile-powered “channels” that showed the big-screen broadcast, an alternate angle view, and an “all-22” camera that is like what coaches view to see the players across the entire field. A “phase II” of the app live video, which had not yet launched at the time of our interview, will include further user-controlled selections for more camera angles and replays. According to Mosier, the Phase II capabilities are supported by the C-Cast system from EVS. The entire new app, he said, was built by developer NeuLion.

Though Mosier said the live video production for the app was “definitely a growing process” that will require further tuning and learning, he said the system already has impressive performance, with delays between live action and app action at “a second to a second and a half.” Mosier said that Nebraska also plans to bring more live action to the mobile app for basketball season, with multiple camera angles including a “slam cam” based near the rims.

Handrails and fan interference of Wi-Fi signals

While the 800 Wi-Fi APs give Nebraska pretty good coverage throughout the facility, Floyd said the IT team knew it would have connectivity challenges in the north and south end zone stands, since neither of those sides have any structural overhangs.

“Since the north and south sides [of the stadium] don’t have overhangs we knew they would be problematic,” Floyd said. One option used in venues including AT&T Park and Levi’s Stadium, the under-the-seat access point, wasn’t an option at Memorial Stadium, Floyd said, because the seats there aren’t high enough to meet safety requirements for keeping bodies away from the antennas.

One creative way CDW and Nebraska brought Wi-Fi to the north and south stands is via Wi-Fi antennas inside railing enclosures, but those are not without their own challenges. Though the railing antennas get a signal close to fans, the long rows of seats at Memorial Stadium — up to 30 in between rows — means that the “waterbags,” or human bodies, can act as signal-blockers for fans in the middle of the rows.

“The first 10 or 12 people on either side get a pretty good signal,” said Floyd, but he added that the fans in the middle are still a challenge to reach. Both Mosier and Floyd said Nebraska will continue to seek ways to upgrade the Wi-Fi network, including possibly putting APs on top of towers or in other creative deployments.

“We knew it would need tuning, and some tuning is easier than others,” Mosier said. “We know we still have pockets of [connectivity] problems. We knew that when we put in a system like this, it wasn’t going to be perfect on day 1.”

Something to brag about

According to Floyd, Memorial Stadium has had a Verizon DAS in place for several years, which fulfilled most of the fans’ basic cellular connectivity needs since he said that “70 to 75 percent” of Cornhusker fans were Verizon customers. However, the new sound system, with its big speaker arrays, has also given room for AT&T and U.S. Cellular to add some DAS equipment of their own, with antennas mounted right inside the speaker enclosures.

Unlike other schools or teams, the Cornhuskers are not pressed to make money off their wireless network, given the stadium sellout streak that dates back to Nov. 3, 1962. But Mosier said that even the Huskers aren’t immune to the lure of the living room couch, with its comfort, HD screen and close-by food and drink.

“We definitely have a blessed situation [with the sellouts],” Mosier said, while allowing that some fans might still prefer sitting at home. “But you can’t match the experience of being at the venue,” he added. “If we can address the connectivity issues, plus add to the stadium experience [with technology], it’s a win-win for us.”

Using the app at Memorial Stadium

Using the app at Memorial Stadium

Niners: More than 1,000 fans used in-seat food delivery at 2nd Levi’s Stadium preseason game

Screen grab from Levi's Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

Screen grab from Levi’s Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

One of the more unique features the San Francisco 49ers are introducing at their new home this year is the ability for all fans to have food and drink delivered to them, no matter which one of Levi’s Stadium’s 68,500 seats they are in. And according to the Niners, the feature is quickly catching on, with more than 1,000 in-seat orders delivered at the second preseason game at Levi’s on Aug. 24.

At a Levi’s media technology tour Wednesday, the Niners’ tech staff provided an in-depth and up-close look at some of the new stadium’s network infrastructure, including a quick glimpse of one of the several data-center rooms. Dan Williams, vice president of technology for the Niners, said that at the first preseason game at Levi’s on Aug. 17, more fans used the “express pickup” service to order food that they could then pick up at concession stand windows than the seat-delivery feature. But at the second game against the San Diego Chargers, more fans went for the in-seat option, perhaps a sign that Niners fans are learning and testing the new stadium services as they go.

“It’s going to be an ongoing education process,” said Martin Manville, business operations analyst for the Niners and one of the key tech leaders on the Levi’s app team. Manville said the Niners had learned a lot about food delivery in test situations at Candlestick Park last season — and some of those lessons are now evident in the Levi’s delivery menu, which is stripped down to ensure the food runners can get grub to fans before it gets cold (or warm, in the case of cold beverages). According to Manville the average delivery time at the Aug. 24 game was between 10 and 12 minutes, but the team expects that “normal” delays during the regular season will be closer to 15 to 20 minutes per order.

Still, that’s not a bad option if you don’t want to leave your seat and you don’t mind the extra $5 delivery charge. One early interesting stat from the food-delivery process is that the Niners found more orders coming from the “cheap seats” in the north and south stands at Levi’s than from the 50-yard-line seats where the high rollers sit. According to Manville, since the fans in the club seats have easy access to numerous uncrowded concession stands they may not see the need for the in-seat options.

Wi-Fi APs: 600 in Levi’s bowl seating

Under-seat Wi-Fi AP at Levi's Stadium.

Under-seat Wi-Fi AP at Levi’s Stadium.

Some other news nuggets from the tech tour day: According to Williams, of the 1,200 Wi-Fi access points at Levi’s, 600 of those are distributed in the seating areas (aka “the bowl”), with the other 600 placed in suites, concourses and other stadium areas. Williams said the Aruba Networks Wi-Fi antennas are basically split into three types — regular enterprise-type APs for suite and office areas, regular outdoor APs for concourse areas, and more specialized versions (including the under-seat APs) for bowl placements.

— For the app itself, the Niners said that there have been 80,000 downloads so far, with almost half of the season ticket holders having put their ticketing information into the app. The “NiNerds,” the geek-dressed help squad that provides fans with personal assistance with the app, is now at about 50 or 60 strong at each event (originally the team had said it wanted to hire 150 such Wi-Fi coaches). The Niners said the NiNerds will be doing more pro-active app education going forward, doing things like approaching fans in concession lines to see if they know about the express line or in-seat ordering options.

— Though Comcast’s 10-year deal with the Niners calls for the cable provider to bring in two 10-gigabit backbone pipes, the Niners are often quoted as saying they have 40 GB of backbone bandwidth. We solved this mystery today: According to Comcast, the other two 10-GB pipes are a redundant channel from another (unnamed) provider. So: the stadium does have four 10 GB bandwidth pipes, by far the most capacity in any stadium we’ve heard of.

— More traffic stats: Though we will break these down in a separate post, the Niners said that for the Aug. 24 game fans used 1.96 Terabytes of Wi-Fi traffic, just a bit lower than the 2.13 TB used at the first preseason game on Aug. 17. The team also provided some DAS stats, claiming fans used another 1.02 TB of cellular data at the two preseason games combined.

Tech tour photos follow… including a sighting of some (shhhh!) Cisco equipment in the data center racks… click on photos for larger images.

Niners VP of technology Dan Williams (left) and COO Al Guido kick off the Wednesday tech tour.

Niners VP of technology Dan Williams (left) and COO Al Guido kick off the Wednesday tech tour.

Ted Girdner, Comcast VP of business services for California, talks stadium networking.

Ted Girdner, Comcast VP of business services for California, talks stadium networking.

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him.

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him.

Mystery Cisco gear inside Levi's Stadium data center. Alert! Intruder!

Mystery Cisco gear inside Levi’s Stadium data center. Alert! Intruder!

Brocade router at Levi's Stadium data center. One of many. As in, many many.

Brocade router at Levi’s Stadium data center. One of many. As in, many many.

Wi-Fi gear in Levi's Stadium data room.

Wi-Fi gear in Levi’s Stadium data room.

Franks and DAS: DGP DAS antennas above food station.

Franks and DAS: DGP DAS antennas above food station.

Screenshot of food feature on Levi's Stadium app. Note the green light buttons to show expected wait times for express option.

Screenshot of food feature on Levi’s Stadium app. Note the green light buttons to show expected wait times for express option.

Obligatory Levi's Stadium selfie. MSR shirts complete the style.

Obligatory Levi’s Stadium selfie. MSR shirts complete the style.