Stadium Tech Report: Nebraska adds Cisco-powered Wi-Fi, IPTV to Memorial Stadium

Memorial Stadium, University of Nebraska. Credit all photos: University of Nebraska.

Memorial Stadium, University of Nebraska. Credit all photos: University of Nebraska.

The state of Nebraska might not have a professional football team, but University of Nebraska fans now have a pro-style wireless experience at home games thanks to a high-density Wi-Fi network and IPTV features recently installed by the school at the 85,000-plus seat Memorial Stadium.

Combined with a new mobile app that delivers multiple live video streams and replay options, the loyal Cornhusker fans — who’ve come to Lincoln, Neb., and filled Memorial Stadium to the brim every game day since the 1960s — now have a wireless fan in-game experience among the best anywhere, collegiate or pro. Installed for use this season by CDW, the new network features Cisco Wi-Fi gear and Cisco’s StadiumVision and StadiumVision Mobile systems, as well as a new app developed by NeuLion and additional video-streaming capabilities from EVS.

“We have the most loyal fans in the country,” said Kelly Mosier, director of digital communications for the University of Nebraska athletic department. Part of putting in a high-definition wireless network, Mosier said, was to meet growing fan connectivity needs and to “stay ahead of the curve” in stadium experiences. “We wanted to reward the fans for being so loyal, and give them a new experience to brag about,” Mosier said.

A look at video options in the Husker app

A look at video options in the Husker app (click picture for bigger view)

One of the first collegiate programs to install big TV screens in its stadium, Nebraska now has a Wi-Fi network with more than 800 access points, both inside Memorial’s seating bowl as well as outside, at entrance and parking areas. In addition to synchronized IPTV broadcasts on stadium flat-panel screens, Nebraska fans also have access to a wide range of live video and video replays of game-day action, thanks to the new game-day app.

According to Dan Floyd, director of information technology for Nebraska athletics, the new network is already a hit, with a peak of 25,000 simultaneous users during the second game this fall. And even though Floyd and Mosier are aware of some problem areas, as Floyd said there has already been “a lot of positive feedback” from the technology upgrades.

Better sound, better connectivity

The network deployment was part of a recent $12.3 million upgrade to the football facility, which has been the home of the Huskers since 1923. Included in the total spend was also a revamp of the public-address and sound systems, but according to Mosier “the biggest chunk” of the spending went to cover the Wi-Fi deployment.

The need for better connectivity became apparent a few years back, Mosier said, and it set in motion a “very long process” of a couple years in length in which Nebraska officials looked at other existing stadium deployments and technology choices before making their decisions. One of the easier picks was selecting Cisco as the main Wi-Fi gear supplier, since Mosier said that most of the rest of the school’s campus was already wired with Cisco networking gear. And a recent deployment of Cisco stadium Wi-Fi at Pinnacle Bank Arena in downtown Lincoln, Neb., where the Husker basketball team plays, helped further push the football decision toward using Cisco.

“We wanted to provide a seamless experience for fans between multiple venues, and to play well with the rest of the campus,” Mosier said. “It was just a no-brainer to use a Cisco [Wi-Fi] product.”

Also in Cisco’s favor was its StadiumVision digital display system and its StadiumVision Mobile product, which both bring advanced IPTV features to static stadium screens as well as to mobile apps. StadiumVision allows for synchronization of programming across a wide array of networked displays, while StadiumVision Mobile supports several live broadcast channels that can be used to provide live content to mobile apps. The Brooklyn Nets use StadiumVision Mobile to bring live action channels to fans who visit the Barclays Center for games.

“We wanted to not just provide the networking infrastructure, but on top of that provide something extra,” Mosier said. “Our fans are pretty savvy, and they are looking for things beyond what the casual fan might be looking for. StadiumVision and StadiumVision Mobile makes sense for our fan base. The ‘extra screen’ approach really lets them control their own video experience.”

Some additional replay options in the Husker app

Some additional replay options in the Husker app

On the Huskers’ game day app, Mosier said, fans at the start of the season could choose between three StadiumVision Mobile-powered “channels” that showed the big-screen broadcast, an alternate angle view, and an “all-22” camera that is like what coaches view to see the players across the entire field. A “phase II” of the app live video, which had not yet launched at the time of our interview, will include further user-controlled selections for more camera angles and replays. According to Mosier, the Phase II capabilities are supported by the C-Cast system from EVS. The entire new app, he said, was built by developer NeuLion.

Though Mosier said the live video production for the app was “definitely a growing process” that will require further tuning and learning, he said the system already has impressive performance, with delays between live action and app action at “a second to a second and a half.” Mosier said that Nebraska also plans to bring more live action to the mobile app for basketball season, with multiple camera angles including a “slam cam” based near the rims.

Handrails and fan interference of Wi-Fi signals

While the 800 Wi-Fi APs give Nebraska pretty good coverage throughout the facility, Floyd said the IT team knew it would have connectivity challenges in the north and south end zone stands, since neither of those sides have any structural overhangs.

“Since the north and south sides [of the stadium] don’t have overhangs we knew they would be problematic,” Floyd said. One option used in venues including AT&T Park and Levi’s Stadium, the under-the-seat access point, wasn’t an option at Memorial Stadium, Floyd said, because the seats there aren’t high enough to meet safety requirements for keeping bodies away from the antennas.

One creative way CDW and Nebraska brought Wi-Fi to the north and south stands is via Wi-Fi antennas inside railing enclosures, but those are not without their own challenges. Though the railing antennas get a signal close to fans, the long rows of seats at Memorial Stadium — up to 30 in between rows — means that the “waterbags,” or human bodies, can act as signal-blockers for fans in the middle of the rows.

“The first 10 or 12 people on either side get a pretty good signal,” said Floyd, but he added that the fans in the middle are still a challenge to reach. Both Mosier and Floyd said Nebraska will continue to seek ways to upgrade the Wi-Fi network, including possibly putting APs on top of towers or in other creative deployments.

“We knew it would need tuning, and some tuning is easier than others,” Mosier said. “We know we still have pockets of [connectivity] problems. We knew that when we put in a system like this, it wasn’t going to be perfect on day 1.”

Something to brag about

According to Floyd, Memorial Stadium has had a Verizon DAS in place for several years, which fulfilled most of the fans’ basic cellular connectivity needs since he said that “70 to 75 percent” of Cornhusker fans were Verizon customers. However, the new sound system, with its big speaker arrays, has also given room for AT&T and U.S. Cellular to add some DAS equipment of their own, with antennas mounted right inside the speaker enclosures.

Unlike other schools or teams, the Cornhuskers are not pressed to make money off their wireless network, given the stadium sellout streak that dates back to Nov. 3, 1962. But Mosier said that even the Huskers aren’t immune to the lure of the living room couch, with its comfort, HD screen and close-by food and drink.

“We definitely have a blessed situation [with the sellouts],” Mosier said, while allowing that some fans might still prefer sitting at home. “But you can’t match the experience of being at the venue,” he added. “If we can address the connectivity issues, plus add to the stadium experience [with technology], it’s a win-win for us.”

Using the app at Memorial Stadium

Using the app at Memorial Stadium

Stadium Tech Report: THE FOOTBALL ISSUE arrives, with extensive coverage of Levi’s Stadium launch and tech reports on all 31 NFL stadiums

STR3_ThumbMobile Sports Report is pleased to announce the arrival of THE FOOTBALL ISSUE, our third Stadium Tech Report for 2014. As the title suggests this long-form report focuses on technology deployments at U.S. football stadiums, with an extensive inside look at the technology inside Levi’s Stadium, the new facility for the San Francisco 49ers. The report is available for free download from our site.

In addition to our Levi’s coverage, the Q3 issue of Stadium Tech Report also includes team-by-team reports on all 31 NFL stadiums, with a focus on Wi-Fi and DAS deployments. Our research found that while there are still 10 stadiums without fan-facing Wi-Fi, there is a lot of innovation going around league-wide, including big new digital displays in Jacksonville and Dallas, and new Wi-Fi and app deployments in other facilities.

Included in the report is an exclusive MSR interview with Michelle McKenna-Doyle, the NFL’s chief information officer, who talks about how the league office acts as a guide to helping teams with their tech deployments. We also have additional insight, analysis, and more tech profiles, and the good news is it is all free to read! Simply head to our report download page and get your free copy today!

We’d like to take a quick moment to thank our report sponsors, without whom we wouldn’t be able to offer such extensive original reporting and analysis free of charge. Our list for the Q3 2014 report includes SOLiD, Crown Castle, TE Connectivity, Extreme Networks, Aruba Networks, Mobilitie and DAS Group Professionals.

Stadium Tech Report: DAS Group Professionals makes a name for itself with Levi’s Stadium DAS

DAS antennas hanging from a Levi's Stadium overhang. Credit, all Levi's photos: Paul Kapsutka, MSR

DAS antennas hanging from a Levi’s Stadium overhang. Credit, all Levi’s photos: Paul Kapsutka, MSR

If you look around at the walls, ceilings and overhangs at Levi’s Stadium, it’s hard to miss the small square boxes with the off-white color and a “DGP” logo in one corner. While the wires hanging out the back of each box make it an easy guess that the equipment has something to do with wireless networks, even many industry insiders may not know the company behind the boxes and the three-letter acronym.

Meet DAS Group Professionals, the Bay Area firm in charge of deploying a distributed antenna system (DAS) to make sure your cell phone gets a good signal at San Francisco 49ers games and any other event inside the 68,500-seat Levi’s Stadium. And while you might not be familiar with DGP, rest assured the company is extremely familiar with cellular deployments for large public venues, having installed similar DAS networks for airports, casinos and hotels, and even for San Francisco’s BART train system. Of course, most of that work was done when the company was called Forza Telecom, before changing its name to DGP earlier this year, another reason why “DGP” may not have rung any bells.

“It may appear like we just fell out of the sky, but we’ve actually built quite a few [DAS] systems,” said Steve Dutto, president of DGP, in a recent phone interview. “We’ve got years of experience.”

DAS antennas above a food stand

DAS antennas above a food stand

DAS: The Rodney Dangerfield of stadium connectivity

One thing that keeps firms like DGP in the shadows is the relative obscurity of DAS itself. While most people generally understand how cell phones work — you turn on your phone, and it connects to an antenna somewhere on a tower or rooftop — in crowded public facilities like stadiums, traditional cellular networks with towers several miles apart can’t handle the concentrated capacity. To provide connectivity for areas with large crowds, the latest tactic is to deploy a DAS, a network of lots of smaller antennas. Originally deployed in places like office buildings, hotels and convention centers, DAS is rapidly gaining favor in stadiums and arenas, helping to alleviate the “no signal” problem that has cropped up in many venues the past few years.

And while stadium Wi-Fi gets lots of headlines whenever it gets deployed — probably thanks again to the widespread understanding of how Wi-Fi works — there are already far more DAS deployments in stadiums than Wi-Fi, mainly because cellular carriers will pay almost all the associated costs of a DAS buildout to make sure their customers get a signal. According to our most recent 2014 State of the Stadium survey, 71.4 percent of our respondents said they had a full DAS at their facility, while only 35.7 percent had fan-facing Wi-Fi.

Steve Dutto, president, DGP

Steve Dutto, president, DGP

How does a DAS work? Usually, either a major cellular carrier or a third-party “neutral” host like DGP will build out the antenna infrastructure, which includes many small antennas and then cables to bring the connections back to a wiring room or data center. There, cellular carriers install their own cell-tower back-end gear to authenticate customers and to provide a connection to the company networks and the Internet. Since it’s in the cellular carriers’ interest to keep their customers connected (and using billable minutes and data), carriers will often pay the full cost of a DAS infrastructure by building and running it themselves. In the case like Levi’s, where DGP is the “neutral host,” DGP builds the infrastructure and then charges cellular carriers to use it.

Such deals are rarely publicized, and Dutto would not comment on how much each carrier was being charged to use the Levi’s DAS — though industry gossip has the figure somewhere around $5 million per carrier per year. And just like fight club, for many deployments the first rule of DAS is that you don’t talk about DAS, because no cellular carrier ever wants to admit that its network might need help. So just like Rodney Dangerfield, DAS often doesn’t get a lot of public respect. But at Levi’s Stadium and many other sports and entertainment venues, DAS is a booming business that would be sorely missed if it wasn’t there.

The ‘dream and the nightmare’ of building the Levi’s DAS

Now that DGP’s 700-plus antenna DAS deployment is up and running at Levi’s, Dutto can breathe a small bit easier. While the network is good business for the company and an obvious prominent calling card for the future, the aggressive deployment timeframe probably isn’t something Dutto is eager to repeat.

“Levi’s Stadium was both a dream and a nightmare [for DGP],” Dutto said, due in part both to the Niners’ aggressive performance expectations and the rapid buildout schedule. Of course, DGP was somewhat used to working quickly with the Niners — when the company put a DAS in Candlestick Park back in 2012 to solve that stadium’s legendary lack of connectivity, Dutto said it was deployed “in about 90 days.”

The success of the Candlestick deployment, Dutto said, led to the Niners offering the Levi’s DAS gig to DGP. With it, however, came the need to match the team’s out-front statements about how the stadium was going to be the best ever in terms of wireless connectivity. And with many people not knowing or not bothering to switch their phones to Wi-Fi, the Levi’s DAS, like most stadium DAS deployments, would probably handle most of the wireless connections.

Door sign for head end equipment room at Levi's Stadium (there are many of these)

Door sign for head end equipment room at Levi’s Stadium (there are many of these)

“We knew that it needed to be significantly better than anywhere else, right at the launch,” Dutto said. While the late addition of an early August soccer game at Levi’s pushed deployment schedules ahead even faster, Dutto said in the end it helped DGP overall.

“I wasn’t a big fan of getting ready for that date [the Aug. 2 soccer game was Levi’s first event] but it was a blessing in disguise,” Dutto said. “The trouble with a network is that you can’t really test it until everyone shows up. We got some good data from that event.”

After the “daily and nightly” discussions with the Niners’ tech team about antenna placements and other matters, the DAS network performed well when it mattered, during the Niners’ two preseason games on Aug. 17 and 24. According to a traffic report from the stadium tech team, the DAS network carried a combined 1.02 terabytes of wireless traffic for the two preseason games, which is on par with activity seen at big events in the past, like Super Bowls. According to Dutto DGP’s internal tests showed that few, if any, calls were dropped or didn’t connect.

“We’re at 98 percent [network performance] already, without [the network] being fully optimized,” Dutto said.

RF challenges and too many iPhone 4 customers

While many of the stadium’s Wi-Fi antennas are well hidden — including the ones in boxes under seats — the DAS antennas are a bit more prominent, especially if you are near where the first overhang comes close to the seats.

“Unfortunately, with Levi’s exposed steel-beam construction, if you want the DAS to work, you’re going to see it,” Dutto said. “It’s hard to be stealthy in there.”

And in many cases there isn’t just one but instead two DGP antennas side by side, which reflects the company’s decision to actually build two parallel DAS systems to better accomodate more wireless carriers. According to Dutto, AT&T and T-Mobile are based on one system, while Verizon Wireless, Sprint and public safety communications are handled by the other one. Currently, all the carriers are live on the DAS except for Sprint, which is still in the process of installing its back-end equipment.

DAS antenna in "Faithful Mile" area

DAS antenna in “Faithful Mile” area

One of the biggest challenges for DGP, Dutto said, comes from outside the stadium, and not inside. Like other open-air stadiums located in city cores, Levi’s Stadium faces significant interference from cellular antennas on nearby office building rooftops, as well as from the Santa Clara Convention Center right across the street.

“Less than a half mile from Levi’s you can see seven different rooftop cell sites,” said Dutto, who said the flat, open terrain around the stadium increases the ability for those signals to interfere with the stadium DAS deployment. Target Field in Minneapolis had some similar problems with cell antennas on nearby office buildings.

“We’ve done a lot of work with the carriers to adjust their macro networks around the stadium,” Dutto said. “We’ll do more of that as we go, and expect it to get better.”

And while Levi’s Wi-Fi network has shown itself to be incredibly robust, Dutto said that cellular connectivity over the DAS might be even faster than Wi-Fi in many instances, especially if fans have later-model phones with 4G LTE.

Even though Dutto said DGP’s testing recorded download speeds of up to 200 Mbps — and 65 Mbps sustained — he acknowledged that many Levi’s patrons might never see those kinds of numbers unless they snap up some of the new iPhone 6 models introduced by Apple this week. According to network stats collected by DGP during the preseason games, a lot of fans may be ready for an upgrade.

“There’s a lot more iPhone 4 users out there than we thought,” Dutto said.

Stadium Tech Report: Aruba Networks chief engineer on building the ‘ultimate stadium’ network at Levi’s

Chuck Lukaszewski, Aruba Networks

Chuck Lukaszewski, Aruba Networks

For Chuck Lukaszewski, getting the job to design the Wi-Fi network at Levi’s Stadium was a bit of a personal perfect storm: Having long desired to build the ultimate stadium Wi-Fi network, he’d finally been matched with a customer with plans and a budget to fulfill that dream.

Lukaszewski, the senior director for outdoor solutions engineering at Levi’s Wi-Fi gear supplier Aruba Networks, said the aggressive vision for connected fan engagement put forth by the San Francisco 49ers and their vice president of technology Dan Williams was a plan Lukaszewski had long wanted to pursue.

“I had a vision of how to build an ‘ultimate stadium,’ and Dan said he wanted an ultimate stadium,” said Lukaszewski in a recent phone interview. “He gave me the freedom to design something that had never really been done before. It was a real meeting of the minds.”

By early results, the joint effort is by all standards a success, with network traffic statistics from the first football game at Levi’s showing a robust Wi-Fi network with Super Bowl levels of user traffic. According to Lukaszewski, by the time kickoff rolled around for the Niners’ preseason game against the Denver Broncos on Aug. 17, the Wi-Fi network at Levi’s Stadium had already surpassed the three most recent Super Bowls when it came to concurrent network usage, and had also exceeded the peak network usage statistics from the recent World Cup finale in Brazil.

While the network still hasn’t been tested in a regular-season game, or with the planned instant-replay feature active on the Levi’s Stadium app, it’s probably safe to say that at the very least, Lukaszewski and Williams have hit the ground running successfully. And while the network team did have some factors in their favor – including the ability to blend the wireless infrastructure into the overall stadium buildout and the budget for 1,200 Wi-Fi APs – there was still significant work that had to be done to produce a wireless experience with the magnitude of Levi’s offering.

Avoiding the ‘circular firing squad of RF’ in antenna deployment

According to Lukaszewski, he’s had experience deploying Wi-Fi networks in many unusual or difficult places, including on oil tankers and on financial-market trading floors. But stadiums, he said, especially open-air arenas like football stadiums, require a much higher level of deployment expertise, especially when you are trying to provide an extremely high level of connectivity to every seat in the stadium, as the Niners wanted to do at their new 68,500-seat home in Santa Clara, Calif.

“The 49ers have a completely unique view of the connectivity experience in a stadium – it’s the whole relationship of a fan with home,” said Lukaszewski. “Technology is just one piece of that. And [wireless] connectivity, is the last yard.”

Under-seat Wi-Fi AP at Levi's Stadium.

Under-seat Wi-Fi AP at Levi’s Stadium.

But just like that last three feet into the end zone, the last yard is often the toughest distance to cover when it comes to providing Wi-Fi connectivity. Unlike cellular operators, who own and control their licensed swath of spectrum, Wi-Fi networks operate in the unlicensed band of airwaves, and have an unknown number and type of end-user customers to support. According to Lukaszewski, designing and deploying a large-scale stadium Wi-Fi network is extremely more difficult than building a similar cellular network, mainly because of the interference situation.

“The classic design for a cellular DAS [distributed antenna system] in a stadium is to just put antennas high up, all the way around the bowl, pointing down,” said Lukaszewski. “That’s completely inefficient for Wi-Fi – we call it the ‘RF circular firing squad.’ Cellular owners can do that because they are exclusive owners of their spectrum and can bathe the bowl in signals. With Wi-Fi we have to ‘share the air’ in a ‘listen before talk’ environment.”

APs under seats, close to the fans

Without getting too deep into the specifics of Wi-Fi networking, what Lukaszewski, Williams and their teams needed to do was to figure out how many Wi-Fi access points (APs) they would need to provide high-quality connectivity to every seat in the house, and where to put them not just to provide the connection, but to also avoid interfering with signals from other close-by APs. While some of the planning can be done beforehand, Lukaszewski said that like in a restaurant kitchen, the final product often involves some hands-on refinement.

“You have a recipe, but you taste it and change it as you go,” Lukaszewski said, comparing the Levi’s deployment to a dinner dish. “We start out by going with the plan, then take a lot of data and make changes in real time.”

In the Niners’ favor was the fact that Levi’s Stadium was built from the ground up with the connectivity in mind, which made it much easier to deploy the network than in older stadiums where wiring and antennas need to be retrofitted into existing structures. Things like the many under-the-seat APs at Levi’s, with their wiring holes and mounting infrastructure designed into blueprints, might require concrete drilling and unsightly cable runs at stadiums built before networking was a pressing need.

“It’s always better when you can design it in,” said Lukaszewski of the wireless infrastructure. “The key is to have lots of low power cells, right among the people. That’s how we extract capacity. To get that you have to mount APs, and get the wiring to them. That can be very expensive in a retrofit. And it’s not always going to look very nice when you do it.”

More recipes for different events

Even with the impressive stats from the initial game, Lukaszewski was fast to note that the Levi’s Stadium wireless network is far from a finished product.

“We still have a lot of work to do – we’re definitely not done trying yet.” Lukaszewski said. “Every game, we want to get a little better.”

Typically in a stadium situation, Lukaszewski said, it may take 5 to 6 big events for the network team to really lock in on the tuning and tweaking necessary to make the network run at optimal performance. But even then, there’s still work to be done since the network requirements often differ greatly for different event types and crowds.

“There can be, more than one [network] recipe,” Luckszewski said. “The use case for a concert, for example, is way different than a football game. In an indoor stadium where you have both basketball and hockey, you may have 3 or 4 recipes, depending on the event.” Levi’s Stadium, he noted, is already scheduled to host several different types of events, including soccer games, high school football games, and an NHL game early next year.

But even as Lukaszewski, Williams and their teams continue to improve on their work, the early returns seem to suggest that the Niners have accomplished what they set out to build at their new home: A connected stadium that provides a fan experience unlike that found in most other large public venues.

“I think the 49ers have set an example for any team, in any sport,” Lukaszewski said.

Holy Terabyte! First football crowd at Levi’s Stadium uses 2.13 TB of Wi-Fi traffic, with nearly 25K fans on Wi-Fi at once

Levi's Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

Levi’s Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

All those predictions about Silicon Valley people using a stadium network more than other fans? It looks like they’re true.

The network numbers are in for the first football game at Levi’s Stadium, and they are pretty amazing: According to Dan Williams, the vice president of technology for the San Francisco 49ers, the Levi’s Stadium Wi-Fi network carried 2.13 terabytes of data during last Sunday’s preseason game, with a peak of 24,775 fans on the Wi-Fi network at the same time. Those numbers are comparable to the latest Super Bowl at MetLife Stadium in New Jersey, where according to AT&T and Verizon there was approximately 2.5 TB of data used on wireless networks.

The kicker to the Niners’ stats — they do NOT include any traffic figures from the Levi’s Stadium DAS, the distributed antenna system that is meant to provide enhanced cellular coverage in the stadium. What follows is an email Q-and-A with Williams, who kindly answered our extensive list of questions. The real question is, if Niners fans are hitting terabyte levels during preseason games, what’s going to happen when the season starts for real? And the instant replay function in the team app is live? Read on for some great insight from Williams. Additional editor’s note: The companies talked about include Aruba Networks, the provider of Wi-Fi gear; Brocade, provider of back-end networking gear and integration; DAS Group Professionals, the integrator and deployment team behind the DAS (the network of small antennas that improve in-building cellular coverage).

Mobile Sports Report: what was the peak number for simultaneous Wi-Fi connections? The average?
Dan Williams: We peaked at 24,775 (roughly 38% of attendance) concurrent connections with an average of 16,862 (roughly 25% of attendance).

MSR: When did connections spike? When did they start and then tail off?
Williams: We had two spikes, 1:02 p.m. [editor’s note: kickoff was 1 p.m.] with a system wide peak of 2.3Gbps and then again at 1:53 p.m. with 1.7Gbps. We averaged more than 1Gbps for more than two hours.

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

MSR: What was the total data tonnage on the Wi-Fi network?
Williams: We offloaded 2.13 Terabytes during the event.

MSR: What were the usage patterns with the app — which feature did people use most?
Williams: We had a great deal of usage throughout Sunday. The food ordering app usage was top of the list with ticketing being a close second while video would take third from a feature standpoint.

MSR: What are the plans with the instant replay feature… when will it be live (and can you explain why it was held back)?
Williams: We felt a lot of folks were happy with the livestream, so we wanted to focus more on a couple of core features with food ordering and ticketing a bit more at this point. Replays will be available to all by the first regular season home game.

MSR: Can you explain exactly how the location feature works… does it require Bluetooth to be on?
Williams: The location service is mainly built around low-energy Bluetooth, BLE. We have a number of beacons placed throughout the open areas and points-of-interest which allow the app to identify your location through proximity. Aruba helped us build this as well. GPS is also used but the primary resource is Bluetooth. The app prompts users to enable Bluetooth to provide improved location awareness.

One of the big screens in Levi's Stadium.

One of the big screens in Levi’s Stadium.

MSR: Can you provide any stats on the DAS performance?
Williams: The DAS held up really well. Like WiFi, we found some areas that need tuning. Unlike WiFi, the carriers protect a lot of their specific data but they have told us they are very happy with the system DGP helped us with. It is important to note our DAS and WiFi have been built to compliment each other and I think between Aruba and DGP, we did that very well. Most come here looking to connect to WiFi but our story internally has been we are going to have an awesome connectivity play regardless of medium.

MSR: Could you guys see any [more] of the 2.4 GHz issues like the one I had?
Williams: As you know, 2.4GHz is limited with non-overlapping channels so we suspect a number of legacy devices may have some problems. That said, we had a ratio of 2:1 with respects to 5GHz to 2.4GHz [usage] which shows a good deal of 2.4GHz usage. We know we still have some optimizations to do in the upper bowl and upper concourse while we continue to fine-tune the main bowl and concourse as we noticed our cell edge was weaker than expected when the stands were full. Our Aruba team did a great job capturing real-time data during the event as there is really no other way to test this stuff without a full venue. We will make some tweaks and continue to learn more from every event we host. Between Aruba, Brocade, and the 49er tech staff, we are not resting on our laurels. We know there is more to do.

Stadium Tech Report: Levi’s Stadium network lives up to hype, but team app still needs work

Levi's Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

Levi’s Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

At the very least San Francisco 49ers fans Sunday could take heart in the fact that the wireless network in Levi’s Stadium largely lived up to its advance billing, performing quite well even as the team on the field sputtered and failed to connect. In its first “real” test with an almost-full house on Sunday the Levi’s Wi-Fi and cellular networks seemed to work well throughout the game, delivering solid speed test results from almost every part of the new 68,500-seat facility, even as Colin Kaepernick and the rest of the 49ers were dealt a 34-0 preseason drubbing by Peyton Manning and the visiting Denver Broncos.

And just like the team, the Niners’ stadium technology lineup still has some weak spots that will hopefully be fixed before the regular season home opener on Sept. 14. Among the disappointments Sunday was a no-show by the highly heralded instant replay feature, the crown jewel of the new Levi’s Stadium app. We also experienced some location-connection problems with one of our devices, exposing what we consider a flaw in the Levi’s app, namely an over-reliance on location technologies to enable key parts of the app, like wayfinding and on-site video streaming.

Ticket scanner with Niners visor to block sun

Ticket scanner with Niners visor to block sun

Some other not-so-advanced technology flaws that could use fine-tuning include the volume level on the main stadium public-address and announcing system, which was so loud that it made it a struggle just to talk to the person next to you for long stretches of time. The ticket scanning machines also seemed to have issues working in the bright sunlight, a problem that found a low-tech fix when ticket personnel placed Niners’ visors around the tops of the machines to shade the scanning area. And many concession stands around the stadium were unable to serve guests or could only take cash because the staff operating the stands said they weren’t given access codes to the point-of-sale systems.

Overall, however, the first football game at Levi’s was a success on many levels, including the fantastic sight lines available from most seats and largely incident-free travel and parking operations, with noticed improvements especially on the VTA light rail front that struggled mightily during the stadium’s opening-event soccer game two weekends ago. Most fans also probably got a little weight loss from the no extra-charge sauna situation, thanks to the cloudless day and bright sun that bathed most of the seats in searing heat for long times after the 1 p.m. start.

Smooth start for early VTA riders

What follows here is a somewhat minute-by-minute account of my trip to the game, and my experience with the network and stadium operations on site.

Mtn View lot sign, not in operation at 9:30 a.m.

Mtn View lot sign, not in operation at 9:30 a.m.

Since I wasn’t given press access to the game, Mobile Sports Report attended like a regular fan, purchasing a single ticket through the NFL Ticket Exchange service on the 49ers’ web site. My plan to get to Levi’s from San Mateo was to drive to downtown Mountain View, park there and take VTA the rest of the way. (I didn’t take CalTrain mainly because I didn’t want to have to sync my return schedule with the CalTrain options going northbound on Sunday.)

Though I was somewhat incredulous about having to buy tickets online — VTA said that the ticket machines in Mountain View would be shut down Sunday to keep big lines from forming — upon further review the VTA app was slick and easy to operate and understand. After purchasing a ticket for $6.50 Saturday night I activated it Sunday, and showed it at the gate where they checked boarding passes. For people who didn’t have tickets there was a tent set up where they could buy a pre-loaded Clipper card for $10 good for a day’s worth of VTA riding. There was an abundance of VTA workers on hand, as well as a large and very obvious police presence. As a nice touch there was also a large bank of porta-potties, and behind the trains there were express buses waiting, according to one VTA employee, in case of crowd overloads.

“We learned some lessons from two weeks ago,” he said.

If there was a glitch in the VTA operations it was with the city of Mountain View — though a couple city lots were designated as places where fans could buy all-day parking passes, and there were clear signs to those lots, at 9:30 a.m. those lots were not yet staffed with anyone to pay; MSR found one sign leaning up against a post, waiting to be deployed. Fans could also park in the CalTrain lot for $5, payable via the CalTrain track podium ticket machines.

Fans transferring from CalTrain to VTA at Mtn View station

Fans transferring from CalTrain to VTA at Mtn View station

I boarded the first VTA train to leave for the stadium, along with many fans who had just gotten off CalTrain. The pleasant, air-conditioned trip took just 27 minutes, passing many Silicon Valley company headquarters and one neighborhood with “no parking here” patrols before stopping pretty much right at the Levi’s Stadium entrance. A few steps later I was in the parking lot, and took the first of many Wi-Fi speed tests and got a signal of 29 Mbps download and 23 Mbps upload, a good sign for network operations.

Looking for Wi-Fi, finding lots of it

DAS antenna in "Faithful Mile" area

DAS antenna in “Faithful Mile” area

Once inside the gates — and past the shaded scanners — I started speed testing in earnest, with the two devices I brought with me: A Motorola Droid 4 on Verizon, and a Samsung Galaxy Note 3 on AT&T. While waiting for the main stadium gates to open at 11 a.m. I got the weakest Wi-Fi signals of the day along the “Faithful Mile” area where promotional booths from sponsors kept early arrivers entertained. Wi-Fi on both devices out there only hit 2 to 3 Mbps on the download, while cell signals on both devices were in the 6-7 Mbps download range. Since I could see multiple DAS radios in the area but not any Wi-Fi access points I wasn’t too surprised; but it was an unusual area not to be blanketed with Wi-Fi, especially since there was good access a couple hundred yards away in the parking lots.

After finally entering the stadium proper, I ran into Niners president Paraag Marathe — who said he was “nervously excited,” and looked ready to start sweating in his suit and tie. “We’ve just got to make sure everything works today,” Marathe said, shaking my hand. Then I went up the escalator and saw the “Kezar pub,” an open-air bar filling the top area above the Intel gate. There, draft beers like Shock Top and Goose Island IPA were available for $11, and bottled beers available for $10.25.

As I started walking around the outside concourse I took my first speed test in the stadium and it blew the needle off the edge: 57.92 Mbps download, 41.00 Mbps upload.

A few minutes later on the inside concourse (where most of the concession stands are) I hit 27.85 Mbps/21.34 Mbps, still impressive. Then I tried to launch the app, and — problem. Apparently the device wasn’t connecting because it wouldn’t show my location on the wayfinding app. Luckily, right in front of me was Racquel, one of the “NiNerds,” the team’s new staff of technical experts who are there to help fans make the app work. But Racquel couldn’t solve my problem, even after we both tried turning on all location services, including Bluetooth.

Racquel the NiNerd

Racquel the NiNerd

“I can try to find another NiNerd who might know more about this device,” offered Racquel, who was visibly dismayed at her failure to help solve my problem. Instead, I moved on, hoping that the problem would solve itself later. But it didn’t.

Failure to locate… and other app problems

After downloading the Levi’s app to both devices over the weekend, I noticed that the first item on the app list of functions — Tickets — required me to “sign in” with my “Stadium Ticket Account,” something I didn’t have and didn’t know how to get. I did figure out how to enter my purchased seat location (which I could have used to order food to my seat, or for the express pick-up option), but I could never get the location feature to work on the Samsung device, which kept me from being able to see the live streaming TV option (I kept getting a message that said, “You must be at the stadium to play this video”). I was able to watch the live TV option on the Motorola device, after turing on location services. But for both devices — and, as it turns out, for everyone in the stadium — the final feature on the app, Game Center, where we were supposed to be able to see all the instant replays we could handle, remained labeled “coming soon.”

Finally sitting in my most excellent seat — section 244, row 3, seat 17 — I noticed that the Motorola device could no longer connect to Wi-Fi, even as the Samsung device was hitting marks in the 15-16 Mbps range. I started tweeting about the problem, and instead of a NiNerd coming to help me I got a personal visit from the Levi’s version of a Jedi Master, namely Dan Williams, the team’s vice president of technology. (Never underestimate the power of a complaining tweet!)

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

On my own, I had guessed that the Droid’s inability to connect came from its having only a 2.4 GHz Wi-Fi radio. If you’re not familiar with Wi-Fi networks, the 2.4 GHz band of unlicensed airwaves is what most first-generation Wi-Fi networks used; more recent devices are able to also use the 5 GHz band of unlicensed airwaves, which simply offer more channels and more bandwidth. The iPhone 5s, for example, mainly uses 5 GHz for Wi-Fi, as does my Samsung Note.

In scanning the available Wi-Fi networks, I had also noticed something else that I thought could be gumming up the Droid’s connection — a bunch of personal Wi-Fi hotspots in the immediate area, including several labeled as GoPro cameras. After curiously examining my Droid 4 — and its slide-out keyboard — Williams and a technician from Wi-Fi gear provider Aruba Networks concurred that my device was getting bogged down in the 2.4 GHz mess, and also wasn’t refreshing the available networks list, a device-specific problem.

Bottom line? Levi’s is no country for old phones.

Initial verdict: Wi-Fi and cellular is world class… but app needs work

After staying into the third quarter — and visiting another friend in section 109, where I got another hefty Wi-Fi speed test (24.42/25.39 Mbps) — I followed the lead of many fans and took an early leave of Levi’s, which meant no lines at the VTA trains and just a couple short delays due to track congestion that stretched the return train trip to 40 minutes. Overall, my travel to and from the stadium from San Mateo took just over an hour each way, a happy stat to report.

My initial verdict is that the Wi-Fi and DAS (cellular) networks delivered as promised, with solid speeds all around the stadium every time I checked. It’s no small accomplishment just to deliver such world-class service to such a crowded space, especially in the middle of Silicon Valley. True to its roots, the crowd Sunday was device-happy, with many iPads and GoPros being carried around as video cameras, in addition to all the phones that were in constant use. It’s a tribute to Williams and his staff, as well as the technology suppliers like Aruba, Brocade, Comcast (backbone bandwidth supplier) and DAS Group Professionals, who built the distributed antenna system (DAS) which brings advanced cellular connectivity inside the gates, to have built a solid network that worked well on its first big test.

The team app, however, did not even come close to living up to its advanced billing. To equal the network I think the app needs more advance instructions, especially on the ticketing/registration options as well as on the location services needed to make everything work. And until we see the multiple-camera angle live replays in action, to me the app is an incomplete project. The good news is, the Niners and their technology teams have several weeks to make improvements, including another preseason game Aug. 24 against the San Diego Chargers.

It’d also be helpful for the team to reach out a bit more to the VTA and players like the City of Mountain View, since the VTA site maps and Mountain View’s parking maps are far from what you would call “advanced design.” I think it’s up to the Niners to help pay for improvements to the city and transportation entities’ technology offerings, simply because of the burden placed on those operators by the fans going to Niners’ games. At the very least, more links from football to getting-there operations seems in order, instead of trusting that all parts of the operation will work in sync.

VTA lines going home

VTA lines going home