Instant replay feature debuts as network rocks at Niners’ home opener at Levi’s Stadium

Instant replay feature on Levi's Stadium app. Credit all photos: Paul Kapustka, MSR

Instant replay feature on Levi’s Stadium app. Credit all photos: Paul Kapustka, MSR

Niners fans may not have liked the final outcome on the scoreboard, but there was no complaining about the wireless network performance for the regular season home opener at Levi’s Stadium.

The somewhat limited debut of the highly anticipated instant replay feature for the Levi’s Stadium app was perhaps the technology highlight, with plays available for viewing seconds after they happened — when the feature finally got going later in the game. Our unofficial speedtests from various points around the stadium showed the Wi-Fi network and DAS network in top form, with solid results in the 20-plus Mbps range at most places for both networks.

We’re still waiting for the official postmortem results stats from the 49ers’ tech crew, but we did get some in-game messages from Niners VP of technology Dan Williams, who said that right before the 5:30 p.m. local time kickoff the usage peak was hit, with more than 19,000 simultaneous users on the Wi-Fi network, using throughput of 3.1 GB per second. Our unofficial guess is that the Niners’ opener may have set new single-game records for data traffic, but we’ll wait until we hear the final totals before we make any such proclamation.

The good news for Niners fans is that everywhere we went in the stadium, including on the top-level cheap-seat decks, the network was strong.

Replay feature sees limited action

If there was any tech downside, it was the limited availability of the instant replay feature. When talking about this feature earlier this year, team officials were adamant that it would be an unbelievable thing, with multiple camera angles and multiple choices of replays to watch. In reality, the feature wasn’t even available for most of the first half, and then when it did start working in the second half it only offered replays of the last two live plays, and two scoring highlights, one of which did not function at first.

Picture of app late in the first half.

Picture of app late in the first half.

Another beef we had with the replay feature was that it wasn’t clearly located on the “Game Center” part of the app; you had to know to click on the down arrow of the screen for the last play replay to appear. We found it only by accident, and would guess that not many fans found it or used it during the Niners’ loss to the Chicago Bears.

When we did watch the replays, however, we were simply amazed by the system’s performance — instant replays were available just seconds after a play had taken place, with a text play by play description as well. But again, there were no options for multiple camera angles, and there were only two “scoring play” highlights available when we last checked, and one (of an early field goal) didn’t have working video.

Other impressions from our stadium visit (which was made possible thanks to a press pass given to MSR by the Niners PR team):

— Concourses are fun places to watch: We watched the national anthem and opening kickoff and drive from the southwest concourse, one of the many places in Levi’s Stadium where you can stand and still see live game action. It was a packed house for the kickoff and first series, with many fans snapping phone pictures of the nearby field. East concourses were also good (though in one section an usher shooed us away from standing two-deep) as long as you kept the setting sun out of your eyes.

Fans take pictures of opening kickoff from southwest concourse.

Fans take pictures of opening kickoff from southwest concourse.

— Food runners were tired on the top deck: We stopped by for a quick chat with some of the food runners on the Pepsi deck on the north end of the stadium, and they looked pretty tired. Again, no official stats here but one runner said he’d been kept busy all game bringing food and drinks to fans who placed their orders via the app. “It’s pretty steep up here,” he said, pointing to the 400-level seating sections. “You get tired going up and down those steps.”

View from the Pepsi seating porch at the north end of Levi's Stadium

View from the Pepsi seating porch at the north end of Levi’s Stadium

— Clubs are the place to be: Several stories about the game noted that club-level seating was often empty, with fans perhaps spending time inside the comfortable beverage/food/gathering areas. Here’s a pregame look inside the United Club, located on the third level of the main building on the stadium’s west side.

United Club during pregame

United Club during pregame

— Signal strength: We also made several phone calls and did speedtests during halftime, when Snoop Dogg surprised us all with a mini-concert on the field. Not once during the game did we see any slowdowns in the network performance; the slowest Wi-Fi speed we found was 11 Mbps on the north concourse, where the team has admitted it didn’t put a lot of antennas (and is working to correct that).

Snoop on da big screen

Snoop on da big screen

— Light rail works fine: Then it was time to go home, and our choice to use the VTA light rail from Mountain View (instead of our press parking pass) was validated, as it took us less than an hour from the start of the line to the end of the line in Mountain View. All in all, an extremely solid home opener for one of the most ambitious stadium technology deployments out there.

Planning a DAS deployment? Check out these AT&T stadium stats from football opening weekend

There’s lots of shared belief out there that fans want to use mobile devices while they are attending sporting events. What we like even more here at Mobile Sports Report are hard numbers that tell us just how much fans are using mobile devices while they are in stadiums and arenas. Though sometimes hard to get, such statistics are great signposts for those who are planning to build their own stadium networks sometime soon, because it gives them a target to shoot for.

Courtesy of our friends over at AT&T, here are some traffic statistics gleaned from AT&T’s distributed antenna system (DAS) deployments in major professional and college football stadiums during last weekend’s games. According to AT&T, the average amount of cellular data used at a pro venue home opener in 2014 was 361 gigabytes, up 59 percent from season-opening games last year. The stats, remember, are only from AT&T DAS networks at the 16 NFL stadiums where AT&T has a DAS network presence.

The top stadium in terms of AT&T DAS traffic was Miami’s Sun Life Stadium with 1,035 GB (or 1.035 TB), followed by AT&T Stadium in Dallas with 889 GB and Atlanta’s Georgia Dome with 696 GB. It will be interesting to see whether or not any of those totals are surpassed by the San Francisco 49ers’ Levi’s Stadium, which hosts its first regular-season home game this Sunday.

On the collegiate side, the average opening-day traffic number was 288 GB, which is huge compared to the 2013 season average of 186 GB per game. The stats are an average taken from 25 Division 1 schools where AT&T has a DAS presence. And in case you were wondering where college football is king, the stats for schools in the South was an average of 343 GB, with the rest of the country checking in at 186 GB. The top three schools in terms of home-opener DAS traffic were Oklahoma with 866 GB, Georgia with 688, and the new stadium at Baylor with 686 GB.

More AT&T DAS stats follow. Remember, this is just a fraction of the actual traffic since many of these venues also have other carriers on DAS networks, as well as Wi-Fi networks in place. Other carriers and stadiums — send us your stats!

AT&T STADIUM DAS STATS

Year-over-Year Professional Mobile Data Increase

o 2014 home opener – 361GB average per venue

o 2013 season average – 227GB average per venue

· 2013 season average to 2014 season opener is a year-over-year increase of more than 59 percent

· 2013 season opener to 2014 season opener is a year-over-year increase of more than 58 percent

Opening Weekend Mobile Data Usage Professional vs. Collegiate

o 2014 professional season opener – 361GB average per venue

o 2014 collegiate season opener – 288GB average per venue

Top 5 Opening Weekend Professional Venues by Total Mobile Data

o Miami, FL – 1035GB

· Equivalent to more than 2.9 million social media post with photos

· Peak hour – 1-2pm ET – 146GB

o Dallas, TX – 889GB

· Equivalent to more than 2.5 million social media post with photos

· Peak hour – 2-3pm CT – 158GB

o Atlanta, GA – 696GB

· Equivalent to nearly 2 million social media post with photos

· Peak hour – 2-3pm ET – 159GB

o Chicago, IL – 452GB

· Equivalent to nearly 1.3 million social media post with photos

· Peak hour – 1-2pm CT – 107GB

o Pittsburgh, PA – 385GB

· Equivalent to more than 1.1 million social media post with photos

· Peak hour – 2-3pm ET – 83GB

Caveats:

· All figures include only data traffic seen on AT&T’s venue-specific mobile network.

· All data metrics come from only venues with a DAS where AT&T’s mobile network is on-air. These metrics are not comprehensive of every game played during the opening weekend for professional or college football.

· This data is compiled from 16 professional football stadiums and 25 division one college football stadiums that had opening week home games where AT&T is on-air on a DAS.

· All 2013 season average data is compiled from only venues with a DAS where AT&T’s mobile network was on-air and games where data was tracked and available.

ADDITIONAL AT&T COLLEGIATE DAS STATS

Regional Breakdown by Mobile Data Usage

o South – 343GB average per venue

o Rest of the US – 186GB average per venue

· Year-over-Year Mobile Data Increase

o 2014 home opener – 288GB average per venue

o 2013 season average – 155GB average per venue

· Top 5 Opening Weekend College Venues by Total Mobile Data

· Norman, OK – 866GB

o Equivalent to more than 2.4 million social media post with photos

· Athens, GA – 688GB

o Equivalent to more than 1.9 million social media post with photos

· Waco, TX – 686GB

o Equivalent to more than 1.9 million social media post with photos

· Auburn, AL – 506GB

o Equivalent to more than 1.4 million social media post with photos

· Los Angeles, CA – 469GB

o Equivalent to more than 1.3 million social media post with photos

Caveats:

· All figures include only data traffic seen on AT&T’s venue-specific mobile network.

· All data metrics come from only venues with a DAS where AT&T’s mobile network is on-air. These metrics are not comprehensive of every game played during the opening weekend for college football.

· This data is compiled from 25 division one college football stadiums that had opening week home games where AT&T is on-air on a DAS.

· All 2013 season average data is compiled from only venues with a DAS where AT&T’s mobile network was on-air and games where data was tracked and available.

· The “South” region is based off how it is qualified by the U.S. Census. It includes the following states: Delaware, District of Columbia, Florida, Georgia, North Carolina, South Carolina, Virginia, West Virginia, Alabama, Kentucky, Mississippi, Tennessee, Arkansas, Louisiana, Oklahoma and Texas.

Stadium Tech Report: DAS Group Professionals makes a name for itself with Levi’s Stadium DAS

DAS antennas hanging from a Levi's Stadium overhang. Credit, all Levi's photos: Paul Kapsutka, MSR

DAS antennas hanging from a Levi’s Stadium overhang. Credit, all Levi’s photos: Paul Kapsutka, MSR

If you look around at the walls, ceilings and overhangs at Levi’s Stadium, it’s hard to miss the small square boxes with the off-white color and a “DGP” logo in one corner. While the wires hanging out the back of each box make it an easy guess that the equipment has something to do with wireless networks, even many industry insiders may not know the company behind the boxes and the three-letter acronym.

Meet DAS Group Professionals, the Bay Area firm in charge of deploying a distributed antenna system (DAS) to make sure your cell phone gets a good signal at San Francisco 49ers games and any other event inside the 68,500-seat Levi’s Stadium. And while you might not be familiar with DGP, rest assured the company is extremely familiar with cellular deployments for large public venues, having installed similar DAS networks for airports, casinos and hotels, and even for San Francisco’s BART train system. Of course, most of that work was done when the company was called Forza Telecom, before changing its name to DGP earlier this year, another reason why “DGP” may not have rung any bells.

“It may appear like we just fell out of the sky, but we’ve actually built quite a few [DAS] systems,” said Steve Dutto, president of DGP, in a recent phone interview. “We’ve got years of experience.”

DAS antennas above a food stand

DAS antennas above a food stand

DAS: The Rodney Dangerfield of stadium connectivity

One thing that keeps firms like DGP in the shadows is the relative obscurity of DAS itself. While most people generally understand how cell phones work — you turn on your phone, and it connects to an antenna somewhere on a tower or rooftop — in crowded public facilities like stadiums, traditional cellular networks with towers several miles apart can’t handle the concentrated capacity. To provide connectivity for areas with large crowds, the latest tactic is to deploy a DAS, a network of lots of smaller antennas. Originally deployed in places like office buildings, hotels and convention centers, DAS is rapidly gaining favor in stadiums and arenas, helping to alleviate the “no signal” problem that has cropped up in many venues the past few years.

And while stadium Wi-Fi gets lots of headlines whenever it gets deployed — probably thanks again to the widespread understanding of how Wi-Fi works — there are already far more DAS deployments in stadiums than Wi-Fi, mainly because cellular carriers will pay almost all the associated costs of a DAS buildout to make sure their customers get a signal. According to our most recent 2014 State of the Stadium survey, 71.4 percent of our respondents said they had a full DAS at their facility, while only 35.7 percent had fan-facing Wi-Fi.

Steve Dutto, president, DGP

Steve Dutto, president, DGP

How does a DAS work? Usually, either a major cellular carrier or a third-party “neutral” host like DGP will build out the antenna infrastructure, which includes many small antennas and then cables to bring the connections back to a wiring room or data center. There, cellular carriers install their own cell-tower back-end gear to authenticate customers and to provide a connection to the company networks and the Internet. Since it’s in the cellular carriers’ interest to keep their customers connected (and using billable minutes and data), carriers will often pay the full cost of a DAS infrastructure by building and running it themselves. In the case like Levi’s, where DGP is the “neutral host,” DGP builds the infrastructure and then charges cellular carriers to use it.

Such deals are rarely publicized, and Dutto would not comment on how much each carrier was being charged to use the Levi’s DAS — though industry gossip has the figure somewhere around $5 million per carrier per year. And just like fight club, for many deployments the first rule of DAS is that you don’t talk about DAS, because no cellular carrier ever wants to admit that its network might need help. So just like Rodney Dangerfield, DAS often doesn’t get a lot of public respect. But at Levi’s Stadium and many other sports and entertainment venues, DAS is a booming business that would be sorely missed if it wasn’t there.

The ‘dream and the nightmare’ of building the Levi’s DAS

Now that DGP’s 700-plus antenna DAS deployment is up and running at Levi’s, Dutto can breathe a small bit easier. While the network is good business for the company and an obvious prominent calling card for the future, the aggressive deployment timeframe probably isn’t something Dutto is eager to repeat.

“Levi’s Stadium was both a dream and a nightmare [for DGP],” Dutto said, due in part both to the Niners’ aggressive performance expectations and the rapid buildout schedule. Of course, DGP was somewhat used to working quickly with the Niners — when the company put a DAS in Candlestick Park back in 2012 to solve that stadium’s legendary lack of connectivity, Dutto said it was deployed “in about 90 days.”

The success of the Candlestick deployment, Dutto said, led to the Niners offering the Levi’s DAS gig to DGP. With it, however, came the need to match the team’s out-front statements about how the stadium was going to be the best ever in terms of wireless connectivity. And with many people not knowing or not bothering to switch their phones to Wi-Fi, the Levi’s DAS, like most stadium DAS deployments, would probably handle most of the wireless connections.

Door sign for head end equipment room at Levi's Stadium (there are many of these)

Door sign for head end equipment room at Levi’s Stadium (there are many of these)

“We knew that it needed to be significantly better than anywhere else, right at the launch,” Dutto said. While the late addition of an early August soccer game at Levi’s pushed deployment schedules ahead even faster, Dutto said in the end it helped DGP overall.

“I wasn’t a big fan of getting ready for that date [the Aug. 2 soccer game was Levi’s first event] but it was a blessing in disguise,” Dutto said. “The trouble with a network is that you can’t really test it until everyone shows up. We got some good data from that event.”

After the “daily and nightly” discussions with the Niners’ tech team about antenna placements and other matters, the DAS network performed well when it mattered, during the Niners’ two preseason games on Aug. 17 and 24. According to a traffic report from the stadium tech team, the DAS network carried a combined 1.02 terabytes of wireless traffic for the two preseason games, which is on par with activity seen at big events in the past, like Super Bowls. According to Dutto DGP’s internal tests showed that few, if any, calls were dropped or didn’t connect.

“We’re at 98 percent [network performance] already, without [the network] being fully optimized,” Dutto said.

RF challenges and too many iPhone 4 customers

While many of the stadium’s Wi-Fi antennas are well hidden — including the ones in boxes under seats — the DAS antennas are a bit more prominent, especially if you are near where the first overhang comes close to the seats.

“Unfortunately, with Levi’s exposed steel-beam construction, if you want the DAS to work, you’re going to see it,” Dutto said. “It’s hard to be stealthy in there.”

And in many cases there isn’t just one but instead two DGP antennas side by side, which reflects the company’s decision to actually build two parallel DAS systems to better accomodate more wireless carriers. According to Dutto, AT&T and T-Mobile are based on one system, while Verizon Wireless, Sprint and public safety communications are handled by the other one. Currently, all the carriers are live on the DAS except for Sprint, which is still in the process of installing its back-end equipment.

DAS antenna in "Faithful Mile" area

DAS antenna in “Faithful Mile” area

One of the biggest challenges for DGP, Dutto said, comes from outside the stadium, and not inside. Like other open-air stadiums located in city cores, Levi’s Stadium faces significant interference from cellular antennas on nearby office building rooftops, as well as from the Santa Clara Convention Center right across the street.

“Less than a half mile from Levi’s you can see seven different rooftop cell sites,” said Dutto, who said the flat, open terrain around the stadium increases the ability for those signals to interfere with the stadium DAS deployment. Target Field in Minneapolis had some similar problems with cell antennas on nearby office buildings.

“We’ve done a lot of work with the carriers to adjust their macro networks around the stadium,” Dutto said. “We’ll do more of that as we go, and expect it to get better.”

And while Levi’s Wi-Fi network has shown itself to be incredibly robust, Dutto said that cellular connectivity over the DAS might be even faster than Wi-Fi in many instances, especially if fans have later-model phones with 4G LTE.

Even though Dutto said DGP’s testing recorded download speeds of up to 200 Mbps — and 65 Mbps sustained — he acknowledged that many Levi’s patrons might never see those kinds of numbers unless they snap up some of the new iPhone 6 models introduced by Apple this week. According to network stats collected by DGP during the preseason games, a lot of fans may be ready for an upgrade.

“There’s a lot more iPhone 4 users out there than we thought,” Dutto said.

Niners: More than 1,000 fans used in-seat food delivery at 2nd Levi’s Stadium preseason game

Screen grab from Levi's Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

Screen grab from Levi’s Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

One of the more unique features the San Francisco 49ers are introducing at their new home this year is the ability for all fans to have food and drink delivered to them, no matter which one of Levi’s Stadium’s 68,500 seats they are in. And according to the Niners, the feature is quickly catching on, with more than 1,000 in-seat orders delivered at the second preseason game at Levi’s on Aug. 24.

At a Levi’s media technology tour Wednesday, the Niners’ tech staff provided an in-depth and up-close look at some of the new stadium’s network infrastructure, including a quick glimpse of one of the several data-center rooms. Dan Williams, vice president of technology for the Niners, said that at the first preseason game at Levi’s on Aug. 17, more fans used the “express pickup” service to order food that they could then pick up at concession stand windows than the seat-delivery feature. But at the second game against the San Diego Chargers, more fans went for the in-seat option, perhaps a sign that Niners fans are learning and testing the new stadium services as they go.

“It’s going to be an ongoing education process,” said Martin Manville, business operations analyst for the Niners and one of the key tech leaders on the Levi’s app team. Manville said the Niners had learned a lot about food delivery in test situations at Candlestick Park last season — and some of those lessons are now evident in the Levi’s delivery menu, which is stripped down to ensure the food runners can get grub to fans before it gets cold (or warm, in the case of cold beverages). According to Manville the average delivery time at the Aug. 24 game was between 10 and 12 minutes, but the team expects that “normal” delays during the regular season will be closer to 15 to 20 minutes per order.

Still, that’s not a bad option if you don’t want to leave your seat and you don’t mind the extra $5 delivery charge. One early interesting stat from the food-delivery process is that the Niners found more orders coming from the “cheap seats” in the north and south stands at Levi’s than from the 50-yard-line seats where the high rollers sit. According to Manville, since the fans in the club seats have easy access to numerous uncrowded concession stands they may not see the need for the in-seat options.

Wi-Fi APs: 600 in Levi’s bowl seating

Under-seat Wi-Fi AP at Levi's Stadium.

Under-seat Wi-Fi AP at Levi’s Stadium.

Some other news nuggets from the tech tour day: According to Williams, of the 1,200 Wi-Fi access points at Levi’s, 600 of those are distributed in the seating areas (aka “the bowl”), with the other 600 placed in suites, concourses and other stadium areas. Williams said the Aruba Networks Wi-Fi antennas are basically split into three types — regular enterprise-type APs for suite and office areas, regular outdoor APs for concourse areas, and more specialized versions (including the under-seat APs) for bowl placements.

— For the app itself, the Niners said that there have been 80,000 downloads so far, with almost half of the season ticket holders having put their ticketing information into the app. The “NiNerds,” the geek-dressed help squad that provides fans with personal assistance with the app, is now at about 50 or 60 strong at each event (originally the team had said it wanted to hire 150 such Wi-Fi coaches). The Niners said the NiNerds will be doing more pro-active app education going forward, doing things like approaching fans in concession lines to see if they know about the express line or in-seat ordering options.

— Though Comcast’s 10-year deal with the Niners calls for the cable provider to bring in two 10-gigabit backbone pipes, the Niners are often quoted as saying they have 40 GB of backbone bandwidth. We solved this mystery today: According to Comcast, the other two 10-GB pipes are a redundant channel from another (unnamed) provider. So: the stadium does have four 10 GB bandwidth pipes, by far the most capacity in any stadium we’ve heard of.

— More traffic stats: Though we will break these down in a separate post, the Niners said that for the Aug. 24 game fans used 1.96 Terabytes of Wi-Fi traffic, just a bit lower than the 2.13 TB used at the first preseason game on Aug. 17. The team also provided some DAS stats, claiming fans used another 1.02 TB of cellular data at the two preseason games combined.

Tech tour photos follow… including a sighting of some (shhhh!) Cisco equipment in the data center racks… click on photos for larger images.

Niners VP of technology Dan Williams (left) and COO Al Guido kick off the Wednesday tech tour.

Niners VP of technology Dan Williams (left) and COO Al Guido kick off the Wednesday tech tour.

Ted Girdner, Comcast VP of business services for California, talks stadium networking.

Ted Girdner, Comcast VP of business services for California, talks stadium networking.

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him.

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him.

Mystery Cisco gear inside Levi's Stadium data center. Alert! Intruder!

Mystery Cisco gear inside Levi’s Stadium data center. Alert! Intruder!

Brocade router at Levi's Stadium data center. One of many. As in, many many.

Brocade router at Levi’s Stadium data center. One of many. As in, many many.

Wi-Fi gear in Levi's Stadium data room.

Wi-Fi gear in Levi’s Stadium data room.

Franks and DAS: DGP DAS antennas above food station.

Franks and DAS: DGP DAS antennas above food station.

Screenshot of food feature on Levi's Stadium app. Note the green light buttons to show expected wait times for express option.

Screenshot of food feature on Levi’s Stadium app. Note the green light buttons to show expected wait times for express option.

Obligatory Levi's Stadium selfie. MSR shirts complete the style.

Obligatory Levi’s Stadium selfie. MSR shirts complete the style.

Stadium Tech Report: Aruba Networks chief engineer on building the ‘ultimate stadium’ network at Levi’s

Chuck Lukaszewski, Aruba Networks

Chuck Lukaszewski, Aruba Networks

For Chuck Lukaszewski, getting the job to design the Wi-Fi network at Levi’s Stadium was a bit of a personal perfect storm: Having long desired to build the ultimate stadium Wi-Fi network, he’d finally been matched with a customer with plans and a budget to fulfill that dream.

Lukaszewski, the senior director for outdoor solutions engineering at Levi’s Wi-Fi gear supplier Aruba Networks, said the aggressive vision for connected fan engagement put forth by the San Francisco 49ers and their vice president of technology Dan Williams was a plan Lukaszewski had long wanted to pursue.

“I had a vision of how to build an ‘ultimate stadium,’ and Dan said he wanted an ultimate stadium,” said Lukaszewski in a recent phone interview. “He gave me the freedom to design something that had never really been done before. It was a real meeting of the minds.”

By early results, the joint effort is by all standards a success, with network traffic statistics from the first football game at Levi’s showing a robust Wi-Fi network with Super Bowl levels of user traffic. According to Lukaszewski, by the time kickoff rolled around for the Niners’ preseason game against the Denver Broncos on Aug. 17, the Wi-Fi network at Levi’s Stadium had already surpassed the three most recent Super Bowls when it came to concurrent network usage, and had also exceeded the peak network usage statistics from the recent World Cup finale in Brazil.

While the network still hasn’t been tested in a regular-season game, or with the planned instant-replay feature active on the Levi’s Stadium app, it’s probably safe to say that at the very least, Lukaszewski and Williams have hit the ground running successfully. And while the network team did have some factors in their favor – including the ability to blend the wireless infrastructure into the overall stadium buildout and the budget for 1,200 Wi-Fi APs – there was still significant work that had to be done to produce a wireless experience with the magnitude of Levi’s offering.

Avoiding the ‘circular firing squad of RF’ in antenna deployment

According to Lukaszewski, he’s had experience deploying Wi-Fi networks in many unusual or difficult places, including on oil tankers and on financial-market trading floors. But stadiums, he said, especially open-air arenas like football stadiums, require a much higher level of deployment expertise, especially when you are trying to provide an extremely high level of connectivity to every seat in the stadium, as the Niners wanted to do at their new 68,500-seat home in Santa Clara, Calif.

“The 49ers have a completely unique view of the connectivity experience in a stadium – it’s the whole relationship of a fan with home,” said Lukaszewski. “Technology is just one piece of that. And [wireless] connectivity, is the last yard.”

Under-seat Wi-Fi AP at Levi's Stadium.

Under-seat Wi-Fi AP at Levi’s Stadium.

But just like that last three feet into the end zone, the last yard is often the toughest distance to cover when it comes to providing Wi-Fi connectivity. Unlike cellular operators, who own and control their licensed swath of spectrum, Wi-Fi networks operate in the unlicensed band of airwaves, and have an unknown number and type of end-user customers to support. According to Lukaszewski, designing and deploying a large-scale stadium Wi-Fi network is extremely more difficult than building a similar cellular network, mainly because of the interference situation.

“The classic design for a cellular DAS [distributed antenna system] in a stadium is to just put antennas high up, all the way around the bowl, pointing down,” said Lukaszewski. “That’s completely inefficient for Wi-Fi – we call it the ‘RF circular firing squad.’ Cellular owners can do that because they are exclusive owners of their spectrum and can bathe the bowl in signals. With Wi-Fi we have to ‘share the air’ in a ‘listen before talk’ environment.”

APs under seats, close to the fans

Without getting too deep into the specifics of Wi-Fi networking, what Lukaszewski, Williams and their teams needed to do was to figure out how many Wi-Fi access points (APs) they would need to provide high-quality connectivity to every seat in the house, and where to put them not just to provide the connection, but to also avoid interfering with signals from other close-by APs. While some of the planning can be done beforehand, Lukaszewski said that like in a restaurant kitchen, the final product often involves some hands-on refinement.

“You have a recipe, but you taste it and change it as you go,” Lukaszewski said, comparing the Levi’s deployment to a dinner dish. “We start out by going with the plan, then take a lot of data and make changes in real time.”

In the Niners’ favor was the fact that Levi’s Stadium was built from the ground up with the connectivity in mind, which made it much easier to deploy the network than in older stadiums where wiring and antennas need to be retrofitted into existing structures. Things like the many under-the-seat APs at Levi’s, with their wiring holes and mounting infrastructure designed into blueprints, might require concrete drilling and unsightly cable runs at stadiums built before networking was a pressing need.

“It’s always better when you can design it in,” said Lukaszewski of the wireless infrastructure. “The key is to have lots of low power cells, right among the people. That’s how we extract capacity. To get that you have to mount APs, and get the wiring to them. That can be very expensive in a retrofit. And it’s not always going to look very nice when you do it.”

More recipes for different events

Even with the impressive stats from the initial game, Lukaszewski was fast to note that the Levi’s Stadium wireless network is far from a finished product.

“We still have a lot of work to do – we’re definitely not done trying yet.” Lukaszewski said. “Every game, we want to get a little better.”

Typically in a stadium situation, Lukaszewski said, it may take 5 to 6 big events for the network team to really lock in on the tuning and tweaking necessary to make the network run at optimal performance. But even then, there’s still work to be done since the network requirements often differ greatly for different event types and crowds.

“There can be, more than one [network] recipe,” Luckszewski said. “The use case for a concert, for example, is way different than a football game. In an indoor stadium where you have both basketball and hockey, you may have 3 or 4 recipes, depending on the event.” Levi’s Stadium, he noted, is already scheduled to host several different types of events, including soccer games, high school football games, and an NHL game early next year.

But even as Lukaszewski, Williams and their teams continue to improve on their work, the early returns seem to suggest that the Niners have accomplished what they set out to build at their new home: A connected stadium that provides a fan experience unlike that found in most other large public venues.

“I think the 49ers have set an example for any team, in any sport,” Lukaszewski said.

Holy Terabyte! First football crowd at Levi’s Stadium uses 2.13 TB of Wi-Fi traffic, with nearly 25K fans on Wi-Fi at once

Levi's Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

Levi’s Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

All those predictions about Silicon Valley people using a stadium network more than other fans? It looks like they’re true.

The network numbers are in for the first football game at Levi’s Stadium, and they are pretty amazing: According to Dan Williams, the vice president of technology for the San Francisco 49ers, the Levi’s Stadium Wi-Fi network carried 2.13 terabytes of data during last Sunday’s preseason game, with a peak of 24,775 fans on the Wi-Fi network at the same time. Those numbers are comparable to the latest Super Bowl at MetLife Stadium in New Jersey, where according to AT&T and Verizon there was approximately 2.5 TB of data used on wireless networks.

The kicker to the Niners’ stats — they do NOT include any traffic figures from the Levi’s Stadium DAS, the distributed antenna system that is meant to provide enhanced cellular coverage in the stadium. What follows is an email Q-and-A with Williams, who kindly answered our extensive list of questions. The real question is, if Niners fans are hitting terabyte levels during preseason games, what’s going to happen when the season starts for real? And the instant replay function in the team app is live? Read on for some great insight from Williams. Additional editor’s note: The companies talked about include Aruba Networks, the provider of Wi-Fi gear; Brocade, provider of back-end networking gear and integration; DAS Group Professionals, the integrator and deployment team behind the DAS (the network of small antennas that improve in-building cellular coverage).

Mobile Sports Report: what was the peak number for simultaneous Wi-Fi connections? The average?
Dan Williams: We peaked at 24,775 (roughly 38% of attendance) concurrent connections with an average of 16,862 (roughly 25% of attendance).

MSR: When did connections spike? When did they start and then tail off?
Williams: We had two spikes, 1:02 p.m. [editor’s note: kickoff was 1 p.m.] with a system wide peak of 2.3Gbps and then again at 1:53 p.m. with 1.7Gbps. We averaged more than 1Gbps for more than two hours.

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

MSR: What was the total data tonnage on the Wi-Fi network?
Williams: We offloaded 2.13 Terabytes during the event.

MSR: What were the usage patterns with the app — which feature did people use most?
Williams: We had a great deal of usage throughout Sunday. The food ordering app usage was top of the list with ticketing being a close second while video would take third from a feature standpoint.

MSR: What are the plans with the instant replay feature… when will it be live (and can you explain why it was held back)?
Williams: We felt a lot of folks were happy with the livestream, so we wanted to focus more on a couple of core features with food ordering and ticketing a bit more at this point. Replays will be available to all by the first regular season home game.

MSR: Can you explain exactly how the location feature works… does it require Bluetooth to be on?
Williams: The location service is mainly built around low-energy Bluetooth, BLE. We have a number of beacons placed throughout the open areas and points-of-interest which allow the app to identify your location through proximity. Aruba helped us build this as well. GPS is also used but the primary resource is Bluetooth. The app prompts users to enable Bluetooth to provide improved location awareness.

One of the big screens in Levi's Stadium.

One of the big screens in Levi’s Stadium.

MSR: Can you provide any stats on the DAS performance?
Williams: The DAS held up really well. Like WiFi, we found some areas that need tuning. Unlike WiFi, the carriers protect a lot of their specific data but they have told us they are very happy with the system DGP helped us with. It is important to note our DAS and WiFi have been built to compliment each other and I think between Aruba and DGP, we did that very well. Most come here looking to connect to WiFi but our story internally has been we are going to have an awesome connectivity play regardless of medium.

MSR: Could you guys see any [more] of the 2.4 GHz issues like the one I had?
Williams: As you know, 2.4GHz is limited with non-overlapping channels so we suspect a number of legacy devices may have some problems. That said, we had a ratio of 2:1 with respects to 5GHz to 2.4GHz [usage] which shows a good deal of 2.4GHz usage. We know we still have some optimizations to do in the upper bowl and upper concourse while we continue to fine-tune the main bowl and concourse as we noticed our cell edge was weaker than expected when the stands were full. Our Aruba team did a great job capturing real-time data during the event as there is really no other way to test this stuff without a full venue. We will make some tweaks and continue to learn more from every event we host. Between Aruba, Brocade, and the 49er tech staff, we are not resting on our laurels. We know there is more to do.