Entry issues snarl start of WrestleMania 32 at AT&T Stadium; crowd uses 6.77 TB of Wi-Fi

While Sunday’s WrestleMania 32 eventually ended up with a record attendance for the WWE’s signature event with 101,763 fans in attendance, some snafus with the entry process had fans reporting as long as three-hour waits to get in, with some pegging a Wi-Fi outage that disabled ticket scanners as one of the roots of the problem.

We don’t as of yet have any on-the-record comments from AT&T Stadium so we can’t say exactly what the problem was. UPDATE, 4/4/16, 2:50 p.m.: According to John Winborn, chief information officer for the Dallas Cowboys Football Club (and the person who runs the network inside AT&T Stadium), despite the rumors that emerged there was no Wi-Fi outage at the stadium — in fact Winborn said the crowd used 6.77 TB of Wi-Fi data Sunday, a single-day stadium Wi-Fi total second only to the 10.1 TB used at Levi’s Stadium for Super Bowl 50. (More details on the Wi-Fi consumption coming soon.)

Wi-Fi rumors aside, on Sunday night Twitter and Instagram were full of photos from irate fans who were stuck outside the venue, with some reportedly unable to get inside or to their seats before the actual matches started. According to a blog post at the Fort Worth Star-Telegram, the WWE and the Stadium issued a joint press release halfway through the event that read:

“To ensure the safety of WWE fans, increased security measures were put in place tonight. We apologize that it may have taken some fans longer than usual to get into AT&T Stadium.”

Several other news outlets reported the problem, with all seeming to peg the issue on a Wi-Fi problem with the ticket scanners. Winborn, however, said there were no Wi-Fi outages in the stadium or with the ticket scanners.

According to the Fort Worth Star-Telegram report, the vitriol on Twitter and other social media outlets may not have been entirely correct; according to the report:

Do not believe the viral reports and unsightly online pictures that fans could not get to their seats by the time the early matches began — they could but they preferred to walk around, buy food, drink and merchandise. But the images and the complaints that flooded the Internet about #WrestleMania32 were so plentiful, and negative, joint press release was issued halfway through the event.

With 101,763 finally in their seats the event was the fifth-largest crowd inside AT&T Stadium, and as such most likely produced a large Wi-Fi and DAS traffic number once things finally got underway. We will update this post and have another when we get the wireless traffic stats from the stadium.

AT&T: Fans used 2.28 TB of cellular data during Sweet 16 and Elite 8 games

Screen Shot 2016-03-31 at 11.45.34 AMAs the march toward the Final Four continued, fans at the Sweet 16 and Elite 8 venues for the men’s NCAA basketball tournament used a combined 2.28 terabytes of data on the AT&T networks in those venues, according to AT&T. The highest weekend total came from games at the Wells Fargo Center in Philadelphia, where 680 GB were used, according to AT&T.

Combined with the 3.6 TB of data AT&T said was used on its networks at first- and second-round sites, that makes a total of 5.88 TB used so far on AT&T cellular and DAS networks at the various hoops arenas. We’d like to hear from other carriers as well, but none have contacted us so far.

Data totals at this weekend’s Final Four in Houston should be interesting, since the host venue, NRG Stadium, doesn’t yet have a Wi-Fi network. Both AT&T and Verizon have beefed up cellular coverage in and around the arena, but without Wi-Fi it may be hard for fans to top last year’s total of 11 TB used at the Final Four at Lucas Oil Stadium in Indianapolis.

Remember — all Final Four games this weekend are on TBS, not CBS! Of course you can also stream the games if that’s easier.

Betting the Under (Part 2): Putting Wi-Fi antennas under seats is the hot new trend in stadium wireless networks

Under-seat Wi-Fi AP at Levi's Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Under-seat Wi-Fi AP at Levi’s Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Part 2 of this story picks up with the decision to put Wi-Fi APs under seats at Levi’s Stadium. If you missed it, here is the link to Part 1.

According to Chuck Lukaszewski, now vice president of wireless strategy and standards at Hewlett Packard Enterprise (formerly very high density architect in the CTO Office of Aruba Networks), Aruba had been testing under-seat AP designs since around 2010, “in one form or another.” There were some initial tests of under-seat AP deployments at Turner Field in Atlanta and at American Airlines Arena in Dallas, but nothing on the scale of AT&T Park’s 2013 deployment, or on the scale Aruba planned to have at Levi’s Stadium when it opened in 2014.

Some of the first under-seat Wi-Fi deployments in other arenas were actually deployed completely under the stands, Lukaszewski said, with signals shooting up through the concrete. Though he said “you could get reasonably good throughput through concrete,” especially for 2.4 GHz frequencies, installing antennas above the concrete was “considerably better,” Lukaszewski said.

Curiously, one of the biggest problems in stadium Wi-Fi deployment — especially for those heavy on overhead antenna use — is negotiating interference between antennas; sometimes, clients can “see” antennas and APs that are across the stadium, and will try to connect to those instead of the AP closest to them, a problem that leads to inefficient bandwidth use. Interference also means you can’t place APs too closely together, making it somewhat of an art to find ways to increase coverage without increasing interference.

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi's Stadium. Photo: Paul Kapustka, MSR

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi’s Stadium. Photo: Paul Kapustka, MSR

What Aruba found in its testing, Lukaszewski said, was that under-seat Wi-Fi AP deployments could be far more dense than overhead-centric designs, mainly because the human bodies in the seats would provide beneficial “blocking” of signals, allowing network designers to place APs more closely together, and to be able to re-use the same Wi-Fi channels in more antennas.

“If you can use human bodies to contain signals, you can have much smaller cells,” Lukaszewski said. Under-seat deployments, he said, “allows us to re-use the same channel less than 100 yards away.”

With more channels available for each AP, the difference in the metric Lukaszewski calls “megabytes per fan” can be “profound” for an under-seat design versus an overhead design, he said.

“We do see trends [in stadium network data] of under-seat being able to deliver well over 100 MB per fan per event, while overhead designs [deliver] significantly under 100 MB per fan per event,” said Lukaszewski.

Dan Williams, the former vice president of technology for the San Francisco 49ers, said he and Lukaszewski were in agreement that under-seat was the best method to deploy at Levi’s Stadium.

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

“I just did not believe in overhead,” said Williams, who said he brainstormed with Aruba’s Lukaszewski on the under-seat idea, which they both brought to the Wi-Fi design at Levi’s. By using under-seat APs, Williams said, the Levi’s Stadium design looked to provide “cones [of bandwidth] around the audience, immersing [fans] in a signal.”

After beating the previous year’s Super Bowl Wi-Fi total at its NFL regular-season opener in 2014, Levi’s Stadium’s Wi-Fi network more than passed its biggest test ever this year, carrying a record 10.1 terabytes of Wi-Fi data during Super Bowl 50. Those numbers are proof of Lukaszewski’s claim: “By far, under seat is better.”

New deployments trending to under-seat

Editor’s note: This excerpt is from our latest STADIUM TECH REPORT, our long-form PDF publication that combines in-depth stadium tech reports with news and analysis of the hottest topics in the world of stadium and large public venue tech deployments. Enjoy this PART 1 of our lead feature, or DOWNLOAD THE REPORT and read the whole story right now!

Even though under-seat deployments can be considerably more expensive, especially in a retrofit situation where deployment requires coring through concrete, many stadiums are now seeming to agree with another Lukaszewski claim, that “the return absolutely justifies the investment.”

At AT&T Stadium in Arlington, Texas, the Cowboys quicked followed their sister park’s lead and installed under-seat APs in force ahead of that venue’s hosting of the inaugural College Football Playoff championship game in January of 2015. John Winborn, chief information officer for the Dallas Cowboys Football Club, said the team worked with AT&T’s “Foundry” innovation centers to produce a smaller, sleeker under-seat AP enclosure that fit well with the stadium’s commitment to aesthetics.

Back on the baseball side, the Giants now have 1,628 Wi-Fi APs in their park, with the vast majority of them under-seat, in all three decks of seating. And the Giants’ main rival to the south, the Los Angeles Dodgers, also used under-seat APs in a recent Wi-Fi upgrade.

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

And if Levi’s Stadium led the way for under-seat Wi-Fi, the new mainly under-seat network at the refurbished Kyle Field at Texas A&M might be the QED on the debate, with ultra-fast network speeds and big data-consumption numbers (including 5.7 TB of Wi-Fi at a game versus Alabama) adding measureable momentum to the under-seat trend. Bill Anderson, CEO of Wi-Fi deployment strategy firm AmpThink, said he was an early disbeliever in under-seat Wi-Fi — until he saw the numbers.

“At first we mocked it, made fun of it,” said Anderson, whose firm has been called in to produce Wi-Fi network designs for several recent Super Bowls, as well as for the Kyle Field design. But when Aruba showed AmpThink the data from under-seat tests and deployments, “that was the ‘a-ha’ moment for us,” Anderson said.

Working with Aruba at Kyle Field, AmpThink was able to collect its own data, which convinced Anderson that under-seat was the way to go if you wanted dense, high-performing networks.

“The really important thing is to get APs closer to the people,” said Anderson. “That’s the future.”

Anderson said some doubters may remain, especially those who try to mix a small amount of under-seat APs with existing overhead deployments, a recipe for lowered success due to the potential interference issues. At Texas A&M, Anderson said AmpThink was able to build a design with far less interference and much greater density than an overhead solution, producing numbers that people have to pay attention to.

“We only know what we’ve observed, but we’re evangelistic supporters” of under-seat designs, Anderson said. “If someone says to you under-seat is hocus-pocus, they’re not looking at the data.”

Not for everyone, but more are trying under-seat

Though proponents of under-seat Wi-Fi all agree on its ability to deliver denser, faster networks, they all also agree that under-seat can be considerably more costly than overhead Wi-Fi, especially in a retrofit situation.

In addition to having to core through concrete seating areas to get conduit to the under-seat APs, the devices themselves need to be sealed, to guard them from weather, drink spills, and the power-washing equipment employed by most stadiums to clean seating areas.

Aruba’s Lukaszewski also noted that under-seat deployments generally use more linear feet of cabling to connect the APs than overhead, which also drives up the cost. Then since under-seat designs tend to use more APs, that also means a higher budget to cover a higher number of devices.

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

For some stadiums, the construction materials used prohibit the under-seat option from even being tried. At the Green Bay Packers’ legendary Lambeau Field, a late-1950s construction design that used lots of concrete and rebar — as well as part of the stadium’s bottom sitting directly in the ground — meant that under-seat Wi-Fi wasn’t an option, according to Wayne Wichlacz, director of information technology for the Packers.

Other stadiums, like the University of Nebraska’s Memorial Stadium, don’t have enough space between the stadium’s bleacher seats and the floor for under-seat APs to be safely installed. And many schools or teams simply don’t have big IT budgets like the $20-million-plus available to Texas A&M that allowed the Kyle Field design to seek the best result possible.

But many of the new stadiums under construction, as well as existing venues that are planning for new best-of-breed networks, have already committed to under-seat Wi-Fi designs, including the Sacramento Kings’ Golden 1 Center, where Ruckus Wireless will implement its first under-seat stadium Wi-Fi network.

Steve Martin, senior vice president and general manager at Ruckus, said the Golden 1 Center design, planned to be the most dense anywhere, will “primarily be underseat,” a choice he said “helps in a lot of ways.”

Foremost is the performance, something Martin said Ruckus has been testing at the Kings’ current home, the Sleep Train Arena. “It [under seat] does give you the isolation for frequency re-use,” he said.

The under-seat design also makes sense in Golden 1 Center since the stadium’s overall design is very open, with lots of glass walls and unobstructed views.

And under-seat deployment is even making inroads into the distributed antenna system (DAS) world, with Verizon Wireless implementing more than 50 under-seat DAS antennas at Levi’s Stadium prior to Super Bowl 50. Mainly installed to cover the bottom-of-the-bowl rows, the under-seat APs helped Verizon manage a record day for DAS traffic, with 7 TB reported on its in-stadium cellular network during the game.

“To get a quality signal, we had to go under seat,” said Brian Mecum, vice president, network, for Verizon Wireless, who said that in that area of the stadium, under seat was the only way to get a quality signal close to the subscriber’s phone. Verizon, he said, helped design the under-seat DAS antenna, and is looking to deploy it in other stadiums soon.

“It’s the first of more,” he said.

END PART 2… HERE IS THE LINK TO PART 1… TO READ THE WHOLE STORY NOW, DOWNLOAD OUR REPORT!

S.F. Giants add more Wi-Fi, ‘virtual reality experience’ to AT&T Park for 2016 season

The view from AT&T Park's left field corner. All photos: Paul Kapustka, MSR (click on any photo for a larger image)

The view from AT&T Park’s left field corner. All photos: Paul Kapustka, MSR (click on any photo for a larger image)

The first ballpark to bring Wi-Fi to its fans is still padding its networking lead, as AT&T Park will have 543 new or upgraded Wi-Fi access points for the 2016 season, according to the San Francisco Giants.

Most of the new APs are of the under-seat variety, completing the team’s three-year plan to put more APs under seats to increase network density and capacity. According to Bill Schlough, senior vice president and chief information officer for the Giants, the park now has a total of 1,628 Wi-Fi APs, the most of any MLB stadium and more than most big football stadiums as well. With 78.2 terabytes of data used during the baseball season and another 20+ TB used during other events, Schlough said AT&T Park’s Wi-Fi network carried more than 100 TB of data in calendar 2015.

Since it’s an even year, the Giants expect to win the World Series again, so the action on the field should be pretty good. If you want to leave reality, however, the Giants can accomodate you in that realm this season with the addition of a “virtual reality experience” at the team’s @Cafe social media spot, located on the concourse behind the left-field bleachers.

Since it's an even year, does that mean another one of these is on order for the Giants?

Since it’s an even year, does that mean another one of these is on order for the Giants?

According to the Giants, fans can be “transported” to Scottsdale Stadium to view practice from spring training, or they can see views from the AT&T Park field, the batting cages and “even Sergio Romo’s car” through a VR headset.

The Giants said fans will also notice an upgrade to the stadium’s LED ribbon boards, which circle the park on the facings of the upper decks. The new Mitsubishi screens, the Giants said, offer 150 percent more pixels than their predecessors, meaning that you might not need those reading glasses to get stat updates or read advertising messsages.

On the DAS side of things, AT&T Park finally has all four major U.S. wireless carriers on its in-house cellular network, with the DAS and Wi-Fi serviced by 13 1-Gbps backbone pipes from AT&T.

Betting the Under: Putting Wi-Fi antennas under seats is the hot new trend in stadium wireless networks

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

What do you typically find under stadium seats on game days? The traditional list might include bags and purses, and get-out-of-the-way items like empty popcorn tubs, used hot dog wrappers and drink cups no longer filled with fluids.

And now you can add Wi-Fi access points and DAS antennas to the list.

A growing trend is emerging to use under-seat antenna placements to bring wireless signals closer to fans, for both Wi-Fi networks as well as cellular distributed antenna system (DAS) deployments. First used to compensate for a lack of overhang or railing placement spaces, under-seat deployments are now winning favor in all sorts of arenas for their ability to use human bodies to help build a more dense network, one that proponents say can carry far more capacity than an infrastructure that relies mainly on overhead antenna placements.

With proof points emerging quickly at venues like the San Francisco 49ers’ Levi’s Stadium and Texas A&M’s Kyle Field, as well as at pioneers AT&T Park and AT&T Stadium, under-seat Wi-Fi deployments may soon become more common, as more integrators and equipment suppliers embrace the under-seat method.

New stadiums under construction including the Sacramento Kings’ Golden 1 Center as well as new Wi-Fi deployments at existing stadiums like Houston’s NRG Stadium and the Bank of America Stadium in Charlotte are also planning to primarily use under-seat Wi-Fi deployments, both for the performance and aesthetic benefits. With such high-profile deployments embracing the method, under-seat APs may become the default placement position going forward, especially as stadium mobile-device usage by fans keeps growing.

History: a need required by architecture

Editor’s note: This excerpt is from our latest STADIUM TECH REPORT, our long-form PDF publication that combines in-depth stadium tech reports with news and analysis of the hottest topics in the world of stadium and large public venue tech deployments. Enjoy this PART 1 of our lead feature, or DOWNLOAD THE REPORT and read the whole story right now!

Giants senior VP and CIO Bill Schlough, at the office

Giants senior VP and CIO Bill Schlough, at the office

When Wi-Fi first arrived in stadiums, the obvious solution to questions about antenna and access point placement seemed evident — just mount them on ceilings, overhangs and walls, like they had always been placed historically. Mostly that decision kept the antennas out of sight, and provided good-enough reception for most network deployments.

But as fan Wi-Fi usage started growing, poor reception areas cropped up, most often in the most expensive seats near the courts or playing fields, where there was often little architectural infrastructure other than the seats themselves. At the San Francisco Giants’ AT&T Park, the stadium where fan-facing Wi-Fi was first installed in 2004, the need for more bandwidth was a big problem that needed to be solved during the 2012-13 offseason, after the team’s second World Series title run in two years had produced record wireless usage.

Even with a Wi-Fi AP placed just about everywhere they could be, the San Francisco Giants’ IT team couldn’t keep up with demand. And the trick that has been tried at some stadiums — putting AP enclosures on handrails — wasn’t an option at AT&T Park, since its lower-bowl seating areas have no railings.

With options limited, that’s when an internal battle commenced around the new idea of placing APs under seats, a plan that met fierce resistance on many fronts.

“We got beaten up pretty bad over the idea [of under-seat APs],” said Bill Schlough, senior vice president and chief information officer for the Giants, who described the 2012-13 offseason as “a very stressful time,” with lots of internal strife and discord. With multiple stakeholders checking in on the plan, including the Giants’ facilities group, the marketing group and the ticketing group, concerns about the loss of under-seat space and the potential health concerns fueled opposition to putting APs under chairs.

But without any railings or overhangs for most of the park’s lower-bowl seats, Schlough and his team had “no other alternative” than to try placing Wi-Fi APs under seats. On the possible health issue, Schlough said the Giants were assured by technology partner (and ballpark title sponsor) AT&T that the deployment would be safe and comply with all FCC regulations; “We were assured that having [an antenna] 18 inches from your butt was the [radio] equivalent of having a cell phone in your pocket,” Schlough said.

On the storage-space concern side, Schlough said the Giants’ IT team made models of the antennas out of cardboard and duct tape, and placed them under seats to see how they worked.

“The [walking] flow through the aisles was good, with the AP models tucked under we never kicked them” during testing, Schlough said. With AT&T assuring the Giants that under-seat was “the way of the future,” the team took a leap of faith and added a large number of under-seat Wi-Fi APs in preparation for the 2013 season, more than doubling the number of APs in the park (to 760 total) in the process.

Under-seat Wi-Fi enclosure at Dodgers Stadium. Photo: Terry Sweeney, MSR

Under-seat Wi-Fi enclosure at Dodgers Stadium. Photo: Terry Sweeney, MSR

Though Schlough and his team “spent a lot of time” communicating with season-ticket holders about the new technology, there was still consternation about what might happen when opening day arrived, and “fans find this box under their seat, and not have a place to put their garlic fries,” Schlough said.

As it turns out, there was almost no resistance to the method; according to Schlough the Giants only had two complaints about the under-seat APs that first day of deployment, which Schlough called “the biggest relief day of my life.”

The success of the under-seat idea was particularly noted at that time by another IT team in the Bay area, the one putting together the wireless plan for the San Francisco 49ers’ new home, Levi’s Stadium, which was being built just to the south in Santa Clara. Testing some under-seat placements of their own at Candlestick Park during that venue’s final season as the Niners’ home, the team building the Levi’s Stadium network became convinced that going under seat was the best way to build the high-density deployment they wanted to have.

END PART 1… HERE IS THE LINK TO PART 2… TO READ THE WHOLE STORY NOW, DOWNLOAD OUR REPORT!

AT&T: NCAA Men’s hoops sites used 3.6 TB of cellular data

Screen shot 2016-03-21 at 10.45.42 PMWhile your bracket was busy getting busted, AT&T said it saw more than 3.6 terabytes of cellular data cross the networks at the eight stadiums hosting the first rounds of the men’s NCAA basketball tournament last weekend.

In Providence, R.I., at the Dunkin’ Donuts Center, AT&T said it saw 926 GB of cellular data cross its networks there during the first round games, a total perhaps helped by the Yale-Duke game in the second round. Stay tuned for more data updates from NCAA regionals as well as from the Final Four. And any other carriers that want to send their numbers along, please do so!