Mobilitie, AT&T beef up DAS coverage at Churchill Downs ahead of Kentucky Derby

White DAS antennas visible on the overhangs at Churchill Downs. Photo: Mobilitie (click on any photo for a larger image).

White DAS antennas visible on the overhangs at Churchill Downs. Photo: Mobilitie (click on any photo for a larger image).

Since it’s legal in horse racing, how about a bet? Who thinks the final AT&T DAS traffic total from the Kentucky Derby weekend will pass 10 terabytes this year? After hitting 5.1 TB during last year’s two days of racing, what are the odds on wireless data used there doubling down again? Anyone want to bet against growth?

For sure, operators of the wireless infrastructure at the track aren’t going to left behind for lack of trying — according to neutral-host DAS provider Mobilitie and DAS design partner AT&T, there will be approximately a 50 percent increase in AT&T’s cellular capacity at Churchill Downs this year, with double coverage in the 1900 MHz band compared to last year, according to AT&T.

Mobilitie president Christos Karmis said in a phone interview this week that “every year, the amount of data used has doubled,” at least since Mobilitie put in the first part of the DAS at the track for the 2013 event. Since then, Karmis said the DAS antenna count has grown from 253 to 290, and the network now has 55 sectors, in and around the track and seating areas as well as in the parking lots. In addition to AT&T, the DAS also carries traffic for Verizon Wireless and T-Mobile, neither of which have reported traffic totals from the event.

Biggest DAS in sports?

Where all the Kentucky Derby race-day social media posts go to find the Internet (aka the head end at the Churchill Downs DAS). Photo: Mobilitie

Where all the Kentucky Derby race-day social media posts go to find the Internet (aka the head end at the Churchill Downs DAS). Photo: Mobilitie

With race day drawing 170,513 fans last year, the Kentucky Derby may well be one of the most challenging events to provide wireless coverage to, with multiple spikes in traffic due to two full days of racing (there are 13 races scheduled for Friday May 6, the Kentucky Oaks Day, and 14 scheduled for Saturday May 7, the Kentucky Derby Day).

According to Karmis, the fans are mainly shooting video and taking pictures during the races, and then posting those images to social media networks during the breaks in between. And then there’s the betting, which can be done on premises via the twinspires app, which drives additional traffic. And then there’s the hats and fashion, which are worth lots of pictures themselves, especially if you see any celebrities during the red-carpet parade.

Though there is still a small amount of Wi-Fi in the main track buildings — at least there was a couple years ago when we did a profile on the new networks — the DAS is the workhorse at the venue, which spreads out far and wide and includes an infield area full of fans as well.

“It’s crazy to try to keep up” with the data demands, Karmis said.

With any luck on our side, we’ll be able to get Verizon and T-Mobile to send over their DAS stats after the event to see if the weekend of races can match the DAS total of 15.9 TB from Super Bowl 50. If the AT&T data alone has gone from 2 TB two years ago to 5.1 TB last year, what’s the total going to be for this year? Put your predictions in the comments below!

AT&T records 29.2 TB of cellular data during Coachella weekends

AT&T's new "drum" antennas at Coachella. Photos: AT&T.

AT&T’s new “drum” antennas at Coachella. Photos: AT&T.

For AT&T, the Coachella Valley Music and Arts Festival isn’t about sounds but about bandwidth: For the past few years, the cellular giant has tested some giant antennas to help connect the 75,000 concert fans who invade the venue in Indio, Calif., for two weekends in April, and this year those fans used up a total of 29.2 terabytes of data, according to AT&T, a new record for the provider at Coachella.

Last year AT&T rolled out something it called the cheese wheel antennas, basically big spheres with the tops and bottoms cut off, and a lot of antenna gear inside. With back-to-back 12 TB weekends, the cheese wheels as well as some drive-up cell towers on wheels (COWS) helped keep music fans connected at the concerts. This year, AT&T added to its antenna arsenal a beefier version of the cheese wheel it called the drum set, which helped record 18.6 TB of data the three days of the first weekend, and another 10.6 this past weekend, for a event total of 29.2 TB. For comparison, Super Bowl 50 back in February racked up a total of 26 TB of wireless traffic, with 10.1 TB on Wi-Fi and 15.9 TB on DAS.

Coachella’s AT&T data is all cellular, no Wi-Fi. In 2014, AT&T brought out its “big ball” antenna at Coachella, where AT&T has seen data usage grow by 20 times since 2011, through last year’s show. With this year’s total of 29.2 TB eclipsing last year’s 24 TB, there still seems to be no end in sight to the mobile-data usage at big events.

UPDATE: WrestleMania 32 sets new Wi-Fi mark at AT&T Stadium; total Wi-Fi + DAS hits 8.6 TB

The Undertaker arrives at AT&T Stadium for WrestleMania 32. Photo: WWE.com

The Undertaker arrives at AT&T Stadium for WrestleMania 32. Photo: WWE.com


UPDATE: Fixes an MSR calculation error on DAS figures.

The 101,763 fans who filled AT&T Stadium Sunday for WrestleMania 32 set new stadium records for Wi-Fi, according to figures provided by AT&T Stadium and AT&T, with 6.77 terabytes of Wi-Fi traffic and an additional 1.9 TB on the AT&T network on the stadium’s DAS for a total wireless figure of 8.6 TB.

The Wi-Fi numbers put Sunday’s signature WWE event (also the biggest WrestleMania by attendance) into second place in our unofficial record-keeping of the largest single-day Wi-Fi traffic stadium events. Only Super Bowl 50 at Levi’s Stadium earlier this year was bigger, with 10.1 TB of Wi-Fi traffic. So far, WrestleMania 32 is also now third in combined Wi-Fi and DAS figures, trailing Super Bowl 50 and Super Bowl XLIX (see charts below).

THE NEW TOP 3 TOTAL USAGE

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB; DAS: 15.9 TB; Total: 26 TB
2. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB; DAS: 6.56 TB**; Total: 12.79 TB**
3. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB; DAS: 1.9 TB*; Total: 8.6 TB*

* = AT&T DAS stats only
** = AT&T, Verizon Wireless and Sprint DAS stats only

THE NEW TOP 5 FOR WI-FI

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB
2. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB
3. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB
4. Alabama vs. Texas A&M, Kyle Field, College Station, Texas, Oct. 17, 2015: Wi-Fi: 5.7 TB
5. College Football Playoff championship game, AT&T Stadium, Arlington, Texas, Jan. 12, 2015: Wi-Fi: 4.93 TB

Wi-Fi not to blame for stadium entry issues

John Winborn, chief information officer for the Dallas Cowboys Football Club, said in an email that the reported claims of the Wi-Fi being offline Sunday — and that being the reason why entry lines were long and slow — were not true. While Winborn did admit that one single Wi-Fi AP (out of the more than 2,000 in the stadium’s network) was offline and there were “a couple issues” with ticket scanners, he said “there were no Wi-Fi issues that would have had a significant impact on ingress.” Other reports have claimed the doors were opened later due to extended show rehearsals, while commenters on MSR’s posts have claimed that a lack of wristbands for stadium-floor seating also led to seating issues even for fans already inside the main building entrances. So far, we have not seen any official explanation for the delays other than the official apology from the stadium and the WWE:

“To ensure the safety of WWE fans, increased security measures were put in place tonight. We apologize that it may have taken some fans longer than usual to get into AT&T Stadium.”

During Sunday’s event Winborn said the Wi-Fi network saw 20,462 concurrent and 34,951 total user connections, some via a network of 150 temporary Wi-Fi APs installed among the seats on the stadium floor.

Entry issues snarl start of WrestleMania 32 at AT&T Stadium; crowd uses 6.77 TB of Wi-Fi

While Sunday’s WrestleMania 32 eventually ended up with a record attendance for the WWE’s signature event with 101,763 fans in attendance, some snafus with the entry process had fans reporting as long as three-hour waits to get in, with some pegging a Wi-Fi outage that disabled ticket scanners as one of the roots of the problem.

We don’t as of yet have any on-the-record comments from AT&T Stadium so we can’t say exactly what the problem was. UPDATE, 4/4/16, 2:50 p.m.: According to John Winborn, chief information officer for the Dallas Cowboys Football Club (and the person who runs the network inside AT&T Stadium), despite the rumors that emerged there was no Wi-Fi outage at the stadium — in fact Winborn said the crowd used 6.77 TB of Wi-Fi data Sunday, a single-day stadium Wi-Fi total second only to the 10.1 TB used at Levi’s Stadium for Super Bowl 50. (More details on the Wi-Fi consumption coming soon.)

Wi-Fi rumors aside, on Sunday night Twitter and Instagram were full of photos from irate fans who were stuck outside the venue, with some reportedly unable to get inside or to their seats before the actual matches started. According to a blog post at the Fort Worth Star-Telegram, the WWE and the Stadium issued a joint press release halfway through the event that read:

“To ensure the safety of WWE fans, increased security measures were put in place tonight. We apologize that it may have taken some fans longer than usual to get into AT&T Stadium.”

Several other news outlets reported the problem, with all seeming to peg the issue on a Wi-Fi problem with the ticket scanners. Winborn, however, said there were no Wi-Fi outages in the stadium or with the ticket scanners.

According to the Fort Worth Star-Telegram report, the vitriol on Twitter and other social media outlets may not have been entirely correct; according to the report:

Do not believe the viral reports and unsightly online pictures that fans could not get to their seats by the time the early matches began — they could but they preferred to walk around, buy food, drink and merchandise. But the images and the complaints that flooded the Internet about #WrestleMania32 were so plentiful, and negative, joint press release was issued halfway through the event.

With 101,763 finally in their seats the event was the fifth-largest crowd inside AT&T Stadium, and as such most likely produced a large Wi-Fi and DAS traffic number once things finally got underway. We will update this post and have another when we get the wireless traffic stats from the stadium.

AT&T: Fans used 2.28 TB of cellular data during Sweet 16 and Elite 8 games

Screen Shot 2016-03-31 at 11.45.34 AMAs the march toward the Final Four continued, fans at the Sweet 16 and Elite 8 venues for the men’s NCAA basketball tournament used a combined 2.28 terabytes of data on the AT&T networks in those venues, according to AT&T. The highest weekend total came from games at the Wells Fargo Center in Philadelphia, where 680 GB were used, according to AT&T.

Combined with the 3.6 TB of data AT&T said was used on its networks at first- and second-round sites, that makes a total of 5.88 TB used so far on AT&T cellular and DAS networks at the various hoops arenas. We’d like to hear from other carriers as well, but none have contacted us so far.

Data totals at this weekend’s Final Four in Houston should be interesting, since the host venue, NRG Stadium, doesn’t yet have a Wi-Fi network. Both AT&T and Verizon have beefed up cellular coverage in and around the arena, but without Wi-Fi it may be hard for fans to top last year’s total of 11 TB used at the Final Four at Lucas Oil Stadium in Indianapolis.

Remember — all Final Four games this weekend are on TBS, not CBS! Of course you can also stream the games if that’s easier.

Betting the Under (Part 2): Putting Wi-Fi antennas under seats is the hot new trend in stadium wireless networks

Under-seat Wi-Fi AP at Levi's Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Under-seat Wi-Fi AP at Levi’s Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Part 2 of this story picks up with the decision to put Wi-Fi APs under seats at Levi’s Stadium. If you missed it, here is the link to Part 1.

According to Chuck Lukaszewski, now vice president of wireless strategy and standards at Hewlett Packard Enterprise (formerly very high density architect in the CTO Office of Aruba Networks), Aruba had been testing under-seat AP designs since around 2010, “in one form or another.” There were some initial tests of under-seat AP deployments at Turner Field in Atlanta and at American Airlines Arena in Dallas, but nothing on the scale of AT&T Park’s 2013 deployment, or on the scale Aruba planned to have at Levi’s Stadium when it opened in 2014.

Some of the first under-seat Wi-Fi deployments in other arenas were actually deployed completely under the stands, Lukaszewski said, with signals shooting up through the concrete. Though he said “you could get reasonably good throughput through concrete,” especially for 2.4 GHz frequencies, installing antennas above the concrete was “considerably better,” Lukaszewski said.

Curiously, one of the biggest problems in stadium Wi-Fi deployment — especially for those heavy on overhead antenna use — is negotiating interference between antennas; sometimes, clients can “see” antennas and APs that are across the stadium, and will try to connect to those instead of the AP closest to them, a problem that leads to inefficient bandwidth use. Interference also means you can’t place APs too closely together, making it somewhat of an art to find ways to increase coverage without increasing interference.

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi's Stadium. Photo: Paul Kapustka, MSR

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi’s Stadium. Photo: Paul Kapustka, MSR

What Aruba found in its testing, Lukaszewski said, was that under-seat Wi-Fi AP deployments could be far more dense than overhead-centric designs, mainly because the human bodies in the seats would provide beneficial “blocking” of signals, allowing network designers to place APs more closely together, and to be able to re-use the same Wi-Fi channels in more antennas.

“If you can use human bodies to contain signals, you can have much smaller cells,” Lukaszewski said. Under-seat deployments, he said, “allows us to re-use the same channel less than 100 yards away.”

With more channels available for each AP, the difference in the metric Lukaszewski calls “megabytes per fan” can be “profound” for an under-seat design versus an overhead design, he said.

“We do see trends [in stadium network data] of under-seat being able to deliver well over 100 MB per fan per event, while overhead designs [deliver] significantly under 100 MB per fan per event,” said Lukaszewski.

Dan Williams, the former vice president of technology for the San Francisco 49ers, said he and Lukaszewski were in agreement that under-seat was the best method to deploy at Levi’s Stadium.

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

“I just did not believe in overhead,” said Williams, who said he brainstormed with Aruba’s Lukaszewski on the under-seat idea, which they both brought to the Wi-Fi design at Levi’s. By using under-seat APs, Williams said, the Levi’s Stadium design looked to provide “cones [of bandwidth] around the audience, immersing [fans] in a signal.”

After beating the previous year’s Super Bowl Wi-Fi total at its NFL regular-season opener in 2014, Levi’s Stadium’s Wi-Fi network more than passed its biggest test ever this year, carrying a record 10.1 terabytes of Wi-Fi data during Super Bowl 50. Those numbers are proof of Lukaszewski’s claim: “By far, under seat is better.”

New deployments trending to under-seat

Editor’s note: This excerpt is from our latest STADIUM TECH REPORT, our long-form PDF publication that combines in-depth stadium tech reports with news and analysis of the hottest topics in the world of stadium and large public venue tech deployments. Enjoy this PART 1 of our lead feature, or DOWNLOAD THE REPORT and read the whole story right now!

Even though under-seat deployments can be considerably more expensive, especially in a retrofit situation where deployment requires coring through concrete, many stadiums are now seeming to agree with another Lukaszewski claim, that “the return absolutely justifies the investment.”

At AT&T Stadium in Arlington, Texas, the Cowboys quicked followed their sister park’s lead and installed under-seat APs in force ahead of that venue’s hosting of the inaugural College Football Playoff championship game in January of 2015. John Winborn, chief information officer for the Dallas Cowboys Football Club, said the team worked with AT&T’s “Foundry” innovation centers to produce a smaller, sleeker under-seat AP enclosure that fit well with the stadium’s commitment to aesthetics.

Back on the baseball side, the Giants now have 1,628 Wi-Fi APs in their park, with the vast majority of them under-seat, in all three decks of seating. And the Giants’ main rival to the south, the Los Angeles Dodgers, also used under-seat APs in a recent Wi-Fi upgrade.

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

And if Levi’s Stadium led the way for under-seat Wi-Fi, the new mainly under-seat network at the refurbished Kyle Field at Texas A&M might be the QED on the debate, with ultra-fast network speeds and big data-consumption numbers (including 5.7 TB of Wi-Fi at a game versus Alabama) adding measureable momentum to the under-seat trend. Bill Anderson, CEO of Wi-Fi deployment strategy firm AmpThink, said he was an early disbeliever in under-seat Wi-Fi — until he saw the numbers.

“At first we mocked it, made fun of it,” said Anderson, whose firm has been called in to produce Wi-Fi network designs for several recent Super Bowls, as well as for the Kyle Field design. But when Aruba showed AmpThink the data from under-seat tests and deployments, “that was the ‘a-ha’ moment for us,” Anderson said.

Working with Aruba at Kyle Field, AmpThink was able to collect its own data, which convinced Anderson that under-seat was the way to go if you wanted dense, high-performing networks.

“The really important thing is to get APs closer to the people,” said Anderson. “That’s the future.”

Anderson said some doubters may remain, especially those who try to mix a small amount of under-seat APs with existing overhead deployments, a recipe for lowered success due to the potential interference issues. At Texas A&M, Anderson said AmpThink was able to build a design with far less interference and much greater density than an overhead solution, producing numbers that people have to pay attention to.

“We only know what we’ve observed, but we’re evangelistic supporters” of under-seat designs, Anderson said. “If someone says to you under-seat is hocus-pocus, they’re not looking at the data.”

Not for everyone, but more are trying under-seat

Though proponents of under-seat Wi-Fi all agree on its ability to deliver denser, faster networks, they all also agree that under-seat can be considerably more costly than overhead Wi-Fi, especially in a retrofit situation.

In addition to having to core through concrete seating areas to get conduit to the under-seat APs, the devices themselves need to be sealed, to guard them from weather, drink spills, and the power-washing equipment employed by most stadiums to clean seating areas.

Aruba’s Lukaszewski also noted that under-seat deployments generally use more linear feet of cabling to connect the APs than overhead, which also drives up the cost. Then since under-seat designs tend to use more APs, that also means a higher budget to cover a higher number of devices.

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

For some stadiums, the construction materials used prohibit the under-seat option from even being tried. At the Green Bay Packers’ legendary Lambeau Field, a late-1950s construction design that used lots of concrete and rebar — as well as part of the stadium’s bottom sitting directly in the ground — meant that under-seat Wi-Fi wasn’t an option, according to Wayne Wichlacz, director of information technology for the Packers.

Other stadiums, like the University of Nebraska’s Memorial Stadium, don’t have enough space between the stadium’s bleacher seats and the floor for under-seat APs to be safely installed. And many schools or teams simply don’t have big IT budgets like the $20-million-plus available to Texas A&M that allowed the Kyle Field design to seek the best result possible.

But many of the new stadiums under construction, as well as existing venues that are planning for new best-of-breed networks, have already committed to under-seat Wi-Fi designs, including the Sacramento Kings’ Golden 1 Center, where Ruckus Wireless will implement its first under-seat stadium Wi-Fi network.

Steve Martin, senior vice president and general manager at Ruckus, said the Golden 1 Center design, planned to be the most dense anywhere, will “primarily be underseat,” a choice he said “helps in a lot of ways.”

Foremost is the performance, something Martin said Ruckus has been testing at the Kings’ current home, the Sleep Train Arena. “It [under seat] does give you the isolation for frequency re-use,” he said.

The under-seat design also makes sense in Golden 1 Center since the stadium’s overall design is very open, with lots of glass walls and unobstructed views.

And under-seat deployment is even making inroads into the distributed antenna system (DAS) world, with Verizon Wireless implementing more than 50 under-seat DAS antennas at Levi’s Stadium prior to Super Bowl 50. Mainly installed to cover the bottom-of-the-bowl rows, the under-seat APs helped Verizon manage a record day for DAS traffic, with 7 TB reported on its in-stadium cellular network during the game.

“To get a quality signal, we had to go under seat,” said Brian Mecum, vice president, network, for Verizon Wireless, who said that in that area of the stadium, under seat was the only way to get a quality signal close to the subscriber’s phone. Verizon, he said, helped design the under-seat DAS antenna, and is looking to deploy it in other stadiums soon.

“It’s the first of more,” he said.

END PART 2… HERE IS THE LINK TO PART 1… TO READ THE WHOLE STORY NOW, DOWNLOAD OUR REPORT!