AT&T sees double DAS usage at College Football Playoff championship game

Screen Shot 2017-01-10 at 12.06.37 PMWireless data use at big sports events keeps continuing to grow, with AT&T reporting that its cellular network traffic from Monday’s College Football Playoff championship game between Clemson and Alabama was double the total from last year’s game.

According to AT&T, it saw fans use a total of 3.8 terabytes of wireless data Monday, on its stadium distributed antenna system (DAS) network at Raymond James Stadium as well as from other network sites in and around the stadium in Tampa. At last year’s championship game in Glendale, Ariz., AT&T saw 1.9 TB of data used on its cell networks. Keep in mind, these numbers are for AT&T networks ONLY, so the total wireless numbers are much larger.

Unfortunately, Verizon Wireless is (so far) declining to report its wireless data statistics from Monday night’s game, a situation we hope they reconsider; we are also still waiting to hear from Sprint and T-Mobile representatives to get their figures from the event. We also have a call in to the stadium authorities to see if we can get figures from the in-stadium Wi-Fi network, so stay tuned. If AT&T’s numbers are any indication, the thrilling 35-31 Clemson victory might just join our list of top single-day wireless event, especially since the event set an attendance record with 74,512 on hand to witness the drama.

NRG Stadium Wi-Fi ready for Super Bowl LI

NRG Stadium. Credit: Houston Texans Instagram

NRG Stadium. Credit: Houston Texans Instagram

Nonexistent a year ago, the new Wi-Fi network at Houston’s NRG Stadium has at least one more live game that administrators can use as a final tune-up before the venue and its wireless infrastructure host Super Bowl LI on Feb. 5.

Live since the start of the current NFL season, the Wi-Fi network deployed by integrator 5 Bars using Extreme Networks Wi-Fi gear has seen growth in fan usage for each subsequent game, according to David Moore, manager of information services for NRG Park.

“The first few games [of the season] it wasn’t heavily promoted, but as we went on usage shot up,” said Moore in a recent phone interview. Though he wouldn’t release specific figures on data use, Moore said that game-day totals near the end of the season saw in the range of 25,000 unique users per game, with data totals in the “4-5 terabyte” range. The stadium’s main tenants, the Houston Texans, will have at least one more home game this weekend when they host the Oakland Raiders in the first round of the NFL playoffs.

Under seat APs visible down seating row. Credit: 5 Bars

Under seat APs visible down seating row. Credit: 5 Bars

While it’s possible that the Texans could be hosting the AFC Championship game if all the higher-seed teams lose en route, this weekend’s game is most likely the last chance the NRG Stadium tech team will have as a dress rehearsal for the Super Bowl. Since the Super Bowl is historically the biggest single-day data-usage event — and has gotten bigger every year — all technical eyes will be on the NRG Stadium network, which only started becoming a reality after the stadium hosted last year’s men’s NCAA basketball Final Four. With a base seating capacity of 72,220, NRG’s Super Bowl crowd should roughly be the same as last year’s at Levi’s Stadium in Santa Clara, Calif.

Fast deployment schedule met

“It was a challenge [to deploy so quickly] and we are still working out the tuning,” said Michelle McKenna-Doyle, the NFL’s chief information officer, in an email interview. “We were fortunate to have the regular season to work on it, but hope to have it Super Bowl ready within next couple weeks. 5 Bars, Extreme Networks, The Texans, Harris County and the host committee all worked hard to make it a reality for our fans.”

At one point in recent history, it seemed like NRG might never get a fan-facing Wi-Fi network since the county-owned facility couldn’t find the budget necessary to bring wireless connectivity to the venue. But with the Super Bowl approaching, a consensus finally pushed through, with 5 Bars winning a deployment bid that still needed a change in equipment from Ruckus (the original supplier in the 5 Bars bid) to Extreme. While no official breakdown of funding shares has been supplied, the reported $6 million-plus cost of the Wi-Fi deployment was likely shared in some fashion by the Texans, the NFL, Verizon Wireless and Extreme.

Ready for the playoffs and Super Bowl! Credit: 5 Bars

Ready for the playoffs and Super Bowl! Credit: 5 Bars

According to Moore, Verizon chipped in to secure guaranteed offload for its cellular customers, as well as its own SSID that will automatically sign on Verizon customers in the facility, a tactic used by Verizon in several other NFL stadiums. Non-Verizon customer fans can use a free xfinitywifi SSID, as Comcast is the backbone supplier for the Wi-Fi network services.

According to Moore there are 1,250 Extreme Wi-Fi APs in the venue now, with 550 of those deployed under seats in the main bowl. While Moore said the under-seat location allows for much denser deployment and better network tuning, the method also caused the most headaches during deployment, beyond the usual cost and struggle of drilling through concrete floors.

First of all, the installers had to bring all the necessary power and cabling infrastructure in, since there was nothing underneath the concrete seat floors, Moore said. In fact, because there are offices underneath some of the seating areas, contractors had to negotiate a “drip pan” that kept moisture from seating power-washing away from the office roofing.

Another “big hiccup” emerged when the original equipment used for the under-seat locations didn’t lock out all the moisture, leading to a full replacement of all the 550 under-seat APs. Moore said the under-seat locations now use Extreme’s highest-grade outdoor-rated AP, the 3965i.

New DAS installed last year

Since Super Bowls also typically set records for cellular DAS usage, it’s no surprise that Verizon also recently updated the DAS at NRG Stadium, reportedly putting $12 million into a new system installed before the 2015 season. According to Moore, the DAS and the Wi-Fi also cover parking areas outside the venue, including a plaza where the Texans typically see 6,000 or more fans gathering before games.

“There’s great coverage” in the parking lots with the new DAS, Moore said. Announcements have not yet been made public, but you can expect that both AT&T and Verizon are busy beefing up the cellular systems in and around the stadium, as typical before any big public event.

With the dust finally settled, Texans fans and the soon-to-arrive Super Bowl fans will have good connectivity for the big day, due in no small part to the efforts of 5 Bars, which Moore complimented repeatedly for the company’s persistence and effort.

According to Moore 5 Bars had brought semi-trailers full of equipment to town ahead of the Final Four, and even used the floor of the old Houston Astrodome to roll out network inventory.

“They were working under the gun, with a limited amount of time to get it done,” said Moore of 5 Bars.

AT&T beefs up DAS at Tampa’s Raymond James Stadium ahead of College Football Playoff championship

Tampa's Raymond James Stadium. DAS antennas visible on light standards. Photos credit: AT&T

Tampa’s Raymond James Stadium. DAS antennas visible on light standards. Photos credit: AT&T

With the college football playoff championship game coming to Tampa’s Raymond James Stadium on Jan. 9, 2017, it’s no surprise that wireless carriers like AT&T have been beefing up coverage in and around the venue ahead of what is historically one of the biggest wireless-use events in sports.

According to our unofficial list, the last two college football playoff games rank fifth and sixth overall in the list of “most Wi-Fi used for a single-day event,” trailing only the last two Super Bowls, WrestleMania 32 and a Texas A&M home game against Alabama. (Note to stadium IT types: If you have a recent event that should be on our list, let us know!) DAS stats from the CFP championship games were also among the top usage totals for single-day events, with such numbers still growing year to year.

DAS antenna visible on red stanchion

DAS antenna visible on red stanchion

For this year’s game at the home of the NFL’s Buccaneers, AT&T said it had increased coverage via the stadium’s DAS by 400 percent, now up to a total of 452 antennas inside the venue. In and around town, AT&T said it had invested more than $9 million in new improvements, including 20 new or enhanced cell sites, ahead of the playoff championship weekend. In addition, AT&T will be deploying 2 cell on wheels or COWs during the event.

MSR TOP 3 TOTAL USAGE

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB; DAS: 15.9 TB; Total: 26 TB
2. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB; DAS: 6.56 TB**; Total: 12.79 TB**
3. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB; DAS: 1.9 TB*; Total: 8.6 TB*

* = AT&T DAS stats only
** = AT&T, Verizon Wireless and Sprint DAS stats only

MSR TOP 5 FOR WI-FI

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB
2. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB
3. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB
4. Alabama vs. Texas A&M, Kyle Field, College Station, Texas, Oct. 17, 2015: Wi-Fi: 5.7 TB
5. College Football Playoff championship game, AT&T Stadium, Arlington, Texas, Jan. 12, 2015: Wi-Fi: 4.93 TB

Will stadiums soon be able to rent their Wi-Fi networks from equipment vendors?

Nationwide Arena. Credit: Columbus Blue Jackets

Nationwide Arena. Credit: Columbus Blue Jackets

If it costs too much to buy a Wi-Fi network for your stadium, why not rent one instead?

A fairly common option in the world of enterprise networking, the ability to rent, or lease a fully operational Wi-Fi network may soon be coming to the world of sports stadiums and other large public venues, if it isn’t here already. Three of the largest Wi-Fi gear suppliers, Cisco, Aruba and Ruckus, already have public offers of network leasing arrangements, where venue owners could pay some kind of monthly or recurring fee for network setup and operation, instead of buying equipment outright. And Cisco, another leader in the marketplace, is rumored to be offering full-control lease-type arrangements for stadium Wi-Fi networks, possibly beginning with the new Wi-Fi network being built at the SAP Center in San Jose.

Though no large sports stadium has yet publicly announced a deal to lease, or in networking lingo, to buy a “Network as a service,” or NaaS, the idea is potentially attractive to stadium owners and operators, many of whom have struggled with the return-on-investment question ever since the idea of putting wireless networks in stadiums has emerged. While cellular carriers have so far borne the lion’s share of the costs of deploying enhanced cellular systems like DAS (distributed antenna system) in stadiums, the question of “who will pay for the Wi-Fi” is still a big one for many venues, especially those that are only filled several times a year.

The benefits of moving to opex vs. capex

Bart Giordano, vice president of business development for the Ruckus business unit at Brocade, said the idea of leasing a stadium network could be attractive especially to venue owners who don’t have the upfront capital necessary to pay for Wi-Fi, a cost that could run into the tens of millions of dollars.

Under new parent Brocade, Ruckus Wi-Fi gear can be obtained via something called the Brocade Network Subscription, a NaaS program that Giordano said “converts it all to opex — you subscribe to the network and pay a monthly service fee.” Under the subscription program, Giordano said Brocade/Ruckus will actually own the equipment, allowing the venue owner the flexibility of being able to return it or upgrade it as needed.

With many stadiums that deployed Wi-Fi several years ago already going through significant upgrades, the idea of a leased network that could be more easily refreshed with new technology might soon gain favor. Though no Ruckus stadium subscribers have yet been announced, Giordano said “some are coming.”

Aruba, now a Hewlett Packard Enterprise company, has a similar subscription model plan for enterprise wireless deployments, one which company representatives said could be used by stadiums as well. Both companies said such deals could possibly come via consulting partnerships, where the consultant firm manages the relationship and deployment/operation details.

Cisco also has a leasing option available for wireless networks, but so far has not made any public announcements of such deals in the sports stadium marketplace. However, there are reports of Cisco taking a more active role in the ownership, deployment and operation of stadium networks, like the Cisco-powered Wi-Fi currently being installed at the SAP Center in San Jose, home of the NHL’s Sharks. So far, neither Cisco nor the Sharks will comment on any business specifics of the new Wi-Fi network other than its use of Cisco gear.

Can leasing work for stadiums?

While the leasing idea for stadiums isn’t new, the business model has met some challenges along the way of the short history of wireless networks in large venues. So far, third-party integrators like Mobilitie, ExteNet Systems and Boingo have crafted lease-like deals in which the venue does not pay the full cost of the network but instead allows the operators to run networks (typically both DAS and Wi-Fi), earning money by leasing space on those networks to wireless carriers or by selling advertising or sponsorships.

Another leasing model, one that crashed and burned, was the one employed by SignalShare, a company now in bankruptcy proceedings with legal claims of fraudulent business practices against it. SignalShare, which also offered venues networks for a monthly cost, may have been hampered by a lack of financial resources, something that shouldn’t be an issue for companies the size of Cisco, HP and Brocade, who will mainly be offering leases on equipment they manufacture themselves. The larger equipment vendors may also not be under as much pressure as SignalShare was to earn revenues on the network operations, which may make them better able to succeed in the NaaS space.

And while the idea sounds good in theory, there are still unanswered questions about how the leases would work, and whether they will make good business sense for both sides. Unlike enterprise operations in traditional offices, stadium networks are far more complex to install and operate, especially those being retrofitted in stadiums built decades ago. Stadium networks also have a much different operational profile, with traffic coming in large spikes rather than daily workday routines.

But stadium networks can also act as public advertisements of sorts, gaining more attention for vendors in PR than perhaps in direct profits. As the market matures and vendors seek out potential customers who have shied away from Wi-Fi in the past due to upfront costs, leasing may be a way forward for both sides — as long as both can find a benefit to the deal.

Washington upgrades Wi-Fi backbone at FedExField

Screen Shot 2016-11-30 at 11.07.38 AMFans at FedExField now have more bandwidth to use on the stadium Wi-Fi network thanks to a recent upgrade that moved the backbone capacity to 10 gigabits per second, according to a release from the team.

Though it’s never been previously publicly announced by the Washington, D.C. NFL franchise, according to sources close to the team the reported network built by Verizon Wireless using Cisco Wi-Fi gear has been in full operation since last season, albeit under a sort of “soft launch” mode done without any promotion.

With more fans now using the network, the team saw a need to increase bandwidth from the 1 Gbps pipe that had been supplying all the Wi-Fi network needs. For many football-stadium Wi-Fi networks, 10-Gbps pipes are a popular choice, with several stadiums opting for multiple such connections. Levi’s Stadium in Santa Clara, Calif., home of the San Francisco 49ers, has four 10 Gbps pipes as part of its backbone configuration, while US Bank Stadium in Minneapolis, the new home of the Minnesota Vikings, has six 10 Gbps links.

For those who are new to stadium Wi-Fi (or who don’t know the difference between backhaul and actual end-user network speeds), the backbone bandwidth is shared by the entire network, which may have multiple tens of thousands of client devices running off any of hundreds of Wi-Fi access points. Typical user bandwidth speeds on Wi-Fi in large networks can range from single-digit megabits per second anywhere up to 20, 40 or even 60 Mbps, depending upon network configuration and density of users. Any fans attending games at FedExField who want to send us a speedtest please do!

Colorado passes on full-stadium Wi-Fi or DAS for Folsom Field

View of the west stands at Folsom Field, home of the University of Colorado football team. Credit all photos: Paul Kapustka, MSR (click on any photo for a larger image)

View of the west stands at Folsom Field, home of the University of Colorado football team. Credit all photos: Paul Kapustka, MSR (click on any photo for a larger image)

With a No. 9 ranking in all the major polls, the University of Colorado football team is experiencing a resurgence this season, which may lead to a rare CU sellout for this weekend’s final home game against Pac-12 rival Utah.

While the Buffs’ on-field performance in 2016 may have ended years of fan frustration, the 50,000+ fans expected to be in attendance at Folsom Field this Saturday may still experience another form of frustration, mainly in trying to get their mobile devices to connect to the Internet. According to school officials, there is no full-stadium, fan-facing Wi-Fi or cellular distributed antenna system (DAS) network in Folsom Field, and no plans to bring either to the venue anytime soon.

Instead, most fans at the on-campus stadium will rely on one of two nearby macro sites, one each from top wireless carriers AT&T and Verizon Wireless. According to Jeff Lipton, director of real estate for CU, the school decided to pass on bringing Wi-Fi or DAS to Folsom for a number of reasons, including the cost of the systems, the infrequent use of the facility, and potential loss of network control to prospective suppliers.

Hard to justify cost of connectivity for lightly used venue

“Nationally, stadiums are challenging” when it comes to cost justifications for installing wireless networks, Lipton said in a recent phone interview. While Lipton claimed that CU “hasn’t been sleeping on this,” saying the school has been reviewing wireless stadium options for several years, he added that CU had concluded that a cellular DAS wasn’t a good fit for Folsom, which has been the home of CU football since 1924.

Error message shown while trying to connect to ESPN's website at Folsom Field on Nov. 19.

Error message shown while trying to connect to ESPN’s website at Folsom Field on Nov. 19.

One of the main problems is, Lipton said, justifying the cost to bring connectivity to a venue that is only used a handful of times a year. In 2016, Colorado had six home games on its football schedule, the main use of 53,613-seat Folsom Field. This past summer there were two concert dates with the Dead and Company band that filled the stadium, and the stadium is also used as the finish line for the annual Bolder Boulder 10k on Memorial Day.

Though the crowds that do come to Folsom would no doubt enjoy better connectivity, right now it’s not in the cards, Lipton said, especially from a cellular DAS perspective.

“We looked at DAS for the main part of the stadium but determined it was not cost effective, and the vendors wanted a connectivity exclusive” for the rest of the campus, a deal Lipton said CU definitely did not want to agree to.

“We like to control our own [wireless] destiny on campus — we’re not going to give that up in a deal to get DAS,” Lipton said. And, Lipton said that “I’m not sure that [technically] in the long term, DAS is the solution” for stadium networks.

The right way to Wi-Fi

What’s more interesting to CU is finding some way or waiting for new technology to emerge to make owning and operating a Wi-Fi network inside Folsom something that makes sense. When it comes to Wi-Fi, Lipton said that CU has been aggressively installing it in many of the 12 billion square feet of building space it manages at the Boulder, Colo., campus.

Around Folsom, there is free public Wi-Fi available at the new Champions Center (an indoor practice field and offices building located adjacent to the east side of Folsom Field) as well as in the attached parking structure. There is also some free Wi-Fi available for suites and club spaces in the newer structures on the east side of Folsom Field, but nothing for the balance of bowl seating in the stadium.

Folsom's east side structure, which does have some Wi-Fi inside suites and club areas.

Folsom’s east side structure, which does have some Wi-Fi inside suites and club areas.

“We have pretty ubiquitous Wi-Fi throughout campus, and we installed it and run it,” said Lipton. In terms of bringing Wi-Fi into the Folsom Field bowl — as well as to the stands at the Coors Event Center, the school’s 11,00-seat basketball facility — that idea is still being studied by an internal working group, Lipton said.

“We recognize that long term, there are some real revenue opportunities [around Wi-Fi networks] that could pay for this later on,” Lipton said. “But it’s not there yet.”

Any Wi-Fi network that does end up getting built inside Folsom would also have to surmount the non-trivial challenge of bringing wireless networks to a facility with parts that are nearly 100 years old. Part of the bowl also sits in the ground, bringing another degree of difficulty to the idea of getting cables underneath the stands (for possible under-seat or railing-based antenna options). But for newer parts of the stadium, including the north end zone structures and the new east side, bringing connectivity to the stands outside wouldn’t be as difficult.

At the 11,064-capacity Coors Event Center, Lipton said there is some CU Wi-Fi inside the building, but it was not designed for full-stadium crowd access down into the seating bowl.

Unable to send texts, or get Internet access

Though Lipton claims that the two Folsom macro sites — one atop the roof on the stadium’s west side building and another on a building across the street — are working “much better” this season, an MSR visit to Folsom for the Nov. 19 home game against Washington State saw almost zero connectivity, on both the cellular and Wi-Fi front.

A look at the newer north and northeast structures at Folsom Field from the east stands.

A look at the newer north and northeast structures at Folsom Field from the east stands.

Though our tests were sporadic, with only one phone in one part of the stadium, our not being able to send a text message with a photo of the stunning Colorado Rockies backdrop was probably something many others experienced inside Folsom last week, where 48,658 fans saw CU beat WSU 38-24. On the Verizon network, we were almost always looking at a “1x” number for connectivity, which pretty much guaranteed no signal all day long.

Even inside the east building’s 5th-floor club area, where we detected the CU campus Wi-Fi network, our phone couldn’t connect, briefly showing a link but then dropping it as soon as we tried to do anything. There was no visible promotion of the CU Wi-Fi, or any instruction about whether fans should use one of two visible SSIDs, one with a “guest” label and one without. Back out in the stands, we tried to get to the ESPN website to see other college scores, but again our device failed to connect.

While Lipton admitted that “traffic on [football] game days can overwhelm” the macro sites, he still thinks any advanced connectivity has to make fiscal sense. As the one who says he signs contracts for such deployments at CU, Lipton said “there’s an art to every deal.” For Folsom Field connectivity, however, that deal hasn’t yet been done.