The MSR Interview: San Francisco Giants CIO Bill Schlough

AT&T Park CIO Bill Schlough shows off his World Series bling.

AT&T Park CIO Bill Schlough shows off his World Series bling.

Who better to talk about stadium Wi-Fi than the guy who was there when it all started? Our guest for our first MSR Interview (part of our Stadium Tech Report Podcast series) is San Francisco Giants senior vice president and chief information officer Bill Schlough, who goes old-school talking about stadium Wi-Fi back in 2004… and brings it to the current day with stats from the most recent season at AT&T Park. Plus, his thoughts on game-day apps and why great connectivity is the real winner. Listen in now!

Hear Bill talk about:

— New Wi-Fi records set… during the Warriors’ playoff run

— Why going under-seat with Wi-Fi was a necessary thing to do

— How the Giants are experimenting with virtual reality

— Why he thinks great connectivity matters most (even more than stadium-app features)

Some story links that offer some history about AT&T Park’s networks from MSR:

S.F. Giants add more Wi-Fi, ‘virtual reality experience’ to AT&T Park for 2016 season

SF Giants fans used 78.2 TB of Wi-Fi data at AT&T Park during 2015 season

Stadium Tech Report: World Series set new wireless records at AT&T Park

Stadium Tech Report: San Francisco’s AT&T Park lives up to its wireless reputation

Giants: NLCS stadium Wi-Fi usage at AT&T Park quadrupled since 2012

SUBSCRIBE TO THE PODCAST:

Here is the link to the podcast on iTunes!

Stadium Tech Report: Carolina Panthers take ownership of DAS, Wi-Fi at Bank of America Stadium

James Hammond, director  of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

James Hammond, director of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

“The fan is the most valuable member of our team,” Jerry Richardson, owner of the Carolina Panthers, is fond of saying.

And it’s become the virtual mission statement for the Charlotte, N.C.-based National Football League franchise. So even though its home field, the Bank of America Stadium, was built relatively recently (1996), technology has come a long way in two decades. And as the Panthers began a four-phase renovation in 2014, they did it with fans’ MVP status in mind, according to James Hammond, director of IT for the Panthers. “It was time for some changes,” he said.

While Carolina was among the first NFL stadiums to install fan-facing Wi-Fi and enhanced cellular networks, the previous DAS and Wi-Fi systems weren’t keeping up with demand and that was starting to adversely impact the Panthers fan experience, Hammond said.

“We chose to perform a rip-and-replace on both DAS and Wi-Fi and take ownership in-house,” Hammond explained. Because the Panthers own and operate BofA Stadium, making those moves was a lot easier than if they were tenants.

Time for an upgrade

Editor’s note: This profile is part of our latest STADIUM TECHNOLOGY REPORT, which includes more stadium profiles as well as looks at Wi-Fi at the Mall of America, and analytics software being used by the Cleveland Browns. DOWNLOAD YOUR FREE COPY of the report today!

The first “fan-centric improvements,” as Hammond called them, came in 2014 in the form of escalators, video big boards and a distributed audio system. As part of the second phase of upgrades, the Panthers then used the 2015 offseason to renovate the club-level suites and tore out the old DAS system while they were at it. And after a careful evaluation of different DAS solutions, they shortlisted two vendors: CommScope and Corning.

CommScope ultimately got the nod; the Panthers then had to decide between the vendor’s ION-B and ION-U DAS systems. “We went with the ION-U, which was quite new and cutting edge at that point, since it had NEMA-rated remotes,” Hammond said. Other systems lacked that kind of weatherproofing and would require additional enclosures – and expense.

CommScope's ION-U powers the new DAS at Bank of America Stadium.

CommScope’s ION-U powers the new DAS at Bank of America Stadium.

“We started over with all new fiber and coax. We did the decommissioning and construction in 90 days, which was pretty quick for a ground-up project,” he said. Beam Wireless Inc. did the design, integration and optimization and is handling the ongoing maintenance of the DAS system; Optical Telecom installed the DAS gear. BofA Stadium now has 256 DAS remotes and more than 600 DAS indoor and outdoor antennas.

AT&T, Verizon and Sprint are the participating DAS carriers; T-Mobile is weighing whether to join the mix during the 2017 off-season.

The Panthers have divided BofA Stadium into 48 DAS zones: 16 zones for the upper bowl, 16 in the lower bowl, and another 16 for concourses, suites, clubs, and offices. Not all zones are used exclusively; carriers choose simulcast patterns that place multiple zones into sectors, and can change them as capacity requirements dictate, Hammond told Mobile Sports Report.

“With some minor design changes to the interior areas, we can accommodate nearly 70 zones,” he explained. “At present the most sectors in use by a carrier is 32. This means the carrier simulcasts across a mix of our 48 zones in order to match them up to 32 carrier sectors.”

Once the new DAS was built and the first couple of events were analyzed, carriers began asking for more frequencies and additional DAS sectors to continue meeting ever-growing demand. In response to the new carrier requests, the first round of DAS upgrades were implemented in the spring of 2016, Hammond said. During the 2015 season, DAS bandwidth was running around 2 GB during games. Hammond said, “With these latest DAS upgrades, we expect the bandwidth numbers to be even higher.”

A DAS remote in a NEMA-rated enclosure.

A DAS remote in a NEMA-rated enclosure.

The impact of the new DAS system was felt immediately upon its debut in July 2015. “It was a much better experience for fans who noticed the improved cellular experience,” Hammond said. Another unexpected benefit: The upgraded DAS helped mitigate bottlenecks with the old Wi-Fi system, which Hammond characterized as “under-designed.”

Going under seat for Wi-Fi upgrade

Unfortunately, there wasn’t time to address any Wi-Fi upgrades before the 2015 football season began, but the Panthers issued an RFP for new Wi-Fi in August 2015 in preparation for Phase 3 renovations that would also include security upgrades and renovations to the upper concourse.

Interested vendors needed to ensure high bandwidth rates as well as high take-rates that allowed three different ISPs (Time Warner Cable/Charter, Level 3 Communications and Windstream) to deliver in excess of 10 GB, though Hammond said they’re starting at a 7-GB threshold.

The Wi-Fi award went to Aruba, now HP Enterprise, in December 2015, and construction began in January 2016 after the last postseason game, when the Panthers beat the Arizona Cardinals to win the NFC championship and a trip to Super Bowl 50.

Similar to Levi’s Stadium and the Dodgers Stadium, the Panthers chose underseat AP enclosures; BofA Stadium sports 770 AP enclosures in the upper and lower bowls out of a total of 1,225 APs, all to ensure maximum coverage and minimal dead spots. The Panthers selected AmpThink to do the Wi-Fi integration and construction; the turnkey contractor also designed and fabricated a custom enclosure for the APs.

Indoor access point inside the stadium.

Indoor access point inside the stadium.

One other innovation in the Panthers’ Wi-Fi installation is that the underseat enclosure is mounted to the riser — the vertical part of the step — but looks like it’s on the tread, the horizontal part, which is intended as a waterproofing measure. “The riser is easier to seal and isn’t affected by pressure washers, which you’re doing constantly with an outdoor stadium,” Hammond said. “And by running pipe through the riser, you don’t have gravity working against you,” which helps keep out water, he explained.

Panthers fans access the stadium Wi-Fi through a portal page after accepting the team’s terms and conditions. From there, they are whitelisted and can automatically join the Wi-Fi network for the rest of the season. Hammond said a fan’s email is requested but not required by the portal page, and there’s a small incentive offered to encourage fans’ email subscriptions.

The new Wi-Fi system got a workout with a soccer game at BofA Stadium at the end of July 2016, then with a Panthers’ Fan Fest the following week. “All the indicators were good, and fan feedback about the system was excellent,” Hammond said. But he cautioned that the two events were not “full bowl” events with smaller attendance numbers (~50,000) than a regular season football game (75,000+). “We will continue to optimize and tune settings as we learn more during events with higher attendance,” Hammond said.

Total budget so far for the technology upgrades totals about $16 million; the DAS build-out was just under $10 million; Wi-Fi was a little more than $6 million, which included additional wired infrastructure, according to the team.

Beacons coming next

And the Panthers aren’t done making technology improvements to their stadium. Phase 4 looks to add Bluetooth beacons and do some refinement of the Panthers app. “My goal during the upcoming season is to look at options for location-aware services,” Hammond said. Some APs have beacons built in; other may need to be added to get the granularity the Panthers want for location awareness.

Hammond also wants to give fans more things to do with the Panthers app and also optimize it for push notifications, even with something as basic and useful as restroom and concessions location information. “As we learn more about fans individually, we can direct them to things of particular interest to them,” he added.

“So far, we are very pleased with the performance of the Wi-Fi and DAS systems,” Hammond said, noting the Panthers will continue to tune frequencies, add zones and increase bandwidth where needed. It’s the sort of attention that smart sporting franchises pay to their most valued team members.

car1

Ookla shares Speedtest data from CenturyLink Field, other stadiums

Ookla ad banner being flown over CenturyLink Field in Seattle. Credit: Ookla

Ookla ad banner being flown over CenturyLink Field in Seattle. Credit: Ookla

Anyone who follows Mobile Sports Report knows that I use the Speedtest app from Ookla to measure stadium network performance whenever I visit a sporting venue. While my one-man tests do show some measure of network power, I always dreamed of harnessing the results from many fans at the same game to see a better picture of the network performance.

Well, Speedtest’s creators think along the same lines, and conducted an experiment during an Aug. 25 Seattle Seahawks preseason game at CenturyLink Field in Seattle. You can read their very thorough post and neat results here, with some interesting twists — for instance, the cellular networks are way faster than the CenturyLink Wi-Fi, according to the Ookla results.

UPDATE: Ookla responded to our email and let us know that on Aug. 25, there were 252 Speedtests at CenturyLink Field, a great sampling to draw results from. Ookla also talked about tests from 12 different events at CenturyLink Field, and said in the email that across those events it saw 1,143 tests conducted.

Ookla also published some test result totals from other stadiums as well, including Levi’s Stadium, AT&T Stadium and Bank of America Stadium, but didn’t say when those tests were recorded, or how many tests were taken.

What we really like, however, is that Ookla’s tests show what our stadium tech report surveys have been showing — that overall, in-stadium network performance is steadily improving. Over time, more data like this can help dispel the still-lingering rumor that stadium networks don’t deliver good connectivity. Now if we could only get Ookla to partner with us to do league-wide or college-comparison speedtests… anyone ready for that idea?

Will cellular carrier aggregation matter in stadium networks?

Kauffman Stadium during 2015 World Series

Kauffman Stadium during 2015 World Series

Over the past few days, both Sprint and Verizon Wireless have made announcements about a technique called “carrier aggregation” (CA for short) for LTE cell networks that basically bonds together different frequency channels to bring more bandwidth to a mobile device. Though the premise sounds great, what we here at MSR HQ haven’t been able to ascertain yet is whether or not this technique will help solve the biggest problem in stadium network situations, namely providing enough capacity for users on the networks installed there.

Sprint has made the most noise this week, with claims of CA demonstrations at Soldier Field in Chicago and Kansas City’s Kauffman Stadium that (they said) showed Sprint devices bonding three different frequency channels to hit download speeds of 230 Mbps, a score way off the charts for any existing stadium networks. (The fastest Wi-Fi and cellular speeds we’ve seen in our short history of stadium tests, by comparison, are in the 60 Mpbs range.) Verizon made a similar announcement about CA being put in across its network, without specifying if the service would be available in stadiums. Other carriers, including AT&T and T-Mobile, are also exploring use of the CA technique. At the very least, some lucky users with newer devices may see leaps in performance thanks to CA deployments, a good thing on any level.

But our bigger question — which hasn’t been answered in the press releases and hasn’t (yet) been answered in email questions to Sprint or Verizon — is whether or not CA will help with overall network capacity, which to us seems to be a more pressing problem at most stadiums as opposed to simple download speeds. I mean, demos are great and it’s cool to see what the upper limits are for one device; but it’d be more impressive if Sprint could guarantee that 230 Mbps mark to every device in the park, should everyone there have a Sprint phone with the capability to perform the CA trick (not all devices in the market today can do so).

Finally using the Clearwire spectrum

What’s also not completely revealed in the press releases is what kind of gear is necessary on the back end of the network to make CA work, and whether or not it makes economic sense to have that gear placed inside stadiums to enable the technique for as many fans as possible. While we understand the basic premise probably better than most (since in a former life yours truly spent several years following and analyzing the Clearwire spectrum holdings at 2.5 GHz) it’s not clear if CA solves any congestion problems, especially for carriers other than Sprint, who only have a limited amount of licensed spectrum in each market they serve.

(Without getting too deep into spectrum geekiness, Sprint on paper probably has more room to grow in the CA space since its 2.5 GHz holdings dwarf other carriers’ licensed bands; but to make use of that spectrum, you need customers with devices that can use that spectrum, and enough cash for a wide network buildout, both of which Sprint may be challenged to find.)

As we understand CA, by bonding channels you can make one device faster since it has more aggregate bandwidth to work with. But it’s not clear that using CA in a stadium environment would make the overall situation any faster than say, three phones using single channels by themselves. Also, since you can’t create new bandwidth, if one phone starts tapping three different channels doesn’t that actually leave less room for other devices that may want to also use those channels? Perhaps with CA the connections would be faster and wouldn’t last as long, thereby freeing up spectrum for other devices; again, there’s not a lot of information yet on the capacity side of the equation, especially in crowded stadiums or at big events where bandwidth needs escalate. If there are any cellular wizards in the audience with more knowledge of the situation, feel free to chime in.

We did get an email response from our old friend John Saw, formerly of Clearwire and now chief technical officer at Sprint. Here’s his explanation of why CA is a good thing for stadiums:

Essentially, sites with bonded channels will drive higher capacities. This will be especially timely and helpful in crowded spaces like Soldier Field where there are surges in capacity demand during live sporting events. Sprint customers with CA enabled phones will enjoy 2X (in the case of 2CA) or 3X (in the case of 3CA) their download speeds, which means that they will get a better data experience with a bigger pipe. But wait – CA will lift all boats and it will also benefit those Sprint customers who have not upgraded to CA enabled phones yet. While they may not enjoy the higher peak speeds enabled by CA phones, their phones will have access to more network resources which means they will also have a better data experience, with no stalling or without that dreaded “windmill effect” in a crowded stadium.

I kind of understand what Saw is talking about here, but I am still having a problem with the math that says all boats will be lifted through the use of CA. Plus, experience and interviews have taught us that across the country, Sprint is behind Verizon and AT&T when it comes to DAS deployments inside stadiums; and, it’s not clear (and hasn’t been answered) whether or not CA can work over a neutral-host DAS deployment where carriers share antennas and other infrastructure.

From an industry-wide standpoint, CA seems like a great thing for all cell phone users since as it progresses devices should be able to utilize whatever bandwidth is around to make performance better. It’s also good to see more technology advancements made on the network side of things, since infrastructure needs all the help it can get to keep up with devices. But right now, we’re not sure if CA is the answer to any of the capacity problems stadium network operators face. Anyone with views that can expand the explanation, feel free to hit the comments section below or send me an email to kaps at mobilesportsreport.com.

Fan-facing Wi-Fi on hold as Coliseum gets ready for Rams’ return to Los Angeles

DAS antennas visible on the LA Coliseum's facade

DAS antennas visible on the LA Coliseum’s facade

Normally when a new professional sports franchise comes to town or opens a new venue, preparations move into overdrive pretty quickly — especially for infrastructure like luxury suites, Wi-Fi and DAS.

But this is the Los Angeles Rams, and nothing about the team’s trajectory is normal, including the technology. So after making good on a longstanding threat to move the team from St. Louis, owner Stan Kroenke in January broke ground on a new, $2.6 billion “NFL Disneyland” venue in the LA suburb of Inglewood. It’s expected to be ready in time for the 2019 season. So for right now, the returning Rams will play at the Los Angeles Coliseum.

While it has an established and robust DAS system, the Coliseum has no dedicated, fan-facing Wi-Fi network, just as the Edward Jones Dome in St. Louis didn’t. But it’s LA, baby. Kanye. Jack. Celebrity sightings means more bandwidth is needed, not less. But the first appearance of Wi-Fi at the venerable Coliseum won’t be for fans, but for operations.

Relying on DAS for fan wireless

Editor’s note: This profile is from our most recent STADIUM TECH REPORT, the Q2 issue which contains a feature story on Wi-Fi analytics, and a sneak peek of the Minnesota Vikings’ new US Bank Stadium. DOWNLOAD YOUR FREE COPY today!

Artist rendering of the proposed new LA football stadium

Artist rendering of the proposed new LA football stadium

So when the Rams play their first pre-season game at the Coliseum in August against the Dallas Cowboys, fans will have to rely on the DAS network for connectivity. AT&T, Verizon Wireless and more recently T-Mobile, are the DAS carriers; Sprint trucks in a COW for USC games.

The Rams will bring in their own Wi-Fi for communications to and from the sidelines, along with their Microsoft Surface tablets, according to Derek Thatcher, IT manager at the Coliseum and an employee of USC, which oversees and administers the venue for Los Angeles County. There will also be private Wi-Fi in the locker rooms and the officiating rooms. Thatcher’s working closely with the Rams and the NFL, including one of the NFL’s frequency coordinators, to ensure everybody has the bandwidth they need.

Separate from the Rams and the NFL, USC is undertaking a major renovation of the Coliseum, home to the university’s storied football team. The $270 million project cost will be funded entirely by USC Athletics from capital gifts, sponsorship revenue, non-USC athletic events at the Coliseum, and donor naming opportunities. “The project will not require any student fees or general university, local, state or federal funds,” the university said on ColiseumRenovation.com.

In addition to significant expansion of luxury suites and press box facilities (which houses most the IT and networking gear for the Coliseum), USC will also be adding public Wi-Fi and is talking with different vendors about their requirements.

DAS antennas inside the concourse

DAS antennas inside the concourse

It is worth noting that Aruba Networks provides wireless networking for most of the adjacent USC campus (more than 5,000 APs), including the Galen Center and USC’s healthcare facilities. Thatcher emphasized the bidding was open to all vendors.

Waiting for the new stadium to be built

The Coliseum’s current capacity is 93,600 and the NFL will use 80,000 seats; post-renovation, capacity will be 77,500, due to replacement of all seats and addition of handrails to the aisles. That’s good news for Wi-Fi engineers, since the Coliseum’s bowl design has no overhangs to speak of; DAS antennas are mounted to poles that ring the stadium, and also above the bowl’s entry and exit tunnels. “Underseat AP design is expensive… we could end up with underseat and handrail,” Thatcher told Mobile Sports Report. “We’re looking at all possible solutions.”

The renovation is scheduled to begin right after USC’s 2017 football season ends and is expected to be done in time for the 2019 opener. The university said it will plan the construction schedule so that 2018 season can still be played at the Coliseum.

The other wild card in the Coliseum’s future is the Los Angeles bid to host the 2024 Summer Olympics. The city has already done the honors before (1932, 1984) and the Coliseum served as the Olympic Stadium both times; a third time hosting would be unprecedented. But because of LA’s experience, coupled with plenty of already-built sporting venues to handle a plethora of events and requirements, it was natural for the US Olympic Committee to turn to LA once Boston bailed.

The city’s bid includes $300 million for additional renovations to the Coliseum.

Rome, Paris and Budapest are also competing to host the games. The winner will be announced by the International Olympic Committee in September 2017.

Kentucky Derby gets it done with DAS: Podcast No. 5 looks at Mobilitie’s neutral-host solution for 170,000 fans

Epsiode 5 of the STADIUM TECH REPORT PODCAST is off and running, in which hosts Phil Harvey and Paul Kapustka examine why the Kentucky Derby doesn’t need Wi-Fi but instead uses DAS to serve the wireless needs of its 170,000 race-day guests. With more than 12 terabytes of data used on race day and 20-plus TB used over the weekend, the Derby DAS was a winner… listen in and find out why!

SUBSCRIBE TO THE PODCAST:

Here is the link to the podcast on iTunes!