The MSR Interview: San Francisco Giants CIO Bill Schlough

AT&T Park CIO Bill Schlough shows off his World Series bling.

AT&T Park CIO Bill Schlough shows off his World Series bling.

Who better to talk about stadium Wi-Fi than the guy who was there when it all started? Our guest for our first MSR Interview (part of our Stadium Tech Report Podcast series) is San Francisco Giants senior vice president and chief information officer Bill Schlough, who goes old-school talking about stadium Wi-Fi back in 2004… and brings it to the current day with stats from the most recent season at AT&T Park. Plus, his thoughts on game-day apps and why great connectivity is the real winner. Listen in now!

Hear Bill talk about:

— New Wi-Fi records set… during the Warriors’ playoff run

— Why going under-seat with Wi-Fi was a necessary thing to do

— How the Giants are experimenting with virtual reality

— Why he thinks great connectivity matters most (even more than stadium-app features)

Some story links that offer some history about AT&T Park’s networks from MSR:

S.F. Giants add more Wi-Fi, ‘virtual reality experience’ to AT&T Park for 2016 season

SF Giants fans used 78.2 TB of Wi-Fi data at AT&T Park during 2015 season

Stadium Tech Report: World Series set new wireless records at AT&T Park

Stadium Tech Report: San Francisco’s AT&T Park lives up to its wireless reputation

Giants: NLCS stadium Wi-Fi usage at AT&T Park quadrupled since 2012

SUBSCRIBE TO THE PODCAST:

Here is the link to the podcast on iTunes!

Stadium Tech Report: Carolina Panthers take ownership of DAS, Wi-Fi at Bank of America Stadium

James Hammond, director  of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

James Hammond, director of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

“The fan is the most valuable member of our team,” Jerry Richardson, owner of the Carolina Panthers, is fond of saying.

And it’s become the virtual mission statement for the Charlotte, N.C.-based National Football League franchise. So even though its home field, the Bank of America Stadium, was built relatively recently (1996), technology has come a long way in two decades. And as the Panthers began a four-phase renovation in 2014, they did it with fans’ MVP status in mind, according to James Hammond, director of IT for the Panthers. “It was time for some changes,” he said.

While Carolina was among the first NFL stadiums to install fan-facing Wi-Fi and enhanced cellular networks, the previous DAS and Wi-Fi systems weren’t keeping up with demand and that was starting to adversely impact the Panthers fan experience, Hammond said.

“We chose to perform a rip-and-replace on both DAS and Wi-Fi and take ownership in-house,” Hammond explained. Because the Panthers own and operate BofA Stadium, making those moves was a lot easier than if they were tenants.

Time for an upgrade

Editor’s note: This profile is part of our latest STADIUM TECHNOLOGY REPORT, which includes more stadium profiles as well as looks at Wi-Fi at the Mall of America, and analytics software being used by the Cleveland Browns. DOWNLOAD YOUR FREE COPY of the report today!

The first “fan-centric improvements,” as Hammond called them, came in 2014 in the form of escalators, video big boards and a distributed audio system. As part of the second phase of upgrades, the Panthers then used the 2015 offseason to renovate the club-level suites and tore out the old DAS system while they were at it. And after a careful evaluation of different DAS solutions, they shortlisted two vendors: CommScope and Corning.

CommScope ultimately got the nod; the Panthers then had to decide between the vendor’s ION-B and ION-U DAS systems. “We went with the ION-U, which was quite new and cutting edge at that point, since it had NEMA-rated remotes,” Hammond said. Other systems lacked that kind of weatherproofing and would require additional enclosures – and expense.

CommScope's ION-U powers the new DAS at Bank of America Stadium.

CommScope’s ION-U powers the new DAS at Bank of America Stadium.

“We started over with all new fiber and coax. We did the decommissioning and construction in 90 days, which was pretty quick for a ground-up project,” he said. Beam Wireless Inc. did the design, integration and optimization and is handling the ongoing maintenance of the DAS system; Optical Telecom installed the DAS gear. BofA Stadium now has 256 DAS remotes and more than 600 DAS indoor and outdoor antennas.

AT&T, Verizon and Sprint are the participating DAS carriers; T-Mobile is weighing whether to join the mix during the 2017 off-season.

The Panthers have divided BofA Stadium into 48 DAS zones: 16 zones for the upper bowl, 16 in the lower bowl, and another 16 for concourses, suites, clubs, and offices. Not all zones are used exclusively; carriers choose simulcast patterns that place multiple zones into sectors, and can change them as capacity requirements dictate, Hammond told Mobile Sports Report.

“With some minor design changes to the interior areas, we can accommodate nearly 70 zones,” he explained. “At present the most sectors in use by a carrier is 32. This means the carrier simulcasts across a mix of our 48 zones in order to match them up to 32 carrier sectors.”

Once the new DAS was built and the first couple of events were analyzed, carriers began asking for more frequencies and additional DAS sectors to continue meeting ever-growing demand. In response to the new carrier requests, the first round of DAS upgrades were implemented in the spring of 2016, Hammond said. During the 2015 season, DAS bandwidth was running around 2 GB during games. Hammond said, “With these latest DAS upgrades, we expect the bandwidth numbers to be even higher.”

A DAS remote in a NEMA-rated enclosure.

A DAS remote in a NEMA-rated enclosure.

The impact of the new DAS system was felt immediately upon its debut in July 2015. “It was a much better experience for fans who noticed the improved cellular experience,” Hammond said. Another unexpected benefit: The upgraded DAS helped mitigate bottlenecks with the old Wi-Fi system, which Hammond characterized as “under-designed.”

Going under seat for Wi-Fi upgrade

Unfortunately, there wasn’t time to address any Wi-Fi upgrades before the 2015 football season began, but the Panthers issued an RFP for new Wi-Fi in August 2015 in preparation for Phase 3 renovations that would also include security upgrades and renovations to the upper concourse.

Interested vendors needed to ensure high bandwidth rates as well as high take-rates that allowed three different ISPs (Time Warner Cable/Charter, Level 3 Communications and Windstream) to deliver in excess of 10 GB, though Hammond said they’re starting at a 7-GB threshold.

The Wi-Fi award went to Aruba, now HP Enterprise, in December 2015, and construction began in January 2016 after the last postseason game, when the Panthers beat the Arizona Cardinals to win the NFC championship and a trip to Super Bowl 50.

Similar to Levi’s Stadium and the Dodgers Stadium, the Panthers chose underseat AP enclosures; BofA Stadium sports 770 AP enclosures in the upper and lower bowls out of a total of 1,225 APs, all to ensure maximum coverage and minimal dead spots. The Panthers selected AmpThink to do the Wi-Fi integration and construction; the turnkey contractor also designed and fabricated a custom enclosure for the APs.

Indoor access point inside the stadium.

Indoor access point inside the stadium.

One other innovation in the Panthers’ Wi-Fi installation is that the underseat enclosure is mounted to the riser — the vertical part of the step — but looks like it’s on the tread, the horizontal part, which is intended as a waterproofing measure. “The riser is easier to seal and isn’t affected by pressure washers, which you’re doing constantly with an outdoor stadium,” Hammond said. “And by running pipe through the riser, you don’t have gravity working against you,” which helps keep out water, he explained.

Panthers fans access the stadium Wi-Fi through a portal page after accepting the team’s terms and conditions. From there, they are whitelisted and can automatically join the Wi-Fi network for the rest of the season. Hammond said a fan’s email is requested but not required by the portal page, and there’s a small incentive offered to encourage fans’ email subscriptions.

The new Wi-Fi system got a workout with a soccer game at BofA Stadium at the end of July 2016, then with a Panthers’ Fan Fest the following week. “All the indicators were good, and fan feedback about the system was excellent,” Hammond said. But he cautioned that the two events were not “full bowl” events with smaller attendance numbers (~50,000) than a regular season football game (75,000+). “We will continue to optimize and tune settings as we learn more during events with higher attendance,” Hammond said.

Total budget so far for the technology upgrades totals about $16 million; the DAS build-out was just under $10 million; Wi-Fi was a little more than $6 million, which included additional wired infrastructure, according to the team.

Beacons coming next

And the Panthers aren’t done making technology improvements to their stadium. Phase 4 looks to add Bluetooth beacons and do some refinement of the Panthers app. “My goal during the upcoming season is to look at options for location-aware services,” Hammond said. Some APs have beacons built in; other may need to be added to get the granularity the Panthers want for location awareness.

Hammond also wants to give fans more things to do with the Panthers app and also optimize it for push notifications, even with something as basic and useful as restroom and concessions location information. “As we learn more about fans individually, we can direct them to things of particular interest to them,” he added.

“So far, we are very pleased with the performance of the Wi-Fi and DAS systems,” Hammond said, noting the Panthers will continue to tune frequencies, add zones and increase bandwidth where needed. It’s the sort of attention that smart sporting franchises pay to their most valued team members.

car1

Texas A&M sees 8.2 TB of Wi-Fi + DAS traffic at Kyle Field for Tennessee game

Texas A&M student recording the halftime show at a 2015 season game. Photo: Paul Kapustka, MSR

Texas A&M student recording the halftime show at a 2015 season game. Photo: Paul Kapustka, MSR

The wireless networks at Texas A&M’s Kyle Field continue to prove their heralded capability, as the school reported seeing 8.2 total terabytes of wireless traffic on its stadium Wi-Fi and DAS networks during the Aggies’ exciting 45-38 double-overtime victory over Tennessee on Oct. 8. Of that number, approximately 4.4 TB was recorded on the stadium’s Wi-Fi network, with another 3.8 TB recorded on the cellular DAS, which carries traffic from AT&T and Verizon Wireless.

Perhaps even more impressive than the total tonnage numbers are the number of unique connections reported during the four-hour span during which Texas A&M recorded statistics: According to a press release from A&M, just more than 52,000 fans connected to the Wi-Fi network at some point, 49 percent out of a total of 106,248 attending the Oct. 8 game. The Wi-Fi network also saw nearly 32,000 fans connected to the Wi-Fi network at the same time, with both numbers surpassing Kyle Field’s previous high totals for both metrics.

Last year, the new Wi-Fi network at Kyle Field saw 5.7 TB of Wi-Fi use for a home game against Alabama, with 37,823 unique clients recorded during pre-game and game time, as well as a 26,318 peak concurrent user count.

No upgrades needed

We are still hoping to circle back with A&M folks to hear more about the second year of Kyle Field’s networks, but sources close to the operation said that so far, neither the Wi-Fi network nor the DAS network has needed any system upgrades since the networks went live last year.

On the DAS side of things, AT&T reported that it saw 2.2 TB of DAS traffic on its Kyle Field networks for Oct. 8; though it’s not confirmed by Verizon, A&M reported a total of 3.8 TB of DAS usage, and since Sprint and T-Mobile are not yet on the DAS, it seems safe to say there was 1.6 TB of Verizon traffic on the DAS Oct. 8. AT&T and Verizon both paid $5 million each to help build the networks at Kyle Field, which had a total price tag of just north of $20 million according to school officials.

World Series fans will have Wi-Fi + DAS at Cleveland’s Progressive Field; cellular only at Wrigley

Screen Shot 2016-10-25 at 11.00.32 AMIf you are lucky, loaded or devoted enough to have tickets to any of this year’s World Series games, you will have more connectivity choices at Cleveland’s Progressive Field than at Chicago’s Wrigley Field. Thanks to a Verizon-built Wi-Fi and a separate DAS network that both came online in 2014, the home of the AL champion Indians should be ready for the bucket-list traffic pressures that come with hosting World Series games.

When the series moves to Chicago later this week, the Wrigley cell network will need to carry the full load, as renovations that are bringing new Wi-Fi and DAS systems to the Friendly Confines are not yet in place (and may not be finished until 2018, according to the Cubs).

However, the cellular networks at Wrigley Field seem to be working just fine — according to AT&T, during the three league championship series games in Chicago, fans who are AT&T customers used 2.6 terabytes of data on the AT&T networks at the ballpark. For the three games at Dodger Stadium, AT&T said it saw a total of 2.0 TB used on its networks. Fans at Progressive Field, meanwhile, used a total of 340 GB during games 1 and 2 of the ALCS on AT&T networks. Expect that number to grow this week.

For fans attending the games in Cleveland (Game 1 is Oct. 25 and Game 2 is Oct. 26; games 6 and 7, if necessary, will also be in Cleveland) there should be plenty of local cellular capacity thanks to the beefing-up brought by carriers ahead of this summer’s Republican convention; we are also guessing that all the big wireless providers are doing their usual big-event preparations, which usually means portable cellular equipment for placement around the stadiums and in other fan areas. Anyone attending the games, send us a speedtest… so we can keep score on the networks!

Kings turn to startups Built.io, Cartogram for Golden 1 Center app development

Screenshots from new Sacramento Kings/Golden 1 Center app. Credit all photos: Paul Kapustka, MSR

Screenshots from new Sacramento Kings/Golden 1 Center app. Credit all photos: Paul Kapustka, MSR

For an arena that seeks to redefine the fan experience at a large public venue, it’s somehow fitting that the Sacramento Kings have turned to startups for two of the core parts of the new stadium app for their new home, Golden 1 Center.

With a main app platform developed by stadium-newcomer Built.io and mapping features provided by Seattle-based startup Cartogram, the Kings have produced a stadium app they hope will help give fans a better game-day experience, by reducing “pain points” in areas like transit, parking and ticketing, while also providing features like concession-ordering and delivery via the app. There are also plans for a wide range of content including replays and stats, and the ability to purchase seat upgrades. Though the new Golden 1 Center is already open, the stadium-app features will likely not be fully tested until the Kings’ first regular-season opener, scheduled for Oct. 27 against the San Antonio Spurs.

And while the app is a centerpiece of the Kings’ very public campaign to tout Golden 1 Center as one of the most technologically advanced stadiums every built, to even begin to reach that title the Kings must first conquer the biggest hurdle most stadium apps have, that being just getting fans to download and use the app in the first place. Ryan Montoya, chief technical officer for the Kings, thinks the app’s focus on services will help drive adoption, especially for a brand-new place where all fans will need help finding there way there, and around once inside.

Flexibility to add more services quickly

“We wanted to build features [in the app] that were actually useful to fans,” said Montoya in a phone interview. “We wanted to push the boundaries, to remove friction and help the arena become more intuitive.”

Directions and parking / transit options

Directions and parking / transit options

Though most of the leading team- and stadium-app platforms, such as those from YinzCam or VenueNext, provide the ability to integrate third-party features, the Kings said they were seeking a more agile app platform, one that Montoya said “would provide us flexibility in real time.” For that core, the Kings turned to a company called Built.io, a San Francisco firm with experience in “assisting large organizations with digital transformations,” but no public claims to team- or venue-app developments.

Neha Sampat, co-founder and CEO of Built.io, said that since its founding in 2007, her company has a lot of experience in helping enterprise company move to cloud-based operations, moves that include support for content management and mobile-device usage.

“What the Kings are trying to do is a large-scale enterprise use case,” said Sampat in a phone interview. “There are a lot of big-data analytics and so much personalization that is dependent on data.”

Sampat said Built.io’s model of a “back end as a service” and its ability to quickly connect other programs’ APIs make it a perfect partner for the Kings, who want to expose multiple services and features through the combined Golden 1 Center/Kings app.

“When Vivek [Ranadive, Kings owner] said 2 years ago the ‘arena will check into you,’ we got excited because that was our vision,” Sampat said. Sampat compared the Kings’ ideas to services Built.io has helped deliver at other large public spaces, like connected cities and convention centers.

Ticketmaster integration

Ticketmaster integration

“The use cases are similar,” Sampat said. “If you’re at a conference, something can happen [to the event app] due to your location.”

Using the Built.io backend, the Kings have been able to add several well-known existing services to the new app, including Appetize for food ordering, and Parking Panda for parking services. There is also integration with Ticketmaster and some new “smart kiosks” from Skidata which Montoya said should be able to allow for up to 1,000 fans per hour to get into the arena, as opposed to limits of around 300 fans per hour via manual ticket-scanning techniques.

The important thing, Montoya said, is to have a backend system that allows for continuous additions to the application, a so-called “bus” theory “to allow us to integrate services seamlessly.”

Maps the way people want them

While the app certainly is pushing the envelope when it comes to functionality, perhaps the most important part of it at the start will be its wayfinding capabilities. With its intimate downtown setting — one placed right in the city center, and not surrounded by parking lots — Golden 1 Center will likely present a challenge for fans in just trying to get to and get inside the venue. Add into that the challenge for any fan simply to find their way around a brand-new facility, and you have an increased need for directions and assistance.

To help fans find their way, the Kings have built features into the app that include map-directed wayfinding to transit and to parking lots, and then into the building and to their seats. To power the wayfinding, the Kings selected Cartogram, a Seattle-based startup that uses Google Maps as its base to power maps that allow for directed searches both inside and outside buildings.

Indoor seating map

Indoor seating map

In a recent phone interview, Cartogram CEO Will Clausen said the Kings had been looking for a mapping solution that could integrate both exterior and interior views, mainly due to the aforementioned challenges of getting directions to a new stadium located in a downtown core. Having a Cartogram employee who worked previously with a Kings employee helped get Cartogram in the door, Clausen said, and now its system drives not only the wayfinding features in the app but other amenities like the ability to see how long lines are at concession stands or bathrooms.

Working closely both with Built.io as well as with the Kings’ wireless networking team, Clausen said the Cartogram software uses information both from Bluetooth beacons as well as from Wi-Fi access points to determine line lengths and wait times. While Bluetooth is great for providing granular location information for a single device, Clausen noted that while not all devices may have the app downloaded, almost all devices in a venue have Wi-Fi active, allowing the Kings’ system to detect density in local areas based on the number of devices its Wi-Fi network can see.

And while other stadium systems like the San Francisco 49ers’ Levi’s Stadium app have wayfinding features with live interactive maps, those maps are different than Google Maps, an app Clausen said most people know and like. Using Google Maps as its base, Clausen said, “gives end users the experience they’re used to.”

In the end, such familiarity may help the Kings’ Golden 1 Center app become more widely used than other stadium apps, which regularly trail social-media apps and email apps for most-used apps in any stadium game-day use measurements. And while relying on startups may be the team-app version of an NBA team relying on rookie players, the Kings’ Montoya is confident the new app will help the Kings deliver on their vision of an advanced fan experience.

“I think we’ve seen a transformation of what a [stadium] app is supposed to be,” Montoya said. “It’s a real evolution.”

The all-important cowbell sound feature for the app.

The all-important cowbell sound feature for the app.

For concerts -- who needs a lighter when the app can provide?

For concerts — who needs a lighter when the app can provide?

IBM lands tech deal for new L.A. soccer stadium

Artist rendering of Banc of California Stadium, slated to open in 2018. Credit: LAFC

Artist rendering of Banc of California Stadium, slated to open in 2018. Credit: LAFC

IBM’s growing sports-venue technology business landed its first soccer-specific client, with the announcement that IBM will lead all technology deployments at the Los Angeles Football Club’s Banc of California Stadium, a venue scheduled to open in 2018.

Like it has at Texas A&M’s Kyle Field and the Atlanta Falcons’ new Mercedes-Benz Stadium, IBM will act as a lead general contractor of sorts for technology at the under-construction 22,000-seat Banc of California Stadium, responsible for picking vendors and leading deployment for such features as Wi-Fi and cellular networks, digital signage, and as yet-to-be determined fan experience applications and services.

The MLS expansion team LAFC, which will begin play in 2018, has a star-studded ownership group that includes names like former pro athletes Magic Johnson and Nomar Garciaparra, actor Will Ferrell and Golden State Warriors owner Peter Guber, among others. The new stadium is being built on the space once held by the Los Angeles Memorial Sports Arena, adjacent to the Los Angeles Memorial Coliseum. According to the team website the stadium will have clear-plastic shields overhead to reduce sun glare and reflect heat, made of the same ETFE plastic used in the clear window sides of the Minnesota Vikings’ U.S. Bank Stadium.

Though IBM was not yet ready to name specific vendors or any specific apps or services that will be available at the new stadium, it did say that its contract with LAFC shows that IBM’s strategy of having a single integrator in charge of all technology deployments isn’t just for huge stadiums or big new projects like Atlanta’s new venue.

Construction-cam shot at home of future Banc of California Stadium. Credit: LAFC

Construction-cam shot at home of future Banc of California Stadium. Credit: LAFC

“As [stadium network] technology evolves, it just becomes more complex, whether it’s a small venue or a large one,” said Jim Rushton, global leader for the sports & entertainment practice for IBM. “Our methodology is the same.”

Just like a lead contractor for plumbing or electricity, Rushton said that IBM’s size and purchasing power gives it an edge that individual venues might not have. Rushton also said that IBM’s ability to oversee all parts of a venue’s technology offerings — from wireless infrastructure to network security and application development — and its ability to integrate technologies from firms other than IBM — can help venues plan more strategically and put together a more complete venue-technology plan than they might be able to do on their own.

Rushton said that IBM’s sports venue practice, which was formally announced a year ago, will be naming more projects underway soon, including some in Europe. IBM is rumored to be the lead technology integrator for the stadium renovation that will be taking place at Notre Dame after this football season, but there has been no announcement of that yet.

Screen Shot 2016-10-13 at 10.57.42 AM