Stadium Tech Report: Network finishes season strong at Niners’ Levi’s Stadium

Arriving at Levi's Stadium for last 2014 season game

Arriving at Levi’s Stadium for last 2014 season game

While the football season didn’t turn out like 49ers fans wanted, the wireless network at the team’s new Levi’s Stadium closed out the year with strong performances to complete a largely glitch-free stretch of events at what is one of the world’s most technologically advanced stadiums.

With more than 2 Terabytes of data used by fans at each of the last two home games for the San Francisco 49ers, the Wi-Fi and DAS networks at Levi’s Stadium closed out a season of superb connectivity that eventually allowed the team to not just offer in-seat food and beverage delivery, but also ordering and delivery of merchandise like hats and T-shirts, an option that was available for the Dec. 20 game against the San Diego Chargers and the Dec. 28 closer vs. the Arizona Cardinals.

According to the Levi’s Stadium network crew, the Wi-Fi network carried 2.34 TB of data for the Chargers game and another 2.11 TB for the Cardinals game, with 20,096 fans using the network on Dec. 20 and 20,164 on Wi-Fi on Dec. 28. Peak concurrent user numbers were 13,700 for the Chargers game, and 14,400 for the season closer.

All season long, our speed tests in various parts of the stadium showed strong signals for both the Wi-Fi as well as the distrbuted antenna system (DAS) enhanced cellular network. At the final game it was no different; we found Wi-Fi download speeds of 16 Mbps on the rooftop deck, 25 Mbps in a suite and a scorching 39 Mbps in the Levi’s 501 Club seats (no doubt in part because there was a Wi-Fi antenna under the seat next to us).

Both Verizon and AT&T 4G LTE services also worked well, consistently scoring download speeds in the 4-6 Mbps range in most places and much higher in others. In short, we didn’t find any flaws in the network coverage in five games of walking all around the stadium, testing everywhere we went.

CalTrain to VTA a smooth ride

Caltrain crown en route to Arizona game

Caltrain crown en route to Arizona game

At the final game, Mobile Sports Report (me) tested out the full public-transit method of getting to the game, starting from San Mateo on CalTrain at 10:51 a.m. The parking lot at the station was almost completely empty, and free since it was Sunday; it’s possible that crowds were lighter since the Niners had been eliminated from postseason play, but nevertheless the ride to Mountain View went without hitch, a good sign for next year when many fans in town for Super Bowl 50 will no doubt be using CalTrain to get from San Francisco to Levi’s.

At the Mountain View CalTrain/VTA station operations were at their best I’ve seen, with more neon-vested VTA helpers offering clear instructions on why you might want to take an express bus instead of the light rail. Insider tip: If the express bus is available, take it, because in our testing it arrived at Levi’s in about half the time as the train trip (~20 minutes as opposed to almost 40 minutes for the light rail option).

Express bus option at Mountain View

Express bus option at Mountain View

The only thing that still remains to be ironed out is the fare confusion in switching from CalTrain to VTA, which are two different operators. On CalTrain there was advertising for a $6 “combo ticket” that would let you ride VTA and could be purchased at the same time you bought your CalTrain pass. But an online single-day ticket purchased via the VTA app was only $4.50, so it’s not clear why you would buy the CalTrain pass. Especially for the Super Bowl, it’d help fans if there was one price and one place to buy a single “Get to Levi’s” public-transit ticket.

Food order arrives as promised

Another thing I tried at the season closer was the in-seat food ordering feature on the Levi’s Stadium app. Sitting in the Levi’s Club section seats on the third level, I consulted the app to order a cold beer and a warm pretzel, which the app said could be delivered in 10 minutes.

Food runner bringing me my cold beer and warm pretzel

Food runner bringing me my cold beer and warm pretzel

After entering credit-card information into the app and hitting the order button the app updated itself with timely notices about the order being prepared, and that it was on its way. I found that information to be very assuring, a sign that things were indeed happening; there was even a big number associated with my order that appeared, apparently to make it easier for the food runner to confirm the order.

The order arrived exactly in 10 minutes’ time, as predicted by the app — it also arrived in a lot of extra packaging, a separate plastic bag for the steel bottle of beer and a paper sack holding a cellophane wrapper-encircled pretzel. Since there is no way to add a gratuity in the app, I gave the runner a cash tip, which seemed appropriate even though there is a $5 charge added to the order cost for the delivery service. I have to admit it felt a little weird to have someone bring me my food and drink but in the time it took to order and deliver I sat in my seat and watched the Niners’ game-winning TD drive so it’s clearly a fan-friendly option.

Video replays work well, for small amount of viewers

Another part of the Levi’s Stadium technology that was at peak performance by year’s end was the app’s instant replay feature. Though it started slowly and had some hiccups early on, by the final game instant replays were appearing in the app even before the next play had concluded (see our feature on how the VenueNext team gets the replays to the app so quickly).

While it’s an impressive addition to the in-game experience, the replays are a feature that only a small amount of fans are watching. According to the team network stats there were only 1,253 unique users watching replays on Dec. 20, and 1,019 during the Dec. 28 game. Total replays viewed for the Chargers game were 6,285, while 4.310 replays were watched during the season closer.

Why aren’t the replays catching on? Our main guess is that the Levi’s Stadium big screens are so clear and so quick to show replays (they also show live action as it’s happening), fans don’t find it necessary to use their phones to watch replays. It’s also possible that many fans in the stadium who are using the network aren’t using the Levi’s Stadium app. Indeed, according to the team network stats, the team app hasn’t yet cracked the top-four apps being used at any of the games this season; for the Dec. 20 game the top apps being used on the network were Amazon cloud drive, Facebook, Google APIs (probably Gmail) and Apple; for Dec. 28 the list was Amazon, Google, Facebook, then Apple.

We’ll try to get some season-long stats to share for both the network and the app features, but our quick conclusion after five live-game visits to Levi’s Stadium this year is that the wireless network and the app both pretty much lived up to their pre-season billing and hype, delivering a wireless game-day experience that is the standard other large public facilities will be measured against, going forward. More photos from our last visit below.

The Microsoft Surface/sideline Wi-Fi unit

The Microsoft Surface/sideline Wi-Fi unit

close-up of cable connection

close-up of cable connection

Niners' Flickr promotion on scoreboard -- very popular

Niners’ Flickr promotion on scoreboard — very popular

Sideline Surface tablets for Niners players and coaches

Sideline Surface tablets for Niners players and coaches

Colin Kaepernick exchanges his radio helmet for his flat-brimmed hat after throwing a TD pass

Colin Kaepernick exchanges his radio helmet for his flat-brimmed hat after throwing a TD pass

View from the Levi's skydeck out over Santa Clara

View from the Levi’s skydeck out over Santa Clara

If you throw a rooftop party, the cheerleaders might visit

If you throw a rooftop party, the cheerleaders might visit

View from the Levi's 501 Club section seats

View from the Levi’s 501 Club section seats

Wi-Fin antenna right under the seat next to me (probably why my speedtest was 40+ Mbps)

Wi-Fin antenna right under the seat next to me (probably why my speedtest was 40+ Mbps)

In-stadium signing help to get fans to light rail

In-stadium signing help to get fans to light rail

End of game view from skydeck

End of game view from skydeck

A final toast to the season at the BNY Club

A final toast to the season at the BNY Club

VTA train line. Only took 15 minutes from here to get on bus.

VTA train line. Only took 15 minutes from here to get on bus.

Caltrain platform at Mountain View. Extra trains helped make ride home easy

Caltrain platform at Mountain View. Extra trains helped make ride home easy

Stadium Tech Report: How VenueNext generates replays for Levi’s Stadium app

replay2During the final home game for the San Francisco 49ers this season, I was somewhat amazed to see a replay appear on the Levi’s Stadium app even before the next play had concluded. Clearly, the VenueNext team behind the app had progressed significantly from the early season, when the app’s replay function struggled somewhat.

During the Dec. 28 game against the Arizona Cardinals, the VenueNext team invited Mobile Sports Report to Levi’s Stadium for a behind-the-scenes look at how the replays are generated; here’s a quick take on how the video moves from playing field cameras to fans’ phones in a matter of seconds.

Step 1: Capturing what the cameras see

How do you make sure you have replays ready? You start by basically capturing all the action from all the cameras in the building. Those views are shuttled back to the main Levi’s Stadium video room, where 15 people have the somewhat enviable job of watching views from all the live cameras in the stadium. (We still haven’t heard a response to our offer of bringing a cooler in exchange for a spare seat next season.)

Inside the video room at Levi's Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Inside the video room at Levi’s Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

According to VenueNext CEO John Paul, who led our tour at Levi’s, VenueNext has its own servers to cache video — here is a look at the Elemental gear used to store the video for the app’s replay function. Paul said that so far, VenueNext has only needed a little more than half of the capacity of the four servers pictured.

Elemental gear in data room next to video room at Levi's Stadium

Elemental gear in data room next to video room at Levi’s Stadium

Step 2: Labeling and thumbnailing the replay

While most of this kind of gear is not new to broadcast operations, the interesting part from an app perspective is the next layer, the choosing and human editing. One of my bigger questions was about how you exactly get a replay set up, labeled with a headline and a thumbnail, and into the app within seconds? The answer is: You write another app, and have two Android-based tablets with football-savvy operators in the press box so they can see the plays live and quickly attach the appropriate info to the replay clips.

In the two pictures below, you can see an over-the-shoulder view of the two tablet apps; the first one is a sort of play-by-play generator, which the operator uses to label the next replay clip as either a run, pass, punt, incomplete, etc., so that a fan looking at the app can quickly figure out what the play might be (“45-yard TD pass,” or something like that). On the second tablet app, another operator sees instant thumbnails gleaned from the video room from all the different camera angles, and when the play is over, that operator picks a thumbnail with a click and the replay is on its way to the app.

First replay tablet app, which adds info about the play

First replay tablet app, which adds info about the play

Second replay tablet app, which adds a thumbnail to the replay

Second replay tablet app, which adds a thumbnail to the replay

Since we only had a few minutes to watch the replay team in action (and since Paul was simultaneously hosting a group of representatives from a prospective future sports-team client), I didn’t get to ask detailed questions about how this all works but it does give you some idea. What’s impressive about the Levi’s Stadium app is that it offers four different camera angles of the same play, as well as a multi-cam view that shows all different angles at once. Though I’m not a fan of the all-at-once option I did find it enjoyable to watch a reply from multiple different perspectives, like watching a pass being caught from both a sideline and end-zone view.

Step 3: Educating fans how to use replay

Initially one of the most-hyped features of the planned stadium app, the replay function has seen only limited use this season, making it somewhat of a disappointment to Niners CEO Jed York. From our perspective, there were two big things holding back extended use of the replay function: The first was the ever-changing appearance of the app itself, which felt like a beta project for most of the season, with a different interface almost every single game; the second thing holding people back from using the app to view replays are the two excellent humungous video boards at Levi’s, which not only show live game action but also quickly show replays, sometimes from those multiple angle views as well.

There might even be a third thing holding fans back from trying out the replay function — that would be just getting fans to use the stadium app in the first place. While usage of the cellular and Wi-Fi networks started at record levels and remained strong from game to game, most fans at Levi’s Stadium were using their devices for other apps, like Twitter, Instagram, Facebook and good old email. As the season progressed, the team started promoting the app and the network more aggressively, adding more-frequent messages to the main video boards and dressing its stadium tech support crew (the “NiNerds”) in neon vests for easier recognition.

What’s next for next season? My guess is that with a year under their belt, the Niners and VenueNext will have a more-aggressive campaign in 2015 to steer more fans to the replay function, which is truly a wonderful enhancement to the game-day experience.

Niners add in-seat merchandise delivery, transit info to Levi’s Stadium app

Screen shot of new merchandise ordering feature in Levi's Stadium app. Credit: VenueNext

Screen shot of new merchandise ordering feature in Levi’s Stadium app. Credit: VenueNext

As previously reported by Mobile Sports Report, the San Francisco 49ers have added in-seat merchandise ordering and public transportation information to the Levi’s Stadium app, ahead of Saturday night’s game between the Niners and the San Diego Chargers.

While the new features were hinted at during a recent technology summit at the stadium, application developer VenueNext made the upgrades official today. Here are the official new features the app will have by Saturday:

— Now Levi’s Stadium app users can order jerseys, novelties and other 49ers apparel to their seat, anywhere in the stadium, paying with credit card or Apple Pay.
— Faithful49 loyalists can redeem yards to purchase food, beverage and merchandise right in the app.
— Users can also get up-to-the-minute transit information for their ride home with TransitScreen (which includes every possible mode of public transport in real time, viewable on one screen)

Photo of directions function in Levi's Stadium app. Credit: MSR

Photo of directions function in Levi’s Stadium app. Credit: MSR

Fans should upgrade their Levi’s Stadium app before coming to the next game, since only the latest version of the app will support the new features. According to the VenueNext press release, the Levi’s app also offers in-stadium access to the NFL Network’s RedZone channel, something which we hadn’t seen in several trips to Levi’s this season.

We’re interested in hearing from any fans who are at the next games at Levi’s about whether or not the transit feature is helpful; while we think it could be a big timesaver (so far at all our visits we regularly see people getting off the light rail train wondering where to go next) we wonder if enough people are aware of all the functionality in the app. Even Niners CEO Jed York has been somewhat surprised at the light takeup of things like the app’s ability to show multiple camera-angle replays.

Niners’ CEO gives Levi’s Stadium operations a ‘B’ grade

Jed York, Niners CEO, speaking at tech summit at Levi's Stadium. Credit all photos: Paul Kapustka, MSR

Jed York, Niners CEO, speaking at tech summit at Levi’s Stadium. Credit all photos: Paul Kapustka, MSR

San Francisco 49ers CEO Jed York, who caused a stir last week by apologizing for his team’s play on Twitter, on Thursday gave the operation of his club’s new Levi’s Stadium an above-average grade of “B” for the first four months of its inaugural season.

Speaking at a technology-focused fan-experience and innovation summit held Thursday at the stadium’s “501” club, York said “I’d give us a ‘B’ on our execution [at Levi’s Stadium] this year,” citing parking issues and foot-traffic flow into the stadium as problems not yet fully solved for the 68,500-seat facility in Santa Clara, Calif.

Given the new stadium’s complicated location — in the middle of a busy corporate-headquarters area and right next to the Great America theme park — it was probably somewhat of a given that there would be parking struggles the first season, as fans, police, traffic directors and stadium workers all figured out how to make the dance work. Though some progress has been made during the season, York said that “parking and just getting people here” remain the biggest issue he sees at Levi’s Stadium.

While the Niners have tried to use technology to solve the problems of getting fans inside (with app-based parking maps and wayfinding), York said that despite plans to funnel fans through the correct gates to get more quickly to their seats, many fans still just head for the gate that’s closest to their parking or train arrival spot, which has sometimes led to big backups at the check-in lines.

On the networking side of things, York said all seemed to be going well with the stadium’s Wi-Fi and cellular networks, which have performed well in a series of ad hoc tests conducted by MSR in visits this season.

“We haven’t had any [network] glitches, but we’ve been doing a lot of tweaking,” York said. The Levi’s Stadium app has been the center of most of that tweaking, with continual upgrades to add features and fix problems like an Android bug that surfaced in a mid-November revision. York said that the Niners have been careful to make sure the human engineering behind the technology is solid before launching new things, like having enough runners to support the feature that allows food to be ordered and delivered to seats.

Levi's Stadium ready for the Pac-12 championship game

Levi’s Stadium ready for the Pac-12 championship game

And though it hasn’t been officially confirmed yet, York said the team might debut a planned feature of having team merchandise available for purchase and delivery to seats at the next home game, Dec. 20 vs. San Diego. John Paul, CEO of Levi’s Stadium app developer VenueNext, also spoke at the summit Thursday and said in an interview that the Dec. 20 game might also see the debut of a new feature that adds in mass transit and Uber wait times. (A good idea for fans going to the stadium is to check on game day morning for any updates to the app.)

On the app side, York said one surprise was the fairly low uptake of fans using the app’s instant replay features. After fans watched 7,800 replays during the regular-season home opener (the first game replays were available in the app), usage has gone down, with less than 4,000 replays watched during the last home game against Seattle.

“We thought that mobile replays would be absolutely a home run feature, but it hasn’t got that much traction,” said York. The culprit, he said, might be the twin HD displays above each end zone at Levi’s Stadium, which are somewhat stunning in their clarity. “We do have great video boards,” York said.

In his talk York also thought out loud about the possibility of using wearable technology to both better help prevent player injuries as well as being able to provide more rich detail for fans, like how hard a hit was on the field. But he also stressed that technology at Levi’s Stadium was not meant to be used for technology’s sake, but instead to improve the experience of being at the game. And improvement on the stadium staff’s execution, York said, will go a long way to making it all work together.

“When we get to an A, or A+, people will really be blown away,” York said.

Levi's Stadium at twilight

Levi’s Stadium at twilight

Levi’s Stadium update: Wi-Fi traffic steady at 2.3+ TB per game, Amazon and Google lead apps use

Scoreboard promo for the Levi's Wi-Fi network

Scoreboard promo for the Levi’s Wi-Fi network

Usage of the Wi-Fi network at the San Francisco 49ers’ new Levi’s Stadium continues to hold steady at 2.3-plus Terabytes of data offloaded per game, according to the most recent statistics provided by the Niners’ network staff.

Though he’s no longer the team’s vice president of technology, former Levi’s network guru Dan Williams remains as a game-day consultant to the team, and he shared some recent network statistics with MSR, including some app usage marks that show Amazon Cloud Drive and Google APIs being among the top two applications being used over the Levi’s Stadium Wi-Fi network.

According to Williams’ numbers, for the Niners’ Nov. 23 day home game against Washington, the Levi’s Stadium Wi-Fi network had 22,095 unique users, 35 percent of the total attendance; the peak number of concurrent users was 14,700, reached at 2:50 p.m. (near halftime), and the total data used was 2.31 TB, between the hours of 9 a.m. and 5 p.m.

During the Thanksgiving day game against the Seahawks, the network had 23,371 users (36 percent of attendees) who used 2.38 TB of data, with a peak of 16,800 concurrent users at 7:15 p.m.

Video-replay statistics from the Levi’s Stadium app apparently do better when there are more home-team highlights to watch. During the victory over Washington, 1,074 fans watched 4,885 replays, with the 30-yard TD pass from Colin Kaepernick to Anquan Boldin racking up 953 total views from 398 unique users — meaning many users are watching the same replay more than once (maybe showing it to people sitting around them?). For the Seahawks game, 1,407 fans watched 3,875 replays, with a Kaepernick pass to Michael Crabtree accounting for the most views, 487 total from 180 unique users.

The top 4 apps used by fans on the network varied a bit from the two close games, but according to Williams Amazon Cloud Drive was the top app for both recent games. For the Seahawks game, the next three top apps were Google APIs, Facebook and SnapChat, while for the Washington game the next three top apps used were stadium video, Google APIs and Apple iTunes.

Niners’ VP of technology, Dan Williams, departs for Dropbox

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him.

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him.

Dan Williams, the human face behind the highly touted wireless network at Levi’s Stadium, has left his position as vice president of technology for the San Francisco 49ers, according to both sources inside the San Francisco 49ers’ stadium network operations and Williams’ own LinkedIn profile.

Williams, who was at his post as of at least last week, is now at online storage concern Dropbox, where his LinkedIn profile has him with the title of “new guy, technical operations.” So far, there is no official statement from the 49ers about Williams’ departure, or about who would take over as head of operations for the Levi’s Stadium network.

Previously a network guru at Facebook, Williams became a the public face for the Niners’ new stadium and its heavily hyped wireless network and app strategy when Kunal Malik, the Niners’ former CTO, left the team earlier this year. The sudden departure is surprising since Williams — a professed avid Niners fan — seemed to enjoy the hands-on work of getting the Levi’s network up and running, even doing “seat calls” during games to help solve connectivity issues. Williams also took great pride in the network’s performance, sending out detailed statistics on bandwidth and app usage.

So far, there has been no public announcement of Williams’ departure. We’ll update this post when we hear more from Dan and/or the 49ers.