Stadium Tech Report: How VenueNext generates replays for Levi’s Stadium app

replay2During the final home game for the San Francisco 49ers this season, I was somewhat amazed to see a replay appear on the Levi’s Stadium app even before the next play had concluded. Clearly, the VenueNext team behind the app had progressed significantly from the early season, when the app’s replay function struggled somewhat.

During the Dec. 28 game against the Arizona Cardinals, the VenueNext team invited Mobile Sports Report to Levi’s Stadium for a behind-the-scenes look at how the replays are generated; here’s a quick take on how the video moves from playing field cameras to fans’ phones in a matter of seconds.

Step 1: Capturing what the cameras see

How do you make sure you have replays ready? You start by basically capturing all the action from all the cameras in the building. Those views are shuttled back to the main Levi’s Stadium video room, where 15 people have the somewhat enviable job of watching views from all the live cameras in the stadium. (We still haven’t heard a response to our offer of bringing a cooler in exchange for a spare seat next season.)

Inside the video room at Levi's Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Inside the video room at Levi’s Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

According to VenueNext CEO John Paul, who led our tour at Levi’s, VenueNext has its own servers to cache video — here is a look at the Elemental gear used to store the video for the app’s replay function. Paul said that so far, VenueNext has only needed a little more than half of the capacity of the four servers pictured.

Elemental gear in data room next to video room at Levi's Stadium

Elemental gear in data room next to video room at Levi’s Stadium

Step 2: Labeling and thumbnailing the replay

While most of this kind of gear is not new to broadcast operations, the interesting part from an app perspective is the next layer, the choosing and human editing. One of my bigger questions was about how you exactly get a replay set up, labeled with a headline and a thumbnail, and into the app within seconds? The answer is: You write another app, and have two Android-based tablets with football-savvy operators in the press box so they can see the plays live and quickly attach the appropriate info to the replay clips.

In the two pictures below, you can see an over-the-shoulder view of the two tablet apps; the first one is a sort of play-by-play generator, which the operator uses to label the next replay clip as either a run, pass, punt, incomplete, etc., so that a fan looking at the app can quickly figure out what the play might be (“45-yard TD pass,” or something like that). On the second tablet app, another operator sees instant thumbnails gleaned from the video room from all the different camera angles, and when the play is over, that operator picks a thumbnail with a click and the replay is on its way to the app.

First replay tablet app, which adds info about the play

First replay tablet app, which adds info about the play

Second replay tablet app, which adds a thumbnail to the replay

Second replay tablet app, which adds a thumbnail to the replay

Since we only had a few minutes to watch the replay team in action (and since Paul was simultaneously hosting a group of representatives from a prospective future sports-team client), I didn’t get to ask detailed questions about how this all works but it does give you some idea. What’s impressive about the Levi’s Stadium app is that it offers four different camera angles of the same play, as well as a multi-cam view that shows all different angles at once. Though I’m not a fan of the all-at-once option I did find it enjoyable to watch a reply from multiple different perspectives, like watching a pass being caught from both a sideline and end-zone view.

Step 3: Educating fans how to use replay

Initially one of the most-hyped features of the planned stadium app, the replay function has seen only limited use this season, making it somewhat of a disappointment to Niners CEO Jed York. From our perspective, there were two big things holding back extended use of the replay function: The first was the ever-changing appearance of the app itself, which felt like a beta project for most of the season, with a different interface almost every single game; the second thing holding people back from using the app to view replays are the two excellent humungous video boards at Levi’s, which not only show live game action but also quickly show replays, sometimes from those multiple angle views as well.

There might even be a third thing holding fans back from trying out the replay function — that would be just getting fans to use the stadium app in the first place. While usage of the cellular and Wi-Fi networks started at record levels and remained strong from game to game, most fans at Levi’s Stadium were using their devices for other apps, like Twitter, Instagram, Facebook and good old email. As the season progressed, the team started promoting the app and the network more aggressively, adding more-frequent messages to the main video boards and dressing its stadium tech support crew (the “NiNerds”) in neon vests for easier recognition.

What’s next for next season? My guess is that with a year under their belt, the Niners and VenueNext will have a more-aggressive campaign in 2015 to steer more fans to the replay function, which is truly a wonderful enhancement to the game-day experience.

Stadium Tech Report: Corning, IBM bringing fiber-based Wi-Fi and DAS to Texas A&M’s Kyle Field

Kyle Field, Texas A&M University. Credit all photos: Texas A&M

Kyle Field, Texas A&M University. Credit all photos: Texas A&M

Editor’s note: The following is an excerpt from our recent Stadium Tech Report series COLLEGE FOOTBALL ISSUE, a 40-page in-depth look at Wi-Fi and DAS deployment trends at U.S. collegiate football stadiums. You can download the full report for free, to get more stadium profiles as well as school-by-school technology deployment capsules for both the SEC and Pac-12 conferences.

When Texas A&M’s newly renovated Kyle Field opens for the 2015 football season, its outside appearance will have changed dramatically. But from a networking perspective, what’s really different is hidden on the inside – namely, an optical fiber infrastructure designed to bring a new level of performance, cost savings and future-proofing to stadium network deployments.

While the use of optical fiber instead of copper cable in large networks isn’t exactly “new” in the core telecom or enterprise networking worlds, in the still-nascent field of stadium network deployments fiber has yet to make large inroads. But the promise of fiber’s ability to deliver much higher performance and greater future-proofing at lower installation costs in stadium situations may get a very visible poster child when Texas A&M’s football facility kicks off the 2015 season with a technology infrastructure designed to be among the most comprehensive in any stadium, collegiate or professional.

With a Wi-Fi network designed to support 100,000 concurrent connections, a robust DAS network with more than 1,000 antennas, and an IPTV deployment with more than 1,000 screens, the IBM-designed network based largely on Corning’s fiber-optical systems is incredibly impressive on paper – and it has already produced some eye-popping statistics this past season, when just a part of it came online during the “Phase 1” period of the two-phase $450 million Kyle Field renovation.

The final, or Phase 2 of the renovation, just now getting underway, began with an implosion of the stadium’s west stands, with reconstruction scheduled to finish in time for the 2015 season with a new, enclosed-bowl structure that will seat 102,512 fans. And if the new network delivers as planned, those fans will be among the most-connected anywhere, with plenty of future-proofing to make sure it remains that way for the foreseeable future – thanks to fiber.

Driving on the left side of the street

What’s going to be new about Kyle Field? According to news reports some of the creature comforts being added include redesigned concession stands, so-called “Cool Zones” with air conditioning to beat the Texas heat, well-appointed luxury suites and new restrooms – including 300 percent more women’s bathrooms.

Scoreboard, Kyle Field

Scoreboard, Kyle Field

According to representatives from the school, the decision to make the new stadium a standout facility extended to its network infrastructure. “Our leadership decided that [the stadium renovation] would be leading edge,” said Matthew Almand, the IT network architect for the Texas A&M University System, the administrative entity that oversees university operations, including those at the flagship school in College Station, Texas. “There were some leaps of faith and there was a decision to be leading edge with technology as well.”

Though the Phase 1 planning had started with traditional copper cable network design for the network, Almand said a presentation by IBM and its “smarter stadium” team changed the thinking at Texas A&M.

“The IBM team came in and did a really good job of presenting the positive points of an optical network,” Almand said.

Todd Christner, now the director, wireless business development at Corning, was previously at IBM as part of the team that brought the optical idea to Texas A&M. While talking about fiber to copper-cable veterans can sometimes be “like telling people to drive on the left side of the street,” Christner said the power, scalability and flexibility of a fiber network fit in well with the ambitious Kyle Field plans.

“The primary driving force [at Texas A&M] was that they wanted to build a state of the art facility, that would rival NFL stadiums and set them apart from other college programs,” Christner said. “And they wanted the fan [network] experience to be very robust.”

With what has to be one of the largest student sections anywhere – Christner said Texas A&M has 40,000 seats set aside for students – the school knew they would need extra support for the younger fans’ heavy data use on smartphones. The school officials, he said, were also concerned about DAS performance, which in the past had been left to outside operators with less than satisfactory results. So IBM’s presentation of a better, cheaper alternative for all of the above found accepting ears.

“It was the right room for us to walk into,” Christner said.

IBM’s somewhat radical idea was that instead of having separate copper networks for Wi-Fi, DAS and IPTV, there would be a single optical network with the capacity to carry the traffic of all three. Though the pitch for better performance, far more capacity, use of less space, and cheaper costs might sound a bit too good to believe, most of it is just the combination of the simple physics advantages of using fiber over copper, which are well known in the core telecom and large-enterprise networking worlds, applied to a stadium situation.

Deploying now and for the future

Corning ONE DAS headend equipment.

Corning ONE DAS headend equipment.

Without going too deeply into the physics or technology, a simple explanation of the benefits stem from the fact that optical fiber can carry far more bandwidth than copper, at farther distances, using less power. That advantage is one reason why fiber is used extensively in core backbone networks, and has been creeping slowly closer to the user’s destination, through deployments like Verizon’s FiOS.

Why hasn’t fiber won over completely? Mainly because in single-user deployments – like to a single home or office – it is still costly to replace systems already in the ground or in the wall with fiber, and for many users fiber’s capacity can be a bit of overkill. Fiber’s main benefits come when lots of bandwidth is needed, and the scale of a project is large, since one main benefit is the elimination of a lot of internal switching gear, which takes up space and consumes lots of power.

Those reasons accurately describe the perfect bandwidth storm happening in networked stadiums these days, where demand seems to keep increasing on a daily basis. Some stadiums that were at the forefront of the wireless-networking deployment trend, like AT&T Park in San Francisco and AT&T Stadium in Arlington, Texas, have been in a near-constant state of infrastructure upgrades due to the ever-increasing needs for more bandwidth. And Isaac Nissan, product manager for Corning ONE, said new equipment like Wi-Fi access points with “smart” or multiple-input antennas are also going to help push the LAN world into more fiber on the back end.

But there’s another drawback to using fiber, which has less to do with technology and more to do with history: Installers, integrators and other hands-on networking folks in general are more comfortable with copper, which they know and have used for decades. Fiber, to many, is still a new thing, since it requires different skills and techniques for connecting and pulling
wires, as well as for managing and administering optical equipment.

“There’s definitely a learning curve for some the RF [industry] people, who have been doing coax for 20 years,” Nissan said. “Fiber is a little different.”

Texas A&M’s Almand admitted that bringing the stadium’s networking group into a new technology – fiber – was a challenge, but one with a worthy payoff.

Copper cable tray hardly filled by optical fiber

Copper cable tray hardly filled by optical fiber

“There’s definitely been a gear-up cycle, getting to a new confidence level [with fiber],” Almand said. But he added that “sometimes it’s good to break out of your comfort zone.”

Lowering the IDF count

Christner said the Corning optical gear is at the center of the Kyle Field deployment, providing support for the fan-facing Wi-Fi as well as Wi-Fi for back of the house operations like point of sale; it also supports the stadium DAS, as well as a network of more than 1,000 IPTV screens. Aruba Networks is the Wi-Fi gear supplier, and YinzCam is helping develop a new Kyle Field app that will include support to use smartphones as remote-control devices for IPTVs in suites.

On the Wi-Fi side, Christner said the finished network will have 600 APs in the bowl seating areas, and another 600 throughout the facility, with a stated goal of supporting 100,000 concurrent 2 Mbps connections. The DAS, Christner said, is slated to have 1,090 antennas in 50 sectors.

With no intermediate switching gear at all, Christner said that for the fiber network in Kyle Field only 12 intermediate distribution frames (the usually wall-mounted racks that support network-edge gear, also called IDFs) would be needed, as opposed to 34 IDFs in a legacy fiber/coax system. In addition to using less power, the cabling needed to support the fiber network is a fraction of what would have been needed for coax.

One of the more striking pictures of the deployment is a 36-inch wide cable tray installed for the original copper-network plan, which is carrying just 10 inches of fiber-optic cable. Christner said the fiber network also provides a cleaner signal for the DAS network, which already had a test run this past season, when 600 DAS antennas were deployed and lit during the 2014 season.

“At the Ole Miss game we had 110,663 fans at the stadium, and according to AT&T on the DAS all their lights were green,” Christner said. “Via our completely integrated fiber optic solution, we are now able to provide the DAS with much higher bandwidth as well,” said Texas A&M’s Almand, who also said that the carriers have responded very positively to the new DAS infrastructure.

Up from the dust – a model for the future?

Antenna and zone gear box near top of stands

Antenna and zone gear box near top of stands

Also included in the design – but not being used – are an additional 4,000 spare fibers at 540 zone locations, which Christner said can be immediately tapped for future expansion needs. And all of this functionality and flexibility, he added, was being built for somewhere between one-third and 40 percent less than the cost of a traditional copper-based solution.

The proof of the network’s worth, of course, will have to wait until after the west stands are imploded, the new ones built, and the final pieces of the network installed. Then the really fun part begins, for the users who will get to play with things like 38 channels of high-def TV on the IPTV screens, to the multiple-angle replay screens and other features planned for the mobile app. At Texas A&M, IBM’s support squad will include some team members who work on the company’s traditionally excellent online effort for the Masters golf tournament, as well as the “smarter stadium” team.

For Texas A&M’s Almand, the start of the 2015 season will mark the beginning of the end, and a start to something special.

“If I were a country singer, I’d write something about looking forward to looking back on this,” Almand said. “When it’s done, it’s going to be something great.”

Niners add in-seat merchandise delivery, transit info to Levi’s Stadium app

Screen shot of new merchandise ordering feature in Levi's Stadium app. Credit: VenueNext

Screen shot of new merchandise ordering feature in Levi’s Stadium app. Credit: VenueNext

As previously reported by Mobile Sports Report, the San Francisco 49ers have added in-seat merchandise ordering and public transportation information to the Levi’s Stadium app, ahead of Saturday night’s game between the Niners and the San Diego Chargers.

While the new features were hinted at during a recent technology summit at the stadium, application developer VenueNext made the upgrades official today. Here are the official new features the app will have by Saturday:

— Now Levi’s Stadium app users can order jerseys, novelties and other 49ers apparel to their seat, anywhere in the stadium, paying with credit card or Apple Pay.
— Faithful49 loyalists can redeem yards to purchase food, beverage and merchandise right in the app.
— Users can also get up-to-the-minute transit information for their ride home with TransitScreen (which includes every possible mode of public transport in real time, viewable on one screen)

Photo of directions function in Levi's Stadium app. Credit: MSR

Photo of directions function in Levi’s Stadium app. Credit: MSR

Fans should upgrade their Levi’s Stadium app before coming to the next game, since only the latest version of the app will support the new features. According to the VenueNext press release, the Levi’s app also offers in-stadium access to the NFL Network’s RedZone channel, something which we hadn’t seen in several trips to Levi’s this season.

We’re interested in hearing from any fans who are at the next games at Levi’s about whether or not the transit feature is helpful; while we think it could be a big timesaver (so far at all our visits we regularly see people getting off the light rail train wondering where to go next) we wonder if enough people are aware of all the functionality in the app. Even Niners CEO Jed York has been somewhat surprised at the light takeup of things like the app’s ability to show multiple camera-angle replays.

Stadium Tech Report: THE COLLEGE FOOTBALL ISSUE looks at university Wi-Fi deployments

collegethumbIf there was a college football playoff for stadium wireless network deployments, which four teams would be in? Electing myself to the committee, I think my top picks would be the same venues we’re profiling in our latest Stadium Tech Report – Baylor, Nebraska, Stanford and Texas A&M. All four are pursuing high-end networks to support a better fan experience, leading the way for what may turn out to be the largest “vertical” market in the stadium networking field – sporting venues at institutions of higher learning.

To be sure, network deployments at major universities in the U.S. are still at the earliest stages — in our reporting for our latest long-form report, we found that at two of the top conferences, the SEC and the Pac-12, only four schools total (two in each conference) had fan-facing Wi-Fi, with only one more planned to come online next year. Why is the collegiate market so far behind the pro market when it comes to network deployment? There are several main reasons, but mostly it comes down to money and mindset, with a lack of either keeping schools on the sidelines.

Leaders look for NFL-type experiences

But at our “playoff” schools, it’s clear that with some ready budget and a clear perspective, college stadiums don’t need to take a back seat to anyone, pro stadiums included. The networks, apps and infrastructure deployed for this season at Baylor’s McLane Stadium and Nebraska’s Memorial Stadium are among the tops anywhere in sports, and the all-fiber infrastructure being put in place at Texas A&M should make that school’s Kyle Field among the most-connected if all work gets completed on time for next football season. Read in-depth profiles on these schools’ deployments, along with team-by-team capsule technology descriptions and an exclusive interview with Mississippi State athletic director Scott Stricklin in our latest report, available for free download from our site.

We’d like to take a second here to thank our sponsors, without whom we wouldn’t be able to offer these comprehensive reports to you free of charge. For our fourth-quarter report our sponsors include Crown Castle, SOLiD, Extreme Networks, Aruba Networks, TE Connectivity, and Corning.

Niners’ CEO gives Levi’s Stadium operations a ‘B’ grade

Jed York, Niners CEO, speaking at tech summit at Levi's Stadium. Credit all photos: Paul Kapustka, MSR

Jed York, Niners CEO, speaking at tech summit at Levi’s Stadium. Credit all photos: Paul Kapustka, MSR

San Francisco 49ers CEO Jed York, who caused a stir last week by apologizing for his team’s play on Twitter, on Thursday gave the operation of his club’s new Levi’s Stadium an above-average grade of “B” for the first four months of its inaugural season.

Speaking at a technology-focused fan-experience and innovation summit held Thursday at the stadium’s “501” club, York said “I’d give us a ‘B’ on our execution [at Levi’s Stadium] this year,” citing parking issues and foot-traffic flow into the stadium as problems not yet fully solved for the 68,500-seat facility in Santa Clara, Calif.

Given the new stadium’s complicated location — in the middle of a busy corporate-headquarters area and right next to the Great America theme park — it was probably somewhat of a given that there would be parking struggles the first season, as fans, police, traffic directors and stadium workers all figured out how to make the dance work. Though some progress has been made during the season, York said that “parking and just getting people here” remain the biggest issue he sees at Levi’s Stadium.

While the Niners have tried to use technology to solve the problems of getting fans inside (with app-based parking maps and wayfinding), York said that despite plans to funnel fans through the correct gates to get more quickly to their seats, many fans still just head for the gate that’s closest to their parking or train arrival spot, which has sometimes led to big backups at the check-in lines.

On the networking side of things, York said all seemed to be going well with the stadium’s Wi-Fi and cellular networks, which have performed well in a series of ad hoc tests conducted by MSR in visits this season.

“We haven’t had any [network] glitches, but we’ve been doing a lot of tweaking,” York said. The Levi’s Stadium app has been the center of most of that tweaking, with continual upgrades to add features and fix problems like an Android bug that surfaced in a mid-November revision. York said that the Niners have been careful to make sure the human engineering behind the technology is solid before launching new things, like having enough runners to support the feature that allows food to be ordered and delivered to seats.

Levi's Stadium ready for the Pac-12 championship game

Levi’s Stadium ready for the Pac-12 championship game

And though it hasn’t been officially confirmed yet, York said the team might debut a planned feature of having team merchandise available for purchase and delivery to seats at the next home game, Dec. 20 vs. San Diego. John Paul, CEO of Levi’s Stadium app developer VenueNext, also spoke at the summit Thursday and said in an interview that the Dec. 20 game might also see the debut of a new feature that adds in mass transit and Uber wait times. (A good idea for fans going to the stadium is to check on game day morning for any updates to the app.)

On the app side, York said one surprise was the fairly low uptake of fans using the app’s instant replay features. After fans watched 7,800 replays during the regular-season home opener (the first game replays were available in the app), usage has gone down, with less than 4,000 replays watched during the last home game against Seattle.

“We thought that mobile replays would be absolutely a home run feature, but it hasn’t got that much traction,” said York. The culprit, he said, might be the twin HD displays above each end zone at Levi’s Stadium, which are somewhat stunning in their clarity. “We do have great video boards,” York said.

In his talk York also thought out loud about the possibility of using wearable technology to both better help prevent player injuries as well as being able to provide more rich detail for fans, like how hard a hit was on the field. But he also stressed that technology at Levi’s Stadium was not meant to be used for technology’s sake, but instead to improve the experience of being at the game. And improvement on the stadium staff’s execution, York said, will go a long way to making it all work together.

“When we get to an A, or A+, people will really be blown away,” York said.

Levi's Stadium at twilight

Levi’s Stadium at twilight

Levi’s Stadium update: Wi-Fi traffic steady at 2.3+ TB per game, Amazon and Google lead apps use

Scoreboard promo for the Levi's Wi-Fi network

Scoreboard promo for the Levi’s Wi-Fi network

Usage of the Wi-Fi network at the San Francisco 49ers’ new Levi’s Stadium continues to hold steady at 2.3-plus Terabytes of data offloaded per game, according to the most recent statistics provided by the Niners’ network staff.

Though he’s no longer the team’s vice president of technology, former Levi’s network guru Dan Williams remains as a game-day consultant to the team, and he shared some recent network statistics with MSR, including some app usage marks that show Amazon Cloud Drive and Google APIs being among the top two applications being used over the Levi’s Stadium Wi-Fi network.

According to Williams’ numbers, for the Niners’ Nov. 23 day home game against Washington, the Levi’s Stadium Wi-Fi network had 22,095 unique users, 35 percent of the total attendance; the peak number of concurrent users was 14,700, reached at 2:50 p.m. (near halftime), and the total data used was 2.31 TB, between the hours of 9 a.m. and 5 p.m.

During the Thanksgiving day game against the Seahawks, the network had 23,371 users (36 percent of attendees) who used 2.38 TB of data, with a peak of 16,800 concurrent users at 7:15 p.m.

Video-replay statistics from the Levi’s Stadium app apparently do better when there are more home-team highlights to watch. During the victory over Washington, 1,074 fans watched 4,885 replays, with the 30-yard TD pass from Colin Kaepernick to Anquan Boldin racking up 953 total views from 398 unique users — meaning many users are watching the same replay more than once (maybe showing it to people sitting around them?). For the Seahawks game, 1,407 fans watched 3,875 replays, with a Kaepernick pass to Michael Crabtree accounting for the most views, 487 total from 180 unique users.

The top 4 apps used by fans on the network varied a bit from the two close games, but according to Williams Amazon Cloud Drive was the top app for both recent games. For the Seahawks game, the next three top apps were Google APIs, Facebook and SnapChat, while for the Washington game the next three top apps used were stadium video, Google APIs and Apple iTunes.