UPDATE: IBM: Sorry! No IPTV deal yet for new Atlanta NFL stadium

Artist's overhead view of new Atlanta NFL stadium

Artist’s overhead view of new Atlanta NFL stadium

Update, 7/9/15, 5 p.m. PT: Sports app developer YinzCam has apparently not yet won the deal to provide IPTV technology for the new NFL stadium being built for the Atlanta Falcons, two days after an IBM exec said that they had.

According to Jim Rushton, global leader and partner in IBM’s sports and entertainment practice, he made some “factually incorrect” statements during a panel presentation Tuesday at the Association of Luxury Suite Directors (ALSD) conference in San Francisco. During his talk Rushton provided some high-level details of IBM’s plans to provide wireless networks and other technologies inside the 71,000-seat, $1.4 billion stadium that is scheduled to open in 2017, and both in his talk and presentation said that app developer YinzCam would be the IPTV technology provider for the new stadium.

However, in a subsequent phone call Thursday evening, Rushton said his statements about YinzCam were “factually incorrect,” and that in fact no contract has yet been awarded for the IPTV technology to be used at the Atlanta stadium. Hence this update to a previous version of the story which led with the YinzCam news, which was mainly new to us since Rushton didn’t name any other potential subcontractors, including the vendors who will be supplying gear for the passive optical network (PON) at the heart of the network or the provider of the Wi-Fi and/or DAS gear that will provide the stadium’s planned wireless connectivity.

However, we will stick with our original speculation, which pegs the leading candidates for optical gear and Wi-Fi equipment as likely Corning and Aruba Networks, who respectively supplied those same technologies for the IBM-led network deployment at Texas A&M University’s Kyle Field, where YinzCam also provided IPTV technology as well as technology for the stadium app. (Daktronics has already been announced as the supplier of the new planned Halo Screen video board.) So if YinzCam hasn’t actually inked a deal for IPTV in Atlanta yet, we will still keep them at the “most likely to win the contract” status.

During his talk Rushton said that network technologies still hadn’t been picked for Atlanta, with “proof of concept” testing still taking place in labs on site at the already-active construction zone. He also would not say whether YinzCam would also be part of the Atlanta Stadium app. YinzCam CEO Priya Narasimhan did not respond to email inquiries about the Atlanta deal (and maybe now we know why). On the Wi-Fi side it will be interesting to see if IBM still chooses to work with Aruba now that Aruba is part of HP after a $3 billion acquisition earlier this year.

Like at Texas A&M, IBM came late to the Atlanta stadium development process, but is claiming that its plan to build an internal fiber backbone for both Wi-Fi and DAS deployments has already saved space, time and money. Rushton said that in Atlanta the DAS headend will be located off the stadium site, a switch that opened up 10,000 square feet of stadium space.

AT&T Stadium, Levi’s Stadium tops for stadium Wi-Fi usage

Niners' Flickr promotion on scoreboard at Levi's Stadium. Photo: Paul Kapustka, MSR

Niners’ Flickr promotion on scoreboard at Levi’s Stadium. Photo: Paul Kapustka, MSR

There’s no competition and no wagering, but if you wanted to find the sports stadium that handles the most Wi-Fi traffic, two of your top finalists would no doubt be Levi’s Stadium and AT&T Stadium, which both recently released some season-long Wi-Fi statistics to Mobile Sports Report, including year-long totals in excess of 40 terabytes for Levi’s Stadium and 50+ TB for AT&T Stadium.

Levi’s Stadium, the brand-new home of the San Francisco 49ers located in Santa Clara, Calif., has carried more than 45 TB of traffic on its Wi-Fi network through 20 events, according to Chuck Lukaszewski, very high density architect in the CTO Office of Aruba Networks, an HP Company (Aruba is the Wi-Fi gear suppler to Levi’s Stadium). During those events — 10 of which were NFL games, the other 10 a list including college games, concerts, a hockey game and a wrestling event — the Levi’s Stadium network saw approximately 415,000 unique users, Lukaszewski said.

Down in Texas, the home of the Dallas Cowboys reported some similar Wi-Fi statistics, with a total tonnage mark of 42.87 TB across 11 NFL games and six college games, according to John Winborn, chief information officer for the Dallas Cowboys Football Club. During those games the AT&T Stadium Wi-Fi network saw more than 500,000 unique connections, Winborn said. Winborn also said that AT&T Stadium saw almost an additional 10 TB in usage from concerts and from hosting the NCAA’s Final Four in 2014, pushing the venue’s Wi-Fi usage mark to 52.17 TB. “This [total] does not include our dirt events (Supercross, Monster Trucks, Rodeos) and other full stadium events that would give us an even higher number,” Winborn added in an email to MSR.

AT&T Stadium at College Football Playoff championship game. Photo: Paul Kapustka, MSR

AT&T Stadium at College Football Playoff championship game. Photo: Paul Kapustka, MSR

For comparison, for football games Levi’s Stadium has a normal capacity of 68,500, with additional seating available (including on-field seats for concerts and other events) that can bring capacity to nearly 80,000. AT&T Stadium has a listed football capacity of 85,000, but that number can also be expanded with standing-room only numbers; according to Wikipedia AT&T Stadium had a record 105,121 fans in attendance for a Cowboys football game on Sept. 21, 2009, and had 108,713 fans in the stadium for the NBA All-Star game on Feb. 14, 2010.

Single-day connections for both pass Super Bowl marks

And while the most recent Super Bowl at the University of Phoenix Stadium in Glendale, Ariz., still holds what we believe to be the highest single-game data mark for Wi-Fi traffic at 6.2 TB, both Levi’s Stadium and AT&T Stadium have had events with Wi-Fi usage totals exceeding 4 TB, with March’s WrestleMania 31 hitting 4.5 TB at Levi’s Stadium and the January College Football Playoff championship game recording 4.93 TB of traffic at AT&T Stadium.

Scoreboard promo for the Levi's Wi-Fi network

Scoreboard promo for the Levi’s Wi-Fi network

Both AT&T Stadium and Levi’s Stadium surpassed the Super Bowl when it came to high-water marks for single-game connected user totals; somewhat ironically, AT&T Stadium set what is probably the highest-ever Wi-Fi connection total of 38,534 unique users (out of 91,174 in attendance) during last season’s home opener against the visiting 49ers. According to Winborn, the total was reached “largely due to heavy in-game promotion around the Wi-Fi upgrades and new stadium app.”

At Levi’s Stadium, the season home opener against the Chicago Bears saw 29,429 unique users on the Wi-Fi network, which was more than the 25,936 unique devices connected to the network at Super Bowl XLIX in Arizona. Levi’s Stadium also saw the highest number of concurrently connected users, 18,900, at the Bears game, compared to a high of 17,322 at the Super Bowl. At AT&T Stadium, Winborn said the season high for concurrently connected users was 27,523, recorded during the Cowboys’ home playoff game against the Detroit Lions.

Looking ahead to Super Bowl 50

According to Aruba’s Lukaszewski, the Wi-Fi network at Levi’s Stadium “did what it was supposed to do” last season, carrying high loads of wireless traffic. One stat the Levi’s team invented for its own network was “amount of time the network spent carrying more than 1 Gbps” — a total that Lukaszewski said reached 21 hours and 30 minutes across the 10 NFL events, and 31 hours 40 minutes across all 20 events.

For the upcoming football season and the hosting of Super Bowl 50 next February, the Levi’s Stadium Wi-Fi network will get some strategic Wi-Fi AP upgrades, specifically along some of the concourse areas where groups of standing fans had effectively blocked signals from under-the-seat APs near the tops of seating rows. Lukaszewski said the stadium team would add additional APs in areas where fans are spending time standing, as well as in concourse and plaza bar areas, where some structures were added during the season. Levi’s Stadium is also planning to deploy temporary under-the-seat APs when additional bleacher seats are added for the Super Bowl, Lukaszewski said.

New Atlanta football stadium picks IBM as lead technology integrator

New Atlanta football stadium under construction. Credit all images: New Atlanta Stadium

New Atlanta football stadium under construction. Credit all images: New Atlanta Stadium

The yet-to-be named new football stadium under construction in Atlanta has selected IBM as its lead technology integrator, somewhat officially welcoming the new 800-pound gorilla to the stadium technology marketplace.

While computing giant IBM has dabbled in sports deployments before — mainly contributing technology as part of its corporate sponsorship of events like The Masters in golf and the U.S. Open for tennis — only recently has Big Blue gotten into the large-venue technology integration game. And while IBM’s recent deal as technology integrator for the revamp of Texas A&M’s Kyle Field was probably its true debut, for the new Atlanta stadium IBM will lead the selection, design and deployment of a wide range of technologies, including but not limited to the core Wi-Fi and DAS networks that will provide in-venue wireless connectivity.

Due to open in March of 2017, the new $1.4 billion stadium is expected to hold 71,000 fans for football, and up to 83,000 fans for other events like basketball or concerts. And while soccer and concerts and basketball will certainly be part of its events schedule, the NFL Atlanta Falcons and owner Arthur Blank are driving the bus on the new building, picking IBM in part to help satisfy a desire to build a venue that will be second to none when it comes to fan experience.

IBM’s size and experience a draw for Atlanta

Interior stadium design rendering

Interior stadium design rendering

In addition to Wi-Fi and DAS network buildouts, IBM will design systems to control the expected 2,000-plus digital displays in the planned stadium and will also oversee other technology-related parts of the stadium, including video security, physical door controls and a video intercom system, according to an announcement made today. IBM will also partner with the stadium owners to develop as yet-undetermined applications to “leverage the power of mobility to create a highly contextual, more personalized game day experience for fans, all through the integration of analytics, mobile, social, security and cloud technologies.”

In a phone interview Thursday, Jared Miller, chief technology officer for Blank’s namesake AMB Sports and Entertainment (AMBSE) group, said IBM’s depth and breadth in technology, applications and design made it a somewhat easy choice as lead technology partner.

Miller said the stadium developers looked at the number of different technology systems that would exist within the building, and ideally wanted to identify a single partner to help build and control them all, instead of multiple providers who might just have a single “silo” of expertise.

Proposed stadium exterior

Proposed stadium exterior

“IBM is unique with its span of technology footprint,” Miller said. He also cited IBM’s ability to not just deploy technology but to also help determine what the technology could be used for, with analytics and application design.

“They’ve looked at the [stadium] opportunity in a different manner, thinking about what we could do with the network once it’s built,” Miller said.

IBM, which also has a sizable consulting business, created a group targeting “interactive experiences” about two years ago, according to Shannon Miller, the North America Fan Experience Lead for the IBM Interactive Experience group. Miller (no relation to Jared Miller), also interviewed by phone Thursday, said IBM had been working with Arthur Blank and the Falcons for more than a year to determine how to make the new stadium “the best fan experience in the world.”

And while IBM is somewhat of a newcomer to the stadium-technology integration game, IBM’s Miller said the company not only understands “how to make digital and physical work together,” but also has resources in areas including innovation, technology development and design that smaller firms may not have. And while the Kyle Field project was ambitious, IBM’s Miller said the Atlanta operation will be much bigger.

“The size and scale of what we’re going to do [in Atlanta] will be unique,” he said.

No suppliers picked yet for Wi-Fi or DAS

For industry watchers, IBM and the Falcons team have not yet picked technology suppliers for discrete parts of the coming wireless network, such as Wi-Fi access points and DAS gear. (Daktronics has already been announced as the supplier of the new planned Halo Screen video board.) But those vendor decisions will likely be coming soon, since the stadium is under a hard deadline to open for the first game of the Major League Soccer season in March of 2017.

“We’re working fast and furious on design, and we want to identify [the gear suppliers] as early as possible,” said AMBSE’s Miller.

IBM and AMBSE did announce that the stadium’s network will be fiber-based, and will probably use Corning as a fiber and Passive Optical Network (PON) technology provider, though that choice was not announced or confirmed. IBM and Corning partnered to install a fiber network core for Wi-Fi and DAS at Texas A&M’s Kyle Field, believed to be the first large fiber network in a large stadium anywhere.

The Atlanta deal puts IBM solidly into the rapidly expanding field of stadium technology integration, which includes companies like CDW (which led network deployments at the University of Nebraska and the University of Phoenix Stadium) as well as stadium ownership groups, like the San Francisco 49ers, and technology name sponsors like AT&T, which has partnered with owners for technology and network deployments at venues like AT&T Park and AT&T Stadium.

Overhead view

Overhead view

Seahawks hit local Wi-Fi record during playoff game with 2.6 Terabytes of traffic; Verizon maintains cone of silence over its investment in network

Screen Shot 2015-01-16 at 11.05.18 AMThe Wi-Fi network that debuted in CenturyLink Field this season had its highest Wi-Fi traffic day last weekend, when 2.6 terabytes of data was carried during the Seattle Seahawks’ Jan. 10 playoff victory over the Carolina Panthers.

According to the Seahawks’ tech staff, 18,899 of the 68,524 fans in attendance used the Wi-Fi network at some point, with a peak concurrent user mark of 15,662. The peak bandwidth utilization of 1.4 Gbps was reached just after Cam Chancellor sealed the win with his electric 90-yard interception touchdown return, the Seahawks tech staff said.

The Wi-Fi numbers represent traffic on both of the separate Wi-Fi networks in the stadium, one of which is reserved exclusively for Verizon Wireless customers. Verizon, which has declined to comment publicly on the specifics of its partnership with the Seahawks, is believed to have bankrolled a major portion of the Wi-Fi deployment at CenturyLink. Before the Wi-Fi partnership between the Seahawks, Wi-Fi gear provider Extreme Networks and Verizon was officially announced on Oct. 29, Verizon claimed it had “added an in-stadium Wi-Fi system” at CenturyLink prior to the start of the 2014 football season, as part of a national football stadium Wi-Fi map Verizon published on Sept. 4. The Wi-Fi deployment was somewhat of a surprise, since team officials had long said they were looking at 2015 as the year they might pull the trigger on a Wi-Fi expenditure. Apparently, having available funding from Verizon helped push the project forward faster than expected; but again, we have no official confirmation or explanation of the exact fiscal participation level of all the partners involved.

For Seattle fans, having Verizon as a Wi-Fi partner has additional perks — in addition to a separate Wi-Fi network for Verizon customers, all fans at the stadium also have access to the NFL Network’s popular RedZone channel, via the new Seahawks stadium app created by YinzCam. Because of various conflicting rights contracts, RedZone isn’t available in most stadiums for fans to watch. The Seahawks also have a number of live-action and replay views available via the app; however, the stadium’s tech team did not have any metrics on fan use of the app or the number of video replays watched.

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Verizon’s reluctance to comment publicly on its Wi-Fi deployments is no surprise; repeated attempts and queries by MSR for interviews with Verizon executives about Wi-Fi deployments are routinely ignored by Verizon representatives, and public quotes like the one from Bobby Morrison, president for the Pacific Northwest and Alaska at Verizon Wireless, in the official press release, don’t offer any details about Verizon’s level of fiscal commitment to the CenturyLink deployment. Verizon has also declined to comment on its Wi-Fi network deployments at Ford Field in Detroit, and at the Staples Center in Los Angeles.

Verizon executives were also conspicuously absent from a stadium-technology event centered on the Wi-Fi network earlier this week, leading some reports to omit Verizon’s considerable participation in the network’s deployment, something NFL CIO Michelle McKenna-Doyle told MSR about earlier this year.

Though no statistics were available from Verizon about the DAS deployment it also put in at CenturyLink Field this year, AT&T did share some DAS stats from CenturyLink for the Seahawks’ Dec. 14 victory over the San Francisco 49ers. During that game, AT&T customer traffic on the AT&T DAS at CenturyLink was 395 GB, according to AT&T. It will be interesting to see how much wireless traffic this weekend’s NFC championship game between the Seahawks and the Packers will generate — we’ll track down as much of it as we can to see if it compares to the 6 TB mark set at the recent College Football Playoff championship game at AT&T Stadium in Arlington, Texas.

Stadium Tech Report: Arizona Cardinals get stadium ready for Super Bowl with Wi-Fi upgrade

University of Phoenix Stadium. Credit all photos: Arizona Cardinals.

University of Phoenix Stadium. Credit all photos: Arizona Cardinals. (click on photos for larger image)

As they get set to host their second Super Bowl this February, the IT team at the University of Phoenix Stadium in Glendale, Ariz., knows now what they didn’t know then: The big game requires a big wireless network. Bigger than you think.

“It’s funny to look back now on that first Super Bowl,” said Mark Feller, vice president of information technology for the Arizona Cardinals, speaking of Roman numeral game XLII, held in the barely 2-year-old facility on Feb. 3, 2008. With a couple Fiesta Bowls and one BCS championship game (2007) under his belt in a facility that opened with Wi-Fi and DAS, Feller said he and his team “thought we had a good handle” on what kind of network was required for a Super Bowl crowd.

The NFL, he said, begged to differ. Those college games might have been big, but the Super Bowl was bigger.

“We had the Fiesta Bowl that year at night, and when the game was over there were people from the NFL there wanting to know when they could set up,” said Feller in a recent phone interview. “This year, we’re much better prepared. We know what the water temperature is this time.”

Rip and replace, with more and better gear

Wi-Fi railing antennas

Wi-Fi railing antennas

For Super Bowl XLIX, scheduled to take place in Glendale on Feb. 1, 2015, Feller and his team have not just tuned up their network — they have done a full rip and replace of the Wi-Fi system, installing new Cisco gear from back end to front, in order to support a wireless game-day demand that is historically second to none. Integrator CDW has led the Wi-Fi effort and Daktronics and ProSound did the installation of new video screens, and neutral host Crown Castle has overseen a revamp of the DAS system, again with more antennas added to bolster coverage. In all, there has been more than $8 million in wireless improvements before this year, Feller said, as well as another $10 million for two new video boards that are each three times larger than what was there before.

“The last three or four years there have been things we knew we needed to improve [before the Super Bowl],” Feller said. After extensive work with the NFL’s technical team — this time well before the Fiesta Bowl — Feller oversaw a “top to bottom” refurbishment that included replacing core Cisco networking gear with newer gear, and new and more Wi-Fi access points that now total somewhere north of the 750 mark, with some more to be added before the big game. The new network, which was in place for the start of the current NFL season, has undergone testing by CDW at each home game, Feller said. CDW also plans to expand the network outside the stadium before the Super Bowl, in part to handle the extra events that take place not just on game day but in the days leading up to the game.

“The plan is to install more [coverage] outside, in the plaza areas,” Feller said.

When it opened in 2006, the $455 million University of Phoenix Stadium was one of the first with full-bowl Wi-Fi, using Cisco gear from the inside out. “Cisco was in here before they called it [their solution] ‘connected stadium’,” Feller said. From core network switches to firewalls to edge switches, this year there is all new Cisco gear in the venue, as well as new 3700 series APs, with panel antennas and antennas in handrails.

“Handrail [antennas] are sometimes a bit of a challenge, because you need to drill through concrete that’s 40 feet up in the air, behind another ceiling,” said Feller, describing one particular design challenge. Another one was mounting antennas on drop rods from the catwalks below the stadium’s retractable roof, to serve the upper-area seating. There are also some new Wi-Fi APs on the front row of the seating bowl, pointing up into the crowd.

“It was a fun project,” Feller said.

Stadium with roof open

Stadium with roof open

All on board for the DAS

The upgrade for the stadium’s DAS, led by Crown Castle, was just finished a few weeks ago, Feller said, and included more coverage outside the stadium as well, with antennas placed on light poles and on the stadium’s shell.

“Crown Castle did a great job of managing the carriers” on what is a 48-sector DAS, Feller said. “It [the upgrade] really required a lot of creative thinking from their engineers.”

Since the stadium was originally designed with wireless in mind, Feller and his team didn’t need to build new head end room for the DAS upgrades. “But I wouldn’t say we have plenty of space left,” he added. “We’ve got a lot of new equipment.”

Though all the major carriers are expected to be on the DAS by the big game, league partner Verizon Wireless should have some special projects up its sleeve for the big game, including another demonstration of its LTE Broadcast technology, which optimizes things like live video over LTE cellular links.

New Cardinals app a preview of Super Bowl version?

The Cardinals also had a new version of the game-day team app for this season, built by stadium-app leader YinzCam. According to Feller the new app supports three different live video feeds, as well as instant replays.

Wi-Fi antenna on railing

Wi-Fi antenna on railing

“It’s really cool to have that ability to watch things like a touchdown pass at the end of the game,” Feller said. And while no details have yet been revealed, in an interview with NFL CIO Michelle McKenna-Doyle earlier this year MSR learned that the league and YinzCam are working on a Super Bowl app with its own new bells and whistles. (Stay tuned for more info on the Super Bowl app.)

In addition to two more regular-season home games in December, the University of Phoenix Stadium will have at least a couple more dry runs to help test the network, during the Dec. 31 Fiesta Bowl and during the NFL’s Pro Bowl, scheduled for Jan. 25. And though the Cardnials lost to the Atlanta Falcons Sunday, at 9-3 they are still tied with the Green Bay Packers for the best record in the NFC, something that has the Phoenix faithful optimistic about the postseason.

“We’re going to get some more test runs, on New Year’s Eve and during the Pro Bowl,” Feller said. “And maybe some home playoff games as well!”

(more photos below)

Wi-Fi antenna in roof rafters

Wi-Fi antenna in roof rafters

More antennas in rafters

More antennas in rafters

Wi-Fin antenna under overhang

Wi-Fi antenna under overhang

Levi’s Stadium ‘NiNerds’ get high-visibility wardrobe upgrade

NiNerd sporting the new neon vest. (Click on any photo for a larger image) Credit all photos: Paul Kapustka, MSR

NiNerd sporting the new neon vest. (Click on any photo for a larger image) Credit all photos: Paul Kapustka, MSR

I needed a one of the NiNerds Sunday at Levi’s Stadium, and thanks to a new wardrobe addition, they were a lot easier to find.

Discovering a problem with the Levi’s Stadium app, I looked around for one of the stadium’s walk-around technology helpers — aka the “NiNerds” — and found one quickly thanks to the new neon-yellow vests many were wearing during Sunday’s game between the San Francisco 49ers and Washington.

Earlier this season, the NiNerds were much more nattily dressed in their gingham-check shirts, bow ties and fake horn-rim glasses. While cool and cute, the outfits proved hard to recognize in the crowd, especially since the NiNerds’ red check shirts looked a lot like the jerseys and t-shirts worn by the many faithful fans. Perhaps in order to make the NiNerds stand out more, the team dressed them in neon Sunday, like Wi-Fi “coaches” in other stadiums have done.

Unfortunately, the NiNerd I talked to wasn’t able to solve my problem (it seems to be related to a known bug in the newest Android release of the stadium app) but I did notice during my visit Sunday that the NiNerds in general seemed to be more numerous and visible, and they even got a nice shout-out on the Levi’s Stadium big screen (see photos below). Below are some thoughts and observations on the network performance, the app performance and the overall fan experience at Levi’s, which I hadn’t visited since the season opener back on Sept. 14.

Wi-Fi network struggles at 2.4 GHz, soars at 5 GHz

Speed test results from outer concourse location, Levi's Stadium, pregame

Speed test results from outer concourse location, Levi’s Stadium, pregame

On the network-performance side of things, the Wi-Fi system seemed as robust as ever for new devices, including the AT&T LG Optimus G Pro I’ve been test-driving lately. With the Android device and its 5 GHz Wi-Fi connection I hit speeds of 31 Mbps download and 29 upload before the game on the Levi’s Stadium outside concourse, and then had a 14 Mbps download connection in my seat in section 229 (south end zone) at kickoff. In the third quarter I wandered up to the top (7th) level of seats, and got a 28/33 Mbps reading while waiting in a concession-stand line.

With my older Verizon Droid 4 device, however, I struggled to connect to the Wi-Fi network. Since the phone only runs on the 2.4 GHz Wi-Fi frequency, it doesn’t do well at Levi’s Stadium, where the Wi-Fi is more heavily tuned for newer, 5 GHz-capable devices.

Full charging station... before the game starts

Full charging station… before the game starts

The best Wi-Fi speedtest I could get with the Droid device was a 1.06 Mbps download/3.04 Mbps upload mark, from the same spot on the outer concourse where the newer device recorded blazing connectivity speeds. Switching it over to its Verizon 4G LTE radio, I was able to get much faster connectivity, including one mark of 21.60/9.58 on the main level inside concourse.

I also ran out of juice on the Verizon phone before the end of the game — which could have been either the device draining due to its inability to get a solid connection, or due to the fact that it’s getting old and the battery doesn’t hold a charge that well anymore. Judging from the crowds around the Levi’s Stadium recharging stations (the picture to the left was taken during pregame) I am not alone in my device-energy woes.

App problem derails beacon test

One of the main tasks I had planned for Sunday was to see how well the beacon-assisted wayfinding feature in the stadium app worked. Only problem was, in the new update of the Levi’s Stadium app that was released in the past week (which I downloaded to both devices Sunday morning), several features were missing, including the “Maps” feature.

Picture of app fail in Levi's Stadium Android app

Picture of app fail in Levi’s Stadium Android app

A NiNerd I talked to outside the stadium on the Faithful Mile area showed me the maps/wayfinding feature on his iPhone, and pulled up a GPS-supported direction message that pointed the correct way for me to enter the stadium. But neither he nor the NiNerd I talked to inside the stadium could figure out why both of my Android devices weren’t showing the maps feature, or several other features on the left-upper-corner pull-down menu.

According to Louise Callagy, vice president of marketing for app developer VenueNext, the new version of the app released this week did have a known Android bug. In an email response Sunday night Callagy said, “we know we have a bug where Android gets confused and won’t return results from the network,” adding that rebooting the device might have fixed the problem; however, I did reboot both devices during the game and the problems were not corrected.

Callagy said the Levi’s network also had issues Sunday in getting location information from the beacons. “Our plan is to re-write the code [for the app] and solve this issue, releasing a new version before the Seahawks game on Thursday,” Callagy added in her email.

On the good-news side the replay function of the Levi’s Stadium app was more impressive than earlier versions, with highlights appearing in the app in mere seconds after the original play had concluded. After the Niners’ first TD pass of the day, I was able to view the highlight of the Colin Kaepernick-to-Anquan Boldin pass just after the extra point had been kicked.

I was also able to see the red light/green light system for restroom wait times that drew so much attention when it was talked about earlier this year. However, in real-life practice it’s doubtful anyone thinks of looking at the app when it’s time to go. (It’s also quite likely that while you are looking at the app for a short restroom line, a natural break in action will occur and restroom lines will predictably lengthen everywhere.) I found a quick trick for Levi’s attendees that might pay off in the future: If the restroom you’re aiming toward has a long line, walk a small bit farther to find its back door — where there is often no line at all.

New version of app, with clearer icons on main screen

New version of app, with clearer icons on main screen

I messed up later in the game, however, in thinking that it would just be easier to find a concession stand than to use the app’s express window ordering function. At least the 10 minutes I spent in line behind three women who were apparently ordering for their entire row (five hotdogs, six orders of fries, two orders of wings, two beers and one large ice water) gave me time to conduct a couple more Wi-Fi speedtests. Next time, I’m using the food-ordering features on the app.

I also made great use of the app’s ability to let fans watch live game action (I chose the feed from the main video screens in the stadium). Since I had to leave early I was on the first VTA train when I saw the game-winning TD run courtesy of the app’s live action broadcast. The live video, incidentally, kept playing seamlessly over the AT&T 4G LTE network as I sped away from the stadium, allowing me to watch the final game-sealing sack as I beat most of the traffic home.

VTA trains a smooth ride, once you figure out how to get on

I also had another smooth ride to and from the stadium using the VTA light rail trains from Mountain View — once I was on the train it took just a little over 30 stress-free minutes both coming and going. Getting on the trains, however, is a process that could still use some work. The Mountain View station, which is logistically hampered by having to share space with the Caltrain tracks and station, has very little signage on game day, and has a lot of confusing temporary gates to try to flow foot traffic toward the ticket-verification checkers.

Packed VTA train en route to Levi's Stadium

Packed VTA train en route to Levi’s Stadium

Once I figured out the maze and was guided across the Caltrain tracks I was directed to one of two waiting trains — but then a VTA staffer looked into the train and told people there were also express buses that wouldn’t stop on the way to Levi’s (unlike the trains, which stop at numerous stations en route). The quizzical advice — nobody said if it was any faster to take the buses — had many people wondering what they were supposed to do, causing a delay in closing the train doors as people made up their minds without any more information.

Once we arrived at the stadium, on the exit platform there was no person or sign directing fans in the proper direction. Good thing for the many newbies on the train (the train 2 hours before kickoff was packed) several of us were veterans and directed everyone down the proper ramp. For the return trip the Mountain View line suffered from similar lack of information and signing — and after one train passed the station with signs that said “Not in Service” we got on a second train that also said “Not in service” but whose doors opened anyway and one person in a yellow vest told everyone, “get inside.”

Overall impressions: Levi’s experience and technology still a work in progress

While I continue to be impressed by the network and app performance at Levi’s Stadium, I also felt several times on Sunday like the technology, the stadium and the entire fan experience is still a work in progress — perhaps something to be expected for a venue in its first year of events. But I have to wonder a bit about releasing a new version of the app in midseason, without apparently testing it enough to make sure it worked well on all devices that might want to use it.

I’m also still skeptical on how well the wayfinding feature will work in real world situations; though it sounds great to be able to get GPS-like directions to places inside the stadium, the reality of trying to walk around looking down at your phone on one of Levi’s Stadium crowded concourses is more likely to lead you into someone’s backside. Anyone with tales to tell of Levi’s Stadium technology experiences, please chime in below in the comments or send me an email to kaps at mobilesportsreport.com. I’d be especially interested to know if anyone else saw my app problems Sunday on Android phones. More Levi’s pictures below.

A NiNerd (no vest) helps fans outside the stadium.

A NiNerd (no vest) helps fans outside the stadium.

Kickoff view from Section 229. Thanks to the Niners for the free media access.

Kickoff view from Section 229. Thanks to the Niners for the free media access.

Niners fans get their phone cameras busy for kickoff ceremonies.

Niners fans get their phone cameras busy for kickoff ceremonies.

Scoreboard plug for the app.

Scoreboard plug for the app.

Scoreboard promo for the NiNerds (one in a series)

Scoreboard promo for the NiNerds (one in a series)

Second in the series. This one got laughs from the crowd.

Second in the series. This one got laughs from the crowd.

Probably the first time many fans heard the term "NiNerds"

Probably the first time many fans heard the term “NiNerds”

Nothing says geek like a bow tie

Nothing says geek like a bow tie