Verizon says it is ‘improving the data crediting process’ to address NFL Mobile data-charge snafus

Verizon said it is “improving the data crediting process” for its popular NFL Mobile app, which has apparently caused many headaches this season with users who claimed the cell provider wasn’t following through with its promise to make watching live NFL action free from any data charges.

If comments on Mobile Sports Report blog posts are any indication of wider unrest, there are many NFL Mobile users who have been erreneously charged for wireless data used while watching the live NFL games provided by the NFL Mobile app. In our blog post announcing Verizon’s claim that all NFL Mobile live action this season would be free of data charges, we guessed that Verizon’s unclear answers about so-called “unlimited” versus metered plans meant that the provider hadn’t fully figured out how to correctly bill users of the app. Seems like we were more right than we wanted to be.

A quick scroll through any of the 20-plus comments our blog post received from frustrated users seems to show that on many levels, Verizon’s billing and customer service reps were on different pages when it came to NFL Mobile data use. After more than a month of inquiries to Verizon about the claims by our commenters, this week we finally received an official reply from a Verizon spokesperson. Here it is:

Verizon is committed to providing live games on NFL Mobile data free to our customers and resolving any related billing disputes. We have made recent adjustments improving the data crediting process to reduce usage alerts and to ensure our customers receive consistent answers when they contact our support organization.

Without actually admitting to any problems, Verizon’s statement about “improving the data crediting process” and other issues seems to be a tacit admission that not all was well, an issue that seems to affect NFL Mobile just about every year.

Football fans, however, may have another choice next season when it comes to watching live games on phones, with recent reports claiming that Verizon’s 4-year, $1 billion deal for exclusive rights won’t be renewed.

Connected Stadium Summit in Amsterdam next week — MSR reader discount available

Amsterdam looks like the place to be for connectivity information next week, especially so for the world of stadium networking with the Connected Stadium Summit on tap Nov. 7-8 at Amsterdam ArenA. Some familiar faces like Jeff Lutes of the Orlando Magic are among the scheduled speakers, along with what looks like a good mix of speakers with topics ranging from DAS and Wi-Fi deployment issues to digital fan engagement and broadcast as well. See below for the organizers’ information links about agenda and free passes for stadium and venue operators, as well as a discount for MSR readers.

The Connected Stadium Summit, the only event focused on enhancing customer experience and future-proofing technology investments is coming your way in one week! Taking place this November 7th – 8th, the event will gather leading organisations in the connected stadium space from the likes of Orlando Magic, Amsterdam ArenA, Feyenoord, Deutsche Telekom and more to connect, learn and share ideas together,

We’ve created a snapshot attendee list, highlighting who you can expect to meet and exchange business cards with. Check out the agenda here and we guarantee you will be in good company.

Free passes for stadia, telecoms operators and venues available here. Alternatively book a delegate place and receive a discount with VIP code MSR20.

Commentary: Time to rethink in-seat delivery?

A beer vendor at Wrigley Field this summer. Credit all photos: Paul Kapustka,, MSR (click on any photo for a larger image)

I have a major scoop: Even though Wrigley Field doesn’t have its new Wi-Fi network installed yet, I can confirm that the Friendly Confines has food and drink delivery to fans in all seats.

And you don’t need an app to order a frosty malt beverage. You simply say, “Hey! Beer man! One over here!” And he walks over and pours you a cold one. Apparently this is not new, but has worked for many, many years.

Though I do jest a bit I hope my point is clear: Sometimes there is a bit too much fascination with technology, especially on the stadium app front, which has not yet been warranted. The main question of this essay is whether or not it’s time to rethink the in-seat ordering and delivery phenomenon, to find what really matters to fans and where technology can deliver better options.

Who really wants in-seat delivery?

Editor’s note: This profile is an excerpt from our latest STADIUM TECH REPORT, our Fall 2017 issue that has in-depth profiles of network deployments at Notre Dame Stadium, Sports Authority Field at Mile High, Colorado State’s new stadium, and the Atlanta Falcons’ new Mercedes-Benz Stadium. DOWNLOAD YOUR FREE COPY of the report today!

I will be the first to admit to being guilty as charged in being over-excited about stadium apps and the idea of things like instant replays on your phone and being able to have food and drink delivered to any seat in the stadium. When the San Francisco 49ers opened Levi’s Stadium four years ago, those two services were fairly unique in the sporting world, and it was cool to see how both worked.

The Niners did a lot of human-engineering study on the food delivery problem, knowing that it was more an issue of getting enough runners to deliver the goods than it was to get the app working right. Even a big glitch at the first-year outdoor ice hockey game at Levi’s Stadium was sort of a confirmation of the idea: That so many people tried to order food deliveries it screwed up the system wasn’t good, but it did mean that it was something people wanted, right?

Turns out, no so much. Recently the Niners officially announced that they are taking a step back on in-seat concessions ordering and deliveries at Levi’s Stadium, limiting it to club areas only. Whatever reasons the Niners give for scaling down the idea, my guess is that it mainly had to do with the fact that it turns out that the majority of people at a football game (or basketball too) may not want to just sit in their seats the whole game, but in fact get up and move around a bit.

The end zone view from the beer garden at Colorado State Stadium.

That may be why most of the new stadiums that have opened in the past couple years have purposely built more “porch” areas or other public sections where fans can just hang out, usually with somewhat of a view of the field. The Sacramento Kings’ nice beer garden on the top level of Golden 1 Center and the Atlanta Falcons’ AT&T Perch come to mind here. For the one or two times these fans need to get something to eat, they are OK with getting up and getting it themselves.

Plus, there’s the fact that at the three or four or more hours you’re going to be at a football game, if you’re drinking beer you’re going to eventually need to get up anyway due to human plumbing. We’ve been fairly out front in saying stadiums should spend more time bringing concession-stand technology into the 21st century, instead of worrying too much about in-seat delivery. It’s good to see there are some strides in this direction, with better customer-facing interfaces for payment systems and things like vending machines and express-ordering lines for simple orders.

While there may be disagreement about whether or not in-seat delivery is a good idea, there is certainly universal disgust for concession lines that are long for no good reason. It’s beyond time for stadiums to mimic systems already in place at fast-food restaurants or coffee shops and bring some of that technology spending to bear in the place that everyone agrees still needs work. Even at the uber-techno Levi’s, regular concession stand lines have been abysmal in their slowness. Maybe the Niners and others guilty of the same crimes will pay more attention to less flashy fixes in this department.

Is drink-only delivery the right move?

The Niners’ revolutionary attempt to bring mobile ordering and in-seat delivery to all fans in a big stadium was part of the app suite from VenueNext, the company the Niners helped start as part of their Levi’s Stadium plans. While VenueNext is regularly adding new pro teams to its stable of customers (in September at Mobile World Congress Americas, the Utah Jazz announced they would switch to VenueNext for the upcoming season), not a single one has tried to copy the Niners’ ambitious deliver-anywhere feature.

The end-zone AT&T Perch at Mercedes-Benz Stadium in Atlanta.

And for Super Bowl 50, the signature event that Levi’s Stadium was in part built for, remember it was the NFL shutting down the idea of in-seat delivery of food and drink, limiting the service instead to just beverage ordering and delivery. It probably makes sense for Mobile Sports Report to put together a list sometime soon about the various attempts at in-seat ordering and delivery around the pro leagues, to see what’s working and what hasn’t. To be clear we are talking here about widespread delivery to all seating areas, and not the wait-staff type delivery systems that have been widely deployed in premium seating areas for years.

Our guess, just from tracking this phenomenon the past several years, is that while such services make sense in premium and club areas, simple logistics and stadium real estate (like narrow aisles or packed, sellout crowds) make in-seat ordering and delivery a human-factor nightmare in most venues.

One experiment worth watching is the system being deployed by the Atlanta Falcons at Mercedes-Benz Stadium as part of the team/stadium app developed by IBM. Instead of working online, the app will let fans pick food items and enter payment information, and then take their phone to the appropriate stand to scan and fulfill the order. Nobody knows yet if this will speed up lines or make the concession process faster, but it is at the very least an attempt to try something new, using technology doing what it does best to eliminate a pain point of going to a game — waiting in line.

And while I will be excited to see the new networks being planned for Wrigley (Wi-Fi and a new DAS are supposed to be online for next season), I’m just as sure that whenever I visit there again, I won’t need an app to have a beer and hot dog brought to my seat. Maybe having more choice in items or having that instant gratification of delivery when you want it is where the world is going today, but on a brilliant summer afternoon at Wrigley Field somebody walking down the aisle every now and then works just fine. With the Cubs winning, the organ playing and the manual scoreboard doing its magic in center field, it’s a welcome reminder that sometimes, technology isn’t always the best or neccessary answer.

Orlando City Stadium adds high-density Wi-Fi for soccer fans

Orlando City Stadium, home of the MLS’s Orlando City SC. Credit all photos: Jenna Cornell (click on any picture for a larger image)

Resilient. Connected. Reliable. Even before its new stadium opened in March of this year, Orlando City SC, the Major League Soccer franchise in central Florida, knew exactly what it wanted from fan-facing Wi-Fi.

Leading that list was networking infrastructure to support the stadium’s 25,500 capacity. The team needed to be able to deliver live streaming video to fans through the team’s LionNation app. And they wanted a way to begin collecting user info and building relationships with fans, according to Renato Reis, CIO for the club.

And with so many professional sports teams having already installed wireless infrastructure, Reis knew there was no reason to reinvent the Wi-Fi wheel. “I had the privilege to travel and interview other organizations,” he said, including the Hard Rock Stadium in Miami where the NFL’s Dolphins and the University of Miami both play football, as well as MetLife Stadium in East Rutherford, N.J., shared by the NFL’s Jets and Giants. Reis said Orlando City SC’s technology drew heavily on the experience and deployment of Levi’s Stadium in Santa Clara, Calif. “I used what I learned,” he laughed.

Paying their own way

Orlando City is barely 3 years old and played its first two seasons in the nearby Citrus Bowl, now called Camping World Stadium.

Under-seat Wi-Fi deployment at Orlando City Stadium.

After some confusion with the City of Orlando, Orange County and the state of Florida over money and budget for a new stadium, Orlando City SC’s ownership abandoned a public-private partnership to go its own way. Orlando City Stadium was built with private funds and opened in time for this year’s opener. Orlando City SC shares the venue with the Orlando Pride, the women’s professional soccer team.

“We had a brand-new stadium and no installed Wi-Fi, two factors that really benefited us,” Reis told Mobile Sports Report. “We planned the position of our antennas and leveraged lessons from other organizations to design something from scratch and build for the future.”

Orlando City SC had some help there. The MLS franchise partnered with managed service provider Spectrum Enterprise, a division of Charter Communications; Spectrum in turn has a longstanding partnership with Orlando City’s equipment vendor, Cisco. Together, they installed networking gear, lots of new fiber-optic cable, and the wireless infrastructure that rides atop the stadium’s 10-Gbps backbone network.

The fan-facing Wi-Fi consists of more than 550 wireless access points around the stadium, or about one AP for every 45 users. The APs are installed under seats, in handrails and on posts. “It was more of a challenge to find the right places, design-wise, for APs to keep them out of people’s line of vision,” Reis said.

Orlando City CIO Renato Reis, posing in front of some cool graffiti and below a Cisco AP.

Supporting streaming video

AP density and processing power were important considerations for Orlando City SC. With such dense coverage, each AP delivers 50-80 Mbps per user, Reis said. That ensures that users of the team’s LionNation app enjoy high performance when using its streaming video capability; users posting to social media or checking email also get faster throughput, he added.

That sort of performance is essential, especially for users of the premium version ($8.99) of the LionNation app. In addition to live-streaming video, premium members get access to behind-the-scenes content, as well as 10 percent discounts off food, drink and merchandise purchases (and points for every dollar spent). They also get priority access to post-season tickets and single-game tickets.

Spectrum helped with the stadium’s engineering and remains active in day-to-day management, said Reis. Spectrum performed three rounds of Wi-Fi tuning and collecting data to see where usage was greatest. No surprise: Entry gates and concession areas, according to Reis. They then made adjustments, repointing APs where needed, thus ensuring bandwidth is available where it’s needed most.

Orlando City SC has also been testing wireless food ordering in one stadium section with 1,500 users since the beginning of the year. “The challenge there isn’t technology but rather logistics,” Reis explained.

Screenshot of the Orlando City app

The team is planning to extend the capability more broadly, but needs more experience to help decide how to proceed. “We’ll probably run the test for the rest of the season and make changes next year,” he said.

Reis’s biggest challenge for the moment is encouraging Wi-Fi usage – and also persuading users to register if they’re not on the app. Even with Orlando City Stadium’s Wi-Fi coverage, most users will stick to cellular (the stadium’s DAS network is serviced by AT&T, Verizon T-Mobile and Sprint), he said.

“The problem I’m trying to solve is who is at the stadium,” Reis explained, adding that the only information he has is that a fan bought four tickets, for example, and when they get scanned at the gates. So how to learn more? “Most landing pages are boring,” he laughed; still, he’s considering offering different incentives for Wi-Fi users to check in.

“Can I loyal-ize you so I can learn what you like, what offers are more appealing, what you enjoy and don’t?” Reis asked. That’s a primary challenge for most sports teams, entertainment companies and ecommerce entities. Luckily for Reis and the Orlando City SC, he’s got the bandwidth, backbone and people resources to learn more about fans and build those relationships going forward.

Bird of a different feather: Atlanta’s Mercedes-Benz Stadium takes tech in a new direction

Mercedes-Benz Stadium, the new home of the Atlanta Falcons. Credit all photos: Paul Kapustka, MSR (click on any picture for a larger image)

As you walk up to it, the striking angular architecture of Atlanta’s Mercedes-Benz Stadium is a way of telling you, before you even set foot inside, that this building is different from any other stadium you may know. When you get inside, see the eight-petal roof and the circular video “halo board” right below it, those feelings are confirmed.

Deeper inside the venue’s construction, the theme is continued with the building’s network technology, which is similarly different if less easily seen. With more fiber optical cabling than perhaps any comparable stadium, the new home of the NFL’s Atlanta Falcons is built more with an eye toward what comes next, while also debuting with systems built with the peak of current knowledge and deployment designs.

Unlike the owners and operators of some other new arenas, the Falcons’ aren’t wasting much bandwidth trying to paint Mercedes-Benz Stadium as the best-ever when it comes to stadium technology. (In fact the stadium network crew is being very closed-mouth about everything, not providing any game-day statistics even though informed rumors tell us that the Wi-Fi network is doing very well.) But come back in 5 years, or even 10 years from now, and see if the decisions made here were able to consistently keep the Falcons’ new roost at the top of the stadium-technology game.

Table stakes, plus a halo board

Editor’s note: This profile is an excerpt from our latest STADIUM TECH REPORT, our Fall 2017 issue that has in-depth profiles of network deployments at Notre Dame Stadium, Colorado State’s new stadium, and the Los Angeles Coliseum. DOWNLOAD YOUR FREE COPY of the report today!

Inside the Falcons’ new roost, with the halo board and roof visible

What’s working now, as the venue enters the Falcons’ 2017 NFL season, includes a Wi-Fi network built with nearly 1,800 Aruba access points. Of the 1,000 of those installed in the main seating bowl, most are mounted underneath the seats, a trend that gained steam a couple years ago and now has numerous proof points behind the higher capacity and faster performance of so-called “proximate” networks. There’s also a neutral-host distributed antenna system, or DAS, for enhanced cellular coverage, built and owned by the arena with space rented out to all four of the major U.S. cellular carriers.

And then there’s the halo board, the circular or oval-shaped video screen that circumnavigates the roof right at the base of the also-innovative eight-petal roof, which is designed to open or close in seven minutes or less. If big video screens are a never-ending trend the Falcons are right out in front with their offerings at Mercedes-Benz Stadium, starting with the halo board and the “mega-column,” a hundred-foot high vertical screen just inside the main entryway. The 2,000-plus other regular-sized screens scattered around the venue should ensure there’s always a display visible, no matter where guests are looking.

And while the team has future plans for video, the one piece of network technology that may matter most is the optical fiber, which can support wider bandwidth and faster speeds than traditional copper cabling. The statistic thrown around often in the Mercedes-Benz Stadium press materials — nearly 4,000 miles of fiber used — is meaningless to most who repeat it, other than it seems like a lot of glass wiring.

What’s more interesting from a stadium-design perspective is not exactly the total but instead the reach of the fiber, as the network designers pushed fiber out to the edges much further than before, betting that by putting more capacity farther into the reaches of the stadium, there will be less needs for big-time network upgrades in the future, when the inevitable need for more bandwidth arrives.

Under-seat Wi-Fi AP

“We like to say we’re future flexible, not future proof,” said Jared Miller, chief digital officer for the Falcons. “Future proof does not exist in technology.”

Informed by Texas A&M

In picking IBM as its lead networking technology partner, the Falcons most certainly gleaned a lot of their stadium network design lessons from the 2015 deployment of a new Wi-Fi network at Texas A&M’s Kyle Field, a project also led by IBM. Making deep use of Corning fiber networking technology, A&M’s Wi-Fi network hit the ground running hard, as a big number of under-seat APs supported several big days of data use by the 100,000-plus Aggie fans who filled the building on home-game weekends.

The basic idea behind using fiber is that optical cabling can carry far more bandwidth at faster speeds than a comparable copper wire. By putting more fiber farther out into all reaches of a stadium, a network can be “future proofed” by being able to support many more new devices on the end of each fiber strand. By not having to string new cabling everywhere to support greater demand, a stadium will theoretically spend far less money in the long haul.

Wi-Fi and DAS will have the bathroom lines covered

From a network backbone perspective, the Falcons took what Texas A&M did and pushed even farther with fiber, taking the glass circuits as close to the edge devices (mainly Wi-Fi APs) as possible. In our mid-August press tour at Mercedes-Benz Stadium, we saw many of the so-called “mini-IDFs,” small closets with three or four pieces of gear in them, mounted on walls throughout the stadium.

“We kept the use of copper as short as possible,” said Miller. “With bandwidth demands continuing to grow at an exponential rate, we need to make sure we keep pace with rapidly evolving technology.”

If there is one big drawback to using fiber, it has to do with the intricacies of dealing with the construction end of building fiber networks, since the cables need to be precisely cut and joined, often with highly specialized equipment. There are also far more network technicians who are trained in copper wiring deployments than in fiber, so personnel issues can also increase costs and complexity.

On the eve of the Falcons’ first regular-season home game, a mid-September playoff rematch with the Green Bay Packers, Miller and his crew had not yet provided any traffic or network throughput statistics from the preseason games at Mercedes-Benz Stadium. However, some inside sources told MSR that the Wi-Fi traffic during the preseason and the early September college game was at the top range of what has been historically seen for NFL game days, so it appears that the system is ready to go.

Holding off on more bells and whistles

Though it may be hard for any other stadium to top the halo board for a while, on some other technology-related items the Falcons and Mercedes-Benz Stadium are taking a step back and not pushing digital solutions to where they might not be needed.

One of the many ‘mini-ISFs,’ this one in the press box

On the stadium app side, for instance, the IBM-developed application does not support some services seen at other NFL or pro-league stadiums, like “blue dot” wayfinding or in-seat food delivery or even express pickup for concessions. Instead, the app uses wayfinding based on static maps, where you need to put in both a location and a desired destination; and on the food-ordering side, fans can enter in an order and their credit card information, but must then take it to a stand to be scanned and fulfilled.

And while the Falcons’ new app does have a fun FAQ chatbot called “Ask Arthur” (for the team’s owner, Arthur Blank), there won’t be any live instant replay features in the app. With all the video screens in the stadium, the Falcons think they have the replay angle thing covered. The Falcons will use the app, however, for expanded digital-ticketing features as well as to help fans find and pay for parking. On the concessions side, the Falcons’ well-reported “fan friendly” pricing with low costs for most stadium food staples might prove more interesting to fans than being able to have food delivered.

Miller was also adamant that fans won’t see any portal or other marketing messages between finding and connecting to the Wi-Fi network.

“You just join ‘AT&Twifi’ and you’re on,” Miller said. “You’re a guest in our house. The last thing we want to do is slow you down from getting on the network.”

The big metal falcon in front of the stadium looks out over downtown Atlanta

More Wi-Fi APs visible under the overhang

A view from the AT&T Porch out through the windows

Niners: No more in-seat merchandise sales or express pickup at Levi’s Stadium

We finally have an official statement from the San Francisco 49ers talking about the team’s decision this season to eliminate in-seat food delivery for the entire stadium and instead offer the service only to those holding club-level seats, news reported first by Mobile Sports Report.

A statement provided to us from Al Guido, president of the Niners, says:

Levi’s Stadium was the first stadium of its size to offer in-seat food and beverage delivery throughout the building, a service no other venue has attempted to date. After conducting a comprehensive offseason review, including analysis of the sections where people took the greatest advantage of the offering and surveys on what matters most to fans sitting in different areas of the venue, we are focusing our in-seat food and beverage ordering service exclusively to non-inclusive Club seating sections for the current season. This change is being done to improve the overall concession service in the venue while allowing us to continue to enhance how this feature can best benefit our fans.

According to the Niners, “non-inclusive” means all club seats where fans don’t have access to the everything-paid-for clubs like the BNY Mellon Clubs on either side of the lower 50-yard-line seats. So to have access to the app-based ordering and delivery, you need to be in a club seat somewhere in the 100 or 200 levels of the stadiums, without a paid-for food and drink plan. The Niners did not provide any numbers about how many or what percentage of Levi’s Stadium fans would continue to have the ability to order concession deliveries.

Also discontinued is the Levi’s Stadium express pickup option, a service the team said was actually stopped last season. The express pickup option allowed fans to order and pay for food and drink ahead of time, and then pick it up at a nearby stand. The Niners also said they are discontinuing the ability for fans to order merchandise and have it delivered to their seats, an option that debuted back in 2014.