Stadium Tech Report: Miami Marlins rely on ExteNet DAS to keep wireless traffic flowing

Marlins Park. Credit all photos: Miami Marlins.

Marlins Park. Credit all photos: Miami Marlins.

If you know anything about Marlins Park, maybe it’s the stadium’s unique retractable roof or the spectacular art that catches your eye. But there’s also something you can’t see that is equally exciting, at least when it comes to the in-stadium connectivity experience: A neutral-host distributed antenna system (DAS) that has more than kept pace with the rapid, continual increase in fan cellular activity.

“When it came to DAS, we were ahead of the game,” said David Enriquez, senior director of information technology for the Miami Marlins, in a recent phone interview. Well before the 37,000-seat stadium opened in 2012, Enriquez said the Marlins’ IT team was researching and planning for enhanced cellular connectivity – even before “DAS” became a hot industry acronym.

“We planned for a DAS even before they were in vogue,” said Enriquez. “We saw it as a necessary evil.”

With the iPhone and all its cataclysmic changes already in motion, Enriquez said the Marlins wanted to avoid what had happened recently at another arena that opened in the Sunshine state without good connectivity.

“What we didn’t want to see was something like what happened in Orlando, when they opened the arena [in 2010], it had bad coverage, and they were crucified in the press for bad [cellular] service,” Enriquez said. “We said, what we’d love to have is the complete opposite of that.”

David Enriquez

David Enriquez

At the opening of Marlins Park, the connectivity inside the walls was better than most, with a full-park Wi-Fi network using gear from Meru Networks and a neutral-host DAS deployed by integrator ExteNet Systems. And though Wi-Fi often gets the headlines when there is talk about stadium networks, in many facilities like Marlins Park, the DAS is an equal workhorse, since many fans still either don’t know how or don’t take the time to switch their devices over to Wi-Fi.

DAS is the workhorse

According to Enriquez, on an average night at the ballpark the Wi-Fi network will handle 40 percent of the wireless traffic, with the DAS taking care of the other 60 percent. That may be because of lack of knowledge, or perhaps satisfaction with the signal the DAS is giving them, Enriquez said.

“Early on, most people, honestly, did not know how to change [their phone] to Wi-Fi,” Enriquez said. Most fans, he added, weren’t typically streaming lots of video — they may, he said, have used the MLB At Bat app to look at a replay or two, but that could all be handled by DAS. “That trend is changing though and we are seeing much more video traffic, especially with the younger generation of guests,” Enriquez said.

Marlins Park outside

Marlins Park outside

“The truth is, many users may not take the time to switch [to Wi-Fi],” Enriquez said. “If they’re getting 4 to 5 bars on their cellular signal, they’re happy.”

Though the Marlins and ExteNet now have five major carriers on their DAS – AT&T, Verizon Wireless, Sprint, T-Mobile and MetroPCS (now part of T-Mobile), Enriquez said there was a bit of the chicken and egg problem at the start.

“Early on, nobody wanted to be the first on (the neutral DAS),” Enriquez said. “ We [the stadium] were just another node. Now, 3 years later, we are a central node in the Miami area and all the carriers are here. We’re a very central location.”

Staying in neutral

Enriquez, who has considerable experience in the large-venue IT world, said that having a neutral host for the DAS eliminates any potential concerns about favoritism between service providers. Even though costs to the team or stadium may be lower if they allow a carrier to take over DAS deployment, Enriquez said that for the Marlins a neutral host was worth the extra price.

“We didn’t want an advantage to be held by one carrier,” Enriquez said. Even if a carrier says it will act as a neutral host, when one carrier owns the deployment, others can “find it hard to believe there will be an equal time slice” when it comes to antenna access.

“We just wanted to avoid that, and make it irrelevant [as a concern],” Enriquez said.

The choice of bringing in an integrator like ExteNet, he said, provides an additional streamlining of operations, as there is now a single point for vendors to interact with to work out technology and deployment issues.

“We wanted to deal with one vendor – I didn’t want to be the middleman between the carriers and the Marlins,” Enriquez said. In that regard, he said, ExteNet has been “wonderful” as a neutral host. “They deal with all the carrier issues that I have no desire to deal with,” Enriquez said.

Less space needed for DAS upgrades

And even as fan cellular bandwidth use continues to grow – requiring carriers to constantly upgrade their systems – Enriquez said that DAS infrastructure is benefiting from improved technology to the point where even as carriers upgrade, their head end footprint is shrinking.

AT&T, for instance, has upgraded its DAS presence in Marlins Park four times over the past 2 years, Enriquez said, to the point where the carrier now has coverage for all four frequency bands. “They [AT&T] have done quite a bit to expand their coverage,” Enriquez said.

Still, the Marlins Park DAS head end hasn’t had to find new space beyond its original 1,500-square foot enclosure.

“Every time someone comes in to replace gear, we have a smaller [DAS] footprint,” Enriquez said. “It’s not going to eat you out of house and home anymore.”

Like other stadium IT directors, Enriquez is still surprised by the amount of wireless traffic generated by the fans who come to the games. “It’s incredible to see the need [for bandwidth” grow,” he said. “But people continue to give our network a thumbs up, we see that in our guest comments all the time. I just don’t know what we would do without the DAS.”

Stadium Tech Report: Upgrades keep San Francisco Giants and AT&T Park at front of stadium DAS and Wi-Fi league

Outside AT&T Park. All photos, Paul Kapustka, Mobile Sports Report. (Click on any photo for larger image)

Outside AT&T Park. All photos, Paul Kapustka, Mobile Sports Report. (Click on any photo for larger image)

What’s it like when the best-connected park in Major League Baseball loses its cellular mojo for a month? This winter the San Francisco Giants found out how fun it isn’t to revisit the days of “no signal,” when a DAS upgrade meant about 30 days of little to no connectivity inside AT&T Park.

“It was painful,” said Bill Schlough, senior vice president and chief information officer for the Giants, during a recent in-person interview at AT&T Park. Though no big sporting events took place during the Feburary-to-March overhaul of the main AT&T distributed antenna system (DAS) head end, Schlough said during that time many of the roughly 200 to 300 employees who work at AT&T Park every day were forced to find daylight to make a call, just like the bad old days before DAS.

“We never really knew how much we rely on DAS [for internal operations], but having it down really drove it home,” said Schlough. The good news on the DAS front was that once the upgrade was complete, the Giants had a lot more space in their previously cramped head-end headquarters. According to Schlough, the new back-end equipment for AT&T’s DAS operations takes up less than 50 percent of the previous gear footprint, room that is likely to be filled with gear from yet another carrier slated to join the AT&T neutral-host DAS later this season.

Painful, but worth it.

Second major upgrade in 5 years of DAS

Giants CIO Bill Schlough (left) talks with workers in the park's main DAS head end facility.

Giants CIO Bill Schlough (left) talks with workers in the park’s main DAS head end facility.

If you’re not familiar with a neutral DAS like the one at AT&T Park, it’s an implementation where there is one set of antennas and internal wiring, and then a “head end” where each carrier puts its cellular-specific networking gear, equipment that identifies and authorizes callers and then connects those calls or messages to fiber links back out to the Internet and beyond. As the lead provider of DAS and as the namesake sponsor of the park it makes sense that AT&T has the biggest DAS requirement on site. Verizon, which has been on the AT&T Park DAS for two years now, actually houses most of its head end gear in a separate facility nearby, and links to the AT&T Park system via fiber.

Part of this year’s DAS renovations include a new room specifically being built for Sprint’s DAS equipment, a sort of re-arrange-the-house construction project that saw the ballpark wall off half its painting services workshop to make space for Sprint’s gear. During our visit we saw workers putting up the racks that will hold the Sprint head end gear, as thick fiber cables snaked in the doorway.

Additional carrier(s) would likely be placed in the same room as AT&T and Verizon, on floor space that used to hold AT&T racks before those were un-drilled from the concrete floor and new racks were installed during the February-March overhaul. According to Schlough, the DAS upgrade (which required minimal tweaks to the previously installed DAS antennas) was the second major rip-and-replace action in the 5 years the DAS has been live at AT&T Park.

DAS performance improves over time; Wi-Fi is good too

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

Though Wi-Fi services in stadiums gets a lot of technology headlines, in many big arenas the DAS is an equal workhorse, connecting people who either don’t know how to or prefer not to connect to Wi-Fi services. Through the first 18 games of the 2014 season, Schlough said AT&T Park was seeing average AT&T traffic loads on the DAS of 150 Megabytes on the download side (fans requesting data) and 50 MB on the upload side (fans sending data). Figures for the Wi-Fi network (which is free to all customers) for the same span of games was an average of 400 MB download, 200 MB upload per game.

Schlough said performance stats for the AT&T portion of the DAS have improved vastly since the distributed antenna system was first put in.

“Just four or five years ago, 97 percent [connection rate] was actually relatively respectable,” Schlough said. Now, Schlough said network connect rates regularly hover in the “four nines” region, with a recent report showing a success rate of 99.9925 percent of all calls or texts going through.

The Wi-Fi network at AT&T Park, the first in any major sporting arena and still among the world’s most expansive, has more than 1,200 access points, many of which are now located beneath the seats. According to Schlough this coming offseason will likely represent the final phase of a stadium-wide deployment effort for the new, under-seat access points, which are installed symmetrically under the seats that are out in the open air.

Giants senior VP and CIO Bill Schlough, at the office

Giants senior VP and CIO Bill Schlough, at the office

Since AT&T Park doesn’t have many railings alongside the seats “in the bowl” or those in the upper decks, the under-the-seat APs were the only choice to extend Wi-Fi connectivity, he said. Though the box-like antennas do take away some under-seat storage area from approximately every 40th seat, Schlough said there haven’t been many complaints from fans about the gear.

What he has seen, however, are many compliments about the network services, especially from fellow professionals in the sports IT world.

“I get friends in the business who come here and send me texts with Speedtests attached, showing how great the Wi-Fi is,” said Schlough. My own ad hoc testing before our interview (albeit during non-game hours) showed speeds of greater than 40 Mbps for Wi-Fi just outside the park near McCovey Cove, and speeds of 25+ Mbps just outside the main gate. Schlough also showed us some of the new iBeacon antennas, which are being tested at MLB parks this summer to provide near-field communication marketing opportunities, like automatically checking fans in to the official At Bat app when they pass by a beacon. It’s just another way the best-connected park in baseball seeks to continue to improve the fan experience.

According to Schlough, the connectivity at AT&T Park doesn’t hurt when it comes to ticket sales.

“People do come here more frequently, I think, because they know there will be good connectivity,” said Schlough. “There’s no compromise. I do think that’s part of why we’re currently riding the third longest sellout streak in MLB history.”

MORE PHOTOS BELOW — CLICK ON IMAGES TO SEE LARGER VERSION

Can you find the iBeacon in the bowels of AT&T Park? It's the small grey box to the left of the other antenna.

Can you find the iBeacon in the bowels of AT&T Park? It’s the small grey box to the left of the other antenna.

Sprint's new DAS room at AT&T Park.

Sprint’s new DAS room at AT&T Park.

A close-up of the under-seat AP. Each AP requires holes drilled through concrete to provide wiring access. APs are weather-sealed, according to the Giants.

A close-up of the under-seat AP. Each AP requires holes drilled through concrete to provide wiring access. APs are weather-sealed, according to the Giants.

Bill Schlough's "old phones" collection. How many of these can you identify?

Bill Schlough’s “old phones” collection. How many of these can you identify?

MLB streaming comes to Google Chromecast


You can now add Google Chromecast to the platforms that stream Major League Baseball broadcasts in the U.S. Subscribers of MLB.TV who own one of the Chromecast sticks that plug into the back of many modern televisions will be very happy.

MLB.TV enables fans to watch home and away games of out of market teams and once subscribed can use not only the Chromecast but also smartphones and tablets to watch games. Be sure to check what qualifies as an out of market team since some areas such as Las Vegas are claimed by multiple teams.

English Premier League gains additional network partners for finals
The Premier League will now be available on additional channels as NBCUniversal has opened up its family of stations to the broadcast of the league’s finale that will consist of 10 matches held on May 11.

It now plans to simultaneously broadcast the games not only on NBC and NBCSN but also on Bravo, Syfy, Oxygen, USA, CNBC, MSNBC, Esquire Network and E!, channels better known for broadcasting anything but exciting soccer.

Rate the MLB announcers
It may be a bit late to join all of the fun but over at Awful Announcing they are having a contest to rate all of Major League Baseball’s individual broadcast teams. It has been ongoing for a few days but is still worth checking in to see how your favorite, or least favorite voices are doing. Do you like or detest homers, are some too bland, off message or just plain head scratching? Time to make your opinion known! The national broadcast teams on stations such as ESPN and Fox Sports will not be included in this poll.

Google lays out details for modular smartphone
Last year when Google first started talking about its “Project Ara” we did not pay that much attention since it was basically just some mockup photos and not much detail. Last week the company put meat on the Ara bones and started explaining what its intentions are in that area.

The goal of the project is simple but grandiose; it wants to revolutionize the smartphone market, and in 1 year. The idea is very simply, a phone that has a number of replaceable components that a user can select to include in their version of the phone. Think of a Lego phone as a comparable. Users could swap in processors, memory, storage and even type of connectors.

USA Today offers rare sports prints
USA Today has launched an endeavor called the USA Today Sports store and to kick it off it is offering customers a chance to buy a select number of images taken of Muhammad Ali early in his career that originally came from the Courier-Journal in Louisville.

While these images will be available for a silent auction fundraiser to benefit the Muhammad Ali Parkinson Center at Barrow Neurological Institute this weekend there are a large number of images available at thestore.

RBI Baseball is here; Best ballparks for craft beer

MLB has released R.B.I Baseball 14 to the joy of long-suffering fans of the game who have been waiting and hoping for years that the title would be reinvented and reinvigorated for a new generation of fans as well as older fans.

The game is the first one for consoles developed by the MLBAM and is available as a digital download for Xbox 360 and PlayStation 3 entertainment systems as well as for iPhones and iPads. The original game was discontinued around 20 years ago.

Quidditch is a real thing?
Apparently not only can you really play quidditch, there is an actual World Cup for the sport. The game that was made famous via the Harry Potter franchise of books and movies, sans magical brooms and a few other minor details, is being played around the globe these days.

Not only is it being played but it is gaining a bit of international support as teams from Canada and Australia will be among the 80 teams participated in the sport’s seventh World Cup in South Carolina last weekend. As a spoiler the University of Texas retained its crown beating Texas State University. Time to start practicing for next year.


Cubs’ fans disappointment may continue for 6 more years

When a new owner comes to town to take over a forlorn franchise there is always hope that they will manage to achieve what so many have failed to do in the past, create a winning atmosphere that will help make up for past failure.

That is what Cubs fans were certainly hoping for when Tom Ricketts et al took over the team. Yet it seems that a stadium and team that do very well in attendance and broadcasting viewership need to wait for new contracts and rebuilding. Or it could be the way the loan is structured and they don’t want to talk about that.

The 10 best ballparks for craft beer
When I started going to baseball games the options for food and beverages were very limited but that has really changed for the better. Now the Daily Meal has taken in upon itself to track down which stadiums have the best craft beer — why is this not a job assignment that I am ever given?

They list #1 as Safeco Field in Seattle, and the overall West Coast is well represented with AT&T and Petco also listed. Also did you know that at Yankee Stadium the space where they sell crafty beer is called ‘beer mixology destination?” Beer Mixology, really?

Qatar World Cup construction worker death rates continue
The World Cup that is due to be played in Qatar has already been under a cloud due to complaints about corruption and the fact that the tournament might have to be moved from summer to winter due to, what else, hot weather. But one issue that also seems to be running under the radar is worker treatment.

According to a piece in Slate 1,200 workers have already died building the facilities for the 2022 event. However FIFA said that it would look into it, which since it is responsible for the awarding of the games in the first place seems like the fox watching the hen house.

Wilson brings digital data collection to basketball
l
Golfers can get a version of a smartwatch that tracks their game and provides real time feedback, there is a host of apps for runners and bicyclists to track their efforts and skiers can get goggles that lay out the run and show speed and slope. Soon recreational basketball will join the digital world with an app that will help Sunday morning athletes track their efforts.

There is more than just an app in the Wilson Smart Basketball program; there is of course a basketball that helps feed in the data. The app and accompanying basketball are scheduled for release for this year’s holiday season.

CBS to get ‘strong’ Thursday Night NFL slate
After outbidding rivals for the newest set of games that the NFL will be playing on Thursday nights CBS appears to be receiving an extra with the deal. It has been reported that the NFL will be taking steps to make those games between what are viewed as strong teams, not necessarily quality teams, but ones that will bring strong viewership.

Of course the quality of the games at the early point of the season, when offensive lines have not yet jelled, rookies are still learning the playbooks and teams have their new coaches is sometimes average at best and add in the fact that teams really only get two days to prepare for Thursday games I am not sure I would put the quality tag on any of these games just yet.

Olympics great for NBCSN; Are mock NFL drafts worthwhile?

The Sochi Winter Olympics are now in the rear view mirror as the sporting world now looks to the next major event, the World Cup, yet NBCSN is ready to continue basking in the winter event’s limelight for just a bit longer.

The network reported its best ratings quarter ever with a 215,000 average viewership daily for the live events that it broadcast during the two-week run. This represents a 231% increase over its 2013 numbers. Even subtracting the Olympics it saw a 58% increase and may indicate that the little-known channel may finally be getting traction with viewers.

Dodger fans can’t watch Dodgers games
With the huge growth of regional sports networks it seems that every MLB team has its own, dedicated network to bring its games to its fans. Somehow it does not always work that way, as New Yorkers can attest from a few years ago.

Now it appears that the fans of the Los Angeles Dodgers are facing viewing issues as its RSN, SportsNet LA, has had issues getting cable players to air the games, causing as much as 70% of greater LA to be blacked out.

Unconscious bias by umpires?
An interesting piece from fivethrityeight.com looks at strike zones but importantly how it can change, and change dramatically in important situations as umpires striving to ensure that they are not a part of the game instead do alter the outcome.

The article can be a bit confusing for some when looking at the graphs but it has some interesting conclusions including that umpires seem happier to call a strike on a borderline pitch with three balls than ball four. Head over and give it a read.

WatchESPN on Amazon TV Fire
Amazon has recently entered the broadcast delivery business with its Amazon Fire TV, a platform that will vie against Apple TV, Roku and others, but sports fans need not worry because ESPN will be there ready and waiting if you adopt the platform.

WatchESPN is available on the Fire TV and using it fans can access a huge amount of live and recorded sporting events along with ESPN’s select offerings of the channels original sports programming such as E:60 and Outside The Lines.


Mock NFL Drafts

The NFL draft is less than two months away and so now is the silly season for a growing industry — NFL mock drafts. You know, when an athlete who has not played in months magically goes from a first rounder to a third rounder on someone’s big board for no apparent reason.

The good folks at Kissing Suzy Kolber have taken the industry to task and show why you should pay little attention to the mock drafts. They point out the inanity of the event and it has always seemed to be that the local beat writer for a team has a much better feel for what your team will do than some talking head in a studio will — that is unless the talking head is using other peoples’ work without recognizing it and we know that never happens!

Friday Grab Bag: NBA first to adopt ads on jerseys?

Apparently it is just a matter of time. ESPN is reporting that new NBA commissioner Adam Silver told an audience at the IMG World Congress of Sports that the move to put ads on NBA jerseys is inevitable and that it will enable its marketing partners to get closer to fans. I guess that translates into owners will be able to take home more money.

Ads on U.S. pro team uniforms has been contentious — MLB has talked about it for more than a decade and in a game in Japan rolled out the look to wide displeasure but it seems inevitable. Teams are always looking for additional revenue and this looks to be money just left on the table.

NFL to have official to official communications
According to MMQB the NFL will equip all NFL on-field officials with a microphone, earpiece and a radio pack so that during games they can communicate wirelessly over an encrypted system to each other for a more efficient game.

I wonder in this day when people can hack into store accounts how long it will take for some person or persons to hack the communications between officials and either broadcast it somewhere or interfere with the chatter?

Buffett wants bracket changes for tournament
If you are like everybody else I know your March Madness bracket was blown up during last weekend’s round of major upsets but not everybody was unhappy — Quicken Loans and Warren Buffett’s offer to pay $1 billion to anyone that picked all winners will go uncollected this year.

However they are not gloating and Buffett, who said that they plan to offer the $1 billion next year, wants to change it so that it will be easier to win. However he has not yet worked out how that can be done.

You can still win millions if you Beat The Streak
March Madness is not the only game in town as with the start of the MLB season there s also the launch of the 14th annual Beat The Streak fantasy contest, this year with a $5.6 million prize, and hopefully someone will finally win this very hard to attain prize.

The Beat The Streak sponsored by Dunkin Donuts game itself is very simple; all a fan has to do is select two players every day and hope that one gets a hit, for 57 consecutive games, breaking baseball’s historic single season hitting streak. Good luck.

Maryland’s departure from ACC gets even more acrimonious

Maryland is one of the many schools that has shopped for a better deal in its collegiate alignment and announced 2 years ago that it was departing from the ACC for the Big Ten in search of its pot of gold. The ACC responded by suing to collect an exit fee.

Now Maryland is striking back and has subpoenaed 10 conference schools and ESPN claiming that the ACC violated its own rules on exit fees and that along with ESPN it tried to lure Big Ten schools, according to the Washington Post.