T moves to eliminate website crashes

Storm-related snags drive online upgrade

By Eric Moskowitz
Globe Staff / February 22, 2011

E-mail this article

Invalid E-mail address
Invalid E-mail address

Sending your article

Your article has been sent.

Text size +

The MBTA, acknowledging that a series of website crashes exacerbated its service problems this winter, has made several technological and personnel changes in an effort to make it possible for commuters to get better information about delays and disruptions.

A spike in Web traffic around snowstorms overwhelmed the T’s site, which failed three times in three weeks, shutting out thousands of people looking for service alerts, schedules, and route maps.

“When we had our first crash I was concerned,’’ said Richard A. Davey, general manager of the MBTA. “But when we had our second — well, our second and then our third — I was very frustrated.’’

The site crashes added to the stress and exasperation for people trying to navigate to work or home on the T — many heeding official advice to avoid driving during a snow emergency — at a time when the weather meant that rail and bus service was inconsistent.

The failures also vexed MBTA officials who have talked about the importance of enhancing communication with customers, while also trying to address well-documented problems with the debt-ridden transit service itself.

Davey called the website “a core part of our business’’ and “critical to our mission.’’ A reliable website could also bring more revenue to the T; Davey has asked the marketing department to explore selling advertising on the website as early as this summer.

“We need to make sure that we are getting customer information right 100 percent of the time,’’ he said. “Because if we know there’s always going to be some issue with service, we need to be in a position to be able to convey that information. . . . My expectation is that the website will be up and running all the time — in particular in critical situations.’’

In an age of real-time information and instant access, people expect websites to work every time.

“We’ve gotten used to there being no busy signals on the Web,’’ said David Weinberger, a senior researcher at Harvard University’s Berkman Center for Internet & Society. “And that’s especially important when the site is providing information that we need in an emergency.’’

Similar outages at a commercial website caused by a spike in traffic — amid, say, buzz over a Super Bowl commercial or a news story about a new product — would cause financial and public headaches. “If it were a commercial site that failed three times, it would be able to measure its losses in very concrete terms,’’ Weinberger said.

In the aftermath of the site crashes, the Massachusetts Bay Transportation Authority has made a series of changes — to hardware, software, and personnel — to try to prevent a recurrence.

“I have a lot of confidence that we will be back to where we were before January, where basically you can count on the site every day being there,’’ said Gary S. Foster, chief technology officer for the MBTA, who oversees not just the website but all network and system hardware and software, including the computing behind the CharlieCard.

Foster, who joined the T three years ago from the private sector, said until last month the site had not experienced a significant outage during his tenure. But the T was taken by surprise by recent Web traffic that defied predictions, he said.

For years, heavy traffic at meant 75,000 visitors in a day — compared with roughly 1.3 million riders. The T cracked six figures just once: when 100,218 went to the website during a snowy and sleeting First Night, on Dec. 31, 2008.

Nothing seemed unusual at the start of this winter. Traffic with the first major storm set a record, with 106,236 visitors Dec. 27, but it was nothing the T couldn’t handle. During the next three storms, though, traffic nearly doubled.

Part of that, Foster said, came when the media — including — directed people hungry for travel information to the T’s website, and thousands clicked in a matter of minutes. The T was caught off guard.

The three-hour failure on Jan. 12 was largely caused by the T’s attempt to redesign the four-year-old “Service Alerts’’ box on the homepage. But that required the database server to run more queries than it could handle, and most visitors got a blank page when they tried to visit during three peak morning hours in the storm.

The T responded by reverting to the old box and tweaking the database, but it had not finished by the next major storm, Jan. 27. The site then crashed for about an hour and a half — again during the morning rush — and the T estimates that as many as 40,000 visitors may have been turned away, on top of the 144,000 who made it through.

Bloggers scoffed, and one TV station described the website as “surprisingly inept.’’ Davey told Foster he wanted an “action plan’’ identifying shortcomings and necessary upgrades, and he removed the webmaster. (The person has been “reassigned,’’ Davey said last week, and is now working elsewhere at the T.)

That weekend, the remaining Web squad, supplemented by others from the MBTA and Department of Transportation IT teams, worked to upgrade and test the site. They also purchased twice the database-processing power and decided that, starting with the next storm, they would have a live person at the office to monitor the website at all hours in a storm.

But fell again Feb. 1, this time in the afternoon rush. The database held up, but the site experienced a bottleneck. Twenty thousand may have been turned away, but still a record 189,259 made it through.

That day, the T made an emergency purchase of two more Web servers and began to reconfigure its load balancer. With sleet following snow the next day, Feb. 2, the site held up all day — amid 170,520 visitors.

The work continues. It may not be complete this week, but it should be finished soon, said Foster, whose goal is to develop twice the capacity the site could ever need. The T has also taken other measures to get information out in a storm, including through Twitter (@mbtagm), a stripped-down website for smartphone users, and by posting copies of its service alerts on the MassDOT site.

And as many as 80,000 smartphone users bypass the T’s site on a busy day to use free or low-cost bus and subway applications produced by third-party developers, using real-time data released by the T.

In the meantime, officials are eager for spring. “Music to my ears,’’ Davey said.

Eric Moskowitz can be reached at