Improving on reality

Want to know about a tourist site? Point a phone at it

By D.C. Denison
Globe Staff / September 7, 2009

E-mail this article

Invalid email address
Invalid email address

Sending your article

Your article has been sent.

  • Email|
  • Print|
  • Reprints|
  • |
Text size +

It has not been easy for seventh-grade teacher Virginia McNally to explain to people what she learned during a two-week “augmented reality games’’ workshop at MIT this summer.

“What I say now is that I was making video games that work with the real world,’’ she said. “They are games, but you play them out in the world, instead of on the couch.’’

Soon, McNally may not have to struggle with unwieldy explanations. Recent developments are pushing “augmented reality’’ toward the mainstream. Until recently, the term has mostly been used in research laboratories and in science fiction to display information over live images of the natural world. Now it’s cropping up in demonstrations of mobile phone applications that have already been launched or are on the way.

Mobile phones running Google’s Android platform, for example, are already capable of displaying information over scenes viewed through the phone’s camera. An application called Layar can display Wikipedia entries over an Android phone’s camera view of a scene. It can, for example, add information about Plymouth Rock when the smartphone is pointed at the landmark.

Apple iPhone enthusiasts are eagerly anticipating the next software upgrade to the iPhone 3GS, expected this fall, which will enable new augmented reality applications. When a user points the phone’s camera at an eating establishment, for instance, recent reviews could appear over the image on the screen. Pan the camera to a nearby sculpture, and the display changes to show information about the artist. Keep panning to a nearby store, and sale items pop up.

Already, a British software development group, Acrossair, has created two apps that enable users to display subway information over real-world images of New York and London. The company is promising a general release as soon as the iPhone upgrade goes live.

Some companies aren’t waiting for Apple - they are adding augmented reality features to their existing iPhone applications by hacking together third-party software. Yelp, the social review service, has enabled an “easter egg’’ that lets iPhone 3GS users overlay information about restaurants and other nearby businesses on top of the phone’s camera view.

After years of working with specialized portable computers, or constraining their ideas to run on limited platforms, software developers and entrepreneurs are focusing their augmented reality efforts on mobile devices. That’s become possible because today’s smartphones boast more powerful processors, global positioning software, high-resolution cameras, and the ability to detect direction and orientation. In addition, a new generation of sensors and bar codes give mobile devices more data to work with.

“Over the last three months we seem to have reached the threshold of what was recently out of reach,’’ said Eric Klopfer, an associate professor at the Massachusetts Institute of Technology in Cambridge and director of the school’s Teacher Education Program, which organized the workshop McNally attended. “We’re on the cusp of a whole new era in augmented reality.’’

Klopfer, who has been designing augmented reality games since 2001, believes looking at annotations on top of real scenes is a more vivid experience than cross-referencing information on a screen with a location by going back and forth between data on the device and the scene in front of the viewer.

“There’s a big difference between looking down at a device and reading, ‘Stand in front of St. Paul’s Cathedral, face north, now turn to the right . . .’ and looking at the real world through a screen that augments reality by simply overlaying information on top of it,’’ Klopfer said.

The MIT Teacher Education Program is using augmented reality to develop games and tool kits to allow students to explore real locations with hand-held computers. The devices display information that highlights environmental and scientific issues that are relevant to the student’s location - like creating games that challenge students to find information overlayed on places in their neighborhoods. But other groups at MIT are working on augmented reality projects in different fields that combine data with live views of locations.

One research team, based at the MIT’s Media Lab, has been working at the other end of the connection between hand-held mobile viewer and the real world. The Camera Culture group, directed by associate professor Ramesh Raskar, recently came up with a system called Bokode that is based on a new way of encoding visual information using light rays from a bar code-like tag. The tags could be used to label items and locations with data that could be read by devices, such as smartphones. Bokode tags contain more data than a standard UPC bar code. That could allow visitors to a museum to read information about a display just by pointing a phone at it.

“The cameras in mobile phones will continue to improve dramatically,’’ Rakash said during a demonstration of the Bokode system.

At this summer’s MIT workshop, teachers from Watertown, Boston, and Lawrence gathered to learn how to use augmented reality to inspire middle school students.

“One of the real benefits of this technology is that kids think this is just the coolest thing,’’ said McNally, the seventh-grade teacher. “I’m always looking for ways to drag the kids outside, and augmented reality games get them engaged and excited to head outdoors. It’s like, ‘Let’s do this!’ ’’

Environmental Detective, an “AR’’ game developed by the MIT Teacher Education Program, is typical of the genre. In the game, which runs on a smartphone, students are told to find the location of a fictional toxic spill by analyzing data overlayed on an actual location, like a campus. The students uncover the information by walking around the campus, following clues displayed on their phones.

Klopfer said he expects educational applications of augmented reality to eventually be joined by more commercial projects.

“The first thing we’re going to see are the tourism apps,’’ he said. “There will also be all sorts of games, and all sorts of things we haven’t thought about yet.’’

The only challenge that remains, Klopfer said, is whether enough power will be available to drive all this interactivity.

“One thing that we haven’t solved is battery life,’’ he said. “You’re going to be out there on your phone using all these sensors and processing power - to drive the GPS, the compass, the Internet, et cetera. And after a few hours . . . you’re going to be out of juice.’’

D.C. Denison can be reached at