NEXT time you're tapping off a private text message or sensitive email in a public place, consider this: someone could be reading every letter you type from up to 60 metres away.
"We can be in the second floor of a building and read a phone on the ground," says computer vision researcher Jan-Michael Frahm, of the University of North Carolina at Chapel Hill.
Frahm and Fabian Monrose, also of UNC-Chapel Hill, have built a program, dubbed iSpy, that can identify text typed on a touchscreen from video footage of the screen or even its reflection in windows or sunglasses. Video from an ordinary mobile phone camera can be used to spy on a person from 3 metres away. And a snoop with a digital SLR camera that shoots HD video could read a screen up to 60 metres away.
The researchers wondered if modern computer vision technologies threatened mobile phone privacy. So they created a program to steal text remotely, using only known techniques.
Their method exploits a feature meant to aid typing on small touchscreens: magnified keys. Letters on a virtual Android or iPhone keyboard pop up in larger bubbles when pressed. The program analyses video footage and identifies the letters based on the bubble locations on screen. Pop-ups for neighbouring letters like E and R can overlap, so the program assigns an accuracy probability to each detected letter. The program correctly identifies letters more than 90 per cent of the time, Frahm says.
The software then identifies words, both individually and in the context of the message being sent. In one test, the team spied on a colleague at a bus stop outside their building, while others around him were also typing on their phones, Frahm says. "The attack is very realistic." To capture passwords, the software simply collects letters and does not perform any word recognition.
"We were surprised at how well that worked," Frahm says of the software, which was presented at the Conference on Computer and Communications Security in Chicago.
Reflections are harder to decode because the screen image is smaller. Still, the program can identify text from video taken with a digital SLR camera from a distance of 12 metres.
Frahm plans to continue using his iPhone, although he is more aware about what he types if he is not in a private place. "At this point, the [phone's] benefits outweigh the threat," he says.
To thwart sneaky snoops, the team suggest disabling the letter magnification feature, or developing privacy screens much like the plastic shields that cover automated teller machine number pads to shield mobile devices.
"This creative and non-intuitive attack proves yet again that we cannot become complacent when adopting new technologies," says Avi Rubin of Johns Hopkins University in Baltimore, Maryland. Rubin recommends taking precautions, like blocking the screen with your hand or body, when using sensitive applications on a mobile phone.