Harvard class project compares iPhone, Windows 7, Android, Blackberry usability; triggers a wave of invective

11.08.2011

The project was simplicity itself: All the students owned smartphones and they lent them to each other to test the three tasks. Each tester had no previous experience with that phone or its UI. The intent was to see whether and how users could figure out each task, without any familiarity with the phone or operating system and without any instruction or guidance.

Each user was videotaped performing each task in sequence and the resulting video was edited to keep it the 10-minute limit imposed by Galletta. The on-screen narrator is one of the students involved in the project.

In the video, the ratings for each task on each phone seem loosely correlated with the number of touches and taps and the length of time each task takes. But the actual criteria are not spelled out. The demonstration is organized by task, so we first see each user trying to make a phone call, then in the same order trying to add a contact, and finally sending a SMS message. Each separate test by the different users shows a close-up of the phone on a flat surface, and the hand or hands of the user.

Taken at face value, the video is most intriguing when one tries to follow the thought process of the novice user, as revealed by his or her touches and taps to the phone's screen. For example, the first test shown is making a phone call with a Samsung Focus, running Windows Phone 7. The user taps correctly on an icon, or "tile," with the image of a phone's handset. But what next appears is the "History" page of recent calls. The user apparently doesn't readily see how to make call from that page, and presses the home button to go back to the homescreen. After trying out a number of other actions, the user taps the phone tile again, finds another soft button at the bottom of the history page and is finally able to type in a phone number and make the call.

That experience won the Samsung device 3 out of 5 possible stars. The iPhone 4, the BlackBerry Storm and the HTC Thunderbolt were all much faster, with just one or two touches needed to bring up the dial pad.