Indeed, with the right technology, the right policies and a little slice of your budget, you can pretty much manage the messaging madness. And new technology likely to emerge from the labs in the next year or two will help bring a little more civilization to the world of e-mail, ensuring its continued place among the most popular and important of all corporate applications.
However, e-mail's problems will accompany it into its second act, especially as users deploy a growing variety of mobile devices and discover new ways of communicating -- such as instant messaging, blogs, wikis and virtual reality spaces you've never even dreamed of. These will offer green pastures for hackers, spammers and phishers, and will require a whole new round of defensive tools, techniques and policies.
While today's efforts to improve e-mail are aimed mostly at curing its ills, research in vendor and university labs points to brave new uses for the humble e-mail message, from knowledge mining to workflow enhancement. Interviews with researchers, futurists and IT managers yielded the following conclusions about the future of e-mail.
1. New technologies, plus economic and political pressures, will eventually tame the malware.
Ray Tomlinson, a principal engineer at BBN Technologies in Cambridge, Mass., calls the struggle against spam, phishing and malware "pretty much a draw" at present. He has a good deal of perspective on these issues, having sent the world's first network e-mail message in 1971.
Tomlinson points with hope, but some exasperation, to alternate -- some would say competing -- proposals for stemming the tide of offensive, malicious and deceptive e-mail.
"It's not so much a hard technical problem; it's a hard business and political problem," Tomlinson says. "The players have vested interests in the various approaches, and they are fighting tooth and nail to get their approaches adopted. It's not the end users who are the bottleneck here."
Microsoft Corp. is pushing its Sender ID Framework, which verifies that a message was actually sent from a server authorized to send mail for the domain owner. John Scarrow, Microsoft's general manager of antispam and antiphishing strategy, says Sender ID has been adopted by 73 percent of Fortune 100 companies and is used for 31 percent of all e-mail messages.
"We are seeing the amount of spam now starting to plateau," he says. "It's a good indication the industry is starting to take a good bite out of the economics of the business."
More good news, Scarrow says, is that while IM and other modes of electronic communication also need to be protected, the technology for doing so is similar to that for e-mail.
Meanwhile, Yahoo Inc. and Cisco Systems Inc. last year submitted to the Internet Engineering Task Force a proposed standard called DomainKeys Identified Mail (DKIM), which, like Sender ID, is designed to guard against spoofing and phishing by authenticating an e-mail sender. DKIM verifies the domain of the sender and also cryptographically verifies the integrity of the message.
In addition to Sender ID, Microsoft has the SmartScreen filter, which uses statistical techniques to learn what's spam and what isn't, and the Phishing Filter add-in for the MSN Search Toolbar. But those tools are not enough, say the folks at Microsoft Research, where some 40 people work on new e-mail technology.
For example, researcher Joshua Goodman says the ultimate solution could be a four-pronged defense against spam called SmartProof. Here's how an experimental version of it works:
-- First, a machine-learning filter, similar to SmartScreen, snags the obvious spam and quarantines it or throws it away. The filter passes on to the user's in-box any message that is from someone on the user's "whitelist."
-- Messages suspected of being spam trigger replies to the senders, challenging them to prove they're not spammers.
-- Senders may respond to the challenge by solving some kind of a puzzle -- one that's easy for a human but hard for an automatic spam generator.
-- Alternately, senders can ensure the delivery of their messages by making credit card-based "micropayments." The payments may go to the recipient, the Internet service provider or a charity, or they can be refunded to the sender if the message turns out not to be spam.
"We thought if we could put all that together, we'd have a great long-term solution," Goodman says. "Obviously, it's a very ambitious plan, and I don't think we ever thought it would happen quickly."
Elsewhere at Microsoft, researchers are working on a prototype called MailScope that monitors e-mail routes and alerts users when significant delays are expected. If MailScope sees persistent delays between, say, Microsoft.com and Berkeley.edu, it warns users on those servers that delays are likely, much as a traffic report notifies drivers of congested routes.
In a related Microsoft project called SureMail, when a message is sent, a system posts a tamperproof notification to a table somewhere on the Internet. E-mail recipients periodically query the table and match notifications with messages received. If they find a notification for which there is no message, they know the message has been lost. Microsoft calls these "silent" losses because they so often go undetected. In controlled experiments over two months, using a variety of e-mail systems and carriers, Microsoft found that one in 140 e-mail messages disappeared without a trace. Delays averaged four minutes but lasted as long as 27 hours.
Despite the extensive research and development, some observers say technology can never completely cure e-mail's ills. Economic and regulatory tools will be needed as well, they say.
"Ultimately, I believe there will be a pay-per-message type of service that charges to ensure that e-mail is spam-free," says CIO Matthew Lynch at ShopKo Stores Inc. in Green Bay, Wis. E-mail carriers will charge companies a penny or two per message and will in exchange certify those messages as legitimate, he says. Lynch also predicts "stronger legislation around this topic."
A combination of technology, policy and market measures will keep e-mail among the top of all corporate applications, most users say. "E-mail will continue to be an integral form of communication," says Matthew Marks, head of integrated user services at Aetna Inc. "The capability to quickly and easily distribute a message with an attachment -- documents, links, objects, etc. -- to a large, dispersed audience with tracking and audit cannot be matched by IM, fax or snail mail."
2. E-mail -- just one of the many communications streams in the workplace -- will become part of a "puddle," or "activity thread."
Although e-mail seems unlikely to be supplanted by alternatives, the job of the IT manager is nevertheless complicated by the emergence of other options.
E-mail is in its "pimply adolescence," says futurist Paul Saffo at the Institute for the Future in Palo Alto, Calif. The problems of spam, phishing and e-mail-borne malware will be conquered, he predicts. In the meantime, he cautions, "you can't treat e-mail in isolation. All of our communications forms are melting away, and we are creating new things out of the puddle of old stuff."
Richard Golden, vice president for IT infrastructure at Circuit City Stores Inc. in Richmond, Va., says these threats will cause corporations to augment their technology defenses with strong policy defenses. He says it's relatively easy to protect e-mail systems with spam filters, virus scanners and the like because the systems are well defined, with discrete messages going from Point A to Point B through corporate IT assets.
"But things are converging into a world that is not as clearly definable as a corporate e-mail system," he says. "I think you'll see more policies about things like blogging, for instance. As the lines blur on the means for communications, it's going to require more focus on the information conveyed, regardless of the means used to convey it."
IBM Research is looking for ways to combine e-mail with other functions and integrate it seamlessly into users' daily activities. "It's not enough to help people manage their e-mail; it's important to help them manage their work," says Dan Gruen, a research scientist at the company's facility in Cambridge, Mass. That involves "connecting all the communications and information feeds around a topic or activity," he says.
For example, an IBM Research proto-type called Activity Explorer is a collaboration tool that pulls together e-mail messages, synchronous communication such as instant messages, screen images, files, folders and to-do lists. A project team can establish "activity threads" containing these feeds and can switch easily between asynchronous and real-time collaboration. An activity thread might include the messages, chats and files exchanged among members of a team that's writing a contract bid, for instance.
A more advanced experimental tool from IBM called Unified Activity Manager does all that and more, linking into other corporate applications such as workflow systems. It not only combines the elements of a current activity but also pulls in those elements from past similar activities. These notions of "activity-centric collaboration" will show up in the next release of Lotus Notes, dubbed Hannover, which is expected to ship next year, Gruen says.
Meanwhile, Microsoft Research has developed a way to combine e-mail, files, Web pages, calendar entries, to-do lists and other materials into one searchable archive. Called "Stuff I've Seen," the prototype uses MS Search to index a user's important content and then offers it through a unified interface with sorting, filtering, previews and thumbnail views.
3. New e-mail applications will emerge, including tools that mine message archives for corporate intelligence.
Even as e-mail yields turf to upstarts like IM, especially among younger users, new uses for e-mail are on the horizon. As companies and individuals begin to systematically archive messages, the e-mail becomes available for data mining, and researchers at a number of companies and universities are developing ways to make these archives more accessible.
For example, Hewlett-Packard Co. researcher Bernardo Huberman is devising ways to "harvest organizational knowledge" by mining the e-mail messages and PowerPoint presentations of employees. His techniques go way beyond the searching and categorization of messages that products do pretty well now. Huberman looks at the strengths of communication bonds among employees and patterns of communication that can reveal both hidden problems and opportunities.
"You can look at an organizational chart and make all sorts of inferences about how people work, but when you look at e-mail patterns, you see how they work in a different way," he says. "You discover leadership roles, such as who's the hub through which most of the e-mails go, that you wouldn't identify from the organizational chart."
The result of such pattern or network analysis might be to reorganize departments, projects or activities around those hubs, Huberman says.
HP Labs is now prototyping a tool called Knowledge Navigator that's based on those principles. It applies text mining, clustering algorithms and statistical analyses to employee e-mails and presentations stored on HP's servers. It could handle a query such as, "Who are the top five experts on topic x?" Huberman says, even when such expertise is not explicitly noted in org charts or personnel records.
Huberman says this kind of knowledge harvesting will be used by companies internally on their employees and externally on customers, resulting in the ability to generate messages and pitches aimed at both groups. "What we will see in the next few years is a very targeted way of placing information in the hands of relevant people," Huberman says. "Sure, it can be annoying, but it's better than getting spam on things you don't care about."
Despite the benefits, he acknowledges that mining messages raises ethical and potential legal issues. "In the next few years, we will see a blurring of the boundaries between what is considered private and public," Huberman says.
Mining employee e-mails is "something the company has an interest in, and we are starting to see that interest grow," says Carl Jones, director of collaboration services at The Boeing Co. in Chicago. He says the company has a knowledge management pilot project that, among other things, examines e-mail messages.
"If you have a business problem, you may be able to mine across the e-mail spectrum and find out, hey, there are people out in the field who are subject- matter experts that can help you," says Jones. But, he adds, "we'll have to be very careful about policies on privacy and so on."
Jon Kleinberg, a professor of computer science at Cornell University , says much can be learned from the networks created by people's activities on the Internet.
"How can you infer that someone is influential?" he says. "Is it the obvious things, like they send and receive the most messages, or is it more subtle things, like they operate at the periphery [of a group] but pull together groups that are otherwise weakly connected?"
Kleinberg says answers to such questions may have profound importance for companies that sell online and rely on word-of-mouth recommendations via customers' e-mail. He's looking into two competing theories as to why that kind of e-mail sometimes leads to snowballing sales and other times fizzles.
"Is it the attractiveness of the product, or is it something about the community of people who are into those kinds of products?" Kleinberg wonders.
He says e-mail pattern analysis could help a company answer questions such as, "Who are the key people to influence?" and "For which products is it worth it, and for which is it not?"
"Social network analysis is one of the great tools for productivity going forward, and very few people understand it," says Thornton A. May, a Computerworld columnist and dean at the IT Leadership Academy at Florida Community College at Jacksonville. "People tend to think of social network analysis as a list of people -- an address book. But it should tell you not just who knows who, but who knows what as well."
Users should see social network analysis as more than a way to find dates or customers. It can "solve problems, create teams or recombine organizations," May says.
Sidebar
E-mail authentication: The choices
Some observers criticize IT vendors for not agreeing on a single, standard way for dealing with evil e-mail. The key e-mail authentication protocols are Microsoft's Sender ID Framework (SIDF), with its Sender of Policy Framework (SPF) records, and the rival Yahoo/Cisco DomainKeys Identified Mail (DKIM).
But a good case can be made that e-mail senders, Internet service providers and e-mail recipients should use both SIDF and DKIM.
"Domain owners are well advised to publish information using both standards, and e-mail recipients can use both standards to help filter spam," says Richi Jennings, an e-mail security analyst at Ferris Research Inc. in San Francisco.
But, he adds, "DKIM is better because the methods used to verify that the sender was authorized to use that domain are stronger. SPF/Sender ID has issues with mail lists and other things that autoforward mail."
DKIM is stronger, Jennings says, because it generates cryptographic hashes of content using keys owned by the e-mail sender's domain, while SIDF is simply based on which IP address the message comes from. "This means that DKIM is harder to set up and a little more expensive in terms of computing horsepower," he says.
John Scarrow, Microsoft's general manager of antispam and antiphishing strategy, agrees that the approaches are complementary. "By utilizing both, e-mail senders receive optimal protection and functionality across the board," he says. He acknowledges that DKIM is better for automatic forwarding by servers, such as when a user configures his Hotmail account to automatically forward messages to his Microsoft account.
But Scarrow argues that DKIM requires users to upgrade to both outbound and inbound message-transfer agents (MTA), such as Microsoft's Exchange Server, and affects "about 10 percent to 15 percent of computing cycles, while SIDF has no outbound impact to the MTA and negligible impact to any computing resources."
Sidebar
The future of electronic communications: Alternate realities
If you think you've conquered the ills of e-mail, you ain't seen nothin' yet, says futurist Paul Saffo.
"Look at all the indignation in the corporate world about salespeople using instant messaging to talk to customers," he says. "Some companies have gone to fire these people and found that all their data is out on a commercial-grade IM system and they can't get to it. Then they have hideous Sarbanes-Oxley problems, plus security problems and everything else."
It will get worse, Saffo predicts, when employees install MMORPGs -- massively multiplayer online role-playing games -- on their desktop computers and when they use their company laptops to travel through online societies like Second Life.
"CIOs worry about improper use of corporate systems by employees. But the bigger problem will be employee use of noncompany systems for company purposes," he says. "If the CIO has a problem getting his head around IM, just wait until he discovers that his employees are creating private lives in Second Life and inviting their clients to come hang out with them."
But at least one IT manager seems to have gotten his head around the problem. "We do not see it as a problem," says Matthew Marks, head of integrated user services at Aetna. "Our Web filtering software blocks specific and whole genres of Web sites from our employees, such as sex, violence, hatred and so on. We already block www.secondlife.com. As more of these types of sites become prevalent, the software will filter these out. We also do not allow IM outside of the company, and we do not allow people to download software from the Internet."
Sidebar
Internet e-mail: Could we start over?
Users complain bitterly about e-mail's perils, but technopundits like to point out that the Internet wasn't designed with privacy, security and commerce in mind. What we need is a brand-new Internet -- new technology from top to bottom, some say. Two of the people who built the Internet in the 1970s have differing views on that idea.
"I think it would be a very useful exercise to ask what a 'clean sheet' Internet would look like," says Vinton Cerf, chief Internet evangelist at Google Inc. and co-inventor of TCP/IP. "Whether it is possible to implement is a different question. Just trying to introduce IPv6 has been tough. On the other hand, the Internet was introduced at a time when the telephone's circuit-switched network and point-to-point data links were basically all the tools available. Who is to say that starting over is impossible?"
Ray Tomlinson would. "It's nonsense; I can't see how that could even be done," says the man who invented network e-mail. "I can see little enclaves building up within corporations, for example, that are using some other technology, especially corporations that have multiple sites and are using things going over the base Internet. They might choose to have an alternative network for that, but they are still going to have connections to the standard Internet."