WASHINGTON (AP) — In the months and early years after 9/11, FBI agents began showing up at Microsoft Corp. more frequently than before, armed with court orders demanding information on customers.
Around the world, government spies and eavesdroppers were tracking the email and Internet addresses used by suspected terrorists. Often, those trails led to the world's largest software company and, at the time, largest email provider.
The agents wanted email archives, account information, practically everything, and quickly. Engineers compiled the data, sometimes by hand, and delivered it to the government.
Often there was no easy way to tell if the information belonged to foreigners or Americans. So much data was changing hands that one former Microsoft employee recalls that the engineers were anxious about whether the company should cooperate.
Inside Microsoft, some called it "Hoovering" — not after the vacuum cleaner, but after J. Edgar Hoover, the first FBI director, who gathered dirt on countless Americans.
This frenetic, manual process was the forerunner to Prism, the recently revealed highly classified National Security Agency program that seizes records from Internet companies. As laws changed and technology improved, the government and industry moved toward a streamlined, electronic process, which required less time from the companies and provided the government data in a more standard format.
The revelation of Prism this month by the Washington Post and Guardian newspapers has touched off the latest round in a decade-long debate over what limits to impose on government eavesdropping, which the Obama administration says is essential to keep the nation safe.
But interviews with more than a dozen current and former government and technology officials and outside experts show that, while Prism has attracted the recent attention, the program actually is a relatively small part of a much more expansive and intrusive eavesdropping effort.
Americans who disapprove of the government reading their emails have more to worry about from a different and larger NSA effort that snatches data as it passes through the fiber optic cables that make up the Internet's backbone. That program, which has been known for years, copies Internet traffic as it enters and leaves the United States, then routes it to the NSA for analysis.
Whether by clever choice or coincidence, Prism appears to do what its name suggests. Like a triangular piece of glass, Prism takes large beams of data and helps the government find discrete, manageable strands of information.
The fact that it is productive is not surprising; documents show it is one of the major sources for what ends up in the president's daily briefing. Prism makes sense of the cacophony of the Internet's raw feed. It provides the government with names, addresses, conversation histories and entire archives of email inboxes.
Many of the people interviewed for this report insisted on anonymity because they were not authorized to publicly discuss a classified, continuing effort. But those interviews, along with public statements and the few public documents available, show there are two vital components to Prism's success.
The first is how the government works closely with the companies that keep people perpetually connected to each other and the world. That story line has attracted the most attention so far.
The second and far murkier one is how Prism fits into a larger U.S. wiretapping program in place for years.
Deep in the oceans, hundreds of cables carry much of the world's phone and Internet traffic. Since at least the early 1970s, the NSA has been tapping foreign cables. It doesn't need permission. That's its job.
But Internet data doesn't care about borders. Send an email from Pakistan to Afghanistan and it might pass through a mail server in the United States, the same computer that handles messages to and from Americans. The NSA is prohibited from spying on Americans or anyone inside the United States. That's the FBI's job and it requires a warrant.
Despite that prohibition, shortly after the Sept. 11 terrorist attacks, President George W. Bush secretly authorized the NSA to plug into the fiber optic cables that enter and leave the United States, knowing it would give the government unprecedented, warrantless access to Americans' private conversations.
Tapping into those cables allows the NSA access to monitor emails, telephone calls, video chats, websites, bank transactions and more. It takes powerful computers to decrypt, store and analyze all this information, but the information is all there, zipping by at the speed of light.
"You have to assume everything is being collected," said Bruce Schneier, who has been studying and writing about cryptography and computer security for two decades.
The New York Times disclosed the existence of this effort in 2005. In 2006, former AT&T technician Mark Klein revealed that the company had allowed the NSA to install a computer at its San Francisco switching center, a key hub for fiber optic cables.
What followed was the most significant debate over domestic surveillance since the 1975 Church Committee, a special Senate committee led by Sen. Frank Church, D-Idaho, reined in the CIA and FBI for spying on Americans.
Unlike the recent debate over Prism, however, there were no visual aids, no easy-to-follow charts explaining that the government was sweeping up millions of emails and listening to phone calls of people accused of no wrongdoing.
The Bush administration called it the "Terrorist Surveillance Program" and said it was keeping the United States safe.
"This program has produced intelligence for us that has been very valuable in the global war on terror, both in terms of saving lives and breaking up plots directed at the United States," Vice President Dick Cheney said at the time.
The government has said it minimizes all conversations and emails involving Americans. Exactly what that means remains classified. But former U.S. officials familiar with the process say it allows the government to keep the information as long as it is labeled as belonging to an American and stored in a special, restricted part of a computer.
That means Americans' personal emails can live in government computers, but analysts can't access, read or listen to them unless the emails become relevant to a national security investigation.
The government doesn't automatically delete the data, officials said, because an email or phone conversation that seems innocuous today might be significant a year from now.
What's unclear to the public is how long the government keeps the data. That is significant because the U.S. someday will have a new enemy. Two decades from now, the government could have a trove of American emails and phone records it can tap to investigative whatever Congress declares a threat to national security.
The Bush administration shut down its warrantless wiretapping program in 2007 but endorsed a new law, the Protect America Act, which allowed the wiretapping to continue with changes: The NSA generally would have to explain its techniques and targets to a secret court in Washington, but individual warrants would not be required.
Congress approved it, with Sen. Barack Obama, D-Ill., in the midst of a campaign for president, voting against it.
"This administration also puts forward a false choice between the liberties we cherish and the security we provide," Obama said in a speech two days before that vote. "I will provide our intelligence and law enforcement agencies with the tools they need to track and take out the terrorists without undermining our Constitution and our freedom."
When the Protect America Act made warrantless wiretapping legal, lawyers and executives at major technology companies knew what was about to happen.
One expert in national security law, who is directly familiar with how Internet companies dealt with the government during that period, recalls conversations in which technology officials worried aloud that the government would trample on Americans' constitutional right against unlawful searches, and that the companies would be called on to help.
Continue reading this story on the...