The secret world of "cyber situational awareness" is a spymaster's wet dream, made all the more alluring by the advent of ultra high speed computing and the near infinite storage capacity afforded by massive server farms and the ubiquitous "cloud."
Lost in the debate surrounding privacy invasion and data mining however, is the key role that internet service providers (ISPs) play as intermediaries and gatekeepers. From their perch, ISPs peer deeply into and collect and analyze the online communications of tens of millions of users simultaneously, in real-time.
Concerted efforts to eliminate online anonymity, in managed democracies and authoritarian regimes alike, are greatly enhanced by the deployment of deep packet inspection (DPI) sensors and software on virtually all networks.
"To unpack this a little" they aver, "all data traffic that courses across the 'net is contained in individual packets that have header (i.e. addressing) information and payload (i.e. content) information. We can think of this as the address on a postcard and the written and visual content of a postcard."
All of which is there for the taking, "criminal evidence, ready for use in a trial," Cryptohippie chillingly informs.