The Inevitability of Archiving Social Networking Data

In 1985, I was part of a team at Carnegie Mellon University that was doing some very radical things, including deploying Internet email to the entire campus.   Some people were quite skeptical about the whole enterprise, fearing in particular that letting "the masses" loose on the Internet would dilute the quality of its content.  In retrospect I can't say they were wrong, but they were certainly swimming against the tide.

I, on the other hand, was insanely optimistic, and knew in my gut that the CMU campus was about to witness the birth of email as a large-scale social medium.  I felt that it would be useful to archive as much as possible of the content of that experiment, as a potential resource for future researchers.  (Privacy concerns were not yet high on anyone's radar.)

So I tried to convince someone -- anyone -- on campus to start archiving such things.  It would have been a major endeavor -- we expected many megabytes of data! -- and thus required the resources of the campus computing center or library.   Both of these institutions turned me down with a variant of, "why would anyone want to look at old email?"  My intuition told me that someone would, but I couldn't make a good enough case to pay for all those magnetic tapes.  The data was lost forever.

Just a few decades later, of course, email archiving is big business, and is actually a legal requirement for many businesses.  No one questions the need for email archiving under many circumstances, and it's hard to recall how outrageous the idea seemed less than 30 years ago.

Today, the situation with regard to archiving social networking data resembles the email archiving landscape a quarter century ago.  People see social networking posts as ephemeral and largely unimportant in the long run.  But we are beginning to see the shortsightedness of that perspective.

To begin with, some businesses -- particularly in heavily regulated industries such as finance -- have begun to use social networking for sensitive matters they'd prefer to keep out of email, precisely because the latter is archived.  That's understandable, but it won't be long before the regulators catch up.  If it makes sense to legally require archiving for email in a given industry, the same logic will apply to social networking data.

Moreover, social networking is part of a great evolution of business data from a structured, file-oriented organization to a looser, communication-oriented organization.  Nobody ever planned for email to become the central repository for corporate data, and yet today by some estimates as much as 90% of that data can be found in email (including attachments).  As social networking communications supplement that pattern, everything that has driven the desire to archive email will similarly drive the archiving of social networking data.

It's very easy to look at the typical social networking post and say, "this isn't worth archiving."  But in 1985, the typical email message might have included a grainy ASCII art picture of a topless woman, hardly an example of precious data.  It just doesn't matter.  Even if the majority of communications are unimportant, the value of archiving the important ones will ultimately justify the maintenance of an archive -- and will drive regulatory requirements to hold on to everything for a certain interval, "just in case."

But although regulatory requirements are a very good reason to archive your data -- it's always good to stay on the right side of the law -- there is an even bettere reason in the long run:  data mining.  When you archive all your communications, you create, in your archive, a potential gold mine of business insight.  From your social network data, for example, you might be able to track customer sentiment and responsiveness over time.  You might be able to figure out who in your organization has used social networking effectively to engage with customers, and who has not.  And by analyzing communication patterns, you might be able to identify key influencers who can help or hurt your business in a big way.

I believe that we're at a turning point in the use of "big data" archives to derive deep analytical insights.  At Mimecast, we're starting to do that with email today.  I find it hard to imagine that the same won't be true of social networking data in a few years, yet I hear the same voices of skepticism I heard in 1985.   The only difference is that now, archiving won't cost a fortune in recording and preserving magnetic tapes.   If anything, that should make the uptake of social network archiving even faster than it was for email archiving.