In 2007, research firm IDC reported that worldwide data storage was at 300 exabytes and projected that data would grow by a factor of ten by 2011.
Now, in 2010, the Internet is a major driver of the data storage explosion. The question is: Can corporate IT afford to continue traditional attitudes and practices concerning data storage as it copes with this explosion?
IT rightfully focuses on mission-critical software applications that will drive the enterprise business, and then identifies what those applications are. The next step is deciding whether the enterprise has the compute power and bandwidth (in the form of servers and networks) needed to run the applications.
Since disk- and tape-based storage has traditionally been cheap, storage is simply added as an afterthought to ensure that the new applications and the data they are going to produce are accommodated adequately. If data needs to be accessed faster on hard drives, the storage media of choice in most enterprises, technicians frequently resort to a technique called “striping,” where only 20 percent of each disk drive is used for access. The technique speeds application inputs and outputs from disk, but leaves 80 percent of disk underutilized, or not utilized at all. This is paid-for disk resource that is sitting -- and that contributes to expanding equipment footprints in data centers that consume square footage and energy.
“Companies need to have a better sense of the data they have and how they are storing it, so the data is not just sitting on any storage media that happens to be available,” says Charles King, principal analyst for Pund-IT Inc. King says that 85 percent of data created by the Internet surge inevitably becomes the responsibility of corporate IT, which must ultimately deploy appropriate data storage architecture, investment, and management practices.
This means looking at technologies like data de-duplication, which reduces storage needs by eliminating data redundancies in online and backup storage. Aggressive solutions from de-dupe vendors have spurred IT adoption of de-duplication over the past two years, and for good reason. A single email with a large attachment copied to four persons, and then backed up and archived, can create up to a 30x size increase (over the original email and attachment) in storage.
Vendors have been equally aggressive in pushing tiered storage that produces faster data access. An example is SSD (solid state disk), which is capable of storing data that is frequently accessed, such as online movie titles, while archiving away to slower hard drives the data that is less often accessed. This storage tiering can be automated with storage management software -- but successful automation depends on IT being able to come up with the right rules for managing the data.
This presents a second IT quandary: Does internal staff have the skills to develop sophisticated storage solutions and then to manage them?
“The exponential growth of data is a major factor that is now contributing to the IT skills shortage in data management,” says Tom Clancy, VP of educational services and productivity at EMC Corp. (NYSE: EMC). “There are also new regulations that require companies to hold onto data longer, and at the same time have immediate access to it. If you don’t have the in-house skills to do this, you have to hire people. The cost of not addressing data management, with the rise of litigation, is too great. In short, the burgeoning amount of data is simply outpacing IT’s ability to manage it.”
Clancy says that companies prefer to hire experienced storage managers -- but finding them is difficult. “Business and IT leadership needs to understand that there is a skills gap in storage management and in the general area of storage,” Clancy says. “Many companies have not yet made the systematic investment in their people, and have strictly focused on the bottom line without thinking of their present and future needs. It takes a forward understanding of the data management crisis to budget for storage management training when funds are tight -- but in the end, an educated person will be much more productive.”
Securely managing data is certainly an issue, Mashka--whether it is on servers or some other storage media.
Wherever it is, it is IT's job to make sure it is in a secure data center area.
The other concern, which I focused on in the article, is what IT DOES with all of this data.
Some data is critical, other data is underutilized and COULD become critical and still other data should be discarded. Data is multiplying so quickly that it is hard for IT (and others) to stay on top of it.
Do I understand right, that when you speak about the Internet data storage - it means, that data are kept at some server somewhere?
So if it is so, is it possible that due to some objective or subjective reason the data will be stolen or lost and what will companies do in that case?
Regulations, statues and laws generally establish retention requirements for RECORDS... and there needs to be a clear understanding that "not all data are records". Each organization needs to either determine what the definition of a record is for their industry segment, or to establish one for their orgnaization.
In general, a record is something that formally documents a transaction, decision, committment, or obligates an organization to do something, or proves that they did something. This is a murky 'definition', it's real clear in a Federal, State or Local government setting... but for private business, unless it's regulated or required, your retention periods are generally set based on known or anticipated business needs.
Once you reach this definition state, you have to establish a policy that says who does what, how they do it, how long records are kept, who keeps them and how they are disposed of (and you need to audit compliance). And yes, this applies to ALL records, regardless of media, form, or format and e-mail is NOT excluded from this when the content meets the definition of a record.
This is an involved process, and when you are creating a records retention schedule from scratch, it requires time and investigation of the processes within the organization that generate records internally or recieve them from external sources. You then need to research to determine if there are required retention periods (like for payroll, HR, product, safety, exposure and other easily categorized records) or to determine what business need they serve. You then set a minimum retention period, and if there is a legitimate business need to retain them longer, you assess what (if any) risk is related to that, and make a determination how long you're willing to keep them.AND you need to destroy them when they meet this period, by means appropriate to the contetn in the records (to ensur eyou protect privacy and any intellectual property)
And you ask, "WHY would anyone ever bother doing this?!?!" Well, in some cases you do it because it's the LAW... in other cases, you do it because by eliminating 'data' that is NOT A RECORD and that serves no apparent business need, you can avoid the cost of storing, managing, and any legal liabilities associated with producing data that you were not obligated to retain and you did it simply because "it's easy and storage is cheap!"
Yes, David, it sure could be a "treasure trove"--although many more employees have gained awareness of this and appear to be keeping their emails out of the corporate mail. In some cases, HR departments have published guidelines on this. In others, employees are being monitored for use of company property for non-company functions. This has scared many away from using coporate email for personal use during businsss. Instead, they use the telephone--or their own mobiles to call or text.
And when you say "save everything" you make me think of another challenge - the fact that so many people have personal email tied into their business emails. Maybe I am just watching too many lawyer shows on TV (okay on my iTouch during my commute) but it would seem like that data is a treasure trove for divorce lawyers and other legal matters.
So it's not just the companies' data that you are storing - but you are storing potentially millions of personal emails. When employees wrote paper letters - work and personal was easily separated. When they talked on rotary phones - not digital phones with recording options - work and personal was easily separated.
But now, think of all the personal data about employees that lives on business mail servers!
Yes, this will require new tools and ways of thinking. The "key" to the "thinking" will be how well the process reporting and analytics can be automated---and the ease of fit with internal IT processes.
Ultimately, though, it still seems to revolve around IT as being the central collection and execution point.
And as you keenly pointed out--there is a lot we don't know about how regulatory and retention requirements will evolve. This literally forces you to save almost everything!
Mary - I think that this issue is further complicated by the challenges of security and privacy which puts this not only in the IT area but in the world of business decisions. As you note “There are also new regulations that require companies to hold onto data longer, and at the same time have immediate access to it" and IT needs counterparts in the business.
How long do we legally need to keep things
How long do we want to keep things when no laws apply
How fast do we need things retrieved
What are the risks for needing this data for legal discovery?
If we have data that has privacy restrictions what impact does that have?
All these factors involving weighing costs and risks- business decisions that go well beyond IT.
Until the context of the process involved in creating the data is captured along with the data - the data problem will continue to spin out of control. It is not just the storage which is a problem, but more importantly - the access to the data. Data captured after the fact is a jumble of direct information, symptoms and side-effects caused by the process of interest. So looking just at the data is the equivalent of doing forensic analysis of the aftermath of an unknown process – and trying to piece together the process (this is very hard to do, and what makes CSI such a compelling show to watch). If there was someone watching the actual events as they unfold (or the actual “murder” process) they could just say – this mess was caused by A shooting B and then running away – and understanding the resulting data (e.g. blood splats on the wall, shell casings on the floor) would be much simpler (and CSI would be a very boring show). To be effective that process context needs to be captured and stored along with the data.
The ThinkerNet does not reflect the views of TechWeb. The ThinkerNet is an informal means of communication to members and visitors of the Internet Evolution site. Individual authors are chosen by Internet Evolution to blog. Neither Internet Evolution nor TechWeb assume responsibility for comments, claims, or opinions made by authors and ThinkerNet bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
Consider the following: By mid-2011, Nielsen estimates, there will be a US smartphone user base of 150 million, with 300 million mobile subscribers and 120 million mobile Web users. And in 2010, over 25 percent of US households are using mobile phones exclusively, eschewing landlines.
It was at a technology conference in New York this past July that Erik Brynjolfsson, director of the MIT Center for Digital Business of the Sloan School of Management, told his audience that coming out of this recession will be unlike any other, in that companies are determined to not reopen jobs that they cut during hard times.
Enterprises that have successfully migrated mission-critical applications to the Internet may balk at the news that a growing number of firms are opting to reverse that cost-saving process.
Now that some of the initial excitement about mobile apps is beginning to wear off, and they are moving toward the realm of mature technology, developers are taking a closer look at the business contexts in which the technology is being used, with an eye to how form might better fit function. In short, they’re finding places where smartphones fit better than tablets, and vice versa.
SIP (Session Initiation Protocol) is an increasingly important Internet integration technology, but there are still areas where critical interoperability work remains to be done -- including on IP PBXs, on SIP-enabled service provider connections, and on fax-over-IP (FoIP) solutions.
Getting to Work on Smart Work: How IT Is Transforming the Implementation of the 'Internet of Things' Organizations in all industry sectors are becoming more instrumented, interconnected, and intelligent -- and that's changing the way they approach virtually every facet of their operations. It's up to IT to help organizations adopt a "Three I's" approach that leverages the emerging Internet of Things and enables them to work smarter. READ THIS eBOOK
your weekly update of news, analysis, and
opinion from Internet Evolution - FREE! REGISTER HERE
Wanted! Site Moderators Internet Evolution is looking for a handful of readers to help moderate the message boards on our site – as well as engaging in high-IQ conversation with the industry mavens on our thinkerNet blogosphere. The job comes with various perks, bags of kudos, and GIANT bragging rights. Interested?
To save this item to your list of favorite Internet Evolution content so you can find it later in your Profile page, click the "Save It" button next to the item.
Cisco's UCS and IBM's zEnterprise have upped the ante for virtualization and 21st century computing. In the future, look for integration of disparate operating systems at the firmware level, self-healing architectures, and workload optimization across entire data centers.
Nielsen’s recent numbers on the increasing use of texting bode well for enterprise networks. Shunning the phone in favor of text messaging could mean reducing bandwidth.
Nielsen’s recent numbers on the increasing use of texting bode well for enterprise networks. Shunning the phone in favor of text messaging could mean reducing bandwidth.
RIM is giving in to demands by India to snoop on encrypted BlackBerry data. It's time to develop cheap or free encryption software for BlackBerrys and other cellular phones.
Two studios have filed suit against an ad broker for placing ads to help monetize P2P sites suspected of copyright infringement. That's taking a dangerous step toward what might be a worthy goal.
By 2014, mobile devices will overtake laptops as the appliance of choice for consumers. But device makers still have some wishes to fulfill, including mobile app simplification and the ability to better perform word processing/spreadsheet functions.
Google's foray into pay-for-view movies may be an indicator that the days of free ad-sponsored content are numbered, or at least that ad sponsorship won't fund nearly enough content.
Online education, improving to better replicate the interactions that occur between teachers and students face-to-face, grew in double digits during the recession. Still, there’s more work to be done.
Google's decision to link VoIP calling of PSTN numbers with Gmail, and to let Google Voice "call" Gmail VoIP clients, will devalue the PSTN and force telcos to fund unprofitable services or create their own VoIP transitions.
Digg
Del.icio.us
Reddit
Email This
TWEET THIS
