For more than 50 years, the disk drive has been at the center of computing architectures and the Internet, but there's mounting evidence that this is going to change powerfully over the next decade.
This past week I attended Storage Networking World, one of the storage industry's more popular meet-ups, and the most discussed topic was flash-based disk storage. Apple Inc. (Nasdaq: AAPL) first introduced NAND (flash) storage on its iPod Shuffle in early 2005, and while some dismissed the announcement as a niche for toys, others began to look at pricing forecasts and realized that, for certain applications, flash-based storage would begin to replace spinning disks in the early part of the next decade.
In fact, selectively, it's already happening today. Virtually every major system and storage supplier has announced support for NAND flash drives in storage arrays. NAND drives have much better response times and faster access times than spinning media. The disadvantages are cost and reliability -- i.e., the number of writes to each memory cell is limited and has to be managed with additional layers (and more cost).
But for Internet applications, where much of the activity is simple reading of the disk, this is potentially good news. Specifically, flash prices are dropping faster than those of spinning media and will begin to replace spinning disks in certain applications. New architectures are possible where the highest-performance Internet data (e.g., indexes, search metadata, certain database activities, etc.) can be serviced from flash-based storage, while less performance-intensive data will be placed on the very highest-capacity, lowest-cost spinning devices, and, in an effort to save energy, spun down when not in use.
It is important to distinguish between high-performance drives and high-capacity drives. The Achilles heel of disk drives is access time limited by rotational delay. Expensive controllers with RAM caches have been designed to mitigate this problem. With the high performance and low access times of NAND storage, the major type of drive that is likely to be initially replaced by NAND devices are high-performance Fibre Channel (FC) drives. Although high-capacity disk drives (SAS or SATA) may eventually be replaced by NAND drives, that is well beyond any near-term planning horizon for users and vendors.
So one key planning question is whether NAND storage will obviate the need for high-performance FC disk drives. And if so, when?
The chart above shows three price curves for flash at varying rates of decline (50 percent, 60 percent, and 70 percent annually). Current trends since last summer show that the actual reduction of NAND prices is about 60 percent per year. At this rate of comparative reduction, FC drives will be obsolete in less than three years time (2011-2012).
While much of the Internet's data will reside on spinning disk for quite some time, it's inevitable that flash technologies will begin to permeate computing architectures and alleviate an age-old problem that mechanical disks were never truly well suited for computer applications.
— David Vellante spent 15 years at IDC and is a founder of The Wikibon Project. He can be reached on Twitter at @dvellante.
From a selfish end-user perspective, this is about heat and noise. After the Mac Cube (http://en.wikipedia.org/wiki/Power_Mac_G4_Cube) I really thought there would be more of a push for silence, but apparently not.
Look at the Dell Adamo. http://www.dell-adamo.nl/wp-content/uploads/2009/01/dell-adamo-white-04_large.jpg
Now they're clearly selling this as a sexy machine, the ads show runway models posing with the thing. But all I see is the ugly air vents on the side. Come on, this would be a slimmer, more beautiful machine w/o a spinning drive, don't you think?
My dream machine: a Thinkpad netbook (for the keyboard and trackpoint) and no moving parts.
I loved your story. Back in about 1974 I could smell the lanolin too.
I agree 100% that there will be a rethink about the CPU/Memory/Disk relationships. Current architectures assume all sorts of conventions (standards) at each point of the interface.
Sooner or later someone with some access to capital will review the CPU/Memory/Disk relationship from the point of view of function- for example a HTML server or a Video server. Everything form the disk sectors on out needs to be defined in view of the intended purpose. I expect 10X performance at the same level of semiconductor/storage technology will be possible.
Wow Rob! That's a tremendous post. Your point is right on, this concept is not new and architects like yourself have faced the spinning media bottleneck forever in this industry.
And for those of you born after 1985, by "sketched out in a notebook" Rob means written in pen or pencil in a 2-fold, sturdy piece of vinyl or leather bound plastic that houses sheets of lined paper :-)
Companies probably do not like the expenses but when considering various laws, particularly those involving privacy, and safeguarding company data, they are willing to sacrifice cost for high performance and durability.
(Besides, they will figure a way to pass the cost to their customers.)
I'm reading the comments with a big grin. Perhaps you would be interested in how the "solid state disk" was invented?
Back in the summer of 1970, I was consulting with the Laboratory Data Products Group at Digital Equipment Corp. Housed in the old woolen mill in Maynard, Mass, the floors were still soaked in lanolin from the blankets made there during the Civil War.
The task at hand was to write a gas chromatographic analysis package to run on a PDP-12. The 12 was a chimera of a machine --- a PDP-8; a LINC-8; and a still-under-development Floating Point Processor (FPP). "Mass" memory on the 12 was a DECtape --- a hand-sized, wide magnetic tape. The FPP was a 17" by 8' rack.
Software assembly with DECtape was a multi-hour process. However, on weekends, there was a "production" 12 used to generate new DECtapes to ship with new 12's. It had a fixed-head hard drive. One head per track. Fast! I could assemble in minutes. The value of a fast, rotating drive with no head-movement delay was embedded in my head.
Several months later, back in my computer lab in Cleveland, running on the PDP-11 that DEC kindly provided in exchange for my successful software effort, there was a DR-11-M (?) Direct Memory Access controller to let a peripheral device move data directly to/from the PDP-11's core memory. In an inventive moment, I realized that one could emulate that fixed-head hard drive in the same manner as the FPP had been connected to the PDP-12, but using a rack-full of memory boards.
It took until 1974 to find a company willing to share the development of what-came-to-be-called the Extended Memory Unit (EMU) [After the fast, flightless bird]. Monolithic Systems in Englewood, CO, a maker of semiconductor memory boards picked up the concept and turned it into a successful product. Rumor was that some EMUs were on nuclear submarines, where there was no gyroscopic torque from a spinning hard drive. I built a very-high-speed data acquisition system where data from explosions went through analog-to-digital converters into the EMU, then sent a single interrupt signal to the host PDP-11/45 computer. To the PDP-11, it seemed that in a microsecond, a disk full of data just appeared on a hard drive.
A graduate student in Computer Science at CWRU studied the EMU performance. Instead of being 17,000 times faster than DEC's RF/RS-11 fixed-head disk [17 millisecond rotational latency versus 1 microsecond access time for the semiconductor memory], the performance was far less, due to the fact that the software device drivers in DEC's OS were written with a rotating drive in mind.
Which brings us to the current "Flash Drive" flurry --- I believe that there will soon be a shift in system architecture. Rather than word-at-a-time or block-at-a-time data transfers, computing will be done where entire large blocks of data are transfered with a simple interrupt signal. Processing algorithms will be re-thought such that FPLA's can do the processing of these large blocks of data in a single clock cycle. The market might evolve with the creation of smart, ultra-fast peripheral devices.
All this was sketched out in a notebook in the mid-1970's. I never throw anything away, so I'll dig through the storeroom and see what explicit ideas I had around this back then.
Well thank you Mary, I appreciate that. And I appreciate your pushing at my assumptions and premise here to make them better. What good is a prediction that's not surprising? (Although I didn't honestly see this as that outlandish).
I see a domino effect coming where the price of flash drops at or near (or below) spinning disk and a leading vendor or a few, in an effort to get competitive advantage and lower the cost of storage, adopt early and push hard. I predict EMC, IBM and HP will lead this charge. Maybe NetApp too.
In re-thinking (and re-thinking) your point about vendor adoption, it strikes me that the unwillingness to switch may be due to the need to re-architect storage systems to accommodate flash. That will cause friction. But I believe these leaders are already doing just that and the rest of the industry will be forced to follow.
So I'll adjust my bottom line a bit. It still comes down to price but I'll concede switching costs are a factor and they are not fully eliminated because flash drives can plug directly into FC slots. I'll admit there are clear costs to re-design array architectures to accommodate flash. The fact that flash will plug into existing interface slots is key because it will drive the first wave but full scale adoption will require redesigns.
New prediction...those vendors who don't re-design their arrays to accommodate flash will begin to lose market share in the 2011 timeframe.
Fascinating stuff, Dave. However, I'm still a skeptic about the big vendors' willingness to make the switch. If I were a wagering person, I'd say you'd have a bet that FC won't lose ground to Flash until way after 2012. Still, your case is well put and this industry is nothing if not surprising.
15,000 rpm even Mary. But it's mechanical. You're comparing a spinning disk with semiconductor speeds. Make it 50,000 rpm (if you could dissipate the heat and handle the data rates) and it's still an order of magnitude slower than semiconductor...not to mention the time it takes to to physically move an actuator arm, which hasn't really improved in years.
The other factor here is processor speeds double every 18 mos but disk drive speeds get worse every 18 mos. Why? because the more data you put under an actuator, the slower disk drives get. It's a problem referred to as 'access density,' meaning the disk has to spin more and the arm has to move more to find the data to fulfill an i/o request. This is why customers 'short-stroke' disk drives - i.e. they leave them empty except for the critical files (e.g. database indicies and the like). There are gazillions of FC drives installed that are basically empty for performance reasons.
Bottom line, compared to the speed of light, mechanical disk drives are sloooooooooooooooooooooooow.
Design good software to minimize the need for faster hardware...interesting concept and one that's been talked about before. One of the major limitations of applications is the need to access spinning disk. For decades we've dreamed of keeping "Data in Memory" (DiM) as a single level store. Flash enables a form of that vision and application developers will exploit it.
Here's what I know...give application developers fast i/o resources and they'll use them in ways your or I never thought of. New supercomputer applications are popping up everyday using the Internet as a resource and the applications are following.
Are you arguing that the state of Internet applications is frozen-- or should be?
My strong belief is Internet applications will exploit flash technologies to create better user experiences-- it's stating the obvious.
The ThinkerNet does not reflect the views of TechWeb. The ThinkerNet is an informal means of communication to members and visitors of the Internet Evolution site. Individual authors are chosen by Internet Evolution to blog. Neither Internet Evolution nor TechWeb assume responsibility for comments, claims, or opinions made by authors and ThinkerNet bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
I had the opportunity recently to meet with Jeff Kubacki the CIO of Kroll, a global risk management consulting firm and a unit of Marsh & McLennan Companies with more than 50 offices worldwide. Kubacki has been the CIO of Kroll for about three years and seems to have a good process for aligning IT strategy with business priorities.
At Twitter’s Chirp developer conference last week, the company confirmed that Twitter has more than 100 million registered users, 300,000 new users per day, and 180 million unique users per month. So I guess it’s inevitable that Twitter would finally start to “grow up.”
The Twittersphere is buzzing with rumors that Yahoo Inc. (Nasdaq: YHOO) is considering shelling out $100 million for Foursquare, the hot startup du jour that offers location-based services.
Facebook's Graph Search may face some profound challenges and risks, first, because Facebook users haven't been thinking of their posts as product reviews; and second, because Facebook will now have to contend with the social-network equivalent of SEO "gaming" of results.
A recent release of the popular TweetDeck app for Twitter power-users gives new life to software that had previously taken a wrong turn. Here's a quick walk-through of the new TweetDeck, to show you why it should be at the top of your Twitter toolkit.
Based on reactions in Nicole's Newsfeed, everyone hates this version of Facebook. This should matter to Facebook now that there's a real competitor on the scene named Google+.
Allowing users to share music and video on Facebook might sound like good news, but is this part of a coherent strategy, or is Facebook just stumbling from idea to idea?
A decade after the dotcom boom, the Internet continues to dramatically change the way that business gets done and individuals communicate. More than a trillion email messages traveled over the Net last year, and dramatic changes loom on the horizon.
New York's Metropolitan Transit Authority is conducting a pilot test of digital kiosks to guide subway users to where they want to go more efficiently and at lower cost.
The whole Amazon.reader debate is a double-stupid. It's stupid to think that there's any e-book buyer who doesn't know Amazon's URL, and it was stupider to let ICANN launch the whole free-form TLD initiative to start with.
While NFC's original goal was to enhance mobile commerce applications, it is finding its way into a number of other uses, which is creating both opportunity as well as challenges for IT departments.
Enterprises would like to move to cloud computing but are hesitant because they are concerned about providers’ ability to secure company data. Here are some tips that help to ensure that if breaches occur, the business is not left holding the bag.
Edmunds separates customers into segments based on the info it collects on its site and from partners, and uses that to push out custom content, said Brian Baron, director of business analytics for Edmunds.com, at Predictive Analytics Innovation Summit.
The automotive website uses propensity modeling to target ads and customer registration forms, said Brian Baron, director of business analytics for Edmunds.com, at Predictive Analytics Innovation Summit.
Expert Integrated Systems: Changing the Experience & Economics of IT In this e-book, we take an in-depth look at these expert integrated systems -- what they are, how they work, and how they have the potential to help CIOs achieve dramatic savings while restoring IT's role as business innovator. READ THIS eBOOK
your weekly update of news, analysis, and
opinion from Internet Evolution - FREE! REGISTER HERE
Wanted! Site Moderators Internet Evolution is looking for a handful of readers to help moderate the message boards on our site – as well as engaging in high-IQ conversation with the industry mavens on our thinkerNet blogosphere. The job comes with various perks, bags of kudos, and GIANT bragging rights. Interested?
To save this item to your list of favorite Internet Evolution content so you can find it later in your Profile page, click the "Save It" button next to the item.
M2M: Rise of the Machines? Not Yet David Weldon In the 1970 science fiction thriller Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M. CLICK FOR MORE
M2M: Rise of the Machines? Not Yet David Weldon In the 1970 science fiction thriller Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M. CLICK FOR MORE
M2M: Rise of the Machines? Not Yet David Weldon In the 1970 science fiction thriller Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M. CLICK FOR MORE
M2M: Rise of the Machines? Not Yet David Weldon In the 1970 science fiction thriller Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M. CLICK FOR MORE
Email This
