The Macrosite for News, Analysis and Opinion about the Future of the Internet
David Vellante

Flash Drives Set to Give Internet a Performance Boost

Written by David Vellante
4/15/2009 21 comments
no ratings
DISCUSS     Email This

For more than 50 years, the disk drive has been at the center of computing architectures and the Internet, but there's mounting evidence that this is going to change powerfully over the next decade.

This past week I attended Storage Networking World, one of the storage industry's more popular meet-ups, and the most discussed topic was flash-based disk storage. Apple Inc. (Nasdaq: AAPL) first introduced NAND (flash) storage on its iPod Shuffle in early 2005, and while some dismissed the announcement as a niche for toys, others began to look at pricing forecasts and realized that, for certain applications, flash-based storage would begin to replace spinning disks in the early part of the next decade.

In fact, selectively, it's already happening today. Virtually every major system and storage supplier has announced support for NAND flash drives in storage arrays. NAND drives have much better response times and faster access times than spinning media. The disadvantages are cost and reliability -- i.e., the number of writes to each memory cell is limited and has to be managed with additional layers (and more cost).

But for Internet applications, where much of the activity is simple reading of the disk, this is potentially good news. Specifically, flash prices are dropping faster than those of spinning media and will begin to replace spinning disks in certain applications. New architectures are possible where the highest-performance Internet data (e.g., indexes, search metadata, certain database activities, etc.) can be serviced from flash-based storage, while less performance-intensive data will be placed on the very highest-capacity, lowest-cost spinning devices, and, in an effort to save energy, spun down when not in use.

It is important to distinguish between high-performance drives and high-capacity drives. The Achilles heel of disk drives is access time limited by rotational delay. Expensive controllers with RAM caches have been designed to mitigate this problem. With the high performance and low access times of NAND storage, the major type of drive that is likely to be initially replaced by NAND devices are high-performance Fibre Channel (FC) drives. Although high-capacity disk drives (SAS or SATA) may eventually be replaced by NAND drives, that is well beyond any near-term planning horizon for users and vendors.

So one key planning question is whether NAND storage will obviate the need for high-performance FC disk drives. And if so, when?

The chart above shows three price curves for flash at varying rates of decline (50 percent, 60 percent, and 70 percent annually). Current trends since last summer show that the actual reduction of NAND prices is about 60 percent per year. At this rate of comparative reduction, FC drives will be obsolete in less than three years time (2011-2012).

While much of the Internet's data will reside on spinning disk for quite some time, it's inevitable that flash technologies will begin to permeate computing architectures and alleviate an age-old problem that mechanical disks were never truly well suited for computer applications.

— David Vellante spent 15 years at IDC and is a founder of The Wikibon Project. He can be reached on Twitter at @dvellante.

DISCUSS     Email This
Current display:       newest comments first       display in chronological order
Page 1 of 3   Next >
ferodynamics
IQ Crew
Saturday April 25, 2009 12:11:05 AM
no ratings

From a selfish end-user perspective, this is about heat and noise.  After the Mac Cube (http://en.wikipedia.org/wiki/Power_Mac_G4_Cube) I really thought there would be more of a push for silence, but apparently not.

Look at the Dell Adamo.  http://www.dell-adamo.nl/wp-content/uploads/2009/01/dell-adamo-white-04_large.jpg

Now they're clearly selling this as a sexy machine, the ads show runway models posing with the thing.  But all I see is the ugly air vents on the side.  Come on, this would be a slimmer, more beautiful machine w/o a spinning drive, don't you think?

My dream machine: a Thinkpad netbook (for the keyboard and trackpoint) and no moving parts.

lpricci49
Rank: Cave Painter
Sunday April 19, 2009 6:22:38 PM
no ratings

I loved your story.  Back in about 1974 I could smell the lanolin too.

I agree 100% that there will be a rethink about the CPU/Memory/Disk relationships.  Current architectures assume all sorts of conventions (standards) at each point of the interface. 

Sooner or later someone with some access to capital will review the CPU/Memory/Disk relationship from the point of view of function- for example a HTML server or a Video server.  Everything form the disk sectors on out needs to be defined in view of the intended purpose.  I expect 10X performance at the same level of semiconductor/storage technology will be possible.

Lawrence Ricci
www.EmbeddedInsider.com

'

 

David Vellante
Thinkernetter
Saturday April 18, 2009 6:53:29 AM
no ratings

Wow Rob! That's a tremendous post. Your point is right on, this concept is not new and architects like yourself have faced the spinning media bottleneck forever in this industry.

And for those of you born after 1985, by "sketched out in a notebook" Rob means written in pen or pencil in a 2-fold, sturdy piece of vinyl or leather bound plastic that houses sheets of lined paper :-)

rswinney
IQ Crew
Saturday April 18, 2009 1:49:57 AM
no ratings

Companies probably do not like the expenses but when considering various laws, particularly those involving privacy, and safeguarding company data, they are willing to sacrifice cost for high performance and durability.

(Besides, they will figure a way to pass the cost to their customers.)

roblake
IQ Crew
Thursday April 16, 2009 5:31:51 PM

Folks -

I'm reading the comments with a big grin.  Perhaps you would be interested in how the "solid state disk" was invented?

Back in the summer of 1970, I was consulting with the Laboratory Data Products Group at Digital Equipment Corp.  Housed in the old woolen mill in Maynard, Mass, the floors were still soaked in lanolin from the blankets made there during the Civil War.

The task at hand was to write a gas chromatographic analysis package to run on a PDP-12.  The 12 was a chimera of a machine --- a PDP-8; a LINC-8; and a still-under-development Floating Point Processor (FPP).  "Mass" memory on the 12 was a DECtape  --- a hand-sized, wide magnetic tape.  The FPP was a 17" by 8' rack.

Software assembly with DECtape was a multi-hour process.  However, on weekends, there was a "production" 12 used to generate new DECtapes to ship with new 12's.  It had a fixed-head hard drive.  One head per track.  Fast!  I could assemble in minutes.  The value of a fast, rotating drive with no head-movement delay was embedded in my head.

Several months later, back in my computer lab in Cleveland, running on the PDP-11 that DEC kindly provided in exchange for my successful software effort, there was a DR-11-M (?) Direct Memory Access controller to let a peripheral device move data directly to/from the PDP-11's core memory.  In an inventive moment, I realized that one could emulate that fixed-head hard drive in the same manner as the FPP had been connected to the PDP-12, but using a rack-full of memory boards.

It took until 1974 to find a company willing to share the development of what-came-to-be-called the Extended Memory Unit (EMU) [After the fast, flightless bird].  Monolithic Systems in Englewood, CO, a maker of semiconductor memory boards picked up the concept and turned it into a successful product.  Rumor was that some EMUs were on nuclear submarines, where there was no gyroscopic torque from a spinning hard drive.  I built a very-high-speed data acquisition system where data from explosions went through analog-to-digital converters into the EMU, then sent a single interrupt signal to the host PDP-11/45 computer.  To the PDP-11, it seemed that in a microsecond, a disk full of data just appeared on a hard drive.

A graduate student in Computer Science at CWRU studied the EMU performance.  Instead of being 17,000 times faster than DEC's RF/RS-11 fixed-head disk [17 millisecond rotational latency versus 1 microsecond access time for the semiconductor memory], the performance was far less, due to the fact that the software device drivers in DEC's OS were written with a rotating drive in mind.

Which brings us to the current "Flash Drive" flurry  ---  I believe that there will soon be a shift in system architecture.  Rather than word-at-a-time or block-at-a-time data transfers, computing will be done where entire large blocks of data are transfered with a simple interrupt signal.  Processing algorithms will be re-thought such that FPLA's can do the processing of these large blocks of data in a single clock cycle.  The market might evolve with the creation of smart, ultra-fast peripheral devices.

All this was sketched out in a notebook in the mid-1970's.  I never throw anything away, so I'll dig through the storeroom and see what explicit ideas I had around this back then.

 

David Vellante
Thinkernetter
Thursday April 16, 2009 1:27:18 PM
no ratings

Well thank you Mary, I appreciate that. And I appreciate your pushing at my assumptions and premise here to make them better. What good is a prediction that's not surprising? (Although I didn't honestly see this as that outlandish).

I see a domino effect coming where the price of flash drops at or near (or below) spinning disk and a leading vendor or a few, in an effort to get competitive advantage and lower the cost of storage, adopt early and push hard. I predict EMC, IBM and HP will lead this charge. Maybe NetApp too. 

In re-thinking (and re-thinking) your point about vendor adoption, it strikes me that the unwillingness to switch may be due to the need to re-architect storage systems to accommodate flash. That will cause friction. But I believe these leaders are already doing just that and the rest of the industry will be forced to follow.

So I'll adjust my bottom line a bit. It still comes down to price but I'll concede switching costs are a factor and they are not fully eliminated because flash drives can plug directly into FC slots. I'll admit there are clear costs to re-design array architectures to accommodate flash. The fact that flash will plug into existing interface slots is key because it will drive the first wave but full scale adoption will require redesigns.

New prediction...those vendors who don't re-design their arrays to accommodate flash will begin to lose market share in the 2011 timeframe.

Mary Jander
Thinkernetter
Thursday April 16, 2009 12:58:35 PM
no ratings

Fascinating stuff, Dave. However, I'm still a skeptic about the big vendors' willingness to make the switch. If I were a wagering person, I'd say you'd have a bet that FC won't lose ground to Flash until way after 2012. Still, your case is well put and this industry is nothing if not surprising.

David Vellante
Thinkernetter
Thursday April 16, 2009 12:26:11 PM
no ratings

15,000 rpm even Mary. But it's mechanical. You're comparing a spinning disk with semiconductor speeds. Make it 50,000 rpm (if you could dissipate the heat and handle the data rates) and it's still an order of magnitude slower than semiconductor...not to mention the time it takes to to physically move an actuator arm, which hasn't really improved in years.

The other factor here is processor speeds double every 18 mos but disk drive speeds get worse every 18 mos. Why? because the more data you put under an actuator, the slower disk drives get. It's a problem referred to as 'access density,' meaning the disk has to spin more and the arm has to move more to find the data to fulfill an i/o request. This is why customers 'short-stroke' disk drives - i.e. they leave them empty except for the critical files (e.g. database indicies and the like). There are gazillions of FC drives installed that are basically empty for performance reasons.

Bottom line, compared to the speed of light, mechanical disk drives are sloooooooooooooooooooooooow.

 

 

David Vellante
Thinkernetter
Thursday April 16, 2009 12:05:02 PM
no ratings

Design good software to minimize the need for faster hardware...interesting concept and one that's been talked about before. One of the major limitations of applications is the need to access spinning disk. For decades we've dreamed of keeping "Data in Memory" (DiM) as a single level store. Flash enables a form of that vision and application developers will exploit it.

Here's what I know...give application developers fast i/o resources and they'll use them in ways your or I never thought of. New supercomputer applications are popping up everyday using the Internet as a resource and the applications are following.

Are you arguing that the state of Internet applications is frozen-- or should be?

My strong belief is Internet applications will exploit flash technologies to create better user experiences-- it's stating the obvious.

Mary Jander
Thinkernetter
Thursday April 16, 2009 11:29:16 AM
no ratings

"The only spinning disks left will be ultra high capacity devices that will maintain an advantage over flash for quite some time to come."

This seems to work against your claim that Fibre Channel will be replaced by flash within the next 3 years.

Even if you go with speeds, FC RPMs are pretty durn high -- up to 7200 for 4-Gbit/s, no?

Now, how much energy they use might be a key factor.

 

Page 1 of 3   Next >
The ThinkerNet does not reflect the views of TechWeb. The ThinkerNet is an informal means of communication to members and visitors of the Internet Evolution site. Individual authors are chosen by Internet Evolution to blog. Neither Internet Evolution nor TechWeb assume responsibility for comments, claims, or opinions made by authors and ThinkerNet bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
previous posts from David Vellante
David Vellante
David Vellante   6/14/2010   5 comments
I've been following the so-called "Stack Wars" in the Internet-enabled enterprise for quite some time.
David Vellante
David Vellante   4/28/2010   14 comments
I had the opportunity recently to meet with Jeff Kubacki the CIO of Kroll, a global risk management consulting firm and a unit of Marsh & McLennan Companies with more than 50 offices worldwide. Kubacki has been the CIO of Kroll for about three years and seems to have a good process for aligning IT strategy with business priorities.
David Vellante
David Vellante   4/19/2010   18 comments
At Twitter’s Chirp developer conference last week, the company confirmed that Twitter has more than 100 million registered users, 300,000 new users per day, and 180 million unique users per month. So I guess it’s inevitable that Twitter would finally start to “grow up.”
David Vellante
The Twittersphere is buzzing with rumors that Yahoo Inc. (Nasdaq: YHOO) is considering shelling out $100 million for Foursquare, the hot startup du jour that offers location-based services.
David Vellante
David Vellante   3/30/2010   36 comments
Google Apps is the single biggest threat Microsoft Corp. (Nasdaq: MSFT) has faced in a decade. This assertion is based on two major factors:
5
of
what.the.ferraro
Twitter Conferences & the Demise of Humanity

10|13|09   |   1:56   |   6 comments


The rise of industry events centered solely on the topic of Twitter is enough to make some people cry. Literally.
Second Shooter
Graphing Facebook Graph Search's Success

1|25|13   |   2:13   |   10 comments


Facebook's Graph Search may face some profound challenges and risks, first, because Facebook users haven't been thinking of their posts as product reviews; and second, because Facebook will now have to contend with the social-network equivalent of SEO "gaming" of results.
Mitch Wagner
TweetDeck Gets a Second Life

11|5|12   |   9:54   |   13 comments


A recent release of the popular TweetDeck app for Twitter power-users gives new life to software that had previously taken a wrong turn. Here's a quick walk-through of the new TweetDeck, to show you why it should be at the top of your Twitter toolkit.
Mitch Wagner
LinkedIn Will Be the Last Social Network Standing

8|31|12   |   2:34   |   15 comments


While Facebook and Twitter get more attention, LinkedIn's going to be the long-term winner.
Reiter's Block
The Web Needs National Grammar Day

2|29|12   |   2:59   |   56 comments


March 4 is National Grammar Day, and you enterprise and consumer bloggers need to pay attention.
what.the.ferraro
Ashton Tweets No More

11|11|11   |   03:01   |   19 comments


After making an egregious blunder on Twitter, Ashton Kutcher is handing off the task of being informed and sensible to a management team.
what.the.ferraro
Interpretive Recitations of Reactions to New Facebook

9|22|11   |   02:49   |   23 comments


Based on reactions in Nicole's Newsfeed, everyone hates this version of Facebook. This should matter to Facebook now that there's a real competitor on the scene named Google+.
Kim Davis
Facebook as Media Platform? Sigh

9|20|11   |   3:07   |   8 comments


Allowing users to share music and video on Facebook might sound like good news, but is this part of a coherent strategy, or is Facebook just stumbling from idea to idea?
what.the.ferraro
Biz Stone Joins AOL as Social Whatever

3|17|11   |   2:27   |   6 comments


Biz Stone is getting paid to be a "social impact strategic advisor" at AOL. We have no idea why.
Wisdom of the Big Chair
The Internet Rocked in 2010!

2|2|11   |   02:05   |   2 comments


A decade after the dotcom boom, the Internet continues to dramatically change the way that business gets done and individuals communicate. More than a trillion email messages traveled over the Net last year, and dramatic changes loom on the horizon.
IETV: the thinkerNet on film
5
of
Kim Davis
Big-Data Can’t Always Sell Wine

5|21|13   |   2:23   |   3 comments


Whole Foods Global Wine Purchaser Doug Bell told me about some of the constraints on using analytics in the US wine market.
Paul J. Fleuranges
Digital Signage Keeps NYC Subway Straphangers on Track

5|6|13   |   3:51   |   No comments


New York's Metropolitan Transit Authority is conducting a pilot test of digital kiosks to guide subway users to where they want to go more efficiently and at lower cost.
Kim Davis
Fast Forward to the Future

4|23|13   |   2:29   |   20 comments


A look back at tech writing in the 90s makes us wonder where enterprise IT will be 20 years from now.
Mitch Wagner
Google Launches Its Most Depressing Service Yet

4|15|13   |   2:59   |   10 comments


Google's new Inactive Account Manager lets you control how Google disposes of your accounts when you die.
Second Shooter
Argument Over Top-Level Domains Is 'Stupid'

4|11|13   |   2:07   |   3 comments


The whole Amazon.reader debate is a double-stupid. It's stupid to think that there's any e-book buyer who doesn't know Amazon's URL, and it was stupider to let ICANN launch the whole free-form TLD initiative to start with.
Kim Davis
Ladies, Your Tablet Awaits

3|21|13   |   2:22   |   37 comments


ePad Femme is the world’s first tablet “made exclusively for women.”
Wisdom of the Big Chair
NFC Moves Into the Mainstream

3|20|13   |   2:16   |   No comments


While NFC's original goal was to enhance mobile commerce applications, it is finding its way into a number of other uses, which is creating both opportunity as well as challenges for IT departments.
Wisdom of the Big Chair
Integrating Security Into Your Cloud Contract

3|19|13   |   3:35   |   No comments


Enterprises would like to move to cloud computing but are hesitant because they are concerned about providers’ ability to secure company data. Here are some tips that help to ensure that if breaches occur, the business is not left holding the bag.
Brian Baron
How Edmunds.com Collects Customer Information

3|18|13   |   1:15   |   No comments


Edmunds separates customers into segments based on the info it collects on its site and from partners, and uses that to push out custom content, said Brian Baron, director of business analytics for Edmunds.com, at Predictive Analytics Innovation Summit.
Brian Baron
How Edmunds.com Uses Analytics to Customize Site

3|14|13   |   0:47   |   No comments


The automotive website uses propensity modeling to target ads and customer registration forms, said Brian Baron, director of business analytics for Edmunds.com, at Predictive Analytics Innovation Summit.
an IBM information resource
sponsored content
big blue blog
an IBM information resource
sponsored content
Expert Integrated Systems: Changing the Experience & Economics of IT
In this e-book, we take an in-depth look at these expert integrated systems -- what they are, how they work, and how they have the potential to help CIOs achieve dramatic savings while restoring IT's role as business innovator.

READ THIS eBOOK
your weekly update of news, analysis, and
opinion from Internet Evolution - FREE!

REGISTER HERE
Wanted! Site Moderators
Internet Evolution is looking for a handful of readers to help moderate the message boards on our site – as well as engaging in high-IQ conversation with the industry mavens on our thinkerNet blogosphere. The job comes with various perks, bags of kudos, and GIANT bragging rights. Interested?

Please email: moderators@internetevolution.com
Internet Evolution – not for thickies
Keep Critical Data With a Knowledge Management System
Taimoor Zubair
Fortune 500 companies lose at least
$31.5 billion a year by failing to share knowledge. A Knowledge Management System (KMS) can help companies significantly reduce these costs.

CLICK FOR MORE
M2M: Rise of the Machines? Not Yet
David Weldon
In the 1970 science fiction thriller
Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M.

CLICK FOR MORE
M2M: Rise of the Machines? Not Yet
David Weldon
In the 1970 science fiction thriller
Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M.

CLICK FOR MORE
M2M: Rise of the Machines? Not Yet
David Weldon
In the 1970 science fiction thriller
Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M.

CLICK FOR MORE
M2M: Rise of the Machines? Not Yet
David Weldon
In the 1970 science fiction thriller
Colossus: The Forbin Project, two giant supercomputers from the United States and Soviet Union secretly join forces to take control of the collective nuclear might of the two countries. In the film, the two machines discover each other's existence, communicate back-and-forth, share their collective data, and cut their human creators out of the process. It is the ultimate example of machine-to-machine communications, or M2M.

CLICK FOR MORE
Yahoo Needs to Break Tumblr in Order to Fix It
Joe Stanganelli
As
Mitch Wagner discussed today, Yahoo is acquiring Tumblr. The big Internet debate at the moment is whether Tumblr will be good or bad for Yahoo. Regardless of their stances on the future of Yahoo itself, many claim that Yahoo will somehow ruin Tumblr.

CLICK FOR MORE