The Macrosite for News, Analysis and Opinion about the Future of the Internet
David Vellante

Flash Drives Set to Give Internet a Performance Boost

Written by David Vellante
4/15/2009 21 comments
no ratings
DISCUSS   Digg   Del.icio.us   Reddit   Email This   TWEET THIS

For more than 50 years, the disk drive has been at the center of computing architectures and the Internet, but there's mounting evidence that this is going to change powerfully over the next decade.

This past week I attended Storage Networking World, one of the storage industry's more popular meet-ups, and the most discussed topic was flash-based disk storage. Apple Inc. (Nasdaq: AAPL) first introduced NAND (flash) storage on its iPod Shuffle in early 2005, and while some dismissed the announcement as a niche for toys, others began to look at pricing forecasts and realized that, for certain applications, flash-based storage would begin to replace spinning disks in the early part of the next decade.

In fact, selectively, it's already happening today. Virtually every major system and storage supplier has announced support for NAND flash drives in storage arrays. NAND drives have much better response times and faster access times than spinning media. The disadvantages are cost and reliability -- i.e., the number of writes to each memory cell is limited and has to be managed with additional layers (and more cost).

But for Internet applications, where much of the activity is simple reading of the disk, this is potentially good news. Specifically, flash prices are dropping faster than those of spinning media and will begin to replace spinning disks in certain applications. New architectures are possible where the highest-performance Internet data (e.g., indexes, search metadata, certain database activities, etc.) can be serviced from flash-based storage, while less performance-intensive data will be placed on the very highest-capacity, lowest-cost spinning devices, and, in an effort to save energy, spun down when not in use.

It is important to distinguish between high-performance drives and high-capacity drives. The Achilles heel of disk drives is access time limited by rotational delay. Expensive controllers with RAM caches have been designed to mitigate this problem. With the high performance and low access times of NAND storage, the major type of drive that is likely to be initially replaced by NAND devices are high-performance Fibre Channel (FC) drives. Although high-capacity disk drives (SAS or SATA) may eventually be replaced by NAND drives, that is well beyond any near-term planning horizon for users and vendors.

So one key planning question is whether NAND storage will obviate the need for high-performance FC disk drives. And if so, when?

The chart above shows three price curves for flash at varying rates of decline (50 percent, 60 percent, and 70 percent annually). Current trends since last summer show that the actual reduction of NAND prices is about 60 percent per year. At this rate of comparative reduction, FC drives will be obsolete in less than three years time (2011-2012).

While much of the Internet's data will reside on spinning disk for quite some time, it's inevitable that flash technologies will begin to permeate computing architectures and alleviate an age-old problem that mechanical disks were never truly well suited for computer applications.

— David Vellante spent 15 years at IDC and is a founder of The Wikibon Project. He can be reached on Twitter at @dvellante.

DISCUSS   Digg   Del.icio.us   Reddit   Email This
Current display:       newest comments first       display in chronological order
Page 1 of 3   Next >
ferodynamics
IQ Crew
Saturday April 25, 2009 12:11:05 AM
no ratings

From a selfish end-user perspective, this is about heat and noise.  After the Mac Cube (http://en.wikipedia.org/wiki/Power_Mac_G4_Cube) I really thought there would be more of a push for silence, but apparently not.

Look at the Dell Adamo.  http://www.dell-adamo.nl/wp-content/uploads/2009/01/dell-adamo-white-04_large.jpg

Now they're clearly selling this as a sexy machine, the ads show runway models posing with the thing.  But all I see is the ugly air vents on the side.  Come on, this would be a slimmer, more beautiful machine w/o a spinning drive, don't you think?

My dream machine: a Thinkpad netbook (for the keyboard and trackpoint) and no moving parts.

lpricci49
IQ Crew
Sunday April 19, 2009 6:22:38 PM
no ratings

I loved your story.  Back in about 1974 I could smell the lanolin too.

I agree 100% that there will be a rethink about the CPU/Memory/Disk relationships.  Current architectures assume all sorts of conventions (standards) at each point of the interface. 

Sooner or later someone with some access to capital will review the CPU/Memory/Disk relationship from the point of view of function- for example a HTML server or a Video server.  Everything form the disk sectors on out needs to be defined in view of the intended purpose.  I expect 10X performance at the same level of semiconductor/storage technology will be possible.

Lawrence Ricci
www.EmbeddedInsider.com

'

 

David Vellante
Thinkernetter
Saturday April 18, 2009 6:53:29 AM
no ratings

Wow Rob! That's a tremendous post. Your point is right on, this concept is not new and architects like yourself have faced the spinning media bottleneck forever in this industry.

And for those of you born after 1985, by "sketched out in a notebook" Rob means written in pen or pencil in a 2-fold, sturdy piece of vinyl or leather bound plastic that houses sheets of lined paper :-)

rswinney
IQ Crew
Saturday April 18, 2009 1:49:57 AM
no ratings

Companies probably do not like the expenses but when considering various laws, particularly those involving privacy, and safeguarding company data, they are willing to sacrifice cost for high performance and durability.

(Besides, they will figure a way to pass the cost to their customers.)

roblake
IQ Crew
Thursday April 16, 2009 5:31:51 PM

Folks -

I'm reading the comments with a big grin.  Perhaps you would be interested in how the "solid state disk" was invented?

Back in the summer of 1970, I was consulting with the Laboratory Data Products Group at Digital Equipment Corp.  Housed in the old woolen mill in Maynard, Mass, the floors were still soaked in lanolin from the blankets made there during the Civil War.

The task at hand was to write a gas chromatographic analysis package to run on a PDP-12.  The 12 was a chimera of a machine --- a PDP-8; a LINC-8; and a still-under-development Floating Point Processor (FPP).  "Mass" memory on the 12 was a DECtape  --- a hand-sized, wide magnetic tape.  The FPP was a 17" by 8' rack.

Software assembly with DECtape was a multi-hour process.  However, on weekends, there was a "production" 12 used to generate new DECtapes to ship with new 12's.  It had a fixed-head hard drive.  One head per track.  Fast!  I could assemble in minutes.  The value of a fast, rotating drive with no head-movement delay was embedded in my head.

Several months later, back in my computer lab in Cleveland, running on the PDP-11 that DEC kindly provided in exchange for my successful software effort, there was a DR-11-M (?) Direct Memory Access controller to let a peripheral device move data directly to/from the PDP-11's core memory.  In an inventive moment, I realized that one could emulate that fixed-head hard drive in the same manner as the FPP had been connected to the PDP-12, but using a rack-full of memory boards.

It took until 1974 to find a company willing to share the development of what-came-to-be-called the Extended Memory Unit (EMU) [After the fast, flightless bird].  Monolithic Systems in Englewood, CO, a maker of semiconductor memory boards picked up the concept and turned it into a successful product.  Rumor was that some EMUs were on nuclear submarines, where there was no gyroscopic torque from a spinning hard drive.  I built a very-high-speed data acquisition system where data from explosions went through analog-to-digital converters into the EMU, then sent a single interrupt signal to the host PDP-11/45 computer.  To the PDP-11, it seemed that in a microsecond, a disk full of data just appeared on a hard drive.

A graduate student in Computer Science at CWRU studied the EMU performance.  Instead of being 17,000 times faster than DEC's RF/RS-11 fixed-head disk [17 millisecond rotational latency versus 1 microsecond access time for the semiconductor memory], the performance was far less, due to the fact that the software device drivers in DEC's OS were written with a rotating drive in mind.

Which brings us to the current "Flash Drive" flurry  ---  I believe that there will soon be a shift in system architecture.  Rather than word-at-a-time or block-at-a-time data transfers, computing will be done where entire large blocks of data are transfered with a simple interrupt signal.  Processing algorithms will be re-thought such that FPLA's can do the processing of these large blocks of data in a single clock cycle.  The market might evolve with the creation of smart, ultra-fast peripheral devices.

All this was sketched out in a notebook in the mid-1970's.  I never throw anything away, so I'll dig through the storeroom and see what explicit ideas I had around this back then.

 

David Vellante
Thinkernetter
Thursday April 16, 2009 1:27:18 PM
no ratings

Well thank you Mary, I appreciate that. And I appreciate your pushing at my assumptions and premise here to make them better. What good is a prediction that's not surprising? (Although I didn't honestly see this as that outlandish).

I see a domino effect coming where the price of flash drops at or near (or below) spinning disk and a leading vendor or a few, in an effort to get competitive advantage and lower the cost of storage, adopt early and push hard. I predict EMC, IBM and HP will lead this charge. Maybe NetApp too. 

In re-thinking (and re-thinking) your point about vendor adoption, it strikes me that the unwillingness to switch may be due to the need to re-architect storage systems to accommodate flash. That will cause friction. But I believe these leaders are already doing just that and the rest of the industry will be forced to follow.

So I'll adjust my bottom line a bit. It still comes down to price but I'll concede switching costs are a factor and they are not fully eliminated because flash drives can plug directly into FC slots. I'll admit there are clear costs to re-design array architectures to accommodate flash. The fact that flash will plug into existing interface slots is key because it will drive the first wave but full scale adoption will require redesigns.

New prediction...those vendors who don't re-design their arrays to accommodate flash will begin to lose market share in the 2011 timeframe.

Mary Jander
Thinkernetter
Thursday April 16, 2009 12:58:35 PM
no ratings

Fascinating stuff, Dave. However, I'm still a skeptic about the big vendors' willingness to make the switch. If I were a wagering person, I'd say you'd have a bet that FC won't lose ground to Flash until way after 2012. Still, your case is well put and this industry is nothing if not surprising.

David Vellante
Thinkernetter
Thursday April 16, 2009 12:26:11 PM
no ratings

15,000 rpm even Mary. But it's mechanical. You're comparing a spinning disk with semiconductor speeds. Make it 50,000 rpm (if you could dissipate the heat and handle the data rates) and it's still an order of magnitude slower than semiconductor...not to mention the time it takes to to physically move an actuator arm, which hasn't really improved in years.

The other factor here is processor speeds double every 18 mos but disk drive speeds get worse every 18 mos. Why? because the more data you put under an actuator, the slower disk drives get. It's a problem referred to as 'access density,' meaning the disk has to spin more and the arm has to move more to find the data to fulfill an i/o request. This is why customers 'short-stroke' disk drives - i.e. they leave them empty except for the critical files (e.g. database indicies and the like). There are gazillions of FC drives installed that are basically empty for performance reasons.

Bottom line, compared to the speed of light, mechanical disk drives are sloooooooooooooooooooooooow.

 

 

David Vellante
Thinkernetter
Thursday April 16, 2009 12:05:02 PM
no ratings

Design good software to minimize the need for faster hardware...interesting concept and one that's been talked about before. One of the major limitations of applications is the need to access spinning disk. For decades we've dreamed of keeping "Data in Memory" (DiM) as a single level store. Flash enables a form of that vision and application developers will exploit it.

Here's what I know...give application developers fast i/o resources and they'll use them in ways your or I never thought of. New supercomputer applications are popping up everyday using the Internet as a resource and the applications are following.

Are you arguing that the state of Internet applications is frozen-- or should be?

My strong belief is Internet applications will exploit flash technologies to create better user experiences-- it's stating the obvious.

Mary Jander
Thinkernetter
Thursday April 16, 2009 11:29:16 AM
no ratings

"The only spinning disks left will be ultra high capacity devices that will maintain an advantage over flash for quite some time to come."

This seems to work against your claim that Fibre Channel will be replaced by flash within the next 3 years.

Even if you go with speeds, FC RPMs are pretty durn high -- up to 7200 for 4-Gbit/s, no?

Now, how much energy they use might be a key factor.

 

Page 1 of 3   Next >
The ThinkerNet does not reflect the views of TechWeb. The ThinkerNet is an informal means of communication to members and visitors of the Internet Evolution site. Individual authors are chosen by Internet Evolution to blog. Neither Internet Evolution nor TechWeb assume responsibility for comments, claims, or opinions made by authors and ThinkerNet bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
previous posts from David Vellante
David Vellante
Lately, I've been hanging around some cloud technologists, cloud service providers, virtualization customers, and security practitioners. I've been asking a lot of basic questions, trying to understand when and how cloud computing/virtualization will be ready to support any application or workload. UPDATED 2/2 4:00 PM
David Vellante
David Vellante   1/25/2010   25 comments
In a shocker of a news flash, the Chinese government has flatly denied any involvement in the cyber attacks on Google (Nasdaq: GOOG) and other Websites.
David Vellante
David Vellante   1/21/2010   5 comments
No matter what you call it -- virtualizaton, cloud computing, or "The Big Switch" -- a return to centralized computing is a widespread trend in our industry. And the general consensus is this movement will continue for a decade or more.
David Vellante
David Vellante   1/14/2010   3 comments
Last August at VMworld, VMware Inc. (NYSE: VMW) CEO Paul Maritz told me in the hallway, “We’re at war with everybody.” He was being flip, but this was no joke. The company had just gone out and paid $400 million-plus (about 20x revenues) for Springsource, an open-source Java framework developer.
David Vellante
David Vellante   1/6/2010   9 comments
Just like every other social media platform these days, Digg has Twitter envy.
5
of
IETV: the thinkerNet on film
5
of
2pm EST
Tue
Feb 23rd
2pm EST
Thu
Mar 4th
3pm EST
Tue
Mar 9th
an IBM information resource
sponsored content
big blue blog
Todd Watson
IBM is announcing today the first of its Power7 processor-based systems and the Power7 processor itself at an event in NYC.
white papers & case studies
an IBM information resource
sponsored content
Smarter Collaboration: How to Thrive in a Challenging Business Environment
Market conditions are changing faster than ever, and organizations need to improve their agility and adaptability in order to provide better service and improve processes. The ability to work with customers, business partners, and employees as effectively as possible - while at the same time holding down costs - is a key to success.

READ THIS eBOOK
your weekly update of news, analysis, and
opinion from Internet Evolution - FREE!

REGISTER HERE
Wanted! Site Moderators
Internet Evolution is looking for a handful of readers to help moderate the message boards on our site – as well as engaging in high-IQ conversation with the industry mavens on our thinkerNet blogosphere. The job comes with various perks, bags of kudos, and GIANT bragging rights. Interested?

Please email: moderators@internetevolution.com
CMP Media LLC
Internet Evolution – not for thickies
Congress Hits the Snooze Button With China
Ira Winkler
In his
recent Congressional testimony, Dennis Blair, the U.S. director of national intelligence, stated that the U.S. is "severely threatened" by cyber attacks and that the recent Google (Nasdaq: GOOG) attacks should serve as a wake-up call.

CLICK FOR MORE
what.the.ferraro
Twitter Conferences & the Demise of Humanity

10|13|09   |   1:56   |   6 comments


The rise of industry events centered solely on the topic of Twitter is enough to make some people cry. Literally.
Steve Saunders' Outernet
Internet Evolution Goes Soccer Crazy

1|28|10   |     |   14 comments


Steve Saunders visits the South African Consulate in New York for a party celebrating not one, but two (2!) major football (soccer!) tournies.
John Soat
Technology Santa Claus

12|23|09   |   2:06   |   2 comments


In the holiday spirit of giving, Technology Santa Clause offers a few words of advice to struggling IT professionals: ‘Be careful what you wish for.’
Reiter's Block
Tweeting for Customer Support

11|18|09   |   2:20   |   No comments


When Reiter gets incensed over incompetent Verizon FiOS order-taking and support, he broadcasts it via Twitter. Did it do any good? How should your company offer Twitter support? Watch this for all the answers.
The Incredible Hultquist
Tweet Less, Get More Clicks

11|9|09   |   2:24   |   No comments


Evidence shows that you can tweet too much. Sites and services like Twitter and Facebook are a good place to reach your audience, but think quality over quantity.
The Incredible Hultquist
Web 2.0 – Just Being There Isn't Enough

11|3|09   |   2:15   |   9 comments


As enterprises leap into the Web 2.0 world of blogging, commenting, and social networking, just 'being there' won't deliver ROI. You may want a 'Web Evangelist' to systematically harvest the feedback in order to polish your product or service.
Thomas S. Kunz
Banks Get Social

10|19|09   |   1:32   |   1 comment


The PNC Financial Group is using social networking for outreach, product development, customer segmentation, branding, and more. It has also developed an iPhone-based virtual wallet application for Gen Y customers.
The Incredible Hultquist
Social Networks & Hiring Pitfalls

10|16|09   |   2:16   |   5 comments


More companies are trolling social networks to find and vet potential job candidates. Beware the pitfalls of blurring the line between personal and professional lives.
Steve Saunders' Outernet
The Coming Internet Bubble: Part 2

Part 2 of 2   |  
See complete series
10|16|09   |   3:38   |   18 comments


How do you recognize an Internet bubble when you see one? Saunders explains how all bubbles have four symptoms in common – and takes a swipe at Google and Twitter into the bargain.
what.the.ferraro
Flo 2.0: Mrs. Brady Gets a Website

10|15|09   |   1:46   |   11 comments


Florence Henderson, of Brady Bunch and Wesson Oil fame, is pitching a tech support site for digitally challenged seniors.
Lee H. Berke
The Decline & Fall of Broadcast Television

2|9|10   |   1:00   |   No comments


Want to know the future of broadcast television? Take a look at broadcast radio’s past.
Tom Nolle
Everything New Is Old Again

2|9|10   |   2:13   |   6 comments


Research shows that the youth of today like Facebook – but not blogging or Twitter. Does that mean Facebook has won, or just that it's not yet out of favor? Will all the services we see today fade into Ovaltine-or-Wheaties status in just a few years?
what.the.ferraro
Email Marketing Gets Desperate

2|8|10   |   2:31   |   4 comments


Promotional emails will use just about anything timely to get people to buy things. Seriously, anything.
Steve Saunders' Outernet
America, Truck Yeah!

2|8|10   |   1:42   |   5 comments


Steve likes his new Dodge Ram 1500, but hates Chrysler's Web non-sales strategy. Rant on, li'l buddy.
what.the.ferraro
Twits Go Wild for Resignation Tweet

2|5|10   |   1:48   |   4 comments


Jonathan Schwartz is the first Fortune 200 CEO to resign via Tweet. Can he walk on water, too?
Full Nelson
Go With the FLO, Part 2

Part 2 of 2   |  
See complete series
2|5|10   |   2:17   |   3 comments


Fritz and his sweater continue their review of Qualcomm's FLO TV.
Singer at C-Level
Goldilocks & the Data Center

2|4|10   |   3:39   |   2 comments


What kinds of companies are doing the most innovation in the data center? Turns out it's midtier enterprises that are taking the "Just Right" approach.
Full Nelson
Go With the FLO, Part 1

Part of 2   |  
See complete series
2|4|10   |   2:39   |   1 comment


Qualcomm's FLO TV gizmo streams live TV shows. Tragically, they include the O'Reilly Factor
Eurotrash
High & Dry in Barcelona

2|3|10   |   1:08   |   No comments


Ray’s heading to Barcelona for the Mobile World Congress, and he’s not happy about it, the miserable git.
Sweeney Blog
No Sex, Please... It's the Super Bowl

2|3|10   |   2:24   |   2 comments


The Super Bowl ads that CBS rejected are turning up online, generating lots of attention but zero revenue for the broadcaster.