Road to Recovery

Data protection management has come a long way in the past decade.  More importantly the features and functionality that are in products these days and what customers have come to expect are now no longer ‘nice to have’ feature in the data center, they are ‘need to have’ features.

Additionally, the term ‘data protection’ is morphing every day and has different meanings to different people.  Questions like ‘is replication data protection?’ or ‘is archive data protection?’ or ‘is DR / BC a function of protection?’ are now common in IT circles.  Each in their own right is a methodology for protecting information or has some play in the grand scheme of data protection.  The reality is, much like every answer in IT, the answer to these questions is ‘it depends’.  Data Protection has many different definitions, which start to expand the scope of what it actually is and more importantly, how it is managed cost effectively across the whole environment.

It is this expanding scope of data protection  where data protection management tools come into play, and the more flexible and granular the tool, the more effective.  It is hard to have good data protection capabilities without having insight to the environment.  First, understanding what type of data lives in the environment, where it is, how it is used and some characteristics about its age or its access frequency helps to determine how to best protect the information.  This is where a data protection management tool that provides some insight to the file system adds a great deal of value.

Next, if archive is a part of data protection (and I would argue that a functional archive, when used properly, is) then a data protection management tool that provides insight to the data in the archive can also help manage the overall protection process within the greater environment.  Knowing if the data in the archive is actually being accessed or if it can be deleted (unless stored for compliant purposes) can help to control archive costs.

If replication is a part of the overall data protection scheme, a data protection management tool that provides insight to this process can also add a great deal of value.  Identifying if links are up, if data is moving between sites and if the data is available, accessible and meets my recovery point objectives at the remote site can ease the concern of recoverability in the event of a disaster.

And finally, providing as much information as possible such as deduplication rates,  tape growth, disk growth (in disk based backup targets – including deduplication targets), as well as providing true analytics into the backup environment to help make decisions as to when to switch from a tape-based solution to a disk-based solutions.  These analytics need to be in-depth enough to show that if some data that is being protected with traditional backup technologies are moved to a next generation solution, such as source-based deduplication, then what affect will it have on the overall backup environment, will it help to better control costs, will it help to increase SLAs?

At a higher level, customers are telling me that they no longer want to manage backup, they just want it to work and they want proof it is working.  As customers move to a more virtualized IT infrastructure, they find that they are being forced to rearchitect their data protection environment and they are now looking to solutions that elevate the process.  IT is looking for tools to make their environment “data protection aware.” As virtual machines are added to the environment they are automatically protected and want notification if they are not so they can mitigate any risk, and let’s face it, backup is all about risk mitigation.  Backup is insurance.  Wouldn’t it be nice if your insurance company had deeper insight to all the cars / drivers in your family and told you when your teenager was speeding on a monthly basis and told you that your premiums are going to go up if they don’t start driving the speed limit before they got the ticket and your premiums increased?

Any tool that IT invests in for a common process, data protection in this case, needs to be flexible enough to allow IT to manage as much of the overall process from a single pain of glass.  Good data protection management tools need to provide IT as much visibility into the overall data protection environment as possible in order to help make good decisions about what data technologies should be invested in, in order to help IT meet its overall SLAs and hence business objectives.

There is no sense spending a great deal of money on rearchitecting a backup environment if there is no insight to the success of the new architecture.  Sooner or later, management needs to have the pretty graphs that prove to someone that the right decisions are being made when it comes to protecting information, or when it comes to how much is spent on data protection or if the SLAs can be met.  Not having good data protection management tool, and spending too much on new data protection architectures while not meeting your SLAs could lead to a RGE (resume generating event).  Data protection management tools today are a need to have, not a nice to have.  Make the investment and put your data protection environment back on the Road to Recovery.

Tweet This Post

The Architecture

This ‘architecture’ diagram, as you can see, is not a typical architecture diagram, but hopefully it can be used to align your business and business objectives with the technologies that are available and can best be applied to solve your issues helping to balance, cost, complexity and compliance.

This diagram can also be used to do a couple of other things.  It can help you begin to classify your data and align your  data to your business objectives.  It also lets you begin to identify what data or data services in your environment that may be more important to you than others and based on this help you to choose areas you may want to outsource or move to the cloud.

As you can tell, there really is not one solution for meeting all your data protection needs.  The challenge comes with managing multiple solutions in an effort to meet your business objectives.  While there are only a few technologies available that allow you to manage your environment across all your RPOs and RTOs, it is important that I point out EMC’s NetWorker is able to do this, centralizing your data protection infrastructure  for ease of management.  It allows you to manage traditional backup, source based deduplicated backup with Avamar, CDP with RecoverPoint, as well as the EMC disk libraries and tape where the data is stored.  Now, I am not saying that NetWorker solves all of your data protection challenges, nor am I suggesting that replacing one traditional backup technology for another is the right answer, but what I am saying is that if you’re looking to have all the feature functionality required to meet all your business objectives and you want easier management, NetWorker is one avenue to get you there.  Additionally, the underlying image of the triangle represents data protection management.  Putting all the new technology in place is one thing, managing it, and ensuring you are now meeting your business needs is another.  EMC’s Data Protection Advisor can help here as well.

This diagram can help customers layout a new, better data protection schema for their environment and start thinking about data protection a bit more strategically versus tactically.  It can also help vendors speak to customers about how they should look at their environment in order to identify specific challenges and the means they need to alleviate these challenges , taking backup, beyond.

Tweet This Post

One of the more thoughtful analysts in the industry, in my opinion is George Crump from Storage Switzerland.  (I like the name and George is as independent as you can get in

this business.)  Yesterday I had the pleasure of briefing George on EMC’s Data Protection Vision.  I like talking with George for a couple of reasons.  First, he gets it.  What does that mean.  Read his material.  He is genuinely trying to educate IT folks on what is really important in the data center and how to address these challenges.  Next, he keeps the ‘pay for’, ‘vendor spin’ to a minimum.  George works hard to just talk about the facts of a product or industry and talk about how products can help without selling.  The reality is, we live in a great technological time.  The problem with IT is that only 50% of the problems are technology related.  The other 50%  is psychological.  IT can’t just implement new technology because its cool or even because it really does solve a problem.  Sometimes new technology is too expensive to implement or the solution that is currently in place had a three year amortization and your only two years into your product life.  Or, more importantly, the new technology may be the greatest technology at the right price but it doesn’t fit into the current IT priorities.  These are all things IT needs to work through when considering whether or not to invest in new technology.  The other thing George and I spoke about was the fact that it gets difficult to be ’strategic’ in IT especially given certain economic times.  A lot of times IT just needs a band-aide or quick fix to move on to more important issues that really drive the business.  I talk about this  a lot, especially when it comes to backup.  Lets face it, it may not be what we all want to hear but backup is not strategic to most environments.  The applications that drive the business are most important.  Backup is about risk mitigation and information availability if everything else fails.  Right, ‘if everything else fails’, and IT typically invests in technology in the front end in an effort to have as little failure as possible.  Meaning, IT doesn’t just buy JBOD with no RAID if they think the environment shouldn’t be put at that kind of risk.  So IT is  already investing in some risk management up front which drives the spend on the back end for data protection.

I wanted to say “Thanks” to George for taking the time to come in and understand the bigger strategy EMC is driving with its products in the data protection space and to talk about our existing successes with the current portfolio.  Hopefully George, as well as all of you, can see how we are helping to put customers on the Road to Recovery.

Tweet This Post

This blog will have multiple parts.  I will introduce my view of a data protection reference architecture and the next few blog posts will talk to components of that architecture.

The other day  I had a very interesting conversation with a colleague of mine in Australia.  He was looking for a data protection reference architecture that he could use to speak to his customer.  As you can imagine having this conversation over the phone could pose to be a difficult challenge.  When the conversation began, my fear was he was looking for an ‘architecture’ diagram that included data protection appliances, backup servers, disk libraries, tape libraries and backup agents.  I quickly realized that this is an impossible conversation to have with him without knowing:

A)     the customer’s environment or challenges

B)      the customer’s business objectives

I find that most vendors don’t know A or B when speaking to a customer about their data protection ‘issues’, but they really should.  Having a more thoughtful conversation with customers in a consultative fashion is more relevant to customers in understanding their challenges and helping to align these challenges to the best possible solution.

I started my conversation with the diagram shown below (Figure 1).  A simple triangle divided horizontally into 4 segments and the middle two segments divided vertically in half.  Each segment represents different business objectives within a company.  As you go around the triangle, you can see that there are different technologies and different methodologies for attacking data protection challenges, which is why there is no longer a “one size fits all” approach when it comes to protecting data today. Let’s face it; the two most important commodities in backup are time and capacity.  One of the primary drivers behind the type of protection that is used is the Recovery Point Objective or RPO.  Different technologies provide different RPOs and each has a different price point as well as there are different processes that can be applied to attach RPOs.

Figure 1

Having a conversation specific to this diagram can have a tremendous amount of value on a number of fronts, including; aligning technology needs with business objectives as well as highlighting critical pain points and beginning a roadmap that helps implement data protection technology based on business needs and budget and put you on the Road to Recovery.

The next post will cover the foundation of the triangle – Archive.

Tweet This Post

I was walking through the data center the other day when I heard one of my colleagues, MJ “Scream”, “I wish I had some ‘Morphine’”.  Well, I have to say I was “Speechless”.  I walked over to where MJ was standing, near the tape library, and when I asked him what was wrong, he replied “there was another backup tape ’Jam‘.”  MJ told me he had been “Working Day and Night” on a major backup problem and he was now bouncing “Off The Wall”.  He told me he was sick of dealing with traditional backup tools and just wanted to get rid of tape.  I told MJ that it was “Human Nature” to feel “Bad” in a time like this but I also told him, “You Are Not Alone”.  I said MJ, “’Keep The Faith’, we all ‘Remember The Time’ when backups ran like a ‘Speed Demon’ and were ‘Unbreakable’, but that is ‘HIStory’, tape isn’t that fast any more given the amount of data we now have.  I also told him that “We are Backup Administrators, we are ‘Invincible’ and ‘Heaven Can Wait’ for us, and while we may not have our issue fixed at the ‘Break Of Dawn’, we would ‘Come Together’ to ‘Heal The World,’ or at least the datacenter’ (I chuckled).  I proceeded to tell him about a revolutionary new backup concept utilizing source-based deduplication technology.  It’s “PYT”, a pretty young thing, but  more importantly it’s here to stay.  EMC  offers it with a product called Avamar , the most efficient variable block,  source-based, deduplication technology on the market that:

• Helps to eliminate tape all together
• Is perfect for VMware environments
• Protects remote offices most efficiently
• Stems the tide of data growth on NAS platforms

Well I thought MJ was going to give me “Trouble” for my comments.  I mean it, all of the sudden I had “Butterflies”, I felt “Threatened” because I knew this guy could be a loose cannon when it came to trying something new, he could be “Dangerous” he may moonwalk over to me and slap me with his glove. Change can be scary.  But just then MJ let out a “Smile” (quite frankly I thought he was going to “Cry”) and said “’I Can’t Help It’, my job is ‘On The Line’ and I ‘Wanna Be Startin’ Somethin’’ soon before my boss tells me to ‘Beat it’” he just felt “2 Bad”.  I told him, “’Don’t Walk Away’ and ‘Whatever Happens’ ‘Billie Jean’ and I were going to help get him out of ‘Trouble’ and together we would replace the tape infrastructure, make backups run 10x faster, provide him with tools that actually verified his backups and make his backup problems ‘Ghosts’”.

I called Billie Jean and at first she said, “’Leave Me Alone’, ‘Why You Wanna Trip On Me’”, but I told her we need her help, so she said she could help MJ and I.  When she asked what the trouble was, I told her that our backup environment was in shams and if MJ didn’t get it fixed, with the right solution that they were going to put MJ on a “Carousel”, that there would be “Blood On The Dance Floor” and he would end up being “Someone In The Dark” “In The Closet”.  Billie Jean hopped on the phone and called “Dirty Diana”, we are all “Just Good Friends” really.  She told her the story and when it came right down to it, it really was “Black or White”.  We needed some “Money”, “2000 Watts”, to replace the old tape libraries with the new Avamar technology and “One More Chance” to fix all of MJ’s backup issues.

I told MJ the plan; we were going to sneak past the guards (that would be simple because “They Don’t Care About Us”) and then replace the old equipment with the new equipment.  MJ asked, “’Is It Scary’ in the datacenter at night?”  I told him we would be fine, that this would not be like his “Childhood” days.  MJ just said, “I Wanna ‘Rock With You’”.  The next night we snuck into the data center like a “Smooth Criminal”.  First, we had to “Get On The Floor” the new Avamar technology.  Next we installed Avamar and it fixed our backup problem right away.  I said, “Man ‘Is It Scary’ or what?”  “Another Part of Me” was just proud of the work we had all accomplished.

The next morning we went into the office of “Little Susie” and knocked on her door (it was always closed because she liked her “Privacy”).  She was MJ’s boss and she was no “Tabloid Junkie” she was a real “Superfly Sister”.   She said, “’Who Is It’”?  We told her and she let us in.  We showed here some reports we had generated from another product we acquired called Data Protection Advisor.  We showed her where all the previous backups had been failing due to problems with network performance, tape libraries and not enough time to back everything up.  Then we showed her that with Avamar we were backing up data in just 1 hour with 100% success because we were seeing 99.5% duplicate data in our NAS environment and that was why we couldn’t meet our backup windows with tape.  We also showed her that our VMware environment could go from 10 to 20 virtual servers per ESX host because backup was no longer the bottleneck keeping us from implementing more virtual guests.  Well she was pretty happy, she said “You Rock My World” and she was not upset that the tape environment was “Gone Too Soon” because it was a true “Heartbreak”.  I told her it was a team effort and we couldn’t have done it without the help of a lot of people including EMC. It was a real “Thriller”.

Tweet This Post

Tweet This Post

Once upon a time (about 18 months ago actually) data protection was considered one of the most boring areas of the storage market.  If ever there was an area ripe for change (in fact, ripe for an entire paradigm shift), it was backup.  Well, ask and ye shall receive.  Data protection is now the most dynamic area in storage today.

In the interest of brevity, I’ll confine this discussion to just backup (although there’s a lot happening in replication too).  First, let’s start with tape.  For most companies, tape now comes in two very distinct flavors: real (the old fashioned kind) and virtual (really a disk library).  For the most part, the virtual kind of tape is eventually written out to real tape for vaulting (as well as costs and long term storage) purposes.  Secondly, there’s deduplication.  Deduplication comes in many flavors, but the net effect is that less data is stored (sometimes, less data is moved over the wire as well).  Deduplication is complicated because not all data de-dups well.  It’s good to know which data does and which data doesn’t (by the way, this often depends on the deduplication solution being used).  Thirdly, there’s virtualization.  Now, virtualization is not a data protection technology, however, it is the impetus behind this inflection point for backing up data.  Virtualization basically destroys the old fashioned “back everything up to tape every night” backup strategy.  Why?  For starters, take the most I/O intensive process in your whole IT operation, backup, and layer it on top of the worst technology available for I/O performance, virtualization.  Also, in a virtualizaed environment, there is a lot more data because there are a lot more servers.  Additionally this data has a ton of redundancy.  Lastly, virtual servers raise a huge number of configuration issues.  It’s not as simple as the old days when a server was really a server, and it was backed up to a physical tape.  If you don’t get this right, recovery can be unbelievably fun (e.g., sorting through tapes to figure out what data was where on a given day) (NOT!).

Enter Data Protection Management (DPM)…

DPM has been one of the fastest growing sectors in storage software for all the reasons stated above.  Put another way, backup is: too expensive, too risky and too hard to properly manage.  These are the problems that DPM solves.   Most DPM products are focused on backup/recovery today,  however this is changing rapidly.  Vendors in the space are hearing from their customer that they to manage the entire stack of data protection technologies including replication as simply, cheaply and with as little risk as possible.  Fundamentally, customers are telling me that they want to be able to apply one service level to critical production data, another to email and a thrid to less important generic user data.  These SLA’s cover everything from recovery time objectives (RTO) to retention periods.  Customers want to  be able to leverage old and new technologies (traditional tape backup and deduplication for example) to create an efficient, cost-effective data protection environment that meets their business requirements for availability.  Using DPM, you not only guarantee efficiency and utilization levels (thus eliminating costs by purchasing more capacity than is needed), you also reduce risk and manage configuration changes and such that occur in today’s hybrid virtual/physical environments.  Customers have told me that they are seeing payback of 12 months or less in real, hard dollars after implementing a DPM solution.  That’s real money.  As an added bonus, they also sleep better at night knowing that their data protection policies (especially policies associated with recoverability) have been rigorously enforced and that they can cleanly demonstrate this via flexible reports to anyone who cares.  To me, DPM goes hand in hand with all the new, emerging data protection technologies.  Given the payback period, one might wonder why every company hasn’t implemented DPM.  The surprising fact is that an ever increasing percentage have implemented DPM and are already reaping the benefits.

Posted by Alan Atkinson

Tweet This Post

Last week I took a course offered by EMC entitled ‘Lean Six Sigma’ – Yellow Belt. This is a training course that is used to help ‘solve problems’ in a given process, typically work related. When I think about where the biggest problem is in IT its in the Backup arena so I thought, what a better place to test it.

Enterprise Strategy Group 2008

There are two components to Lean Six Sigma. Lean or Leaning a process is about removing excess from a process to make it more efficient. For backup, moving as much data out of the backup stream as possible would increase backup efficiency.  Deleting unnecessary data or archiving static data in the production storage can cut down on as much as 50% of the data in the backup, ‘leaning’ the process.

Next, when looking at Six Sigma, we learned about the DMAIC process. That is:

• Define – Business case, scope, problem statement, goals
• Measure – Process flow, run charts, Pareto charts
• Analyze – Cause / Effect, waste identification
• Improve – Waste removal, improve plan, control charts
• Control – Monitor to prevent repeat failure, control charts, control plan

First, as I was thinking about this, I kept coming to the measure phase. If you don’t currently measure your backup process, unless of course only when there is a recovery failure, then perhaps its time to invest in a tool to help measure the current process.  This measurement will allow you to identify current problems, serving as a benchmark against wich you can measure the success of your ‘leaning’.  So, if we apply the steps in the DMAIC process to your typical backup environment, here is what it may look like.

Define

In the first step or the define step the objective is to describe the business case and problem statement and set SMART (Specific, Measureable, Attainable, Realistic and Timely) goals. Again, the key will be in the measurements but typically with backup you want to measure recovery success, which ends up being a result, most of the time, of backup success. The objective is to take a look at your existing recovery success rate, and hence your backup success rate and identify what you would like the percentage of successful backups and recoveries to be. I would guess in most cases shooting for 100% would be the requirement, but perhaps 99% is fine. So the problem statement would be: data recoveries fail more than 54% of the time and this data loss contributes to employee frustration and can translate into significant risk for the company during a legal disclosure process. The reasons these recoveries fail are due to a flawed, multi-step process that needs to be examined and fixed in order to yield a 99% success rate when it comes to recoveries. This process specifically affects backup administrators on a daily basis. When the process is fixed and recoveries yield a 99% success rate, the customer benefits, the end users, executives and customers satisfaction will keep the company’s corporate costs low and drive  repeat business.  Additionally, in the define phase, it may be good to create an IPO diagram. IPO stands for; Input, process, output.

Measure

In the Measure phase you will want to make sure you have metrics that can identify the following; recovery success rates, backup success rates, dollars lost due to failed data recovery, customer complaints due to failed data recovery, and the costs to backup and recover data for the environment. These would be the key metrics to understand and to fix the problems that are uncovered. It will be important to establish a baseline to improve upon, and this is where having tools in place (such as DPA) can help tremendously throughout this process. This will also be a good place to ‘map out’ the current process flow and make sure to identify what is in scope and what is not in scope in order to avoid ‘scope creep’. To review the process flow it may be useful to create a process flow chart using a whiteboard and post it notes. Brainstorm all of the steps and put them on the whiteboard in random placement. Organize in time sequence and then fill in the missing steps and review for completeness. This will be helpful for the analyze phase.

Analyze

Next comes the Analyze phase. This is one of the best places to start to identify ‘waste’ in the process and see how the process can be ‘leaned’. (It is also, particularity for the backup process, a good place to see where the data can be ‘leaned’.) Identify waste and poor performing areas of the process for both the backup and recoveryflow. Brainstorm as to where the process breaks down and what pieces may fail. It will be important to take a look at the overall daily trends as well as the weekly trends to see if there are any anomalies in the process. Typically backups are daily incremental and weekly fulls, so you want to make sure that there are no flaws in either process in order to achieve 99% data recoverability. It may be useful to develop a ‘Cause & Effect Diagram’, such as the one shown below, to find all of the problems.

Improve

Now comes the Improve phase. Utilize the whiteboard and your Post Its again to review a new process flow. Don’t let the existing tools limit where you mind may go. Think out of the box. If part of the problem is to recover data 99% of the time ‘company wide’ and that includes remote offices, there may be a reason to use other tools at these offices in order to meet the objectives. Build out a ‘mistake proof’ process. Don’t worry, you can analyze the costs afterwards, but identify the ‘best case scenario’. Be sure to document the new process.  The next step is to implement some of your changes and see how your new process is working out. It will be very important to utilize the same tools and  measure the new process against the old . You will want to utilize the same charts as before with the new data to show improvement. Based on your results you still may want to take a look at refining the process a bit more now that it is in action and new, unexpected issues pop up as a result of the new process.

Control

Finally, you will want to make sure you control the new process. It will be important to use the same tools to continually monitor and manage the process and to make sure you stay within the new specifications of 99% recovery. If there is ever a situation where you fall outside of the range, you will need to review the process again, identify where the process broke down, fix it and go through the whole DMAIC process again. There is no sense going through all of the prior work to not manage it afterwards to make sure the process stays in compliance.

Keep in mind, requirements may change and other outside factors such as data growth can, and will, have an impact on this process and may force you to re-look at the process or the tools used to manage the process. By continuing to measure and control the process, you will see when you start to fall outside of the critical success criteria and need to make adjustments, but it will also allow you to operate at a much higher recovery level than you have in the past.  Trust me, follow the lean Six Sigma process, it will help you take backup, beyond and put you on the “Road to Recovery”.

Posted by Steve Kenniston

Tweet This Post