What you will learn from this tip: What you can do to overcome poor technical support from your storage vendors.
While the complexity of storage products has grown, vendor service levels have been steadily falling over the last few years, according to interviews of Fortune 1000 storage professionals conducted by TheInfoPro Inc., an independent research network. "We have seen a drop in some of our vendors' willingness to understand our business," says Wendy Betts, distributed systems storage manager at Hewitt Associates LLC, a human resources outsourcing and services company in Lincolnshire, Ill.
TheInfoPro's latest survey attributed the drop in vendor service levels to four factors:
- Increased product complexity that requires greater knowledge and expertise.
- Vendor alliances and mergers that initially result in less integrated solutions and less knowledgeable support staffs.
- The convergence of storage, security and networking increases the number of professionals involved in delivering and supporting a product or service.
- Competitive pressure among vendors creates higher technical support force turnover.
Growing product complexity
It's not uncommon today for a storage architect to manage 50 TB of storage. This may include hundreds to thousands of servers, dozens of different path management products, host bus adapter (HBA) firmware levels and driver releases, fabric zones, storage switch firmware levels, storage frame firmware/software versions and thousands of LUNs. Add in a virtualization layer and you can see why storage teams are overwhelmed. In addition, storage professionals are taking on more strategic roles within the business and getting involved with enterprise document management, regulatory compliance, email management and archiving.
((Content component not found.)) "My biggest storage-related pain point is the complexity of the solutions," says one storage administrator who works in the U.S. government. "We have to educate people, and there are so many moving parts that it's hard to find people to support them."
To alleviate some of the problems complexity brings, storage departments at larger companies are dedicating vendor technical support teams to specific tasks such as the installation of new systems, product maintenance and firmware updates. However, many companies that have relied on vendors for these vital support services have horror stories: how a vendor's system engineer or cabling contractor accidentally powered off a primary storage array while upgrading the storage service processor; or a storage support engineer who pulled out the wrong spindle on the wrong storage array while replacing a failed disk drive.
"We will not tolerate a storage support organization that is less knowledgeable of storage technologies than our staff," says Hewitt Associates' Betts.
More points of failure
The conventional wisdom for getting the fastest response to an outage is to have "one throat to choke" -- a primary vendor to call when things go south. Even though many companies try to limit the number of vendors in their environment, there will always be the need to support legacy apps, the desire to minimize vendor lock-in and the advantage of occasionally installing best-of-breed products. Any of these factors can result in products from multiple vendors.
To keep up with the pace of innovation, primary storage vendors are acquiring startups and forming partnerships with other companies. Recent notable examples include EMC Corp.'s acquisition of Rainfinity, Hewlett-Packard's purchase of AppIQ, IBM buying DataPower, NetApp partnering with Kazeon Systems Inc. and Sun Microsystems Inc. snapping up StorageTek. As a result, even if you purchased all of your storage from one primary vendor, there may be as many as 10 organizations within that vendor that need to diagnose your problem before a solution or workaround is determined.
"Client support tends to gradually decline once one company is acquired by another," says a storage manager working for a large manufacturing company.
The high turnover rate for support personnel at primary storage vendors also contributes to declining support levels. Good vendor support personnel won't hesitate to jump to another company to secure a better salary. Vendors under pressure to satisfy Wall Street will sometimes lay off staff to make their quarterly numbers. Acquisitions usually result in staff reductions to eliminate "redundant" positions. The movement of support specialists is a concern because even a support person with an extensive technical background will need time to learn the ins and outs of a new product.
"The tech support is getting really bad; it's frustrating that we know more of the answers up front than their people do," says a storage manager working for an educational services company.
A better working relationship
One way to get better technical support is to treat your primary vendor's technical support representative as a trusted member of your storage team. Have the technical sales rep participates in the development and execution of your business strategy. This includes communicating common storage strategies across the organization, educating business peers on storage dependencies and setting up proper performance expectations, as well as ensuring that the storage infrastructure is protected against failure.
Nonetheless, it will take at least three months for a vendor's technical support team to get up to speed with a company's staff, organization, priorities and business climate. Furthermore, vendor technicians, once they get up to speed at a company, are often transferred to other "troubled" accounts. This practice continues to erode the trust between storage professionals and their technical support teams, and further contributes to low support ratings from users.
"The support staff is only reading a checklist and doesn't have experience," says a storage manager at a multinational pharmaceutical firm. "Plus, the response time is much too long."
In an attempt to buttress the effectiveness of their support organizations, primary storage providers sometimes limit or delay the number of products they certify/support when used with their products. For example, EMC delayed support for McData Corp.'s Eclipse; for its SAN Volume Controller, IBM delayed support for EMC's Clariion; and HP delayed support for NetApp's V-Series products. However, this strategy tends to anger users.
Out-of-control storage consumption
According to TheInfoPro, SAN capacity doubled in the last year and NAS storage storage quadrupled. Business users are complaining about the time it takes to purchase and provision storage. Furthermore, the state of capacity planning and forecasting is inadequate. Storage professionals can't project data warehouse, application and file growth accurately, and find themselves reacting to storage demands rather than proactively planning storage growth. And vendors, of course, are all too willing to sell more and larger boxes rather than help users better manage what they have.
"We won't even buy a midrange unit from them again because the service put us down for weeks at a time," says a storage manager at a technology company.
In light of the rapid growth of networked storage, it's easy to forget that five years ago a storage team or an enterprise-wide SAN were the exceptions. Storage organizations are relatively new entities in many companies. But a variety of new technological innovations could threaten the relatively new storage teams.
For example, to support rapidly growing NAS systems, many storage organizations have to work closer with their networking administrators. Wide-area file services are becoming a popular way to support remote offices, but they require the IP networking and storage admin teams to coordinate their efforts. Security officers are starting to set new guidelines for storage. This means that a larger number of professionals are involved in data storage.
While some storage vendors foresaw these convergences, they still didn't invest sufficiently in their sales and technical support organizations to ensure that their people were knowledgeable in all of the different technologies that comprise an effective storage strategy. Storage professionals are frustrated with having to educate multiple technical support teams from the same company.
"Our biggest reason for getting rid of them is the structure of their support organization," says a storage director at a large media company. "It's horrible."
The good news is that vendors are aware that their technical support organizations are getting low marks. They know they must invest in technical support and upgrade their services to remain competitive. Today, vendors can't get away with selling a complex product and then walking away, forcing the user to turn to a high-priced consultant for help. At TheInfoPro, we're seeing positive feedback on customer ratings for vendor "management vision" and "technical innovation."
However, one thing is certain: If Cisco Systems Inc., EMC, HP, IBM, NetApp, Sun and Symantec are unable to convert high ratings in these areas into superior product support, a more support-minded startup will be ready to replace them.
This piece originally appeared in the May 2006 issue of "Storage" magazine
Do you know…
How to safely relocate your SAN
