Case studies in network centralization and decentralization

[Network] centralization and decentralization can be major overhauls. If you are asking people to accept the pain of converting to a new system, you should be proposing a system that is not only cheaper but also better for them.

More on network design Click here to download a PDF version of this chapter. For more information on this topic, see our FAQ guide on designing networks for speed and reliability.

There is an old adage that often appears on buttons and bumper stickers: "Cheap, fast, good. Pick two." This pointed statement reveals a time-tested truism. In general, you must sacrifice one of those three items to achieve the other two. In fact, if someone tries to claim that they provide all three simultaneously, look under the tablecloth of their slick demo and check for hidden wires. This section describes some examples that achieved or promised to achieve all three. Some, like the purchasing example, were a great overall success. Others had mixed results.

Consolidate Purchasing

In this example, centralization resulted in better products more quickly delivered for less money. An SA group was able to position itself to approve all computer-related purchasing for its division. In fact, the group was able to have the purchasing agent who handled such purchases moved into its group so they could work closely on contracts, maintenance agreements, and so on. As a result, the group was able to monitor what was being purchased. Particular purchases, such as servers, would alert the SAs to contact customers to find out what special requirements the server would create: Did it need machine room space, special networking, or configuration? This solved a problem whereby customers would blindside the SAs with requests for major projects. Now the SAs could contact them and schedule these large projects. As a side benefit, the group was able to do a better job of asset management. Because all purchasing was done through one system, there was one place where serial numbers of new equipment were captured. Previous attempts at tracking assets had failed because it depended on such data to be collected by individuals who had other priorities.

Centralized purchasing's biggest benefit was the fact that the SAs now had knowledge of what was being purchased. They noticed certain products being purchased often and arranged volume purchasing deals for them. Certain software packages were preordered in bulk. Imagine the customers' surprise when they tried to purchase a particular package and instead received a note saying that their department would be billed for one-fiftieth of a 50-license package purchased earlier that year and were given a password they could use to download the software package and manuals. That certainly beat waiting for it to be delivered!

The most pervasive savings came from centralizing the PC purchasing process. Previously, customers had ordered their own PCs and spent days looking through catalogs, selecting each individual component to their particular needs. The result was that the PC repair center had to handle many types of motherboards, cards, and software drivers. Although a customer might take pride in saving the $10 by selecting a nonstandard video card, he or she would not appreciate the cost of a technician at the PC repair department spending half a day to get it working. With the repair group unable to stock such a wide variety of spare parts, the customers were extremely unhappy with having to wait weeks for replacement parts to arrive.

The average time for a PC to be delivered under the old system had been 6 weeks. It would take a week to determine what was to be ordered and push it through the purchasing process. The vendor would spend a couple of weeks building the PC to the specifications and delivering it. Finally, another week would pass before the SAs had time to load the OS, with possibly an additional week if there were difficulties. A company cannot be fast paced if every PC requires more than a month to be delivered. To make matters worse, new employees had to wait weeks before they had a PC. This was a morale killer and reflected badly on the company. The temporary solution was that management would beg the SAs to cobble together a PC out of spare parts to be used until the person's real PC was delivered. Thus, twice as much work was being done because two complete PC deliveries were required.

The centralized purchasing group was able to solve these problems. The group found that by standardizing the PC configuration, a volume discount could be used to reduce cost. In fact, the group was able to negotiate a good price even though it had negotiated four configurations: server, desktop, ultralight laptop, and ultrapowerful laptop. Fearing that people would still opt for a custom configuration, the group used some of the savings to ensure that the standard configuration would be more powerful, with better audio and video than any previously purchased custom PC. Even if the group matched the old price, the savings to the PC repair department would be considerable. The ability to stock spare parts would be a reduction in lost productivity by customers waiting for repairs.

The purchasing group realized that it wouldn't be able to push a standard on the customers, who could simply opt for a fully custom solution if the standard configuration wasn't to their liking. Therefore, purchasing made sure to pull people to their standard by making it amazingly good. With the volume discounts, the price was so low and quality so high that most of the remaining customization options would result in a less powerful machine for more money. How could anyone not choose the standard? Using pull instead of push is using the carrot, not the stick, get a mule to move forward.

One more benefit was achieved. Because the flow of new machines being purchased was relatively constant, the purchasing group was able to preorder batches of machines that would be preloaded with their OS by the SAs. New employees would have a top-notch PC installed on their desk the day before they arrived. They would be productive starting on the very first day.

Ordering time for PCs was reduced from 6 weeks to 6 minutes. When faced with the choice between ordering the exact PC they wanted and waiting 6 weeks or waiting 6 minutes and getting a PC that was often more powerful than they required, for less money, it was difficult to reject the offer. Any company that is rapidly growing, purchasing a lot of computerrelated items, or deploying PCs should consider these techniques.

Outsourcing

Outsourcing is often a form of centralization. Outsourcing is a process by which an external company is paid to provide certain technical functions for a company. Some commonly outsourced tasks are running the corporate PC helpdesk, remote access, WAN and LAN services, and computer deployment operations. Some specific tasks, such as building the infrastructure to support a particular application—web site, e-commerce site, enterprise resource planning (ERP) system—are outsourced, though probably vendors refer to that process "professional services" instead.

The process of outsourcing usually involves centralization to reduce redundant services and to standardize processes. Outsourcing can save money by eliminating the political battles that were preventing such efficiencies. When executives are unable to overcome politics through good management, outsourcing can be a beneficial distraction.

Advocates emphasize that outsourcing lets a company focus on its core competency rather than on all the technological infrastructure required to support that core. Some companies become bogged down in supporting their infrastructure, to the detriment of their business goals. In that situation, outsourcing can be an appealing solution.

The key in outsourcing is to know what you want and to make sure that it is specified in the contract. The outsourcing company isn't required to do anything that isn't in the contract. Although the salespeople may paint an exciting picture, once the contract is signed, you should expect only what is specified in ink. This can be a particular problem when the outsourced services had been provided previously in-house.

We've seen three related problems with signing an outsourcing contract. Together, they create an interesting paradox. Outsourcing to gain new technical competence means that the people you are negotiating with have more technical competence than you do. This gives the outsourcing firm the power seat in the negotiations. Second, to accurately state your requirements in the contract, you must have a good understanding of your technical needs; however, if your executive management had a good handle on what was needed and was skilled at communicating this, you wouldn't need outsourcing. Finally, companies sometimes don't decide to outsource until their computing infrastructure has deteriorated to the point that outsourcing is being done as an emergency measure and are thus too rushed or desperate to keep the upper hand in negotiations. These companies don't know what they want or how to ask for it and are in too much of a panicked rush to do adequate research. As you can imagine, this spells trouble. It is worth noting that none of these situations that specifically allow for technology knowledge refer to the buying company.

You should research the outsourcing process, discuss the process with peers at other companies, and talk with customer references. Make sure that the contract specifies the entire life cycle of services—design, installation, maintenance and support, decommissioning, data integrity, and disaster recovery—SLAs penalties for not meeting performance metrics, and a process for adding and removing services from the contract. Negotiating an outsourcing contract is extremely difficult, requiring skills far more sophisticated than our introduction to negotiating.

Some outsourcing contracts are priced below cost in order for the vendor to be considered a preferred bidder on project work; it's on this project work that outsourcing deals make money for the supplier. Contracts usually specify what is "in scope" of the contract and casually mention a standard rate for all other "out-of-scope" work. The standard rate is often very high, and the outsource organization hopes to identify as much out-of-scope work as possible. Clients don't usually think to send such work out to bid to receive a better rate.

There are outsourcing consultants who will lead you through the negotiating process. Be sure to retain one who has no financial ties to the outsourcing firms that you are considering.

Don't Hide Negotiations

When you outsource anything, your job becomes quality assurance. Some people think that after outsourcing, the contract will simply take care of itself. In reality, you must now learn now to monitor SLAs to make sure that you get all the value out of the contract. It is common for items such as documentation or service/network architecture diagrams to be specified on the contract, but not delivered until explicitly requested.

Executives at one company were very proud of their decision to outsource when, after a year of service, the metrics indicated that calls to the helpdesk were completed, on average, in 5 minutes. This sounded good, but why were employees still complaining about the service they received? Someone thought to ask how this statistic could be true when so many calls included sending a technician to the person's desk. A moderate percentage of calls like that would destroy such an excellent average. It turned out that the desk-side support technicians had their own queue of requests, which had their own time-to-completion metrics. A call to the helpdesk was considered closed when a ticket was passed on to the desk-side technician's queue, thus artificially improving the helpdesk's metrics. Always ask for a detailed explanation of any metrics you receive from a vendor, so that you can clearly relate them to SLAs.

While you are trying to get the most out of your contract, the outsourcing company is trying to do the same. If the contract is for "up to $5 million over 5 years," you can be assured that the account executive is not going to let you spend only $4.5 million. Most outsourcing companies hold weekly meetings to determine whether they are on schedule for using the entire value of the contract as quickly as possible; they penalize their sales team for coming in "under contract." Does the contract charge $1,000 per server per month? "How can we convince them that a new service they've requested needs a dedicated host rather than loading it onto an existing machine?" will be asked at every turn. Here's the best part: If they can get you to spend all $5 million of a 5-year contract in only 4.5 years, those last 6 months usually won't be at the discounted rate you negotiated. How can anyone predict what their IT needs will be that far out? This is the most dangerous aspect of long-term outsourcing contracts.

Make sure that your contract specifies an exit strategy. When starting a long-term contract, the outsourcing company usually retains the right to hire your current computing staff. However, the contract never says that you get them back if you decide that outsourcing isn't for you. Many contracts fail to guarantee that your former staff will remain on-site for the duration of the contract. The company may decide to use their skills at another site! Even switching to a different outsourcing company is difficult, because the old company certainly isn't going to hand over its employees to the competition. Make sure that the contract specifies what will happen in these situations so that you do not get trapped. Switching back to in-house service is extremely difficult. Eliminate any noncompete clauses that would prevent you from hiring back people.

Our coverage of outsourcing is admittedly centric to our experiences as SAs. Many books give other points of view. Some are general books about outsourcing (Gay and Essinger 2000, Rothery and Robertson 1995); by contrast, Williams (1998) gives a CIO's view of the process. Mylott (1995) discusses the outsourcing process with a focus on managing the transfer of MIS duties. Group Staff Outsource (1996) has a general overview of outsourcing. Kuong (2000) discusses the specific issue of provisioning outsourced web application service provider services. Jennings and Passaro (1999) is an interesting read if you want to go into the outsourcing business yourself. Finally, Chapman and Andrade (1997) discuss how to get out of an outsourcing contract and offer an excellent sample of outsourcing horror stories.

The first edition of this book was written during the outsourcing craze of the late 1990s. We had numerous warnings about the negative prospects of outsourcing, many of which came true. Now the craze is over, but off-shoring is the new craze. Everything old is new again.

Network centralization and decentralization
Introduction
The basics
Candidates for centralization
Candidates for decentralization
The icing
Conclusion/exercises

Reproduced from the Addison-Wesley Professional book The Practice of System and Network Administration, 2nd Edition, by Thomas A. Limoncelli, Christina J. Hogan and Strata R. Chalup. ISBN 978-0321492661. Copyright 2007, Addison-Wesley Professional. Reproduced by permission of Pearson Education Inc., 800 East 96th St., Indianapolis, IN 46240. Written permission from Pearson Education Inc. is required for all other uses.