Nine Real-Life IT Horror Stories
InfoWorld Staff, InfoWorld
Nothing can screw up a well-managed network faster than the people for whom you built it. Whether it's user error, optimistic expectations, or simply that bastard Murphy, IT's job is rarely predictable.
Lucky for you, there are lessons to be learned from others' misfortunes. So rather than wait to make your own forehead-shaped dent in the office wall, familiarize yourself with the screwups detailed below. It will make you that much more prepared to safeguard your IT environment from the ever-evolving boneheaded tendencies of those you serve.
Stupid user trick No. 1: Home is where the malware is It happens at least once a year, and this year it happened twice, writes one IT admin: "And though we make the point with memos and lectures, there always seems to be someone who gives their work PC to the kids at night."
[ Users are by no means alone when it comes to hard-headedness in the IT world. See "Stupid user tricks 3: IT admin follies " and "True IT confessions " for real-world tales of folks who should know better fouling up. ]
The situation is familiar: To save on expenses, folks buy fewer home PCs, but their kids want to use them more than ever. Enter the corporate laptop into the home Web surfing environment -- a recipe for disaster for IT.
And it's not just kids playing games and doing homework. It's spouses using social networking -- and that uncle nobody talks about surfing porn on your corporate machines.
"Our security tends to be better than the average home box, but that won't protect you forever if you actually run out and look for attack sites," our admin warns. Sooner or later, one of your user's laptops will get compromised, leaving your network exposed to infection the next time he or she logs on at the office.
"We've gotten better at catching these compromised machines early, so instead of it being the big problem it used to be, last year it mainly just confirmed our investment in end-client security," the admin says.
The worst offender? A procurement manager who was found to have a keyboard logger installed on his company-issued laptop. "And this was a guy who spent several $100K a year online for the company," the admin informs us.
Solution: End-point security goes a long way toward preventing infected machines from gaining access to the corporate net, but they'll never be 100 percent effective. Web browsers are the gateway to hell when it comes to attack entry points. Let your users surf helter skelter and your attack potential goes way up. The only preventative measure: a strong fair-use policy and a management staff that'll enforce it.
Moral: Users will continue to break your official-use policy as long as money is tight and they believe the consequences are minor. Include disciplinary action in your policy, and make sure users know you're tracking Web site visits and system access. Otherwise, you are simply setting yourself up for disaster. Another solution: Sponsor employee discounts on netbooks. That way, your users will be less tempted to transform company property into their home PCs.
Stupid user trick No. 2: Message to self: E-mail isn't for everything Sometimes all it takes is a well-meaning IT management decision to set stupid users in motion, writes P. Lindo, an IT admin at a New York-based organization with more than 1,000 e-mail inboxes, which the firm first maxed out at 100MB per mailbox, then at 500MB.
"In 2007, we hired a new IT manager who got placed in charge of e-mail management," says Lindo. "He saw the load of user requests for larger mailbox space and decided this was where he was going to make a big difference."
And so he set about purchasing enough hardware to increase individual mailbox sizes to 1GB -- barely.
"He also used all the user requests to get backing to upgrade everyone to Office 2007 -- the one with the new Outlook mailbox search," Lindo says.
Throw in a new policy for teaching users proper inbox maintenance, and watch inbox utilization hover at a manageable 75 percent -- until you put policy into practice.
"Turns out users don't read documents titled 'Proper Inbox Space Management,'" Lindo says. What they see instead is the fine print that tells them they now have 1GB of mailbox space. And then they start using Outlook's handy new search feature to turn their e-mail clients into personal information managers.
"Nobody deletes attachments anymore. Instead they leave them in their inboxes so that they can run quick searches against them, where all they need to remember is a rough description of the attachment and the name of the person who might have sent it to them," Lindo explains.
Worse, they send attachments to themselves just so the doc will be in the inbox somewhere.
"Our mail servers got maxed out inside of three months."
The small saving grace?
"We actually saw a 35 percent decrease in the amount these users used their network home directories," Lindo reports. "Outlook became the main network gateway for personal storage. So we were able to repurpose some storage from the file server machines on the e-mail infrastructure, but we still had to make several large and unscheduled server purchases to keep up with new demand."
Solution: A big inbox may sound like a good idea, but proper capacity planning is an even better one. Moreover, planning for 75 percent utilization is a recipe for trouble. Instead, target 50 percent or less, or run a pilot project before committing. A low-cost SAN can help here as well; adding capacity to one of those is significantly easier than installing new servers.
Moral: If it seems like everyone's working harder these days it's because they are. Users will utilize any tool you put in front of them to get the job done. And if they're more familiar with their e-mail client than other network resources, they'll use it as a substitute -- as long as you let them. Expand your definition of "desktop management" to include reaching out to users to train them on the tools your company is spending money on.
Stupid user trick No. 3: Outsourcing Web development to the corner office Here's a thought: Don't let the CEO design your company's customer-facing Web site just so he can save a few bucks, advises an IT consultant.
"We tried to sell a medium-sized company client on both a network install and a Web site design project," the consultant says. "We got the install contract, but the CEO figured he could design his site himself.
"When his general manager -- who was also his wife -- called us back in, she pulled the site up and it was hard not to wince. He'd used an open source editor with what looked like every freely downloadable template, fonts, and flashy widgets he could find. It looked like a teenage MySpace page."
[ More manager mishaps when meddling in IT can be found in "More stupider user tricks: IT horror stories redux " ]
Sure, the company's product information was now available on the Web, but the lack of customer-facing tools and analysis features did not bode well for the company's Web future.
"Even the Webmaster e-mail link didn't work," the consultant says. "Needless to say, the site was not attractive to customers, so Web revenue was low, and all those new and expanding Web marketing possibilities were crippled. The same CEO who built the site started spouting about how the rumors of e-commerce revenue were false."
Solution: Today, Web site design is cheap. From local outfits to eBay or Craigslist, the cost of a decently designed Web page has dropped from thousands of dollars per page to hundreds -- or less. Stop being penny-wise and pound-foolish.
Moral: Company Web sites can't be an afterthought investment, especially for small businesses. Not just an important face to your customer, your Web site is possibly the best way to analyze exactly who your customers are and how to sell to them. Treat it professionally, and you can leverage it for additional opportunities, including market research, customer analysis, and more.
Stupid user trick No. 4: Keep your enemies close, but your Linux talent closer Going open source can save big bucks -- unless you leave your entire open source infrastructure in the hands of a single college intern, warns an admin at a small IT services firm.
"I finally find a small-business client who made the jump to Linux -- well, Linux and HP-UX due to a silo app they had to run for two big clients," the admin says. "Our new client had used his college intern to setup the basic network, but the kid had left for summer vacation a day earlier and suddenly the network was down. We were the first outfit in the phone book that didn't shy away from the phrase 'Debian on the desktop.'"
When the admin and his cohorts arrived, all the client's server lights were green, but nobody was connecting to anything and no one could log in to the system.
"We had to restore the servers from the ground up, which took about an hour. Everything was humming after that, so we took the time to sit down with the CEO and discuss plans for the network," the admin says. Stoked to locate someone unafraid to talk about open source software, the admin and his team got a little carried away shooting the bull with the CEO and stayed for more than an hour.
"As we were on the way out, the servers dumped again," the admin says. "Same story as before. Not wanting to lose our new penguin client, we rolled up our sleeves, restored the servers, and started digging for root cause."
What they found was a cron job set up off root.
"The cron 'cd'ed to a backup directory that tried to remove the files from a lengthy list of source directories, including several that didn't exist," the admin says. "Seems the kid had been changing these on the fly for some reason -- and he apparently liked doing sys admin as root. Academics."
Solution: Protect root access. Test your cron jobs. And maintain those server backup images.
Moral: Linux has definite benefits, but there's no denying that managing it requires a certain skill set. It's not something to trust entirely to an intern.
Stupid user trick No. 5: Facebook Face it, even the most stringent social networking policies can't diffuse the ticking time bomb that is Facebook. Throw in a little Jäger, some IT naivete, and you're set for devastating corporate embarrassment.
"About a year ago, I get a call from a junior VP who's yelling at me that he's desperate and needs me to do a 'recall on Facebook,'" says one admin who wishes to remain anonymous. "I try and get a word in edgewise, but he's ranting about what crap Web technology is and why computer people can't just leave well enough alone and how everything was fine when we just used the telephone. Then he ends with, 'Is it done yet?'"
"'Is what done?'"
"'The Facebook recall, for @#$%'s sake.'"
Which gave our admin the obvious pleasure of asking, "What the @#$% is a 'Facebook recall'?"
As it turns out, the junior VP had updated his Facebook page from his phone while having a few drinks with some senior VPs and potential new clients.
"He stated that he'd dated one of the clients' wives and made some nasty comment about what she looked like naked," the admin says. "All his college buddies were on Facebook in their college group, and he knew her when they were both at school. Turns out that's where she met her husband, too, and he was on the college Facebook group as well, which the genius junior VP figured out when he got back to the table and started a conversation about Facebook with the potential clients."
As for the "Facebook recall," it appears that the junior VP thought updating his Facebook page was like sending an e-mail in Outlook.
"I told him how to lock down his page, but apparently that was a little late," the admin says. "We didn't get that account."
Solution: There really isn't one, other than trying to make sure your users have some idea of where the power of IT ends and the big, bad world begins.
Moral: The beauty of social networking is that it connects you with millions of other people. The danger of social networking is that it connects you with millions of other people.
Stupid user trick No. 6: Offshoring while under the influence of MBA Whoever said offshoring was idiot-proof? After all, it often involves upper management -- potentially the worst IT offenders of all.
"We got a new CIO just before the bubble burst back in 2000," says D. Aubrey, who at that time worked at a Web services firm with a solid market position that it now had to defend against upstarts. "She was one of those MIS MBAs -- emphasis on MBA. All you press types started writing stories about the benefits of outsourcing around then, so she jumped on the trend, canned our Web dev team, and outsourced the whole shebang to an outfit in Mumbai that worked for $25 an hour."
The plan looked good on paper -- until you looked at the paper.
"We got a hold of the plan spreadsheet she presented to the CEO, and all she'd done was compare the cost of software tools and staff from in-house to out-of-house, so obviously the savings looked huge," Aubrey says.
"Then came the phone bill, which I think had quadrupled for that project," he adds. "And the security audit bill, since the data our Web dev guys were working with was quite a king-size waffle of personal customer data. And the hardware/services bill for moving our data out of the outsource outfit's internal datacenter -- which as far as we could tell was four servers in a closet somewhere -- and into a professional data hosting facility in Europe."
If that weren't enough, the final product -- a redesigned site -- "looked so average it might as well have been beige."
As it turned out, the new CIO had outsourced not just development, but project management and QA as well.
"There was literally nobody on our side proofing the work. They just kept showing her screenshots and she kept approving them until the day the redesign flipped," recalls Aubrey.
The volume of customer complaints about the site's new look and lack of functionality was put to a stop by the site itself, which crashed twice on the first day.
"The CEO ordered her to pull the plug and go back to the previous design," Aubrey says. "When they added it all up, she'd spent about 75 percent of the original project budget and had nothing to show for it."
"Normally, we'd have just snickered as they walked her out the door, but this was a down economy and this crap just cost us about five months of competitive advantage," he says.
The company never recovered. And though our intrepid offshorer was the first out the door, the rest of the crew followed by year's end.
"I'm not saying outsourcing doesn't work," Aubrey says. "But it takes a hell of a lot more planning than just comparing staff costs."
Solution: Go back and reread No. 3, and then realize that this submitter didn't go far enough. Web site development doesn't have to be isolated to be cheap.
Moral: If the Web site is a key revenue stream, do not entrust site development to a single exec.
Stupid user trick No. 7: Duct tape doesn't fix everything "This one still makes us laugh over beers," says H. Foreman, an admin at a Midwest-based organization.
"We were growing pretty well in 2004 and 2005, so we opened an office across the street," Foreman says. To connect the two offices, they decided to buy two microwave bridges.
"The setup is easy enough that we were able to do the job ourselves, though we had professional carpenters install the bridges to the walls of each building, just under the roof, pointing through double-paned office glass, so we would have no weather worries."
Success carried over into 2006, when the company decided to extend its leases.
"As part of the deal, they get permission to put up a fancy sign near the top of both buildings -- indoors but facing outward through the windows," Foreman says. "The day the sign goes up, our network goes down for about 15 minutes. We're still doing the basic set of troubleshooting diagnostics when it suddenly comes back up. Our guy shrugs, verifies everything again, and lets it go."
The next morning wasn't as forgiving. The network went down and stayed down.
"The basic software diagnostics aren't working, so we go to physical link monitoring," Foreman says. "Pretty quick, we see that one of the bridges isn't responding anymore. Upstairs we go."
Apparently the bridges had been in the way of the signs.
"The outfit that put up the signs just detached the bridges and moved them -- outside," Foreman says. "There was a balcony on the upper floor and they just moved both bridges out there and then duct-taped both of them to the railings.
"What kills us is that the network somehow recovered the first time," he says. "The duct tape across the street held, but the one on our side slipped off during the night and the bridge fell eight stories, bounced off the dumpster, and landed behind it. The sign installers apparently left a note explaining what they'd done with the receptionist across the street and she hadn't passed it on."
Naturally, Foreman and company had fun pointing the finger at the sign company in front of the CEO, who then ran out to chew out the install rep.
"But as soon as he left the room, the CIO, who is a really good ex-tech, pointed out that if we knew someone was going to be doing construction around a critical piece of network infrastructure, why the hell hadn't we gone up there to check it? Especially after the network went down during the construction process," Foreman says. "He had a point."
Solution: The basic network monitoring software this company was evidently using is as good a technology solution as you need in this instance. Without such software, however, this would have been a much nastier adventure.
Moral: What the CIO said. Construction around network infrastructure requires personal attention from your IT staff. Remote monitoring is no substitute for "eyes-on" during critical times.
Stupid user trick No. 8: Executive privilege This one hits close to home, as some tech magazine editors epitomize the worst kind of user an IT admin can encounter: those who have read so much about IT that they simply assume hands-on expertise. What results are "special requests" of IT not unlike those we find dealing with higher-up execs.
Let me set the stage: I was working as a technical editor for an IT magazine some years back and happened to be in the executive editor's office three days in a row when this little drama went down. I can't remember whether Windows 95 or Windows 98 had just come out, but it was one of those two. The executive editor had requested the new OS on his honking Toshiba notebook -- a $6,000 box, the price of which I still can't fathom. IT had obliged and installed it. He'd happily used it for a day, taken the box home, and when he returned the next morning it was dead. Windows wouldn't boot. The conversation went something like:
IT tech: "So what did you do?"
Executive editor: "Nothing, it just didn't reboot."
IT tech: "It couldn't have just stopped for no reason. Did you install something?"
Executive editor: "No. Really. It just wouldn't reboot."
IT tech: [sigh] "OK. Fine. I'll fix it."
The next day, the tech returned the notebook, Win 95/98 fully reinstalled and working fine. The day goes well; no crashes. The next morning, the executive editor returns yet again with a $6,000 paperweight. I'm in his office for this part and had to work hard not to shoot coffee out my nose.
IT tech: "Come on, you had to have done something. Everything was working yesterday!"
Executive editor: "No, really. I didn't install a thing. I was just working and organizing."
IT tech [suspicious]: "What do you mean 'organizing'?"
Executive editor: "You know, just arranging folders so that I can find things more easily."
IT tech [still suspicious]: And which folders were you 'organizing'?"
Executive editor [annoyed]: "What does that matter?"
IT tech [equally annoyed]: "Trust me. Which ones?"
Executive editor: "My personal folder, the issue folders, the system folder --"
IT tech [squeezing his eyes shut]: "What did you do in the system folder?"
Executive editor [slowly dawning]: "Uh, well it was so messy. They had one folder for 16-bit DLLs and another for 32-bit DLLs, so I thought it'd be more efficient if they were all, you know, in a single folder."
I'm not sure who the tech wanted to kill more: the executive editor for what he did or me for sitting there, shoulders shaking, beet red, with my mouth clenched shut and tears coming out of my eyes.
Solution: Don't let your users become case studies for denying administrative access to local machines. Deny them administrative access to begin with. With senior execs, however, it still takes a social engineering degree to keep that rule enforced. That's a line of patter every IT guy needs to develop.
Moral: Even Microsoft computers don't suddenly quit for no reason. There's always a guilty user somewhere on the chain of causality. Find him early and you can avoid a large load of trouble down the line.
Stupid user trick No. 9: User populations are like bacterial ecosystems from distant planets This particular stupid user trick hails from my days as an IT consultant, when our clients' CIO types, who had read about Shadow Copy, immediately wanted to engage on it. After all, in many cases they'd paid for it already, so they wanted it up and running right away.
Rolling out Shadow Copy was easy -- once we had Windows 2000 on every desktop and a working Active Directory domain controller. Then I used my vaunted writing skills to pen a short and sweet "Shadow Copy Advisory" memo and e-mailed it to every user. We followed that up with personal visits to all the managers in the company, explaining how the feature worked and what they needed to tell their employees about it.
The upshot was that My Docs was now being shadow-copied for every user, so all those folders they had on their desktops should be moved to My Docs to make sure everything got backed up to the network automatically.
In retrospect, I might as well have been asking them to bite off their own fingers for my amusement.
Everyone nodded excitedly, but nobody had any intention of using it.
To be fair, this was our fault as much as theirs. Assuming that users will put data exactly where they say they will is a newb mistake. But like true consulting newbs, we set up a backup policy to perform daily backups of "data" folders -- the shadow-copied stuff and the file shares users said they were going to be using -- and weekly snaps of the full server. Desktop backups relied entirely on users making My Docs their sole data dump.
Naturally, when a nasty virus hit and took out a large percentage of the desktops and simultaneously dumped two out of three servers, we found only 8 percent of users had been taking advantage of Shadow Copy. The rest were simply screwed. Worse, we found out that they had decided to build new "informal" network shares right off the server's hard disk (exactly where we hadn't expected them to), so those files were lost, too.
Solution: First, realize that you will never get away from users using their desktops as data storage. Ever. That's why it's called the "desktop." Whatever desktop backup strategy you employ, it needs to cover the desktop -- My Docs and any personal folders they've built themselves -- automatically. On the server side, you need a daily snap, so just thank God for block-level change technology.
Moral: Great ideas are fine, but you have to weigh them against every user's inherent resistance to change. User populations are like bacterial ecosystems from distant planets. You can't predict with very much precision how they'll evolve, so things like backup need to use the word "holistic" rather than "targeted."
Related articles Stupid user tricks 3: IT admin follies IT heroes toil away unsung in miserable conditions -- unsung, that is, until they make a colossally stupid mistake More stupider user tricks: IT horror stories redux Idiot-proof your enterprise with these 10 hard-luck lessons of boneheaded IT miscues Even dirtier IT jobs: The muck stops here More dirty tech deeds, done dirt cheap The 7 dirtiest jobs in IT Somebody's got to do them -- and hopefully that somebody isn't you True IT confessions Supergeeks fess up to some of the dumbest things they've ever done -- and the lessons they learned as a result Stupid QA tricks: Colossal testing oversights The trick to nipping IT miscues is testing, testing, testing, as these hard-luck lessons in boneheaded quality assurance attest Stupid hacker tricks, part two: The folly of youth Tech-savvy delinquents set the Net aflame with boneheaded exploits that earn them the wrong kind of fame Programming IQ test: Round 2 Acing last year's quiz was nothing. Only true hacker heroes will survive Round 2 Linux admin IQ test How much do you really know about the free OS? Apple IQ test How much do you really know about the house that Jobs built? Web IQ test Think you know all there is to know about the World Wide Web? Take our quiz to find out