ad info
   personal technology

 custom news
 Headline News brief
 daily almanac
 CNN networks
 CNN programs
 on-air transcripts
 news quiz

CNN Websites
 video on demand
 video archive
 audio on demand
 news email services
 free email accounts
 desktop headlines

 message boards




Opinion: Raleigh, we have a problem

August 5, 1999
Web posted at: 12:09 p.m. EDT (1609 GMT)

by Joe Barr


   Linuxbierwanderung: On the Linux trail

   Message Board: Linux

   Sign up for the Computer Connection email service

   For more computing stories


(IDG) -- Ramon Fernandez (an assumed name to protect the victim's true identity), and others like him, are important. Though not yet Linux users, they are Linux shoppers. They are much too important to be sold a copy of Linux that includes tech support, only to be ignored when they try to use it.

What makes Ramon and others like him so important is that their counsel will be sought by those in the enterprise when it comes time to decide which distribution will best meet their next operating system requirement.

I know Ramon has tried Linux before, because I've given him distributions to play with. I know he has discussed Linux with his customers, too, because he has talked to me more than once about their concerns regarding Linux support. This time around, I gave him a copy of Red Hat 6.0 -- the expensive spread -- which comes with tech support.

Now, I've personally never had a problem with Red Hat tech support. But I've heard a number of complaints about it from other members of the Austin LUG. One of the founders of the Austin LUG, for example, bought that same 6.0 version of Red Hat earlier this year. He ran into a snag during installation and decided to ask Red Hat about it. But when he called the number for tech support, he was told outright that there was no Red Hat telephone support.

Another member of the same LUG wrote recently:

If you are considering purchasing Red Hat for their support, I would save the money, or advise others to save their money. Go buy some RAM or something. I had RH support once. I posed some dial-up configuration problems to them and got "punted."

So there's been a low-level rumbling of discontent for some time -- and it isn't anything new. That buzz has escalated now and again to open mockery. One wag on a mailing list suggested that the recent Linux hack reported by the New York Times news service was the result of the user having called Red Hat support for assistance in installing ipchains.

But these problems are of the sort that can be attributed to a good tech having a bad day, or simply to a bad tech. They are the kind of problem that will happen from time to time in any large support organization, no matter how good it is.

In the words of the French philosopher: There is no support

The problems that Ramon ran into, on the other hand, are far more serious. They're systemic and reflect the support organization as a whole. And they're bad. Raleigh, are you listening? We have a problem.

  LinuxWorld's home page
  LinuxWorld free e-mail alerts
  LinuxWorld features & columns
  Get Media Grok and The Industry Standard Intelligencer delivered for free
 Reviews & in-depth info at's personal news page
  Year 2000 World
  Questions about computers? Let's editors help you
  Subscribe to's free daily newsletter for IT leaders
  Search in 12 languages
 News Radio
 * Computerworld Minute
 * Fusion audio primers

First, a word about Ramon. He may not represent the typical Linux newcomer, either in his level of experience or in his possible influence on its future. Ramon prefers to gnaw on raw executable code. To him, an advanced assembler just makes things too cushy for the programmer. C language? Please, that's for sissies! Give Ramon a requirement to extend DOS or to debug a device driver, a copy of debug, and a quiet corner, and he is a happy man.

But Linux is new to him, and when he ran into this particular problem he decided it was something he probably needed help with. The home computer that he uses as a distribution testbed is an old CompuAdd 486 EISA/VESA with an on-board Adaptec SCSI controller. Red Hat 5.2 installed on this machine without a hitch, but something is different in 6.0, and it just wouldn't boot after installation.

Diary of a call for help

Day 1: July 7, 1999

Ramon sent the following message to Red Hat support:

I have gone through what appears to be a successful install of Red Hat 6.0 but when I boot the system locks up with a screen full of information. The first line is: <1> Unable to handle kernel paging request at virtual address 000076A9

Almost immediately, he received a reply from Red Hat saying:

Your request for personal assistance has been received. You should expect a response from us within 24 hours.

It's hard to find anything to fault with that, isn't it? Nearly instantaneous response, a unique incident number, and a promise to respond within 24 hours. It sounds great, actually. The problem is, over the next two weeks that's as good as it got.

Day 2: July 8, 1999

As promised, Red Hat responded within 24 hours of Ramon's post for help. He received an e-mail suggesting that he remove any memory holes he had enabled, that he switch the order of the cards in his system, and that he append aic7xxx=extended,no_reset to his linux command line at the LILO prompt.

Ramon easily did all of the above and reinstalled Linux. But still the system refused to boot, so he dutifully reported back to Red Hat.

Day 3: July 9, 1999

On the evening of July 9, Red Hat support informed Ramon that his ticket was being upgraded so the tier 2 technicians could take a look at it. At the time, Ramon felt they were doing the right thing.

True, it was taking a little longer than he had hoped (it would now be Day 4 before he talked to the right level of support), but he felt it might be a more difficult and obscure bug than a front-line support tech would be able to resolve.

The notion that Red Hat's tech support realized this as well and was passing it up the ladder reassured him, albeit only temporarily, that they knew what it was about.

Days 4 to 6: July 10 through 12, 1999

No word from Red Hat.

Day 7: July 13, 1999

It was Day 7 before Ramon heard from Red Hat's tier 2 techs. They requested that he get the latest errata and create new boot images, and they provided him detailed instructions on how to go about doing so.

By this time, however, Ramon's patience and understanding attitude were beginning to wane. Granted, Day 3 (July 9) had been a Friday, and even support people are allowed an occasional weekend off, but what happened to Monday?

What happened, it turned out, was that Ramon took the time to update the incident online with additional information. That this action reset his 24-hour clock and delayed him yet another day did not sit well with Ramon.

Worse, when he did hear from tier 2 it appeared that they hadn't read any of the information he had previously and patiently supplied. The tech spoke about problems with Gateway hardware they had run across in the past. Ramon had never said anything to suggest he had a Gateway box.

Ramon has worked in the past in front-line customer support, servicing mainframe hardware and operating systems. He has had to escalate to tier 2, too. He's no stranger to the game, and he got the feeling that the Red Hat techs were stalling -- giving him busywork to keep him off their case for another 24 hours.

Day 8: July 14, 1999

The next day, now a full week since the problem was originally reported, Ramon wrote his third or fourth note to the Red Hat support manager. Not only was he understandably impatient with the lack of progress, he was feeling ignored. This isn't a good thing.

He wrote to the support manager:

This problem doesn't seem to be getting any closer to resolution. What is the escalation procedure? Is anyone reading these messages to I have yet to receive any reply confirming the receipt of any e-mail to this address.

A deadly sin, that. It's bad enough to get no response from a tech support e-mail address, but when the manager responsible isn't replying either, the whole operation looks suspect.

Meanwhile, tier 2 suggested that he double-check his LILO configuration and rebuild the MBR. He did that, but no joy. It appeared as if tier 2 was firing blindly at the problem.

Day 9: July 15, 1999

Still no progress on the problem. Still no reply from the support manager. Ramon, nothing if not a resourceful man, decided to try a different approach. He wrote to the Red Hat Sales Department:

I know this is sales, but I have sent at least four e-mail messages to over the past week and have yet to receive any response. Not even an acknowledgement or a go-to!! I have an open incident which isn't getting resolved and need some help.

Day 10: July 16, 1999

No word from Red Hat.

Days 11 to 12: July 17 through 18, 1999

It was the weekend again, and Ramon knew there would be nothing new from Red Hat, so he decided to look elsewhere for help. I had bragged to him in the past about the support available from the Linux community itself. So he posted a message in a Linux newsgroup completely describing his configuration and the problem. He even mapped the call trace as best he could.

But all the great things I've been telling Ramon about support from the Linux community came to naught. He never received a reply to his post. Not even one, as he pointed out to me ruefully, to say he was stupid for asking. I weakly suggested that maybe the problem was too obscure for anyone else to have had a similar experience. I was closer to the truth than I knew.

Day 13: July 18, 1999

Ramon decided to try telephone support, thinking that if he could just talk the problem through with someone who knew the Linux kernel, instead of conducting an e-mail exchange with a built-in 24-hour lag time (or longer) between thoughts, and if he were then able to offer additional information between Red Hat responses, they might be able to puzzle it out together.

But, no. Voice support doesn't have visibility to problems reported online. But the support person did listen, and suggested that there were many known problems with the Adaptec driver. Surely that must be the problem. And a new Adaptec driver should be available soon.

Day 13 also brought a long-awaited response to his e-mails to the support manager -- possibly as a result of his e-mail to the Sales Department asking for contact information for the CEO. The manager wrote:

I have looked at the incident you reference below and you have been assigned a level 2 technician that is trying to work through your problem. If you do not receive a reply today to your latest update to the incident (At 7/15/99 0:16 AM) within 24 hours please contact me again and I will seek a quick and complete resolution of your problem. I am also very sorry for the delay that you experienced in receiving a reply from this address.

But later that same day, and nearly two weeks after the problem was first reported, Red Hat support threw in the towel and offered Ramon a full refund. Ramon, however, was just getting started. And by now, even I was trying to keep the game alive a little longer.

The eagle lands

Have I mentioned that Ramon is a bit stubborn when it comes to being stumped by a computer, even if it is running an alien operating system? Well, he is. Ramon believed at first, as did Red Hat, that it could very well be a problem related to the Adaptec driver. After all, he had experienced difficulties with that driver while installing earlier versions of Linux.

But he kept going back to the call trace he had mapped and wondering why there were so many APM related calls when his machine didn't support APM.

He knew that the CompuAdd BIOS sometimes lied about APM support. If he ran OS/2 on that system, it recognized that he didn't have APM. But if he booted from DOS and queried the BIOS, it told him that it did. Ramon figured that if he completely removed the APM support from the BIOS, Linux just might boot.

So he did just that, he built his own BIOS without APM. And Linux did just as he suspected it might, it booted cleanly on the first try with the modified BIOS.

Ramon wrote me saying:

The Eagle has landed! Straight ahead I see GNOME for the first time. Taking APM out of the BIOS did the trick. So maybe the only systems in the world that could experience this problem are the CompuAdd EISA/VESA motherboard systems.

The trouble with Red Hat

There is no doubt that this problem was difficult and obscure, so I don't fault Red Hat for not being able to come up with a fix, early or late. But the facts of the case leave Red Hat support looking inadequate to provide technical support for the enterprise.

Ramon points out that he was getting "personal" support, and that Red Hat does offer a higher level of support for commercial customers. But I'm not sure if that changes anything. And I'm not sure that Red Hat's commercial support would have found the problem, either -- or demonstrated a better response rate.

In spite of the poor showing by Red Hat tech support, Ramon still rates Red Hat as his favorite distribution. While Caldera OpenLinux 2.2 manages to load in spite of the rogue APM code in the BIOS, it's dreadfully slow in doing so. He's also tried PHT TurboLinux on the same hardware, with the old BIOS and the new BIOS, without success.

He was surprised to receive "The Letter" from Red Hat, especially since he received it before finding the cure, but he doesn't see anything sinister behind the invitation to participate in the IPO at ground-floor prices, and neither do I. Evidently, his name circulated widely within Red Hat during those two weeks.

Red Hat may be gripped by IPO fever and simply not performing up to par at present ... I don't know. But I certainly hope that the major problems seen here with its tech support, the length of time it took before the techs told Ramon "No mas!" and the silence with which his request for escalation was answered, are fixed. Both are eminently fixable, but Red Hat has to want to fix them.

You can't charge money for tech support and then ignore your customers. You can't simply go silent when you don't have a ready answer. And you can't reset the clock on a promised response based on a technicality. It makes you look bad. Real bad.

Are you listening, Raleigh?

Companies race to support Linux applications
July 5, 1999
Opinion: Get online for (free) Linux support!
June 16, 1999
Linux fans fear Red Hat takeover
March 26, 1999

Novell invests in Red Hat, four Internet companies to push NDS
(Network World Fusion)
Red Hat launches Linux e-commerce server
Red Hat expands into Europe, buys Linux vendor
Red Hat picks Global Knowledge as training partner
(InfoWorld Electric)
Compaq, Red Hat ink joint development pact
Red Hat files for IPO
(InfoWorld Electric)
Putting Red Hat 6.0 through its paces
Note: Pages will open in a new browser window
External sites are not endorsed by CNN Interactive.

Red Hat
Note: Pages will open in a new browser window
External sites are not endorsed by CNN Interactive.
Enter keyword(s)   go    help

Back to the top   © 2001 Cable News Network. All Rights Reserved.
Terms under which this service is provided to you.
Read our privacy guidelines.