August 24, 1999 Web posted at: 11:24 a.m. EDT (1524 GMT) by Lynda Radosevich

From...

(IDG) -- These days, if your business strategy does not include running a dot-com with hypergrowth traffic rates, you are considered passé. But increased traffic is destined to make Web sites operate more slowly and crash -- unless IT professionals build the behind-the-scenes servers, software, and networks to handle heavy-duty use before the crowds come.

MORE COMPUTING INTELLIGENCE
	IDG.net home page
InfoWorld home page
InfoWorld forums home page
InfoWorld Internet commerce section
Get Media Grok and The Industry Standard Intelligencer delivered for free
Reviews & in-depth info at IDG.net
IDG.net's personal news page
Year 2000 World
Questions about computers? Let IDG.net's editors help you
Subscribe to IDG.net's free daily newsletter for IT leaders
Search IDG.net in 12 languages
News Radio
	Fusion audio primers
	Computerworld Minute

In a recent panel discussion, four technology officers at companies with dynamic, high-volume Web sites discussed what they have learned about scaling back-end systems. Back-end systems comprise the core server components -- including applications, databases, and supporting technologies, such as load-balancing software -- rather than static HTML Web servers.

The panel included Marc Hansen, chief technology officer (CTO) at iEngineer.com, a San Francisco-based engineering portal, and former vice president of systems architecture at J. Crew; Rick Lamb, chief operating officer at Bottle Rocket, a New York-based online game site; Jon Tulk, CTO at HealthAxis.com, an online health insurance company in Norristown, Penn., and former CTO at music site N2K; and Dan Woods, CTO at TheStreet.com, a New York-based financial news site. InfoWorld Deputy Features Editor Lynda Radosevich moderated the forum.

The panelists described the experience and wise practices they picked up preparing their companies' Web sites for high-volume traffic.

Expect Web site traffic demands and bottlenecks to be moving targets

Jon Tulk: "The traffic goes up, and you see that you're starting to get a little slow in one area. You figure out where [the bottleneck is] coming from, whether it's a network issue, whether it's CPU capacity, [or whether it's related to the] memory disc, database, or whatever. You go in and resolve that problem; then the next day, the system's getting slow again. You're almost never chasing the same problem twice."

Rick Lamb: Referring to a former job at an online game company: "We had no idea what the spikes and surges would be against our system."

Focus on the back-end database and overall architecture

Marc Hansen: "Performance isn't any one thing; it's a good fundamental architecture, and then it's hundreds of things. We fine-tuned our cache on our database so well -- using a main cache and gigabytes of memory -- [that] our disk activity was extraordinarily low, except for the disk log. Eventually our bottleneck became the rotation to the hard disk. As time goes on, the bottlenecks will change ... but if you keep all the historical data, you can go back to it and predict what's going to happen."

Dan Woods: "Segment your architecture so you have lots of smaller machines ... handling one application."

Marc Hansen: "Think of [structuring your architecture] like constructing a building. We could order a big pile of lumber and start nailing it together, or we could create a clear set of plans before we start. When it comes to creating the plans, there are home builders and there are architects who work on large-scale buildings. There is a continuum of skills available. Matching the skills to the size of the project is important. Choosing the correct initial architecture is also important. If you start with a wood-frame house, it will be difficult to grow it into a 40-story office building.

Select a high-quality application server, and buy scalable database servers

Jon Tulk: "[The application server] should provide application-server load balancing, session-state management, database-connection pooling, thread pooling, data or content caching, and, ideally, some monitoring, measurement, and management facilities. We have selected Bluestone Sapphire/Web, which is an excellent product."

Marc Hansen: "We're talking about real databases here, not PCs with [Microsoft] SQL Server on them. Put in place all the properties you would put in place for a mainframe. ... In fact, I would recommend you hire yourself an IBM data center manager."

Rick Lamb: "[This] is absolutely true for application-server and database-server layers: At the Web-server layer, PCs running Linux can be a perfectly acceptable solution and slightly more cost-effective."

Marc Hansen: "I agree with Rick that Linux works fine for HTTP servers, but I would differ in that I believe that the salary cost of maintaining skills on another OS [such as Linux] and hardware cost more than the slight savings in hardware [that Linux can provide]."

Build a good data model up front

Jon Tulk: "The data model we had was not designed that well up front, basically because those of us who first built the site didn't have database backgrounds. We didn't realize ... how hard it is to change a data model once you're on your way. And we carried that history with us for the next five years. It affected the performance of the site. It affected the ease of integrating new features and adding data fields and columns to the data tables. It affected the ability to replicate the database, and that affected the ability to do distribution."

Rick Lamb: "Design your database into subject areas and entities -- don't rush the design. Review all application requirements and loads prior to committing to a database implementation. Test the system on paper, in diagrams. Find and productively employ experienced data modelers."

Get good database administrators, and pay them well

Jon Tulk: "With help from a couple of consulting DBAs [database administrators], we added a lot of indexes [to the database], and we had a huge difference in performance. The load on our server went down almost in half at the same time as our traffic nearly tripled -- this was going into last Christmas season -- and it was all to do with optimizations. It's a lot easier to fix this up front by finding the good data modelists, and with a good design."

Marc Hansen: "We were shocked to discover the market price [for a DBA], which was about double what we budgeted -- about six figures." However, "It turned out to be the best investment we ever made in terms of solving scalability problems. A good DBA pays [a company back in hardware savings]."

Dan Woods: "If you can find really smart people, I would pay them above market rates. You should never have them thinking about their compensation. They get 15 to 20 headhunter calls a month, and [compensation] should be a nonissue.

"I've got a playbook on [hiring good DBAs]. First, the person must be personally secure. They must not brag about how much they know and what they did. They must be interested in your business and what you want to do. Second, they must understand the role of technology -- to help business, not to be an end in itself. Third is, are they smart? And the last is whether they have experience. I find if you run that playbook over and over, you get an environment where you have really smart people who attract other really smart people and nobody is a bully or fearmonger."

Cache data and pieces of static content that will be used to construct a page

Dan Woods: "People get drunk on dynamic content with unproven business value, such as push and personalization. ... Keep everything, if possible, on static pages; avoid dynamic content as much as possible. The way you handle that for a large-scale Web site is, you have a high-speed templating agent that is working against application components that are cached."

Marc Hansen: "Dan's comments about static content are true, but there are some kinds of Web businesses that don't lend themselves to static content, like e-commerce. For example, at J. Crew, when you want to buy something, we actually verify inventory in real time in our inventory management system that ran on CICS on [an IBM] System 390."

Dan Woods: "I would much rather do something like take the screen of updates from your real-time inventory and update all the static pages in the background, then run the site using Thunderstone [dynamic publishing software], so you're delivering flat pages all the time. Then you've got the maximum scalable system."

Do capacity planning for traffic spikes and ongoing growth

Jon Tulk: "Traffic planning is an absolute must, and it's hard to do when you start, because you don't have enough data to predict off of. After building some data, you can use a spreadsheet to create a simple traffic prediction model based on the historical data. Get more sophisticated later as the need demands and time permits. Also, it is good to choose some performance metrics that you will measure over time. One useful metric is the number of Web pages [served] per minute, per CPU. This can be used to predict hardware requirements against the traffic model or to monitor system performance changes over time."

Dan Woods: "We figure that [building capacity for] six to eight times the average load is what we need to not have to turn anybody away. ... Our big days are around a million page views. We think that on the biggest day, we'll probably get two and a half to 3 million [page views]."

Marc Hansen: "Measure everything that's going on, and keep the data. Because your business people will decide to run a promotion, and they'll say, 'Remember that promotion we did last week? We're going to run one like it but with double the traffic.' Now you can go back and look at all your instrumented data and see what's going to happen."

Use outsourcing to host sites more reliably and less expensively than you can in-house, but develop and prototype new applications in-house, where you can make changes more rapidly

Rick Lamb: "You simply can't take upon yourself the complete responsibility of putting together a bulletproof design, reacting to what are usually pretty aggressive schedule demands. We use consultants on the design side. We use consultants on the program management side. ... There is tremendous value to capturing and preserving the intelligence from consultants. ... It's often difficult to enable a high degree of cooperation between outsource consultants and in-house staff, but you've got to overcome that with human management."

Dan Woods: "It is very difficult to duplicate internally what you can attain from the first-tier colocation providers. I mean, these organizations have separate power feeds from different grids, multiple UPSes [uninterruptible power supplies], multiple generators, and full-time staff to maintain the power and environment infrastructure. It would never be cost-effective for us to do that on an individual basis."

Jon Tulk: "At N2K, we sourced our own machines ... because we already had a computer facility. And we preferred having the system right there in the next room. Likewise, at HealthAxis, we're bringing the site home, because for the next few months, while we're really building the thing and changing hardware, architecture, and software, it's going to be advantageous to have really tight access to [the system].

If your company is going public, establish the focus of the business and the stability of the Web site before the initial public offering

Dan Woods: "Once you become public, you have a whole new set of people talking to your senior management. The Wall Street analysts are going to ask, 'What are you guys doing about disaster recovery?' ... A lot of hard questions are being asked about environments."

Lynda Radosevich is deputy features editor for InfoWorld.