[ale] Linking Linuxes

[Jeff Barber]

Dan Newcombe writes:
> 
> > I spoke with a rep from SGI that mentioned you could take SGI WebForce 
> > servers and link them together to handle the incoming load.
> > Is there a way you can do that with Linux boxes or is this guy blowing
> > smoke?
> > He said that Netscape has 8 SGI's linked together at www.netscape.com and 
> > that's how they handled 1 Billion hits last month on their site.
> 
> Yahoo does something like that - you hit www.yahoo.com, and you may be 
> sent to www1, www2, www3, etc...
> 
> It may be some kinda proxy server, or special DNS setup which gives a 
> different address each time to keep them rotating.  The main concern 
> would be how do you let all 8 share data?  Do they all have copies of 
> the same data on thier machines?  Or are they linked via a fast 
> ethernet internally, with external connections on another net segment?

I understand that some sites do this with DNS as you describe, but that
this can cause problems when DNS lookups get cached.  That is, if you
ask your local DNS server machine to lookup www.yahoo.com and get www3's
IP address, chances are good that your neighbor is going to get the same
address when doing the same lookup.  Of course, that still serves to
distribute the load somewhat, but if a single site is acting as a caching
server for a large subnet, it can lead to overloads on particular machines.

According to Netscape, they actually have a special hack in their browser
that adds a digit to the host name *before* doing the DNS lookup to the
Netscape domain (I don't know the algorithm for what digit to add) -- thus
ensuring a fairly uniform spread across the server pool.  Unfortunately,
your average web site doesn't have that luxury.

BTW, according to Netscape folks at their recent developer conference,
they have other machines than SGIs in their pool of servers, including
Suns, HPs, DEC Alphas, etc.  (I guess you wouldn't expect to hear that
from an SGI rep though, eh? :-)


-- Jeff