Rate limiting options with load balanced servers

I'm looking for ways to rate limit page views due to users who get hold
of some tool and attempt to copy hundreds of thousands of pages from our
web sites. If we had a single web server, there appear to be several
options (mod_throttle, mod_bandwidth, mod_cband, etc.). We have
multiple web servers behind hardware load balancers, so no single server
has a view of all the traffic.

My current thought is to put dedicated firewalls in front of the load
balancers to throttle connections above a certain limit - we are
currently in a hosting situation with shared firewalls and do not have
this level of control.

Does anyone have alternate suggestions? We are running Apache 1.3 with
intentions of migrating most sites to Apache 2.2 in the near future.

Thanks,
Jim

how about changing your load balancer scheduling algorithm to one that binds
individual users to a single back-end server? then you can apply your choice of
available rate-limiting options that you've already turned up.

you didn't mention what sort of load balancing hardware you're using, here is
documentation of some generic / widely available load-balancing algorithms c/o
the linux ipvs project:

http://www.linuxvirtualserver.org/docs/scheduling.html

and if you can't adjust your load balancers, you could consider setting up a
high-availability proxy server (instead of hardware load balancing), and
employing your rate limiting options on the proxy server. there's a good essay
(and proxy software) for this approach here:

http://haproxy.1wt.eu/

good luck!
-sean

--
sean dreilinger - http://durak.org/sean/


Jim Hayter wrote:
> I'm looking for ways to rate limit page views due to users who get hold
> of some tool and attempt to copy hundreds of thousands of pages from our
> web sites. If we had a single web server, there appear to be several
> options (mod_throttle, mod_bandwidth, mod_cband, etc.). We have
> multiple web servers behind hardware load balancers, so no single server
> has a view of all the traffic.
>
> My current thought is to put dedicated firewalls in front of the load
> balancers to throttle connections above a certain limit - we are
> currently in a hosting situation with shared firewalls and do not have
> this level of control.
>
> Does anyone have alternate suggestions? We are running Apache 1.3 with
> intentions of migrating most sites to Apache 2.2 in the near future.
>
> Thanks,
> Jim