DoS attack revealed against Probert Encyclopaedia

Matt Probert wrote:

 > During the period August 4th to August 5th The Probert Encyclopaedia
 > was subjected to a crude denial of service attack, allegedly
 > originating from IP address 69.56.130.138.
 >
 > The attack took the form continuous random strings of different words
 > being sent to our search engine, which inevitably resulted in a server
 > overload.
 >
 > Naturally this is an occupational hazard for any web site. The
 > question is, however, how best to minimise the impact?
 >
 > We shall conduct our own investigations and trials into ways to try
 > and capture non-genuine search queries and quickly reject them, but in
 > the meantime, for the benefit of all webmasters who operate search
 > facilities and the like, anyone got any ideas?
 >
 > Matt

Hoy Matt,

If you use iptables as a firewall then use it to limit TCP inbpund packets
to prevent DOS.

example:

iptables -t nat -N syn-flood
#limit 12 connections per second
iptables -t nat -A syn-flood -m limit --limit 12/s --limit-burst 24 \
-j RETURN
iptables -t nat -A syn-flood -j DROP
#Check for DOS
iptables -t nat -A PREROUTING -i $EXT_IFACE -d $DEST_IP -p tcp --syn \
-j syn-flood

The rules limit new inbound connections (packets with the SYN set) to 12 per
second after 24 connections per second have been seen.

Or
# Drop packets from hosts with more than 16 active connections
iptables -t nat -A PREROUTING -i EXT_IFACE -p tcp --syn -d $DEST_IP \
-m iplimit --iplimit-above 16 -j DROP

YMMV

Checkout iptables firewalling for more @ <a style='text-decoration: underline;' href="http://www.netfilter.org" target="_blank">http://www.netfilter.org</a>

--
There are 10 types of people in this world
Those that understand binary and those that don't<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

comments.DeleteThis@probertencyclopaedia.com (Matt Probert) wrote in
news:411636cf.91721037@news.ntlworld.com:

 > During the period August 4th to August 5th The Probert Encyclopaedia
 > was subjected to a crude denial of service attack, allegedly
 > originating from IP address 69.56.130.138.
 >
 > The attack took the form continuous random strings of different words
 > being sent to our search engine, which inevitably resulted in a server
 > overload.
 >
 > Naturally this is an occupational hazard for any web site. The
 > question is, however, how best to minimise the impact?
 >
 > We shall conduct our own investigations and trials into ways to try
 > and capture non-genuine search queries and quickly reject them, but in
 > the meantime, for the benefit of all webmasters who operate search
 > facilities and the like, anyone got any ideas?
 >
 > Matt

There is a simple solution that does not require you guessing whether a
search is legitimate or not.

Limit the number of searches an IP can make within a given period of time
(15 seconds or so).

No human will be inconvenienced because it is extremely unlikely they will
do multiple searches in less than 15 secconds.

So, nobody can make 60 concurrent searches. They can only make 4 searches
per minute.

For someone to take your site down again, they would have to employ the
more sofisticated DDOS (distributed) attack instead of the simple DOS
attack they are using now. Not impossible but way above the capability of
most script kiddies.


--
Edward Alfert
<a style='text-decoration: underline;' href="http://www.rootmode.com/" target="_blank">http://www.rootmode.com/</a>
Multiple Domain Hosting and Reseller Hosting Plans
Promotional Code (Recurring $5/month Discount): newsgroup<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

## comments.DeleteThis@probertencyclopaedia.com (Matt Probert) wrote:

 >During the period August 4th to August 5th The Probert Encyclopaedia
 >was subjected to a crude denial of service attack, allegedly
 >originating from IP address 69.56.130.138.
 >
 >The attack took the form continuous random strings of different words
 >being sent to our search engine, which inevitably resulted in a server
 >overload.
 >
 >Naturally this is an occupational hazard for any web site. The
 >question is, however, how best to minimise the impact?
 >
 >We shall conduct our own investigations and trials into ways to try
 >and capture non-genuine search queries and quickly reject them, but in
 >the meantime, for the benefit of all webmasters who operate search
 >facilities and the like, anyone got any ideas?
 >
 >Matt
 >

DisBoards, a forum I frequent, has a search once per 40seconds
feature. It may come in the forum software itself. This in case you
wanted to search the software, or email the disboards.com folks and
ask if they'll share with you how they did it. I wish I could be
more helpful, I do not know how to do it myself.

I took the braaaaainbench test on php last night. Don't get the
results til Monday. Was quite interesting. The test itself taught
me cool php features <voice="discoverychannelnatureshow">Witness
the recalcitrant Windows' Visual Basic programmer, tail twitching,
moving ever closer to a complete switch into the Php world.</voice>


Sherry
--
me(www.CactusBlossom.org) fyi(http://allmyfaqs.com/faq.pl?How_to_post)
Before you criticize someone you should walk a mile in their shoes. That
way when they get angry, you're a mile away and have their shoes.<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

Edward Alfert wrote:
 > comments RemoveThis @probertencyclopaedia.com (Matt Probert) wrote in
 > news:411636cf.91721037@news.ntlworld.com:
 >
  >> During the period August 4th to August 5th The Probert Encyclopaedia
  >> was subjected to a crude denial of service attack, allegedly
  >> originating from IP address 69.56.130.138.
  >>
  >> The attack took the form continuous random strings of different words
  >> being sent to our search engine, which inevitably resulted in a
  >> server overload.
  >>
  >> Naturally this is an occupational hazard for any web site. The
  >> question is, however, how best to minimise the impact?
  >>
  >> We shall conduct our own investigations and trials into ways to try
  >> and capture non-genuine search queries and quickly reject them, but
  >> in the meantime, for the benefit of all webmasters who operate search
  >> facilities and the like, anyone got any ideas?
  >>
  >> Matt
 >
 > There is a simple solution that does not require you guessing whether
 > a search is legitimate or not.
 >
 > Limit the number of searches an IP can make within a given period of
 > time (15 seconds or so).
 >
 > No human will be inconvenienced because it is extremely unlikely they
 > will do multiple searches in less than 15 secconds.
 >
 > So, nobody can make 60 concurrent searches. They can only make 4
 > searches per minute.
 >
 > For someone to take your site down again, they would have to employ
 > the more sofisticated DDOS (distributed) attack instead of the simple
 > DOS attack they are using now. Not impossible but way above the
 > capability of most script kiddies.

The PPC Script I use for my search site limits searches for each IP to a
time I can set. I would try to get you that code but the programer has the
script encrypted.. Plus its PHP and I think you use CGI....<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

On Sun, 8 Aug 2004 14:39:30 -0400, Viper wrote:

 > Edward Alfert wrote:
  >> comments.DeleteThis@probertencyclopaedia.com (Matt Probert) wrote in
  >> news:411636cf.91721037@news.ntlworld.com:
  >>
   >>> During the period August 4th to August 5th The Probert Encyclopaedia
   >>> was subjected to a crude denial of service attack, allegedly
   >>> originating from IP address 69.56.130.138.
   >>>
   >>> The attack took the form continuous random strings of different words
   >>> being sent to our search engine, which inevitably resulted in a
   >>> server overload.
   >>>
   >>> Naturally this is an occupational hazard for any web site. The
   >>> question is, however, how best to minimise the impact?
   >>>
   >>> We shall conduct our own investigations and trials into ways to try
   >>> and capture non-genuine search queries and quickly reject them, but
   >>> in the meantime, for the benefit of all webmasters who operate search
   >>> facilities and the like, anyone got any ideas?
   >>>
   >>> Matt
  >>
  >> There is a simple solution that does not require you guessing whether
  >> a search is legitimate or not.
  >>
  >> Limit the number of searches an IP can make within a given period of
  >> time (15 seconds or so).
  >>
  >> No human will be inconvenienced because it is extremely unlikely they
  >> will do multiple searches in less than 15 secconds.
  >>
  >> So, nobody can make 60 concurrent searches. They can only make 4
  >> searches per minute.
  >>
  >> For someone to take your site down again, they would have to employ
  >> the more sofisticated DDOS (distributed) attack instead of the simple
  >> DOS attack they are using now. Not impossible but way above the
  >> capability of most script kiddies.
 >
 > The PPC Script I use for my search site limits searches for each IP to a
 > time I can set. I would try to get you that code but the programer has the
 > script encrypted.. Plus its PHP and I think you use CGI....


How does one encrypt a php script?


Karim
--
<a style='text-decoration: underline;' href="http://www.cheapesthosting.com" target="_blank">http://www.cheapesthosting.com</a> - Innovative Web Hosting since 1998
Online calendars with event email notification - Online Photo Album (RSS
Enabled)<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

Karim wrote:
  >>
  >> The PPC Script I use for my search site limits searches for each IP
  >> to a time I can set. I would try to get you that code but the
  >> programer has the script encrypted.. Plus its PHP and I think
  >> you use CGI....
 >
 >
 > How does one encrypt a php script?
 >
SourceGuardian <a style='text-decoration: underline;' href="http://www.sourceguardian.com/" target="_blank">http://www.sourceguardian.com/</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

Matt Probert wrote:

 > During the period August 4th to August 5th The Probert Encyclopaedia
 > was subjected to a crude denial of service attack, allegedly
 > originating from IP address 69.56.130.138.
 >
 > The attack took the form continuous random strings of different words
 > being sent to our search engine, which inevitably resulted in a server
 > overload.
 >
 > Naturally this is an occupational hazard for any web site. The
 > question is, however, how best to minimise the impact?
 >
 > We shall conduct our own investigations and trials into ways to try
 > and capture non-genuine search queries and quickly reject them, but in
 > the meantime, for the benefit of all webmasters who operate search
 > facilities and the like, anyone got any ideas?

I always ask myself "what does Google do?", "what does Ebay do?".

They must get bombarded by all sorts of things.

I quite like the idea of limiting searches to a time frame per IP
address, but I am not sure what overhead this has.

One often sees solutions that involve processing, sometimes elaborate,
again overhead would be my concern here.

In the end, and I suppose this is what Google and Ebay do, one just has
to be geared up for a flood of requests with sufficient processing and
serving capacity.

--
Charles Sweeney
<a style='text-decoration: underline;' href="http://CharlesSweeney.com" target="_blank">http://CharlesSweeney.com</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

On 8 Aug 2004 15:17:52 GMT Edward Alfert <ealfert DeleteThis @rootmode.com> broke
off from drinking a cup of tea at RootMode LLC to write:

 >
 >Limit the number of searches an IP can make within a given period of time
 >(15 seconds or so).
 >

A sound idea.

But how might one implement this in practice?

One problem we discovered, was that people were submitting a search
request, and then quite innocently resubmitting it repeatedly before
the results had returned - a problem which is further compounded when
the server is already busy and the search takes longer.

A practical way to stop this was to insert a "Please wait, we're
dealing with it" page, thus:

#!/usr/bin/perl

$datadir = "/home/sites/home/web";
$hostname = "www.mydomain.com";

$chklen = $ENV{'CONTENT_LENGTH'};

if ($chklen > 100)
{
  exit;
}

if ($ENV{'REQUEST_METHOD'} eq 'GET')
{
$input = $ENV{QUERY_STRING};
}

#Start the HTML with some search engine type words
print "Content-type: text/html\n\n";
print "<HTML>\n";
print "<HEAD>\n";
print "<BASE HREF=http://$hostname/index.html>";
print "<TITLE>Search</TITLE>\n";
print "<link rel=stylesheet type=\"text/css\" href=\"style.css\">\n";
print "<meta http-equiv=\"Refresh\" content=\"0;
URL=http://www.mydomain.com/cgi-bin/actual_search_script.pl?$input\">\n";
print "</HEAD>\n";
print "<BODY>\n";

print "<blockquote><p> <p> <p>\n";

print "<h1 align=center>Your search is being processed.</h1>\n";
print "<p><h1 align=center>Please Wait...</h1>\n";

open(FILE,"/proc/loadavg");
$_ = <FILE>;
close(FILE);

@pairs = split(/ /, $_);
$load = @pairs[1];
$load =~ s/ //g;

if ($load > 5)
{
  print "<p><h1 align=center>We are currently dealing with a lot
of searches.<br>Please be patient!</h1>\n";
}

print "</blockquote>\n";
print "</body>\n";
print "</html>\n";

exit;

This script relies on the remote browser to honour the meta-refresh
tag to pass the search request on to a second script which is the real
search script, which I realise is not guarranteed - but that's just
tough!

It also has built in a check of server load, and will display an
appropriate message if the server is very busy.

Perhaps not the most elegant way of doing things, but it does work <g>

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

comments.DeleteThis@probertencyclopaedia.com (Matt Probert) wrote in
news:41165585.3833942@news.ntlworld.com:

 > On 8 Aug 2004 15:17:52 GMT Edward Alfert <ealfert.DeleteThis@rootmode.com> broke
 > off from drinking a cup of tea at RootMode LLC to write:
 >
  >>
  >>Limit the number of searches an IP can make within a given period of
  >>time (15 seconds or so).
  >>
 >
 > A sound idea.
 >
 > But how might one implement this in practice?
 >
 > One problem we discovered, was that people were submitting a search
 > request, and then quite innocently resubmitting it repeatedly before
 > the results had returned - a problem which is further compounded when
 > the server is already busy and the search takes longer.
 >
 > A practical way to stop this was to insert a "Please wait, we're
 > dealing with it" page, thus:
 >
<snip>
 >
 > This script relies on the remote browser to honour the meta-refresh
 > tag to pass the search request on to a second script which is the real
 > search script, which I realise is not guarranteed - but that's just
 > tough!
 >
 > It also has built in a check of server load, and will display an
 > appropriate message if the server is very busy.
 >
 > Perhaps not the most elegant way of doing things, but it does work <g>
 >
 > Matt


Can't you just put in an IP check at the same part of your script that
checks the load?

Of course, you would have to log the IPs of every search and before doing a
search verifying the amount of time since the last search by that IP. This
would increase the load on the server but it might be worth it if you are
constatntly being attacked.



--
Edward Alfert
<a style='text-decoration: underline;' href="http://www.rootmode.com/" target="_blank">http://www.rootmode.com/</a>
Multiple Domain Hosting and Reseller Hosting Plans
Promotional Code (Recurring $5/month Discount): newsgroup<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

Matt Probert wrote:

 >
 > One problem we discovered, was that people were submitting a search
 > request, and then quite innocently resubmitting it repeatedly before
 > the results had returned

same as a "pay now" or similar button. visitors sometimes need a
reminder not to be click happy.

 > a problem which is further compounded when
 > the server is already busy and the search takes longer.
 >
 > A practical way to stop this was to insert a "Please wait, we're
 > dealing with it" page, thus:
 >
 > #!/usr/bin/perl ...
 >
 > ...
 > This script relies on the remote browser to honour the meta-refresh
 > tag to pass the search request on to a second script which is the real
 > search script, which I realise is not guarranteed - but that's just
 > tough!
 >
 > It also has built in a check of server load, and will display an
 > appropriate message if the server is very busy.
 >
 > Perhaps not the most elegant way of doing things, but it does work <g>

Yep - just add a link with text "please follow this link if you are not
redirected in 7 seconds" or somesuch and you've covered the meta refresh
issue.

but a thought occurs. this is a different issue (to the DoS) and one
that can surely be solved with your server side script. may I say
"cache"? your server could trivially be trained to know what searches
it's running or run recently, it's also not a major leap for your script
to stash the most recent (and/or most frequent) results in a pre-canned
format for lightweight processing.

--
William Tasso
Recommended reading ...
<a style='text-decoration: underline;' href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a>
<a style='text-decoration: underline;' href="http://www.aww-faq.org/" target="_blank">http://www.aww-faq.org/</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

On Sun, 08 Aug 2004 17:38:26 GMT, Matt Probert wrote:

 > On 8 Aug 2004 15:17:52 GMT Edward Alfert <ealfert RemoveThis @rootmode.com> broke
 > off from drinking a cup of tea at RootMode LLC to write:
 >
  >>
  >>Limit the number of searches an IP can make within a given period of time
  >>(15 seconds or so).
  >>
 >
 > A sound idea.
 >
 > But how might one implement this in practice?
 >
 > One problem we discovered, was that people were submitting a search
 > request, and then quite innocently resubmitting it repeatedly before
 > the results had returned - a problem which is further compounded when
 > the server is already busy and the search takes longer.
 >
 > A practical way to stop this was to insert a "Please wait, we're
 > dealing with it" page, thus:
 >
 > #!/usr/bin/perl
 >
 > $datadir = "/home/sites/home/web";
 > $hostname = "www.mydomain.com";
 >
 > $chklen = $ENV{'CONTENT_LENGTH'};
 >
 > if ($chklen > 100)
 > {
  > exit;
 > }
 >
 > if ($ENV{'REQUEST_METHOD'} eq 'GET')
 > {
 > $input = $ENV{QUERY_STRING};
 > }
 >
 > #Start the HTML with some search engine type words
 > print "Content-type: text/html\n\n";
 > print "<HTML>\n";
 > print "<HEAD>\n";
 > print "<BASE HREF=http://$hostname/index.html>";
 > print "<TITLE>Search</TITLE>\n";
 > print "<link rel=stylesheet type=\"text/css\" href=\"style.css\">\n";
 > print "<meta http-equiv=\"Refresh\" content=\"0;
 > URL=http://www.mydomain.com/cgi-bin/actual_search_script.pl?$input\">\n";
 > print "</HEAD>\n";
 > print "<BODY>\n";
 >
 > print "<blockquote><p> <p> <p>\n";
 >
 > print "<h1 align=center>Your search is being processed.</h1>\n";
 > print "<p><h1 align=center>Please Wait...</h1>\n";
 >
 > open(FILE,"/proc/loadavg");
 > $_ = <FILE>;
 > close(FILE);
 >
 > @pairs = split(/ /, $_);
 > $load = @pairs[1];
 > $load =~ s/ //g;
 >
 > if ($load > 5)
 > {
  > print "<p><h1 align=center>We are currently dealing with a lot
 > of searches.<br>Please be patient!</h1>\n";
 > }
 >
 > print "</blockquote>\n";
 > print "</body>\n";
 > print "</html>\n";
 >
 > exit;
 >
 > This script relies on the remote browser to honour the meta-refresh
 > tag to pass the search request on to a second script which is the real
 > search script, which I realise is not guarranteed - but that's just
 > tough!
 >
 > It also has built in a check of server load, and will display an
 > appropriate message if the server is very busy.
 >
 > Perhaps not the most elegant way of doing things, but it does work <g>
 >

Log the ip address and let the database log the current timestamp. For
example in SQL Server let the default for the datetime column be GetDate().
Every time a new request comes in, search for the ip address and look at
the last time stamp and decide if you want to run it. In the database, put
an index on the ip address and keep the table small so that the queries run
very fast. This means keep only the last timestamp for each ip address. Use
cookies instead of ip addresses since more than one person can come from an
ip address.

I wouldn't call what's happening to you a DosAttach. If people keep hitting
the submit button, disable the button after the first hit.


Karim
--
<a style='text-decoration: underline;' href="http://www.cheapesthosting.com" target="_blank">http://www.cheapesthosting.com</a> - Innovative Web Hosting since 1998
Online calendars with event email notification - Online Photo Album (RSS
Enabled)<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

William Tasso wrote:

 > "cache"? your server could trivially be trained to know what searches
 > it's running or run recently, it's also not a major leap for your
script
 > to stash the most recent (and/or most frequent) results in a pre-canned
 > format for lightweight processing.

I use a system to cache the current search results.

Previously when you hit the back button, the script would make a fresh
trip to the database for the same search.

This would occur every time a user went back to the results. Very
inefficient and pointless.

I used this code in PHP to cache the results:

$expire = date('D, d M Y H:i:s T', gmmktime(0,0,0,gmdate("m"),(gmdate
("d")+1),gmdate("Y")));
header("Expires: $expire");
header("Cache-Control: max-age=86400, min-fresh=86400, max-stale=0");

Now when the user hits back, they get the cached page instantly.

As I discovered, you have to be careful where you use it, sometimes you
want a fresh page to update session variables.

--
Charles Sweeney
<a style='text-decoration: underline;' href="http://CharlesSweeney.com" target="_blank">http://CharlesSweeney.com</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia 

On 8 Aug 2004 16:50:44 GMT William Tasso <SpamBlocked RemoveThis @tbdata.com>
broke off from drinking a cup of tea at Chaos to write:

 >Yep - just add a link with text "please follow this link if you are not
 >redirected in 7 seconds" or somesuch and you've covered the meta refresh
 >issue.

Though this does allow the imaptient to resubmitt an already running
search. Hence I don't use use it.

 >
 >but a thought occurs. this is a different issue (to the DoS) and one
 >that can surely be solved with your server side script. may I say
 >"cache"? your server could trivially be trained to know what searches
 >it's running or run recently, it's also not a major leap for your script
 >to stash the most recent (and/or most frequent) results in a pre-canned
 >format for lightweight processing.

Trivial? I'm tired of saying I'm not a server expert <g> but it
doesn't seem to cache search requests, nor do intermediate caches
along the way, it appears, though that may be a misperception.

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: DoS attack revealed against Probert Encyclopaedia