Protecting files being downloaded

Il Sun, 18 Jul 2004 09:51:15 -0700, Purl Gurl ha scritto:

 > lamp wrote:
 >
  >> How do I configure my apache webserver to deny clients downloading all
  >> the files in my site using wget -r <a style='text-decoration: underline;' href="http://www.mysite.com" target="_blank">www.mysite.com</a> ?
 >
 >
 > In reality, you cannot. Wget is a very hostile application.
 >
 > You can use mod_rewrite to browser match "wget" in the
 > remote agent string. This works for most cases. Simply
 > match "wget" and deny access or more favored, send it
 > to a sticky honeypot.

It is incredible how a simple program could do these things. I discovered
it trying to download some images I liked, but it started to download
every page of the site.


 > This wget application is hostile and a certain shame upon GNU Dot Org
 > for writing and making available an application designed to inflict harm
 > upon others. Through their wget application, GNU Dot Org displays a
 > complete lack of respect for the internet community.

So, do you believe this program could steal every file in a directory?
Isn't it possibile to protect any directory at all? I am facing these
topics for a few time only, and I don't know how to do it.

Francesco<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded 

lamp wrote:

 > Purl Gurl wrote:
  > > lamp wrote:

(snipped)

   > >> How do I configure my apache webserver to deny clients downloading all
   > >> the files in my site using wget -r <a style='text-decoration: underline;' href="http://www.mysite.com" target="_blank">www.mysite.com</a> ?

  > > In reality, you cannot. Wget is a very hostile application.

 > It is incredible how a simple program could do these things. I discovered
 > it trying to download some images I liked, but it started to download
 > every page of the site.

You used the "recursive" command line switch.


 > So, do you believe this program could steal every file in a directory?
 > Isn't it possibile to protect any directory at all? I am facing these
 > topics for a few time only, and I don't know how to do it.

No, wget, like Perl LWP, does not "steal" files but rather
has an ability to recursively download files which you make
publically available. Anyone can download your public files;
no stealing involved.

Unlike Perl LWP, GNU Dot Org has written in features which
are directly intended to defeat your security. Doing so
is very wrongful behavior. Yes, Perl LWP can be written to
do the same, but violating site security is not a feature
already written into Perl's LWP; a Perl programmer does this.

GNU Dot Org taunts its wget, based in part, on its ability
to defeat site security through user command line options.

Nothing illegal at all. However, it is extremely rude to
clobber a site, like yours, with recursive downloads, even
if you want to disallow this.

Look at browser matching to defeat wget but keep in mind,
GNU Dot Org wrote in coding to defeat this, which is very
disrepectful of the internet community.


Purl Gurl<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded 

Purl Gurl wrote:

 > lamp wrote:
  > > Purl Gurl wrote:
   > > > lamp wrote:

(snipped)


 > Look at browser matching to defeat wget but keep in mind,
 > GNU Dot Org wrote in coding to defeat this, which is very
 > disrepectful of the internet community.

Here is an example using mod_rewrite with Apache:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*wget.*$
RewriteRule ^.*$ - [F]

That will produce a "forbidden" message but will not
stop wget from hitting every file on your server, if
your pages have been previously indexed and stored.

It is likely wget will produce as many forbidden messages
as there are files on your server; you are clobbered
regardless. Your files are not downloaded but legit
visitors are abused by "robbing" them of bandwidth.

Contrasting this, if the person using wget has not
previously indexed your site, usually only a single
forbidden message is generated. Nonetheless, that
person then can use command line options to defeat
your security; he wins, everyone else loses.

My example up there is very verbose, this is, no
shortcuts are used. This is best when writing
regular expressions (regex) to prevent errors.
Once working, then you can shortcut.

Check the case sensitivity for wget. I cannot remember
if it comes in as "Wget" or as "wget" for a browser.

This wget program is not a serious problem, least in
my experience. I cannot recall the last time wget
appeared in our logs. Fairly rare, maybe three or
four times a month.

However, I have written in programming which automatically
banishes any server downloading at a high speed rate,
for selected directories. That works very well.

I would not worry too much about wget but rather focus
on abusive spiders such as Link Walker and similar.


Purl Gurl<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded 

 > No, wget, like Perl LWP, does not "steal" files but rather
 > has an ability to recursively download files which you make
 > publically available. Anyone can download your public files;
 > no stealing involved.
 >

I am trying wget with some sites I did long time ago.
I have seen that some sites don't allow wget to retrieve everything, for
example, for some sites of mine, I only get the index.html page
So, it is possible do disallow people getting everything of a site. You
told me to use mod_rewrite, colud you tell me more, maybe with an example?
And how do I make some directory pubblicly available, or unavailable?

Thanks

Francesco<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded 

lamp wrote:

  > > No, wget, like Perl LWP, does not "steal" files but rather
  > > has an ability to recursively download files which you make
  > > publically available. Anyone can download your public files;
  > > no stealing involved.

 > I am trying wget with some sites I did long time ago.
 > I have seen that some sites don't allow wget to retrieve everything, for
 > example, for some sites of mine, I only get the index.html page
 > So, it is possible do disallow people getting everything of a site. You
 > told me to use mod_rewrite, colud you tell me more, maybe with an example?
 > And how do I make some directory pubblicly available, or unavailable?


Sorry, not about to write an Apache book!

There are dozens and dozens of security methods available
for controlling access to both directories and files.

A good beginning point, is researching and reading about .htaccess

<a style='text-decoration: underline;' href="http://www.purlgurl.net/manual/howto/htaccess.html" target="_blank">http://www.purlgurl.net/manual/howto/htaccess.html</a>

Use of .htaccess is the most popular and most basic method.
Easy to learn, easy to use.


Search the net with your favorite search engine. You will find
more resources than you can read in a year. Some key expressions,

Apache security
Apache htaccess
Apache mod_security
Apache mod_rewrite
Apache mod_throttle
Apache password
Apache authentication
Apache access control
Apache security tips
Apache user directories
Apache cgi security

Those key expressions will keep you reading for months.

Purl Gurl<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded 

Ok, I'll start to study, I am really interested.

Thanks to your answers.

Francesco
 >> Stay informed about: Protecting files being downloaded 

Purl Gurl wrote:

*SNIPPED*
 > This wget application is hostile and a certain shame upon
 > GNU Dot Org for writing and making available an application
 > designed to inflict harm upon others. Through their wget
 > application, GNU Dot Org displays a complete lack of respect
 > for the internet community.

This HAS to be a troll.....but I'll bite.

Yet you *claim* to be exceptionally intelligent and "educated". Don't blame
the authors of wget for its misuse. Wget is an exceptionally useful tool
for a number of reasons. If you don't know the reasons why someone would
legitimately want to use wget then you wouldn't understand the explanation
I can give.

Those who ask "why", usually, wouldn't understand the answer.

*PLONK*

James
--
Anyone who cannot cope with mathematics is not fully human. At best he
is a tolerable subhuman who has learned to wear shoes, bathe and not
make messes in the house.
-- Lazarus Long, "Time Enough for Love"<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded 

Centurion wrote:

 > Purl Gurl wrote:

 > *SNIPPED*

  > > This wget application is hostile and a certain shame upon
  > > GNU Dot Org for writing and making available an application
  > > designed to inflict harm upon others. Through their wget
  > > application, GNU Dot Org displays a complete lack of respect
  > > for the internet community.

 > This HAS to be a troll.....but I'll bite.

 > Yet you *claim* to be exceptionally intelligent and "educated". Don't blame
 > the authors of wget for its misuse. Wget is an exceptionally useful tool
 > for a number of reasons. If you don't know the reasons why someone would
 > legitimately want to use wget then you wouldn't understand the explanation
 > I can give.

 > Those who ask "why", usually, wouldn't understand the answer.

 > *PLONK*


Well gosh, yours is certainly a delightful display of
ignorance and hatred. You must be one of the boys from
Apache Dot Org.


Purl Gurl<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Protecting files being downloaded