Nutch
    ~

IseeDeadPeople
: 06.12.2005
: 18000
# : 10, 2009 1:32 pm
, 10 , . , :
Nutch 19827+4606 586.49

.. , .. , , 50 ..

.. .. .. , robots.txt, ..
:
User-agent: *
Crawl-delay: 5

User-agent: nutch
Disallow: /



.. , , / .. .. , . ?

, "User-agent: nutch" -, .. .. nutch
.. nutch Nutch ( ..) ?


, , .htaccess
/ "Nutch" ?

.. , 403 ()..
(403 - / .. ?)

/ / robots.txt :
:
User-agent: *
Crawl-delay: 5


( - )

_______


.. nutch
.. .. ..
http://m.lsl.com/

IseeDeadPeople
: 06.12.2005
: 18000
# : 10, 2009 1:37 pm
,
:
User-agent: *
Crawl-delay: 5



, .,
, . ..

.. 1000 ..
, , , , , , 1000/5 (=200) .

, , ., . ..
:
User-agent: *
Crawl-delay: 5

, () .
http://m.lsl.com/

Soeti
: 05.06.2009
: 616
# : 10, 2009 5:28 pm
user-agent
- .

http://www.globalsecurity.org/robots.txt

. , .

? , . - .

. :
:

User-agent: CazoodleBot/Nutch-0.9-dev
Disallow: /

User-agent: CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com; [email protected])
Disallow: /

.

..... !( )
    ~