自己網(wǎng)站的ROBOTS.TXT屏蔽的記錄，以及一些代碼和示例：

屏蔽后臺(tái)目錄，為了安全，做雙層管理后臺(tái)目錄/a/xxxx/，蜘蛛屏蔽/a/，既不透露后臺(tái)路徑，也屏蔽蜘蛛爬后臺(tái)目錄

緩存，阻止蜘蛛爬靜態(tài)緩存文件

下載，阻止蜘蛛爬下載目錄，若無(wú)用，刪除下載目錄

編輯器，阻止蜘蛛爬編輯器，也防止編輯器目錄被發(fā)現(xiàn)產(chǎn)生安全隱患

郵件，阻止蜘蛛爬靜態(tài)郵件模板

其他頁(yè)面，無(wú)收錄價(jià)值頁(yè)面屏蔽

圖片，阻止蜘蛛爬除JPG/jpg類文件之外的任何類型圖片

核心文件目錄，阻止蜘蛛直接爬include及其子目錄（函數(shù)/類庫(kù)/模型/模板等）

媒體目錄，阻止爬播放類型媒體目錄，若無(wú)用，刪除該目錄

附加參數(shù)頁(yè)面，阻止蜘蛛爬帶參數(shù)的頁(yè)面

RAR ZIP GZ文件類型

無(wú)效蜘蛛、惡意蜘蛛屏蔽

指定sitemap.xml位置

?

目錄屏蔽：

User-agent: *

Disallow: /a/

Disallow: /cache/

Disallow: /download/

Disallow: /editors/

Disallow: /email/

Disallow: /extras/

Disallow: /images/

Disallow: /includes/

Disallow: /media/

Disallow: /pub/

Disallow: /nddbc.html

Disallow: /page_not_found.php

Disallow: /login.html

Disallow: /privacy.html

Disallow: /conditions.html

Disallow: /contact_us.html

Disallow: /gv_faq.html

Disallow: /discount_coupon.html

Disallow: /unsubscribe.html

Disallow: /shopping_cart.html

Disallow: /ask_a_question.html

Disallow: /popup_image_additional.html

Disallow: /product_reviews_write.html

Disallow: /tell_a_friend.html

Disallow: /pages-popup_image.html

Disallow: /popup_image_additional.html

Disallow: /login.html

?

阻止蜘蛛爬非jpg圖片（限制產(chǎn)品圖片格式為jpg）

User-agent: Googlebot

Allow: .jpg$

Disallow: .jpeg$

Disallow: .gif$

Disallow: .png$

Disallow: .bmp$

?

阻止蜘蛛爬壓縮文件

User-agent: *

Disallow: .zip$

Disallow: .rar$

Disallow: .gz$

Disallow: .tar $

?

?

制定sitemap地址

Sitemap: http://www.xxx.jp/sitemap.xml

?

其他無(wú)效蜘蛛、惡意蜘蛛屏蔽：

User-Agent: almaden

Disallow: /

User-Agent: ASPSeek

Disallow: /

User-Agent: Axmo

Disallow: /

User-Agent: BaiduSpider

Disallow: /

User-Agent: booch

Disallow: /

User-Agent: DTS Agent

Disallow: /

User-Agent: Downloader

Disallow: /

User-Agent: EmailCollector

Disallow: /

User-Agent: EmailSiphon

Disallow: /

User-Agent: EmailWolf

Disallow: /

User-Agent: Expired Domain Sleuth

Disallow: /

User-Agent: Franklin Locator

Disallow: /

User-Agent: Gaisbot

Disallow: /

User-Agent: grub

Disallow: /

User-Agent: HughCrawler

Disallow: /

User-Agent: iaea.org

Disallow: /

User-Agent: lcabotAccept

Disallow: /

User-Agent: IconSurf

Disallow: /

User-Agent: Iltrovatore-Setaccio

Disallow: /

User-Agent: Indy Library

Disallow: /

User-Agent: IUPUI

Disallow: /

User-Agent: Kittiecentral

Disallow: /

User-Agent: iaea.org

Disallow: /

User-Agent: larbin

Disallow: /

User-Agent: lwp-trivial

Disallow: /

User-Agent: MetaTagRobot

Disallow: /

User-Agent: Missigua Locator

Disallow: /

User-Agent: NetResearchServer

Disallow: /

User-Agent: NextGenSearch

Disallow: /

User-Agent: NPbot

Disallow: /

User-Agent: Nutch

Disallow: /

User-Agent: ObjectsSearch

Disallow: /

User-Agent: Oracle Ultra Search

Disallow: /

User-Agent: PEERbot

Disallow: /

User-Agent: PictureOfInternet

Disallow: /

User-Agent: PlantyNet

Disallow: /

User-Agent: QuepasaCreep

Disallow: /

User-Agent: ScSpider

Disallow: /

User-Agent: SOFT411

Disallow: /

User-Agent: spider.acont.de

Disallow: /

User-Agent: Sqworm

Disallow: /

User-Agent: SSM Agent

Disallow: /

User-Agent: TAMU

Disallow: /

User-Agent: TheUsefulbot

Disallow: /

User-Agent: TurnitinBot

Disallow: /

User-Agent: Tutorial Crawler

Disallow: /

User-Agent: TutorGig

Disallow: /

User-Agent: WebCopier

Disallow: /

User-Agent: WebZIP

Disallow: /

User-Agent: ZipppBot

Disallow: /

User-Agent: Xenu

Disallow: /

User-Agent: Wotbox

Disallow: /

User-Agent: Wget

Disallow: /

User-Agent: NaverBot

Disallow: /

User-Agent: mozDex

Disallow: /

User-Agent: Sosospider

Disallow: /

User-Agent: Baidupider

Disallow: /

?

轉(zhuǎn)載于:https://www.cnblogs.com/zsqx5e/p/3996553.html

總結(jié)

以上是生活随笔為你收集整理的ROBOTS.TXT屏蔽笔记、代码、示例大全的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。