在nutch中，默認情況下尊重robot.txt的配置，同時不提供配置項以忽略robot.txt。
以下是其中一個解釋。即作為apache的一個開源項目，必須遵循某些規定，同時由于開放了源代碼，可以簡單的通過修改源代碼來忽略robot.txt的限制。

From the point of view of research and crawling certain pieces of the web, and i strongly agree with you that it should be configurable. But because Nutch being an Apache project, i dismiss it (arguments available upon request). We should adhere to some ethics, it is bad enough that we can just DoS a server by setting some options to a high level. We publish source code, it leaves the option open to everyone to change it, and i think the current situation is balanced enough.
Patching it is simple, i think we should keep it like that :)

以下為修改源代碼的方法：【未驗證】
修改類org.apache.nutch.fetcher.FetcherReducer.java
將以下內容注釋掉：

if (!rules.isAllowed(fit.u.toString())) {// unblockfetchQueues.finishFetchItem(fit, true);if (LOG.isDebugEnabled()) {LOG.debug("Denied by robots.txt: " + fit.url);}output(fit, null, ProtocolStatusUtils.STATUS_ROBOTS_DENIED,CrawlStatus.STATUS_GONE);continue;}

總結

以上是生活随笔為你收集整理的Nutch关于robot.txt的处理的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。