步驟1：準(zhǔn)備好eclipse、eclipse svn插件、mysql準(zhǔn)備好，mysql使用utf-8編碼

步驟2：mysql建庫，建表：

CREATE?DATABASE?nutch?;?? ? ? ? ? ?? CREATE TABLE `webpage` ( `id` varchar(767) NOT NULL, `headers` blob, `text` mediumtext DEFAULT NULL, `status` int(11) DEFAULT NULL, `markers` blob, `parseStatus` blob, `modifiedTime` bigint(20) DEFAULT NULL, `score` float DEFAULT NULL, `typ` varchar(32) CHARACTER SET latin1 DEFAULT NULL, `baseUrl` varchar(767) DEFAULT NULL, `content` longblob, `title` varchar(2048) DEFAULT NULL, `reprUrl` varchar(767) DEFAULT NULL, `fetchInterval` int(11) DEFAULT NULL, `prevFetchTime` bigint(20) DEFAULT NULL, `inlinks` mediumblob, `prevSignature` blob, `outlinks` mediumblob, `fetchTime` bigint(20) DEFAULT NULL, `retriesSinceFetch` int(11) DEFAULT NULL, `protocolStatus` blob, `signature` blob, `metadata` blob, PRIMARY KEY (`id`) ) ENGINE=InnoDB ROW_FORMAT=COMPRESSED DEFAULT CHARSET=utf8mb4;

`id` varchar(767) NOT NULL 這個(gè)在我本機(jī)是不能成功的，只能最大設(shè)置為100 所以改為：`id` varchar(100) NOT NULL

步驟3：從?https://svn.apache.org/repos/asf/nutch/tags/release-2.1? 拉下代碼，在本地創(chuàng)建java project。本人因?yàn)樵囼?yàn)過很多次，所以在此取項(xiàng)目名稱為test。

步驟4：加src文件

在project explorer下右擊項(xiàng)目，選擇properties。進(jìn)入java build path ，在source選項(xiàng)卡，刪除src文件夾，選擇“Add Folder?”，在這里把conf,src/bin,src/java,src/test,src/testresources，以及src/plugin文件夾下各個(gè)插件的src和test也加入進(jìn)來。最終可以看到如下界面（test為項(xiàng)目名稱）：

在每個(gè)eclipse 項(xiàng)目文件夾下有 .classpath文件，打開?.classpath文件能看到：內(nèi)容基本是這樣的。 <classpathentry kind="src" path="conf"/> <classpathentry kind="src" path="src/java"/> <classpathentry kind="src" path="src/test"/> <classpathentry kind="src" path="src/plugin/protocol-file/src/test"/> <classpathentry kind="src" path="src/plugin/protocol-httpclient/src/test"/> <classpathentry kind="src" path="src/plugin/subcollection/src/test"/> <classpathentry kind="src" path="src/plugin/parse-html/src/test"/> <classpathentry kind="src" path="src/plugin/urlfilter-automaton/src/test"/> <classpathentry kind="src" path="src/plugin/parse-html/src/java"/> <classpathentry kind="src" path="src/plugin/parse-tika/src/test"/> <classpathentry kind="src" path="src/plugin/lib-http/src/test"/> <classpathentry kind="src" path="src/plugin/parse-tika/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-regex/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-domain/src/java"/> <classpathentry kind="src" path="src/plugin/scoring-link/src/java"/> <classpathentry kind="src" path="src/plugin/index-anchor/src/test"/> <classpathentry kind="src" path="src/plugin/protocol-http/src/java"/> <classpathentry kind="src" path="src/plugin/urlnormalizer-regex/src/test"/> <classpathentry kind="src" path="src/plugin/urlfilter-prefix/src/java"/> <classpathentry kind="src" path="src/plugin/scoring-opic/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-domain/src/test"/> <classpathentry kind="src" path="src/plugin/protocol-file/src/java"/> <classpathentry kind="src" path="src/plugin/urlnormalizer-regex/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-suffix/src/java"/> <classpathentry kind="src" path="src/plugin/language-identifier/src/java"/> <classpathentry kind="src" path="src/plugin/lib-regex-filter/src/test"/> <classpathentry kind="src" path="src/plugin/language-identifier/src/test"/> <classpathentry kind="src" path="src/plugin/subcollection/src/java"/> <classpathentry kind="src" path="src/plugin/urlnormalizer-basic/src/test"/> <classpathentry kind="src" path="src/plugin/index-basic/src/java"/> <classpathentry kind="src" path="src/plugin/urlnormalizer-pass/src/test"/> <classpathentry kind="src" path="src/plugin/creativecommons/src/java"/> <classpathentry kind="src" path="src/bin"/> <classpathentry kind="src" path="src/plugin/protocol-httpclient/src/java"/> <classpathentry kind="src" path="src/plugin/tld/src/java"/> <classpathentry kind="src" path="src/plugin/urlnormalizer-basic/src/java"/> <classpathentry kind="src" path="src/plugin/index-basic/src/test"/> <classpathentry kind="src" path="src/plugin/lib-http/src/java"/> <classpathentry kind="src" path="src/plugin/protocol-ftp/src/java"/> <classpathentry kind="src" path="src/plugin/index-anchor/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-validator/src/java"/> <classpathentry kind="src" path="src/plugin/index-more/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-suffix/src/test"/> <classpathentry kind="src" path="src/plugin/creativecommons/src/test"/> <classpathentry kind="src" path="src/plugin/microformats-reltag/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-regex/src/test"/> <classpathentry kind="src" path="src/plugin/lib-regex-filter/src/java"/> <classpathentry kind="src" path="src/plugin/index-more/src/test"/> <classpathentry kind="src" path="src/plugin/urlnormalizer-pass/src/java"/> <classpathentry kind="src" path="src/plugin/urlfilter-automaton/src/java"/> <classpathentry kind="src" path="src/testresources"/>

步驟5：加入lib包：

?? ? ? 切換到Libaries選項(xiàng)卡，“Add Library"->"IvyDE Managed Dependencies"->"Next",選擇“Project”，選擇ivy\ivy.xml文件。點(diǎn) Ok。eclipse會自動下載依賴的jar包。

在這個(gè)過程中或許會報(bào)錯(cuò)，看到錯(cuò)誤信息是因?yàn)?span style="line-height:1.5; font-size:10pt; font-family:'Courier New'; color:rgb(128,0,0)">org.restlet.jse包下載不到。解決方法是：ivy\ivy.xml中找到 <dependency?org="org.restlet.jse"?name="org.restlet"?rev="2.0.5"?conf="*->default"?/> ????<dependency?org="org.restlet.jse"?name="org.restlet.ext.jackson"?rev="2.0.5"? ??????conf="*->default"?/> 部分，注釋掉。在網(wǎng)上手動找到這兩個(gè)包，放在lib包下，加入到Libaries中。
接著加入plugin文件夾下各個(gè)插件的ivy.xml文件。手動一個(gè)一個(gè)加進(jìn)去。

步驟6：在"Order and Export"選項(xiàng)卡，將 conf ? ?top

步驟7：數(shù)據(jù)庫配置以及其他配置信息

打開/conf/gora.properties?，刪除文件中所有內(nèi)容，寫入mysql配置： ############################### #?MySQL?properties????????????# ############################### gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true gora.sqlstore.jdbc.user=root gora.sqlstore.jdbc.password=123456

在/conf/gora-sql-mapping.xml?修改 ?<primarykey?column="id"?length="240"/> 在?/conf/nutch-site.xml輸入： <property> <name>http.agent.name</name> <value>Your Nutch Spider</value> </property>
<property> <name>http.accept.language</name> <value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value> <description>Value of the “Accept-Language” request header field. This allows selecting non-English language as default one to retrieve. It is a useful setting for search engines build for certain national group. </description> </property>
<property> <name>parser.character.encoding.default</name> <value>utf-8</value> <description>The character encoding to fall back to when no other information is available</description> </property>
<property> ? <name>plugin.includes</name> ?<value>protocol-httpclient|protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</value> ?<description>Regular expression naming plugin directory names to ? include. ?Any plugin not matching this expression is excluded. ? In any case you need at least include the nutch-extensionpoints plugin. By ? default Nutch includes crawling just HTML and plain text via HTTP, ? and basic indexing and search plugins. In order to use HTTPS please enable? ? protocol-httpclient, but be aware of possible intermittent problems with the? ? underlying commons-httpclient library. ? </description> </property>
<property> <name>storage.data.store.class</name> <value>org.apache.gora.sql.store.SqlStore</value> <description>The Gora DataStore class for storing and retrieving data. Currently the following stores are available: …. </description> </property>
<property> ? <name>plugin.folders</name> ? <value>./src/plugin</value> ? <description>Directories where nutch plugins are located. ?Each ? element may be a relative or absolute path. ?If absolute, it is used ? as is. ?If relative, it is searched for on the classpath.</description> </property>?
在根目錄下的build.xml中找到如下代碼

<target?name="resolve-default"?depends="clean-lib,?init"?description="-->?resolve?and?retrieve?dependencies?with?ivy">??

??<ivy:resolve?file="${ivy.file}"?conf="default"?log="download-only"?/>??

??<ivy:retrieve?pattern="${build.lib.dir}/[artifact]-[revision].[ext]"?symlink="false"?log="quiet"?/>??

??<antcall?target="copy-libs"?/>??

?</target>??

將pattern="${build.lib.dir}/[artifact]-[revision].[ext]"替換為pattern="${build.lib.dir}/[artifact]-[type]-[revision].[ext]"

步驟8：配置抓取url

在test項(xiàng)目下創(chuàng)建文件夾urls，在urls下創(chuàng)建文件seeds.txt ，寫你要抓取的網(wǎng)站。我寫的是http://www.163.com。

步驟9：運(yùn)行org.apache.nutch.crawl.Crawler

打開Crawler文件，“Run As” -> “Run Configurations” ，在“Arguments”選項(xiàng)卡的“Program Arguments”，輸入 ? “urls -depth 3 -topN 5”，點(diǎn)"Run"。哈哈，報(bào)錯(cuò)了吧。報(bào)錯(cuò)信息類似于“?Failed?to?set?permissions?of?path:?\tmp\hadoop-Administrator\mapred\staging\Administrator1712398257\.??”的錯(cuò)誤。這是hadoop的一個(gè)問題。解決方法是，修改/hadoop-1.0.2/src/core/org/apache/hadoop/fs/FileUtil.java里面的checkReturnValue，注釋掉即可。當(dāng)然最簡單的辦法是在網(wǎng)上找一個(gè)修改過的包，替換一下FileUtil.class。 再次運(yùn)行，哈哈 執(zhí)行成功到此結(jié)束。
祝各位好運(yùn)吧。

參考資料：

http://cosmo1987.iteye.com/blog/1826971 http://nlp.solutions.asia/?p=180 http://wiki.apache.org/nutch/RunNutchInEclipse? 遇到的問題： 1?報(bào)?Exception?in?thread?"main"?java.lang.RuntimeException:?job?failed:?name=parse,?jobid=job_local_0004 根據(jù)在網(wǎng)上查到的問題可能很多首先 nutch-default.xml 中配置?<name>plugin.folders</name><value>./src/plugin</value> ??其次查找?hadoop.log文件。

總結(jié)

以上是生活随笔為你收集整理的nutch2.1在windows平台上使用eclipsedebug 存储在mysql的搭建过程的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。