場(chǎng)景：爬蟲(chóng)網(wǎng)頁(yè)列表上的鏈接，用于再次爬蟲(chóng)，主要功能是列表翻頁(yè)功能。

代碼參考：

package com;import com.gargoylesoftware.htmlunit.BrowserVersion; import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController; import com.gargoylesoftware.htmlunit.WebClient; import com.gargoylesoftware.htmlunit.html.DomElement; import com.gargoylesoftware.htmlunit.html.DomNodeList; import com.gargoylesoftware.htmlunit.html.HtmlDivision; import com.gargoylesoftware.htmlunit.html.HtmlElement; import com.gargoylesoftware.htmlunit.html.HtmlPage;public class BlogAutoClick {public static void main(String[] args){String sUrl="IP";//網(wǎng)址 //webclient設(shè)置 final WebClient webClient = new WebClient(BrowserVersion.CHROME); //創(chuàng)建一個(gè)webclient webClient.getOptions().setJavaScriptEnabled(true); // 啟動(dòng)JS webClient.getOptions().setUseInsecureSSL(true);//忽略ssl認(rèn)證 webClient.getOptions().setCssEnabled(false);//禁用Css，可避免自動(dòng)二次請(qǐng)求CSS進(jìn)行渲染 webClient.getOptions().setThrowExceptionOnScriptError(false);//運(yùn)行錯(cuò)誤時(shí)，不拋出異常 webClient.getOptions().setThrowExceptionOnFailingStatusCode(false); webClient.setAjaxController(new NicelyResynchronizingAjaxController());// 設(shè)置Ajax異步 //登錄 int i_clickCount=0;try { HtmlPage page = (HtmlPage) webClient.getPage(sUrl); webClient.waitForBackgroundJavaScript(10000);//等待1秒 boolean flag=true;while(flag){//本頁(yè)鏈接點(diǎn)擊HtmlDivision articlediv=(HtmlDivision)page.getElementById("article_list"); DomNodeList<HtmlElement> articleas=articlediv.getElementsByTagName("a"); for(HtmlElement a:articleas){HtmlPage llpage=(HtmlPage)a.click();DomNodeList<DomElement> h1name= llpage.getElementsByTagName("h1");String strname=h1name.get(0).asText(); i_clickCount++;System.out.print("article:"+strname+"is auto clicked at times:"+i_clickCount);}//翻頁(yè)HtmlDivision pagediv=(HtmlDivision)page.getElementById("papelist");DomNodeList<HtmlElement> pageas=pagediv.getElementsByTagName("a");boolean nextpage=false;for(HtmlElement a:pageas){if (a.asText().equals("下一頁(yè)")) {page=(HtmlPage)a.click();webClient.waitForBackgroundJavaScript(10000);//等待1秒 nextpage=true;//下一頁(yè)存在break;}}if(nextpage==false) flag=false;//沒(méi)有下一頁(yè)了，退出循環(huán)}}catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } } }

總結(jié)

以上是生活随笔為你收集整理的HtmlUnit爬取页面列表链接的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。