DHT抓取程序開(kāi)源地址：https://github.com/h31h31/H31DHTDEMO

數(shù)據(jù)處理程序開(kāi)源地址:https://github.com/h31h31/H31DHTMgr

DHT系列文章:

1.[搜片神器] 之P2P中DHT網(wǎng)絡(luò)爬蟲(chóng)原理

2.[搜片神器]之DHT網(wǎng)絡(luò)爬蟲(chóng)的代碼實(shí)現(xiàn)方法

3.[搜片神器]之DHT網(wǎng)絡(luò)爬蟲(chóng)的C++程序初步開(kāi)源

--------------------------------------------------------------------------------------------------------------------

看懂此文章需要提前看明白上面的系列文章,還需要你有TCP網(wǎng)絡(luò)編程和bencode編碼方法基礎(chǔ)上,如果都看不明白,可以到娛樂(lè)區(qū)http://www.sosobta.com??去看看,休息下...

?

在介紹了這么多期文章后,最后介紹BT網(wǎng)絡(luò)里面一個(gè)比較重要種子下載協(xié)議,方便大家知道如何從DHT網(wǎng)絡(luò)直接下載種子的問(wèn)題.

先說(shuō)下我們目前下載電影等文件是如何下載的,比如我們有個(gè)BT種子,就可以去下載對(duì)應(yīng)的文件,但如果我們只有個(gè)文件名字,如何去找BT種子呢?

首先我們可以去通過(guò)搜索得到磁連接,然后就由此字符串去下載對(duì)應(yīng)的種子文件和電影等信息,但如果沒(méi)有網(wǎng)站讓你下載種子,我們又當(dāng)如何去搜索這個(gè)種子呢?

目前我們下載BT種子有兩種方式: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??

通過(guò)HTTP直接從WEB服務(wù)器上下載,這種直接方便,比如從迅雷服務(wù)器上下載種子,

再就是通過(guò)BT軟件從網(wǎng)絡(luò)里面去獲取BT網(wǎng)絡(luò)里面專門(mén)有個(gè)下載種子的協(xié)議文件,只能下載種子,然后種子下載好后就可以交給BT軟件來(lái)下載數(shù)據(jù)了.

如何從DHT網(wǎng)絡(luò)下載種子,必須先看兩個(gè)協(xié)議文章:

http://www.bittorrent.org/beps/bep_0009.html

http://www.bittorrent.org/beps/bep_0010.html

這里面有介紹,但還是需要說(shuō)明一下如何操作的流程方便大家更好的理解.

?我們的代碼流程必須還是基于?DHT抓取程序開(kāi)源地址：https://github.com/h31h31/H31DHTDEMO?之上,因?yàn)槭菑腄HT網(wǎng)絡(luò)里面獲取數(shù)據(jù),

需要我們?cè)诖酥喜僮骱罄m(xù)流程.

之前的DHT有SEARCH的相關(guān)代碼來(lái)搜索這個(gè)HASH對(duì)應(yīng)的哪些IP在提供下載.

/* This is how you trigger a search for a torrent hash. If port (the second argument) is non-zero, it also performs an announce.Since peers expire announced data after 30 minutes, it's a good idea to reannounce every 28 minutes or so. */if(searching) {//m_dht.dht_random_bytes((void*)hashList[2],20);if(m_soListen >= 0)m_dht.dht_search(hashList[2], 0, AF_INET, DHT_callback, this);if(s6 >= 0)m_dht.dht_search(hashList[2], 0, AF_INET6, DHT_callback, this);searching = 0;}

搜索到對(duì)方返回的IP信息和端口號(hào)后,大家可以分析dht.c里面的函數(shù)代碼dht_periodic(const void *buf, size_t buflen,const struct sockaddr *fromAddr, int fromlen,time_t *tosleep,dht_callback *callback, void *closure)函數(shù)里面的ANNOUNCE_PEER返回請(qǐng)求里面帶有對(duì)方表明自己此BT種子對(duì)應(yīng)的認(rèn)證碼peerid.

dht_periodic(const void *buf, size_t buflen,const struct sockaddr *fromAddr, int fromlen,time_t *tosleep,dht_callback *callback, void *closure) 函數(shù)里面的ANNOUNCE_PEERcase ANNOUNCE_PEER:_dout("Announce peer!From IP:%s:%d\n",inet_ntoa(tempip->sin_addr),tempip->sin_port);new_node(id, fromAddr, fromlen, 1);if(id_cmp(info_hash, zeroes) == 0) {_dout("Announce_peer with no info_hash.\n");send_error(fromAddr, fromlen, tid, tid_len,203, "Announce_peer with no info_hash");break;}if(!token_match(token, token_len, fromAddr)) {_dout("Incorrect token for announce_peer.\n");send_error(fromAddr, fromlen, tid, tid_len,203, "Announce_peer with wrong token");break;}if(port == 0) {_dout("Announce_peer with forbidden port %d.\n", port);send_error(fromAddr, fromlen, tid, tid_len,203, "Announce_peer with forbidden port number");break;}if(callback) {(*callback)(closure, DHT_EVENT_ANNOUNCE_PEER_VALUES, info_hash,(void *)fromAddr, port,id);//此ID就是peerid,}

知道了對(duì)應(yīng)的IP,端口號(hào),還有種子ID號(hào),就可以向?qū)Ψ桨l(fā)送請(qǐng)求了.

獲取HASH是通過(guò)UDP網(wǎng)絡(luò),但下載BT種子是通過(guò)TCP來(lái)處理,相當(dāng)于別人是TCP服務(wù)器,我們連接過(guò)去,直接下載對(duì)應(yīng)PEERID的種子就行了.

BT種子在DHT網(wǎng)絡(luò)下載流程 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

?先看http://www.bittorrent.org/beps/bep_0010.html協(xié)議介紹,我們必須先握手 ?

• handshake message

?

?此包構(gòu)造比較簡(jiǎn)單,按照格式進(jìn)行組裝就行了,然后發(fā)送出去,對(duì)方就會(huì)回應(yīng)自己是什么客戶端的軟件提供種子下載.

void CH31BTMgr::Encode_handshake() {//a byte with value 19 (the length of the string that follows);//the UTF-8 string "BitTorrent protocol" (which is the same as in ASCII);//eight reserved bytes used to mark extensions;//the 20 bytes of the torrent info hash;//the 20 bytes of the peer ID.char btname[256];memset(btname,0,sizeof(btname));sprintf(btname,"BitTorrent protocol");char msg[1280];memset(msg,0,sizeof(msg));msg[0]=19;memcpy(&msg[1],btname,19);char ext[8];memset(ext,0,sizeof(ext));ext[5]=0x10;memcpy(&msg[20],ext,8);memcpy(&msg[28],m_hash,20);memcpy(&msg[48],m_peer_id,20);int res1=Write(msg, 68);//TCP發(fā)送消息 }

在發(fā)送握手后,我們可以接著發(fā)送種子數(shù)據(jù)請(qǐng)求包,需要學(xué)習(xí)http://www.bittorrent.org/beps/bep_0009.html 里面的內(nèi)容:　　

extension header The metadata extension uses the extension protocol (specified in BEP 0010 ) to advertize its existence. It adds the "ut_metadata" entry to the "m" dictionary in the extension header hand-shake message. This identifies the message code used for this message. It also adds "metadata_size" to the handshake message (not the "m" dictionary) specifying an integer value of the number of bytes of the metadata.Example extension handshake message:{'m': {'ut_metadata', 3}, 'metadata_size': 31235} extension message The extension messages are bencoded. There are 3 different kinds of messages:0 request 1 data 2 reject The bencoded messages have a key "msg_type" which value is an integer corresponding to the type of message. They also have a key "piece", which indicates which part of the metadata this message refers to.In order to support future extensability, an unrecognized message ID MUST be ignored.

這就需要會(huì)bencode的相關(guān)代碼,這個(gè)大家可以網(wǎng)上搜索進(jìn)行編譯,如果實(shí)現(xiàn)搞不定,可以留下郵箱我將此類代碼發(fā)送給你,其實(shí)也是網(wǎng)上收集整理的.

void CH31BTMgr::Encode_Ext_handshake() {entry m;m["ut_metadata"] = 0;entry e;e["m"]=m;char msg[200];char* header = msg;char* p = &msg[6];int len = bencode(p, e);int total_size = 2 + len;namespace io = detail;io::write_uint32(total_size, header);io::write_uint8(20, header);io::write_uint8(0, header);int res1=Write(msg, len + 6); }

如果別人回應(yīng)的是2,那就直接退出吧,說(shuō)明別人拒絕了你.

如果回應(yīng)是1,則返回的是數(shù)據(jù)區(qū),每塊是16K大小,最后一包不是.

?

data The data message adds another entry to the dictionary, "total_size". This key has the same semantics as the "metadata_size" in the extension header. This is an integer.The metadata piece is appended to the bencoded dictionary, it is not a part of the dictionary, but it is a part of the message (the length prefix MUST include it).If the piece is the last piece of the metadata, it may be less than 16kiB. If it is not the last piece of the metadata, it MUST be 16kiB.Example:{'msg_type': 1, 'piece': 0, 'total_size': 3425} d8:msg_typei1e5:piecei0e10:total_sizei34256eexxxxxxxx... The x represents binary data (the metadata).

下面給出如何進(jìn)行提交我需要第幾包的數(shù)據(jù)代碼:

void CH31BTMgr::write_metadata_packet(int type, int piece) {ASSERT(type >= 0 && type <= 2);ASSERT(piece >= 0);entry e;e["msg_type"] = type;e["piece"] = piece;char const* metadata = 0;int metadata_piece_size = 0;if (type == 1){e["total_size"] = 14132;int offset = piece * 16 * 1024;//metadata = m_tp.metadata().begin + offset;metadata_piece_size = (std::min)(int(14132 - offset), 16 * 1024); }char msg[200];char* header = msg;char* p = &msg[6];int len = bencode(p, e);int total_size = 2 + len + metadata_piece_size;namespace io = detail;io::write_uint32(total_size, header);io::write_uint8(20, header);io::write_uint8(m_message_index, header);int res1=Write(msg, len + 6); }

在接收到一包請(qǐng)求后我們才可以繼續(xù)下一包的請(qǐng)求,下面給了我們?nèi)绾谓馕鲞@一包的問(wèn)題代碼:

// 處理一個(gè)完整的包數(shù)據(jù) bool CH31BTMgr::DeCodeFrameData(char * buffer,int buflen) {char * p = (char *)mhFindstr(buffer, buflen, "ut_metadatai", 12);if(p) {m_message_index=atoi(&p[12]);if(m_message_index==2){return false;}write_metadata_packet(0,0);char filename[256];memset(filename,0,sizeof(filename));sprintf(filename,"%s\\torrent.txt",m_workPath);DelFile(filename);} p = (char *)mhFindstr(buffer, buflen, "metadata_sizei", 14);if(p) {m_metadata_size=atoi(&p[14]);m_fileCnt=(int)(m_metadata_size/16384)+1;} p = (char *)mhFindstr(buffer, buflen, "msg_typei", 9);if(p) {int type1=atoi(&p[9]);if(type1==1){p = (char *)mhFindstr(buffer, buflen, "piecei", 6);if(p) {int piece=atoi(&p[6]);p = (char *)mhFindstr(buffer, buflen, "total_sizei", 11);if(p) {int total_size=atoi(&p[11]);p = (char *)mhFindstr(buffer, buflen, "ee", 2);if(p) {//保存數(shù)據(jù)FILE* pfile=NULL;char filename[256];memset(filename,0,sizeof(filename));sprintf(filename,"%s\\torrent.txt",m_workPath);char openmethod[5]="a";if(piece==0)sprintf(openmethod,"w");if((pfile=fopen(filename,openmethod))!=NULL){if((piece+1)*16*1024<total_size){fseek(pfile,(piece)*16*1024,SEEK_SET);fwrite(&p[2],1,16*1024,pfile);write_metadata_packet(0,piece+1);fclose(pfile);}else{fwrite(&p[2],1,total_size-(piece)*16*1024,pfile);fclose(pfile);ManageTorrentFileToRealFile(filename);}}}}}}else if(type1==2){return false;}} return true; }

void * mhFindstr(const void *haystack, size_t haystacklen,const void *needle, size_t needlelen)
{
const char *h =(const char *) haystack;
const char *n =(const char *) needle;
size_t i;

/* size_t is unsigned */
if(needlelen > haystacklen)
return NULL;

for(i = 0; i <= haystacklen - needlelen; i++) {
if(memcmp(h + i, n, needlelen) == 0)
return (void*)(h + i);
}
return NULL;
}

?

?

接下來(lái)說(shuō)下如何進(jìn)行快速調(diào)試的問(wèn)題: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

? ? ? ?第一次調(diào)試也很天真的等著DHT網(wǎng)絡(luò)上的數(shù)據(jù)過(guò)來(lái),需要等很久,而且調(diào)試總是發(fā)現(xiàn)別人不回應(yīng),要么就是拒絕,經(jīng)過(guò)一段時(shí)間后,

問(wèn)朋友總是不對(duì)問(wèn)題,結(jié)果是協(xié)議沒(méi)有構(gòu)造對(duì).下面就需要注意的地方總結(jié)下:

1.一定要接收到別的人PEERID后才能夠與別人交流,不然別人肯定不理你;

2.構(gòu)造協(xié)議調(diào)試不能夠在外網(wǎng)絡(luò)上調(diào)試,最好大家將mono-monotorrent源代碼下載回來(lái),調(diào)試分析下,本地開(kāi)啟服務(wù)器;

3.通過(guò)本地與mono-monotorrent進(jìn)行調(diào)試,你就可以分析出是哪里不對(duì)的問(wèn)題,是不是協(xié)議哪些封裝得不對(duì)的問(wèn)題.

4.通過(guò)DHT網(wǎng)絡(luò)下載回來(lái)的種子肯定是最新的,WEB下載的可能還沒(méi)有呢..

5.通過(guò)協(xié)議下載回來(lái)的種子好像沒(méi)有announce-list,不知道為什么不提供一些內(nèi)容,可能還有些什么關(guān)鍵地方?jīng)]有下載,分析mono-monotorrent代碼里面就是不提供下載,希望高手指點(diǎn).

6.TCPClient接收數(shù)據(jù)區(qū)需要開(kāi)到16K以上,這樣方便處理,當(dāng)然如果會(huì)前后拼接包就更好.

7.如果需要bencode相關(guān)的編碼C++代碼,可以在此留言或者給h31h31#163.com發(fā)郵件.

?

如果此文章看不太明白,請(qǐng)先看看之前的文章,分析調(diào)試下代碼,再來(lái)學(xué)習(xí)此文章可能就比較懂一些.

希望有了解的朋友更好的交流和進(jìn)步.在此留言學(xué)習(xí)討論.

?

希望大家多多推薦哦...大家的推薦才是下一篇介紹的動(dòng)力...

?

?

轉(zhuǎn)載于:https://www.cnblogs.com/miao31/p/3332819.html

總結(jié)

以上是生活随笔為你收集整理的[搜片神器]直接从DHT网络下载BT种子的方法的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。