ProjectDiscovery組織開源了很多自動化掃描的內(nèi)部工具和研究，它們都是基于Go語言編寫，并且在實際滲透中有極大的作用。我非常喜歡這個組織開源的軟件，它也是我學(xué)習(xí)Go語言的動力之一，所以計劃寫一個系列文章來研究下它們的代碼。

介紹

幾個特性：

• 基于syn/connect兩種模式掃描
• 多種輸入類型支持，包括HOST / IP / CIDR表示法。
• 自動處理多個子域之間的重復(fù)主機(jī)
• Stdin和stdout支持集成到工作流中
• 易于使用的輕量級資源
  

? naabu -host hackerone.com_____ ___ ___ _/ / __ __/ _ \/ _ \/ _ \/ _ \/ // / /_//_/\_,_/\_,_/_.__/\_,_/ v2.0.3projectdiscovery.io[WRN] Use with caution. You are responsible for your actions [WRN] Developers assume no liability and are not responsible for any misuse or damage. [INF] Running SYN scan with root privileges [INF] Found 4 ports on host hackerone.com (104.16.100.52) hackerone.com:80 hackerone.com:443 hackerone.com:8443 hackerone.com:8080

掃描方式

掃描相關(guān)的代碼在 v2/pkg/scan目錄

cdn check

顧名思義，跟蹤一下，發(fā)現(xiàn)cdn檢查調(diào)用的是github.com/projectdiscovery/cdncheck中的項目。

通過接口獲取一些CDN的ip段，判斷ip是否在這些ip段中

// scrapeCloudflare scrapes cloudflare firewall's CIDR ranges from their API func scrapeCloudflare(httpClient *http.Client) ([]string, error) {resp, err := httpClient.Get("https://www.cloudflare.com/ips-v4")if err != nil {return nil, err}defer resp.Body.Close()data, err := ioutil.ReadAll(resp.Body)if err != nil {return nil, err}body := string(data)cidrs := cidrRegex.FindAllString(body, -1)return cidrs, nil }// scrapeIncapsula scrapes incapsula firewall's CIDR ranges from their API func scrapeIncapsula(httpClient *http.Client) ([]string, error) {req, err := http.NewRequest(http.MethodPost, "https://my.incapsula.com/api/integration/v1/ips", strings.NewReader("resp_format=text"))if err != nil {return nil, err}req.Header.Set("Content-Type", "application/x-www-form-urlencoded")resp, err := httpClient.Do(req)if err != nil {return nil, err}defer resp.Body.Close()data, err := ioutil.ReadAll(resp.Body)if err != nil {return nil, err}body := string(data)cidrs := cidrRegex.FindAllString(body, -1)return cidrs, nil }// scrapeAkamai scrapes akamai firewall's CIDR ranges from ipinfo func scrapeAkamai(httpClient *http.Client) ([]string, error) {resp, err := httpClient.Get("https://ipinfo.io/AS12222")if err != nil {return nil, err}defer resp.Body.Close()data, err := ioutil.ReadAll(resp.Body)if err != nil {return nil, err}body := string(data)cidrs := cidrRegex.FindAllString(body, -1)return cidrs, nil }// scrapeSucuri scrapes sucuri firewall's CIDR ranges from ipinfo func scrapeSucuri(httpClient *http.Client) ([]string, error) {resp, err := httpClient.Get("https://ipinfo.io/AS30148")if err != nil {return nil, err}defer resp.Body.Close()data, err := ioutil.ReadAll(resp.Body)if err != nil {return nil, err}body := string(data)cidrs := cidrRegex.FindAllString(body, -1)return cidrs, nil }func scrapeProjectDiscovery(httpClient *http.Client) ([]string, error) {resp, err := httpClient.Get("https://cdn.projectdiscovery.io/cdn/cdn-ips")if err != nil {return nil, err}defer resp.Body.Close()data, err := ioutil.ReadAll(resp.Body)if err != nil {return nil, err}body := string(data)cidrs := cidrRegex.FindAllString(body, -1)return cidrs, nil }

connect掃描

naabu的connect掃描就是簡單的建立一個tcp連接

// ConnectVerify is used to verify if ports are accurate using a connect request func (s *Scanner) ConnectVerify(host string, ports map[int]struct{}) map[int]struct{} {for port := range ports {conn, err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", host, port), s.timeout)if err != nil {delete(ports, port)continue}gologger.Debugf("Validated active port %d on %s\n", port, host)conn.Close()}return ports }

syn掃描

syn掃描只能在unix操作系統(tǒng)上運行，如果是windows系統(tǒng)，會切換到connect掃描。

syn掃描的原理是只用發(fā)一個syn包，節(jié)省發(fā)包時間，而完整的tcp需要進(jìn)行三次握手。

獲取空閑端口

初始化時，獲取空閑端口，并監(jiān)聽這個端口

import github.com/phayes/freeportfunc NewScannerUnix(scanner *Scanner) error {rawPort, err := freeport.GetFreePort()if err != nil {return err}scanner.listenPort = rawPorttcpConn, err := net.ListenIP("ip4:tcp", &net.IPAddr{IP: net.ParseIP(fmt.Sprintf("0.0.0.0:%d", rawPort))})if err != nil {return err}scanner.tcpPacketlistener = tcpConnvar handlers Handlersscanner.handlers = handlersscanner.tcpChan = make(chan *PkgResult, chanSize)scanner.tcpPacketSend = make(chan *PkgSend, packetSendSize)return nil }

監(jiān)聽網(wǎng)卡

獲取網(wǎng)卡名稱

SetupHandlerUnix 監(jiān)聽網(wǎng)卡

const (maxRetries = 10sendDelayMsec = 10chanSize = 1000packetSendSize = 2500snaplen = 65536readtimeout = 1500 )func SetupHandlerUnix(s *Scanner, interfaceName string) error {inactive, err := pcap.NewInactiveHandle(interfaceName)if err != nil {return err}err = inactive.SetSnapLen(snaplen)if err != nil {return err}readTimeout := time.Duration(readtimeout) * time.Millisecondif err = inactive.SetTimeout(readTimeout); err != nil {s.CleanupHandlers()return err}err = inactive.SetImmediateMode(true)if err != nil {return err}handlers := s.handlers.(Handlers)handlers.Inactive = append(handlers.Inactive, inactive)handle, err := inactive.Activate()if err != nil {s.CleanupHandlers()return err}handlers.Active = append(handlers.Active, handle)// Strict BPF filter// + Packets coming from target ip// + Destination port equals to sender socket source porterr = handle.SetBPFFilter(fmt.Sprintf("tcp and dst port %d and tcp[13]=18", s.listenPort))if err != nil {s.CleanupHandlers()return err}s.handlers = handlersreturn nil }

從網(wǎng)卡中過濾數(shù)據(jù)包 tcp and dst port %d and tcp[13]=18

%d 即第一步獲取的空閑端口，tcp[13]=18 即tcp的第十三位偏移的值為18，即僅抓取TCP SYN標(biāo)記的數(shù)據(jù)包。

監(jiān)聽數(shù)據(jù)

通過pcap監(jiān)聽數(shù)據(jù)

func TCPReadWorkerPCAPUnix(s *Scanner) {defer s.CleanupHandlers()var wgread sync.WaitGrouphandlers := s.handlers.(Handlers)for _, handler := range handlers.Active {wgread.Add(1)go func(handler *pcap.Handle) {defer wgread.Done()var (eth layers.Ethernetip4 layers.IPv4tcp layers.TCP)// Interfaces with MAC (Physical + Virtualized)parserMac := gopacket.NewDecodingLayerParser(layers.LayerTypeEthernet, e, &ip4, &tcp)// Interfaces without MAC (TUN/TAP)parserNoMac := gopacket.NewDecodingLayerParser(layers.LayerTypeIPv4, &ip4, &tcp)var parsers []*gopacket.DecodingLayerParserparsers = append(parsers, parserMac, parserNoMac)decoded := []gopacket.LayerType{}for {data, _, err := handler.ReadPacketData()if err == io.EOF {break} else if err != nil {continue}for _, parser := range parsers {if err := parser.DecodeLayers(data, &decoded); err != nil {continue}for _, layerType := range decoded {if layerType == layers.LayerTypeTCP {if !s.IPRanger.Contains(ip4.SrcIP.String()) {gologger.Debugf("Discarding TCP packet from non target ip %s\n", ip4.SrcIP.String())continue}// We consider only incoming packetsif tcp.DstPort != layers.TCPPort(s.listenPort) {continue} else if tcp.SYN && tcp.ACK {s.tcpChan <- &PkgResult{ip: ip4.SrcIP.String(), port: int(tcp.SrcPort)}}}}}}}(handler)}wgread.Wait() }

如果dstport為我們監(jiān)聽的端口，并且標(biāo)志位是 syn+ack，就將端口和ip加入到結(jié)果中。

發(fā)送數(shù)據(jù)包

核心內(nèi)容是從之前監(jiān)聽的tcp發(fā)送。

// SendAsyncPkg sends a single packet to a port func (s *Scanner) SendAsyncPkg(ip string, port int, pkgFlag PkgFlag) {// Construct all the network layers we need.ip4 := layers.IPv4{SrcIP: s.SourceIP,DstIP: net.ParseIP(ip),Version: 4,TTL: 255,Protocol: layers.IPProtocolTCP,}tcpOption := layers.TCPOption{OptionType: layers.TCPOptionKindMSS,OptionLength: 4,OptionData: []byte{0x05, 0xB4},}tcp := layers.TCP{SrcPort: layers.TCPPort(s.listenPort),DstPort: layers.TCPPort(port),Window: 1024,Seq: s.tcpsequencer.Next(),Options: []layers.TCPOption{tcpOption},}if pkgFlag == SYN {tcp.SYN = true} else if pkgFlag == ACK {tcp.ACK = true}err := tcp.SetNetworkLayerForChecksum(&ip4)if err != nil {if s.debug {gologger.Debugf("Can not set network layer for %s:%d port: %s\n", ip, port, err)}} else {err = s.send(ip, s.tcpPacketlistener, &tcp)if err != nil {if s.debug {gologger.Debugf("Can not send packet to %s:%d port: %s\n", ip, port, err)}}} }// send sends the given layers as a single packet on the network. func (s *Scanner) send(destIP string, conn net.PacketConn, l ...gopacket.SerializableLayer) error {buf := gopacket.NewSerializeBuffer()if err := gopacket.SerializeLayers(buf, s.serializeOptions, l...); err != nil {return err}var (retries interr error)send:if retries >= maxRetries {return err}_, err = conn.WriteTo(buf.Bytes(), &net.IPAddr{IP: net.ParseIP(destIP)})if err != nil {retries++// introduce a small delay to allow the network interface to flush the queuetime.Sleep(time.Duration(sendDelayMsec) * time.Millisecond)goto send}return err }

其他

修改ulimit
大多數(shù)類UNIX操作系統(tǒng)（包括Linux和macOS）在每個進(jìn)程和每個用戶的基礎(chǔ)上提供了系統(tǒng)資源的限制和控制（如線程，文件和網(wǎng)絡(luò)連接）的方法。 這些“ulimits”阻止單個用戶使用太多系統(tǒng)資源。

import (_ "github.com/projectdiscovery/fdmax/autofdmax" )

修改ulimit,只針對unix系統(tǒng)

fdmax.go

// +build !windowspackage fdmaximport ("runtime""golang.org/x/sys/unix" )const (UnixMax uint64 = 999999OSXMax uint64 = 24576 )type Limits struct {Current uint64Max uint64 }func Get() (*Limits, error) {var rLimit unix.Rlimiterr := unix.Getrlimit(unix.RLIMIT_NOFILE, &rLimit)if err != nil {return nil, err}return &Limits{Current: uint64(rLimit.Cur), Max: uint64(rLimit.Max)}, nil }func Set(maxLimit uint64) error {var rLimit unix.RlimitrLimit.Max = maxLimitrLimit.Cur = maxLimit// https://github.com/golang/go/issues/30401if runtime.GOOS == "darwin" && rLimit.Cur > OSXMax {rLimit.Cur = OSXMax}return unix.Setrlimit(unix.RLIMIT_NOFILE, &rLimit) }

隨機(jī)IP PICK

import "github.com/projectdiscovery/ipranger"

ipranger 實現(xiàn)就是來自masscan的隨機(jī)化地址掃描算法

隨機(jī)化地址掃描

在讀取地址后，如果進(jìn)行順序掃描，偽代碼如下

for (i = 0; i < range; i++) {scan(i); }

但是考慮到有的網(wǎng)段可能對掃描進(jìn)行檢測從而封掉整個網(wǎng)段，順序掃描效率是較低的，所以需要將地址進(jìn)行隨機(jī)的打亂，用算法描述就是設(shè)計一個打亂數(shù)組的算法，Masscan是設(shè)計了一個加密算法，偽代碼如下

range = ip_count * port_count; for (i = 0; i < range; i++) {x = encrypt(i);ip = pick(addresses, x / port_count);port = pick(ports, x % port_count);scan(ip, port); }

隨機(jī)種子就是i的值，這種加密算法能夠建立一種一一對應(yīng)的映射關(guān)系，即在[1…range]的區(qū)間內(nèi)通過i來生成[1…range]內(nèi)不重復(fù)的隨機(jī)數(shù)。同時如果中斷了掃描，只需要記住i的值就能重新啟動，在分布式上也可以根據(jù)i來進(jìn)行。

如果對這個加密算法感興趣可以看 Ciphers with Arbitrary Finite Domains 這篇論文。

可緩存的hashmap

ipranger中使用了github.com/projectdiscovery/hmap/store/hybrid

看了下代碼，是一個帶緩存功能的hashmap，也帶有超時時間。

所有添加的目標(biāo)(ip)會加入到緩存中，讓我想到ksubdomain中也有實現(xiàn)類似的功能，不過是在內(nèi)存中進(jìn)行，導(dǎo)致目標(biāo)很多的時候內(nèi)存操作會有點問題。如果用這個庫應(yīng)該可以解決這個問題 。

總結(jié)

naabu的代碼架構(gòu)很清晰，一個文件完成一個功能，通過看文件名就知道這個實現(xiàn)了什么功能，所以看代碼的時候很輕松，naabu也模仿masscan中的部分代碼，將它go化，值得學(xué)習(xí)。

但是從代碼來看，naabu只是實現(xiàn)了在linux上的syn掃描，在Windows上會使用三次握手的tcp連接(基于pcap，可以實現(xiàn)在windows上組合tcp發(fā)包的，但naabu沒有實現(xiàn))，所以在windows上掃描速度和一般掃描器幾乎無差別。

naabu的目標(biāo)添加是先循環(huán)讀取目標(biāo)一遍，如果目標(biāo)cidr地址很大，會造成很多內(nèi)存占用(雖然也會有硬盤緩存)，如果邊讀取邊發(fā)送就沒有這種煩惱，但naabu不是這樣的。

關(guān)注私我獲取【網(wǎng)絡(luò)安全學(xué)習(xí)攻略】

《新程序員》：云原生和全面數(shù)字化實踐50位技術(shù)專家共同創(chuàng)作，文字、視頻、音頻交互閱讀

總結(jié)

以上是生活随笔為你收集整理的【安全工具】projectdiscover之naabu 端口扫描器源码学习的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。