當前位置：首頁 > 编程资源 > 综合教程 >内容正文

综合教程

搭建ES搜索引擎,实时导入mysql数据进行查询

發布時間：2024/8/5 综合教程 47 生活家

生活随笔收集整理的這篇文章主要介紹了搭建ES搜索引擎,实时导入mysql数据进行查询小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

eslogstashikhead

一、安裝es和插件

1. 安裝環境
2. 安裝插件
3. 創建索引并使用

二、安裝Logstash導入數據

1. 安裝Logstash
2. 安裝logstash-input-jdbc插件
3. 實時同步mysql數據

三、整合java

環境: centos6 + elasticsearch-2.4.5 + mysql-5.1.37
插件:

elasticsearch-head管理
elasticsearch-analysis-ik分詞器
Logstash
logstash-input-jdbc

一、安裝es和插件

1. 安裝環境

下載上傳解壓步驟跳過, https://www.elastic.co/cn/downloads/past-releases
編輯配置文件
vim ./config/elasticsearch.yml
單節點es配置:

cluster.name: lin-es # 集群名
node.name: node-1 # 節點名
network.host: 192.168.37.151 # 主機ip

防止腦裂配置

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping_timeout: 120s
client.transport.ping_timeout: 60s
discovery.zen.ping.unicast.hosts: ["192.168.37.151","192.168.37.152","192.168.37.153"]

創建新用戶,如:es 啟動es不能使用root賬戶

useradd es #創建用戶
passwd es #修改密碼
chown -R es:es elasticsearch-2.4.5 #授權目錄給es用戶

注意: 安裝jdk路徑不要放在/root目錄下,會造成切換到es用戶時,找不到java環境,放在/opt或者/usr都行(坑)

./bin/elasticsearch #啟動
./bin/elasticsearch -d #后臺啟動

啟動成功,訪問: http://192.168.37.151:9200

2. 安裝插件

elasticsearch-head: 管理es的web后臺
https://github.com/mobz/elasticsearch-head
https://github.com/mobz/elasticsearch-head/releases

elasticsearch-analysis-ik: 中文分詞器
https://github.com/medcl/elasticsearch-analysis-ik
https://github.com/medcl/elasticsearch-analysis-ik/releases

切換到es目錄
安裝head: bin/plugin install file:/usr/soft/elasticsearch-head-master.zip
安裝ik: bin/plugin install file:/usr/soft/elasticsearch-analysis-ik-1.10.5.zip

安裝成功,訪問: http://192.168.37.151:9200/_plugin/head/

head管理后臺

3. 創建索引并使用

創建索引

命令創建:curl -XPUT http://192.168.37.151:9200/ik

ES的動作是以http方法來決定的: 常用的http方法: GET/PUT/POST/DELETE
curl后跟的參數分別是：

-X 指定http請求的方法
-HEAD GET(查詢) POST(更新) PUT(新增) DELETE(刪除)
-d 指定要傳輸的數據

設置分詞器規則:
ik_max_word 和 ik_smart的區別是,前者可以更細粒度的劃分詞匯,后者在查詢效果上更占優勢.

curl -XPOST http://192.168.37.151:9200/ik/ikType/_mapping -d'{
        "properties": {
            "content": {
                "type": "string",
                "index":"analyzed",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            }
       }
}'

添加數據:

curl -XPOST http://192.168.37.151:9200/ik/ikType/1 -d'
{"content":"美國留給伊拉克的是個爛攤子嗎"}
'
curl -XPOST http://192.168.37.151:9200/ik/ikType/2 -d'
{"content":"公安部：各地校車將享最高路權"}
'
curl -XPOST http://192.168.37.151:9200/ik/ikType/3 -d'
{"content":"中韓漁警沖突調查：韓警平均每天扣1艘中國漁船"}
'
curl -XPOST http://192.168.37.151:9200/ik/ikType/4 -d'
{"content":"中國駐洛杉磯領事館遭亞裔男子槍擊 嫌犯已自首"}
'

查詢數據:

curl -XGET http://192.168.37.151:9200/ik/ikType/_search?pretty -d'{
    "query" : { "term" : { "content" : "中國" }}
}'

分析數據:

curl -XPOST http://192.168.37.151:9200/ik/ikType/_analyze?pretty -d'{
  "analyzer": "ik_max_word",
  "text": "美國留給伊拉克的是個爛攤子嗎"
}'

結果:

{
	"tokens": [{
			"token": "美國",
			"start_offset": 0,
			"end_offset": 2,
			"type": "CN_WORD",
			"position": 0
		},
		{
			"token": "留給",
			"start_offset": 2,
			"end_offset": 4,
			"type": "CN_WORD",
			"position": 1
		},
		{
			"token": "伊拉克",
			"start_offset": 4,
			"end_offset": 7,
			"type": "CN_WORD",
			"position": 2
		},
		{
			"token": "伊",
			"start_offset": 4,
			"end_offset": 5,
			"type": "CN_WORD",
			"position": 3
		},
		{
			"token": "拉",
			"start_offset": 5,
			"end_offset": 6,
			"type": "CN_CHAR",
			"position": 4
		},
		{
			"token": "克",
			"start_offset": 6,
			"end_offset": 7,
			"type": "CN_WORD",
			"position": 5
		},
		{
			"token": "個",
			"start_offset": 9,
			"end_offset": 10,
			"type": "CN_CHAR",
			"position": 6
		},
		{
			"token": "爛攤子",
			"start_offset": 10,
			"end_offset": 13,
			"type": "CN_WORD",
			"position": 7
		},
		{
			"token": "攤子",
			"start_offset": 11,
			"end_offset": 13,
			"type": "CN_WORD",
			"position": 8
		},
		{
			"token": "攤",
			"start_offset": 11,
			"end_offset": 12,
			"type": "CN_WORD",
			"position": 9
		},
		{
			"token": "子",
			"start_offset": 12,
			"end_offset": 13,
			"type": "CN_CHAR",
			"position": 10
		},
		{
			"token": "嗎",
			"start_offset": 13,
			"end_offset": 14,
			"type": "CN_CHAR",
			"position": 11
		}
	]
}

二、安裝Logstash導入數據

1. 安裝Logstash

官網: https://www.elastic.co/products/logstash
下載解壓執行安裝:
bin/logstash -e 'input { stdin {} } output { stdout {} }'

2. 安裝logstash-input-jdbc插件

插件使用ruby語言開發,所以要安裝ruby環境
yum install ruby -y
切換目錄并安裝,等待(^_)

bin/logstash-plugin install logstash-input-jdbc
或者跳過檢查直接安裝
bin/logstash-plugin install --no-verify logstash-input-jdbc

3. 實時同步mysql數據

提前準備mysql驅動jar包,創建 jdbc.conf 和 jdbc.sql 文件

jdbc.conf

input {
    stdin {
    }
    jdbc {
      # 數據庫地址  端口  數據庫名
      jdbc_connection_string => "jdbc:mysql://localhost:3306/es"
      # 數據庫用戶名
      jdbc_user => "root"
      # 數據庫密碼
      jdbc_password => "123456"
      # mysql java驅動地址 
      jdbc_driver_library => "/usr/soft/logstash-5.3.2/lib/mysql-connector-java-5.1.8.jar"
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "50000"
      # sql 語句文件
      statement_filepath => "/usr/soft/logstash-5.3.2/jdbc.sql"
      # 定時字段 各字段含義（由左至右）分、時、天、月、年，全部為*默認含義為每分鐘都更新
      schedule => "* * * * *"
    }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
}

output {
    elasticsearch {
        hosts => ["192.168.37.151:9200"]
        # index名稱
        index => "web"
        # type名稱
        document_type => "t_web"
        # 文檔_id
        document_id => "%{id}"
    }
    stdout {
        codec => json_lines
    }
}

jdbc.sql

SELECT
id,
name,
domain
FROM
web

執行導入
bin/logstash -f jdbc.conf

數據導入結果

三、整合java

v2.4.x版本
https://www.cnblogs.com/yzlsthl/p/9096541.html
https://www.cnblogs.com/anxbb/p/9383221.html

 <dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>2.4.5</version>
</dependency>

package es.utli;

import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * Created by lyf on 2020/4/20.
 * ES v2.5.4
 */
public class ESUtil {
    static TransportClient client;

    static {
        Map<String, String> map = new HashMap<String, String>();
        map.put("cluster.name", "lin-es");
        Settings.Builder settings = Settings.builder().put(map);
        try {
            client = TransportClient.builder().settings(settings).build()
                    .addTransportAddress(
                            new InetSocketTransportAddress(InetAddress.getByName("es"),
                            Integer.parseInt("9300")));
        } catch (UnknownHostException e) {
            e.printStackTrace();
        }
    }

    public static void info() {
        List<DiscoveryNode> nodes = client.connectedNodes();
        for (DiscoveryNode node : nodes) {
            System.out.println(node.getHostAddress());
        }
    }

    /**
     * 查詢
     * @param indexName 索引名稱
     * @param typeName  索引類型
     * @param filedName 列名
     * @param value     查詢內容
     * @param pageIndex
     * @param pageSize
     * @return
     */
    public static List<Map<String, Object>> query(String indexName, String typeName,
                                           String filedName, String value,
                                           Integer pageIndex,
                                           Integer pageSize) {

        QueryBuilder qb = QueryBuilders.matchPhraseQuery(filedName, value);
        SearchResponse response = client.prepareSearch(indexName)
                .setTypes(typeName)
                .setQuery(qb)
                .setFrom(pageIndex)
                .setSize(pageSize)
                .execute()
                .actionGet();
        List<Map<String, Object>> result = new ArrayList<>();
        for (SearchHit hit : response.getHits().getHits()) {
            result.add(hit.getSource());
        }
        return result;
    }
}

https://www.jianshu.com/p/fbe11ac6d204
https://www.liangzl.com/get-article-detail-3366.html
https://esdoc.bbossgroups.com/

參考:

https://www.jianshu.com/p/40e33c84693d
https://blog.csdn.net/u013850277/article/details/88753348

總結

以上是生活随笔為你收集整理的搭建ES搜索引擎,实时导入mysql数据进行查询的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。