搭建ES搜索引擎,实时导入mysql数据进行查询
eslogstashikhead
一、安裝es和插件
1. 安裝環境
2. 安裝插件
3. 創建索引并使用二、安裝Logstash導入數據
1. 安裝Logstash
2. 安裝logstash-input-jdbc插件
3. 實時同步mysql數據三、整合java
環境: centos6 + elasticsearch-2.4.5 + mysql-5.1.37
插件:elasticsearch-head管理
elasticsearch-analysis-ik分詞器
Logstash
logstash-input-jdbc
一、安裝es和插件
1. 安裝環境
下載上傳解壓步驟跳過, https://www.elastic.co/cn/downloads/past-releases
編輯配置文件
vim ./config/elasticsearch.yml
單節點es配置:
cluster.name: lin-es # 集群名
node.name: node-1 # 節點名
network.host: 192.168.37.151 # 主機ip防止腦裂配置
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping_timeout: 120s
client.transport.ping_timeout: 60s
discovery.zen.ping.unicast.hosts: ["192.168.37.151","192.168.37.152","192.168.37.153"]創建新用戶,如:es 啟動es不能使用root賬戶
useradd es #創建用戶
passwd es #修改密碼
chown -R es:es elasticsearch-2.4.5 #授權目錄給es用戶注意: 安裝jdk路徑不要放在/root目錄下,會造成切換到es用戶時,找不到java環境,放在/opt或者/usr都行(坑)
./bin/elasticsearch #啟動
./bin/elasticsearch -d #后臺啟動啟動成功,訪問: http://192.168.37.151:9200
2. 安裝插件
elasticsearch-head: 管理es的web后臺
https://github.com/mobz/elasticsearch-head
https://github.com/mobz/elasticsearch-head/releases
elasticsearch-analysis-ik: 中文分詞器
https://github.com/medcl/elasticsearch-analysis-ik
https://github.com/medcl/elasticsearch-analysis-ik/releases
切換到es目錄
安裝head: bin/plugin install file:/usr/soft/elasticsearch-head-master.zip
安裝ik: bin/plugin install file:/usr/soft/elasticsearch-analysis-ik-1.10.5.zip
安裝成功,訪問: http://192.168.37.151:9200/_plugin/head/
head管理后臺
3. 創建索引并使用
創建索引
命令創建:curl -XPUT http://192.168.37.151:9200/ik
ES的動作是以http方法來決定的: 常用的http方法: GET/PUT/POST/DELETE
curl后跟的參數分別是：
-X 指定http請求的方法
-HEAD GET(查詢) POST(更新) PUT(新增) DELETE(刪除)
-d 指定要傳輸的數據
設置分詞器規則:
ik_max_word 和 ik_smart的區別是,前者可以更細粒度的劃分詞匯,后者在查詢效果上更占優勢.
curl -XPOST http://192.168.37.151:9200/ik/ikType/_mapping -d'{
"properties": {
"content": {
"type": "string",
"index":"analyzed",
"analyzer": "ik_max_word",
"search_analyzer": "ik_smart"
}
}
}'添加數據:
curl -XPOST http://192.168.37.151:9200/ik/ikType/1 -d'
{"content":"美國留給伊拉克的是個爛攤子嗎"}
'
curl -XPOST http://192.168.37.151:9200/ik/ikType/2 -d'
{"content":"公安部：各地校車將享最高路權"}
'
curl -XPOST http://192.168.37.151:9200/ik/ikType/3 -d'
{"content":"中韓漁警沖突調查：韓警平均每天扣1艘中國漁船"}
'
curl -XPOST http://192.168.37.151:9200/ik/ikType/4 -d'
{"content":"中國駐洛杉磯領事館遭亞裔男子槍擊 嫌犯已自首"}
'查詢數據:
curl -XGET http://192.168.37.151:9200/ik/ikType/_search?pretty -d'{
"query" : { "term" : { "content" : "中國" }}
}'分析數據:
curl -XPOST http://192.168.37.151:9200/ik/ikType/_analyze?pretty -d'{
"analyzer": "ik_max_word",
"text": "美國留給伊拉克的是個爛攤子嗎"
}'結果:
{
"tokens": [{
"token": "美國",
"start_offset": 0,
"end_offset": 2,
"type": "CN_WORD",
"position": 0
},
{
"token": "留給",
"start_offset": 2,
"end_offset": 4,
"type": "CN_WORD",
"position": 1
},
{
"token": "伊拉克",
"start_offset": 4,
"end_offset": 7,
"type": "CN_WORD",
"position": 2
},
{
"token": "伊",
"start_offset": 4,
"end_offset": 5,
"type": "CN_WORD",
"position": 3
},
{
"token": "拉",
"start_offset": 5,
"end_offset": 6,
"type": "CN_CHAR",
"position": 4
},
{
"token": "克",
"start_offset": 6,
"end_offset": 7,
"type": "CN_WORD",
"position": 5
},
{
"token": "個",
"start_offset": 9,
"end_offset": 10,
"type": "CN_CHAR",
"position": 6
},
{
"token": "爛攤子",
"start_offset": 10,
"end_offset": 13,
"type": "CN_WORD",
"position": 7
},
{
"token": "攤子",
"start_offset": 11,
"end_offset": 13,
"type": "CN_WORD",
"position": 8
},
{
"token": "攤",
"start_offset": 11,
"end_offset": 12,
"type": "CN_WORD",
"position": 9
},
{
"token": "子",
"start_offset": 12,
"end_offset": 13,
"type": "CN_CHAR",
"position": 10
},
{
"token": "嗎",
"start_offset": 13,
"end_offset": 14,
"type": "CN_CHAR",
"position": 11
}
]
}二、安裝Logstash導入數據
1. 安裝Logstash
官網: https://www.elastic.co/products/logstash
下載解壓執行安裝:
bin/logstash -e 'input { stdin {} } output { stdout {} }'
2. 安裝logstash-input-jdbc插件
插件使用ruby語言開發,所以要安裝ruby環境
yum install ruby -y
切換目錄并安裝,等待(_)
bin/logstash-plugin install logstash-input-jdbc
或者跳過檢查直接安裝
bin/logstash-plugin install --no-verify logstash-input-jdbc3. 實時同步mysql數據
提前準備mysql驅動jar包,創建 jdbc.conf 和 jdbc.sql 文件
jdbc.conf
input {
stdin {
}
jdbc {
# 數據庫地址 端口 數據庫名
jdbc_connection_string => "jdbc:mysql://localhost:3306/es"
# 數據庫用戶名
jdbc_user => "root"
# 數據庫密碼
jdbc_password => "123456"
# mysql java驅動地址
jdbc_driver_library => "/usr/soft/logstash-5.3.2/lib/mysql-connector-java-5.1.8.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
# sql 語句文件
statement_filepath => "/usr/soft/logstash-5.3.2/jdbc.sql"
# 定時字段 各字段含義（由左至右）分、時、天、月、年，全部為*默認含義為每分鐘都更新
schedule => "* * * * *"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => ["192.168.37.151:9200"]
# index名稱
index => "web"
# type名稱
document_type => "t_web"
# 文檔_id
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}jdbc.sql
SELECT
id,
name,
domain
FROM
web執行導入
bin/logstash -f jdbc.conf
數據導入結果
三、整合java
v2.4.x版本
https://www.cnblogs.com/yzlsthl/p/9096541.html
https://www.cnblogs.com/anxbb/p/9383221.html
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.4.5</version>
</dependency>package es.utli;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* Created by lyf on 2020/4/20.
* ES v2.5.4
*/
public class ESUtil {
static TransportClient client;
static {
Map<String, String> map = new HashMap<String, String>();
map.put("cluster.name", "lin-es");
Settings.Builder settings = Settings.builder().put(map);
try {
client = TransportClient.builder().settings(settings).build()
.addTransportAddress(
new InetSocketTransportAddress(InetAddress.getByName("es"),
Integer.parseInt("9300")));
} catch (UnknownHostException e) {
e.printStackTrace();
}
}
public static void info() {
List<DiscoveryNode> nodes = client.connectedNodes();
for (DiscoveryNode node : nodes) {
System.out.println(node.getHostAddress());
}
}
/**
* 查詢
* @param indexName 索引名稱
* @param typeName 索引類型
* @param filedName 列名
* @param value 查詢內容
* @param pageIndex
* @param pageSize
* @return
*/
public static List<Map<String, Object>> query(String indexName, String typeName,
String filedName, String value,
Integer pageIndex,
Integer pageSize) {
QueryBuilder qb = QueryBuilders.matchPhraseQuery(filedName, value);
SearchResponse response = client.prepareSearch(indexName)
.setTypes(typeName)
.setQuery(qb)
.setFrom(pageIndex)
.setSize(pageSize)
.execute()
.actionGet();
List<Map<String, Object>> result = new ArrayList<>();
for (SearchHit hit : response.getHits().getHits()) {
result.add(hit.getSource());
}
return result;
}
}https://www.jianshu.com/p/fbe11ac6d204
https://www.liangzl.com/get-article-detail-3366.html
https://esdoc.bbossgroups.com/
參考:
https://www.jianshu.com/p/40e33c84693d
https://blog.csdn.net/u013850277/article/details/88753348
總結
以上是生活随笔為你收集整理的搭建ES搜索引擎,实时导入mysql数据进行查询的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 《塔瑞斯世界》副业赚钱方法
- 下一篇： 《崩坏：星穹铁道》酒馆夜谭天环族特辑获取