Kubernetes环境下如何运行Coherence缓存集群
生活随笔
收集整理的這篇文章主要介紹了
Kubernetes环境下如何运行Coherence缓存集群
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
Oracle官方出了一個如何在Docker環境下運行Coherence的技術文檔，大家可以參考：
https://github.com/oracle/docker-images/tree/master/OracleCoherence
但是對于一個熟悉Coherence的老司機來說，簡單搭建起來只是個初步方案，在客戶的環境總是各種特性和定制化配置，所以本文研究的也是如何將已經客戶化的Coherence架構構建在Kubernetes開源框架上。
背景架構說明
話不多說，找一個客戶的典型的Coherence架構
?
?架構說明：
- Coherence的節點有不同的角色，分成了存儲節點，代理節點，管理節點和客戶端
- 不同的角色需要配置不同的Cacheconfig文件
- 整個架構是屬于Coherence Extend的方式，客戶端通過tcp長連接的方式連入Coherence集群
- 存儲節點存儲數據，代理節點，管理節點和客戶端都不存儲數據，代理節點主要負責調度客戶端請求到Coherence集群中。
?
Kubernetes環境下架構
?
?區別在于:
- 客戶端WebLogic Server基本都是基于Replication Controller部署成一個個的Pod
- 后端的CacheServer角色的節點以及負責代理的節點都是部署成兩組不同的Replication Controller
- cacheserver中每個pod都通過flanneld綁定不同的ip,proxy server節點也是綁定不同的ip以及相同的端口9099
現在問題是proxy server的每個ip都是不固定的，這樣在weblogic端就需要指定后端動態變化的ip連入集群。初步想法是讓proxy pod綁定服務，然后通過servicename接入。
所以首先需要配置的是dns,作用是每個weblogic pod都能通過dns解析servicename,并轉換到proxy server的具體地址。
?
構建Coherence Proxy ?Images
其實coherence cacheserver和proxyserver可以定制一個image,然后通過不同的參數來切換不同的配置文件，但這里采用最簡單的辦法，針對不同角色構建不同的images.
在?/home/weblogic/docker/OracleCoherence/dockerfiles/12.2.1.0.0 目錄下加入一個文件proxy-cache-config.xml作為proxyserver啟動的配置參數。
[root@k8s-node-1 12.2.1.0.0]# cat proxy-cache-config.xml <?xml version="1.0"?> <!DOCTYPE cache-config SYSTEM "cache-config.dtd"> <cache-config><caching-scheme-mapping><cache-mapping><cache-name>*</cache-name><scheme-name>distributed-scheme</scheme-name></cache-mapping></caching-scheme-mapping><caching-schemes><!-- Distributed caching scheme. --><distributed-scheme><scheme-name>distributed-scheme</scheme-name><service-name>DistributedCache</service-name><thread-count>50</thread-count><backup-count>1</backup-count><backing-map-scheme><local-scheme><scheme-name>LocalSizeLimited</scheme-name></local-scheme> </backing-map-scheme><autostart>true</autostart><local-storage>false</local-storage></distributed-scheme><local-scheme><scheme-name>LocalSizeLimited</scheme-name><eviction-policy>LRU</eviction-policy><high-units>500</high-units><unit-calculator>BINARY</unit-calculator><unit-factor>1048576</unit-factor><expiry-delay>48h</expiry-delay></local-scheme><proxy-scheme> <service-name>ExtendTcpProxyService</service-name> <thread-count>5</thread-count> <acceptor-config> <tcp-acceptor> <local-address> <address>0.0.0.0</address> <port>9099</port> </local-address> </tcp-acceptor> </acceptor-config> <autostart>true</autostart> </proxy-scheme> </caching-schemes> </cache-config>注意這里的address,0.0.0.0意味著可以綁定任何生成的ip.
?
修改Dockerfile,最好建立一個新的Dockerfile.proxy
[root@k8s-node-1 12.2.1.0.0]# cat Dockerfile.proxy # LICENSE CDDL 1.0 + GPL 2.0 # # ORACLE DOCKERFILES PROJECT # -------------------------- # This is the Dockerfile for Coherence 12.2.1 Standalone Distribution # # REQUIRED BASE IMAGE TO BUILD THIS IMAGE # --------------------------------------- # This Dockerfile requires the base image oracle/serverjre:8 # (see https://github.com/oracle/docker-images/tree/master/OracleJava) # # REQUIRED FILES TO BUILD THIS IMAGE # ---------------------------------- # (1) fmw_12.2.1.0.0_coherence_Disk1_1of1.zip # # Download the Standalone installer from http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html # # HOW TO BUILD THIS IMAGE # ----------------------- # Put all downloaded files in the same directory as this Dockerfile # Run: # $ sh buildDockerImage.sh -s # # or if your Docker client requires root access you can run: # $ sudo sh buildDockerImage.sh -s ## Pull base image # --------------- FROM oracle/serverjre:8# Maintainer # ---------- MAINTAINER Jonathan Knight# Environment variables required for this build (do NOT change) ENV FMW_PKG=fmw_12.2.1.0.0_coherence_Disk1_1of1.zip \FMW_JAR=fmw_12.2.1.0.0_coherence.jar \ORACLE_HOME=/u01/oracle/oracle_home \PATH=$PATH:/usr/java/default/bin:/u01/oracle/oracle_home/oracle_common/common/bin \CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"ENV COHERENCE_HOME=$ORACLE_HOME/coherence# Copy files required to build this image COPY $FMW_PKG install.file oraInst.loc /u01/ COPY start.sh /start.sh COPY proxy-cache-config.xml $COHERENCE_HOME/conf/proxy-cache-config.xmlRUN useradd -b /u01 -m -s /bin/bash oracle && \echo oracle:oracle | chpasswd && \chmod +x /start.sh && \chmod a+xr /u01 && \chown -R oracle:oracle /u01USER oracle# Install and configure Oracle JDK # Setup required packages (unzip), filesystem, and oracle user # ------------------------------------------------------------ RUN cd /u01 && $JAVA_HOME/bin/jar xf /u01/$FMW_PKG && cd - && \$JAVA_HOME/bin/java -jar /u01/$FMW_JAR -silent -responseFile /u01/install.file -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation ORACLE_HOME=$ORACLE_HOME && \rm /u01/$FMW_JAR /u01/$FMW_PKG /u01/oraInst.loc /u01/install.fileENTRYPOINT ["/start.sh"]區別在于需要把剛才的文件copy到容器中去
然后修改start.sh
[root@k8s-node-1 12.2.1.0.0]# cat start.sh #!/usr/bin/env sh#!/bin/sh -e -x -utrap "echo TRAPed signal" HUP INT QUIT KILL TERMmain() {COMMAND=serverSCRIPT_NAME=$(basename "${0}")MAIN_CLASS="com.tangosol.net.DefaultCacheServer"case "${1}" inserver) COMMAND=${1}; shift ;;console) COMMAND=${1}; shift ;;queryplus) COMMAND=queryPlus; shift ;;help) COMMAND=${1}; shift ;;esaccase ${COMMAND} inserver) server ;;console) console ;;queryPlus) queryPlus ;;help) usage; exit ;;*) server ;;esac }# --------------------------------------------------------------------------- # Display the help text for this script # --------------------------------------------------------------------------- usage() {echo "Usage: ${SCRIPT_NAME} [type] [args]"echo ""echo "type: - the type of process to run, must be one of:"echo " server - runs a storage enabled DefaultCacheServer"echo " (server is the default if type is omitted)"echo " console - runs a storage disabled Coherence console"echo " query - runs a storage disabled QueryPlus session"echo " help - displays this usage text"echo ""echo "args: - any subsequent arguments are passed as program args to the main class"echo ""echo "Environment Variables: The following environment variables affect the script operation"echo ""echo "JAVA_OPTS - this environment variable adds Java options to the start command,"echo " for example memory and other system properties"echo ""echo "COH_WKA - Sets the WKA address to use to discover a Coherence cluster."echo ""echo "COH_EXTEND_PORT - If set the Extend Proxy Service will listen on this port instead"echo " of the default ephemeral port."echo ""echo "Any jar files added to the /lib folder will be pre-pended to the classpath."echo "The /conf folder is on the classpath so any files in this folder can be loaded by the process."echo "" }server() {PROPS=""CLASSPATH=""MAIN_CLASS="com.tangosol.net.DefaultCacheServer"start }console() {PROPS="-Dcoherence.localstorage=false"CLASSPATH=""MAIN_CLASS="com.tangosol.net.CacheFactory"start }queryPlus() {PROPS="-Dcoherence.localstorage=false"CLASSPATH="${COHERENCE_HOME}/lib/jline.jar"MAIN_CLASS="com.tangosol.coherence.dslquery.QueryPlus"start }start() {if [ "${COH_WKA}" != "" ]thenPROPS="${PROPS} -Dcoherence.wka=${COH_WKA}"fiif [ "${COH_EXTEND_PORT}" != "" ]thenPROPS="${PROPS} -Dcoherence.cacheconfig=extend-cache-config.xml -Dcoherence.extend.port=${COH_EXTEND_PORT}"fiCLASSPATH="/conf:/lib/*:${CLASSPATH}:${COHERENCE_HOME}/conf:${COHERENCE_HOME}/lib/coherence.jar"CMD="${JAVA_HOME}/bin/java -cp ${CLASSPATH} ${PROPS} -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=proxy-cache-config.xml ${JAVA_OPTS} ${MAIN_CLASS} ${COH_MAIN_ARGS}"echo "Starting Coherence ${COMMAND} using ${CMD}"exec ${CMD} }main "$@"主要是在最后的java -cp中的修改。修改相應的build文件
最后build image(-s是指用standardalong版本)
sh buildProxyServer.sh -v 12.2.1.0.0 -s?
構建Coherence CacheServer Images
按照類似方法，先新建一個storage-cache-server.xml
[root@k8s-node-1 12.2.1.0.0]# cat storage-cache-config.xml <?xml version="1.0"?> <!DOCTYPE cache-config SYSTEM "cache-config.dtd"> <cache-config><caching-scheme-mapping><cache-mapping><cache-name>*</cache-name><scheme-name>distributed-pof</scheme-name></cache-mapping></caching-scheme-mapping><caching-schemes><distributed-scheme><scheme-name>distributed-pof</scheme-name><service-name>DistributedCache</service-name><backing-map-scheme> <local-scheme/> </backing-map-scheme> <listener/><autostart>true</autostart><local-storage>true</local-storage></distributed-scheme></caching-schemes> </cache-config>然后建立一個Dockerfile.cacheserver
[root@k8s-node-1 12.2.1.0.0]# cat Dockerfile.cacheserver # LICENSE CDDL 1.0 + GPL 2.0 # # ORACLE DOCKERFILES PROJECT # -------------------------- # This is the Dockerfile for Coherence 12.2.1 Standalone Distribution # # REQUIRED BASE IMAGE TO BUILD THIS IMAGE # --------------------------------------- # This Dockerfile requires the base image oracle/serverjre:8 # (see https://github.com/oracle/docker-images/tree/master/OracleJava) # # REQUIRED FILES TO BUILD THIS IMAGE # ---------------------------------- # (1) fmw_12.2.1.0.0_coherence_Disk1_1of1.zip # # Download the Standalone installer from http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html # # HOW TO BUILD THIS IMAGE # ----------------------- # Put all downloaded files in the same directory as this Dockerfile # Run: # $ sh buildDockerImage.sh -s # # or if your Docker client requires root access you can run: # $ sudo sh buildDockerImage.sh -s ## Pull base image # --------------- FROM oracle/serverjre:8# Maintainer # ---------- MAINTAINER Jonathan Knight# Environment variables required for this build (do NOT change) ENV FMW_PKG=fmw_12.2.1.0.0_coherence_Disk1_1of1.zip \FMW_JAR=fmw_12.2.1.0.0_coherence.jar \ORACLE_HOME=/u01/oracle/oracle_home \PATH=$PATH:/usr/java/default/bin:/u01/oracle/oracle_home/oracle_common/common/bin \CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"ENV COHERENCE_HOME=$ORACLE_HOME/coherence# Copy files required to build this image COPY $FMW_PKG install.file oraInst.loc /u01/ COPY start.sh /start.sh COPY storage-cache-config.xml $COHERENCE_HOME/conf/storage-cache-config.xmlRUN useradd -b /u01 -m -s /bin/bash oracle && \echo oracle:oracle | chpasswd && \chmod +x /start.sh && \chmod a+xr /u01 && \chown -R oracle:oracle /u01USER oracle# Install and configure Oracle JDK # Setup required packages (unzip), filesystem, and oracle user # ------------------------------------------------------------ RUN cd /u01 && $JAVA_HOME/bin/jar xf /u01/$FMW_PKG && cd - && \$JAVA_HOME/bin/java -jar /u01/$FMW_JAR -silent -responseFile /u01/install.file -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation ORACLE_HOME=$ORACLE_HOME && \rm /u01/$FMW_JAR /u01/$FMW_PKG /u01/oraInst.loc /u01/install.fileENTRYPOINT ["/start.sh"]?
最后修改start.sh,關鍵語句是
CMD="${JAVA_HOME}/bin/java -cp ${CLASSPATH} ${PROPS} -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.cacheconfig=storage-cache-config.xml ${JAVA_OPTS} ${MAIN_CLASS} ${COH_MAIN_ARGS}"?
最后build image
sh buildCacheServer.sh -v 12.2.1.0.0 -s通過docker images看到
[root@k8s-node-1 12.2.1.0.0]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 1213-domain v2 326bf14bb29f About an hour ago 2.055 GB oracle/coherence 12.2.1.0.0-cacheserver 57a90e86e1d2 20 hours ago 625 MB oracle/coherence 12.2.1.0.0-proxy 238c85d61468 23 hours ago 625 MB?
在master節點創建一系列ReplicationController
coherence-proxy.yaml
[root@k8s-master ~]# cat coherence-proxy.yaml apiVersion: v1 kind: ReplicationController metadata:name: coherence-proxy spec:replicas: 2template:metadata:labels:coherencecluster: "proxy"version: "0.1"spec:containers:- name: coherenceproxyimage: oracle/coherence:12.2.1.0.0-proxyports:- containerPort: 9099 --- apiVersion: v1 kind: Service metadata:name: coherenceproxysvclabels:coherencecluster: proxy spec:type: NodePortports:- port: 9099protocol: TCPtargetPort: 9099nodePort: 30033selector:coherencecluster: proxy?
coherence-cacheserver.yaml
[root@k8s-master ~]# cat coherence-cacheserver.yaml apiVersion: v1 kind: ReplicationController metadata:name: coherence-cacheserver spec:replicas: 2template:metadata:labels:coherencecluster: "mycluster"version: "0.1"spec:containers:- name: coherencecacheserverimage: oracle/coherence:12.2.1.0.0-cacheserver?
kubectl create -f coherence-proxy.yamlkubectl create -f coherence-cacheserver.yaml然后看看pod是否啟動成功
[root@k8s-master ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE coherence-cacheserver-96kz7 1/1 Running 0 1h 192.168.33.4 k8s-node-1 coherence-cacheserver-z67ht 1/1 Running 0 1h 192.168.33.3 k8s-node-1 coherence-proxy-j7r0w 1/1 Running 0 1h 192.168.33.5 k8s-node-1 coherence-proxy-tg8n8 1/1 Running 0 1h 192.168.33.6 k8s-node-1?
登錄進去后確定coherence集群成員狀態，看member成員的個數基本確定已經都加入集群了。
MasterMemberSet(ThisMember=Member(Id=8, Timestamp=2017-06-16 10:45:04.244, Address=192.168.33.6:38184, MachineId=42359, Location=machine:coherence-proxy-tg8n8,process:1, Role=CoherenceServer)OldestMember=Member(Id=1, Timestamp=2017-06-16 10:24:16.941, Address=192.168.33.4:46114, MachineId=10698, Location=machine:coherence-cacheserver-96kz7,process:1, Role=CoherenceServer)ActualMemberSet=MemberSet(Size=4Member(Id=1, Timestamp=2017-06-16 10:24:16.941, Address=192.168.33.4:46114, MachineId=10698, Location=machine:coherence-cacheserver-96kz7,process:1, Role=CoherenceServer)Member(Id=2, Timestamp=2017-06-16 10:24:20.836, Address=192.168.33.3:44182, MachineId=23654, Location=machine:coherence-cacheserver-z67ht,process:1, Role=CoherenceServer)Member(Id=7, Timestamp=2017-06-16 10:45:02.144, Address=192.168.33.5:39932, MachineId=47892, Location=machine:coherence-proxy-j7r0w,process:1, Role=CoherenceServer)Member(Id=8, Timestamp=2017-06-16 10:45:04.244, Address=192.168.33.6:38184, MachineId=42359, Location=machine:coherence-proxy-tg8n8,process:1, Role=CoherenceServer))MemberId|ServiceJoined|MemberState1|2017-06-16 10:24:16.941|JOINED,2|2017-06-16 10:24:20.836|JOINED,7|2017-06-16 10:45:02.144|JOINED,8|2017-06-16 10:45:04.244|JOINEDRecycleMillis=1200000RecycleSet=MemberSet(Size=4Member(Id=3, Timestamp=2017-06-16 10:25:25.741, Address=192.168.33.6:37346, MachineId=50576)Member(Id=4, Timestamp=2017-06-16 10:25:25.74, Address=192.168.33.5:46743, MachineId=65007)Member(Id=5, Timestamp=2017-06-16 10:43:45.413, Address=192.168.33.5:37568, MachineId=44312)Member(Id=6, Timestamp=2017-06-16 10:43:45.378, Address=192.168.33.6:33362, MachineId=42635)))TcpRing{Connections=[7]} IpMonitor{Addresses=3, Timeout=15s}?
確定service狀態
[root@k8s-master ~]# kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE coherenceproxysvc 10.254.22.102 <nodes> 9099:30033/TCP 1h kubernetes 10.254.0.1 <none> 443/TCP 26dcoherenceproxysvc已經啟動。
?
開始Coherence客戶端WebLogic Pod的配置
因為需要修改setDomainEnv.sh文件，將客戶端的coherence配置文件寫入，所以轉入weblogic目錄
[root@k8s-node-1 1213-domain]# pwd /home/weblogic/docker/OracleWebLogic/samples/1213-domain新建一個proxy-client.xml
[root@k8s-node-1 1213-domain]# cat proxy-client.xml <?xml version="1.0"?> <!DOCTYPE cache-config SYSTEM "cache-config.dtd"> <cache-config><caching-scheme-mapping><cache-mapping><cache-name>*</cache-name><scheme-name>extend-dist</scheme-name></cache-mapping></caching-scheme-mapping><caching-schemes> <remote-cache-scheme><scheme-name>extend-dist</scheme-name><service-name>ExtendTcpCacheService</service-name><initiator-config><tcp-initiator><remote-addresses><socket-address><address>coherenceproxysvc</address><port>9099</port> </socket-address> </remote-addresses> <connect-timeout>10s</connect-timeout> </tcp-initiator> <outgoing-message-handler> <request-timeout>5s</request-timeout> </outgoing-message-handler> </initiator-config> </remote-cache-scheme> </caching-schemes> </cache-config>需要注意的是address要指到service的名稱，依靠dns去解析。
?
修改Dockerfile,核心是加入JAVA_OPTIONS和CLASSPATH.
FROM oracle/weblogic:12.1.3-generic# Maintainer # ---------- MAINTAINER Bruno Borges <bruno.borges@oracle.com># WLS Configuration # ------------------------------- ARG ADMIN_PASSWORD ARG PRODUCTION_MODEENV DOMAIN_NAME="base_domain" \DOMAIN_HOME="/u01/oracle/user_projects/domains/base_domain" \ADMIN_PORT="7001" \ADMIN_HOST="wlsadmin" \NM_PORT="5556" \MS_PORT="7002" \PRODUCTION_MODE="${PRODUCTION_MODE:-prod}" \JAVA_OPTIONS="-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=/u01/oracle/proxy-client.xml" \CLASSPATH="/u01/oracle/coherence.jar" \PATH=$PATH:/u01/oracle/oracle_common/common/bin:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin:/u01/oracle# Add files required to build this image USER oracle COPY container-scripts/* /u01/oracle/ COPY coherence.jar /u01/oracle/ COPY proxy-client.xml /u01/oracle/# Configuration of WLS Domain WORKDIR /u01/oracle RUN /u01/oracle/wlst /u01/oracle/create-wls-domain.py && \mkdir -p /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security && \echo "username=weblogic" > /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security/boot.properties && \ echo "password=$ADMIN_PASSWORD" >> /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security/boot.properties && \echo ". /u01/oracle/user_projects/domains/base_domain/bin/setDomainEnv.sh" >> /u01/oracle/.bashrc && \ echo "export PATH=$PATH:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin" >> /u01/oracle/.bashrc && \cp /u01/oracle/commEnv.sh /u01/oracle/wlserver/common/bin/commEnv.sh && \rm /u01/oracle/create-wls-domain.py /u01/oracle/jaxrs2-template.jar # Expose Node Manager default port, and also default http/https ports for admin console EXPOSE $NM_PORT $ADMIN_PORT $MS_PORTWORKDIR $DOMAIN_HOME # Define default command to start bash. CMD ["startWebLogic.sh"]然后build image
docker build -t 1213-domain:v2 --build-arg ADMIN_PASSWORD=welcome1 .?
構建weblogic pod作為一個客戶端
[root@k8s-master ~]# cat weblogic-pod.yaml apiVersion: v1 kind: Pod metadata:name: weblogic spec:containers:- name: weblogicimage: 1213-domain:v2ports:- containerPort: 7001?
啟動以后，通過log確定weblogic啟動時確實把我們客戶化的參數加入
[root@k8s-master ~]# kubectl logs weblogic . . JAVA Memory arguments: -Djava.security.egd=file:/dev/./urandom . CLASSPATH=/u01/oracle/wlserver/../oracle_common/modules/javax.persistence_2.1.jar:/u01/oracle/wlserver/../wlserver/modules/com.oracle.weblogic.jpa21support_1.0.0.0_2-1.jar:/usr/java/jdk1.8.0_101/lib/tools.jar:/u01/oracle/wlserver/server/lib/weblogic_sp.jar:/u01/oracle/wlserver/server/lib/weblogic.jar:/u01/oracle/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/u01/oracle/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar:/u01/oracle/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.1.0.0.jar:/u01/oracle/wlserver/common/derby/lib/derbyclient.jar:/u01/oracle/wlserver/common/derby/lib/derby.jar:/u01/oracle/wlserver/server/lib/xqrl.jar:/u01/oracle/coherence.jar . PATH=/u01/oracle/wlserver/server/bin:/u01/oracle/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/usr/java/jdk1.8.0_101/jre/bin:/usr/java/jdk1.8.0_101/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin:/u01/oracle/oracle_common/common/bin:/u01/oracle/oracle_common/common/bin:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin:/u01/oracle . *************************************************** * To start WebLogic Server, use a username and * * password assigned to an admin-level user. For * * server administration, use the WebLogic Server * * console at http://hostname:port/console * *************************************************** starting weblogic with Java version: java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Starting WLS with line: /usr/java/jdk1.8.0_101/bin/java -server -Djava.security.egd=file:/dev/./urandom -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/oracle/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=/u01/oracle/proxy-client.xml -Djava.endorsed.dirs=/usr/java/jdk1.8.0_101/jre/lib/endorsed:/u01/oracle/wlserver/../oracle_common/modules/endorsed -da -Dwls.home=/u01/oracle/wlserver/server -Dweblogic.home=/u01/oracle/wlserver/server -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server?
?
部署一個HelloWorld.war文件，核心的index.jsp代碼是:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <%@page import="java.util.*"%> <%@page import="com.tangosol.net.*"%> <%@ page contentType="text/html;charset=windows-1252"%><html><body> This is a Helloworld test</body> <h3><%String mysession;NamedCache cache;cache = CacheFactory.getCache("demoCache");cache.put("eric","eric.nie@oracle.com");%>Get Eric Email:<%=cache.get("eric").toString()%></h3></html>?
[root@k8s-master ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE coherence-cacheserver-96kz7 1/1 Running 0 1h 192.168.33.4 k8s-node-1 coherence-cacheserver-z67ht 1/1 Running 0 1h 192.168.33.3 k8s-node-1 coherence-proxy-j7r0w 1/1 Running 0 1h 192.168.33.5 k8s-node-1 coherence-proxy-tg8n8 1/1 Running 0 1h 192.168.33.6 k8s-node-1 weblogic 1/1 Running 0 1h 192.168.33.7 k8s-node-1?
部署后訪問
?
?查看后面的weblogic日志
<Jun 16, 2017 10:32:42 AM GMT> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.> <Jun 16, 2017 10:32:42 AM GMT> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.> <Jun 16, 2017 10:32:42 AM GMT> <Warning> <Server> <BEA-002611> <The hostname "localhost", maps to multiple IP addresses: 127.0.0.1, 0:0:0:0:0:0:0:1.> <Jun 16, 2017 10:32:42 AM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.> 2017-06-16 10:36:29.193/280.892 Oracle Coherence 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational configuration from "jar:file:/u01/oracle/coherence/lib/coherence.jar!/tangosol-coherence.xml" 2017-06-16 10:36:29.451/281.086 Oracle Coherence 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational overrides from "jar:file:/u01/oracle/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml" 2017-06-16 10:36:29.479/281.114 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified 2017-06-16 10:36:29.511/281.146 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "cache-factory-config.xml" is not specified 2017-06-16 10:36:29.525/281.159 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "cache-factory-builder-config.xml" is not specified 2017-06-16 10:36:29.526/281.161 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/custom-mbeans.xml" is not specifiedOracle Coherence Version 12.1.3.0.0 Build 52031Grid Edition: Development mode Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.2017-06-16 10:36:29.672/281.306 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded cache configuration from "file:/u01/oracle/proxy-client.xml"; this document does not refer to any schema definition and has not been validated. 2017-06-16 10:36:30.225/281.860 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Created cache factory com.tangosol.net.ExtensibleConfigurableCacheFactory 2017-06-16 10:36:30.507/282.142 Oracle Coherence GE 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connecting Socket to 10.254.203.94:9099 2017-06-16 10:36:30.534/282.169 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error connecting Socket to 10.254.203.94:9099: java.net.ConnectException: Connection refused 2017-06-16 10:46:11.056/862.693 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Restarting Service: ExtendTcpCacheService 2017-06-16 10:46:11.156/862.791 Oracle Coherence GE 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connecting Socket to 10.254.22.102:9099 2017-06-16 10:46:11.188/862.837 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connected Socket to 10.254.22.102:9099?
問題和定位
關于DNS解析是否正確，可以通過下面命令
iptables -L -v -n -t nat查看路由是否正確，主要是看coherenceproxysvc的路由是否到正確的pod和端口.
[root@k8s-node-1 1213-domain]# iptables -L -v -n -t nat Chain PREROUTING (policy ACCEPT 5 packets, 309 bytes)pkts bytes target prot opt in out source destination 30708 1898K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */3 172 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCALChain INPUT (policy ACCEPT 0 packets, 0 bytes)pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)pkts bytes target prot opt in out source destination 2057 126K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCALChain POSTROUTING (policy ACCEPT 4 packets, 240 bytes)pkts bytes target prot opt in out source destination 4100 283K MASQUERADE all -- * !docker0 192.168.33.0/24 0.0.0.0/0 28494 1716K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */2 277 RETURN all -- * * 192.168.122.0/24 224.0.0.0/24 0 0 RETURN all -- * * 192.168.122.0/24 255.255.255.255 0 0 MASQUERADE tcp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-655350 0 MASQUERADE udp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-655350 0 MASQUERADE all -- * * 192.168.122.0/24 !192.168.122.0/24 Chain DOCKER (2 references)pkts bytes target prot opt in out source destination 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0 Chain KUBE-MARK-DROP (0 references)pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000Chain KUBE-MARK-MASQ (6 references)pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000Chain KUBE-NODEPORTS (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp dpt:300330 0 KUBE-SVC-BQXHRGVXFCEH2BHH tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp dpt:30033Chain KUBE-POSTROUTING (1 references)pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000Chain KUBE-SEP-67FRRWLKQK2OD4HZ (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.2 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.33.2:53Chain KUBE-SEP-GIM2MHZZZBZJL55J (2 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.0.105 0.0.0.0/0 /* default/kubernetes:https */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: SET name: KUBE-SEP-GIM2MHZZZBZJL55J side: source mask: 255.255.255.255 tcp to:192.168.0.105:443Chain KUBE-SEP-IM4M52WKVEC4AZF3 (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.6 0.0.0.0/0 /* default/coherenceproxysvc: */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp to:192.168.33.6:9099Chain KUBE-SEP-LUF3R3GRCSK6KKRS (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.2 0.0.0.0/0 /* kube-system/kube-dns:dns */0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.33.2:53Chain KUBE-SEP-ZZECWQBQCJPODCBC (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.5 0.0.0.0/0 /* default/coherenceproxysvc: */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp to:192.168.33.5:9099Chain KUBE-SERVICES (2 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.254.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:4430 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.254.254.254 /* kube-system/kube-dns:dns cluster IP */ udp dpt:530 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.254.254.254 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:530 0 KUBE-SVC-BQXHRGVXFCEH2BHH tcp -- * * 0.0.0.0/0 10.254.22.102 /* default/coherenceproxysvc: cluster IP */ tcp dpt:90990 0 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCALChain KUBE-SVC-BQXHRGVXFCEH2BHH (2 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-ZZECWQBQCJPODCBC all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ statistic mode random probability 0.500000000000 0 KUBE-SEP-IM4M52WKVEC4AZF3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-67FRRWLKQK2OD4HZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-GIM2MHZZZBZJL55J all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-GIM2MHZZZBZJL55J side: source mask: 255.255.255.2550 0 KUBE-SEP-GIM2MHZZZBZJL55J all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-LUF3R3GRCSK6KKRS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */?
可優化之處：
- 構建image腳本沿用官方修改，很多不必要的地方可以刪除
- cacheserver和proxy的image可以采用同一個，通過不同的參數調用不同的命令啟動。
?
轉載于:https://www.cnblogs.com/ericnie/p/7029083.html
總結
以上是生活随笔為你收集整理的Kubernetes环境下如何运行Coherence缓存集群的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 硬盘坏道如何屏蔽?最全攻略都在这了！
- 下一篇： 使用httpclient下载 页面、图片