lvs调度异常RealServer出现大量VIP连接VIP的ESTABLISHED连接导致web站点拒绝服务
系统架构:
调度算法: DR模式
DR:Keepalived+Lvs IP:10.251.146.83 VIP:10.251.146.254
RealServer01:jboss4 IP:10.251.146.69 VIP:10.251.146.254
RealServer02:jboss4 IP:10.251.146.70 VIP:10.251.146.254
网站性质:jboss 集群使用session 粘帖的方式,网站为现在学习视频站点,每一个视频学习时间为40-50分钟,学习完成后会生成相应的试卷;
故障现象:网站平时很正常,但是运行一段时间后,在两台RealServer 也就是69/70,上面在1-2分钟时间内出现1.6万个ESTABLISHED 连接,这些连接源ip,目标ip都是10.251.146.254,具体见下图:
10.251.146.69:
tcp        0      0 10.251.146.254:44837        10.251.146.254:80           ESTABLISHED 31263/java
tcp        0      0 10.251.146.254:21501        10.251.146.254:80           ESTABLISHED 31263/java
tcp        0      0 10.251.146.254:6049         10.251.146.254:80           ESTABLISHED 31263/java
tcp        0      0 10.251.146.254:5517         10.251.146.254:80           ESTABLISHED 31263/java
tcp        0      0 10.251.146.254:80           10.251.146.254:8647         ESTABLISHED 31263/java
tcp        0      0 10.251.146.254:80           10.251.146.254:57387        ESTABLISHED 31263/java
tcp        1      0 10.251.146.254:25155        10.251.146.254:80           CLOSE_WAIT  31263/java
tcp        0      0 10.251.146.254:80           10.251.146.254:23079        ESTABLISHED 31263/java
tcp        0      0 10.251.146.254:80           10.251.146.254:39253        ESTABLISHED 31263/java
10.251.146.70:
tcp        0      0 10.251.146.254:42878        10.251.146.254:80           ESTABLISHED 15684/java
tcp        0      0 10.251.146.254:80           10.251.146.254:55489        ESTABLISHED 15684/java
tcp        0      0 10.251.146.254:80           10.251.146.254:1936         ESTABLISHED 15684/java
tcp        1      0 10.251.146.254:9713         10.251.146.254:80           CLOSE_WAIT  15684/java
tcp        0      0 10.251.146.254:80           10.251.146.254:57413        ESTABLISHED 15684/java
tcp        1      0 10.251.146.254:62270        10.251.146.254:80           CLOSE_WAIT  15684/java
tcp      647      0 10.251.146.254:80           10.251.146.254:26017        ESTABLISHED -
tcp        0      0 10.251.146.254:54823        10.251.146.254:80           ESTABLISHED 15684/java
tcp        0      0 10.251.146.254:13816        10.251.146.254:80           ESTABLISHED 15684/java
tcp        0      0 10.251.146.254:80           10.251.146.254:23712        ESTABLISHED 15684/java
tcp        0      0 10.251.146.254:80           10.251.146.254:21401        ESTABLISHED 15684/java
tcp        1      0 10.251.146.254:2145         10.251.146.254:80           CLOSE_WAIT  15684/java
| 附件 | 大小 | 
|---|---|
|  10.251.146.69建立链接截图 | 49.04 KB | 
|  10.251.146.70建立链接截图 | 52.43 KB | 
|  ipvsadm -lnc | 42.49 KB | 
 
      