lvs调度异常RealServer出现大量VIP连接VIP的ESTABLISHED连接导致web站点拒绝服务

系统架构:
调度算法: DR模式
DR:Keepalived+Lvs IP:10.251.146.83 VIP:10.251.146.254
RealServer01:jboss4 IP:10.251.146.69 VIP:10.251.146.254
RealServer02:jboss4 IP:10.251.146.70 VIP:10.251.146.254

网站性质:jboss 集群使用session 粘帖的方式,网站为现在学习视频站点,每一个视频学习时间为40-50分钟,学习完成后会生成相应的试卷;
故障现象:网站平时很正常,但是运行一段时间后,在两台RealServer 也就是69/70,上面在1-2分钟时间内出现1.6万个ESTABLISHED 连接,这些连接源ip,目标ip都是10.251.146.254,具体见下图:
10.251.146.69:
tcp 0 0 10.251.146.254:44837 10.251.146.254:80 ESTABLISHED 31263/java
tcp 0 0 10.251.146.254:21501 10.251.146.254:80 ESTABLISHED 31263/java
tcp 0 0 10.251.146.254:6049 10.251.146.254:80 ESTABLISHED 31263/java
tcp 0 0 10.251.146.254:5517 10.251.146.254:80 ESTABLISHED 31263/java
tcp 0 0 10.251.146.254:80 10.251.146.254:8647 ESTABLISHED 31263/java
tcp 0 0 10.251.146.254:80 10.251.146.254:57387 ESTABLISHED 31263/java
tcp 1 0 10.251.146.254:25155 10.251.146.254:80 CLOSE_WAIT 31263/java
tcp 0 0 10.251.146.254:80 10.251.146.254:23079 ESTABLISHED 31263/java
tcp 0 0 10.251.146.254:80 10.251.146.254:39253 ESTABLISHED 31263/java
10.251.146.70:
tcp 0 0 10.251.146.254:42878 10.251.146.254:80 ESTABLISHED 15684/java
tcp 0 0 10.251.146.254:80 10.251.146.254:55489 ESTABLISHED 15684/java
tcp 0 0 10.251.146.254:80 10.251.146.254:1936 ESTABLISHED 15684/java
tcp 1 0 10.251.146.254:9713 10.251.146.254:80 CLOSE_WAIT 15684/java
tcp 0 0 10.251.146.254:80 10.251.146.254:57413 ESTABLISHED 15684/java
tcp 1 0 10.251.146.254:62270 10.251.146.254:80 CLOSE_WAIT 15684/java
tcp 647 0 10.251.146.254:80 10.251.146.254:26017 ESTABLISHED -
tcp 0 0 10.251.146.254:54823 10.251.146.254:80 ESTABLISHED 15684/java
tcp 0 0 10.251.146.254:13816 10.251.146.254:80 ESTABLISHED 15684/java
tcp 0 0 10.251.146.254:80 10.251.146.254:23712 ESTABLISHED 15684/java
tcp 0 0 10.251.146.254:80 10.251.146.254:21401 ESTABLISHED 15684/java
tcp 1 0 10.251.146.254:2145 10.251.146.254:80 CLOSE_WAIT 15684/java

Forums:

randomness