求助:lvs 分发表连接建立过慢导致客户请求出现超时,服务器无法访问!

章博士及各位同仁:
公司使用LVS时遇到一个很急手的问题,lvs 分发表连接建立过慢导致客户请求出现超时,服务器无法访问!
表现:客户端请求失败,报连接超时,服务器无法访问。

详细信息如下:
Network structure:

test client -> firewall -> lvs server -> realserver
2.2.2.200 -> 2.2.2.6(VIP's public ip) -> 192.168.5.20(VIP) -> 192.168.5.33
firewall nat: 2.2.2.6-> 192.168.5.20
only one real server

lvs:
1.DR mode
2.through arptables implement lvs to realserver distribution
3. realserver's arptables:
[root@DEVICE01 ~]# arptables -L -n
Chain IN (policy ACCEPT)
target source-ip destination-ip source-hw destination-hw hlen op hrd pro
DROP 0.0.0.0/0 192.168.5.20 00/00 00/00 any 0000/0000 0000/0000 0000/0000
DROP 0.0.0.0/0 192.168.5.21 00/00 00/00 any 0000/0000 0000/0000 0000/0000

Chain OUT (policy ACCEPT)
target source-ip destination-ip source-hw destination-hw hlen op hrd pro
mangle 0.0.0.0/0 192.168.5.20 00/00 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-s 192.168.5.32
mangle 0.0.0.0/0 192.168.5.21 00/00 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-s 192.168.5.32
mangle 192.168.5.20 0.0.0.0/0 00/00 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-d 192.168.5.32
mangle 192.168.5.21 0.0.0.0/0 00/00 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-d 192.168.5.32

Chain FORWARD (policy ACCEPT)
target source-ip destination-ip source-hw destination-hw hlen op hrd pro

2.2。2.6(VIP)访问不了时:
1, LVS,REALSERVER能接到请求包。

2, LVS与REALSERVER的连接状态建立需要时间较长(几分钟),下面是有问题时,LVS上的连接情况。

3, 状态要由SYN_RECV变为ESTABLISH才能访问。但每个连接在SYN_RECV状态为60秒,到时将被关闭,客户端提示超时。

4, 一直刷新至到某次连接变为ESTABLISH,这时所有来自此服务器的访问就正常了。

[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:56 SYN_RECV 2.2.2.200:53817 192.168.5.20:80 192.168.5.33:80#开始连接
TCP 00:58 NONE 2.2.2.205:0 192.168.5.20:80 192.168.5.33:80
TCP 00:57 SYN_RECV 2.2.2.205:1542 192.168.5.20:80 192.168.5.33:80
TCP 00:03 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:58 SYN_RECV 2.2.2.200:53817 192.168.5.20:80 192.168.5.33:80
TCP 00:53 NONE 2.2.2.205:0 192.168.5.20:80 192.168.5.33:80
TCP 00:52 SYN_RECV 2.2.2.205:1542 192.168.5.20:80 192.168.5.33:80
TCP 00:59 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:52 SYN_RECV 2.2.2.200:53817 192.168.5.20:80 192.168.5.33:80
TCP 00:47 NONE 2.2.2.205:0 192.168.5.20:80 192.168.5.33:80
TCP 00:46 SYN_RECV 2.2.2.205:1542 192.168.5.20:80 192.168.5.33:80
TCP 00:53 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80

..............................

[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:05 SYN_RECV 2.2.2.200:53817 192.168.5.20:80 192.168.5.33:80
TCP 00:59 SYN_RECV 2.2.2.200:35216 192.168.5.20:80 192.168.5.33:80
TCP 00:00 NONE 2.2.2.205:0 192.168.5.20:80 192.168.5.33:80
TCP 00:50 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:01 SYN_RECV 2.2.2.200:53817 192.168.5.20:80 192.168.5.33:80
TCP 00:56 SYN_RECV 2.2.2.200:35216 192.168.5.20:80 192.168.5.33:80
TCP 00:47 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:00 SYN_RECV 2.2.2.200:53817 192.168.5.20:80 192.168.5.33:80#连接被关,客户端报连接超时
TCP 00:54 SYN_RECV 2.2.2.200:35216 192.168.5.20:80 192.168.5.33:80
TCP 00:45 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:53 SYN_RECV 2.2.2.200:35216 192.168.5.20:80 192.168.5.33:80
TCP 00:44 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:52 SYN_RECV 2.2.2.200:35216 192.168.5.20:80 192.168.5.33:80
TCP 00:43 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
............................................
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:31 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
TCP 00:21 SYN_RECV 2.2.2.200:35217 192.168.5.20:80 192.168.5.33:80
TCP 00:40 SYN_RECV 2.2.2.200:35218 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:30 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
TCP 00:20 SYN_RECV 2.2.2.200:35217 192.168.5.20:80 192.168.5.33:80
TCP 00:39 SYN_RECV 2.2.2.200:35218 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:30 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
TCP 00:20 SYN_RECV 2.2.2.200:35217 192.168.5.20:80 192.168.5.33:80
TCP 00:39 SYN_RECV 2.2.2.200:35218 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 00:29 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
TCP 00:19 SYN_RECV 2.2.2.200:35217 192.168.5.20:80 192.168.5.33:80
TCP 00:38 SYN_RECV 2.2.2.200:35218 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 14:52 ESTABLISHED 2.2.2.200:35219 192.168.5.20:80 192.168.5.33:80#连上了
TCP 00:52 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
TCP 00:16 SYN_RECV 2.2.2.200:35218 192.168.5.20:80 192.168.5.33:80
[root@LVS01 ~]# ipvsadm -Lnc | grep 192.168.5.33
TCP 14:58 ESTABLISHED 2.2.2.200:35219 192.168.5.20:80 192.168.5.33:80
TCP 00:38 NONE 2.2.2.200:0 192.168.5.20:80 192.168.5.33:80
TCP 00:02 SYN_RECV 2.2.2.200:35218 192.168.5.20:80 192.168.5.33:80

Forums:

randomness