lvs集群，停掉一台realServer，不能马上转发到另外一台上面

Submitted by xuyufei20540 on 周日, 2009-11-22 10:54

我的lvs服务器配置如下:
总共三台服务器
vip是：172.16.3.30
lvs的机器是：172.16.3.28
俩台realServer：分别是172.16.3.16和172.16.3.17
--------------------------------------------
在lvs上面执行了如下的脚本：

VIP=172.16.3.30
RIP2=172.16.3.17
RIP3=172.16.3.16
GW=172.16.1.254
# set the Virtual IP Address
/sbin/ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev eth0:0
#Clear IPVS table
/sbin/ipvsadm -C
#set LVS
/sbin/ipvsadm -A -t $VIP:8080 -s sh -p 5 -M 255.255.255.0
/sbin/ipvsadm -a -t $VIP:8080 -r $RIP2 -g -w 1
/sbin/ipvsadm -a -t $VIP:8080 -r $RIP3 -g -w 1
/sbin/ipvsadm --set 2 2 2
#Run LVS
/sbin/ipvsadm

--------------------------------------
在realServer：16和17上面分别执行
#!/bin/bash
VIP=172.16.3.30
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p

--------------------------------------
通过vip:172.16.3.30都可以正常访问web应用了，但是我从ipvsadm -Lcn里面查看到很多条记录
都是指向172.16.3.16的。
当我把172.16.3.16这台机器关掉之后，网页就无法用vip:172.16.3.30继续访问了，
用ipvsadm -Lcn查看到的记录并没有转发到172.16.3.17的记录。
但是等上一段时间后，直到ipvsadm -Lcn查看不到172.16.3.16的记录后，又可以继续用vip:172.16.3.30访问网页，也看到转发到了172.16.3.17

-------------------------------------
但是以上结果并不理想啊：停掉一台，应该把转向这台的记录都清除掉才对啊 ，我这里哪里配置有问题，还是只能等到超时去清除持久的记录.请有经验的朋友帮忙回答一下.
万分感谢.

Forums:

sery

周一, 2009-11-23 01:33

lvs无故障隔离以及失败切换框架，要实现这个功能，需要配合keepalived等工具实现。具体可参考我的文章http://blog.formyz.org/?p=46

祝你好运

xuyufei20540

周二, 2009-11-24 05:53

以下是我启动keepalived的log
-------------------------------------
Nov 24 11:48:35 backup26 Keepalived: Starting Keepalived v1.1.17 (11/24,2009)
Nov 24 11:48:35 backup26 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Nov 24 11:48:35 backup26 Keepalived_vrrp: Registering Kernel netlink reflector
Nov 24 11:48:35 backup26 Keepalived_vrrp: Registering Kernel netlink command channel
Nov 24 11:48:35 backup26 Keepalived_vrrp: Registering gratutious ARP shared channel
Nov 24 11:48:35 backup26 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Nov 24 11:48:35 backup26 Keepalived_vrrp: Configuration is using : 37227 Bytes
Nov 24 11:48:35 backup26 Keepalived: Starting VRRP child process, pid=3244
Nov 24 11:48:35 backup26 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Nov 24 11:48:40 backup26 Keepalived_vrrp: VRRP_Instance(VI_CACHE) Transition to MASTER STATE
Nov 24 11:48:45 backup26 Keepalived_vrrp: VRRP_Instance(VI_CACHE) Entering MASTER STATE
Nov 24 11:48:45 backup26 Keepalived_vrrp: VRRP_Instance(VI_CACHE) setting protocol VIPs.
Nov 24 11:48:45 backup26 Keepalived_vrrp: VRRP_Instance(VI_CACHE) Sending gratuitous ARPs on eth0 for 172.16.3.30
Nov 24 11:48:45 backup26 Keepalived_vrrp: VRRP_Group(VGM) Syncing instances to MASTER state
Nov 24 11:48:45 backup26 avahi-daemon[2691]: Registering new address record for 172.16.3.30 on eth0.
Nov 24 11:48:50 backup26 Keepalived_vrrp: VRRP_Instance(VI_CACHE) Sending gratuitous ARPs on eth0 for 172.16.3.30

keepalived.conf的配置如下
-----------------------------------------------------

global_defs {
router_id LVS_CNC_1
}

vrrp_sync_group VGM {
group {
VI_CACHE
}
}

################################################################
# vvrp_instance define #
################################################################

vrrp_instance VI_CACHE {
state MASTER
interface eth0
lvs_sync_daemon_inteface eth0
virtual_router_id 51
priority 180
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.3.30
}
}

##############################################################
# virtual machine setting #
##############################################################

# setting port 8080 forward
virtual_server 172.16.3.30 8080 {
delay_loop 6
lb_algo wlc
lb_kind DR
# persistence_timeout 20
protocol TCP

real_server 172.16.3.16 8080 {
weight 100
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 8080
}
}

real_server 172.16.3.17 8080 {
weight 100
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 8080
}
}

}

-----------------------------------------------------------
以上配置以后，还需要对lvs进行配置吗？
我如果不对lvs进行编写脚本，用ipvsadm去查看并没有记录内容
我配置之后是可以转发了，但是停掉其中正在转发的relServer，页面用ipvsadm 或者ipvsadm -Lcn查看，还是会有转向停掉的那台的记录，
不是应该停掉之后会被清除掉 的吗？哪里配置有问题？
请指点一下

xuyufei20540

周三, 2009-11-25 10:34

哦，原来是编译keepalived的时候出了问题，现在都ok了，编译keepalived的时候要把kernel 的head弄过去

Anonymous (没有被验证)

周六, 2011-05-21 07:48

virtual_server 192.168.1.101 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP

real_server 127.0.0.1 80 {
weight 1
TCP_CHECK{
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}

real_server 192.168.1.100 80 {
weight 1
TCP_CHECK{
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}

[root@localhost keepalived-1.2.1]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.101:80 rr persistent 50
-> 127.0.0.1:80 Local 1 0 0

May 21 23:20:15 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
May 21 23:20:15 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
May 21 23:32:43 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
May 21 23:32:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
May 21 23:32:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
May 21 23:32:44 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.101
May 21 23:32:44 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.1.100 added
May 21 23:32:44 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.100 added
May 21 23:32:44 localhost avahi-daemon[3051]: Registering new address record for 192.168.1.101 on eth0.
May 21 23:32:49 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.101

我先在配置文件中把127.0.0.1 放在前面，怎么访问都是第一台机器127.0.0。1
后来我在配置文件中把192.168.1.102放在前面，怎么访问都是192.168.1.102