Tag Archives: Networking

Configuring a SPAN or mirror port on Open vSwitch

This howto describes how to configure a mirror port on your Open vSwitch. The goal is to install a new guest to act as IDS/IPS system. This guest is configured with 2 virtual network interfaces. The first interface will have an IP address and will be used to manage the guest. The other interfaces will be connected to the mirror port on Open vSwitch. This means that it will see all mirrored traffic.

My setup

Host OS : Ubuntu Quantal Quetzal 12.04 with libvirtd
Networking : The virtual machines are all connected to a OpenvSwitch bridge and are using RFC 1918 ip addresses. Since I only have a single external IP, my host runs a firewall that NATs certain ports towards the virtual machines.

XML configuration of guest

This is a copy of the interface declarations in the config file.

<interface type='bridge'>
 <mac address='52:54:bb:bb:11:11'/>
 <source bridge='ovsbr0'/>
 <virtualport type='openvswitch'>
 </virtualport>
 <model type='virtio'/>
</interface>
<interface type='bridge'>
 <mac address='52:54:bb:bb:11:12'/>
 <source bridge='ovsbr0'/>
 <virtualport type='openvswitch'>
 </virtualport>
 <model type='virtio'/>
</interface>

Since the MAC address has to be unique, we can use that as an identifier to configure the mirror port on Open vSwitch.

Configuring the mirror port on Open vSwitch

This script should be named ‘qemu’ and should be placed in ‘/var/libvirt/hooks’.

#!/bin/bash
# Written by Thomas Elsen
# You can use this at your own risk.
#
# The following to variables should be set before using the script.
# MAC containts the mac address from the interface that will receive
# all mirrored traffic.
MAC="52:54:bb:bb:11:12"
#GUEST should point to the name of the guest
GUEST="ids"

if [ $1 = $GUEST ];
then
        if [ $2 = 'started' ];
        then
                IFACE=`ifconfig | grep $MAC | awk '{print $1;}'`
                ovs-vsctl clear bridge ovsbr0 mirrors
                ovs-vsctl -- --id=@m create mirror name=mirror0 -- add bridge ovsbr0 mirrors @m -- --id=@capt get Port $IFACE -- set mirror mirror0 output_port=@capt select_all=1
                exit 0
        fi
fi

echo "Nothing to do : $1 $2" | logger
exit 0

After installing the script, make sure to set the 2 variables to the right values and give it the right permissions.

# chmod 755 /etc/libvirt/hooks/qemu

Using the above script will make sure that the mirror port is created when the guest is started. To make sure that libvirtd will use this new script, we have to restart it.

# /etc/init.d/libvirt-bin restart

Next step

In the next article I’ll use this new guest to run snort. Snort is an Open Source IDS sensor.

IPv6 : ping: sendmsg: Network is down

SETUP

I have 2 linux machines which are directly connected by means of an OpenVPN tunnel using tap interfaces. This means that 1 interface of each machine sits in the same virtual layer 2 network.

Interface of machine A:

root@machineA ~ # ip -6 ad show dev ovsbr1
6: ovsbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 
    inet6 2001:470:7071::1/64 scope global 
       valid_lft forever preferred_lft forever

Interface of machine B:

root@machineB:~# ip -6 ad show tap0
10: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 100
    inet6 2001:470:7071::2/128 scope global 
       valid_lft forever preferred_lft forever

Both machines can ping each other.

root@machineA ~ # ping6 -c 1 2001:470:7071::2
PING 2001:470:7071::2(2001:470:7071::2) 56 data bytes
64 bytes from 2001:470:7071::2: icmp_seq=1 ttl=64 time=34.9 ms

--- 2001:470:7071::2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 34.949/34.949/34.949/0.000 ms

And the other way around:

root@machineB:~# ping6 -c 1 2001:470:7071::1
PING 2001:470:7071::1 (2001:470:7071::1): 56 data bytes
64 bytes from 2001:470:7071::1: seq=0 ttl=255 time=34.233 ms

--- 2001:470:7071::1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 34.233/34.233/34.233 ms

Also, just to prove that both machines are really pinging each other, I’ll print the mac addresses and neighbour caches.

root@machineA ~ # ip link show dev ovsbr1
6: ovsbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT 
    link/ether 62:b0:18:37:1d:47 brd ff:ff:ff:ff:ff:ff
root@machineA ~ # ip -6 neigh
2001:470:7071::2 dev ovsbr1 lladdr 26:d5:0d:eb:93:72 router REACHABLE
root@machineA ~ #

And same info for machine B:

root@machineB:~# ip link show dev tap0
10: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 100
    link/ether 26:d5:0d:eb:93:72 brd ff:ff:ff:ff:ff:ff
root@machineB:~# ip -6 neigh
2001:470:7071::1 dev tap0 lladdr 62:b0:18:37:1d:47 router REACHABLE
root@machineB:~#

FYI : neighbor cache in IPv6 is similar to arp cache in IPv4.

MachineA has a route for 2001:470:7071:200::/56 pointing towards machineB

root@machineA ~ # ip -6 route  | grep 200::
2001:470:7071:200::/56 via 2001:470:7071::2 dev ovsbr1  metric 1 
root@machineA ~ # ip -6 route get 2001:470:7071:200::5
2001:470:7071:200::5 from :: via 2001:470:7071::2 dev ovsbr1  src 2001:470:7071::1  metric 0 
    cache 
root@machineA ~ #

Some extra info from machine A:
– Ubuntu Quantal. Linux kernel 3.5.0.
– The bridge is an OpenvSwitch bridge.

Problem

On machineA, I can’t connect or ping to hosts behind machineB. This is what happens.

root@machineA ~ # ping6 -c 1 2001:470:7071:200::5
PING 2001:470:7071:200::51(2001:470:7071:200::5) 56 data bytes
ping: sendmsg: Network is down
^C
--- 2001:470:7071:200::5 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@machineA ~ #

I also checked with tcpdump, but nothing is send over the interface when I try to ping.

Solution

None yet. If you have an idea, let me know.
[update on 2013/02/26] Started working after a reboot. Too bad I didn’t find the real reason.

Switch your KVM from regular bridge to Open vSwitch

This howto focusses on the steps that you have to take to switch from the regular bridge to Open vSwitch.

 My SETUP

My server runs Debian 6.0.6 Squeeze. It has only 1 external IP address. That IP address is bound to eth0.

It currenlty has a br0 configured with an internal (RFC1918) ip address. All virtual machines are connected to that bridge. I’m using iptables and natting to make sure that all traffic on port 80 is forwarded to my webserver, port 25 to the mailserver etc.

Bridge configuration in /etc/network/interfaces

auto br0
iface br0 inet static
  address 172.16.11.1
  network 172.16.11.0
  netmask 255.255.255.0 
  broadcast 172.16.11.255
  bridge_stp on
  bridgefd 0 
  bridge_maxwait 0 
  pre-up brctl addbr br0
  post-down brctl delbr br0
  post-up /etc/network/firewallscript.sh

Building the packages

Since Debian doesn’t provide any packages for Open-vSwitch on Squeeze, I decided to create that package myself.

I’m going to use the latest version since I’m interested in the latest features as well. Let’s start by cloning the source tree.

 git clone git://openvswitch.org/openvswitch

Building the package itself. I’ve omitted the output.

cd openvswitch
./boot.sh
./configure
dpkg-buildpackage -b

This resulted in the following list of packages being created.

rivy@shell:~/src/openvswitch/src/openvswitch$ ls -l ../
total 13736
drwxr-xr-x 19 rivy rivy 4096 Nov 20 15:07 openvswitch
-rw-r--r-- 1 rivy rivy 272814 Nov 20 15:07 openvswitch-brcompat_1.9.90-1_amd64.deb
-rw-r--r-- 1 rivy rivy 664470 Nov 20 15:07 openvswitch-common_1.9.90-1_amd64.deb
-rw-r--r-- 1 rivy rivy 297856 Nov 20 15:07 openvswitch-controller_1.9.90-1_amd64.deb
-rw-r--r-- 1 rivy rivy 2328744 Nov 20 15:07 openvswitch-datapath-dkms_1.9.90-1_all.deb
-rw-r--r-- 1 rivy rivy 2395538 Nov 20 15:07 openvswitch-datapath-source_1.9.90-1_all.deb
-rw-r--r-- 1 rivy rivy 6181626 Nov 20 15:07 openvswitch-dbg_1.9.90-1_amd64.deb
-rw-r--r-- 1 rivy rivy 32656 Nov 20 15:07 openvswitch-ipsec_1.9.90-1_amd64.deb
-rw-r--r-- 1 rivy rivy 26310 Nov 20 15:07 openvswitch-pki_1.9.90-1_all.deb
-rw-r--r-- 1 rivy rivy 1610624 Nov 20 15:07 openvswitch-switch_1.9.90-1_amd64.deb
-rw-r--r-- 1 rivy rivy 45264 Nov 20 15:07 openvswitch-test_1.9.90-1_all.deb
-rw-r--r-- 1 rivy rivy 4902 Nov 20 15:07 openvswitch_1.9.90-1_amd64.changes
-rw-r--r-- 1 rivy rivy 48778 Nov 20 15:07 ovsdbmonitor_1.9.90-1_all.deb
-rw-r--r-- 1 rivy rivy 84260 Nov 20 15:07 python-openvswitch_1.9.90-1_all.deb
rivy@shell:~/src/openvswitch/src/openvswitch$

In case you don’t want to build the package. openvswitch_1.9.90-1_amd64.tgz.

Installing and configuring

Now install those packages on the server that has KVM running.

dpkg -i openvswitch-*

Make sure all dependencies are met. If not, run ‘apt-get -f install’ and rerun the installation.

Now, make sure we provide compatibility with the Linux bridge. Uncomment and change ‘no’ to ‘yes’ in /etc/default/openvswitch-switch

BRCOMPAT=yes

Make sure the module is built and will be loadable at boot time.

module-assistant auto-install openvswitch-datapath

Adapt the bridge settings in /etc/network/interfaces

#auto br0
iface br0 inet static
  address 172.16.11.1
  network 172.16.11.0
  netmask 255.255.255.0 
  broadcast 172.16.11.255
  bridge_stp on
  bridgefd 0 
  bridge_maxwait 0 
#  pre-up ovs-vsctl add-br br0 ( not needed as the switch config is kept in DB)
#  post-down ovs-vsctl del-br br0
  post-up /etc/network/firewallscript.sh

Replace the existing ifup scripts with scripts that make use of the new Open vSwitch. Therefor we keep a copy of the old files and create 2 new files.

cd /etc/kvm
mv kvm-ifdown kvm-ifdown-original
mv kvm-ifup kvm-ifup-original

Contents of new  /etc/kvm/kvm-ifup

#!/bin/sh
switch='br0'
/sbin/ifconfig $1 0.0.0.0 up
ovs-vsctl --if-exists del-port ${switch} $1
ovs-vsctl add-port ${switch} $1

Contents of new /etc/kvm/kvm-ifdown

#!/bin/sh
switch='br0'
/sbin/ifconfig $1 0.0.0.0 up
ovs-vsctl del-port ${switch} $1

Make sure that the new module is loaded before ‘bridge’. I did this by adding ‘openvswitch’ to ‘/etc/modules’.

echo 'openvswitch' >> /etc/modules

Next step shutdown all guests and remove the old bridge. When the bridge is removed, you can unload the old ‘bridge’ module and load the new ‘openvswitch’ module.

virsh list
virsh shutdown <all your guests>
ifconfig br0 down
rmmod bridge
modprobe openvswitch
service openvswitch-controller start
service openvswitch-switch start
service networking restart
service libvirt-bin restart

If all goes well, you guests should be using the new Open vSwitch.

To check if this is correct, issue the following command. You should have a vnetX interface for each guest.

# ovs-vsctl show
c9900989-6f68-4da6-b739-23b106efcab5
 Bridge "br0"
   Port "vnet6"
     Interface "vnet6"
   Port "vnet5"
     Interface "vnet5"
   Port "vnet4"
     Interface "vnet4"
   Port "vnet3"
     Interface "vnet3"
   Port "vnet0"
     Interface "vnet0"
   Port "vnet1"
     Interface "vnet1"
   Port "br0"
     Interface "br0"
       type: internal
   Port "vnet2"
     Interface "vnet2"
 ovs_version: "1.9.90"