Network topology used in this howto

There are many ways of using Quantum. In fact, as many as there are network topologies. But we can't cover all of them. It is therefore left as an exercise to the reader to adapt this howto to his own need.

In this howto, we will assume that you have to your disposal, a bunch of IP addresses available on the 1.2.3.0/24 network. The IP address 1.2.3.1 will be the default gateway provided by your network provider, who granted you the use of a block of IP starting at 1.2.3.5 and ending at 1.2.3.9. Of course, in real life, you may need more IP addresses, but this is enough for this howto. The address 1.2.3.9 will be the "main" IP address of your server: it will be assumed that your server will be installed with a fresh Debian installation using this IP address.

Also, in this howto, we will setup a single router per tenant (if you don't know already, a "tenant" in OpenStack terms, can be thought as a customer). The customer router will be setup with the public IP address 1.2.3.5 and will route traffic for a LAN on 10.0.0.0/24, with 10.0.0.1 as default gateway for this LAN.

This small below drawing will help understanding better. The green network is the public network, while the blue one is the virtual network which Quantum creates.

quantum_network_topology.png

Setting-up quantum-plugin-openvswitch-agent

On your compute node, make sure that quantum-plugin-openvswitch-agent (this howto covers only this case, which doesn't need specific hardware). Make sure that Open vSwitch is installed and running, and that the local_ip directive in /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini is set to the management IP address of your compute node (it is normally set automatically by the debconf and postinst scripts). Then make sure that the daemon is up and running (ps axuf, then check the logs in /var/log/quantum). Once this is done, create the br-int bridge using Open vSwitch:

# ovs-vsctl add-br br-int

Creating the tenant LAN

Using python-quantumclient, simply issue the following commands on the shell, which will create a network called "testnetwork" with 10.0.0.0/24 attached to it as subnet:

# quantum net-create testnetwork
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 710620b7-63be-4a04-9168-35e19d7ca995 |
| name                      | testnetwork                          |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 1                                    |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 5f3fe6621a884758b37ca30506c161ff     |
+---------------------------+--------------------------------------+

# quantum subnet-create -- testnetwork 10.0.0.0/24
Created a new subnet:
+------------------+--------------------------------------------+
| Field            | Value                                      |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.0.0.2", "end": "10.0.0.254"} |
| cidr             | 10.0.0.0/24                                |
| dns_nameservers  |                                            |
| enable_dhcp      | True                                       |
| gateway_ip       | 10.0.0.1                                   |
| host_routes      |                                            |
| id               | b5cd1984-4a45-4233-90b1-19eb4f7a6e32       |
| ip_version       | 4                                          |
| name             |                                            |
| network_id       | 710620b7-63be-4a04-9168-35e19d7ca995       |
| tenant_id        | 5f3fe6621a884758b37ca30506c161ff           |
+------------------+--------------------------------------------+

Take a note of the ID of the net and subnet which you just created (eg: 710620b7-63be-4a04-9168-35e19d7ca995 for the network, and b5cd1984-4a45-4233-90b1-19eb4f7a6e32 for the subnet in this example).

If you wish, you can also set the DNS server for that subnet (in this example, sets the famous 8.8.8.8 Google public DNS resolver):

quantum subnet-update b5cd1984-4a45-4233-90b1-19eb4f7a6e32 --dns_nameservers list=true 8.8.8.8

Creating the tenant router

Now, we create a tenant router called "testrouter":

# quantum router-create testrouter
Created a new router:
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| external_gateway_info |                                      |
| id                    | 0d259ad9-7727-44f2-9d86-a8d682073d87 |
| name                  | testrouter                           |
| status                | ACTIVE                               |
| tenant_id             | 5f3fe6621a884758b37ca30506c161ff     |
+-----------------------+--------------------------------------+

Note the ID of the router (0d259ad9-7727-44f2-9d86-a8d682073d87 in this example). Then we attach to it a network interface which will be on the tenant LAN. This network interface will be assigned automatically the IP address of the default gateway for the tenant LAN (eg: 10.0.0.1/24). The first parameter is the ID of the router we just created, and the 2nd one is the ID of tenant LAN subnet:

# quantum router-interface-add 0d259ad9-7727-44f2-9d86-a8d682073d87 b5cd1984-4a45-4233-90b1-19eb4f7a6e32
Added interface to router 0d259ad9-7727-44f2-9d86-a8d682073d87

Mapping your public IP address network in Quantum

Low level setup of the node running quantum-server

Now is the time to man 1.2.3.0/24 in Quantum. Before doing it, we need to configure the machine running openstack-proxy-node. What is needed to be done, is to create a "br-ex" bridge, attached to your eth0 interface, which will connect your quantum-server and all of your virtual machines to the public IP network.

WARNING: this operation is potentially dangerous! If you do a mistake with the bridging, it is possible that you will loose access to your server. So make sure you can recover from mistakes, either by having physical access to your server, or by having a KVM over IP access. When you will run "ovs-vsctl add-port br-ex eth0" (see below), instantly, you will loose any kind of connectivity until the rest of the commands are run.

So, we prepare the below script:

set -x

ovs-vsctl add-br br-ex
ovs-vsctl add-port br-ex eth0
ifconfig br-ex 1.2.3.9 netmask 255.255.255.0 up
ifconfig eth0 0.0.0.0 up
route add default gw 1.2.3.1

Then run GNU screen (or tmux, or anything like that...) to make sure that this script will run until it is finished, even if you loose connectivity with ssh.

Creating the public IP network in Quantum

If everything works as expected, then you can create the ext_net network as per below:

# quantum net-create ext_net -- --router:external=True
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | cd5dea18-1d25-492d-88b1-c439a0a93dad |
| name                      | ext_net                              |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 2                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 5f3fe6621a884758b37ca30506c161ff     |
+---------------------------+--------------------------------------+

and attach a subnet to it:

# quantum subnet-create --allocation-pool start=1.2.3.5,end=1.2.3.8 ext_net 1.2.3.0/24 -- --enable_dhcp=False
Created a new subnet:
+------------------+----------------------------------------+
| Field            | Value                                  |
+------------------+----------------------------------------+
| allocation_pools | {"start": "1.2.3.5", "end": "1.2.3.8"} |
| cidr             | 1.2.3.0/24                             |
| dns_nameservers  |                                        |
| enable_dhcp      | False                                  |
| gateway_ip       | 1.2.3.1                                |
| host_routes      |                                        |
| id               | 5fd4033b-3609-4abc-bc17-2da0c1b1ccf9   |
| ip_version       | 4                                      |
| name             |                                        |
| network_id       | cd5dea18-1d25-492d-88b1-c439a0a93dad   |
| tenant_id        | 5f3fe6621a884758b37ca30506c161ff       |
+------------------+----------------------------------------+

Note that these IP addresses, in the range from 1.2.3.5 to 1.2.3.8 are called "floating IPs" in OpenStack. Later on, you can assign some floating IPs to virtual machines (see below).

Defining the default gateway for the external network

First, get the IDs of your ext_net and router:

# quantum net-list
+--------------------------------------+-------------+------------------------------------------------------+
| id                                   | name        | subnets                                              |
+--------------------------------------+-------------+------------------------------------------------------+
| 710620b7-63be-4a04-9168-35e19d7ca995 | testnetwork | b5cd1984-4a45-4233-90b1-19eb4f7a6e32 10.0.0.0/24     |
| cd5dea18-1d25-492d-88b1-c439a0a93dad | ext_net     | 5fd4033b-3609-4abc-bc17-2da0c1b1ccf9 1.2.3.0/24      |
+--------------------------------------+-------------+------------------------------------------------------+
# quantum router-list
+--------------------------------------+------------+-----------------------+
| id                                   | name       | external_gateway_info |
+--------------------------------------+------------+-----------------------+
| 0d259ad9-7727-44f2-9d86-a8d682073d87 | testrouter | null                  |
+--------------------------------------+------------+-----------------------+

Then simply use the router-gateway-set command using the router ID as first parameter, and the ext_net ID as second parameter:

# quantum router-gateway-set 0d259ad9-7727-44f2-9d86-a8d682073d87 cd5dea18-1d25-492d-88b1-c439a0a93dad
Set gateway for router 0d259ad9-7727-44f2-9d86-a8d682073d87

All this should be enough configuration to have a working networking setup.

Booting your first VM inside the tenant LAN

Before starting the virtual machine, check that you really do have 2 networks, as expected:

# quantum net-list
+--------------------------------------+-------------+------------------------------------------------------+
| id                                   | name        | subnets                                              |
+--------------------------------------+-------------+------------------------------------------------------+
| 710620b7-63be-4a04-9168-35e19d7ca995 | testnetwork | b5cd1984-4a45-4233-90b1-19eb4f7a6e32 10.0.0.0/24     |
| cd5dea18-1d25-492d-88b1-c439a0a93dad | ext_net     | 5fd4033b-3609-4abc-bc17-2da0c1b1ccf9 1.2.3.0/24      |
+--------------------------------------+-------------+------------------------------------------------------+

Then you can boot the VM (note: you can use "nova image-list" to list AMI images):

# nova boot --flavor 1 --image 7f2cae48-a017-4950-9f2c-d45e618367ca --nic net-id=710620b7-63be-4a04-9168-35e19d7ca995 firstinst
+-------------------------------------+--------------------------------------+
| Property                            | Value                                |
+-------------------------------------+--------------------------------------+
| OS-EXT-STS:task_state               | scheduling                           |
| image                               | tty-linux                            |
| OS-EXT-STS:vm_state                 | building                             |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000001                    |
| flavor                              | m1.tiny                              |
| id                                  | d0c9b41f-56e4-499e-9d00-0414a8aab922 |
| security_groups                     | [{u'name': u'default'}]              |
| user_id                             | 87d27a6fcac94ab7b5c2e3da89339bc1     |
| OS-DCF:diskConfig                   | MANUAL                               |
| accessIPv4                          |                                      |
| accessIPv6                          |                                      |
| progress                            | 0                                    |
| OS-EXT-STS:power_state              | 0                                    |
| OS-EXT-AZ:availability_zone         | None                                 |
| config_drive                        |                                      |
| status                              | BUILD                                |
| updated                             | 2013-04-09T07:49:56Z                 |
| hostId                              |                                      |
| OS-EXT-SRV-ATTR:host                | None                                 |
| key_name                            | None                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                 |
| name                                | firstinst                            |
| adminPass                           | vsJ7iPvBXxAL                         |
| tenant_id                           | 5f3fe6621a884758b37ca30506c161ff     |
| created                             | 2013-04-09T07:49:55Z                 |
| metadata                            | {}                                   |
+-------------------------------------+--------------------------------------+
# nova list
+--------------------------------------+-----------+--------+----------------------+
| ID                                   | Name      | Status | Networks             |
+--------------------------------------+-----------+--------+----------------------+
| d0c9b41f-56e4-499e-9d00-0414a8aab922 | firstinst | ACTIVE | testnetwork=10.0.0.2 |
+--------------------------------------+-----------+--------+----------------------+

Accessing the sshd of your virtual machine from within your quantum-network machine

Since Quantum is using IP namespaces, you have to use namespaces to access your VM. To do that, we first list all IP namespaces:

# ip netns
qdhcp-710620b7-63be-4a04-9168-35e19d7ca995
qrouter-0d259ad9-7727-44f2-9d86-a8d682073d87

We can then check the IP addresses of our testrouter router this way:

# ip netns exec qrouter-0d259ad9-7727-44f2-9d86-a8d682073d87 ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

qg-5cb2227b-60 Link encap:Ethernet  HWaddr fa:16:3e:ba:61:29
          inet addr:1.2.3.5  Bcast:1.2.3.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:feba:6129/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:813 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:53506 (52.2 KiB)  TX bytes:1462 (1.4 KiB)

qr-f2ed7a54-66 Link encap:Ethernet  HWaddr fa:16:3e:e2:9b:61
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fee2:9b61/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:228 errors:0 dropped:0 overruns:0 frame:0
          TX packets:149 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:19189 (18.7 KiB)  TX bytes:18120 (17.6 KiB)

As you can see, this shows the 1.2.3.5 and 10.0.0.1 IP addresses assigned to the router. If everything works, then you should be able to ping your virtual machine:

# ip netns exec qrouter-0d259ad9-7727-44f2-9d86-a8d682073d87 ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=16.9 ms
64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=0.282 ms
64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=0.234 ms
64 bytes from 10.0.0.2: icmp_req=4 ttl=64 time=0.229 ms
^C
--- 10.0.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 0.229/4.422/16.943/7.229 ms

And also ssh into it, with access to the public internet from within this virtual machine:

# ip netns exec qrouter-0d259ad9-7727-44f2-9d86-a8d682073d87 ssh root@10.0.0.2
The authenticity of host '10.0.0.2 (10.0.0.2)' can't be established.
RSA key fingerprint is 60:d3:89:19:28:f3:f7:c9:08:9c:bd:3e:23:86:c6:d9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.2' (RSA) to the list of known hosts.
root@10.0.0.2's password: <password is password...>

Chop wood, carry water.

# ifconfig
eth0      Link encap:Ethernet  HWaddr FA:16:3E:F7:12:76
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:208 errors:0 dropped:0 overruns:0 frame:0
          TX packets:253 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:26560 (25.9 KiB)  TX bytes:23116 (22.5 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=49 time=26.985 ms
64 bytes from 8.8.8.8: seq=1 ttl=49 time=11.038 ms
64 bytes from 8.8.8.8: seq=2 ttl=49 time=10.944 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 10.944/16.322/26.985 ms

Assigning a public IP address to a virtual machine

Why?

Above, we could access the virtual machine using the IP namespace. But that is kind of cheating, since normally a customer will not be able to login in your management network. It is therefore possible to assign floating IP addresses to virtual machines. You can see these as a kind of NAT thing: the public IP address will redirect to the LAN IP.

Creating floating IPs

This is super easy. No parameter, since everything is already configured. Just do this:

quantum floatingip-create ext_net
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 1.2.3.6                         |
| floating_network_id | cd5dea18-1d25-492d-88b1-c439a0a93dad |
| id                  | 27f9989b-3128-440c-9d42-21706c560bc4 |
| port_id             |                                      |
| router_id           |                                      |
| tenant_id           | 5f3fe6621a884758b37ca30506c161ff     |
+---------------------+--------------------------------------+

The IP address will be taken from the pool of IPs you created when defining the ext_net subnet.

Assigning a floating IP address to a VM

First, list your VM ID and its port:

# nova list
+--------------------------------------+-----------+--------+----------------------+
| ID                                   | Name      | Status | Networks             |
+--------------------------------------+-----------+--------+----------------------+
| d0c9b41f-56e4-499e-9d00-0414a8aab922 | firstinst | ACTIVE | testnetwork=10.0.0.2 |
+--------------------------------------+-----------+--------+----------------------+
# quantum port-list -- --device_id d0c9b41f-56e4-499e-9d00-0414a8aab922
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                       |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 1a1a6984-344a-419c-ab5f-bb26280a17e2 |      | fa:16:3e:f7:12:76 | {"subnet_id": "b5cd1984-4a45-4233-90b1-19eb4f7a6e32", "ip_address": "10.0.0.2"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+

Then assign the floating IP address to it:

# quantum floatingip-associate 27f9989b-3128-440c-9d42-21706c560bc4 1a1a6984-344a-419c-ab5f-bb26280a17e2
Associated floatingip 27f9989b-3128-440c-9d42-21706c560bc4

Now, you can check that the IP address 1.2.4.6 is really redirecting to 10.0.0.2:

quantum floatingip-show 27f9989b-3128-440c-9d42-21706c560bc4
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.0.0.2                             |
| floating_ip_address | 1.2.4.6                              |
| floating_network_id | cd5dea18-1d25-492d-88b1-c439a0a93dad |
| id                  | 27f9989b-3128-440c-9d42-21706c560bc4 |
| port_id             | 1a1a6984-344a-419c-ab5f-bb26280a17e2 |
| router_id           | 0d259ad9-7727-44f2-9d86-a8d682073d87 |
| tenant_id           | 5f3fe6621a884758b37ca30506c161ff     |
+---------------------+--------------------------------------+

Normally, doing a ping of 1.2.4.6 should now work from anywhere on the internet.

Security groups

Security groups with Nova

Before you can ssh into your virtual machine, you need to allow the ssh port. By default, everything is blocked on the floating IP address. This is done this way:

nova secgroup-add-rule default tcp 22 22 0.0.0.0/0

Then normally, you should be able to ssh into your VM:

# ssh root@1.2.4.6
The authenticity of host '1.2.4.6 (182.54.237.6)' can't be established.
RSA key fingerprint is 60:d3:89:19:28:f3:f7:c9:08:9c:bd:3e:23:86:c6:d9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '182.54.237.6' (RSA) to the list of known hosts.
root@182.54.237.6's password: 

Chop wood, carry water.

#

Listing security groups rules is done this way:

# nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

ICMP echo requests can also be added:

# nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

but I saw it working by default anyway.

Security groups with Quantum

The new way to configure security groups is to do it through Quantum. This is now the default with the latest version of the Debian packages for Grizzly (as of 24th june 2013). A "default" security group should exist. If it's not there, then you can create it this way:

# quantum security-group-create default

Then allow ssh and ICMP:

#quantum security-group-rule-create --direction ingress --protocol tcp --port_range_min 22 --port_range_max 22 default
#quantum security-group-rule-create --protocol icmp --direction ingress default