20521
Comment: replace placeholder CONTROLLER_HOSTNAME because puppet has troubles handling upper case and lower case and because hostname CONTROLLER_HOSTNAME does not work for the lack of dot in the hostname
|
20563
add proxy
|
Deletions are marked like this. | Additions are marked like this. |
Line 12: | Line 12: |
* 1 master/controller node: | * 1 master/proxy/controller node (named controller in the following): |
HOWTO: Openstack on Debian GNU/Linux unstable (sid)
Nova
This howto aims to provide guidelines to install & set up a multi-node Openstack-Compute (aka Nova) environment.
This environment will include 3 hosts :
- 2 compute nodes:
- compute1: pubnet@eth0=10.142.6.31 / privnet@eth1=169.254.6.31
- compute2: pubnet@eth0=10.142.6.32 / privnet@eth1=169.254.6.32
- 1 master/proxy/controller node (named controller in the following):
- controller: pubnet@eth0=10.142.6.33 / privnet@eth1=169.254.6.33
Choices:
- Virtualization technology: kvm/libvirt
Networking mode: ?VlanManger + multi_host
Services on compute* nodes :
- puppet agent
- nova-compute
- nova-network
- nova-api (metadata only)
On controller node:
- puppet master
- puppet agent
- mysql database
- keystone
- glance (local storage)
- nova-api
- nova-scheduler
- nova-novncproxy
PREREQUISITES
Things to prepare beforehand :
- Machines :
- They should have two network interfaces to ensure security. If only one interface is used the private part is more exposed to attacks coming from the public part.
- a "public" one to communicate with the outside world
- a "private" one for the guests VLans
- They should have two network interfaces to ensure security. If only one interface is used the private part is more exposed to attacks coming from the public part.
- Network :
- public network
private network. If the machines are not on a LAN, create one with OpenVPN.
- fixed ip range for guests
- number of networks for guests
- network size for guests
- public "floating" IPs (optional)
- Base distribution :
- Debian GNU/Linux squeeze (will be upgraded to wheezy)
IMPORTANT
This HOWTO is valid for the OpenStack Nova, Glance and Keystone packages labelled 2012.1, currently available in Debian unstable (sid) and might need some adjustments with later versions.
Upgrade to Wheezy
Edit /etc/apt/sources.list to read :
deb http://ftp.fr.debian.org/debian/ wheezy main deb-src http://ftp.fr.debian.org/debian/ wheezy main deb http://security.debian.org/ wheezy/updates main deb-src http://security.debian.org/ wheezy/updates main # squeeze-updates, previously known as 'volatile' deb http://ftp.fr.debian.org/debian/ squeeze-updates main deb-src http://ftp.fr.debian.org/debian/ squeeze-updates main
Then :
apt-get update apt-get dist-upgrade -y reboot
Puppet
Install puppet agent (on the three nodes):
apt-get install -y puppet augeas-tools
Install puppetmaster (only on the controller node)
apt-get install -y puppetmaster sqlite3 libsqlite3-ruby libactiverecord-ruby git
Configure the puppet agents
On all the nodes
Enable pluginsync & configure the hostname of the puppetmaster
augtool << EOT set /files/etc/puppet/puppet.conf/agent/pluginsync true set /files/etc/puppet/puppet.conf/agent/server <controller.hostname> save EOT
Configure the Puppet Master
On the controller node only.
- * Enable storedconfig and configure database
augtool << EOT set /files/etc/puppet/puppet.conf/master/storeconfigs true set /files/etc/puppet/puppet.conf/master/dbadapter sqlite3 set /files/etc/puppet/puppet.conf/master/dblocation /var/lib/puppet/server_data/storeconfigs.sqlite save EOT
- Create a dummy site manifest
cat > /etc/puppet/manifests/site.pp << EOT node default { notify { "Hey ! It works !": } } EOT
- * Restart puppetmaster
service puppetmaster restart
Test the puppet agents
⚠ Warning ⚠:
- With sqlite3 as database backend, only one puppet agent can run at once.
On each node run
puppet agent -vt --waitforcert 60
And while the puppet agent is waiting, on the master/controller run :::
puppetca sign -a
There should be no error and you should see a message saying "Hey ! It works !"
Install the openstack modules and the true manifest
Get the modules
cd /etc/puppet/modules git clone git://git.labs.enovance.com/puppet.git . git checkout openstack git submodule init git submodule update
Copy the example manifest for use by the puppetmaster and edit the values you want to change (IP addresses, passwords, …)
cp /etc/puppet/modules/examples/openstack_compute_multihost.pp /etc/puppet/manifests/site.pp
Installation
Note
- Openstack 2012.1 being currently only in the "unstable" sources, we need to enable sid packages to install it
echo "deb http://ftp.fr.debian.org/debian/ sid main" >> /etc/apt/sources.list apt-get update
Run puppet agent on the controller node
puppet agent -vt
Run puppet agent on the each compute node
puppet agent -vt
There should be no errors … (hopefully 'O_o)
Checking if it really works
The required services are advertised in the database
root@controller:~# nova-manage service list Binary Host Zone Status State Updated_At nova-consoleauth controller nova enabled :-) 2012-05-03 08:56:29 nova-scheduler controller nova enabled :-) 2012-05-03 08:56:31 nova-cert controller nova enabled :-) 2012-05-03 08:56:32 nova-compute compute1 nova enabled :-) 2012-05-03 08:56:50 nova-network compute1 nova enabled :-) 2012-05-03 08:56:49 nova-compute compute2 nova enabled :-) 2012-05-03 08:56:47 nova-network compute2 nova enabled :-) 2012-05-03 08:56:48
A file named 'openrc.sh' has been created in /root on the controller node. Source it & check the nova api works
root@controller:~# source openrc.sh root@controller:~# nova list +----+------+--------+----------+ | ID | Name | Status | Networks | +----+------+--------+----------+ +----+------+--------+----------+ root@controller:~# nova flavor-list +----+-----------+-----------+------+-----------+------+-------+-------------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | +----+-----------+-----------+------+-----------+------+-------+-------------+ | 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 | | 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | | 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | | 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | | 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | +----+-----------+-----------+------+-----------+------+-------+-------------+ root@controller:~# nova image-list +----+------+--------+--------+ | ID | Name | Status | Server | +----+------+--------+--------+ +----+------+--------+--------+
The openstack cluster is quite empty and useless like this, let's upload an image in glance::
root@controller:~# wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img … root@controller:~# glance add name="CirrOS 0.3" disk_format=qcow2 container_format=ovf < cirros-0.3.0-x86_64-disk.img Uploading image 'CirrOS 0.3' ================================================================[100%] 7.73M/s, ETA 0h 0m 0s Added new image with ID: 949bbc5c-e6fa-4ec3-91cb-65cbb6123c54 root@controller:~# glance index ID Name Disk Format Container Format Size ------------------------------------ ------------ ------------- ----------------- ---------- 949bbc5c-e6fa-4ec3-91cb-65cbb6123c54 CirrOS 0.3 qcow2 ovf 9761280
Does it show up in nova ?
root@controller:~# nova image-list +--------------------------------------+-----------------+--------+--------+ | ID | Name | Status | Server | +--------------------------------------+-----------------+--------+--------+ | 949bbc5c-e6fa-4ec3-91cb-65cbb6123c54 | CirrOS 0.3 | ACTIVE | | +--------------------------------------+-----------------+--------+--------+
We'll create a network for our vms (replace 169.254.200.0 with values to fit your setup. These will be private addresses for the instances)
root@controller:~# nova-manage network create --fixed_range_v4 169.254.200.0/24 --num_networks 1 --network_size 256 --label private --vlan 2000 root@controller:~# nova-manage network list id IPv4 IPv6 start address DNS1 DNS2 VlanID project uuid 1 169.254.200.0/24 None 169.254.200.3 None None 2000 None 71681e09-c072-4281-b5b4-37f26ddc97bf
And some floating (public) ips (choose an IP range addressable on your network)::
root@controller:~# nova-manage floating create --ip_range 10.142.6.224/27 root@controller:~# nova-manage floating list None 10.142.6.225 None nova eth0 None 10.142.6.226 None nova eth0 …
Now create a keypair (for ssh access) and save the output in a file
root@controller:~# nova keypair-add test_keypair > test_keypair.pem root@controller:~# chmod 600 test_keypair.pem
Boot an instance and get the console log
root@controller:~# nova boot --image 949bbc5c-e6fa-4ec3-91cb-65cbb6123c54 --flavor 1 --key_name test_keypair FirstTest --poll +-------------------------------------+--------------------------------------+ | Property | Value | +-------------------------------------+--------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | instance-00000001 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | accessIPv4 | | | accessIPv6 | | | adminPass | yab49fMqVHJf | | config_drive | | | created | 2012-05-03T10:09:00Z | | flavor | m1.tiny | | hostId | | | id | 06dd6129-f94a-488d-9670-7171491899e5 | | image | CirrOS 0.3 | | key_name | meh | | metadata | {} | | name | FirstTest | | progress | 0 | | status | BUILD | | tenant_id | d1c9085272d542eda98f7e08a1a779d6 | | updated | 2012-05-03T10:09:00Z | | user_id | cd04222b81004af5b0ff20c840fb629e | +-------------------------------------+--------------------------------------+ root@controller:~# nova console-log FirstTest …
Allocate a floating ip and associate it to the instance::
root@controller:~# nova floating-ip-create +--------------+-------------+----------+------+ | Ip | Instance Id | Fixed Ip | Pool | +--------------+-------------+----------+------+ | 10.142.6.225 | None | None | nova | +--------------+-------------+----------+------+ root@controller:~# nova add-floating-ip FirstTest 10.142.6.225 …
Update the rules for the default security group (allow icmp & ssh)::
root@controller:~# nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0 +-------------+-----------+---------+-----------+--------------+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-------------+-----------+---------+-----------+--------------+ | icmp | -1 | -1 | 0.0.0.0/0 | | +-------------+-----------+---------+-----------+--------------+ root@controller:~# nova secgroup-add-rule default tcp 22 22 0.0.0.0/0 +-------------+-----------+---------+-----------+--------------+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-------------+-----------+---------+-----------+--------------+ | tcp | 22 | 22 | 0.0.0.0/0 | | +-------------+-----------+---------+-----------+--------------+
We now should be able to ping the instance::
root@controller:~# ping -c 1 10.142.6.225 PING 10.142.6.225 (10.142.6.225) 56(84) bytes of data. 64 bytes from 10.142.6.225: icmp_req=1 ttl=63 time=0.626 ms --- 10.142.6.225 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.626/0.626/0.626/0.000 ms
And ssh into it with the identity we created before::
root@controller:~# ssh -i test_key cirros@10.142.6.225 $ uname -a Linux cirros 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 x86_64 GNU/Linix $ exit Connection to 10.142.6.114 closed.
Et voilà !
Swift
Assuming three machines installed with squeeze, the primary node being the openstack mgmt.host node and no puppet or puppetmaster installed.
swift primary node
apt-get install libmysql-ruby ruby-activerecord-2.3 sqlite3 puppetmaster puppet ruby-sqlite3
Puppet configuration:
diff --git a/puppet/puppet.conf b/puppet/puppet.conf index b18fae3..ce4ed22 100644 --- a/puppet/puppet.conf +++ b/puppet/puppet.conf @@ -7,6 +7,8 @@ factpath=$vardir/lib/facter templatedir=$confdir/templates prerun_command=/etc/puppet/etckeeper-commit-pre postrun_command=/etc/puppet/etckeeper-commit-post +pluginsync=true +storeconfigs=true [master] # These are needed when the puppetmaster is run by passenger commit 507105065306433eec8f03dd72ab52ccaf268ccc Author: root <root@sd-16961.dedibox.fr> Date: Mon Apr 2 15:04:53 2012 +0200 configure database storage diff --git a/puppet/puppet.conf b/puppet/puppet.conf index ce4ed22..af220e9 100644 --- a/puppet/puppet.conf +++ b/puppet/puppet.conf @@ -9,10 +9,19 @@ prerun_command=/etc/puppet/etckeeper-commit-pre postrun_command=/etc/puppet/etckeeper-commit-post pluginsync=true storeconfigs=true +server=mgmt.host [master] # These are needed when the puppetmaster is run by passenger # and can safely be removed if webrick is used. ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY +storeconfigs=true +# Needed for storeconfigs=true +dbadapter=mysql +dbname=puppet +dbuser=puppet +dbpassword=password +dbserver=localhost +dbsocket=/var/run/mysqld/mysqld.sock
Setup mysql for puppet:
mysqladmin create puppet mysql -e "grant all on puppet.* to 'puppet'@'localhost' identified by 'password';"
Install openstack modules for puppet:
cd /etc/puppet git clone git://git.labs.enovance.com/openstack-puppet-modules.git modules && cd modules && git submodule init && git submodule update cp /etc/puppet/modules/swift/examples/multi.pp /etc/puppet/manifests/site.pp
commit 8eb77223e25bfff1284612417efedd228e0c6696 Author: root <root@sd-16961.dedibox.fr> Date: Mon Apr 2 15:37:19 2012 +0200 use tap0 for lan diff --git a/puppet/manifests/site.pp b/puppet/manifests/site.pp index a915aea..9b890b0 100644 --- a/puppet/manifests/site.pp +++ b/puppet/manifests/site.pp @@ -28,7 +28,7 @@ $swift_shared_secret='changeme' # assumes that the ip address where all of the storage nodes # will communicate is on eth1 -$swift_local_net_ip = $ipaddress_eth0 +$swift_local_net_ip = $ipaddress_tap0 Exec { logoutput => true }
Enable puppet autosign for all hosts:
echo '*' > /etc/puppet/autosign.conf
Deploy swift configuration on the proxy:
chown -R puppet:puppet /var/lib/puppet/ puppet agent --certname=swift_storage_1 --server=mgmt.host --verbose --debug --test /etc/init.d/xinetd reload
swift secondary nodes
deb http://ftp.fr.debian.org/debian/ wheezy main deb http://ftp.fr.debian.org/debian/ sid main apt-get install python2.7=2.7.2-8 python2.7-minimal=2.7.2-8 libpython2.7=2.7.2-8 python-prettytable=0.5-1 echo libpython2.7 hold | dpkg --set-selections echo python2.7 hold | dpkg --set-selections echo python2.7-minimal hold | dpkg --set-selections echo python-prettytable hold | dpkg --set-selections apt-get install puppet ruby-sqlite3 puppet agent --certname=swift_storage_3 --server=openstack-online-0001.dedibox.fr --verbose --debug --test
create swift ring
puppet agent --certname=swift_proxy --server=openstack-online-0001.dedibox.fr --verbose --debug --test
propagate the swift configuration
puppet agent --certname=swift_storage_1 --server=openstack-online-0001.dedibox.fr --verbose --debug --test puppet agent --certname=swift_storage_2 --server=openstack-online-0001.dedibox.fr --verbose --debug --test puppet agent --certname=swift_storage_3 --server=openstack-online-0001.dedibox.fr --verbose --debug --test
check that it works
On proxy / mgmt.host :
# cd /etc/puppet/modules/swift/ext # ruby swift.rb getting credentials: curl -k -v -H "X-Storage-User: test:tester" -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0 verifying connection auth: curl -k -v -H "X-Auth-Token: AUTH_tk5d5a63abdf90414eafd890ed710d357b" http://127.0.0.1:8080/v1/AUTH_test Testing swift: swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat found containers/objects: 0/0 Uploading file to swift with command: swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing upload my_container /tmp/foo1 tmp/foo1 Downloading file with command: swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing download my_container tmp/foo1 Dude!!!! It actually seems to work, we can upload and download files!!!!
Swift/Horizon
Edit /etc/keystone/default_catalog.templates like this:
catalog.RegionOne.object-store.publicURL = http://mgmt.host:8080/v1/AUTH_$(tenant_id)s catalog.RegionOne.object-store.adminURL = http://mgmt.host:8080/ catalog.RegionOne.object-store.internalURL = http://mgmt.host:8080/v1/AUTH_$(tenant_id)s catalog.RegionOne.object-store.name = 'Object Store Service'
diff --git a/swift/proxy-server.conf b/swift/proxy-server.conf index 83dda1e..8364fe7 100644 --- a/swift/proxy-server.conf +++ b/swift/proxy-server.conf @@ -7,7 +7,8 @@ user = swift [pipeline:main] # ratelimit? -pipeline = healthcheck cache tempauth proxy-server +#pipeline = healthcheck cache tempauth proxy-server +pipeline = healthcheck cache tokenauth keystone proxy-server [app:proxy-server] use = egg:swift#proxy @@ -28,3 +29,17 @@ use = egg:swift#healthcheck use = egg:swift#memcache # multi-proxy config not supported memcache_servers = 127.0.0.1:11211 + +[filter:tokenauth] +paste.filter_factory = keystone.middleware.auth_token:filter_factory +service_port = 5000 +service_protocol = http +service_host = 127.0.0.1 +auth_port = 35357 +auth_protocol = http +auth_host = 127.0.0.1 +admin_token = ADMIN + +[filter:keystone] +paste.filter_factory = keystone.middleware.swift_auth:filter_factory +operator_roles = admin, swiftoperator, projectmanager
/etc/init.d/swift-proxy restart
swift command line
apt-get install swift swift -U $OS_TENANT_NAME:$OS_USERNAME list