|
|
Openstack上使用BOSH部署CloudFoundry（二）
第二篇 ：部署BOSH
micro bosh部署完毕后，要利用micro bosh部署bosh，先从bosh-cli target到micro bosh上
[html] view plaincopy
- ubuntu@bosh-cli:~/deployments$ bosh target 10.23.51.26:25555
- Target already set to `microbosh-openstack'
- ubuntu@bosh-cli:~/deployments$ bosh login
- Your username: admin
- Enter password: *****
- Logged in as `admin'
默认用户密码为admin admin[html] view plaincopy
- ubuntu@bosh-cli:~/deployments$ bosh deployments
发现还没有部署任何项目。那么，下面就着手一步步部署bosh，本文只专注部署过程以及碰到的问题，bosh模块简介等不涉及。
BOSH stemcell
部署micro bosh 用的是mirco bosh stemcell，而部署bosh则用 bosh stemcell，两个stemcell的区别在上一篇中已经提到。
[html] view plaincopy
- bosh public stemcells --all
选择openstack 最新版本
[html] view plaincopy
- bosh download public stemcell bosh-stemcell-openstack-kvm-0.7.0.tgz
上传bosh stemcell
[html] view plaincopy
- bosh upload stemcell bosh-stemcell-openstack-kvm-0.7.0.tgz
bosh-release编译包先从github或自己的代码库中取最新版本，这里是用github上的代码。
[html] view plaincopy
[html] view plaincopy
- git submodule update --init –recursive
[html] view plaincopy
- bosh create release –with-tarball --force
不出情况的话，会生成release包。这一步得看网速和网络环境了，rubygem你懂的。有快有慢，如果有提示哪个模块没有，其实可以进入文件夹，手工去git clone。
如中间碰到某些gem搞不下来， 那是rubygem问题了，多多重试。最不济就用goagent，虽然在micro bosh上再搞个goagent有点别扭。
[html] view plaincopy
- ...
- ...
- Release version:11.1-dev
- Release manifest:/home/ubuntu/bosh-release/dev_releases/bosh-dev1-11.1-dev.yml
- Release tarball (88.8M):/home/ubuntu/bosh-release/dev_releases/bosh-dev1-11.1-dev.tgz
上传bosh-release
[html] view plaincopy
- bosh upload release ~/bosh-release/dev_releases/bosh-dev-11.1-dev.tgz
编写bosh部署文件bosh.ymlbosh.yml 定义了安装的模块，以及IP、内存等信息
[html] view plaincopy
- ---
- name: bosh-openstack #bosh名称，部署完bosh deployments列出来的就是
- director_uuid: f6d5ef18-441d-40a4-b895-0a49f72097d4 #bosh status ,把uudi copy过来
-
- release:
- name: bosh-dev # bosh releases,查看上传的release,name和version要对应上
- version: 11.1-dev
-
- compilation:
- workers: 1
- network: default
- reuse_compilation_vms: true
- cloud_properties:
- instance_type: jae.bosh # 此处的flavor要自己提前定义
-
- update:
- canaries: 1
- canary_watch_time: 90000
- update_watch_time: 90000
- max_in_flight: 1
- max_errors: 1
-
- networks:
- - name: floating
- type: vip
- cloud_properties: {}
- - name: default
- type: dynamic
- dns:
- - 10.23.54.108 # CHANGE: Micro Bosh IP
- cloud_properties:
- security_groups:
- - default
-
- resource_pools:
- - name: small
- network: default
- size: 8
- stemcell:
- name: bosh-stemcell
- version: 0.7.0
- cloud_properties:
- instance_type: m3.small-no-delete #自定义flavor
- env:
- bosh:
- password: $6$D0NatspPe$CscRzCu.SHVOBW0V114PyfXbESPLbOmx9TpHGEWKAYtGD3IXDCqUzdfinyjWTMxccder7AFuy0Z5mej2yzR5f.
-
- jobs:
- - name: powerdns
- template: powerdns
- instances: 1
- resource_pool: small
- networks:
- - name: default
- default: [dns, gateway]
- - name: floating
- static_ips:
- - 10.23.54.121 # Change: Floating IP 1
-
- - name: nats
- template: nats
- instances: 1
- resource_pool: small
- networks:
- - name: default
- default: [dns, gateway]
-
- - name: postgres
- template: postgres
- instances: 1
- resource_pool: small
- persistent_disk: 2048
- networks:
- - name: default
- default: [dns, gateway]
-
- - name: redis
- template: redis
- instances: 1
- resource_pool: small
- networks:
- - name: default
- default: [dns, gateway]
-
- - name: director
- template: director
- instances: 1
- resource_pool: small
- persistent_disk: 4096
- networks:
- - name: default
- default: [dns, gateway]
- - name: floating
- static_ips:
- - 10.23.54.122 # Change: Floating IP 2
-
- - name: blobstore
- template: blobstore
- instances: 1
- resource_pool: small
- persistent_disk: 8192
- networks:
- - name: default
- default: [dns, gateway]
-
- - name: openstack_registry
- template: openstack_registry
- instances: 1
- resource_pool: small
- networks:
- - name: default
- default: [dns, gateway]
- - name: floating
- static_ips:
- - 10.23.54.126 # Change: Floating IP 3
-
- - name: health_monitor
- template: health_monitor
- instances: 1
- resource_pool: small
- networks:
- - name: default
- default: [dns, gateway]
-
- properties:
- env:
-
- dns:
- user: powerdns
- password: powerdns
- address: 10.23.54.121 # Change: Floating IP 1
- replication:
- basic_auth: replication
 RoqsenPsNGX6 - user: replication
- password: powerdns
- database:
- name: powerdns
- port: 5432
- #recursors:
- # - 10.23.51.10
- webserver:
- password: powerdns
- port: 8081
-
- nats:
- user: nats
- password: nats
- address: 0.nats.default.bosh-openstack.bosh
- listen_address: 0.0.0.0
- port: 4222
-
- postgres:
- user: postgres
- password: postgres
- address: 0.postgres.default.bosh-openstack.bosh
- listen_address: 0.0.0.0
- port: 5432
- database: bosh
-
- redis:
- address: 0.redis.default.bosh-openstack.bosh
- listen_address: 0.0.0.0
- port: 25255
- password: redis
-
- director:
- name: bosh-openstack
- address: 10.23.54.122 # Change: Floating IP 2
- port: 25555
- encryption: false
-
- blobstore:
- address: 0.blobstore.default.bosh-openstack.bosh
- port: 25250
- backend_port: 25251
- agent:
- user: agent
- password: agent
- director:
- user: director
- password: director
-
- openstack_registry:
- address: 10.40.62.126 # Change: Floating IP 3
- http:
- port: 25777
- user: registry
- password: registry
- db:
- max_connections: 32
- pool_timeout: 10
-
- hm:
- http:
- port: 25923
- user: hm
- password: hm
- director_account:
- user: admin
- password: admin
- intervals:
- prune_events: 30
- poll_director: 60
- poll_grace_period: 30
- log_stats: 60
- analyze_agents: 60
- agent_timeout: 180
- rogue_agent_alert: 180
- loglevel: info
- email_notifications: false
- tsdb_enabled: false
- pagerduty_enable: false
- varz_enabled: false
-
- openstack:
- auth_url: http://ip:5000/v2.0/tokens # CHANGE: OpenStack Auth URL
- username: xx # CHANGE: OpenStack Username
- api_key: xx # CHANGE: OpenStack Password
- tenant: jae # CHANGE: OpenStack Tenant
- default_key_name: xx2 # CHANGE: OpenStack default Keyname
- default_security_groups: ["default"] # CHANGE: OpenStack default Security Group
- private_key: ~/.ssh/xx2.pem # CHANGE: OpenStack Private Key
部署bosh
[html] view plaincopy
- ubuntu@bosh-cli:~/deployments$ bosh deployment bosh.yml
- Deployment set to `/home/ubuntu/deployments/bosh-openstack/bosh.yml'
- ubuntu@bosh-cli:~/deployments$ bosh deploy
部署成功后，bosh deployments 和 bosh vms查看部署情况。
[html] view plaincopy
- ubuntu@bosh-cli:~/deployments$ bosh deployments
-
- +----------------+-------------------+---------------------+
- | Name | Release(s) | Stemcell(s) |
- +----------------+-------------------+---------------------+
- | bosh-openstack | bosh-dev/11.1-dev | bosh-stemcell/0.7.0 |
- +----------------+-------------------+---------------------+
-
- Deployments total: 1
- ubuntu@bosh-cli:~/deployments$ bosh vms
- Deployment `bosh-openstack'
-
- Director task 33
-
- Task 33 done
-
- +----------------------+---------+---------------+---------------------------+
- | Job/index | State | Resource Pool | IPs |
- +----------------------+---------+---------------+---------------------------+
- | blobstore/0 | running | small | 10.23.51.71 |
- | director/0 | running | small | 10.23.51.69, 10.23.54.122 |
- | health_monitor/0 | running | small | 10.23.51.80 |
- | nats/0 | running | small | 10.23.51.79 |
- | openstack_registry/0 | running | small | 10.23.51.77, 10.23.54.126 |
- | postgres/0 | running | small | 10.23.51.73 |
- | powerdns/0 | running | small | 10.23.51.41, 10.23.54.121 |
- | redis/0 | running | small | 10.23.51.66 |
- +----------------------+---------+---------------+---------------------------+
-
- VMs total: 8
遇到的问题
其实部署bosh，官网等都有详细的文档。但是过程中的问题总是神出鬼没，所以这一个环节显得很有必要。
问题一：openstack用户的quota配额当执行bosh deploy的时候，问题就集中出现了，第一个碰到的问题是openstack用户的quota配额问题。
[html] view plaincopy
- Creating bound missing VMs
- small/2: Expected([200, 202]) <=> Actual(413 Request Entity Too Large)
- request => {:connect_timeout=>60, :headers=>{"Content-Type"=>"application/json", "X-Auth-Token"=>"38fe51b931184a30a287e71bc37cc05d", "Host"=>"10.23.54.150:8774", "Content-Length"=>422}, :instrumentor_name=>"excon", :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/excon-0.16.2/data/cacert.pem", :ssl_verify_peer=>true, :write_timeout=>60, :host=>"10.23.54.150", :path=>"/v2/69816bacecd749f9ba1d68b3c8bae1f1/servers.json", :port=>"8774", :query=>"ignore_awful_caching1362453746", :scheme=>"http", :body=>"{\"server\":{\"flavorRef\":\"25\",\"imageRef\":\"e205b9ec-0e19-4500-87fe-ede3af13b227\",\"name\":\"vm-b875d6d8-81ce-483b-bfa8-d6d525aaf280\",\"metadata\":{},\"user_data\":\"eyJyZWdpc3RyeSI6eyJlbmRwb2ludCI6Imh0dHA6Ly8xMC4yMy41MS4zNToy\\nNTc3NyJ9LCJzZXJ2ZXIiOnsibmFtZSI6InZtLWI4NzVkNmQ4LTgxY2UtNDgz\\nYi1iZmE4LWQ2ZDUyNWFhZjI4MCJ9LCJkbnMiOnsibmFtZXNlcnZlciI6WyIx\\nMC4yMy41NC4xMDgiXX19\\n\",\"key_name\":\"jae2\",\"security_groups\":[{\"name\":\"default\"}]}}", :expects=>[200, 202], :method=>"
 OST"} - response => #<Excon::Response:0x00000004edec30 @body="{\"overLimit\": {\"message\": \"Quota exceeded for instances: Requested 1, but already used 10 of 10 instances\", \"code\": 413}}", @headers={"Retry-After"=>"0", "Content-Length"=>"121", "Content-Type"=>"application/json; charset=UTF-8", "X-Compute-Request-Id"=>"req-c5427ed2-62af-47b9-98a6-6f114893d8fc", "Date"=>"Tue, 05 Mar 2013 03:22:27 GMT"}, @status=413> (00:00:02)
看response部分：
[html] view plaincopy
- response => #<Excon::Response:0x00000004edec30 @body="{\"overLimit\": {\"message\": \"Quota exceeded for instances: Requested 1, but already used 10 of 10 instances\", \"code\": 413}}"
由于quota，不能再创建instance了。但是bosh部署的时候要创建8个组件vm + n 个workers。后面还有cf-release的几十个vm呢，必须要调整。通过openstack dashboard 修改此用户的各种配额，包括instance数，vcpu数，mem，volume等，尽量够大，省得再改麻烦。
改后再bosh deply，如果想从头再来，可以删除部署过的半拉子项目：
[html] view plaincopy
- bosh deployments
-
- bosh delete deployment {depoly-name}
这个命令会把创建过的vm，volume全删掉。另外要记得删掉bosh当前目录有个缓存的yml。不然会出现vm找不到的情况，因为缓存yml已经记录了创建过的vm，你删掉了可不就找不着了。
问题二：创建vm的线程问题。
[html] view plaincopy
- Director task 22
-
- Preparing deployment
- binding deployment (00:00:00)
- binding releases (00:00:00)
- binding existing deployment: VM `413db010-cb45-45fb-97f5-819cd4f4b6fe' is out of sync: it reports itself as `blobstore/0' but there is no instance reference in DB (00:00:00)
- Error 3/9 00:00:00
-
- Error 400004: VM `413db010-cb45-45fb-97f5-819cd4f4b6fe' is out of sync: it reports itself as `blobstore/0' but there is no instance reference in DB
这个问题google良久，没有找到可靠的办法。是跟创建vm的最大线程有关。由于本文的bosh-release用的是年前编译的，没有更新。查看代码，创建vm的最大线程数为32，
于是手动改了，新代码里写个变量了，可以create bosh release前改下配置。部署bosh是由micro bosh的direct统一调度的，所以要先 ssh到micro bosh去修改源码。
[html] view plaincopy
密码为admin，在micro_bosh.yml设置的。上一篇中有提到：Openstack上使用BOSH部署CloudFoundry（一）
代码位于/var/vcap/packages/director/bosh/director/lib/director/jobs/update_deployment.rb的85行左右。新版本上已经改为变量了：
[html] view plaincopy
- https://github.com/cloudfoundry/bosh/blob/master/director/lib/director/jobs/update_deployment.rb#L85
我是把这一行改成:
[html] view plaincopy
- ThreadPool.new(:max_threads => 64).wrap do |thread_pool|
然后重启micro bosh 的director
[html] view plaincopy
- root@bm-c3ffd8f1-46be-4936-becc-03c426317ade:sudo su -
- root@bm-c3ffd8f1-46be-4936-becc-03c426317ade:/var/vcap/bosh/bin# ./monit restart director
另外，担心worker过多也会造成问题，直接把worker减到1。
问题三：Error 400007: `director/0' is not running after update[html] view plaincopy
- ubuntu@bosh-cli:~/deployments$ bosh vms
- Deployment `bosh-openstack'
-
- Director task 29
-
- Task 29 done
-
- +----------------------+---------+---------------+---------------------------+
- | Job/index | State | Resource Pool | IPs |
- +----------------------+---------+---------------+---------------------------+
- | blobstore/0 | running | small | 10.23.51.71 |
- | director/0 | failing | small | 10.23.51.69, 10.23.54.122 |
- | health_monitor/0 | running | small | 10.23.51.80 |
- | nats/0 | running | small | 10.23.51.79 |
- | openstack_registry/0 | running | small | 10.23.51.77, 10.23.54.126 |
- | postgres/0 | running | small | 10.23.51.73 |
- | powerdns/0 | running | small | 10.23.51.41, 10.23.54.121 |
- | redis/0 | running | small | 10.23.51.66 |
- +----------------------+---------+---------------+---------------------------+
果然director 的状态是failing，进去openstack dashboard，手动重启下对应的vm，然后再次bosh deploy，又碰到一两次IT论坛模块的这个问题，一样的方法解决了。这个方法
比较旁门，还是得把问题根源解决。
本文到此结束，有不足不处望指正，多多交流，微博：http://weibo.com/kingjames3
下篇讲解如何用bosh来部署cloudfoundry。
摘自：http://www.csdn123.com/k_james/article/details/8638769
Openstack上使用BOSH部署CloudFoundry（二）
|
|
京ICP备14020293号-2
本网站内容均收集于互联网，如有问题请联系QQ:389897944予以删除
窥视卡
雷达卡
楼主
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡