aboutsummaryrefslogtreecommitdiffstats
path: root/meta-openstack
diff options
context:
space:
mode:
authorVu Tran <vu.tran@windriver.com>2014-07-10 13:48:28 -0400
committerBruce Ashfield <bruce.ashfield@windriver.com>2014-07-17 00:14:05 -0400
commitce99c21125614dfa72428a9ef3ed563db8c99c93 (patch)
treeae5f2bfe88a8fc7dc90d23374d32186145bd88a7 /meta-openstack
parent362dd9c1793f8d50fc4255205b0a7ae34e72f3fa (diff)
downloadmeta-cloud-services-ce99c21125614dfa72428a9ef3ed563db8c99c93.tar.gz
meta-cloud-services-ce99c21125614dfa72428a9ef3ed563db8c99c93.tar.bz2
meta-cloud-services-ce99c21125614dfa72428a9ef3ed563db8c99c93.zip
rally add readme doc
Signed-off-by: Vu Tran <vu.tran@windriver.com>
Diffstat (limited to 'meta-openstack')
-rw-r--r--meta-openstack/Documentation/README.rally335
1 files changed, 335 insertions, 0 deletions
diff --git a/meta-openstack/Documentation/README.rally b/meta-openstack/Documentation/README.rally
new file mode 100644
index 0000000..352ef25
--- /dev/null
+++ b/meta-openstack/Documentation/README.rally
@@ -0,0 +1,335 @@
+Summary
+=======
+
+This document is not intended to provide detail of how Rally in general
+works, but rather it highlights how Rally is integrated/configured into
+meta-cloud-services and also describes how Rally is tested to ensure that
+Rally Verification and Benchmarking components are working correctly.
+
+
+Rally Overview
+==============
+
+Rally - OpenStack benchmarking at a scale - is intended to provide the
+community with a benchmarking tool that is capable of performing "specific",
+"complicated" and "reproducible" test cases on "real deployment" scenarios.
+
+Rally has 4 main components:
+
+* Server Providers - provides virtual servers
+* Deploy Engines - deploys OpenStack cloud on servers that are presented by Server Providers
+* Verification - runs tempest
+* Benchmarking - allows to write parameterized benchmark scenarios & run them against the cloud
+
+
+Rally Deployment
+================
+
+Rally is configured to use existing deployment (by using deployment configuration file
+/etc/rally/deployments/existing.json) and therefore Rally components "Server Providers"
+and "Deploy Engines" are not used heavily.
+
+
+Rally Verification
+==================
+
+By default, Rally verification performs the following steps:
+
+* git clone tempest source from upstream
+* setup virtualenv for this tempest
+* setup testr environment with virtualenv created above
+* create tempest.conf for this tempest
+* use testr and subunit.run module to run tempest
+
+However, meta-cloud-services already includes tempest which is also configured/modified
+to have low failure/error testcases, therefore it's desired to use this tempest
+(without using virtualenv) instead of letting Rally to download tempest and running it
+on virtualenv.
+
+The option "existing_tempest_config" in /etc/rally/rally.conf can be used to configure
+Rally to either use the existing tempest or to download from upstream.
+
+If the option "existing_tempest_config" is not set then Rally follows the default path.
+If "existing_tempest_config" is set to absolute path of tempest config folder (which
+contains tempest "tools" and .testr.conf, e.g. /etc/tempest) then Rally uses this
+existing tempest. By default, "existing_tempest_config" is set to "/etc/tempest/".
+
+
+Build Configuration Options
+===========================
+
+To have Rally and tempest included in final built image, include layer
+meta-openstack-controller-test-config into Controller build and
+layer meta-openstack-compute-test-config into Compute build.
+
+
+Test Steps
+==========
+
+Please note: the following commands/steps are carried on Controller
+node, unless otherwise explicitly indicated.
+
+ # Start Controller and Compute node
+ $ ps aux | grep rally
+root 13954 4.2 0.0 130856 27500 ? Ss 17:05 0:00 python /usr/bin/rally-api --log-dir=/var/log/rally
+
+ $ curl http://localhost:8877
+{"versions": [{"status": "CURRENT", "media_types": [{"base": "application/json", "type": "application/vnd.openstack.rally.v1+json"}], "id": "v1", "links": [{"href": "http://localhost:8877/v1", "rel": "self"}], "updated_at": "2014-01-07T00:00:00Z"}], "description": "Rally is a Benchmark-as-a-Service project for OpenStack.", "name": "OpenStack Rally API"}
+
+ $ rally deployment create --filename=/etc/rally/deployments/existing.json --name=wr-ops
++--------------------------------------+----------------------------+--------+------------------+--------+
+| uuid | created_at | name | status | active |
++--------------------------------------+----------------------------+--------+------------------+--------+
+| 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | 2014-07-07 17:09:11.160741 | wr-ops | deploy->finished | |
++--------------------------------------+----------------------------+--------+------------------+--------+
+
+ $ rally deployment list
++--------------------------------------+----------------------------+--------+------------------+--------+
+| uuid | created_at | name | status | active |
++--------------------------------------+----------------------------+--------+------------------+--------+
+| 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | 2014-07-07 17:09:11.160741 | wr-ops | deploy->finished | * |
++--------------------------------------+----------------------------+--------+------------------+--------+
+
+ $ rally use deployment --uuid=5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb
+Using deployment: 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb
+~/.rally/openrc was updated
+
+ $ rally deployment check
+keystone endpoints are valid and following services are available:
++------------+----------------+-----------+
+| services | type | status |
++------------+----------------+-----------+
+| barbican | keystore | Available |
+| ceilometer | metering | Available |
+| cinder | volume | Available |
+| ec2 | ec2 | Available |
+| glance | image | Available |
+| heat | orchestration | Available |
+| heat-cfn | cloudformation | Available |
+| keystone | identity | Available |
+| neutron | network | Available |
+| nova | compute | Available |
+| swift | object-store | Available |
++------------+----------------+-----------+
+
+ $ rally show flavors
++----+-----------+-------+----------+-----------+-----------+
+| ID | Name | vCPUs | RAM (MB) | Swap (MB) | Disk (GB) |
++----+-----------+-------+----------+-----------+-----------+
+| 1 | m1.tiny | 1 | 512 | | 1 |
+| 2 | m1.small | 1 | 2048 | | 20 |
+| 3 | m1.medium | 2 | 4096 | | 40 |
+| 4 | m1.large | 4 | 8192 | | 80 |
+| 5 | m1.xlarge | 8 | 16384 | | 160 |
++----+-----------+-------+----------+-----------+-----------+
+
+ $ rally show images
++--------------------------------------+--------------+----------+
+| UUID | Name | Size (B) |
++--------------------------------------+--------------+----------+
+| 3ec2f841-b179-47d7-908a-edb766b65c5e | myfirstimage | 13167616 |
++--------------------------------------+--------------+----------+
+
+ $ rally show keypairs
++------+-------------+
+| Name | Fingerprint |
++------+-------------+
++------+-------------+
+
+ $ rally show networks
++--------------------------------------+--------------------------------------------+------+
+| ID | Label | CIDR |
++--------------------------------------+--------------------------------------------+------+
+| 7a9cf643-d9ab-404e-8155-65134bd937fb | TEMPEST_NET | None |
++--------------------------------------+--------------------------------------------+------+
+
+ $ rally verify list
+There are no results from verifier. To run a verifier, use:
+rally verify start
+
+ $ rally verify start
+Configuring existing Tempest for deployment 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb
+Verifier has been successfully configured to use existing Tempest
+tempest.api.network.test_extensions.ExtensionsTestXML
+ test_list_show_extensions[gate,smoke] FAIL
+setUpClass (tempest.api.network.test_networks
+ BulkNetworkOpsJSON) FAIL
+tempest.api.orchestration.stacks.test_stacks.StacksTestJSON
+ test_stack_crud_no_resources[gate,smoke] OK 6.56
+ test_stack_list_responds[gate,smoke] OK 0.02
+setUpClass (tempest.api.network.test_load_balancer
+ LoadBalancerXML) FAIL
+tempest.api.compute.test_quotas.QuotasTestJSON
+ test_get_default_quotas[gate,smoke] OK 1.10
+ test_get_quotas[gate,smoke] OK 0.02
+tempest.api.identity.admin.v3.test_domains.DomainsTestJSON
+ test_create_update_delete_domain[gate,smoke] OK 5.67
+
+...
+
+(Start Rally verification by running all on tempest testcases which are marked as smoke)
+
+ $ rally verify list
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status |
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+| a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished |
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+
+ $ rally verify show a5575bd2-d132-4522-b625-eed324396dda
+Total results of verification:
+
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status |
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+| a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished |
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+
+Tests:
+
++--------------------------------------------------------------------------------------------------------------------------------------------+------------+--------+
+| name | time | status |
++--------------------------------------------------------------------------------------------------------------------------------------------+------------+--------+
+| process-returncode | 0.0 | FAIL |
+| setUpClass (tempest.api.compute.servers.test_create_server.ServersTestJSON) | 0.0 | FAIL |
+| setUpClass (tempest.api.compute.servers.test_create_server.ServersTestManualDisk) | 0.0 | FAIL |
+| setUpClass (tempest.api.compute.servers.test_create_server.ServersTestXML) | 0.0 | FAIL |
+| setUpClass (tempest.api.network.test_floating_ips.FloatingIPTestJSON) | 0.0 | FAIL |
+| setUpClass (tempest.api.network.test_floating_ips.FloatingIPTestXML) | 0.0 | FAIL |
+| setUpClass (tempest.api.network.test_load_balancer.LoadBalancerJSON) | 0.0 | FAIL |
+
+...
+
+ $ rally verify results a5575bd2-d132-4522-b625-eed324396dda
+(lot of details is printed out. Just make sure that there is no error when running this command)
+
+ $ rally verify detailed a5575bd2-d132-4522-b625-eed324396dda
+(the out put is the same to the output of command "rally verify show" plus all the error/warning
+messages from running tempest)
+
+ $ rally verify start --set identity
+root@controller:~# rally verify start --set identity
+tempest.api.identity.admin.test_services.ServicesTestJSON
+ test_create_get_delete_service[gate,smoke] OK 1.06
+ test_list_services[gate,smoke] OK 0.82
+tempest.api.identity.admin.v3.test_services.ServicesTestJSON
+ test_update_service[gate] OK 3.65
+tempest.api.identity.admin.test_tenant_negative.TenantsNegativeTestJSON
+ test_create_tenant_by_unauthorized_user[gate,negative] OK 1.71
+ test_create_tenant_request_without_token[gate,negative] OK 0.73
+ test_create_tenant_with_empty_name[gate,negative] OK 0.89
+
+...
+
+(Start Rally verification by running all tempest testcases in tempest/api/identiy)
+
+ $ rally verify list
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status |
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+| a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished |
+| b05bbc4c-eab4-4296-9020-d8305aa6457b | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | identity | 216 | 0 | 2014-07-07 17:59:35.358702 | finished |
++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
+
+ $ rally -v task start /etc/rally/tasks/example.json
+================================================================================
+Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is started
+--------------------------------------------------------------------------------
+2014-07-07 18:06:30.608 9641 INFO rally.orchestrator.api [-] Benchmark Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 on Deployment 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb
+2014-07-07 18:06:30.609 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Check cloud.
+2014-07-07 18:06:30.766 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Completed: Check cloud.
+2014-07-07 18:06:30.766 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation.
+2014-07-07 18:06:30.785 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation of scenarios names.
+2014-07-07 18:06:30.788 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Completed: Task validation of scenarios names.
+2014-07-07 18:06:30.788 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation of syntax.
+
+...
+
+================================================================================
+Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is finished.
+--------------------------------------------------------------------------------
+
+test scenario Authenticate.keystone
+args position 0
+args values:
+{u'context': {u'users': {u'concurrent': 30,
+ u'tenants': 3,
+ u'users_per_tenant': 50}},
+ u'runner': {u'concurrency': 5, u'times': 100, u'type': u'constant'}}
++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
+| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
+| authenticate.keystone | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 |
+| total | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 |
++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
+
+(This test step runs a benchmarking Rally task defined by /etc/rally/tasks/example.json)
+
+ $ rally task list
++--------------------------------------+----------------------------+----------+--------+-----+
+| uuid | created_at | status | failed | tag |
++--------------------------------------+----------------------------+----------+--------+-----+
+| 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | 2014-07-07 18:06:30.584805 | finished | False | |
++--------------------------------------+----------------------------+----------+--------+-----+
+
+ $ rally task detailed 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77
+================================================================================
+Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is finished.
+--------------------------------------------------------------------------------
+/etc/rally/tasks/example.json
+test scenario Authenticate.keystone
+args position 0
+args values:
+{u'context': {u'users': {u'concurrent': 30,
+ u'tenants': 3,
+ u'users_per_tenant': 50}},
+ u'runner': {u'concurrency': 5, u'times': 100, u'type': u'constant'}}
++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
+| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
+| authenticate.keystone | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 |
+| total | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 |
++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
+
+ $ rally task plot2html 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 --out output.html
+ $ ls -l output.html
+-rw-r--r-- 1 root root 23028 Jul 7 18:12 output.html
+
+ # scp output.html into a host machine and use Browser on host machine to open
+ this ouput.html file. A nice graphical report for this benchmark is displayed
+ on Browser. Make sure the tale in Section "Table for task results" matches with
+ table printed out by "rally task detailed" above.
+
+
+Rally Built-In Unit Tests
+=========================
+
+This section describes how to run Rally built-in unit
+tests which are located at:
+
+ /usr/lib64/python2.7/site-packages/rally/tests
+ /usr/lib64/python2.7/site-packages/rally/tests_ci
+
+To run Rally built-in unit test with nosetests:
+
+ $ cd /usr/lib64/python2.7/site-packages/rally/tests
+ $ nosetests -v
+
+----------------------------------------------------------------------
+Ran 647 tests in 7.598s
+
+FAILED (errors=3, failures=2)
+
+ $ /usr/lib64/python2.7/site-packages/rally/tests_ci
+ $ nosetests -v
+
+----------------------------------------------------------------------
+Ran 1 test in 2.620s
+
+OK
+
+
+References
+==========
+
+https://wiki.openstack.org/wiki/Rally