diff options
Diffstat (limited to 'meta-openstack/Documentation/README.rally')
| -rw-r--r-- | meta-openstack/Documentation/README.rally | 335 |
1 files changed, 335 insertions, 0 deletions
diff --git a/meta-openstack/Documentation/README.rally b/meta-openstack/Documentation/README.rally new file mode 100644 index 00000000..352ef253 --- /dev/null +++ b/meta-openstack/Documentation/README.rally @@ -0,0 +1,335 @@ +Summary +======= + +This document is not intended to provide detail of how Rally in general +works, but rather it highlights how Rally is integrated/configured into +meta-cloud-services and also describes how Rally is tested to ensure that +Rally Verification and Benchmarking components are working correctly. + + +Rally Overview +============== + +Rally - OpenStack benchmarking at a scale - is intended to provide the +community with a benchmarking tool that is capable of performing "specific", +"complicated" and "reproducible" test cases on "real deployment" scenarios. + +Rally has 4 main components: + +* Server Providers - provides virtual servers +* Deploy Engines - deploys OpenStack cloud on servers that are presented by Server Providers +* Verification - runs tempest +* Benchmarking - allows to write parameterized benchmark scenarios & run them against the cloud + + +Rally Deployment +================ + +Rally is configured to use existing deployment (by using deployment configuration file +/etc/rally/deployments/existing.json) and therefore Rally components "Server Providers" +and "Deploy Engines" are not used heavily. + + +Rally Verification +================== + +By default, Rally verification performs the following steps: + +* git clone tempest source from upstream +* setup virtualenv for this tempest +* setup testr environment with virtualenv created above +* create tempest.conf for this tempest +* use testr and subunit.run module to run tempest + +However, meta-cloud-services already includes tempest which is also configured/modified +to have low failure/error testcases, therefore it's desired to use this tempest +(without using virtualenv) instead of letting Rally to download tempest and running it +on virtualenv. + +The option "existing_tempest_config" in /etc/rally/rally.conf can be used to configure +Rally to either use the existing tempest or to download from upstream. + +If the option "existing_tempest_config" is not set then Rally follows the default path. +If "existing_tempest_config" is set to absolute path of tempest config folder (which +contains tempest "tools" and .testr.conf, e.g. /etc/tempest) then Rally uses this +existing tempest. By default, "existing_tempest_config" is set to "/etc/tempest/". + + +Build Configuration Options +=========================== + +To have Rally and tempest included in final built image, include layer +meta-openstack-controller-test-config into Controller build and +layer meta-openstack-compute-test-config into Compute build. + + +Test Steps +========== + +Please note: the following commands/steps are carried on Controller +node, unless otherwise explicitly indicated. + + # Start Controller and Compute node + $ ps aux | grep rally +root 13954 4.2 0.0 130856 27500 ? Ss 17:05 0:00 python /usr/bin/rally-api --log-dir=/var/log/rally + + $ curl http://localhost:8877 +{"versions": [{"status": "CURRENT", "media_types": [{"base": "application/json", "type": "application/vnd.openstack.rally.v1+json"}], "id": "v1", "links": [{"href": "http://localhost:8877/v1", "rel": "self"}], "updated_at": "2014-01-07T00:00:00Z"}], "description": "Rally is a Benchmark-as-a-Service project for OpenStack.", "name": "OpenStack Rally API"} + + $ rally deployment create --filename=/etc/rally/deployments/existing.json --name=wr-ops ++--------------------------------------+----------------------------+--------+------------------+--------+ +| uuid | created_at | name | status | active | ++--------------------------------------+----------------------------+--------+------------------+--------+ +| 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | 2014-07-07 17:09:11.160741 | wr-ops | deploy->finished | | ++--------------------------------------+----------------------------+--------+------------------+--------+ + + $ rally deployment list ++--------------------------------------+----------------------------+--------+------------------+--------+ +| uuid | created_at | name | status | active | ++--------------------------------------+----------------------------+--------+------------------+--------+ +| 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | 2014-07-07 17:09:11.160741 | wr-ops | deploy->finished | * | ++--------------------------------------+----------------------------+--------+------------------+--------+ + + $ rally use deployment --uuid=5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb +Using deployment: 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb +~/.rally/openrc was updated + + $ rally deployment check +keystone endpoints are valid and following services are available: ++------------+----------------+-----------+ +| services | type | status | ++------------+----------------+-----------+ +| barbican | keystore | Available | +| ceilometer | metering | Available | +| cinder | volume | Available | +| ec2 | ec2 | Available | +| glance | image | Available | +| heat | orchestration | Available | +| heat-cfn | cloudformation | Available | +| keystone | identity | Available | +| neutron | network | Available | +| nova | compute | Available | +| swift | object-store | Available | ++------------+----------------+-----------+ + + $ rally show flavors ++----+-----------+-------+----------+-----------+-----------+ +| ID | Name | vCPUs | RAM (MB) | Swap (MB) | Disk (GB) | ++----+-----------+-------+----------+-----------+-----------+ +| 1 | m1.tiny | 1 | 512 | | 1 | +| 2 | m1.small | 1 | 2048 | | 20 | +| 3 | m1.medium | 2 | 4096 | | 40 | +| 4 | m1.large | 4 | 8192 | | 80 | +| 5 | m1.xlarge | 8 | 16384 | | 160 | ++----+-----------+-------+----------+-----------+-----------+ + + $ rally show images ++--------------------------------------+--------------+----------+ +| UUID | Name | Size (B) | ++--------------------------------------+--------------+----------+ +| 3ec2f841-b179-47d7-908a-edb766b65c5e | myfirstimage | 13167616 | ++--------------------------------------+--------------+----------+ + + $ rally show keypairs ++------+-------------+ +| Name | Fingerprint | ++------+-------------+ ++------+-------------+ + + $ rally show networks ++--------------------------------------+--------------------------------------------+------+ +| ID | Label | CIDR | ++--------------------------------------+--------------------------------------------+------+ +| 7a9cf643-d9ab-404e-8155-65134bd937fb | TEMPEST_NET | None | ++--------------------------------------+--------------------------------------------+------+ + + $ rally verify list +There are no results from verifier. To run a verifier, use: +rally verify start + + $ rally verify start +Configuring existing Tempest for deployment 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb +Verifier has been successfully configured to use existing Tempest +tempest.api.network.test_extensions.ExtensionsTestXML + test_list_show_extensions[gate,smoke] FAIL +setUpClass (tempest.api.network.test_networks + BulkNetworkOpsJSON) FAIL +tempest.api.orchestration.stacks.test_stacks.StacksTestJSON + test_stack_crud_no_resources[gate,smoke] OK 6.56 + test_stack_list_responds[gate,smoke] OK 0.02 +setUpClass (tempest.api.network.test_load_balancer + LoadBalancerXML) FAIL +tempest.api.compute.test_quotas.QuotasTestJSON + test_get_default_quotas[gate,smoke] OK 1.10 + test_get_quotas[gate,smoke] OK 0.02 +tempest.api.identity.admin.v3.test_domains.DomainsTestJSON + test_create_update_delete_domain[gate,smoke] OK 5.67 + +... + +(Start Rally verification by running all on tempest testcases which are marked as smoke) + + $ rally verify list ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ +| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status | ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ +| a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished | ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ + + $ rally verify show a5575bd2-d132-4522-b625-eed324396dda +Total results of verification: + ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ +| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status | ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ +| a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished | ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ + +Tests: + ++--------------------------------------------------------------------------------------------------------------------------------------------+------------+--------+ +| name | time | status | ++--------------------------------------------------------------------------------------------------------------------------------------------+------------+--------+ +| process-returncode | 0.0 | FAIL | +| setUpClass (tempest.api.compute.servers.test_create_server.ServersTestJSON) | 0.0 | FAIL | +| setUpClass (tempest.api.compute.servers.test_create_server.ServersTestManualDisk) | 0.0 | FAIL | +| setUpClass (tempest.api.compute.servers.test_create_server.ServersTestXML) | 0.0 | FAIL | +| setUpClass (tempest.api.network.test_floating_ips.FloatingIPTestJSON) | 0.0 | FAIL | +| setUpClass (tempest.api.network.test_floating_ips.FloatingIPTestXML) | 0.0 | FAIL | +| setUpClass (tempest.api.network.test_load_balancer.LoadBalancerJSON) | 0.0 | FAIL | + +... + + $ rally verify results a5575bd2-d132-4522-b625-eed324396dda +(lot of details is printed out. Just make sure that there is no error when running this command) + + $ rally verify detailed a5575bd2-d132-4522-b625-eed324396dda +(the out put is the same to the output of command "rally verify show" plus all the error/warning +messages from running tempest) + + $ rally verify start --set identity +root@controller:~# rally verify start --set identity +tempest.api.identity.admin.test_services.ServicesTestJSON + test_create_get_delete_service[gate,smoke] OK 1.06 + test_list_services[gate,smoke] OK 0.82 +tempest.api.identity.admin.v3.test_services.ServicesTestJSON + test_update_service[gate] OK 3.65 +tempest.api.identity.admin.test_tenant_negative.TenantsNegativeTestJSON + test_create_tenant_by_unauthorized_user[gate,negative] OK 1.71 + test_create_tenant_request_without_token[gate,negative] OK 0.73 + test_create_tenant_with_empty_name[gate,negative] OK 0.89 + +... + +(Start Rally verification by running all tempest testcases in tempest/api/identiy) + + $ rally verify list ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ +| UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status | ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ +| a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished | +| b05bbc4c-eab4-4296-9020-d8305aa6457b | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | identity | 216 | 0 | 2014-07-07 17:59:35.358702 | finished | ++--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ + + $ rally -v task start /etc/rally/tasks/example.json +================================================================================ +Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is started +-------------------------------------------------------------------------------- +2014-07-07 18:06:30.608 9641 INFO rally.orchestrator.api [-] Benchmark Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 on Deployment 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb +2014-07-07 18:06:30.609 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Check cloud. +2014-07-07 18:06:30.766 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Completed: Check cloud. +2014-07-07 18:06:30.766 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation. +2014-07-07 18:06:30.785 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation of scenarios names. +2014-07-07 18:06:30.788 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Completed: Task validation of scenarios names. +2014-07-07 18:06:30.788 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation of syntax. + +... + +================================================================================ +Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is finished. +-------------------------------------------------------------------------------- + +test scenario Authenticate.keystone +args position 0 +args values: +{u'context': {u'users': {u'concurrent': 30, + u'tenants': 3, + u'users_per_tenant': 50}}, + u'runner': {u'concurrency': 5, u'times': 100, u'type': u'constant'}} ++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ +| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count | ++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ +| authenticate.keystone | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | +| total | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | ++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ + +(This test step runs a benchmarking Rally task defined by /etc/rally/tasks/example.json) + + $ rally task list ++--------------------------------------+----------------------------+----------+--------+-----+ +| uuid | created_at | status | failed | tag | ++--------------------------------------+----------------------------+----------+--------+-----+ +| 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | 2014-07-07 18:06:30.584805 | finished | False | | ++--------------------------------------+----------------------------+----------+--------+-----+ + + $ rally task detailed 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 +================================================================================ +Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is finished. +-------------------------------------------------------------------------------- +/etc/rally/tasks/example.json +test scenario Authenticate.keystone +args position 0 +args values: +{u'context': {u'users': {u'concurrent': 30, + u'tenants': 3, + u'users_per_tenant': 50}}, + u'runner': {u'concurrency': 5, u'times': 100, u'type': u'constant'}} ++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ +| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count | ++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ +| authenticate.keystone | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | +| total | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | ++-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ + + $ rally task plot2html 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 --out output.html + $ ls -l output.html +-rw-r--r-- 1 root root 23028 Jul 7 18:12 output.html + + # scp output.html into a host machine and use Browser on host machine to open + this ouput.html file. A nice graphical report for this benchmark is displayed + on Browser. Make sure the tale in Section "Table for task results" matches with + table printed out by "rally task detailed" above. + + +Rally Built-In Unit Tests +========================= + +This section describes how to run Rally built-in unit +tests which are located at: + + /usr/lib64/python2.7/site-packages/rally/tests + /usr/lib64/python2.7/site-packages/rally/tests_ci + +To run Rally built-in unit test with nosetests: + + $ cd /usr/lib64/python2.7/site-packages/rally/tests + $ nosetests -v + +---------------------------------------------------------------------- +Ran 647 tests in 7.598s + +FAILED (errors=3, failures=2) + + $ /usr/lib64/python2.7/site-packages/rally/tests_ci + $ nosetests -v + +---------------------------------------------------------------------- +Ran 1 test in 2.620s + +OK + + +References +========== + +https://wiki.openstack.org/wiki/Rally |