Testing FIWARE Lab Node locally

In order to ensure that the OpenStack instance is installed property, a set of manual tests are executed by the FIWARE Lab team in order to guarantee the correct execution of them. Those E2E tests are based on the historical problems that we found in the FIWARE Lab nodes once it was up and ready in the past. Some of these problems include no route to the virtual machine, unable to execute the cloudinit process or unable to delete properly the allocated resources of a user. Therefore, these E2E tests cover all the problematic scenarios that arose during the FIWARE Lab existence.

The tests are divided into Mandatory and Optional. This is because not all FIWARE Lab nodes implement the Object Storage components and hence these tests are not required for all nodes. The language that is used to specify those scenarios is the Gherkin language^. Gherkin is plain-text language with well-designed structure to define the behaviour of a system. It is designed to the learn and use in a very easy way not only by programmers. Besides, it can describe concisely examples to describe business rules in most real-world domains. An example of Gherkin document is:

Example of Gherkin language

  Feature: Access to the OpenStack instance

    Scenario: Registering a user into OpenStack

      Given the OpenStack account name of a valid user

      And the valid password of this user

      When the user requests access to the OpenStack instance with those data

      Then the OpenStack account service (Keystone) authenticates the user

      And Keystone redirect to the OpenStack Portal (Horizon).

The main keywords In Gherkin are:

  • Feature

  • Scenario

  • Given, When, Then, And, But (Steps)

  • Background

  • Scenario outline

  • Examples

We will take different scenarios in order to complete the lists of tests that we want to apply for each new FIWARE Lab node. Currently those tests are executed manually but It does not mean that it will continue in a manual way, In the future, we will probably adopt a Behaviour Driven Development solution to automatically execute those tests.

Feature 1: Creation and management of Security groups

In this scenario, we want to test the basic operations related to the creation and management of security groups. It is comprised of the following scenarios:

Feature about the creation and management of Security Groups

  Feature 1: Creation and management of Security groups

    Scenario 1.1: Create a Security Group

      Given a registered user in OpenStack environment.

      When the user requests creation of a Security Group with name "SGtest".

      Then OpenStack creates the corresponding Security Group with the name "SGtest".

      And this Security Group has no rules assigned to it.

    Scenario 1.2: Create a Security Group Rules for a Security Group

      Given a registered user in OpenStack environment.

      And a Security Group previously created with name "SGtest".

      When the user requests creation of a Security Group Rule for SSH access.

      Then the OpenStack creates the corresponding Security Group Rules associated to the Security Group "test".

      And this Security Group has the corresponding Security Group Rule associated to it.

Feature 2: Creation of key pair

In this scenario, we want to test the basic operations related to the creation and management of key pair content. It is comprised of the following scenario:

Feature about the creation of key pair

  Feature 2: Creation of key pair

    Scenario 2.1: Create a Key Pair

    Given a registered user in OpenStack environment.

    When the user requests creation of a KeyPair with the name "testKeyPair".

    Then the OpenStack creates the corresponding KeyPair.

    And the user can download it.

Feature 3: Allocate and associate Floating IPs

In this scenario, we want to test the basic operations related to the allocation and association of Floating IPs to a specific user. It is compound the by following scenarios:

Feature about the allocation and association of floating IPs

  Feature 3: Allocate and associate Floating IPs

    Scenario 3.1: Allocate an IP to the user project

      Given a registered user in OpenStack environment.

      When the user requests the allocation of an IP from the pool "public-ext-net-01".

      Then the OpenStack allocates an IP.

    Scenario 3.2: Associate a floating IP to a Virtual Machine

      Given a registered user in OpenStack environment.

      And a previous deployed virtual machine with name "testVM".

      And a previous allocate public IP.

      When the user requests the association of the floating IP to a "testVM".

      Then the OpenStack associates the floating IP with the corresponding Virtual Machine.

Feature 4: Creation of network and associate to a Virtual Machine

In this scenario, we want to test the basic operations related to the creation of networks and routers and associate them to a virtual machine in order to tests if the neutron service is properly configured. It is comprised of the following scenarios:

Feature about creation of network and associate to a Virtual Machine

  Feature 4: Creation of network and associate to a Virtual Machine

    Scenario 4.1: Create a network with its corresponding subnetwork

      Given a registered user in OpenStack environment.

      When the user requests the creation of the network "networktest".

      And the addition of subnet with name "subnettest", network address "195.134.187.0/10" and DNS Name Server "8.8.8.8".

      Then the Neutron service creates the network with the corresponding subnetwork.

    Scenario 4.2: Create a new router

      Given a registered user in OpenStack environment.

      When the user requests the creation of a router "routertest".

      Then the OpenStack creates a new router.

    Scenario 4.3: Assign an interface to the previous router

      Given a registered user in OpenStack environment.

      And a previous created router "routertest".

      And a previous deployed network "networktest" with subnetwork "subnettest".

      When the user requests the addition of a new interface to the router.

      And select this network and subnetwork.

      Then the OpenStack adds the interface to the router.

    Scenario 4.4: Set a gateway to the previous router

      Given a registered user in OpenStack environment.

      And a previous created router "routertest".

      And a public access network "public-ext-net-01".

      When the user requests to set the Gateway to this external network "public-ext-net-01".

      Then the OpenStack sets the Gateway to this router.

    Scenario 4.5: Check the access to a virtual machine

      Given a registered user in OpenStack environment.

      And a previous created network "networktest".

      And a router "reoutertest" properly configured in terms of gateway and interface.

      When the user requests to create a virtual machine associated to this network.

      Then the OpenStack allocates the virtual machine.

      And the user can access through SSH to the new instance.

Feature 5: Working with new Instances

In this scenario, we want to test if a new instance can be deployed and it is correctly managed in terms of network connectivity. We also check access to the OpenStack metadata service in order to receive different actions from it. Last but not least, the test will include the communications that we have to do from the instance to the world in order to be sure that the network interfaces are correctly configured in both directions. It is comprised of the following scenarios:

Feature about working with instances

  Feature 5: Working with Virtual Machines

    Scenario 5.1: Create a new Virtual Machine

      Given a registered user in OpenStack environment.

      And a base image with the name "base_ubuntu_14.04" exists.

      And a flavor "m1.small" exists.

      And a Key pair "testKeyPair" exists.

      And a Security Group "SGtest" exists.

      And a network "node-int-net-01" exists.

      When the user requests the creation of the instance name "testinstance".

      And the previous Flavor, Key pair, Security Group and network are selected.

      Then the OpenStack deploy a virtual machine with that name.

    Scenario 5.2: Delete a virtual machine

      Given a registered user in OpenStack environment.

      And a virtual machine with the name "testinstance" exists.

      When the user requests the deletion of this virtual machine.

      Then the OpenStack deletes the virtual machine without any problem.

    Scenario 5.3: Create again a new Virtual Machine

      Given a registered user in OpenStack environment.

      And a base image with the name "base_ubuntu_14.04" exists.

      And a flavor "m1.small" exists.

      And a Key pair "testKeyPair" exists.

      And a Security Group "SGtest" exists.

      And a network "node-int-net-01" exists.

      And a virtual machine with the name "testinstance" was previously deleted.

      When the user requests the creation of the instance name "testinstance".

      And the previous Flavor, Key pair, Security Group and network are selected.

      Then the OpenStack deploy a virtual machine with that name.

    Scenario 5.4: Create a Snapshot from a virtual machine

      Given a registered user in OpenStack environment.

      And a virtual machine with the name "testinstance" exists.

      When the user requests the creation of a snapshot of that image with name "demo-instance-snapshot".

      Then the OpenStack creates the snapshot with this name.

    Scenario 5.5: Create a virtual machine from a snapshot image

      Given a registered user in OpenStack environment.

      And a previous snapshot with the name "demo-instance-snapshot".

      And a flavor "m1.small" exists.

      And a Key pair "testKeyPair" exists.

      And a Security Group "SGtest" exists.

      And a network "node-int-net-01" exists.

      And a virtual machine with the name "testinstance" does not exist.

      When the user requests the creation of the instance name "testinstance2".

      And the previous Flavor, Key pair, Security Group and network are selected.

      Then the OpenStack deploy a virtual machine with that name.

    Scenario 5.6: Check the access to a virtual machine created from an image

      Given a registered user in OpenStack environment.

      And a previously created virtual machine "testinstance2" exists.

      And a public IP is available from the pool "public-ext-net-01".

      When the user requests to associate this public IP to the virtual machine.

      Then the OpenStack associates them.

      And the user can access through SSH to the new instance.

    Scenario 5.7: Check the access to the metadata service from a virtual machine

      Given a registered user in OpenStack environment.

      And a previously created virtual machine "testinstance2" exists.

      And this virtual machine is accessible using SSH.

      When the user accesses to the virtual machine using SSH.

      And request the access to the metadata service through the command "curl http://169.254.169.254/1.0/meta-data".

      Then command line responses with the corresponding information about the metadata service.

    Scenario 5.8: Check the access to the world from a deployed virtual machine

      Given a registered user in OpenStack environment.

      And a previously created virtual machine "testinstance2" exists.

      And this virtual machine is accessible using SSH.

      When the user accesses to the virtual machine using SSH.

      And the user executes a simple ping command "ping www.google.com".

      Then ping command responses correctly with the information of this server.

Feature 6: Working with volumes

In this scenario, we want to test the creation and association of volumes work properly on the new OpenStack environment. It is comprised of the following scenarios:

Feature about working with volumes

  Feature 6: Check volume management

    Scenario 6.1: Check the creation of a volume

      Given a registered user in OpenStack environment.

      When the user requests the creation of a volume with the name "testvolume".

      And the description "a test volume to be deleted".

      And a size of 1 Gb.

      Then the OpenStack creates the volume properly with status "available".

    Scenario 6.2: Delete a volume

      Given a registered user in OpenStack environment.

      And a volume with name "testvolume" exists.

      When the user requests the deletion of this volume.

      Then the OpenStack deletes the volume properly.

    Scenario 6.3: Attach a volume to an instance

      Given a registered user in OpenStack environment.

      And a virtual machine with the name "testinstance2" exists.

      And a volume with the name "testvolume" is created properly.

      When the user tries to attach the volume to the instance.

      Then the OpenStack attaches the volume to the instance.

      And the status is "in-use".

      And the attachment is "1".

    Scenario 6.4: Check a volume to an instance

      Given a registered user in OpenStack environment.

      And a virtual machine with the name "testinstance2" exists.

      And a volume with the name "testvolume" is attached to this virtual machine.

      When the user accesses to the virtual machine through SSH client.

      And execute the command "sudo fdisk -l".

      Then the commands response with the current partition table.

      And it contains the description of a disk "Disk /dev/vdb: 1073 MB, 1073741824 bytes".

      And it says us "Disk /dev/vdb doesn't contain a valid partition table" which means that it is ready to mount this new partition.

Feature 7: Check list of images

In this scenario, we want to test if the node has installed the list of base images. They are the only base images that are allowed to be used in the FIWARE Lab environment for security reasons. It is comprised of the following scenario:

Feature about checking the list of images

  Feature 7: Check image list

    Scenario 7.1: Check the current list of image list

      Given a registered user in OpenStack environment.

      When the user requests the list of available images.

      Then the OpenStack response with the list of available images.

      And it should contain "base_ubuntu_16.04", "base_ubuntu_14.04", "base_debian_8" and "base_centos_7".

Feature 8: Object Storage management

In this scenario, we want to test if we can work with object storage service. This is an optional test and only will be executed if the FIWARE Lab node currently install the corresponding service. It is comprised of the following scenario:

Feature about object storage management

  Feature 8: Object Storage management

    Scenario 8.1: Create a container

      Given a registered user in OpenStack environment.

      When the user requests the creation of a new container with the name "testcontainer".

      Then the OpenStack creates the container.

      And the number of Objects is "0".

      And the Size is "0 bytes".

    Scenario 8.2: Upload a text file on a container

      Given a registered user in OpenStack environment.

      And a container with the name "testcontainer" exists.

      And this container is empty.

      When the user requests the upload of a simple text file on this container.

      And the name is "testfile".

      Then the OpenStack uploads the file into the container.

      And the after 1 minute, the Objects number change to "1".

      And the Size is not "0 bytes".

    Scenario 8.3: Download an object from a Container

      Given a registered user in OpenStack environment.

      And a container with the name "testcontainer" exists.

      And an object with the name "testfile" exists.

      When the user requests the download of the object.

      Then the OpenStack downloads the object.

      And the user can open properly and see the content of the file.

    Scenario 8.4: Delete an object from a Container

      Given a registered user in OpenStack environment.

      And a container with the name "testcontainer" exists.

      And an object with the name "testfile" exists.

      And this is the only object on that container.

      When the user requests the deletion of the object.

      Then the OpenStack deletes the object.

      And the after 1 minute, the Objects number change to "0".

      And the Size is "0 bytes".

    Scenario 8.5: Delete a Container

      Given a registered user in OpenStack environment.

      And a container with the name "testcontainer" exists.

      And this container is empty.

      When the user requests the deletion of the container.

      Then the OpenStack deletes the container.

Report the results of the execution of the tests

The report consists of a detailed description of the manual tests provided to the node together with the table of the results of the execution of those tests. As it is possible that there may be several errors in different iterative steps, the report will keep the previous result of the execution of the tests.

Sometimes if the number of errors is very high, we can decide to stop the execution of the tests and inform to the future FIWARE Lab node that they have to resolve the current errors in order to follow with the manual execution of the tests.

Additionally, the results will include some comments about the problem that was found and in same case, if the reporter knows the solution, instructions to correct the problem and pass the scenario’s test.

You can see in the Annex A: Template of report of the Testing FIWARE Lab Node locally, the format of this report.