Skip to content

Development Continuous Integration tests

Jump to bottom
Pekka Aho edited this page Apr 13, 2022 · 4 revisions

TESTAR Continuous Integration Workflow

The TESTAR project uses the GitHub Actions Workflow feature together with a serie of Gradle tasks to test that the tool starts, connects and correctly detects desktop and web applications widgets.

The workflow file .github/workflows/gradle.yml contains three jobs to build TESTAR inside a Windows, Linux and MacOS environments using Java 1.8.
Windows job also contains multiple steps that run gradle tasks specifically defined to test different TESTAR functionalities.

Gradle tasks

testar/build.gradle file contains all these different test tasks grouped in group = 'test_testar_workflow'.

These tasks run TESTAR using Windows command prompt, read the messages from the TESTAR output buffer, and finally determines if the tasks have completed successfully based on the output buffer information.

Example of a job step:

name: Run desktop_generic protocol with TESTAR using COMMAND_LINE
run: ./gradlew runTestDesktopGenericCommandLineOk

that executes a gradle task:

// Default COMMAND_LINE execution to run Notepad
// Test TagsForSuspiciousOracle feature using the UIATags.UIAAutomationId
// Force TESTAR to find a Suspicious Title (MenuBar widget)
// And verify it reading output command line
task runTestDesktopGenericCommandLineSuspiciousTitle(type: Exec, dependsOn:'installDist') {
    // Save TESTAR output buffer information
    standardOutput = new ByteArrayOutputStream()
    // Assign this task to the workflow group
    group = 'test_testar_workflow'
    description ='runTestDesktopGenericCommandLineSuspiciousTitle'
    // Indicate the TESTAR distributed folder directory
    workingDir 'target/install/testar/bin'
    // Prepare a TESTAR command line execution, with the desired protocol and the required settings
    commandLine 'cmd', '/c', 'testar sse=test_gradle_workflow_desktop_generic AlwaysCompile=true ApplicationName="notepad_command_and_suspicious" TagsForSuspiciousOracle="NotExist;UIAAutomationId" SuspiciousTitles=".*MenuBar.*" ShowVisualSettingsDialogOnStartup=false Mode=Generate SUTConnector=COMMAND_LINE SUTConnectorValue="C:\\\\Windows\\\\System32\\\\notepad.exe" Sequences=1 SequenceLength=2'

    // After TESTAR execution
    doLast {
        // Read TESTAR output buffer information to
        String output = standardOutput.toString()
        // Verify MenuBar UIAAutomationId has been detected
        if(output.readLines().any{line->line.contains("Discovered suspicious widget 'UIAAutomationId' : 'MenuBar")}) {
            println "\n${output} \nTESTAR has successfully detected MenuBar Suspicious Title using TagsForSuspiciousOracle UIAAutomationId! "
        } else {
            throw new GradleException("\n${output} \nERROR: TESTAR didnt detect MenuBar Suspicious Title using TagsForSuspiciousOracle UIAAutomationId")
        }
    }
}
  • Test different TESTAR features will require multiple tasks with different protocols and settings.
  • Some TESTAR features have additional software dependecies, such as Selenium WebDriver or OrientDB. The use of these software should be defined as additional and dependent tasks.

TESTAR protocols and settings

TESTAR tool makes use of a Java and test.settings protocol files to define the behaviour of the interaction with the System Under Test (SUT), something that we will need to configure if we want to test specific TESTAR features.

To allow the customization of these protocols, the folder testar/resources/workflow/settings has been created. These protocols are not copied with the default TESTAR tool compilation, it is necessary to initialize the copy of these protocols with the existing gradle task init_workflow_test:

  1. Prepare TESTAR protocols: gradle init_workflow_test
  2. Build TESTAR: gradle build
  3. Prepare installed distribution of TESTAR: gradle installDist
  4. Run desktop_generic protocol with TESTAR using COMMAND_LINE: gradle runTestDesktopGenericCommandLineOk 
- name: Prepare TESTAR protocols
  run: ./gradlew init_workflow_test
- name: Build TESTAR with Gradle
  run: ./gradlew build
- name: Prepare installed distribution of TESTAR with Gradle
  run: ./gradlew installDist
- name: Run desktop_generic protocol with TESTAR using COMMAND_LINE
  run: ./gradlew runTestDesktopGenericCommandLineOk

TESTAR software dependencies

All jobs are executed inside GitHub virtual environments on which it is necessary to download and prepare some software such as Selenium WebDriver executable and an OrientDB testar database.

In our case we have created additional gradle tasks to download and execute these configurations:

  • Webdriver gradle tasks download last chromedriver version with the use of the dependency dependsOn: downloadAndUnzipChromedriver 
// Download last win32 chromedriver 
task downloadChromedriver(type: Download, dependsOn: installDist) {
    group = 'test_testar_workflow'
    description ='downloadChromedriver'

    // This web URL give us the version information we need
    ProcessBuilder builder = new ProcessBuilder().command('cmd', '/c', 'curl https://chromedriver.storage.googleapis.com/LATEST_RELEASE');
    Process process = builder.start();
    process.waitFor()

    // Download last chromedriver executable, but do not overwrite if already exists
    src 'https://chromedriver.storage.googleapis.com/' + process.text + '/chromedriver_win32.zip'
    dest 'target/install/testar/bin'
    overwrite false
}

// Extract Chromedriver from the downloaded file
task downloadAndUnzipChromedriver(type: Copy, dependsOn: downloadChromedriver) {
    group = 'test_testar_workflow'
    description ='downloadAndUnzipChromedriver'

    // Extract chromedriver executable from the zip file
    from zipTree(new File(downloadChromedriver.dest, 'chromedriver_win32.zip'))
    into downloadChromedriver.dest
}
  • State Model OrientDB tasks download version 3.0.34 and creating a new testar database with the use of the dependency dependsOn: createDatabaseOrientDB 
// Download OrientDB 3.0.34 from the official wep page
task downloadOrientDB(type: Download, dependsOn: installDist) {
    // This orientdb3 URL contains the specific version we need
    src 'https://s3.us-east-2.amazonaws.com/orientdb3/releases/3.0.34/orientdb-3.0.34.zip'
    dest 'target/install/testar/bin'
    overwrite false
}

// Verify that we downloaded the correct OrientDB file
task verifyOrientDB(type: Verify, dependsOn: downloadOrientDB) {
    // Because we always use the same orientdb3 URL we can verify this is the desired file
    src new File(downloadOrientDB.dest, 'orientdb-3.0.34.zip')
    algorithm 'MD5'
    checksum 'a0619e5522e7d849d4f1067bf08e3857'
}

// Extract OrientDB binary files
task downloadAndUnzipOrientDB(type: Copy, dependsOn: verifyOrientDB) {
    group = 'test_testar_workflow'
    description ='downloadAndUnzipOrientDB'

    // Extract all OrientDB files from the zip
    from zipTree(new File(downloadOrientDB.dest, 'orientdb-3.0.34.zip'))
    into downloadOrientDB.dest
}

// Execute a command line to create a testar OrientDB database with admin admin credentials
task createDatabaseOrientDB(type: Exec, dependsOn: downloadAndUnzipOrientDB){
    group = 'test_testar_workflow'
    description ='createDatabaseOrientDB'

    // Save output and errpr buffer information
    standardOutput = new ByteArrayOutputStream()
    errorOutput = new ByteArrayOutputStream()
    workingDir 'target/install/testar/bin/orientdb-3.0.34/bin'
    commandLine 'cmd', '/c', 'console.bat CREATE DATABASE plocal:../databases/testar admin admin'
    // Ignore errors creating the database because we check the output buffer message
    ignoreExitValue true
    doLast {
        String output = standardOutput.toString()
        
        // If error creating database because already exists is ok
        if (execResult.getExitValue()==1 && errorOutput.toString().contains("Cannot create new database 'testar' because it already exists")) {
            println "\n${output} \ntestar OrientDB database already exists"
        }
        // Check if testar database created sucessfully
        else if(output.readLines().any{line->line.contains("Database created successfully.")}) {
            println "\n${output} \ntestar OrientDB database created successfully"
        } else {
            throw new GradleException("\n${output} \nERROR: Creating testar OrientDB database")
        }
    }
}

TESTAR artifact result

GitHub Actions Workflow feature also allow us to define and upload a file or folder as a result of a job step. Basically we want to indicate that the artifact of the test step is the TESTAR output folder, that contains the HTML, screenshots, logs, etc...

name: Save runTestDesktopGenericCommandLineSuspiciousTitle HTML report artifact
uses: actions/upload-artifact@v2
with:
  name: runTestDesktopGenericCommandLineSuspiciousTitle-artifact
  path: D:/a/TESTAR_dev/TESTAR_dev/testar/target/install/testar/bin/notepad_command_and_suspicious

Because it is necessary to indicate the naming of the specific path to be uploaded as an artifact and because the name of the TESTAR output folder is based on the dynamic timestamp, we are doing a copy of the output folder name based on the ApplicationName setting value.

// Example: testar/resources/workflow/settings/test_gradle_workflow_desktop_generic/Protocol_test_gradle_workflow_desktop_generic.java
/**
* This method is called after the last sequence, to allow for example handling the reporting of the session
*/
@Override
protected void closeTestSession() {
   super.closeTestSession();
   try {
      // Copy original TESTAR folder result testar/bin/output/timestamp
      File originalFolder = new File(OutputStructure.outerLoopOutputDir).getCanonicalFile();
      // To the desired Artifact testar/bin/ApplicationName
      File artifactFolder = new File(Main.testarDir + settings.get(ConfigTags.ApplicationName,""));
      FileUtils.copyDirectory(originalFolder, artifactFolder);
   } catch(Exception e) {System.out.println("ERROR: Creating Artifact Folder");}
}

This means that is better to use different ApplicationName setting values in the gradle test tasks to have different TESTAR artifacts for the test tasks.

  • gradle task runTestDesktopGenericCommandLineSuspiciousTitle

uses the ApplicationName="notepad_command_and_suspicious

  • Windows job step name: Save runTestDesktopGenericCommandLineSuspiciousTitle HTML report artifact

saves the artifact from path: D:/a/TESTAR_dev/TESTAR_dev/testar/target/install/testar/bin/notepad_command_and_suspicious

Clone this wiki locally