Integrate with your Continuous Integration process

Automation with HipTest publisher – continuous integration

Once the tests have been successfully generated from an HipTest project, it is possible to integrate them in a continuous integration process. We will not discuss here the details on how to integrate (as it will change, based on your CI tool) but describe the necessary steps to integrate HipTest in your CI tool.

In this documentation, we will consider the following pre-requisites:

  • you have structured the project for automation and you have been able to generate tests with hiptest-publisher
  • you want to have the results of the HipTest tests execution back to HipTest
  • you already have a CI server and task for your project

Create a test run dedicated to the CI

First, create a test run dedicated to the CI. This test run should only contain the tests that have been fully automated.
This test run should not be used for manual testing, only for the results sent by the CI tool. If you need to complete your validation process with some manual testing, use other test runs to set the results.

Once the test run is created, click on the Hiptest step menu menu, then on the “Automate” action:

Opening the Hiptest automate link

You should now see the following modal:

Automate Hiptest menu content

Note: each command line displayed in the modal can be copied to the clipboard.

There are three command lines displayed in the modal. Let’s detail them a bit further.

Start Hiptest for free banner

Installing hiptest-publisher on the CI

In order to be able to synchronize both the test run and the code, and to push the results back to Hiptest, you need to have hiptest-publisher installed on the machine that will run the tests.
This machine also need to have Ruby installed (as you did previously on the developer machine). Then, you can run the command :

gem install hiptest-publisher

It will install or update the tool. Note that this is a good practice to have this command line ran before executing the tests every time, in order to ensure that the latest version of hiptest-publisher is installed.

Updating the tests from the test run

The second command line should look something like that:

hiptest-publisher --config-file <path to your config file> --test-run-id 215 --without=actionwords --verbose

As the previous command, it must be executed before running the tests, so copy the command in the pre-build part of your CI job configuration.

Its role is to ensure that you are executing the correct version of the tests (the one that is fixed in the test run). It will also add some annotations to the tests, so HipTest will be able to match the results and the tests.

One important thing to note here is the argument “–without=actionwords”. It ensures that only the tests will be regenerated, not the implementation you did for the action words.

Adding “–verbose” is a good practice as it will output more information while generating tests. It can be useful for debugging.

Pushing the results back to HipTest

The third command line should look like this:

hiptest-publisher --config-file <path to your config file> --push <path to your results file> --push-format junit --verbose

Its role is to push the results back to HipTest, so it should be executed after the tests.
One good practice is to copy it in the post-build part of your CI job, not directly after the command running the tests, otherwise it might not be ran if some test fails (that depends on your CI tool configuration).

The command varies according to the type of report generated by your test framework, so ensure you have selected the correct format before copying the command line.

Once every command has been added to the CI job, run it once manually to ensure everything is working, and you should see the results appear in the test run you created previously.


I used CucumberJs that generates a JUnit format but it doesn’t push the test results to HipTest. Why?

When CucumberJS exports JUnit reports, it does not place the test UIDs.
Here’s a quick workaround for this kind of issue:

  •  in packages.json, use this command line to select the cucumberjs-junit package:
"cucumberjs-junit": "git://"
  • when running the tests, use this command line:
node_modules/.bin/cucumber-js --format=json | node_modules/cucumber-junit/bin/cucumber-junit --features-as-suites > junit_output.xml

Now, you can push the junit_output.xml file to HipTest and it should work fine 🙂

I have executed my tests on multiple environments, how can I push the results for these environments back to a test run?

You can specify an environment name in the hiptest-publisher command using the --execution-environment option:

hiptest-publisher --config-file <path to your config file> --push <path to your results file> --test-run-id <id of the test run> --push-format <format of the execution report> --execution-environment <name of the environment>

If the environment with the given name does not exist, a new environment will be created.

Download the ebook

By submitting this form, you agree to our Terms of Use and Privacy Policy

Hiptest is FREE for open source projects

We actually utilize a number of open source components, which is why Hiptest service is free for any open source project. We are giving back to the community!

Your open source project must meet the following criteria to be approved:

  • Your project is licensed under a license approved by the Open Source Initiative
  • Your project source code is available for download
  • Your project has a publicly accessible website