I have been using JIRA Software for the past few years for issue tracking, and overall I have been very happy with it. For testing management, in the past, I have used TestLink, but decided it was time I looked into finding something that integrated with JIRA so I could manage all of my testing in one place. A quick search yielded the three main test management add-ons for JIRA are: Zephyr, Xray and SynapseRT and I have looked into all three. Because many comparisons have already been written for Zephyr, I decided that I would focus this post on comparing Xray and SynapseRT.
When I first started researching Xray and SynapseRT, by looking over the documentation, I found the terminology much easier to understand for SynapseRT. In Xray, they use terms like Test, Test Set, Text Execution and Test Run, while in SynapseRT they use terms like Test Case, Test Suite, Test Cycle, Test Plan and Test Case Run. Even though the concepts can be mapped between the two, I find the language of SynapseRT much more familiar.
|Test||Test Case||A test case scenario|
|Test Set||Test Suite||A way of many a group of test cases|
|Test Execution||Test Cycle||A group of test cases to be executed|
|Test Plan||Test Plan||A group of test cycles to executed|
|Test Run||Test Case Run||The execution of a test case|
Creating a Test Case
Both SynapseRT and Xray implement their test cases as JIRA issue types, which leverages all of the functionality that goes along with that, including being able to add custom fields, etc. Creating a test case is quite similar between the two: they both allow for adding steps that include both test data, and expected results. Both also allow for adding attachments, but Xray has a slight advantage here, because it allows you to add attachments not only to the test case but also at each step level, which may be useful. Xray also allows you to specify whether a test is manual or automated, but this could can be done in SynapseRT using a custom field.
One big difference between the two, is how they manage test case pre-conditions. SynapseRT does not have any concept of pre-conditions, so if you need pre-conditions you need to create a custom field. Xray, on the other hand, treats pre-conditions as a distinct issue type. This has the advantage of re-use; if the same pre-conditions are used in many tests, you can create the Pre-Conditions issue type, and assign it the various tests.
Organizing Test Cases
Any but the most simple project will have dozens, to hundreds of test cases, and so being able to organize them is important. Xray has the concept of a Test Set, which is also a JIRA issue type. Once you create a Test Set, you can add Tests to it, and since it is an issue type, you can also add a description and other information to it. By creating several Test Sets you can group your tests into different categories.
SynapseRT, on the other hand, does not have a separate issue type to organize tests, but instead has the concept of a Test Suite (again terminology I’m more familiar with). You can create a Test Suite, and then at Test Cases to it. The nice thing about Test Suites is that they can be nested, and so you can create a hierarchy of Test Cases, similar to creating a folder tree, with Test Cases as the files. Overall I found this approach by SynapseRT to be a far superior way of organizing test cases.
When it comes to test planning, Xray uses another issue type for this, called a Test Execution. You create a Test Execution issue type and then you can add Tests to this Test Execution. Adding Tests is quite straight forward, and can either be added as individual Tests, or as Test Sets. Once a Test Execution has been created, individual Tests can be executed and tracked.
As of version 2.0, Xray has introduced a Test Plan issue type. You create a Test Plan, and can then assign Tests to it. Then, from within the Test Plan, you can create one or more Test Executions with some or all of the Tests that have been added to the Test Plan. This approach almost makes the Test Executions, like a classic test cycle, but since Test Executions can be created independently as well, I find this approach a bit confusing.
In test planning, SynapseRT again takes a slightly different approach. Instead of separate Test Execution and Test Plan issue types, it just has the Test Plan issue type. Once you have created a Test Plan, you can add Test Cases, either as individual Test Cases, or from a Test Suite. I found the adding of tests to be a little more slick in Xray, but it works well enough in SynapseRT. Once you have added your Test Cases to the Test Plan, you can then add one or more Test Cycles to the Test Plan, and then decide which Test Cases to be included in each Test Cycle. Again, for my software testing processes, this approach of a test plan, with multiple test cycles works much better, as I can create a test plan for a planned release, but then run several test cycles within it.
Once the various tests have been planned, both Xray and SynapseRT allow you to assign testing work to testers, this can either be done at a Test Execution/Test Cycle level or at an individual Test/Test Case level.
Executing tests for both Xray and SynapseRT are quite similar. Once a test case has been added to a Test Execution/Test Run, you can execute the test. When executing a test, both Xray and SynapseRT, display the test case with its test steps, and allow you to mark each step as Pass or Fail (SynapseRT also allows for Blocked and NA, while Xray allows for Executing and Aborted ). To save time, both allow you to batch mark all the steps in a test case as well.
If a defect is found during testing, both Xray and SynapseRT allow you to either link an existing issue to the test execution or to create a new issue. The issue then gets linked to the test case, and shows up as part of the test execution and part of test reports.
While both offer similar functionality, Xray’s interface for this looks a little slicker, and I especially like that when passing/failing tests, you only need one click, instead of clicking, and selecting from a drop down list, like in SynapseRT.
Tracking and Reporting
Tracking testing progress, for both Xray and SynapseRT, can be done by viewing the Test Plan. Again, each takes a slightly different approach.
In Xray, the focus is on individual test cases, where each test is listed with its latest status. Each test can also be expanded to view the Test Executions that the test has been run in. The overall execution status also just lists the latest status for each test.
In SynapseRT, the focus is more on the Test Cycles. Information is displayed in a grid, with the Test Cases, along with results for each Test Cycle, making it easier to see the status of each test case in its various Test Cycles. Below the main grid, the status of each Test Cycle is shown, including the number of Test Cases that passed/failed/etc. and the status of any defects that were logged during the Test Cycle.
SynapseRT also includes several Dashboard gadget for tracking Test Plan and Test Cycle status, including a Test Case burn down gadget that shows the number of Test Cases that are remaining to be executed for each test.
Because both tools use native JIRA issues types, JIRA filters and reports can also be used for tracking and reporting.
Both add-on also include Requirements management and traceability, but this is something I didn’t evaluate to any large extent, and so I didn’t include it in this review.
SynapseRT and Xray are available for purchase from the Atlassian Marketplace. As is common practice for add-ons, both offer a starter price of $10 for 10 user, and for me, as independent software developer, is my price point. As the number of users goes up, however, Xray starts to get more expensive, for example at 50 Users, Xray is $880 while SynapseRT is $590, so if you have a larger organization, the price may be something to consider.
|Test Process Workflow|
|Look and Feel|
After evaluating both tools, I found that they both do a good job of creating, managing and executing tests, and I feel I would be satisfied with either. Xray has a slightly more polished UI, and now that they have added Test Plans is quite good. Overall, however, I found SynapseRT seems to match my testing practices better, and the general workflow seems more intuitive to me. I also like the results reporting betting, in SynapseRT. Both are available on trial basis and Xray also has a Sandbox server to play with, so my best suggestion would be to try both and see which works better for you.