Trundl | June 17, 2021 | Guide
Xray is a pragmatic Test Management software trusted by several organizations of global repute. Follow these steps to use Xray’s features in Jira for improved Test Management.
Test Management is the process of managing various phases of testing with a view to maintain quality, efficient testing of a software application. The method involves organizing, controlling, and ensuring traceability of the testing process in order to deliver high-quality products. To accomplish this goal, it is pivotal that teams work within a proven process framework with a tool that is both effective and simple. Read on to know more…
Contents
- An overview of Test Case Management for Jira, powered by Xray
- Xray & Jira: Best of both worlds
- Take advantage of Xray / Jira marriage
- Benefits of Test Case Management with Xray in Jira
- Getting started with initial setup for Xray in Jira
- Creating a manual Test Case
- What a full Jira view looks like
- What if one of the tests fail?
- FAQ
- Know your experts
- Need help with Xray in Jira?
An overview of test case management for Jira, powered by Xray
About Xray: Xray, the leading quality assurance and test management application for Jira, is available for all three hosting types: Cloud, Server and Data Center, and offers complete traceability features. With 6000 customers, over 4.5 million active users in 70 countries and more than 130 of its client organizations featuring in the fortune 500 companies list, Xray is also one of the only test management solutions that grew in terms of market share in the last quarter. Delivering over 200 different features, Xray currently has over a hundred million test cases configured within the tool. Offering a high degree of flexibility, Xray supports both manual and automated tests and used for organizing, planning and executing tests.
About Jira: Jira -the flagship project of Atlassian- is about managing issue tracking and collaboration. Since its inception in 2002, the power-packed product has undergone continuous improvement with over 15 years of Agile evolution to its credit.
The cornerstone of Jira is “a single source of truth”, and the tool is designed to bestow more power in the hands of developers because they’re adept at solving problems with code. While context and breaking down silos constitute this single source of truth, developers should be well integrated with QA to be able to release faster and get better code faster. This is to ensure they’re more vested in writing their own tests, and better equipped to handle testing applications.
In terms of native capabilities with Jira, managing full lifecycle test cases demand more manual effort. However, when it comes to a full fledged solution, particularly on the side of robust reporting, Jira exceeds expectations when augmented. Enter Xray, the missing link that helps Jira accomplish the same. The complete test management tool, when combined with Jira, bridges the gap between your requirements, what needs to be tested and how well your requirements are covered.
Xray & Jira: Best of both worlds
- Does not require any other software in order to run
- Supports the entire testing life cycle: test planning, test design, test execution and test reporting
- Uses special Jira issue types, so you can effectively use all Jira benefits/functions
- Enables users the power to configure screens, set workflows, add custom fields, and more
- Automated tests with native support for BDD methodology
- Powerful coverage analysis and test environments for fast deployment
Take advantage of Xray / Jira marriage
One of the main advantages of using Xray is: Xray issues are in fact also Jira issues. For test specification, it will create a test case issue type as well as Pre-Condition issue types for organizing your test sets. For planning purposes, you will get a test plan issue type, and then for test execution issue type. Xray does not implement permission control, which is left up to Jira, and it is done the same way a Jira administrator would do for any other type of Jira issue.
The learning curve for any Jira user will be minimized, as when you look at Xray issues, in fact, you will be looking at something that is virtually like any other Jira issue. This ensures everybody is in sync with the project from the same frame of reference.
Benefits of test case management with Xray in Jira
One of the biggest advantages of test case management with Jira and Xray is to bring your development and testing teams together on a single platform, which would eventually open up a plethora of opportunities in terms of visibility, and provide a comprehensive context without having to hop between different tool sets.
Since Xray also uses a lot of Jira capabilities, it provides many gadgets and widgets to create Xray specific dashboards, which can give a lot of information to the team and ensure that everybody is on the same page as far as the overall testing is concerned.
Getting started with Xray in Jira
Log into your Jira, go to ‘settings’, and then ‘manage apps’. Once you are in ‘managed apps’ page, you can view a full section of settings in Xray.
These are basic settings which would not impact how you’re going to execute it, but for the convenience of viewing and managing its results.
Issue Type Mapping
Xray also allows you to bring in multiple custom fields that ensure you can use some of the settings here as well. However, a key aspect of this general configuration is Issue Type Mapping, as it provides test cases, test execution and pretty much everything.
Xray needs to know exactly against what Jira issue types are we going to cover the requirements, provide the coverage, and the kind of initial types we could use to create bugs and defects as part of that testing process.
Trundl’s Tip: Jira is a tool with infinite possibilities. Given Xray’s flexibility, one can choose whatever issue type is available in their Jira to mark it as requirement issue.
Defect Issue Type
For the defect tracking (see image above), some teams like to use the term ‘defect’ or specifically ‘bug types in Jira’ to identify bugs or defects in their system. Drag and drop any of such issue types into the specific column to designate them as a specific bug or defect type as part of Xray’s scope.
Requirement Coverage
It is important to tell Xray what needs to be considered for requirements to be identified as covered/ not covered/ not okay/ not run. In this segment, you’ll find many options which you can use, such as used version test sets, etc. Each team could have different ways of considering a requirement to be covered based on the particular methodology they adopt, and Xray offers multiple options to accomplish that. There are in-built settings with regards to which requirement statuses to ignore or consider.
Requirement Project
As you can see, we’ve added a few projects here. Let us use a bookstore project for this demo.
Jira is an open tool which can be used for various purposes, be it development or administration, and this is a perfect platform to showcase a specific project, bring all the specific functionalities onto different screens, and various options that you see at a project level.
Trundl’s Tip: Once you set these up, you can move on to a project that will take you to a Scrum Board or a Kanban Board–feel free to choose any methodology for your development.
Make sure you have a Board configured with a Board view set up where you can see the coverage of your requirements as a status on one of the cards. This will enable you to clearly identify by looking at the card, see what’s going on with that particular story’s testing of that particular requirement.
Standard Jira Navigation
Go to projects, choose the project that you would like to work on, and you will land up on the Sprint Board as shown in the image. You can identify the requirements that you have to write test case against, and that’s when the Xray magic begins.
Let’s assume we identified story as one of our requirement types. Next, when you move to story, you’ll have this test coverage panel added to your story screen or the requirements screen right away.
This is where you can manage all the test specific information pertaining to a particular story. All the other information that you see here are default Jira fields, which you can modify, add or remove as per team requirements.
The fields particular to Xray are the requirements status, which will be based on the test that gets added to this particular story. Rest of them are the three buttons depicted as follows:
These are unique Xray-specific buttons, and unless you add a project to requirement covered, these buttons will not appear if you go to any other project which doesn’t belong to the list. This is where the testing team can see what’s going on with requirements. Isn’t that a cool way to spare your colleagues who’re not conducting testing all the information overload?
Creating an issue
You can create tests from multiple sources. Use the create button, just like you’re creating any other requirements.
Click on the create button and identify the issue type. Issue type here is ‘test’, which is promoted by Xray, and it will be added to the project.
Take advantage of the Xray button
Alternately, you could also create a test right from this one:
Using the button which Xray provides comes with an advantage. When you use the default Jira create button, it doesn’t bring the information from the requirement into your test, and the user has to add that information all over again. On the other hand, the Xray button automatically copies the description and summary of the particular requirement into your test. This eases the burden of test teams to a significant extent.
Creating Custom Fields
Once you add this general information, you will find that Xray also inherits a lot of Jira properties, so you have the flexibility to add more sets of custom fields that are particular to your testing processes and your audit processes.
Summary and Reporter are the only mandatory fields here, everything else can be left optional.
Creating Test Type
Then you go to the test types, where all these information are provided by Xray configuration. You don’t have to do anything additional in terms of configuring a screen for the test type, which is permitted by Xray. Once you install it, these types become available automatically.
Once you go into the test details, you will be able to see what type of tests can be created in Xray.
We can choose different options here, and depending on what type of tests you’re creating, Xray will automatically switch to the corresponding fields on that particular screen. Let’s select ‘cucumber’, and it automatically gives us a scenario outline with specific options.
Creating a manual test case
Let’s go ahead and create a manual test case. You can create these steps in Xray by putting the information into different fields actions, the data that you’re expecting to see here, and expected results from those steps.
- It could be daunting to create several steps manually, but fret not, Xray provides an ‘import option’ once you create a test. This allows you to import additional steps into the test case using different methods.
- Once you create the test, you can see that it automatically asks for that information on to the story itself.
Note that the requirement status has changed from ‘uncovered’ to ‘not run’. You’ll also get to see how many tests are part of this specific requirement.
What a full Jira view looks like
An advantage of using Xray for Jira teams is that the user experience is easy because of familiarity with screen layout, different navigation methods that are followed, etc.
Create Test Details
Once you open a test in a specific tab, you can go to test details to see all the information.
If you click on it, it will expand and reveal all the information pertaining to that particular step. Sometimes you have 10, 15, 20, 30 test steps for each scenario to be covered, so you can use a CSV. If you have a master test, this becomes a handy feature to perform a quick search.
Trundl’s Tip: You can enhance the Scrum Cards with the execution status, but this is only available in Jira Server or Data Center. Once Atlassian enables a certain mechanism for Xray to provide this feature in Cloud, users can augment their Scrum or Kanban with the execution status. Watch out, this feature will be highly useful!
Importing steps from an existing test
What if you want to reuse a lot of information already created? Be prepared to pick and choose, as it will provide you a list of all the steps that are part of the test, pull-in information, and corresponding tables. Xray also takes care of things when you’re trying to import steps from non-compatible test types.
Importing steps using CSV
You will be able to create a template, add all the information and just click on CSV import to get all the steps in here. Once you add the test, you need to be able to also have a ‘Pre-Condition’, and guess what? You don’t even have to move out of your test screen for that.
You’d be able to do everything pertaining to this particular test, right from this screen. Xray provides all the information that you need to be able to work around your test itself.
Setting Pre-Conditions
To create any new Pre-Condition, just click on ‘create Pre-Condition’, and it will open up the Pre-Condition tab.
- If you just want to associate a Pre-Condition, go ahead and start typing in here, as Xray is smart enough to provide the details when trying to search across Xray screens.
Organizing tests
Method I: Different teams like to pursue organizing tests at different levels. For instance, teams that are accustomed to traditional testing methodologies use tools like HP or IBM Doors are accustomed to adding their tests with different folders and Xray covers those aspects as well. If you want to use that specific functionality, you can use Xray test repository.
You’ll be able to organize various different tests into folders as well. This is to organize your test to ensure quicker access.
Method II: Another way of doing it is to group your tests into different test sets.
This could be for a specific testing that you’re administering, a specific phase of the testing, etc. To organize a test, make sure that you can add your test cases into different test sets. That way you can inherit or use that test set in different levels in the execution phase. As you can see here, there is an ‘associate test sets’ button.
Once you start typing, and Xray’s smart search will bring up the test set that you’ve created.
It will also show how many test cases the test possesses, key aspects of it, etc. You can also add additional columns as per preference. Now that you’ve added a test set, go ahead and plan your test and then run it too.
Use the same screen to go for ‘test plan’, click on ‘associate test plans’, and start typing it. The smart search will bring up the test plan that has already been created, so you simply add your test in here as shown below.
Things become so much easier when all concerned team members know exactly which test set to add their test to, and one doesn’t have to go back and do everything at once. Next, you will be able to go back and identify your test in the test plan, and start creating executions as well.
Trundl’s Tip: Consider the Test Repository as a launchpad to most of the things that you’d like to do. You can also multi-select or create test sets directly from within the test repository. By clicking on the directory structure, for instance, or multi- selecting these tests, you can add them to a test set to create a new test set, or even a test plan, and so on and so forth.
Initiating the execution process
It’s not necessary that all your tests will be part of a test plan, so you can always go for an ad-hoc execution. Sometimes, you could create random tests which you want to give a flexibility with just an execution.
As you can see below, you have a ‘test run’ option here.
Xray will show clearly that it is an ad-hoc execution.
Adding test environment
Tests environments in Xray are more like labels, and not something configurable or like any other drop down list that you see in Jira as that comes with certain limitations. The solution is to use Addons which are available to manage those labels.
Trundl’s Tip: In both Server and Data Center versions, these labels give you additional flexibility. However, this may lead to errors if you type the wrong Test Environment. In Cloud, Test Environments are created by the Project Manager at the project level, so it is a part of the dropdown list. The good news is, this will also be a part of future versions of Data Center and Server versions of Xray, and can be pre-created by whoever is in charge.
Running the Test Execution
Let’s go to the Test Plan and try to run the execution. All you need to do is just click on the hyperlink, and it will take you to a Test Plan Board showing the overall status of your executions based on different test levels.
Once you are on a Test Plan Board, you don’t have to go back to a previous screen where you’ll be able to see the status of each of these Tests as well as their overall execution status.
Let’s go back to the test plan itself and create the execution. Simply add all the tests or a specific test that you created for test execution and finally, run those tests.
What if one of the Tests fail?
You should be able to pick and choose which test you want to run as you proceed into the test execution phase. Once you review the test plan and see all the tests you’re adding here and feel something is amiss, you will have an option to add it right from here.
This is where organizing your test into test sets would comes handy. You’d be able to use this test sets in here, and then add everything that is part of the test set in one go. That way, you don’t have to pick and choose every single test from this section, but include all the tests corresponding to that test set into your test plan with one click.
Creating Executions
The only difference is that it doesn’t have the word ‘ad-hoc’ in here.
This enables you to differentiate whether it’s a planned execution or an ad-hoc execution. It will also ask you whether you want to redirect it and here you’ll still be able to use the same environment variables that you created in the previous stage.
The page below is an execution.
That particular test plan will allow tracking of all the information pertaining to this execution, and you will be able to run a different set of executions without having to affect the results of the previous execution. You will also be able to see all the different tests that we have added to this particular execution, alongside their status, and view the requirements each of these tests are covering, and find all of them hyperlinked.
Got questions? You’ll be able to click through them and go to the specific information as well. This segment will also show what test set this belongs to, and if the test has been assigned to an individual, it will show that information as well.
Assigning Tests
Once you’ve arrived at test execution, you will be able to see if this test has been assigned to somebody. If not, go ahead and assign it to a team member.
Comment, Execution, Defects:
Here, you get to see the overall execution status. While these are things you can configure, but at this point, these are some of those common test execution statuses that you will see in a lot of testing teams and testing processes. It starts with to-do process, and depending on how you’re proceeding with the rest of the steps, the information changes. Here you’ll be able to see this comment, execution, defects, and execution evidences, which are at the overall test level.
If you want to add any comments, there is a rich text editor which allows you to add all the formatting that you would like to incorporate.
You can also attach evidences here by choose relevant attachments. Take screenshots and copy paste that information, and the same will be captured.
You wil also be able to see the rest of the details such as test description, what requirement a particular test is covering, etc. If you have any other questions, simply click through it and view the status of that particular requirement. The Pre-Condition can also be viewed here.
Test Step:
Once you come into the test steps status, note that each step has its own context. Again, just like the way you did everything in the test case level, you can repeat the same here. Add comments:
Let us assume you want to unsubscribe from the newsletter.
- Since it has failed, let’s add a defect in for this particular step. Click on ‘create defect’, which will open up this information as seen below. Xray automatically copies that step into the description so that you can provide this additional information.
- Once you fail a step, you can see that its color has been updated and the overall execution of this particular test case shows ‘failed’.
- Go back to the execution again, and you’ll be able to see this information. Note that one of the tests has failed and the two other test cases are on to-do in status, and the information has been captured.
- Go back to the test plan to see the same information, specifically the execution in which a particular test has failed.
Checking Status:
Now that the test has failed, you’d want to run another execution in a different environment, and irrespective of what happened with the dev environment, you’ll be able to run this execution.
Once you complete the execution of all the test cases, you’ll be able to see the status of particular requirement status showing up there, or what’s going on with the particular coverage.
Need help with setting up Xray in Jira? Contact our experts today.
FAQ
Does Xray data count against Jira Cloud limits?
Yes. Every Xray issue is a Jira issue, and hence it will count. However, in terms of licensing, it has no impact whatsoever.
Yes. Xray has a test repository within the test step library, so you can start typing in and it will search for those Cucumber steps for you.
No, there are only three test types: Manual, BDD and the cucumber. Once you switch it, you lose that information. So as you go to automated tests, most likely the test specification is in your source code or script files. Try cloning it to have links to figure out which ones will have the automated version and the manual version for you.
Yes. If you’re looking at different testing frameworks that would probably fit into, you could use cucumber or BDD to invoke them. You could also use your generic tests and they would map onto a single test case. For instance, if you’re using JUnit, each method that you’re using would map onto an individual test case, and those results would show up in your in the test execution results.
What about generating historical graphs?
Yes, there’s a gadget for historical test coverage which is provided out of the box with Xray.
Can I define test parameters for a test case ?
In the next version, which will be released in a early Q3 or late Q2, Xray will be having that feature.
Remember that when you’re exporting feature files, unless you have some type of label that you’ll place it there, the feature files are the test specification. It is when you execute that test that you specify what environment you’re running it in. So a test case is agnostic to the test environment, and since this execution is being carried out elsewhere in an external system, you would have to state what testing environment you used when you push the results back into Xray.
Your best bet, if you have some type of labels within those tests, is to create a JQL query that will look for the ones that have specific labels for and just include those. That’s one way of doing it. The other one is adding all the tests with it, which are within that test set and then removing those.
Is there an option to directly execute tests or to convert a test to a tested execution?
Test execution is a mandatory process for executing your test cases, and then you have test runs within those executions. So a test case is not converted into a test execution, a test execution uses, or has test case specifications within that execution. However, you can create ad hoc executions right from the test case.
Can we add more than one test in an ad-hoc test execution?
Yes, but not from within the test case itself. You would probably use that in the test repository and select the ones that you want and create a doc test execution for that.
Is there any other unique identifier other than class name dot module name?
Xray constructs these unique identifiers based on the framework that you’re using. So if you’re using JUnit, that will be the only one that you can use at the time being.
Is it possible to create a test execution from the test plan view?
Yes. Within the test plan view, there are several ways of doing it. You can grab the list from the traditional Jira like view, or you can go into the test board at which you have a similar view of your test plan which looks like a repository. Select everything you want and just create an execution.
Can we edit the test steps, action, expected results during execution?
No. If you decide to change test steps as you go along, having a test case specification becomes meaningless. If you still want to change that, you can actually go back to the test specification and change those tests steps you desire. Expect Xray to warn you when you’re looking at a result for which the test steps have changed.
How do I filter searching for specific test cases from an execution?
You probably want to select or have a filter which will just show you the test cases and another one which would just show you the test executions. JQL is likely to help you in this case.
JQL itself is, can be very complex. Get in touch with us for expert help with guidelines for the JQL or Xray related JQL.
Is Xray a good tool for 500-1000 test executions?
Yes. Rest assured, large organizations worldwide work with Xray as a trusted test management tool without any limitations. Good practices must be followed when using Xray, especially in terms of reporting, test coverage and traceability.
How can you follow the evolution of test writing?
What most customers are still using is scripted tests. If they’re manual or automated, they’re actually specified. Xray is looking into data-driven, exploratory tests and actually has an exploratory exploratory testing application which integrates with Jira (and x-ray of course). That’s where all the benefits come really stand out. Nonetheless, we do know that some customers are venturing out into exploratory testing, which takes testing to a whole new level, and Xray is also prepared for that.
Is there a built in feature to record videos and directly attach during creation?
It is not built-in within Xray, but you can use your own capture and then add it in the feature. In the exploratory testing application, you can record videos and upload that, and create the defect directly from within the exploratory testing application.
Can x-ray add feature for confirmation before deleting labels?
Remember that labels is not an Xray property, but a Jira property. There can be some workflow or some other type of app that you can use to keep track of it.
Yes. Many customers just use subtests execution which are like sub tasks. So you can implement your workflows that will prevent a requirement from moving to the next stage while their sub tasks have not yet been completed. You can treat an execution as a sub task and until it is fully completed.
Is it possible to export the reports in Excel ? Currently it is only supporting all the CSV.
Yes. Have a look at document generator feature, which is located in the right-hand corner. It is a reporting capability where you define the template of what you want. However, this is not possible for the built-in reports, such as the test coverage report, etc. You can only export them in CSV, but you can get the same results from using document generator.
Know your experts
HELDER BISCAIA is the Head of Sales at Xblend, responsible for ensuring that worldwide customers are aware of how Xray can optimize the overall test management process. He is based in Aveiro, Portugal.
MANOHAR GOLI is a founding member and CTO for Trundl, running global teams focused on everything from new Atlassian configurations, hosting migrations to integrations. Trundl is an implementer of Xray for Atlassian Customers. Manohar lives in Atlanta, Georgia.
Need help with Xray in Jira?
Feel free to contact us, we will get in touch with you ASAP
