Getting started
Installation of server components
Install MySQL and PHP according to instructions supplied with each piece of software. Apache / Linux users, please review instructions supplied with MySQL and PHP regarding options that must be used to compile Apache web server. Use default installation options. There are no special requirements for TCMS.
TCMS software installation and database initialization
See Readme.htm
Estimate time is about 5 minutes.
Initial configuration
You (as TCMS administrator) will initially create user accounts and define values for all the pulldown menus to the best of your knowledge and future outlook.
Estimate time is about 30 minutes.
Creating your test cases
Have all members of your team create or manually convert existing Test Cases specifying appropriate properties such as Product, Component, Test component, and Test SubComponent, etc.
Provide instructions if necessary. If you are converting existing Test Cases it may be faster to modify database tables directly using MySQL CC tool.
Estimate time is about 15 minutes per test case per user.
The process
Verify / Modify Test Case assignment
Before starting tests on each new version, You (as QA manager) should verify test case assignment to members of your team specifying appropriate Version and Priority.
This task can be accomplished by utilizing bulk assignment functionality.
- Query Test Cases by Product, Component, Group, SubGroup, or Text in order to select TCs that are likely to be assigned to person "A".
If you wish to find TCs that are not currently assigned to anyone then check "Unassigned only" checkbox.
- Query the database; verify that results are as expected; if necessary, go back and repeat this process.
- Check checkbox next to CaseID number for the test cases you wish to assign to person "A".
Click on "Check-All" link to select all rows.
- Click on "Assign" button, specify assignment properties, and submit it.
- Repeat steps 1 to 3 for person "B", "C", etc.
- Query the database to make sure there are no more unassigned or incorrectly assigned test cases
Now, members of your team can see what is assigned to them, for which version, what needs to be tested and in what priority.
At the same time, TCMS system "knows" which test results are expected.
Note that members of your team can assign / reassign test cases to themselves or other members as necessary. This functionality is not restricted to any particular user account. It relies on intelligence and good judgment of your team members and considered a valuable feature since it takes this assignment routine off your back.
Estimate time is about 1-2 hours.
Testing
Determine reasonable balance in terms what data should be collected and entered into each Test Result (besides required fields).
Define conditions when "Failed" test needs to be recorded and when not.
For example,
test case says "type -h into the prompt and see if help is displayed";
user typed "-?" instead;
help did not appear;
user should repeat this with "-h" without recording "Failed", that's obvious,
but if system crashed in the process perhaps tester should record "Partial" result instead of "Passed" even if his mistake not directly related to the Test Case conditions.
Have your team record results as the tests go along. Discourage users from using yellow stickers and "I will enter it later" approach. It is much more productive to copy-paste text from the screen rather than write-it-down and type-back-in.
Test Results
When all tests are completed and all results recorded manager (and everybody else) can query the system as explained in examples 1 and 2 to see if there are any "Failed" results. Eventually, this process will mature to the point when no emails like "Team! what is the status?" would be necessary.
Frequently asked questions
| Q: |
What sort of user levels exist |
| A: |
There is one predefined user account called "root". When you login as root, you will have a link to Administration interface where you can add/change menu options, add users, change passwords, and perform certain DB maintenance tasks. This root account should be used only for administration purposes because you would want to have personalized entries in the history for the rest of the activities. In fact, most of the users in your group should know about this account and use it to add menu options as necessary without having to wait for someone in charge to do this. |
| |
|
| Q: |
Are there permission levels for admin, lead/manager, external tester and guest? |
| A: |
No, there are no permission levels, however this may be added in the next major release. Right now, all users can use the system equally, however certain things available only to the 'root' user. TCMS does not have external users since it is intended for internal use within an organization. |
| |
|
| Q: |
Are there any reports ? |
| A: |
Reports, reports. Everyone is looking for reports :-)
There are no separate reports in this release. The "Search/Query" form allows you to produce 3 types of "reports" - Test Cases, Test Results, and History. You can query the database and print results if necessary. There is a small "Print" button on most of the pages which will produce output more suitable for printing. Then, use browser's Print functionality.
Example 1
We want to find what is not tested for version 1.1
In "Assignment" box specify Version to 1.1
Click on "Find Not Tested" button
The result displays 3 test cases that do not have any test results recorded.
Example 2
Now, we want to find "What has failed or partially completed in version 1.1, build 1, and we assume that tester could perform this test more than one time"
In "Assignment" box set Version to 1.1
Set Build to 1
Select "any of subsequent test results" option
Select "Failed", "Failed (no retest)", and "Partial" checkboxes
Click "Find Test Results" button
The result is one row (TC # 1).
Click on "1" in CaseID column (not very noticeable, but there is a link)
Look at "Recorded Test Results" table.
Here is what happened.
This test was performed total 10 times for Version 1.1, build 1.
First time it has failed, a second later it was partial,
and a second later it passed.
Test was performed by an automated script using some sort of "retry" loop.
On the next day, user root (not good idea to use root account, though) noticed the problem and re-tested this case
manually 7 more times - all Passed.
It is also Passed in a later build # 99 on the 25th.
Ideally, failed and partial result would contain some useful comments and script output, so that we can read what happened. ;)
BTW, results with sequence numbers between 3 and 12 called scope, i.e. multiple tests for the same version+build.
In these examples, we used only 2-3 fields to query the database. As you can imagine, there are many more combinations.
Perhaps, this can be classified as tabular reports down to build layer plus "scope" capabilities. There is also a summary (totals) at the end of the report. If you wish to print it, there is a small print button on top of the page that will produce a printable version.
I should mention that when we developed TCMS our customer already had about 3500 test cases and printable reports did not make much sense, because nobody was going to print a book after each build (once a week) and actually be able to do anything useful with it in between the builds. So, printable reports died before even being born, instead the query mechanism evolved into interactive reports. |
| |
|
| Q: |
Detailing what the status of a test case is [ not run, pass, fail ] |
| A: |
For "not run" (does not have any test results for that version recorded) you need to run a separate query (see example # 1).
To find-out the status for a particular Version for which test is complete specify that Version in Assignment box and use "Find Test Results" button.
In query result, column "Summary of Test Results" is split into two sub columns. "Last" means last known result. "Entire scope" gives you an idea of what had happened if there were multiple test runs on the same version [ plus build ].
Basically, if you see anything in red color, you would want to investigate it by clicking on that item. |
| |
|
| Q: |
Are build and versions linked such that a build number will not show up for a version that does not have that number? |
| A: |
TCMS does not automatically know your build number. This is something your users pick from the list of available builds menu or your automation scripts can determine or configured to. The system will not work with an empty build field, but you can certainly specify a dummy value like "none" or "0" just to make things work. TCMS should be Ok with this; however, you will loose this layer of tracking. If we go back to example # 2, everything tested for version 1.1.none is going to be your one large scope and version 1.2 is another scope, and so on. |
| |
|
| |
|
|
|
