Hi, while writing a script for a test, the SUT crashed. This got me thinking "what should I do with my test script until the bug is fixed? Shall I drop it, shall I run the script and crash? ".
I'm thinking about running the script and querying the bug tracking system. Then I will report an error, either the bug isn't fixed or my test script must be updated. I'm not sure what to do with a crash. Restarting the SUT allows me to run more test scripts.
I'm curious how others deal with such situations.
How do you handle (SUT) bugs & crashes in your test scripts?
Re: How do you handle (SUT) bugs & crashes in your test scripts?
What I do if I know I have a blocking defect is the following...
1) write a snippet of code that checks the status of a defect (you can do this if you have some sort of access to your defect database, otherwise you can skip this step).
2) If the defect is not marked fixed, or if you are skipping step #1, write a WARNING or a FAILURE to the log file that the following bit of test code will be skipped until defect #XXX is fixed.
3) skip that section of test and continue on, assuming that the defect will not affect the next set of tests.
The thinking is, if I just stop testing there, and there still are tests that can be run and are not affected by the defect, then I would be missing on importing testing in other areas of the AUT. I do not want to skip testing when it could still be accomplished.
1) write a snippet of code that checks the status of a defect (you can do this if you have some sort of access to your defect database, otherwise you can skip this step).
2) If the defect is not marked fixed, or if you are skipping step #1, write a WARNING or a FAILURE to the log file that the following bit of test code will be skipped until defect #XXX is fixed.
3) skip that section of test and continue on, assuming that the defect will not affect the next set of tests.
The thinking is, if I just stop testing there, and there still are tests that can be run and are not affected by the defect, then I would be missing on importing testing in other areas of the AUT. I do not want to skip testing when it could still be accomplished.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!
Ciege...
Ciege...
Re: How do you handle (SUT) bugs & crashes in your test scripts?
How often do you check if the SUT has crashed? Or handle validation errors?
They both leave the SUT in a state from where it's difficult to continue. Do you kill the SUT, restart it & continue?
They both leave the SUT in a state from where it's difficult to continue. Do you kill the SUT, restart it & continue?
Re: How do you handle (SUT) bugs & crashes in your test scripts?
I have a running second thread that checks for any unhandled exceptions in my AUT. This thread will halt testing when one occurs. Since it is an unhandled exception in the AUT I want to preserve state and know my latest steps so that I can get it addressed.How often do you check if the SUT has crashed?
Depends on the error... You have to know your AUT and your test code. You also have to know what expected values are and if an incorrect value will affect your future tests. If an incorrect value will affect a future test you need to fail your test, put your AUT back in a known good state and continue testing. Or, just fail your test and stop testing so you can manually verify the failure and determine the error.Or handle validation errors? They both leave the SUT in a state from where it's difficult to continue. Do you kill the SUT, restart it & continue?
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!
Ciege...
Ciege...
