Possible to notify at point of failure?

Ask general questions here.
Gav
Posts: 27
Joined: Thu Oct 20, 2011 10:52 am

Possible to notify at point of failure?

Post by Gav » Thu Oct 20, 2011 11:06 am

Hi,

We're evaluating Ranorex and it's looking pretty good. We're now at the stage of determining how we can handle recovery scenarios and notifications.

We are running a suite of tests, with each recording configured to try to continue its steps if there's a failure. Each case in the suite is then configured to continue to the next case if the recordings can't continue.

But rather than wait until the suite of tests has finished to get a notification and report (e.g. by email using the Send Mail module) is it possible to have an alert as soon as there's a failure in any of the recordings or cases?

Thanks,

Gav

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Possible to notify at point of failure?

Post by Support Team » Thu Oct 20, 2011 12:29 pm

Hi,
Gav wrote:But rather than wait until the suite of tests has finished to get a notification and report (e.g. by email using the Send Mail module) is it possible to have an alert as soon as there's a failure in any of the recordings or cases?
If I can follow your description you want a notification if something fails, right? Or do you want a message box which pauses the automation?

Regards,
Peter
Ranorex Team

Gav
Posts: 27
Joined: Thu Oct 20, 2011 10:52 am

Re: Possible to notify at point of failure?

Post by Gav » Thu Oct 20, 2011 1:59 pm

Hi Peter,

That's right - we would like the tests to continue without pause, but to 'raise a flag' as soon as something fails so we can start working on it, rather than waiting for the report at the end.

Gav

User avatar
sdaly
Posts: 238
Joined: Mon May 10, 2010 11:04 am
Location: Dundee, Scotland

Re: Possible to notify at point of failure?

Post by sdaly » Thu Oct 20, 2011 2:36 pm

I like this idea too :D

If the ranorex assertions fire an even upon failure, we could add a handler for that event and ping out an email...

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Possible to notify at point of failure?

Post by Support Team » Thu Oct 20, 2011 3:47 pm

Hi,
Gav wrote:That's right - we would like the tests to continue without pause, but to 'raise a flag' as soon as something fails so we can start working on it, rather than waiting for the report at the end.
Currently it is possible to use the Ranorex Blog Customizing Ranorex Reports or you use the Tear Down part of the Test Case to handle this.
sdaly wrote:If the ranorex assertions fire an even upon failure, we could add a handler for that event and ping out an email...
It would be possible to implement an EventHandler, but currently we don't know what this handler should do? Should this handler allow to execute "whatever" code or just logging code? Only on failure or even on success?

Would it be possible to post us a few samples, where the other two solutions aren't enough and you need an EventHandler?

Thank you.

Regards,
Peter
Ranorex Team

User avatar
sdaly
Posts: 238
Joined: Mon May 10, 2010 11:04 am
Location: Dundee, Scotland

Re: Possible to notify at point of failure?

Post by sdaly » Thu Oct 20, 2011 4:39 pm

Hi Peter

The way I would imagine to use it would be like this -

Code: Select all

//register a handler for the validate 'failure' event
Validate.Failure += new EventHandler(Failure);

public static void Failure(object sender, RanorexEventArgs e){
sendEmail("Assertion failed - location is " + e.className + "::"  + e.memberName + " - " + e.errorMessage);	
}

User avatar
sdaly
Posts: 238
Joined: Mon May 10, 2010 11:04 am
Location: Dundee, Scotland

Re: Possible to notify at point of failure?

Post by sdaly » Thu Oct 20, 2011 6:32 pm

...I should probably explain this a bit more!...

We use NUnit to control our tests - we determine failure by surrounding each test in a try/catch block. I would like to send out an email notification upon failure. We could add some code to our tear downs to send an email if the test has failed. We would have to remember to add this bit of logic to each of our teardowns though.

The advantages I see of having an event fired upon failure is -
>Say we have a validate.exists within a code module/helper method such as "Login()"...if a 'failure' event was fired here (with stacktrace details) which invokes a method (our handler) that sends an email, then we would get an email to say failure has occurred at 'Login' and not just that test x has failed.
>We could log failures to our DB for each validate, again with the class/member details...this would allow us to report on failing areas such as 'Login', instead of just being able to see failures at test level. We could quickly see 10 tests have failed but 10 'Login's have failed and therefore identify that there is a login issue.
>If we have a test which has several validates wrapped in a try/catch because we want it to try all steps, then we would get an email notification for each of the failure points, not just one big failure at the end of the test.

In general, it would just be a nice little feature which would open up some powerful opportunities...

I hope that makes sense :?

Sorry for hijacking this thread...all seems relevant though :oops:

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Possible to notify at point of failure?

Post by Support Team » Fri Oct 21, 2011 10:23 am

Hey Scott,

We discussed this internally and we could implement an event handler.
But actually we don't know which arguments we should return.
We could return the first class/method which forces the validate? We could even return you the stacktrace, but for the most users the stacktrace is too complex and difficult to handle and thus not useful.

So what would you like to know when such an event handler is called?

Regards,
Peter
Ranorex Team

User avatar
sdaly
Posts: 238
Joined: Mon May 10, 2010 11:04 am
Location: Dundee, Scotland

Re: Possible to notify at point of failure?

Post by sdaly » Fri Oct 21, 2011 11:04 am

Hi Peter

I guess the class/method which triggered the validate would be adequate and would keep it simple...so that, the validation message and the exception message? :mrgreen:

Cheers
Scott

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Possible to notify at point of failure?

Post by Support Team » Fri Oct 21, 2011 12:12 pm

I added a feature request to our internal bug tracking system.
sdaly wrote:the validation message and the exception message?
You want the validation exception and the EventHandler exception inside the Report, right?

Regards,
Peter
Ranorex Team

User avatar
sdaly
Posts: 238
Joined: Mon May 10, 2010 11:04 am
Location: Dundee, Scotland

Re: Possible to notify at point of failure?

Post by sdaly » Fri Oct 21, 2011 1:06 pm

Hi Peter

Those are the things I would like passed through the event args! So then in the custom event handler, you can do whatever!

Thanks
Scott

Gav
Posts: 27
Joined: Thu Oct 20, 2011 10:52 am

Re: Possible to notify at point of failure?

Post by Gav » Fri Oct 21, 2011 2:43 pm

Please excuse my noobness, but if the feature request goes ahead, does it mean our developer guys will be able to get Ranorex to send notifications as soon as there is a failure in the test suite, while the test suite then attempts to carry on with other tests?

Gav

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Possible to notify at point of failure?

Post by Support Team » Mon Oct 24, 2011 10:24 am

This feature request would just provide another way to react to validation failures using code, e.g. by sending an email.
Gav wrote:does it mean our developer guys will be able to get Ranorex to send notifications as soon as there is a failure in the test suite, while the test suite then attempts to carry on with other tests?
That feature is already available. You just need to attach a custom logger that sends an email if a failure is reported. See following blog on how to customize Ranorex reports and the "Send Mail" automation module:
http://www.ranorex.com/blog/customizing-ranorex-reports
http://www.ranorex.com/forum/send-mail- ... t2356.html

Regards,
Alex
Ranorex Team

Gav
Posts: 27
Joined: Thu Oct 20, 2011 10:52 am

Re: Possible to notify at point of failure?

Post by Gav » Mon Oct 31, 2011 11:21 am

Hi Alex,

Apologies again, but I just need to be absolutely clear on this before I get our devs to have a look at it for me. Will the custom logger allow a message to be sent immediately that a step fails (regardless that the script will attempt to continue), without having to wait until the suite completes and the report containing failures and results is presented at the end?

Gav

User avatar
sdaly
Posts: 238
Joined: Mon May 10, 2010 11:04 am
Location: Dundee, Scotland

Re: Possible to notify at point of failure?

Post by sdaly » Mon Oct 31, 2011 12:11 pm

Gav, that is correct, you can do that. I implemented a solution using a custom logger last week (never even knew it existed!)... I'm going to write a blog post next week on it so I'll post you a link once I've done so...