Ranorex Forum

My company works in the medical device field so we have to validate all tools we are using and report to the FDA. I am kind of stuck how we could prove to the FDA that the instrumented app we are testing with Ranorex is equivalent to the non-instrumented app we will ship to the user.

Has anybody else encountered this problem or is there any guidance from Ranorex?

Please provide more information.

What type of application are you testing?
Is it on a mobile device? Web-based? Flash/Flex?

krstcs wrote:Please provide more information.

What type of application are you testing?
Is it on a mobile device? Web-based? Flash/Flex?

Sorry, I was really unclear. It's about mobile apps on iOS and Android.

I am trying to come up with a line of reasoning that demonstrates that doing most of the testing with an instrumented app is equivalent to testing the non-instrumented app.

I would guess this is a common problem for medical companies that want to go into mobile.

If we automated with Appium for example we don't have to instrument the app so it's easier to explain.

Ranorex instrumentation attaches their own libraries to assist with automation for mobile applications. The behavior of the application should be exactly the same but on a bit level, they are different. Do not release a instrumented application either, as the library is not designed for end-user use, only automation.

In my opinion, it would be exceedingly difficult to justify to an investigator that they are equivalent.

Ranorex does, indeed, change the app's files, even if they don't really change the functionality. Therefore, it would be really difficult, without doing a very extensive binary analysis, to say for certain that injecting Ranorex into the app didn't change something, for good or bad, that makes it different than what is released to your production environment. Even then, it might not be truly possible.

But, that is my opinion, having worked in the US Defense industry for 15 years before starting where I'm at now.

krstcs wrote:In my opinion, it would be exceedingly difficult to justify to an investigator that they are equivalent.

Ranorex does, indeed, change the app's files, even if they don't really change the functionality. Therefore, it would be really difficult, without doing a very extensive binary analysis, to say for certain that injecting Ranorex into the app didn't change something, for good or bad, that makes it different than what is released to your production environment. Even then, it might not be truly possible.

But, that is my opinion, having worked in the US Defense industry for 15 years before starting where I'm at now.

Is there a statement from Ranorex on this? It seems they are losing a big target market that way. I don't understand why they even need instrumentation. Appium doesn't for example.

merklh wrote:I am kind of stuck how we could prove to the FDA that the instrumented app we are testing with Ranorex is equivalent to the non-instrumented app we will ship to the user.

The instrumented and the non-instrumented app are definitely not equivalent, not on a binary nor on a functional level (e.g. Ranorex "inserts" code to get notified of events).

That said, every automated testing tool (instrumenting and non-instrumenting) will have influences on the AUT that a human user would not have (and vice versa). For example, even if Appium does not instrument an app, it uses special APIs to identify controls when searching for the control and to interact with it (touch). So even in this case, other code will be executed in the AUT as if a human user would interact with the AUT.

On the other hand, human testers tend to make systematic (and random) mistakes, too.
And with automation, you may be able to do more tests or increase test coverage.

IMHO it's just a matter of risk management: you either "risk" doing automated testing and possibly influence the AUT or not automate your testing at all, with all consequences. My choice is clear (automate everything!), but I may be biased

merklh wrote:If we automated with Appium for example we don't have to instrument the app so it's easier to explain.

AFAIK Appium uses the Android MonkeyRunner API under the hood, which Ranorex also uses to automate system apps (see this chapter in the user guide). So, this way could work with Ranorex, too, but without instrumenting your app you cannot use the recorder to record actions.

Regards,
Alex
Ranorex Team

Support Team wrote: AFAIK Appium uses the Android MonkeyRunner API under the hood, which Ranorex also uses to automate system apps (see this chapter in the user guide). So, this way could work with Ranorex, too, but without instrumenting your app you cannot use the recorder to record actions.

Alex, Can you elaborate on this sentence a little more? We would be fine without recording at all or doing recording with instrumented apps. Could we automate our own app without instrumentation by treating it like a system app? How about iOS?

Hi merklh,

the monkeyrunner API was initially designed to control an Android device from outside of an Android application. This approach allows you to test/control applications on Android based devices without instrumenting the application itself. Please find more information about the monkeyrunner on the Android Developer side.
Of course, you can also use this approach to test your application without instrumenting. Just follow the steps mentioned in our User Guide. Please also note the points which are mentioned under the Restrictions header.
Thank you.

On iOS, the situation is a bit more complicated since Apple is very restricted and does not allow to interact with the device outside it's sand boxed environment. In this case, our instrumentation ensures a reliable and stable object recognition within the application.

Please let me know if you have any further questions.

Regards,
Markus (S)

Hi

If you need to further test your mobile application with ranorex I would propose the following steps:

1) Create a list of regression test cases (most critical, mostly used by your customers
2) Write a test plan (Title: Validation of Ranorex and Instrumented Apps) where you compare the not instrumented and instrumented version
3) Perform all test cases manually with the original (not instrumented) version
4) Perform all test cases manually with the instrumented version (deployed by Ranorex)
5) Automate the test cases and execute (with Ranorex)
--> All results have to be identical
7) Write a test report with all the results
Conclusion: The behavior of all 3 test environments is the same - testing on the instrumented apps shows equal results

For medical devices you have to validate your test equipment before the results created with this test equipment can used as formal correct product validation results. This above procedure is equal to validate an external temperature sensor for example, where you prof that it delivers correct results.

If your App is FDA rated as 'high level of concern', I would discuss that with FDA representatives before.
Maybe they have additional request to prove equality

Hope this helps
Speedboat