Agent-based execution sometimes fails with Remoting Exception

Bug reports.
WZMan
Posts: 1
Joined: Fri Aug 06, 2021 8:57 am

Agent-based execution sometimes fails with Remoting Exception

Post by WZMan » Fri Aug 06, 2021 12:16 pm

Hello everyone,

we are using Agent-based remote execution for testing some GUI components of our application with Ranorex automatically.
The testcases are executed on the command line in the context of a GitLab pipeline, and we are checking the agent exit code on the command line to determine whether a particular test was successful or not.

This works perfectly most of the time. However, in some cases, running a test with the remote agent returns exit code -1, which indicates failure ('Test run aborted due to error.') Checking the .rxzlog logs retrieved by the agent shows that running the test itself was actually successful however, with no errors whatsoever.

We then detected the following error in the Ranorex Agent log on the remote machine where the tests are executed:

Code: Select all

2021-08-06 09:01:01.8586 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Ranorex.Dex.Common.Types.Progress.TestExeState 
2021-08-06 09:01:03.8762 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Ranorex.Dex.Common.Types.Progress.TestExeState 
2021-08-06 09:01:05.8961 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Ranorex.Dex.Common.Types.Progress.TestExeState 
2021-08-06 09:01:06.2627 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:13 Background task 'Failed to resume test process via proxy. Killing process...' failed: System.Runtime.Remoting.RemotingException: Failed to write to an IPC Port: The pipe is being closed.


Server stack trace: 
   at System.Runtime.Remoting.Channels.Ipc.IpcPort.Write(Byte[] data, Int32 offset, Int32 size)
   at System.Runtime.Remoting.Channels.ChunkedMemoryStream.WriteTo(Stream stream)
   at System.Runtime.Remoting.Channels.Ipc.IpcClientHandler.SendRequest(IMessage msg, ITransportHeaders headers, Stream contentStream)
   at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at Ranorex.Dex.Components.TestExecution.Env.ReplayServiceProxy.GetCurrentExeState()
   at Ranorex.Dex.Components.TestExecution.Steps.ExecuteBinTestStep.<GetStatus>b__26_0(IProcess p). 
2021-08-06 09:01:07.9081 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Execution finished with result Error 
2021-08-06 09:01:07.9081 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Publishing test results 
2021-08-06 09:01:07.9081 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Publishing xxx
2021-08-06 09:01:07.9081 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Test results published 
2021-08-06 09:01:07.9081 Ranorex.Dex.Components.Domain.ExecutionAgent Job finished 
2021-08-06 09:01:07.9081 Ranorex.Dex.Components.Domain.ExecutionAgent Saving job results ... 
2021-08-06 09:01:07.9871 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:5 Job changed: Job 5 in state Executing issued by xxx -> Job 5 in state Executed issued by xxx. 
2021-08-06 09:01:07.9871 Ranorex.Dex.Components.Domain.ExecutionAgent ...Test results saved 
2021-08-06 09:01:08.1646 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:6 AgentChanged: Agent(Id=1 Name=0002022-01 DisplayName=RanorexAgent State=ExecutingTestSuite StateDetail=Test results published LastUpdated=01.01.01 00:00 +00:00 Tags=) -> Agent(Id=1 Name=xxx DisplayName=RanorexAgent State=Idle StateDetail=Test results published LastUpdated=01.01.01 00:00 +00:00 Tags=). 
It seems that although the test was successful, running the agent still threw an exception, eventually resulting in the non-zero exit code.

We have already disabled BitBridge for our solution since we do not need it and it may be associated with that kind of error according to some other posts in the forum. The described error still occurs however.

Our setup looks like follows:
Local OS: Windows Server 2016
Remote OS: Windows Server 2016 (this is were application and remote agent runs)
GUI Technology: PowerBuilder 2019 R3
Run Environment: GitLab 14.11
Ranorex Studio Version: 9.3.2

Has anybody else experienced this issue with Ranorex Agent execution? If so, were you able to solve it?

Thanks a lot!

JasonY
Posts: 9
Joined: Fri May 28, 2021 3:51 pm
Contact:

Re: Agent-based execution sometimes fails with Remoting Exception

Post by JasonY » Fri Aug 06, 2021 7:48 pm

Hello WZMan,

I recommend you review some of this documentation. We have some information that I think can help with the issue you are experiencing: https://www.ranorex.com/help/latest/ran ... g-and-faq/

Best Regards
Image

qa-auto
Posts: 86
Joined: Mon Aug 05, 2019 10:46 pm

Re: Agent-based execution sometimes fails with Remoting Exception

Post by qa-auto » Mon Aug 16, 2021 8:30 pm

JasonY wrote:
Fri Aug 06, 2021 7:48 pm
Hello WZMan,

I recommend you review some of this documentation. We have some information that I think can help with the issue you are experiencing: https://www.ranorex.com/help/latest/ran ... g-and-faq/

Best Regards
Which section of the link specifically would help with the issue reported?

I have 20 agents in my env, and we are starting to get this error:



2021-08-10 16:11:49.5500 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:13 Background task 'Failed to resume test process via proxy. Killing process...' failed: System.Runtime.Remoting.RemotingException: Port is Busy: All pipe instances are busy.




Server stack trace:
at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)
at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)
at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)



Exception rethrown at [0]:
at Ranorex.Dex.Components.TestExecution.Env.ReplayServiceProxy.GetCurrentExeState()
at Ranorex.Dex.Components.TestExecution.Steps.ExecuteBinTestStep.<GetStatus>b__26_0(IProcess p).
2021-08-10 16:11:49.5650 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:13 Background task 'Failed to resume test process via proxy. Killing process...' failed: System.ArgumentNullException: Value cannot be null.
Parameter name: value
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at Ranorex.Dex.Components.TestExecution.Steps.ExecuteBinTestStep.<GetStatus>b__26_0(IProcess p).
2021-08-10 16:11:49.5650 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Execution finished with result Error
2021-08-10 16:11:49.5650 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Publishing test results
2021-08-10 16:11:49.5650 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Publishing C:\Users\ranoadmin\AppData\Local\Temp\Ranorex\bc54f6b383a895b32106035220afc7cd\TestSuite_20210810_132135.rxzlog to http://192.168.26.142:8081/api/files/
2021-08-10 16:11:49.5650 Ranorex.Dex.Components.TestExecution.Steps.ProgressLogger Test results published
2021-08-10 16:11:49.5650 Ranorex.Dex.Components.Domain.ExecutionAgent Job finished
2021-08-10 16:11:49.5650 Ranorex.Dex.Components.Domain.ExecutionAgent Saving job results ...
2021-08-10 16:11:49.5880 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:13 Background task 'Failed to kill test process' failed: System.ComponentModel.Win32Exception (0x80004005): Access is denied
at System.Diagnostics.Process.Kill()
at Ranorex.Dex.Components.TestExecution.Steps.ExecuteBinTestStep.TryKillProcess(IProcess p).
2021-08-10 16:11:49.5930 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:5 Job changed: Job 82 in state Executing issued by Lorinda Desroches -> Job 82 in state Executed issued by Lorinda Desroches.
2021-08-10 16:11:49.5930 Ranorex.Dex.Components.Domain.ExecutionAgent ...Test results saved
2021-08-10 16:11:49.5930 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:6 AgentChanged: Agent(Id=1 Name=CFPB3 DisplayName=CFPB3 State=ExecutingTestSuite StateDetail=Test results published LastUpdated=1/1/0001 12:00:00 AM +00:00 Tags=) -> Agent(Id=1 Name=CFPB3 DisplayName=CFPB3 State=Idle StateDetail=Searching item 'Self' in 'Base' LastUpdated=1/1/0001 12:00:00 AM +00:00 Tags=).
2021-08-10 16:28:42.2245 Ranorex.Dex.Infrastructure.RanorexEventSource EventId:11 Background task 'Delete(C:\ProgramData\Ranorex Agent\Uploads\a192469a798784444cc8937ebf3529ce.zip)' started.

qa-auto
Posts: 86
Joined: Mon Aug 05, 2019 10:46 pm

Re: Agent-based execution sometimes fails with Remoting Exception

Post by qa-auto » Mon Aug 16, 2021 8:35 pm

Nothing is blocked, but I see 10001 is not listed. Here's 1 of our agent vm's:

We've been running 2 years with this config so I don't think that's the culprit.


C:\Users\ranoadmin>netsh firewall show state

Firewall status:
-------------------------------------------------------------------
Profile = Domain
Operational mode = Disable
Exception mode = Enable
Multicast/broadcast response mode = Enable
Notification mode = Disable
Group policy version = Windows Firewall
Remote admin mode = Disable

Ports currently open on all network interfaces:
Port Protocol Version Program
-------------------------------------------------------------------
10000 UDP Any (null)
8081 TCP Any (null)
29189 TCP Any (null)
31000 UDP Any (null)
31000 TCP Any (null)
3389 TCP Any (null)

IMPORTANT: Command executed successfully.
However, "netsh firewall" is deprecated;
use "netsh advfirewall firewall" instead.
For more information on using "netsh advfirewall firewall" commands
instead of "netsh firewall", see KB article 947709
at http://go.microsoft.com/fwlink/?linkid=121488 .



C:\Users\ranoadmin>netstat -ano | findstr -i SYN_SENT

C:\Users\ranoadmin>

qa-auto
Posts: 86
Joined: Mon Aug 05, 2019 10:46 pm

Re: Agent-based execution sometimes fails with Remoting Exception

Post by qa-auto » Mon Aug 16, 2021 8:41 pm

Would changing some of the agents (of 20) to something other than 8081 make a difference?

qa-auto
Posts: 86
Joined: Mon Aug 05, 2019 10:46 pm

Re: Agent-based execution sometimes fails with Remoting Exception

Post by qa-auto » Mon Aug 16, 2021 8:42 pm

Is it not cleaning up after creating a named pipe?