APM Remote fails to generate a new full download package due to corruption of previous download pack


 Product(s):APM Implementation and Performance Management
 Version(s):7.11
 Environment:N/A
 Area:N/A
 Subarea:N/A

Problem

Remote Tablet suddenly stopped synchronizing successfully. In reviewing the error log files, the following error came up:

2019-07-14 02:38:20.7844Z Error  Ivara.Replication.Remote.XMLGen.XMLGenerator [50] XMLGenerator threw exception EXCEPTION OCCURRED:SqlException A transport-level error has occurred when sending the request to the server. (provider: TCP  Provider, error: 0 - An existing connection was forcibly closed by the remote host.)    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.SNIWritePacket(SNIHandle handle, SNIPacket packet, UInt32& sniError, Boolean canAccumulate, Boolean callerHasConnectionLock)
   at System.Data.SqlClient.TdsParserStateObject.WriteSni(Boolean canAccumulate)
   at System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte flushMode, Boolean canAccumulate)
   at System.Data.SqlClient.TdsParserStateObject.ExecuteFlush()
   at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch(String text, Int32 timeout, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj, Boolean sync, Boolean callerHasConnectionLock, Byte[] enclavePackage)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean  describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean  asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at Ivara.Replication.Remote.XMLGen.XMLGenerator.ProcessSQLtoXML(IvaraDataTable IvaraTable, List`1& PrimaryColList, Boolean bIsInterfaceClassMultiNode, String queryString, String InterfaceClassName, DbConnection dbConnection)
   at Ivara.Replication.Remote.XMLGen.XMLGenerator.GenerationThread(Object o)
Win32Exception An existing connection was forcibly closed by the remote host
2019-07-14 02:38:32.9250Z Error BENTLEY\SERVERNAME$ Ivara.PL.FatalExceptionPolicy [11] Database Connect Error:
   DBErrorType     = 'dbConnection'
   Native SQL Code = 10054
   ODBC sqlstate   = 08S01
   ODBC errortext  = '[Microsoft][ODBC Driver 13 for SQL Server]TCP Provider: An existing connection was forcibly closed by the remote host.
'
(SessionID:e50cd015-a995-4d62-8603-658cf516856b) caught at: [?Refresh@oPersistentObject@OQ@EXPLink@Ivara@@$$FQE$AAM_N_N@Z,f:\src\apm\explink\opersistentobject.cpp,229]

This was followed by:

2019-07-14 02:39:01.7566Z Error  System.ServiceModel [40]  http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.TraceHandledException.aspxHandling an exception. Exception details: Ivara.Replication.Remote.RemoteService.PackageGenerationException: Exception during package file generation, please  check the Application server logs for details.
   at Ivara.Replication.Remote.RemoteService.RemoteSession.ReadRequest(Int32 nOffset, Int32 nCount)
   at SyncInvokeReadRequest(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)IvaraServer.exeIvara.Replication.Remote.RemoteService.PackageGenerationException, RemoteService, Version=7.11.0.0, Culture=neutral, PublicKeyToken=nullException  during package file generation, please check the Application server logs for details.   at Ivara.Replication.Remote.RemoteService.RemoteSession.ReadRequest(Int32 nOffset, Int32 nCount)
   at SyncInvokeReadRequest(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)Ivara.Replication.Remote.RemoteService.PackageGenerationException: Exception during package file generation, please check the Application server logs  for details.
   at Ivara.Replication.Remote.RemoteService.RemoteSession.ReadRequest(Int32 nOffset, Int32 nCount)
   at SyncInvokeReadRequest(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

What this means is that there was a SQL Database related error whereby the Server lost connection to the Database while the Download Package was being generated for the Remote Computer. This corrupted the XML Download Package file for that Remote Computer. Due to the corrupted Download Package file not being able to be accessed, APM errors every time it tries to regenerate the download package for this particular machine.

Solution

This problem may occur when the APM database or services are taken down while a Remote tablet is in the process of synchronizing. If Remote tablets are set to auto-synchronize, user should disconnect the devices from the docking stations or internet connection during planned outages and outage manager should monitor server manager to ensure that no Remote Tablets are synchronizing (transacting) before taking down the APM database or APM Server Service.

Should the problem happen the solution is to:

1. restart the APM Server Service

2. wait for the service to start completely and the download package expiry time to elapse (usually set to 1 hour)

3. retry synchronizing the Remote tablet.

This will successfully clear the corrupted the Remote XML Download Package and allow for a new Download Package to be generated and the Remote Computer to successfully synchronize.