I am supporting an application that accesses an on-premise CRM via the XRM SDK. Our environments have multiple CRM Front End servers that are accessed through a HAProxy load balancer. We recently made a changes to the environment to round robin the calls to the CRM FE servers to balance the load to the servers, instead of sticking to a single node. The server machine keys have been aligned and accessing CRM from the browser works fine with no issues or errors. However when our application (a web service itself) makes calls to CRM via the SDK we can are occasionally seeing errors in the trace logs on the CRM servers and 500 errors in the IIS Logs. No exception is raised to the calling code and the data is being retrieved and saved correctly.
Below is the error from the log
<TraceRecord xmlns="schemas.microsoft.com/.../TraceRecord" Severity="Error"><TraceIdentifier>msdn.microsoft.com/.../System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier><Description>Throwing an exception.</Description><AppDomain>/LM/W3SVC/1/ROOT-1-132053368025369459</AppDomain><Exception><ExceptionType>System.ServiceModel.Security.MessageSecurityException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Message security verification failed.</Message><StackTrace> at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message&amp; message, TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.VerifyIncomingMessage(Message&amp; message, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationStates)
at System.ServiceModel.Channels.SecurityChannelListener`1.ServerSecurityChannel`1.VerifyIncomingMessage(Message&amp; message, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationState)
at System.ServiceModel.Channels.SecurityChannelListener`1.SecurityReplyChannel.ProcessReceivedRequest(RequestContext requestContext, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelListener`1.ReceiveItemAndVerifySecurityAsyncResult`2.OnInnerReceiveDone()
at System.ServiceModel.Channels.SecurityChannelListener`1.ReceiveItemAndVerifySecurityAsyncResult`2.InnerTryReceiveCompletedCallback(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
at System.Runtime.InputQueue`1.Dispatch()
at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace><ExceptionString>System.ServiceModel.Security.MessageSecurityException: Message security verification failed. ---&gt; System.ServiceModel.Security.Tokens.SecurityContextTokenValidationException: The SecurityContextSecurityToken with context-id=urn:uuid:e7a18557-43d9-4395-b7b6-8f9b59d72882 (key generation-id=) is not registered.
at System.ServiceModel.Security.WSSecureConversation.SecurityContextTokenEntry.ReadTokenCore(XmlDictionaryReader reader, SecurityTokenResolver tokenResolver)
at System.ServiceModel.Security.WSSecurityTokenSerializer.ReadTokenCore(XmlReader reader, SecurityTokenResolver tokenResolver)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlReader reader, SecurityTokenResolver tokenResolver, IList`1 allowedTokenAuthenticators, SecurityTokenAuthenticator&amp; usedTokenAuthenticator)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlDictionaryReader reader, Int32 position, Byte[] decryptedBuffer, SecurityToken encryptionToken, String idInEncryptedForm, TimeSpan timeout)
at System.ServiceModel.Security.ReceiveSecurityHeader.ExecuteFullPass(XmlDictionaryReader reader)
at System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessageCore(Message&amp; message, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message&amp; message, TimeSpan timeout)
--- End of inner exception stack trace ---</ExceptionString><InnerException><ExceptionType>System.ServiceModel.Security.Tokens.SecurityContextTokenValidationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The SecurityContextSecurityToken with context-id=urn:uuid:e7a18557-43d9-4395-b7b6-8f9b59d72882 (key generation-id=) is not registered.</Message><StackTrace> at System.ServiceModel.Security.WSSecureConversation.SecurityContextTokenEntry.ReadTokenCore(XmlDictionaryReader reader, SecurityTokenResolver tokenResolver)
at System.ServiceModel.Security.WSSecurityTokenSerializer.ReadTokenCore(XmlReader reader, SecurityTokenResolver tokenResolver)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlReader reader, SecurityTokenResolver tokenResolver, IList`1 allowedTokenAuthenticators, SecurityTokenAuthenticator&amp; usedTokenAuthenticator)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlDictionaryReader reader, Int32 position, Byte[] decryptedBuffer, SecurityToken encryptionToken, String idInEncryptedForm, TimeSpan timeout)
at System.ServiceModel.Security.ReceiveSecurityHeader.ExecuteFullPass(XmlDictionaryReader reader)
at System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessageCore(Message&amp; message, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message&amp; message, TimeSpan timeout)</StackTrace><ExceptionString>System.ServiceModel.Security.Tokens.SecurityContextTokenValidationException: The SecurityContextSecurityToken with context-id=urn:uuid:e7a18557-43d9-4395-b7b6-8f9b59d72882 (key generation-id=) is not registered.
at System.ServiceModel.Security.WSSecureConversation.SecurityContextTokenEntry.ReadTokenCore(XmlDictionaryReader reader, SecurityTokenResolver tokenResolver)
at System.ServiceModel.Security.WSSecurityTokenSerializer.ReadTokenCore(XmlReader reader, SecurityTokenResolver tokenResolver)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlReader reader, SecurityTokenResolver tokenResolver, IList`1 allowedTokenAuthenticators, SecurityTokenAuthenticator&amp; usedTokenAuthenticator)
at System.ServiceModel.Security.ReceiveSecurityHeader.ReadToken(XmlDictionaryReader reader, Int32 position, Byte[] decryptedBuffer, SecurityToken encryptionToken, String idInEncryptedForm, TimeSpan timeout)
at System.ServiceModel.Security.ReceiveSecurityHeader.ExecuteFullPass(XmlDictionaryReader reader)
at System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessageCore(Message&amp; message, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.VerifyIncomingMessage(Message&amp; message, TimeSpan timeout)</ExceptionString></InnerException></Exception></TraceRecord>
We have checked the time on the servers as this can cause authentication issues and these are all OK.
Do I need to be concerned by these errors, or is it just spurious?