Extending dispatcher to prevent message replay

Extending dispatcher to prevent message replay

• Daniela Torres

  First of all, thank you very much for your thoughts. I agree with you when you said asynchronous is better than synchrounous. However, in some situations the end user is expecting the response immediately. Imagine this scenario: unfornately, in Portugal we continue to have many people that goes to a branch to order a bank transfer between accounts and waits for a response from the bank operator. The operator clicks the button on the UI and wait for a message. Even if under the hood all stuff is asynchronous, the customer is waiting for the response: "ok, you have the money in your other account".
  
  But suppose that the system is with problems and the response doesn't arrive. The customer irritated says "you are all imcompetent. I am hurry. Please I want to cancel the order, and I will order again later in the ATM". The operator clicks the button "Cancel the transfer". Now, what should happen Imagine that the original order is in CICS preparing to commit (my orchestration is waiting for a message from the mainframe gateway), when the cancel message arrives to the middleware: what should I do Wait for the original orchestration to finish before process the cancelation message Or something else
  
  Another identical scenario is the withdraw in the ATM. If there is a Timeout, the ATM asks to cancel the transaction.
  
  By the way, I almost forgot it: for now, and to not change the client code programing model I will follow your sugestion to implement a message inspector and an operation invoker. In the future we will try to convince our consumers to change his mindset to asynchrounous.
  
  Thanks again
  
  
  
  

• Phil Graham

  I am trying to understand how what you are trying to do is different from the reliable session feature that we have in the product and which is addressing message loss, duplication detection and ordering.

  What is in your scenario that doesn't allow you to use reliable sessions

  
  
  

• alekz

  Interesting question!!! I can see that you are running in to this situation because you are trying to solve a problem which sounds more asynchronous with a synchronous MEP (message exchange pattern).

  As far as I understand, I think the problem is the extensive time taken by the service to complete its task. But there are some handy ways to solve this rather than polling the result from the client side.

  1. You can switch to the full duplex mode from the simplex mode (request/response or rather). Using a duplex channel you can let the client send the request and forget about it. Then the service will process it and *calls back* the client when the response is ready. Of course you can unleash the power of WS-RM to make sure that your messages are delivered and processed as expected.

  2. If your client (and the service) is more unstable you could think about using an MSMQ channel. This way, your client will send the message and if the service is unavailable the request will be queued in your outgoing message queue. Once the connection is re-established the MSMQ infrastructure will do the necessary transmission for you. Same thing applies to the service end when it tries to send the response back to the client.

  But still if you think you need a way to intercept the calls by extending the service model layer try to use an implementation of IOperationInvoker and hook it up to the service model via an IOperationBehavior. You can create some sort of a server side cache (or in other words a repository) in your IOperationInvoker which looks up for the response based on your input parameters.

  Glad if this helps. I'm really looking forward to hearing your thoughts :).

  Cheers

  UPDATE: Ouch! Too late :). Clemens had posted a better answer when I hit "Post" button on this window! ;-)

   

  
  
  

• mikkk

  I am quite happy that your answer and my answer mention the same things ;) Your answer is as good as mine, Buddhike

  
  
  

• LeBlah

  You can have a separate cancellation queue, holding cancel messages.

  The "checkCancel" method will empty the queue and add the cancellations to a local storage (an array for example).

  Now, for each incoming message you'll first call the "checkCancel" method to make sure no cancellations messages are waiting, and than look for the incoming message transaction id in the cancellations storage.

  This way, if the connection was lost and the MSMQ stored both the original request and the cancel request, both messages will arrive to the server once the connection is restored, and via checking the cancellation queue first you make sure the cancelled order is not executed.

  
  

• Tr1na

  That's a very good question. I have more than one answer for you :-)

  Answer 1 is the behavior mechanics response:

  • Create a message inspector that checks the message against a tracking list you maintain somewhere. If you have already seen the message, add a property to the message indicating that.
  • Create an operation invoker that looks at the operation context and suppresses the invocation if the message has already been seen.
  • On the way out, inject the previous answer into the reply in the message inspector.

  Answer 2 is the channel mechanics response:

  • Write your own protocol channel that implements the functionality on the messaging level and compose that into a binding.

  The problem with answers 1 and 2 is that you will have to figure how long you want to maintain the tracking list (repository) and the associated responses. An hour A day A week How big can that repository grow and what does it mean for your performance Does that make you vulnerable for DoS attacks How does it compose with scale-out clustering

  Answer 3 is the architectural response:

  I would consider your scenario more a question of the overall architectural approach on the application level rather than just one of the underlying messaging infrastructure. If you know from the start that request/response is a risk in your scenario because you might time out the client, you should try to avoid using it.

  Let's think the bank transfer scenario through and separate out a few concerns. I am assuming that you want to make this as reliable as possible.

  A) You need reliable delivery of the information/request towards the backend, because you want to ensure that the data doesn't get lost or duplicated in transit for whatever reason.

  B) You need to receive timely feedback when the requested activity has been performed and need access to the result.

  C) You want to avoid that the business activity that the user intended is only executed exactly once.

  I'll also say that the bank transfer scenario is a bit of a problematic example, because it is quite complex when you drill into the details, but let's go with it:

  A) is a function of the transport you choose. If you need reliable delivery, you have two choices with WCF:

  1. Reliable Session will give you a level of reliability that equivalent to what TCP does for IP. As long as both communicating parties are alive and able to maintain the session and interruptions in the communication path do not exceed a certain timeout period, reliable session will ensure that messages are delivered exactly once and, if you ask for it, in order in case you send multiple messages within the same session.

  2. The MSMQ binding will give you end-to-end durable and transacted message delivery, which will function even if the receiving service is currently not running, the network goes down for an extended period of time, or if the processing transaction on the receiver fails for some reason (including power loss, etc.)

  The additional benefit of using a queue for messages (especially those that represent data of immediate monetary value such as purchase orders or bank transfers which you really don't want to lose) is that because that the service is pulling from the queue rather than the client pushing into the service, the server can be tuned a lot better for optimal reliability and overall throughput. Because the service pulls, it does so on its own schedule and won't be suffering from stress due to sudden spikes in traffic as it would in a Request/Response scenario.

  B) is a matter of what kind of response strategy you choose.

  Request/Response is very convenient from a programming perspective on the client side, but not an optimal strategy if you have long running activities that require reliability. You should consider that every nanosecond of "hang time" between your request and the expected response makes it increasingly more likely that one of the thousands of moving pieces (software, hardware, power, etc.) involved in processing the request fails. So one strategy to make things more reliable is to break the work into small steps and save the interim results on the way (just like I am making incremental backups of this into notepad as I write it).

  One way to break up the work is to choose reliable one-way delivery (possibly with the MSMQ binding). You prepare the request and save it on the application level so you can track it. Then you hand it to the service via WCF and the infrastructure gives you a delivery QoS as I explained in A) based on what binding you pick. Once the service has the data, it saves it. Then the service does its transactional processing. If that fails, the service safely rolls back and retries.

  On success -- this here is the request/response alternative -- the service notifies the client with a one-way message about the outcome or the client comes around and checks the transaction status periodically (polling). The notification can be done with a callback contract for bindings/transports that support duplex and a separate endpoint for those that don't. The polling option applies if there is no communication path from server to client.

  It's a sequence of little steps, but each step can independently recover and each step is transactable.

  C) is an explicit application function and builds on what I just wrote for B)

  The client and the service will likely track activities with some form of transaction number. That is the "request id" you mentioned. If the client resubmits such a transaction for some reason (this might include cancellation/compensation), it creates a new message with the same data content, transaction number included. However, the message is a different one, and sent at a different time, which has implications for security and expiration times, etc.

  The service will look at the transaction number and see in its persisted tracking data store whether the operation has already been performed. If so, it either ignores the request or yields the same reply as for the original request; otherwise it runs the transaction or, possibly, a compensation. That's somewhat similar to my answer 2 above, but it's integrated with the application's transactional data store, not a function of the messaging infrastructure per se.

  The summary is that the level of reliability you seem to be looking for is best achieved by using one of the reliable transport options explained above and by architecting away the request/response and synchronicity constraint.

  I hope that made sense and helps a bit
  Clemens

  
  
  

• samarrin

  
  Maybe Reliable Session Implementation does the work but I wiil need to undestand better the reliable session.
  
  Suppose this situation: I expose a service, let's say a bank account transfer, to my business channels (Internet, Phone, etc.). This service is exposed as a request-reply paradigm. Basicaly, this service instantiates an workflow orchestration to do do some validations, talks to a mainframe gateway to execute the transfer, do some logging , and gives the response. In the client code it was specified a timeout which was exceeded. But in the service side everything is ok: the transfer was executed.
  
  Then, since the client doesn't know what happened in the service side, it begins (another process) to play the same request until it has a response with ok or nok (insuficient money for example). In the service side I want to detect that that message was already processed and give the original response message from a repository, maybe getting from a repository message.
  
  I hope it is clear the scenario.
  
  Thank you
  
  

• Hassmann

  It is not about sync or async it is about duplicate messages. Tell me the truth you just want to avoid the problem of a user clicking twice in the home banking web page J

  They way I solve that is by generating tickets for transactions. The ticket lasts for what 5 minutes 10 minutes whatever! That is the time you've got to make the transaction then it is automatically scavenged.

  Once you submit your message you put the ticket in a header and have your inspector, policy assertion, soap extension, HttpModule, pipeline handler o what ever *** that intercepts the message to look into the bag of tickets if the ticket is present it removes it and executes the transaction.

  That means that if you execute it twice, the second time your are not going to find the ticket in the server and you are going to avoid the duplicate message.

  Doing it the other way around is not safe, how much should you keep your logs how much infrastructure do you need to log all sent messages

  I haven’t seen how the reliable spec has been implemented, but it definitely does the same thing, that is why it needs session, because it generates tickets J

  
  
• Extending dispatcher to prevent message replay