Multiple outstanding calls using Asynchronous RCF?

[gentzel]

My company has a product that uses RCF extensively for a client-server application.

One oddity about how we use RCF is that there are three distinct RcfClients used for the client/server communication. This was done before my tenure and the justification was that one of the clients could have fairly long running calls which should not prevent some other calls from being processed (i.e. allow out-of-band calls separate from the main processing). All of the calls on these RcfClients are synchronous.

My question is whether RCF can support such a model on a single RcfClient by using asynchronous RCF. In other words, can asynchronous RCF differentiate between multiple outstanding calls and appropriately route their responses when they are received?

Thanks.

[jarl]

Hi,

That's correct - you need to use distinct RcfClient<>'s for concurrent asynchronous calls. RCF currently does not support multiple outstanding calls on a single connection.

[Jeremy Viehland]

Jarl,

Before choosing to abandon RCF, would you be able to comment on the technical feasibility of implementing multiple asynchronous calls on a single RcfClient / TCP connection?

We have a terrible hack that we've had to implement to work around this lack of this feature. In order to implement server-side load balancing, all of our incoming TCP connections had to have an embedded GUID so the load balancer could forward the entire set of connections to the same back-end server. This has unfortunately tied our software to a particular load-balancer. This, in turn, prevents us from taking advantage of geographical load balancing offered by some vendors.

We're now faced with a choice of whether to attempt to refactor RCF to implement this feature or to abandon our RCF implementation in favor of an RPC library that can use a single TCP connection.

-Jeremy

p.s., Here is the important part of the failing unit test -- perhaps we're doing something incorrectly?

Error thrown:

Code: Select all

\RCF\include\RCF/Future.hpp(241): [Thread: 32312][Time: 31] RCF exception thrown. Error message: Error: multiple concurrent calls attempted on the same RcfClient<> object. To make concurrent calls, use multiple RcfClient<> objects instead.

Code: Select all

/// @test confirming that one RcfClient object cannot make two asynchronous calls.
TEST(RCFTest, test_two_async_requests_from_one_RcfClient_objectrr) {
  cdsLogDebug("Entered");
  RcfInitDeinit rcfInit;
  int rcfLogLevel = 2; // 0=off, 1=asserts, 2 = default, 4 = max
  enableLogging(LogToFunc(cdsLogRcf), rcfLogLevel);

  RCFTestServer rcfTestServer;
  TcpEndpoint tcpEndpoint(RCFTestServer::ServerHost.getPtr(), RCFTestServer::ServerPort);
  RcfServerAutoPtr rcfServerAutoPtr;
  rcfServerAutoPtr.reset(new RcfServer(tcpEndpoint));
  rcfServerAutoPtr->bind<I_RCFTest>(rcfTestServer);
  ThreadPoolPtr threadPoolPtr(new RCF::ThreadPool(2));
  rcfServerAutoPtr->setThreadPool(threadPoolPtr);
  rcfServerAutoPtr->start();

  RCFTestPtr rcfClient1(new RcfClient<I_RCFTest>(tcpEndpoint));

  const int expected = 42;
  
  Future<long> fPrintSlowDuration;
  fPrintSlowDuration = rcfClient1->PrintSlow( AsyncTwoway( bind(onCallCompleted<long>, rcfClient1, fPrintSlowDuration)), "print me slowly");

  Future<int> futureValue;
  ASSERT_NO_THROW(futureValue = rcfClient1->GetValue( AsyncTwoway( bind(onCallCompleted<int>, rcfClient1, futureValue))), RCF::Exception);
  cdsLogDebug("OK");
}

[jarl]

Hi Jeremy,

The use case with the load balancers is quite convincing - thanks for bringing that up. I think we definitely need to fix this.

What I will do is make this the top priority feature for the next release. Once I start working on it I will be in a better place to determine if there are any serious obstacles. I don't think there will be, but the devil is in the details... In any case, I can keep you posted on progress.

[Jeremy Viehland]

Jarl,

Could I add another request to that?

(There's no point in requesting a feature without nailing down the requirements.)

There is another unit test that would greatly reduce the amount of refactoring that we would require in order to take advantage of the proposed multiple outstanding asynchronous calls. At the moment, as I mentioned, in our code there are three RcfClient objects. If it were possible to cause those three objects to share a ClientTransport object (which from the name I am assuming is responsible for the TCP connection), then we would eliminate the need to re-write our three RCF interfaces (and eliminate any bugs that might be introduced in the process). Note that our TCP connection is using transport filters to add OpenSSL (not sure if this matters, but we can't afford to have that break, so thought I'd mention it).

Below is a second failing unit test that we had hoped would allow a simple solution to our problems. In this test, the calls are synchronous but if the transport object could be shared and these calls additionally be asynchronous in a backwards compatible way, then a small change on our end would provide many benefits. Not only would we be able to utilize geographic load balancing features of some vendors, but we wouldn't have to maintain Tcl scripts for ours! (Oops, did I give away their name?)

Thank you for your quick response!

-Jeremy

Code: Select all

/// @test that shows two RcfClient objects can NOT share the same transport.
TEST(RCFTest, DISABLED_test_two_sync_requests_from_two_RcfClient_objects_on_two_threads_with_shared_transport) {
  cdsLogDebug("Entered");
  RcfInitDeinit rcfInit;
  enableLogging(LogToFunc(cdsLogRcf), 2); // 0=off, 1=asserts, 2 = default, 4 = max

  RCFTestServer rcfTestServer;
  TcpEndpoint tcpEndpoint(RCFTestServer::ServerHost.getPtr(), RCFTestServer::ServerPort);
  RcfServerAutoPtr rcfServerAutoPtr;
  rcfServerAutoPtr.reset(new RcfServer(tcpEndpoint));
  rcfServerAutoPtr->bind<I_RCFTest>(rcfTestServer);
  ThreadPoolPtr threadPoolPtr(new RCF::ThreadPool(2));
  rcfServerAutoPtr->setThreadPool(threadPoolPtr);
  rcfServerAutoPtr->start();
  
  RCFTestPtr rcfClient1(new RcfClient<I_RCFTest>(tcpEndpoint));
  RCFTestPtr rcfClient2(new RcfClient<I_RCFTest>(tcpEndpoint));

  // Copy the transport
  auto& transport = rcfClient1->getClientStub().getTransport();
  RCF::ClientTransportAutoPtr transportAutoPtr(&transport);
  rcfClient2->getClientStub().setTransport(transportAutoPtr);

  // Client 1 is a slow sync call on another thread.
  struct RcfCaller : public Runnable {
    RcfCaller(RCFTestPtr rcfClient) : rcfClient_(rcfClient) {}
    virtual void run() {
      rcfClient_->PrintSlow("print me slowly");
    }
  private:
    RCFTestPtr rcfClient_;
  };
  Ref<RcfCaller> rcfCaller(new RcfCaller(rcfClient1));
  WorkerThread workerThread(rcfCaller);

  // Client 2 is multiple sync calls on the current thread.
  Thread::sleep(500);
  rcfClient2->PrintFast("quick call from another thread");
  Thread::sleep(500);
  rcfClient2->PrintFast("quick call from another thread");

  rcfClient2->getClientStub().releaseTransport();
  cdsLogDebug("OK");
}

[jarl]

Yep, I'm aiming to get that fixed as well. It is pretty much the same issue.

[Jeremy Viehland]

Jarl,

Thank you! Looking forward to the next release!

-Jeremy