Clock skew issues

RCF support and general discussion.
Post Reply
Volker
Posts: 26
Joined: Wed May 23, 2012 3:27 pm

Clock skew issues

Post by Volker »

Hi Jarl,

we observed some issues within our software due to time modifications (I must think of the famous Rocky Horror Picture Show whenever I make the Time Warp within our simulator ;) ). I found and fixed some locations and I stumbled over the following code sniplets :

(Platform.cpp from RCF 2.0.1.100, also included in older versions)

Code: Select all

    // TODO: any issues with monotonicity of gettimeofday()? <--- Yes !
    boost::uint32_t getCurrentTimeMs()
    {
        static struct timeval start = {0};
        static bool init = false;
        if (!init)
        {
            init = true;
            gettimeofday(&start, NULL);
        }

        struct timeval now;
        gettimeofday(&now, NULL);

        long seconds =  now.tv_sec - start.tv_sec;
        long microseconds = now.tv_usec - start.tv_usec;
        boost::uint64_t timeMs = boost::uint64_t(seconds)*1000 + microseconds/1000;
        timeMs = timeMs & 0xFFFFFFFF;
        return static_cast<boost::uint32_t>(timeMs) - OffsetMs;
    }
Yes, I can confirm that this is an issue. I replaced gettimeofday with clock_gettime( CLOCK_MONOTONIC, ...), fixes the problem for us but may introduce unwanted dependencies on other platforms (Linux needs librt for that, Mac OS X seems to lack this function).

(SessionTimeoutService.cpp from RCF 2.0.1.100, also included in older versions)

Code: Select all

    void SessionTimeoutService::onTimer()
    {
        mSessionsTemp.resize(0);

        mpRcfServer->enumerateSessions(std::back_inserter(mSessionsTemp));

        for (std::size_t i=0; i<mSessionsTemp.size(); ++i)
        {
            SessionStatePtr sessionStatePtr = mSessionsTemp[i].lock();
            if (sessionStatePtr)
            {
                RcfSessionPtr rcfSessionPtr = sessionStatePtr->getSessionPtr();
                if (rcfSessionPtr)
                {
                    boost::uint32_t lastTouched = rcfSessionPtr->getTouchTimestamp();
                    if (lastTouched)
                    {
                        RCF::Timer lastTouchedTimer( lastTouched );
                        if (lastTouchedTimer.elapsed(mSessionTimeoutMs))
                        {
                            rcfSessionPtr->disconnect();
                        }
                    }
                }
            }
        }
    }
I don't understand the line 'if (lastTouched)'. My wild guess is that this prevents a session currently created from getting disconnected. But - as the 'touch time' is directly derived from method getCurrentTimeMs and getCurrentTimeMs may also return 0 - the rare case that an inactive session remains undetected can happen too.

(include/RCF/thread/posix_event.hpp and include/RCF/thread/impl/posix_event.ipp from RCF 2.0.1.100, also included in older versions)

Code: Select all

posix_event::posix_event()
{
>>
+  ::pthread_condattr_init(&condattr_);
+  ::pthread_condattr_setclock(&condattr_, CLOCK_MONOTONIC);
>>

  int error = ::pthread_cond_init(&cond_, 0);
  RCF_VERIFY(error == 0, Exception(_RcfError_ThreadingError("pthread_cond_init()"), error));
}

  template <typename Lock>
  bool timed_wait(Lock& lock, boost::uint32_t waitMs)
  {
      BOOST_ASSERT(lock.locked());

>>
+     struct timespec ts = {0};
 +    clock_gettime(CLOCK_MONOTONIC, &ts);
>>
      // Add waitMs to current time.
      ts.tv_sec += (waitMs / 1000);
      boost::uint32_t remainderMs = waitMs % 1000;
      ts.tv_nsec += (remainderMs * 1000 * 1000);

      // Check for overflow in tv_nsec.
      if (ts.tv_nsec >= 1000*1000*1000)
      {
          BOOST_ASSERT(ts.tv_nsec < 2*1000*1000*1000);
          ts.tv_sec += 1;
          ts.tv_nsec -= 1000*1000*1000;
      }

      // POSIX automatically unlocks and locks the mutex.

      int ret = ::pthread_cond_timedwait(&cond_, &lock.mutex().mutex_, &ts); // Ignore EINVAL.
      if (ret == ETIMEDOUT)
      {
        return false;
      }       

      return true;
  }

Again a monotonic clock to get rid of problems due to clock skew.

Kind Regards,

Volker

Volker
Posts: 26
Joined: Wed May 23, 2012 3:27 pm

Re: Clock skew issues

Post by Volker »

Hello Jarl,

I assume you're quite busy. I just need your confirmation about the points mentioned above so we can write some patches for our next release.

Kind Regards

Volker

jarl
Posts: 238
Joined: Mon Oct 03, 2011 4:53 am
Contact:

Re: Clock skew issues

Post by jarl »

Hi Volker,

This is definitely a valid issue. Are you seeing non-monotonic time when you change the clock settings on the system, or what is the contributing factor?

If you can send me all the files you've changed, I'll integrate the changes, with some #ifdef protection as well, so the code can be enabled or disabled.

Thanks.
Kind Regards

Jarl Lindrud
Delta V Software
http://www.deltavsoft.com

Volker
Posts: 26
Joined: Wed May 23, 2012 3:27 pm

Re: Clock skew issues

Post by Volker »

Yes, when I change the system clock I observe RCF connections getting closed. This happens with those RCF connections where we added a session timeout.

I will send you our patches for Platform.cpp and posix_event.hpp/ipp but you may have to add some ifdef's for the MacOS users. And I learnt that there may also be some AIX xlC users out there but as far as I remember AIX was quite conform to the posix standard (We should ask scobiej).

So far I have no solution for the SessionTimeoutService. Would be adding and using a member like 'touchedAtLeastOnce' a solution ... ?

Regards,

Volker

jarl
Posts: 238
Joined: Mon Oct 03, 2011 4:53 am
Contact:

Re: Clock skew issues

Post by jarl »

Great - if you send the patches through I will make the changes here as well...
Kind Regards

Jarl Lindrud
Delta V Software
http://www.deltavsoft.com

Volker
Posts: 26
Joined: Wed May 23, 2012 3:27 pm

Re: Clock skew issues

Post by Volker »

Sorry for the delay, pls find the necessary patches in the attachments.

I tested those patches against RCF-2.0.0.2685 and RCF-2.0.1.101 on Ubuntu Linux. The monotonic_clock - patches additionally need linking against librt. There may be issues when trying to use these patches on other platforms (notably Mac OS X).

One thing I'm not sure about :

- The session timeout will be ignored until a call to 'RcfSession::touch' has been taken place for the first time after an active session has been accomplished. Is this correct ?
- I added a reset for the session timeout member 'RcfSession::mTouchTimestamp' in method 'RcfSession::disconnect'. Is this necessary / correct ?

Kind Regards

Volker
Attachments
RCF_session_timeout.patch
(3.22 KiB) Downloaded 478 times
RCF_current_time_monotonic_clock.patch
(1.22 KiB) Downloaded 493 times
RCF_condition_monotonic_clock.patch
(1.39 KiB) Downloaded 480 times

Volker
Posts: 26
Joined: Wed May 23, 2012 3:27 pm

Re: Clock skew issues

Post by Volker »

Hello Jarl,

today I stumbled over release 2.1.0.0 and it seems to me that the clock skew-problem is still present. Do you plan to address this problem in the next release ?

Kind Regards

Volker

jarl
Posts: 238
Joined: Mon Oct 03, 2011 4:53 am
Contact:

Re: Clock skew issues

Post by jarl »

Hi Volker,

I've just imported the changes you previously attached - thanks for that. I will send you a patch you can apply to RCF 2.1 , just so you can verify that it works the way you expect it to. You will need to define RCF_USE_CLOCK_MONOTONIC in your build, for the changes to take effect.
Kind Regards

Jarl Lindrud
Delta V Software
http://www.deltavsoft.com

Post Reply