Open Source Synthesis SyncML Engine: PIM data synchronization and more ()

headers satyadev mussai | 3 May 2010 14:56 · /Ximian/

_______________________________________________
os-libsynthesis mailing list
os-libsynthesis@...
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis

On Fri, 2010-04-30 at 17:12 +0100, Lukas Zeller wrote:
> as promised I added server progress events now to the engine. See
> updates in "luz" branch (in particular b15e49512c (engine 3.4.0.7:
> Progress events now session local and available for server sessions)).
> 
> It works exactly the same as in clients - SessionStep returns with
> stepCmd==STEPCMD_PROGRESS when there is a progress event to show.

Works like a charm. Thanks a lot! I merged your "luz" branch into our
master on moblin.org and in the last few days, ran the SyncEvolution
testing using that new code, with no regressions so far. We'll use this
in SyncEvolution 1.0.

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

On Mon, 2010-04-26 at 13:40 +0100, Patrick Ohly wrote:
> On Wed, 2010-04-21 at 09:39 +0200, Patrick Ohly wrote:
> > I have a meeting in my Evolution calendar which I synchronize with
> > SyncEvolution. I noticed that the outgoing VEVENT has start and end
> > time
> > converted to UTC times which are one hour off.
> > 
> > Here's the stripped down event:
> > 
> > BEGIN:VCALENDAR
> > PRODID:-//Ximian//NONSGML Evolution Calendar//EN
> > VERSION:2.0
> > METHOD:PUBLISH
> > BEGIN:VTIMEZONE
> > TZID:GMT Standard Time
> > BEGIN:STANDARD
> > DTSTART:16010101T020000
> > TZOFFSETFROM:+0100
> > TZOFFSETTO:+0000
> > RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
> > END:STANDARD
> > BEGIN:DAYLIGHT
> > DTSTART:16010101T010000
> > TZOFFSETFROM:+0000
> > TZOFFSETTO:+0100
> > RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
> > END:DAYLIGHT
> > END:VTIMEZONE
> > BEGIN:VEVENT
> > SUMMARY:test event
> > DTSTART;TZID=GMT Standard Time:20100421T150000
> > DTEND;TZID=GMT Standard Time:20100421T160000
> > UID:040000008200E00074C5B7101A82E0080000000000B5A586D0A4CA01000000000000000010
> > CLASS:PUBLIC
> > PRIORITY:5
> > DTSTAMP:20100415T035521Z
> > TRANSP:OPAQUE
> > STATUS:CONFIRMED
> > SEQUENCE:4
> > END:VEVENT
> > END:VCALENDAR
> 
> This fails because RRULE2toInternal() in rrule.cpp fails to convert the
> RRULE properties into the internal format. It bombs out here:
> 
>   switch (freq)
>   {
> ...
>     case 'Y' :
>       if (byday == "" ||
>          (byday.length() == 2 && byday[0] ==
> RRULE_weekdays[startwday][0] &&
>                                  byday[1] ==
> RRULE_weekdays[startwday][1]))
>       {
> ...
>       }
>       else
>       {
>         goto incompat;
>       }
> 
> It reaches the "goto incompat" because byday == "-1SU".
> 
> Recurrence rules with "last day in month ..." are common for VTIMEZONEs.
> Beat, is that really something that is not handled by the Synthesis code
> at the moment? What would be necessary to add support for it?

I'm still wondering about this.

In the meantime I looked into VTIMEZONEs which parse successfully. What
I found is that FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10 (as in the example
above) is supported, by converting it to the equivalent
FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU.

However, there is a sanity check that compares the start month:
  if (DayMonthCombi( freq, byday,bymonth,startmonth )) {
    freq    = 'M';         // a different model will be chosen for this case,
    interval= 12*interval; // which is in fact aequivalent
    bymonth = "";
  } // if

In the example above, DTSTART=16010101T020000 and thus startmonth !=
bymonth. DayMonthCombi() then returns false.

Does the month in DTSTART really matter in this case? I would expect
that the first recurrence will be in the month specified in BYMONTH
which is >= than the month in DTSTART, so one could apply the conversion
to FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU by setting the month in DTSTART
to the value from BYMONTH.

Does that make sense?

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Hello Patrick,

Thanks for the analysis. I changed the rrule parsing routine to allow returning a adjusted DTSTART so the
engine now correctly detects this iCalendar's timezone as GMT. I just pushed these changes into the luz
branch on synthesis indefero.

Best Regards,

Lukas

On May 4, 2010, at 22:40 , Patrick Ohly wrote:

> On Mon, 2010-04-26 at 13:40 +0100, Patrick Ohly wrote:
>> On Wed, 2010-04-21 at 09:39 +0200, Patrick Ohly wrote:
>>> I have a meeting in my Evolution calendar which I synchronize with
>>> SyncEvolution. I noticed that the outgoing VEVENT has start and end
>>> time
>>> converted to UTC times which are one hour off.
>>> 
>>> Here's the stripped down event:
>>> 
>>> BEGIN:VCALENDAR
>>> PRODID:-//Ximian//NONSGML Evolution Calendar//EN
>>> VERSION:2.0
>>> METHOD:PUBLISH
>>> BEGIN:VTIMEZONE
>>> TZID:GMT Standard Time
>>> BEGIN:STANDARD
>>> DTSTART:16010101T020000
>>> TZOFFSETFROM:+0100
>>> TZOFFSETTO:+0000
>>> RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
>>> END:STANDARD
>>> BEGIN:DAYLIGHT
>>> DTSTART:16010101T010000
>>> TZOFFSETFROM:+0000
>>> TZOFFSETTO:+0100
>>> RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
>>> END:DAYLIGHT
>>> END:VTIMEZONE
>>> BEGIN:VEVENT
>>> SUMMARY:test event
>>> DTSTART;TZID=GMT Standard Time:20100421T150000
>>> DTEND;TZID=GMT Standard Time:20100421T160000
>>> UID:040000008200E00074C5B7101A82E0080000000000B5A586D0A4CA01000000000000000010
>>> CLASS:PUBLIC
>>> PRIORITY:5
>>> DTSTAMP:20100415T035521Z
>>> TRANSP:OPAQUE
>>> STATUS:CONFIRMED
>>> SEQUENCE:4
>>> END:VEVENT
>>> END:VCALENDAR
>> 
>> This fails because RRULE2toInternal() in rrule.cpp fails to convert the
>> RRULE properties into the internal format. It bombs out here:
>> 
>>  switch (freq)
>>  {
>> ...
>>    case 'Y' :
>>      if (byday == "" ||
>>         (byday.length() == 2 && byday[0] ==
>> RRULE_weekdays[startwday][0] &&
>>                                 byday[1] ==
>> RRULE_weekdays[startwday][1]))
>>      {
>> ...
>>      }
>>      else
>>      {
>>        goto incompat;
>>      }
>> 
>> It reaches the "goto incompat" because byday == "-1SU".
>> 
>> Recurrence rules with "last day in month ..." are common for VTIMEZONEs.
>> Beat, is that really something that is not handled by the Synthesis code
>> at the moment? What would be necessary to add support for it?
> 
> I'm still wondering about this.
> 
> In the meantime I looked into VTIMEZONEs which parse successfully. What
> I found is that FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10 (as in the example
> above) is supported, by converting it to the equivalent
> FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU.
> 
> However, there is a sanity check that compares the start month:
>  if (DayMonthCombi( freq, byday,bymonth,startmonth )) {
>    freq    = 'M';         // a different model will be chosen for this case,
>    interval= 12*interval; // which is in fact aequivalent
>    bymonth = "";
>  } // if
> 
> In the example above, DTSTART=16010101T020000 and thus startmonth !=
> bymonth. DayMonthCombi() then returns false.
> 
> Does the month in DTSTART really matter in this case? I would expect
> that the first recurrence will be in the month specified in BYMONTH
> which is >= than the month in DTSTART, so one could apply the conversion
> to FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU by setting the month in DTSTART
> to the value from BYMONTH.
> 
> Does that make sense?
> 
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 
> 
> _______________________________________________
> os-libsynthesis mailing list
> os-libsynthesis@...
> http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis

Lukas Zeller (luz@...)
- 
Synthesis AG, SyncML Solutions  & Sustainable Software Concepts
info@..., http://www.synthesis.ch

On Fri, 2010-05-07 at 12:53 +0100, Lukas Zeller wrote:
> Thanks for the analysis. I changed the rrule parsing routine to allow
> returning a adjusted DTSTART so the engine now correctly detects this
> iCalendar's timezone as GMT. I just pushed these changes into the luz
> branch on synthesis indefero.

Confirmed, works for me.

I should have mentioned that I had already made similar changes as you
on the moblin.org master branch (check for NULL obj in debug macros,
enable debug logging inside vtimezone code when invoked by mimedir
profile parser).

My own debug logging patches go a bit further:
- log an error when parsing a VTIMEZONE fails
- logging of libical timezone conversion
- logging of timezone matching

I'm trying to track down an unexpected timezone failure with this which
occurs when normal Evolution events are involved.

Do you agree with these changes? Would you mind merging them? I rebased
them on your "luz" branch.

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Hi Patrick,

On May 10, 2010, at 9:04 , Patrick Ohly wrote:

> Confirmed, works for me.

Thanks!

> I should have mentioned that I had already made similar changes as you
> on the moblin.org master branch (check for NULL obj in debug macros,
> enable debug logging inside vtimezone code when invoked by mimedir
> profile parser).
> 
> My own debug logging patches go a bit further:
> - log an error when parsing a VTIMEZONE fails
> - logging of libical timezone conversion
> - logging of timezone matching
> 
> I'm trying to track down an unexpected timezone failure with this which
> occurs when normal Evolution events are involved.
> 
> Do you agree with these changes? Would you mind merging them? I rebased
> them on your "luz" branch.

The attempt to init the time zones after reading config (fc000b0f45 (TSyncAppBase: moved system time zone
parsing after config reading)) creates a chicken and egg problem. The config reading process relies
itself on the time zone stuff being initialized, most prominently for <definetimezone>, but also for
other tags that have ISO8601 timestamp values.

Maybe the dilemma can be solved by adding an optional, second initialize run after config reading that can
be switched on specifically to track down timezone init problems. Another option would be to use the
ConferrPuts() channel, which is ready before config is read by definition.

Best Regards,

Lukas Zeller (luz@...)
- 
Synthesis AG, SyncML Solutions  & Sustainable Software Concepts
info@..., http://www.synthesis.ch

Hello!

I've just pushed some experimental code to a "doxygen" branch on
moblin.org:

commit 7d9ecc9316d12412cc72c1c41a0889dbeeeb06a2
Author: Patrick Ohly <patrick.ohly@...>
Date:   Tue May 11 14:12:42 2010 +0200

    log call location of debug messages

    For someone unfamiliar with the source code, it is sometimes hard
    to locate the code which produced a certain message. This patch
    solves this issue like this:
    - produce a fully cross-referenced HTML version of the source
      with Doxygen
    - extend the logging functions and macros so that each message
      comes with file name, line number and function name information
    - turn the time stamp in the HTML output into a link to the source
      code line, with the function name as title

    The reason for picking the time stamp was that making other text a
    link would interfere with the visual markup of it (color, bold, etc.).

    In such an enriched log one can:
    - hover over the time stamp to see function name in a bubble help
      and the URL (and thus source code location) at the bottom of the
      browser window (at least in Firefox)
    - click on the time stamp to jump to the source code

    Because this feature makes both the library and the logs larger,
    it is turned off by default. Configure with --enable-debug-logs
    to enable it when using autotools. Macro tricks are used to
    produce the same code as before when the feature is off.

    Limitations:
    - Without time stamp logging, no links are recorded.
    - Each change in the Synthesis source code forces a complete
      new run of Doxygen. This makes compilation quite a bit slower,
      but the only alternative would be to accept that logs and source dump
      are inconsistent in the changed parts.
    - The links contain the absolute file path (making it a bit harder
      to share logs and source code dumps).
    - The TClientEngineInterface::debugPuts() API introduced for and used
      by SyncEvolution leads to links which contain the wrong absolute
      path to the Doxygen HTML files (inside libsynthesis instead of
      SyncEvolution).
    - The "prefix" parameter in that same API is still ignored.

You can an example of an extended log and the corresponding Doxygen
files here:
http://downloads.syncevolution.org/tmp/synthesis-html.tar.gz

Unpack it in /tmp, then point your web browser to
file:///tmp/home/pohly/.cache/syncevolution/scheduleworld__1 <at> client_+test_+1-2010-05-11-14-04/syncevolution-log.html

Is that something that would be acceptable for core libsynthesis? I find
it useful as it is, despite the limitations.

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

On Mon, 2010-05-10 at 16:48 +0100, Lukas Zeller wrote:
> Hi Patrick,
> 
> On May 10, 2010, at 9:04 , Patrick Ohly wrote:
> 
> > Confirmed, works for me.
> 
> Thanks!
> 
> > I should have mentioned that I had already made similar changes as you
> > on the moblin.org master branch (check for NULL obj in debug macros,
> > enable debug logging inside vtimezone code when invoked by mimedir
> > profile parser).
> > 
> > My own debug logging patches go a bit further:
> > - log an error when parsing a VTIMEZONE fails
> > - logging of libical timezone conversion
> > - logging of timezone matching
> > 
> > I'm trying to track down an unexpected timezone failure with this which
> > occurs when normal Evolution events are involved.
> > 
> > Do you agree with these changes? Would you mind merging them? I rebased
> > them on your "luz" branch.
> 
> The attempt to init the time zones after reading config (fc000b0f45
> (TSyncAppBase: moved system time zone parsing after config reading))
> creates a chicken and egg problem. The config reading process relies
> itself on the time zone stuff being initialized, most prominently for
> <definetimezone>, but also for other tags that have ISO8601 timestamp
> values.

I see.

> Maybe the dilemma can be solved by adding an optional, second
> initialize run after config reading that can be switched on
> specifically to track down timezone init problems.

That would imply redoing some of the work. What about the following
(code in master, as I need to fix that anyway):

commit 7bbec4fd20cde059973d4d1c840fcee85d947e60
Author: Patrick Ohly <patrick.ohly@...>
Date:   Tue May 11 16:57:35 2010 +0200

    time zone init and logging: split into part before and after config parsing

    This patch reverts the "move system time zone parsing after config
    reading" part of commit fc000b0f45. As Lukas pointed out, config parsing
    already depends on time zones, for example for <definetimezone>.

    But this patch keeps the finishConfig() callback and uses it to invoke
    a second global time zone function. On Linux, that function is used
    to add the libical time zone definitions and logs them at the same time.
    Builtin time zones are added at the original time, before parsing the
    config.

    This is a compromise between having no time zones while parsing the config
    and not logging the libical time zones. It works for SyncEvolution, because
    it doesn't reference the additional time zones in the config.

>  Another option would be to use the ConferrPuts() channel, which is
> ready before config is read by definition.

That won't help, because I need this information in the normal log file
to simplify remote debugging.

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Hello Patrick,

that's a very cool addition indeed  Thanks!

I'm about to integrate it here, and I have added config options for it, such that I can choose to create
txmt:// links (which directly open TextMate or BBEdit code editors on my Mac with the cursor on the right
line) instead of doxygen. A second config option allows to override the default base path (the one you
define with SYDEBUG_LOCATION).

I'll post the changes soon together with the other pending changes on my branch.

Best Regards,

Lukas

On May 11, 2010, at 16:22 , Patrick Ohly wrote:

> Hello!
> 
> I've just pushed some experimental code to a "doxygen" branch on
> moblin.org:
> 
> commit 7d9ecc9316d12412cc72c1c41a0889dbeeeb06a2
> Author: Patrick Ohly <patrick.ohly@...>
> Date:   Tue May 11 14:12:42 2010 +0200
> 
>    log call location of debug messages
> 
>    For someone unfamiliar with the source code, it is sometimes hard
>    to locate the code which produced a certain message. This patch
>    solves this issue like this:
>    - produce a fully cross-referenced HTML version of the source
>      with Doxygen
>    - extend the logging functions and macros so that each message
>      comes with file name, line number and function name information
>    - turn the time stamp in the HTML output into a link to the source
>      code line, with the function name as title
> 
>    The reason for picking the time stamp was that making other text a
>    link would interfere with the visual markup of it (color, bold, etc.).
> 
>    In such an enriched log one can:
>    - hover over the time stamp to see function name in a bubble help
>      and the URL (and thus source code location) at the bottom of the
>      browser window (at least in Firefox)
>    - click on the time stamp to jump to the source code
> 
>    Because this feature makes both the library and the logs larger,
>    it is turned off by default. Configure with --enable-debug-logs
>    to enable it when using autotools. Macro tricks are used to
>    produce the same code as before when the feature is off.
> 
>    Limitations:
>    - Without time stamp logging, no links are recorded.
>    - Each change in the Synthesis source code forces a complete
>      new run of Doxygen. This makes compilation quite a bit slower,
>      but the only alternative would be to accept that logs and source dump
>      are inconsistent in the changed parts.
>    - The links contain the absolute file path (making it a bit harder
>      to share logs and source code dumps).
>    - The TClientEngineInterface::debugPuts() API introduced for and used
>      by SyncEvolution leads to links which contain the wrong absolute
>      path to the Doxygen HTML files (inside libsynthesis instead of
>      SyncEvolution).
>    - The "prefix" parameter in that same API is still ignored.
> 
> You can an example of an extended log and the corresponding Doxygen
> files here:
> http://downloads.syncevolution.org/tmp/synthesis-html.tar.gz
> 
> Unpack it in /tmp, then point your web browser to
> file:///tmp/home/pohly/.cache/syncevolution/scheduleworld__1 <at> client_+test_+1-2010-05-11-14-04/syncevolution-log.html
> 
> Is that something that would be acceptable for core libsynthesis? I find
> it useful as it is, despite the limitations.
> 
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 
> 
> _______________________________________________
> os-libsynthesis mailing list
> os-libsynthesis@...
> http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis

Lukas Zeller (luz@...)
- 
Synthesis AG, SyncML Solutions  & Sustainable Software Concepts
info@..., http://www.synthesis.ch

On Wed, 2010-05-12 at 17:43 +0100, Lukas Zeller wrote:
> that's a very cool addition indeed  Thanks!

Glad you like it 

> I'm about to integrate it here, and I have added config options for
> it, such that I can choose to create txmt:// links (which directly
> open TextMate or BBEdit code editors on my Mac with the cursor on the
> right line) instead of doxygen. A second config option allows to
> override the default base path (the one you define with
> SYDEBUG_LOCATION).

Something similar might also be useful on Linux. These Doxygen runs are
fairly slow. Something for later...

> I'll post the changes soon together with the other pending changes on my branch.

Note that I updated "master" and "doxygen" once more today. Please pull
again before merging.

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

2	March 2013
4	December 2012
1	October 2012
1	September 2012
1	August 2012
14	June 2012
19	May 2012
7	April 2012
17	March 2012
2	February 2012
1	December 2011
12	October 2011
34	September 2011
18	August 2011
13	July 2011
19	June 2011
3	May 2011
2	April 2011
3	February 2011
5	January 2011
23	December 2010
5	November 2010
1	October 2010
10	September 2010
3	August 2010
4	June 2010
22	May 2010
8	April 2010
49	March 2010
50	February 2010
22	January 2010
27	December 2009
34	November 2009
62	October 2009
80	September 2009
29	August 2009
43	July 2009
72	June 2009
10	May 2009

Newbie: Create a new instance of engine + pass a valid XML to be stored in a DBMS

Re: Server progress events

Re: wrong interpretation of timezone information

Re: wrong interpretation of timezone information

Re: wrong interpretation of timezone information

Re: wrong interpretation of timezone information

logging the source code location of debug messages

Re: wrong interpretation of timezone information

Re: logging the source code location of debug messages

Re: logging the source code location of debug messages

Mon	Tue	Wed	Thu	Fri	Sat	Sun

					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31