Patrick Ohly | 29 Aug 10:46 2014
Picon

avoiding disk writes during sync without changes

Hello!

For IVI, I am working on cloud syncing. When syncing against a WebDAV
server, SyncEvolution runs a two-way sync with binfile client on the
WebDAV side and a server with admin data handled by SyncEvolution on the
other side.

One goal for IVI is to minimize or better, avoid disk writes, because
flash storage must last as long as possible.

The most common case is that nothing changed on either side. In this
case, libsynthesis unnecessarily updates nonce (even if not used; I've
already patched that (1)), sync anchors (again, I have a patch for this:
skip writing of admin data after detecting the special case (2)) and the
change log in the binfile client.

This last write happens in TBinfileImplDS::changeLogPreflight(). The
changes are minor, just a few bytes change. I suspect that these are the
time stamps and modcount embedded in the log.

Would it be possible to check in changeLogPreflight() how significant
the changes are? If there were no item changes, what would be the effect
of not updating the header?

There are two cases where that can happen with the attached patch:
- the processes crashes
- SyncEvolution skips the session shutdown, see (2).

(1) For local sync, <requestedauth>none</requestedauth> and
<requiredauth>none</requiredauth> are used. Patch attached. Okay?
(Continue reading)

Patrick Ohly | 11 Jul 15:27 2014
Picon

comparison of PHOTO data

Hello!

I'm currently investigating why a comparison of two PHOTO fields of
different length returns "field equal". PHOTO is defined as:

      <field name="PHOTO" type="string" compare="conflict" merge="fillempty"/>

I think the specific situation is that the two values contain a null
byte somewhere in the middle, and the part before that is equal.

I think the following code does the comparison, doesn't it?

sInt16 TStringField::compareWith(TItemField &aItemField, bool aCaseInsensitive)
{
  sInt16 result;
  PULLFROMPROXY;
  if (aItemField.isBasedOn(fty_string)) {
    TStringField *sfP = static_cast<TStringField *>(&aItemField);
    #ifdef STREAMFIELD_SUPPORT
    sfP->pullFromProxy(); // make sure we have all chars
    #endif
    // direct compare possible, return strcmp
    if (aCaseInsensitive)
      result=strucmp(fString.c_str(),sfP->fString.c_str());
    else
      result=strcmp(fString.c_str(),sfP->fString.c_str());
  }
...

We have std::string as value and therefore can store null bytes as part
(Continue reading)

anuj chauhan | 19 May 15:16 2014
Picon

How to run syncevolution at bare minimum processing.

Hi ,


I am trying to run 300 concurrent sync-sessions from my machine with a server.
I have a quad-core machine with 8gb ram and these processes are consuming about 90-94% of the cpu usage.I want to tune syncevolution so that it could run on bare minimum processing.For this i have done following modification :

1.      Commented out the code which gets the Server DevInf. I suspect, this will reduce load on Server and client as well.

2.       Removed the file synccompare. During these concurrent runs, top command showed high CPU usage for Perl just after syncevolution completed its exercise. By removing this file, this reduced the client load and processing.

3.       Reduced the log level to 0. Number of IO went down along with the disk usage.

 

For server load testing i am not concerned with data loss and want minimal logging.

Can you suggest some more pieces of code/functionality which can be disabled so that the overall CPU usage goes down?



Regards
Anuj Chauhan

_______________________________________________
os-libsynthesis mailing list
os-libsynthesis@...
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Patrick Ohly | 14 May 21:15 2014
Picon

Re: mixing properties with and without group tag (was: Re: suppressempty + arrays)

On Wed, 2014-05-14 at 17:44 +0200, Lukas Zeller wrote:
> Hello Patrick,
> 
> On 12.05.2014, at 18:25, Patrick Ohly <patrick.ohly@...> wrote:
> 
> > [...] 
> >> and a property which has a group tag must not reuse any of these
> >> unassigned group tag values.
> > 
> > Actually, a "property which has a group *field*" - it doesn't matter
> > whether the current property has a group tag value. This check was
> > missing. Attached a patch adding it, in a brute-force manner. Does that
> > look right?
> 
> Yes, it looks right to me. I guess you've found out in the meantime if it also *works* right :-)

Yes, it works for me. But it's good that you had a look at it anyway,
because my vCard profile certainly isn't representative.

I'll include this and the other patches (including the removal of
backslash escaping in parameters!) in the master branch of libsynthesis
on freedesktop.org once it passed all my tests. There's also a memory
leak fix for SWAP().

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
Patrick Ohly | 12 May 12:55 2014
Picon

suppressempty + arrays

Hello!

What's the intended behavior of suppressempty for arrays?

        "suppressempty": This optional boolean value. can be set to true if this property must
        never be sent empty (without values). This is the case for most vCalendar fields, for example.

The default seems to be suppressempty=no (judging from the "can be set
to true"). When I have arrays with empty entries, will those be
generated?

Example:

- 35 :     string RELATEDNAMES    [   0,   0,     0] : <array with 3
elements>
                                     -- element    0 : <empty>
                                     -- element    1 : <empty>
                                     -- element    2 : <empty>
...
- 83 :     string LABEL           [   0,   0,     0] : <array with 3 elements>
                                     -- element    0 : "Spouse"
                                     -- element    1 : "Manager"
                                     -- element    2 : "Assistant"

        <property name="X-ABRELATEDNAMES" groupfield="GROUP_TAG">
          <value field="RELATEDNAMES"/>
          <position field="RELATEDNAMES" repeat="array" increment="1" minshow="0"/>
          <parameter name="X-ABLabel" rule="HAVE-ABLABEL-PARAMETER" sharedfield="yes">
            <value field="LABEL"/>
          </parameter>
        </property>
...
        <property name="X-ABLabel" suppressempty="yes" groupfield="GROUP_TAG" rule="HAVE-ABLABEL-PROPERTY">
          <value field="LABEL" repeat="array" increment="1" minshow="0"/>
          <position field="LABEL" repeat="array" increment="1" minshow="1"/>
        </property>

This leads to:

BEGIN:VCARD
VERSION:3.0
PRODID:-//Synthesis AG//NONSGML SyncML Engine V3.4.0.47//EN
REV:20140512T093706Z
UID:syuid384197.212266690650521
N:parserbug=;;;;
FN:parserbug=
X-EVOLUTION-FILE-AS:parserbug=
NICKNAME:user17
NOTE:triggers parser bug in Funambol 3.0: trailing = is mistaken for soft l
 ine break=
item3.X-ABLabel:Spouse
item2.X-ABLabel:Manager
item1.X-ABLabel:Assistant
END:VCARD

The X-ABRELATEDNAMES properties were not generated. The labels should be
redundant, but some peers get confused. Google preserves them as
stand-alone X-ABLabel without tag. DAViCal preserves them with tag,
which then happened to confuse SyncEvolution's conversion code (separate
issue).

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
Patrick Ohly | 7 May 20:20 2014
Picon

[+ x] for next field

Hello!

The documentation mentions:

        In addition there is a special form of array index that can be
        used to access fields in a field list
        (see 10.1) by index instead of by name. This special form of
        array index starts with a + sign as the
        first character after the opening [ as shown in the following
        example:
        <!-- a sample field list -->
        <fieldlist>
        <field name="NAME" type="string"/>
        <field name="TEL_1" type="telephone"/>
        <field name="TEL_2" type="telephone"/>
        <field name="TEL_3" type="telephone"/>
        </fieldlist>

        /* sample script to access the telephone numbers by index
        instead of by name */
        integer i;
        telephone a,b,c;
        a = TEL_1[+0]; // this is the same as: a=TEL_1
        b = TEL_1[+1]; // this is the same as: b=TEL_2
        c = TEL_1[+2]; // this is the same as: c=TEL_3

Does this also work for fields which are arrays? Like this:

      <field name="TEL"         array="yes" type="telephone" compare="conflict"/>
      <field name="TEL_FLAGS"   array="yes" type="integer"   compare="conflict"/> <!-- offset 0 -->
      <field name="TEL_LABEL"   array="yes" type="string"    compare="conflict"/> <!-- offset 1 -->
      <field name="TEL_ID"      array="yes" type="integer"   compare="conflict"/> <!-- offset 2 -->
      <field name="TEL_SLOT"    array="yes" type="integer"   compare="never"/>    <!-- offset 3 -->

      a = TEL[+1][3]; // same as TEL_FLAGS[3]

I tried it briefly, but got a syntax error which implied that the first
square brackets were treated like a normal array index in TEL.

What would it take to make this work?

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
anuj chauhan | 7 May 13:07 2014
Picon

Syncevoltion throwing error on log level < 4

Hi ,


I am facing a issue on reducing log level<4.Syncevolution is

throwing following error:


#[ERROR] basic_string::_S_construct null not valid


 Please suggest how to resolve this problem.




Thanks & Regards

Anuj Chauhan



_______________________________________________
os-libsynthesis mailing list
os-libsynthesis@...
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Rajesh Kumar Pawar | 28 Apr 14:53 2014
Picon

Duplication of data during a merge scenario

Hi,

 

I've been trying to work on a merging  scenario using syncevolution as a client.

 

The steps that I followed in this scenario are as follows

1.       Created a contact with TEL;CELL:12456399

2.       Perform slow sync with server

3.       Modify the field in client vCard as TEL;CELL:12456300

4.       Wait for 5min to ensure sufficient difference between the modification time stamps.

5.       Modified the field as TEL;CELL:12456391 on server.

6.       Perform a two-way sync.

 

vCard before sync:

BEGIN:VCARD

REV:20140425T113334Z

N:last;first;middle;prefix;suffix

TEL;CELL:12456399

VERSION:2.1

UID:1000101

X-CT:PERSON

END:VCARD

 

vCard after sync

BEGIN:VCARD

REV:20140425T165221Z

N:last;first;middle;prefix;suffix

TEL;CELL:12456391

TEL;CELL:12456300

VERSION:2.1

UID:1000101

X-CT:PERSON

END:VCARD

 

What was expected is that the server wins the conflict as the data was modified last at server and result be as follows

 

BEGIN:VCARD

REV:20140425T165221Z

N:last;first;middle;prefix;suffix

TEL;CELL:12456391

VERSION:2.1

UID:1000101

X-CT:PERSON

END:VCARD

 

Instead the data was duplicated in the vCard

 

While for server the data remains as modified i.e. TEL;CELL:12456391

 

 

Fieldlist entries:

      <field name="TEL" array="yes" compare="conflict" type="telephone"/>

      <field name="TEL_FLAGS" array="yes" compare="conflict" type="integer"/>

 

While going through the code in file SyncSource.cpp found the following lines.

 

        "      <!-- conflict strategy: Newer item wins\n"

        "           You can set 'server-wins' or 'client-wins' as well\n"

        "           if you want to give one side precedence\n"

        "      -->\n"

        "      <conflictstrategy>newer-wins</conflictstrategy>\n"

        "\n"

    

This shows that syncevolution should have discarded the local entry and replaced it with the entry from server.

Kindly suggest a way to correct or control this merging.

 

Thank You,

Rajesh

_______________________________________________
os-libsynthesis mailing list
os-libsynthesis@...
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Anuj | 25 Apr 11:12 2014
Picon

How to stop asking for server DevInf

Hi lukas,
While doing sync with server I observed that server is sending a dev inf  and my evolution client is changing
the behaviour according to servers dev inf  for further sync sessions.How do I make   my client stop asking
for servers dev inf?

Regards
Anuj Chauhan
Sachin Gupta | 25 Apr 11:33 2014
Picon

SyncML Server Performance using Syncevolution

Hi Lukas/Patrick,

I have to run some test cases using syncevolution as client to test a SyncML server performance. Expectation is to put load of around 2500 users syncing concurrently for an hour.

Can you suggest how i can test SyncML Server performance and have 2500 users/syncevolutions connecting simultaneously?

Regards
Sachin 
_______________________________________________
os-libsynthesis mailing list
os-libsynthesis@...
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Patrick Ohly | 11 Apr 16:52 2014
Picon

UID + CardDAV

Hello!

In my syncing between local storage and CardDAV I have the following
problem: some CardDAV servers expect the UID to be unchanged (Apple
does, Google does not). EDS as local storage and probably also
traditional SyncML clients do not store the UID that they received from
CardDAV, so when writing back their data as-is I get an error from the
CardDAV server.

I used to have a hack in the SyncEvolution CardDAV backend which put
back the UID, but I noticed that this hack isn't working in all cases
and therefore would like to replace it.

libsynthesis already does a read/update/write cycle
("replace_writes_all_fields") when preparing items for the CardDAV
backend. My thinking therefore was:
     1. Add UID to vCard field list.
     2. Don't generate or parse it by default (in particular not when
        talking to EDS).
     3. Enable it when exchanging data with CardDAV.
     4. During the read/update/write the engine should see an unassigned
        UID in the updated data and a set, non-empty UID in the data
        that it needs to update, and thus preserve the UID.

Unfortunately it doesn't work like that. Full log and field list/profile
attached.

It deviates from my expectation in having a <empty> (and not
<unassigned>) UID after parsing the incoming data (which did not have a
UID property). Why is that?

2014-04-11 14:18:02.574] 'Item_Parse' - parsing SyncML item,
SyncOp=replace, format=plain-text,
LocalID=b36b6c92e727bc498bfacb938403b6cd.vcf [--][++] [->end] [->enclosing]
      * [2014-04-11 14:18:02.574] Created new item of datatype
        'vCard30', localID='b36b6c92e727bc498bfacb938403b6cd.vcf'
        remoteID=''
      * [2014-04-11 14:18:02.574] Parsing: 
      * [2014-04-11 14:18:02.574]
      *  BEGIN:VCARD
VERSION:3.0
PRODID:-//Synthesis AG//NONSGML SyncML Engine V3.4.0.47//EN
REV:20140411T141802Z
N:Doe;John;1;Mr.;Sr.
...
[no UID here!]
...
[2014-04-11 14:18:02.575] Successfully parsed: 
  * [2014-04-11 14:18:02.575] Item
    LocalID='b36b6c92e727bc498bfacb938403b6cd.vcf', RemoteID='',
    operation=replace, size: [maxlocal,maxremote,actual]
  * [2014-04-11 14:18:02.575] 
-  0 :    integer SYNCLVL         [ n/a,   0,     0] : <unassigned>
-  1 :  timestamp REV             [   0,   0,     0] : 2014-04-11T14:18:02Z (TZ: UTC)
-  2 :     string UID             [   0,   0,     0] : <empty>
-  3 :     string N_LAST          [   0,   0,     3] : "Doe"
-  4 :     string N_FIRST         [   0,   0,     4] : "John"

Note the <empty> UID after parsing.

...
[2014-04-11 14:18:02.577] startDataWrite called, status=0
  * [2014-04-11 14:18:02.577] TStdLogicDS::logicProcessRemoteItem
    0x24e6240 starting, SyncOp=replace, RemoteID='',
    LocalID='b36b6c92e727bc498bfacb938403b6cd.vcf'
  * [2014-04-11 14:18:02.577] Testing filter '' against item:
  * [2014-04-11 14:18:02.577] Filter test result is TRUE
  * [2014-04-11 14:18:02.577] TStdLogicDS: Need read-modify-write
    (cause: replace_writes_all_fields ) -> retrieve original item from
    DB
  * ...
  * [2014-04-11 14:18:02.624] Parsing: 
  * [2014-04-11 14:18:02.624] 
BEGIN:VCARD
VERSION:3.0
UID:6f354d698b7ccd22
...
[2014-04-11 14:18:02.624] Successfully parsed: 
  * [2014-04-11 14:18:02.624] Item
    LocalID='b36b6c92e727bc498bfacb938403b6cd.vcf', RemoteID='',
    operation=replace, size: [maxlocal,maxremote,actual]
  * [2014-04-11 14:18:02.624] 
-  0 :    integer SYNCLVL         [   0, n/a,     0] : <unassigned>
-  1 :  timestamp REV             [   0,   0,     0] : 2014-04-11T08:43:42Z (TZ: UTC)
-  2 :     string UID             [   0,   0,    16] : "6f354d698b7ccd22"
-  3 :     string N_LAST          [   0,   0,     3] : "Doe"
-  4 :     string N_FIRST         [   0,   0,     4] : "John"

This is the UID straight from the CardDAV server. And then it gets lost:

[2014-04-11 14:18:02.626] TStdLogicDS: Item updated with contents from remote
  * [2014-04-11 14:18:02.626] Item
    LocalID='b36b6c92e727bc498bfacb938403b6cd.vcf', RemoteID='',
    operation=replace, size: [maxlocal,maxremote,actual]
  * [2014-04-11 14:18:02.626] 
-  0 :    integer SYNCLVL         [   0, n/a,     0] : <unassigned>
-  1 :  timestamp REV             [   0,   0,     0] : 2014-04-11T14:18:02Z (TZ: UTC)
-  2 :     string UID             [   0,   0,     0] : <empty>
-  3 :     string N_LAST          [   0,   0,     3] : "Doe"
-  4 :     string N_FIRST         [   0,   0,     4] : "John"

With Apple, that leads to a somewhat misleading error message:

[<?xml version='1.0' encoding='UTF-8'?> 
<error xmlns='DAV:'> 
<no-uid-conflict xmlns='urn:ietf:params:xml:ns:carddav'>

<href
xmlns='DAV:'>/addressbooks/__uids__/user01/addressbook/b36b6c92e727bc498bfacb938403b6cd.vcf</href> 
</no-uid-conflict> 
<error-description
xmlns='http://twistedmatrix.com/xml_namespace/dav/'>UID already used in
another resource</error-description> 
</error>]
  * [2014-04-11 14:18:02.681] Running post_send hooks
  * [2014-04-11 14:18:02.681] ah_post_send (#0), code is 403 (want 401),
    WWW-Authenticate is (none)
  * [2014-04-11 14:18:02.681] Request ends, status 403 class 4xx, error
    line: 
403 Forbidden
  * [2014-04-11 14:18:02.681] Running destroy hooks.
  * [2014-04-11 14:18:02.681] Request ends.
  * [2014-04-11 14:18:02.681] error code from SyncEvolution access
    denied (remote, status 403): PUT: bad HTTP status: <status 1.1, code
    403, class 4, Forbidden>

Any suggestion?

As a workaround, I am now doing the following in my incoming script:

      if (UID==EMPTY) {
        UID=UNASSIGNED;
      }

But that still doesn't work. Now the update incoming item has UID ==
<unassigned>, but the engine removes the non-empty UID regardless.

What exactly is the purpose of the read/update/write cycle if it
overwrites all fields completely?

Hmm, perhaps the problem is this:
[2014-04-11 14:40:53.456] Field options after CTCap analyzing:
  * [2014-04-11 14:40:53.456] 
- SYNCLVL              : n/a          maxoccur=0, maxsize=0 (unlimited)
- REV                  : AVAILABLE    maxoccur=1, maxsize=0 (unlimited)
- UID                  : AVAILABLE    maxoccur=1, maxsize=0 (unlimited)

Looks like the sending side with EDS as storage did advertise UID
support, although the property wasn't actually enabled.

Yes, CtCap contains it:

<Property><PropName>UID</PropName><MaxOccur>1</MaxOccur></Property>

However, on the EDS side the property is not active (as intended):

Parsing: 
[2014-04-11 14:40:53.520] 
BEGIN:VCARD
VERSION:3.0
UID:pas-id-5347FEF300000000
...
[2014-04-11 14:40:53.520] parseMimeDir: property not parsed (unknown or
not storable): UID:pas-id-5347FEF300000000 
P
  * [2014-04-11 14:40:53.520] Successfully parsed: 
  * [2014-04-11 14:40:53.520] Item LocalID='pas-id-5347FEF300000000',
    RemoteID='29022eb7c6cc694dc3fe901edd794d04.vcf',
    operation=wants-replace, size: [maxlocal,maxremote,actual]
  * [2014-04-11 14:40:53.521] 
-  0 :    integer SYNCLVL         [   0, n/a,     0] : <unassigned>
-  1 :  timestamp REV             [   0,   0,     0] : 2014-04-11T14:40:53Z (TZ: UTC)
-  2 :     string UID             [   0,   0,     0] : <empty>

Could it be a bug that the disable property shows up in the CtCap?

Darn, probably a circular dependency again: we have to be ready to send
CtCap before we know the peer, so the rule mechanism (which depends on
knowing the peer) can't be used to influence the CtCap. Right?

--

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Attachment (syncevolution-log.html.gz): application/x-gzip, 96 KiB
Attachment (00vcard-fieldlist.xml): application/xml, 8 KiB
Attachment (01vcard-profile.xml): application/xml, 19 KiB
_______________________________________________
os-libsynthesis mailing list
os-libsynthesis@...
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis

Gmane