Patrick J. Kelsey | 11 May 2007 04:53

DumpFile.pm patch

Hello,
 
I am having very good success in migrating a 310 MiB VSS database that has a number of 'destroyed' items and incorrect timestamps to svn, but only with the following patch to DumpFile.pm.  The symptom that prompted me to make this change was a "file not found" error from svnadmin load when it encountered an entry in the dumpfile that was attempting to copy to a subdirectory in the orphaned directory whose parent directories had not yet been created.  I haven't fully investigated the exact sequence of events that leads to this condition, since the intuitively simple (perhaps naive) patch below seems to have been quite successful for me.  The gist of the change is this: if the parent path of the target path of a move operation does not exist, then create it (code largely copied from the add handler).
 
I am still working on a thorough review of the results of the conversion, but so far a number of diffs of projects at head and at a few labels are coming up clean.
 
 
Index: script/Vss2Svn/Dumpfile.pm
===================================================================
--- script/Vss2Svn/Dumpfile.pm  (revision 309)
+++ script/Vss2Svn/Dumpfile.pm  (working copy)
<at> <at> -408,6 +408,21 <at> <at>
         return 0;
     }
+### HACK
+    my $success = $self->{repository}->exists_parent ($itempath);
+    if(!defined($success)) {
+        $self->add_error("Path consistency failure while trying to move "
+            . "item '$oldpath' to '$itempath' at revision $data->{revision_id}; skipping");
+        return 0;
+    }
+    elsif ($success == 0) {
+        $self->add_error("Parent path missing while trying to move "
+            . "item '$oldpath' to '$itempath' at revision $data->{revision_id}: adding missing "
+            . "parents");
+        $self->_create_svn_path ($nodes, $itempath);
+    }
+### ENDHACK
+
     my $node = Vss2Svn::Dumpfile::Node->new();
     $node->set_initial_props($itempath, $data);
     $node->{action} = 'add';
 
Regards,
Patrick
_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user
Stephen Lee | 11 May 2007 12:09
Favicon

Re: DumpFile.pm patch

Patrick J. Kelsey wrote:
> I am having very good success in migrating a 310 MiB VSS database that 
> has a number of 'destroyed' items and incorrect timestamps to svn, but 
> only with the following patch to DumpFile.pm.
You seem to have independently come up with the same patch (except for 
the exact error message) as the DumpFile.pm section of my patch from 
February.

http://lists.pumacode.org/pipermail/vss2svn-users-lists.pumacode.org/2007-February/000927.html
contains patch against revision 285

http://lists.pumacode.org/pipermail/vss2svn-users-lists.pumacode.org/2007-February/001022.html
contains patch against revision 300

Although I probably confused it somewhat by talking about several other 
things in the same email!

Perhaps worth someone with access checking in at least the DumpFile.pm 
part of the patch...

--

-- 
Stephen Lee <Stephen.Lee <at> hexagonmetrology.com>
Software Engineer, Vision Group - Pro-Measure Leader
Wilcox Associates Inc. (U.K.)

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Patrick J. Kelsey | 11 May 2007 14:02

RE: DumpFile.pm patch

Ah, I should have searched the list for patches to DumpFile.pm after I concocted this one.  Well you can
certainly have all the glory :)  and consider my report another vote for integration of your patch.

Patrick

________________________________

From: vss2svn-users-bounces <at> lists.pumacode.org on behalf of Stephen Lee
Sent: Fri 5/11/2007 6:09 AM
To: Vss2Svn Users
Subject: Re: DumpFile.pm patch

Patrick J. Kelsey wrote:
> I am having very good success in migrating a 310 MiB VSS database that
> has a number of 'destroyed' items and incorrect timestamps to svn, but
> only with the following patch to DumpFile.pm.
You seem to have independently come up with the same patch (except for
the exact error message) as the DumpFile.pm section of my patch from
February.

http://lists.pumacode.org/pipermail/vss2svn-users-lists.pumacode.org/2007-February/000927.html
contains patch against revision 285

http://lists.pumacode.org/pipermail/vss2svn-users-lists.pumacode.org/2007-February/001022.html
contains patch against revision 300

Although I probably confused it somewhat by talking about several other
things in the same email!

Perhaps worth someone with access checking in at least the DumpFile.pm
part of the patch...

--
Stephen Lee <Stephen.Lee <at> hexagonmetrology.com>
Software Engineer, Vision Group - Pro-Measure Leader
Wilcox Associates Inc. (U.K.)

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Attachment (winmail.dat): application/ms-tnef, 4914 bytes
_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user
Stephen Lee | 11 May 2007 14:18
Favicon

Re: DumpFile.pm patch

Patrick J. Kelsey wrote:
> Ah, I should have searched the list for patches to DumpFile.pm after I
> concocted this one.  Well you can certainly have all the glory :)  and
> consider my report another vote for integration of your patch.

Don't particularly care for glory... but kinda recognised the code and 
realised the similarity... and figured that one of our patches to 
DumpFile.pm might count as a code review of the other! ;)

--

-- 
Stephen Lee <Stephen.Lee <at> hexagonmetrology.com>
Software Engineer, Vision Group - Pro-Measure Leader
Wilcox Associates Inc. (U.K.)

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Toby Johnson | 11 May 2007 19:56
Picon

Patches and tickets (was: DumpFile.pm patch)

Stephen Lee wrote:
> Patrick J. Kelsey wrote:
>> I am having very good success in migrating a 310 MiB VSS database 
>> that has a number of 'destroyed' items and incorrect timestamps to 
>> svn, but only with the following patch to DumpFile.pm.
> You seem to have independently come up with the same patch (except for 
> the exact error message) as the DumpFile.pm section of my patch from 
> February.
>
>
http://lists.pumacode.org/pipermail/vss2svn-users-lists.pumacode.org/2007-February/000927.html 
>
> contains patch against revision 285
>
>
http://lists.pumacode.org/pipermail/vss2svn-users-lists.pumacode.org/2007-February/001022.html 
>
> contains patch against revision 300
>
>
> Although I probably confused it somewhat by talking about several 
> other things in the same email!
>
>
> Perhaps worth someone with access checking in at least the DumpFile.pm 
> part of the patch...
>

Thanks for the patch guys; I guess if two people independently submit 
the same patch then I can assume it works! I've committed this in r310.

Stephen, if you have any other suggested patches which were not 
committed and which don't exist as an open Ticket 
(http://www.pumacode.org/projects/vss2svn/report/1), please feel free to 
refresh my memory (I know you were discussing several issues w/ Dirk at 
the time and I'm not familiar w/ that whole conversation). I can give 
you or others access to create/modify tickets directly if you'd like (I 
had this turned on globally but was getting ticket spam, which Trac 
makes very hard to delete).

Since myself and the handful of committers for this project don't have 
much time to work on this anymore, perhaps we need to come up with a 
better way for submitted patches to be accepted. I obviously don't want 
to just commit every patch that is submitted via email, but I certainly 
don't want to lose track of the good ones either. Thoughts on how to do 
this are welcome.

toby

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Larson, Aaron (SWCOE | 13 May 2007 06:24
Picon
Favicon

RE: Patches and tickets (was: DumpFile.pm patch)

>>>>> "TJ" == Toby  <Johnson> writes:

TJ> Thanks for the patch guys; I guess if two people independently
TJ> submit the same patch then I can assume it works! I've committed
TJ> this in r310.

The description of this patch certainly sounds like the problem I've
been having with my databases as well.  Any chance a nightly build
could be done that includes the patch? 

TJ> Since myself and the handful of committers for this project don't
TJ> have much time to work on this anymore, perhaps we need to come up
TJ> with a better way for submitted patches to be accepted. 

I wish I could help.  For whatever reason, when I run from source, the
results are very different than the nightly build, otherwise I'd
volunteer as a tester.

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Toby Johnson | 15 May 2007 04:02
Picon

Re: Patches and tickets

Larson, Aaron (SWCOE) wrote:
>>>>>> "TJ" == Toby  <Johnson> writes:
>>>>>>             
>
> TJ> Thanks for the patch guys; I guess if two people independently
> TJ> submit the same patch then I can assume it works! I've committed
> TJ> this in r310.
>
> The description of this patch certainly sounds like the problem I've
> been having with my databases as well.  Any chance a nightly build
> could be done that includes the patch? 
>   

A new nightly build has been uploaded with this change.

toby

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Larson, Aaron (SWCOE | 15 May 2007 19:40
Picon
Favicon

RE: Patches and tickets

>>>>> "TJ" == Toby  <Johnson> writes:
AL> Larson, Aaron (SWCOE) wrote:

AL> The description of this patch certainly sounds like the problem
AL> I've been having with my databases as well.  Any chance a nightly
AL> build could be done that includes the patch?

TJ> A new nightly build has been uploaded with this change.

Thanks.  Sadly, it didn't fix my problem. :-(

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Aleksey Nogin | 17 May 2007 23:47

vss2svn --auto-props do not convert "svn:eol-style=native" files correctly?

If I use the vss2svn (latest HEAD version) with the --auto_props option 
and the auto-props file sets svn:eol-style=native one some of the files, 
the files would not be converted correctly. The problem here is that the 
dump file is _required_ to only have the LF line terminators on files 
marked with svn:eol-style=native (see 
http://svn.haxx.se/dev/archive-2006-07/0981.shtml ), but vss2svn does 
not make any attempt to do that...

Should I file an issue report on this?

Aleksey

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

finnied | 18 May 2007 04:44
Favicon

svnadmin: File not found


Dirk & All,

I came up with, and coded a solution for this problem. The patch is supplied below, if anyone is interested.

To summarise, this is to fix the "svnadmin load" problem where it emits the following error:

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv

svnadmin: File not found: transaction '20678-1', path 'orphaned/_YEDAAAAA/S01/InstallShield/WebGate/Setup Files/Uncompressed Files/Language Independent/OS Independent/setup.bmp'

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The main problem stemmed from the following query in the BUILDACTIONHIST phase:

    my $sql = 'SELECT * FROM PhysicalAction ORDER BY timestamp ASC, '
            . 'itemtype ASC, priority ASC, sortkey ASC, action_id ASC';

If 2 or more projects were added at the exact same time, then depending on their physical VSS name, they may be returned by this query in the wrong order (i.e. ADDing a subproject before its parent project has been ADDed).

To fix this, I decided to extend the use of "parentdata" in the PhysicalAction table, and instead treat it like a project "depth" attribute. During the MERGEPARENTDATA phase, all of the parent records are updated with their project path depth as an integer, such that the depth increases the further a project/file gets from the root project.

The query is then changed to:

    my $sql = 'SELECT * FROM PhysicalAction ORDER BY timestamp ASC, '
            . 'itemtype ASC, priority ASC, parentdata ASC, sortkey ASC, action_id ASC';

The solution ensures that subprojects will have a parentdata value greater than that of their parent project, and therefore be returned by this query after their parent project.

This fixed the problem for me. If anyone can think of a better solution, go for it. Delving into this problem has made me truly realise how badly designed VSS is !

A disclaimer - PERL is not my forte, so if there is a cleverer way of doing it, I'd be happy for people to edit my efforts :-)

Dave

Index: script/vss2svn.pl
===================================================================
--- script/vss2svn.pl        (revision 311)
+++ script/vss2svn.pl        (working copy)
<at> <at> -526,13 +526,13 <at> <at>
 
     my($sth, $rows, $row);
     $sth = $gCfg{dbh}->prepare('SELECT * FROM PhysicalAction '
-                               . 'WHERE parentdata = 1');
+                               . 'WHERE parentdata > 0');
     $sth->execute();
 
     # need to pull in all recs at once, since we'll be updating/deleting data
     $rows = $sth->fetchall_arrayref( {} );
 
-    my($childrecs, $child, $id);
+    my($childrecs, $child, $id, $depth);
     my <at> delchild = ();
 
     foreach $row ( <at> $rows) {
<at> <at> -543,6 +543,8 <at> <at>
                           . "'$row->{action_id}'");
         }
 
+        $depth = &GetPathDepth($row);
+
         foreach $child ( <at> $childrecs) {
             &UpdateParentRec($row, $child);
             push( <at> delchild, $child->{action_id});
<at> <at> -558,6 +560,72 <at> <at>
 }  #  End MergeParentData
 
 ###############################################################################
+#  GetPathDepth
+###############################################################################
+sub GetPathDepth {
+    my($row) = <at> _;
+
+    # If we've already worked out the depth of this row, return it immediately
+    if ($row->{parentdata} > 1) {
+        return $row->{parentdata};
+    }
+
+    my($maxParentDepth, $depth, $parents, $parent);
+
+    # Get the row(s) corresponding to the parent(s) of this row, and work out
+    # the maximum depth
+    
+    my $sql = <<"EOSQL";
+SELECT
+    *
+FROM
+    PhysicalAction
+WHERE
+    parentdata > 0
+    AND physname = ?
+    AND actiontype = ?
+EOSQL
+
+    my $sth = $gCfg{dbh}->prepare($sql);
+    $sth->execute( <at> { $row }{qw(parentphys actiontype)} );
+
+    $parents =  $sth->fetchall_arrayref( {} );
+    $maxParentDepth = 0;
+    foreach $parent ( <at> $parents) {
+        $depth = &GetPathDepth($parent);
+        $maxParentDepth = ($depth > $maxParentDepth) ? $depth : $maxParentDepth;
+    }
+
+    # Depth of this path becomes one more than the maximum parent depth
+    $depth = $maxParentDepth + 1;
+
+    # Update the row for this record
+    &UpdateDepth($row, $depth);
+
+    return $depth;
+}  #  End GetPathDepth
+
+###############################################################################
+#  UpdateDepth
+###############################################################################
+sub UpdateDepth {
+    my($row, $depth) = <at> _;
+
+    my $sql = <<"EOSQL";
+UPDATE
+    PhysicalAction
+SET
+    parentdata = ?
+WHERE
+    action_id = ?
+EOSQL
+
+    my $sth = $gCfg{dbh}->prepare($sql);
+    $sth->execute( $depth, $row->{action_id} );
+
+}  #  End UpdateParentRec
+
+###############################################################################
 #  GetChildRecs
 ###############################################################################
 sub GetChildRecs {
<at> <at> -739,7 +807,7 <at> <at>
     my($sth, $row, $action, $handler, $physinfo, $itempaths, $allitempaths);
 
     my $sql = 'SELECT * FROM PhysicalAction ORDER BY timestamp ASC, '
-            . 'itemtype ASC, priority ASC, sortkey ASC, action_id ASC';
+            . 'itemtype ASC, priority ASC, parentdata ASC, sortkey ASC, action_id ASC';
 
     $sth = $gCfg{dbh}->prepare($sql);
     $sth->execute();

This e-mail message and any attachments may contain confidential, proprietary or non-public information. This information is intended solely for the designated recipient(s). If an addressing or transmission error has misdirected this e-mail, please notify the sender immediately and destroy this e-mail. Any review, dissemination, use or reliance upon this information by unintended recipients is prohibited. Any opinions expressed in this e-mail are those of the author personally.


Dirk <vss2svn <at> nogga.de>
Sent by: vss2svn-users-bounces <at> lists.pumacode.org

27-Mar-2007 06:51 PM

Please respond to
Vss2Svn Users <vss2svn-users <at> lists.pumacode.org>

To
Vss2Svn Users <vss2svn-users <at> lists.pumacode.org>
cc
Subject
Re: Fw: ssphys error: failed to read necessary amount of data from        input        file





Dirk schrieb:
>
>> 86197        TUADAAAA        EDADAAAA        1        ADD            
>>    1        0        \N
>> 86198        LUADAAAA        TUADAAAA        1        ADD      
>>  /orphaned/_YEDAAAAA/S01/InstallShield/WebGate/Setup
>> Files/Uncompressed Files/Language Independent/        1        
>> 0        \N

Ok, the problem is, that you managed to add a subproject to a project at
exactly the same timestamp as you added the project to its parent
project. vss2svn is confused about this fact, and as a result it first
adds the subproject, then the project itself. The above two actions must
actually be reversed.

Here are the corresponding lines from your physical history without any
confidential information

> 58593    TUADAAAA    \N    LUADAAAA    ADD    Language
Independent/    1    1139237015    xxxx    0    \N    1    AAAADAUT    
1    \N    \N
> 98983    EDADAAAA    \N    TUADAAAA    ADD    OS Independent/    1    
1139237015    xxxx    0    \N    1    AAAADADE    1    \N    \N


TUADAAAA is added to LUADAAAA at timestamp 1139237015 .
EDADAAAA is added to TUADAAAA also at timestamp 1139237015 .


Gimme some time to think about a solution.


Dirk

_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user


_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/listinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-control.subversion.vss2svn.user

Gmane