Dave Johansen | 26 Jun 01:05 2016
Picon
Gravatar

1.8.6 failing tests on Fedora

I just tried updating to 1.8.6 on Fedora and there are 4 test failures. Here's the output:
======================================================================
FAIL: test_fetch_movetotrunk (test_fetch_command.TestStupidPull)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/durin42-hgsubversion-49d324e11856/tests/test_fetch_command.py", line 256, in test_fetch_movetotrunk
    self.assertMultiLineEqual(refgraph, graph)
  File "/builddir/build/BUILD/durin42-hgsubversion-49d324e11856/tests/test_util.py", line 734, in assertMultiLineEqual
    msg)
AssertionError: 'o  changeset: 0:02996a5980ba (r3)\n   branch:\n   tags:      tip\n   summary:   [truncated]... != 'o  changeset: 0:02996a5980ba (r)\n   branch:\n   tags:      tip\n   summary:    [truncated]...
- o  changeset: 0:02996a5980ba (r3)
?                                -
+ o  changeset: 0:02996a5980ba (r)
     branch:
     tags:      tip
     summary:   move to trunk
     files:     a dir/b
 


======================================================================
FAIL: test_fetch_revert (test_fetch_command.TestStupidPull)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/durin42-hgsubversion-49d324e11856/tests/test_fetch_command.py", line 242, in test_fetch_revert
    self.assertMultiLineEqual(refgraph, graph)
  File "/builddir/build/BUILD/durin42-hgsubversion-49d324e11856/tests/test_util.py", line 734, in assertMultiLineEqual
    msg)
AssertionError: 'o  changeset: 3:937dcd1206d4 (r4)\n|  branch:\n|  tags:      tip\n|  summary:   [truncated]... != 'o  changeset: 3:937dcd1206d4 (r)\n|  branch:\n|  tags:      tip\n|  summary:    [truncated]...
- o  changeset: 3:937dcd1206d4 (r4)
?                                -
+ o  changeset: 3:937dcd1206d4 (r)
  |  branch:
  |  tags:      tip
  |  summary:   revert2
  |  files:     a dir/b
  |
- o  changeset: 2:9317a748b7c3 (r3)
?                                -
+ o  changeset: 2:9317a748b7c3 (r)
  |  branch:
  |  tags:
  |  summary:   revert
  |  files:     a dir/b
  |
- o  changeset: 1:243259a4138a (r2)
?                                -
+ o  changeset: 1:243259a4138a (r)
  |  branch:
  |  tags:
  |  summary:   changefiles
  |  files:     a dir/b
  |
- o  changeset: 0:ab86791fc857 (r1)
?                                -
+ o  changeset: 0:ab86791fc857 (r)
     branch:
     tags:
     summary:   init
     files:     a dir/b
 


======================================================================
FAIL: test_svn_keywords (test_template_keywords.TestLogKeywords)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/durin42-hgsubversion-49d324e11856/tests/test_template_keywords.py", line 53, in test_svn_keywords
    '''.strip())
AssertionError: 'rev: 2 svnrev: svnpath: svnuuid:\n <at> \n|\n  rev: 1 svnrev: svnpath: svnuuid:\no\n|\n  rev: 0 svnrev: svnpath: svnuuid:\no' != 'rev: 2 svnrev: svnpath: svnuuid:\n <at> \n|\n  rev: 1 svnrev:3 svnpath:/trunk svnuuid:df2126f7-00ab-4d49-b42c-7e981dde0bcf\no\n|\n  rev: 0 svnrev:2 svnpath:/trunk svnuuid:df2126f7-00ab-4d49-b42c-7e981dde0bcf\no'
-------------------- >> begin captured stdout << ---------------------
  rev: 2 svnrev: svnpath: svnuuid:
<at>
|
  rev: 1 svnrev: svnpath: svnuuid:
o
|
  rev: 0 svnrev: svnpath: svnuuid:
o



--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: test_svn_revsets (test_template_keywords.TestLogKeywords)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/durin42-hgsubversion-49d324e11856/tests/test_template_keywords.py", line 67, in test_svn_revsets
    self.assertEqual(ui._output, '0:2 1:3 ')
AssertionError: '0: 1: ' != '0:2 1:3 '

--
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to hgsubversion-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.
Jun Wu | 23 Jun 21:14 2016

[PATCH 1 of 2] maps: do not open database when initializing sqlite revmap

# HG changeset patch
# User Jun Wu <quark@...>
# Date 1466704673 -3600
#      Thu Jun 23 18:57:53 2016 +0100
# Node ID 8e89b4ee52680563306e649765e48a16dc579cb7
# Parent  3ba9ca659fd0395c5b8faacb0f2515ed1033f33f
# Available At https://bitbucket.org/quark-zju/hgsubversion-revmap
#              hg pull https://bitbucket.org/quark-zju/hgsubversion-revmap -r 8e89b4ee5268
maps: do not open database when initializing sqlite revmap

Since every db operation goes through _transaction, which will open the
database on demand, it is unnecessary to open the database during __init__.

diff --git a/hgsubversion/maps.py b/hgsubversion/maps.py
--- a/hgsubversion/maps.py
+++ b/hgsubversion/maps.py
 <at>  <at>  -566,7 +566,6  <at>  <at>  class SqliteRevMap(collections.MutableMa

         self._db = None
         self._hashes = None
-        self._opendb()
         self.firstpulled = 0
         self._updatefirstlastpulled()
         # __iter__ is expensive and thus disabled by default

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Jun Wu | 23 Jun 12:12 2016

[PATCH 1 of 2] maps: do not ask sqlite for row count

# HG changeset patch
# User Jun Wu <quark@...>
# Date 1466675167 -3600
#      Thu Jun 23 10:46:07 2016 +0100
# Node ID 96540f1bad17d5c98bab2a6b14bb8c6282419739
# Parent  f21605bcda2489483ffdbdb32cbcbdd83ca88391
# Available At https://bitbucket.org/quark-zju/hgsubversion-revmap
#              hg pull https://bitbucket.org/quark-zju/hgsubversion-revmap -r 96540f1bad17
maps: do not ask sqlite for row count

"SELECT COUNT(1) FROM x" is not O(1) for sqlite and can be slow on large
tables. This patch changes the count to be backed by a file instead.

The change exposes a risk that the number may become inaccurate, if
__setitem__ is called with a same key multiple times. But we don't do that
during pull, and only use __len__ to calculate how many revisions pulled,
or test if the map is empty. So it would be fine.

diff --git a/hgsubversion/maps.py b/hgsubversion/maps.py
--- a/hgsubversion/maps.py
+++ b/hgsubversion/maps.py
 <at>  <at>  -426,6 +426,7  <at>  <at>  class RevMap(dict):
             revmap.exportrevmapv1(tmppath)
             os.rename(tmppath, self._filepath)
             hgutil.unlinkpath(revmap._dbpath)
+            hgutil.unlinkpath(revmap._rowcountpath, ignoremissing=True)
             return self._readmapfile()
         if ver != self.VERSION:
             raise hgutil.Abort('revmap too new -- please upgrade')
 <at>  <at>  -554,10 +555,13  <at>  <at>  class SqliteRevMap(collections.MutableMa

     lastpulled = util.fileproperty('_lastpulled', lambda x: x._lastpulledpath,
                                    default=0, deserializer=int)
+    rowcount = util.fileproperty('_rowcount', lambda x: x._rowcountpath,
+                                 default=0, deserializer=int)

     def __init__(self, revmap_path, lastpulled_path):
         self._filepath = revmap_path
         self._dbpath = revmap_path + '.db'
+        self._rowcountpath = self._dbpath + '.rowcount'
         self._lastpulledpath = lastpulled_path

         self._db = None
 <at>  <at>  -608,6 +612,7  <at>  <at>  class SqliteRevMap(collections.MutableMa
     def clear(self):
         hgutil.unlinkpath(self._filepath, ignoremissing=True)
         hgutil.unlinkpath(self._dbpath, ignoremissing=True)
+        hgutil.unlinkpath(self._rowcountpath, ignoremissing=True)
         self._db = None
         self._hashes = None
         self._firstpull = None
 <at>  <at>  -635,10 +640,8  <at>  <at>  class SqliteRevMap(collections.MutableMa
         return iter(rows)

     def __len__(self):
-        # 'WHERE rev >= 0' hints sqlite to use the rev index
-        with self._transaction() as db:
-            return db.execute('SELECT COUNT(1) FROM revmap ' +
-                              'WHERE rev >= 0').fetchone()[0]
+        # rowcount is faster than "SELECT COUNT(1)". the latter is not O(1)
+        return self.rowcount

     def __setitem__(self, key, binha):
         revnum, branch = key
 <at>  <at>  -653,6 +656,8  <at>  <at>  class SqliteRevMap(collections.MutableMa

     def __delitem__(self, key):
         for row in self._querybykey('DELETE', key):
+            if self.rowcount > 0:
+                self.rowcount -= 1
             return
         # For performance reason, self._hashes is not updated
         raise KeyError(key)
 <at>  <at>  -687,6 +692,10  <at>  <at>  class SqliteRevMap(collections.MutableMa
         self._db.executemany(
             'INSERT OR REPLACE INTO revmap (rev, branch, hash) ' +
             'VALUES (?, ?, ?)', rows)
+        # If REPLACE happens, rowcount can be wrong. But it is only used to
+        # calculate how many revisions pulled, and during pull we don't
+        # replace rows. So it is fine.
+        self.rowcount += len(rows)

     def _opendb(self):
         '''Open the database and make sure the table is created on demand.'''
 <at>  <at>  -723,7 +732,12  <at>  <at>  class SqliteRevMap(collections.MutableMa
         with self._transaction('EXCLUSIVE'):
             map(self._db.execute, self.TABLESCHEMA)
             if version == RevMap.VERSION:
+                self.rowcount = 0
                 self._importrevmapv1()
+            elif not self.rowcount:
+                self.rowcount = self._db.execute(
+                    'SELECT COUNT(1) FROM revmap').fetchone()[0]
+
             # "bulk insert; then create index" is about 2.4x as fast as
             # "create index; then bulk insert" on a large repo
             map(self._db.execute, self.INDEXSCHEMA)

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Kostia Balytskyi | 22 Jun 18:43 2016

[PATCH] svnmeta: avoid recreating svnmeta object for every revision

# HG changeset patch
# User Kostia Balytskyi <ikostia@...>
# Date 1466613817 25200
#      Wed Jun 22 09:43:37 2016 -0700
# Node ID 8544755f2ce2467346eb0a179bde9b19f84a1310
# Parent  f21605bcda2489483ffdbdb32cbcbdd83ca88391
svnmeta: avoid recreating svnmeta object for every revision

Before this change svnmeta object had a reference to a repo, but not
vice-versa, so a new meta was constructed on every revision lookup,
which made commands like 'hg log -r "r1 + r2 + ... + r100"' read
./hg/svn/rev_map many times.

This change introduces a reference cycle between a repo object and a
svnmeta object. In my opinion this is ok because repo is a long-living
object anyway so we should be fine if it gets collected by mark-and-sweep
rather than reference counting. However,  <at> quark told me that hgsubversion
tries to avoid that, so if reviewers are not happy with this, I can
introduce a config option that will enable cycle creation and stay
disabled by default.

diff --git a/hgsubversion/util.py b/hgsubversion/util.py
--- a/hgsubversion/util.py
+++ b/hgsubversion/util.py
 <at>  <at>  -332,6 +332,14  <at>  <at>  def getsvnrev(ctx, defval=None):
     '''Extract SVN revision from commit metadata'''
     return ctx.extra().get('convert_revision', defval)

+def _getsvnmetaobject(repo):
+    try:
+        meta = repo._svnmeta
+    except AttributeError:
+        meta = repo.svnmeta(skiperrorcheck=True)
+    repo._svnmeta = meta
+    return meta
+
 def revset_fromsvn(repo, subset, x):
     '''``fromsvn()``
     Select changesets that originate from Subversion.
 <at>  <at>  -340,7 +348,7  <at>  <at>  def revset_fromsvn(repo, subset, x):

     rev = repo.changelog.rev
     bin = node.bin
-    meta = repo.svnmeta(skiperrorcheck=True)
+    meta = _getsvnmetaobject(repo)
     if not meta.revmapexists:
         raise hgutil.Abort("svn metadata is missing - "
                            "run 'hg svn rebuildmeta' to reconstruct it")
 <at>  <at>  -352,7 +360,6  <at>  <at>  def revset_svnrev(repo, subset, x):
     Select changesets that originate in the given Subversion revision.
     '''
     args = revset.getargs(x, 1, 1, "svnrev takes one argument")
-
     rev = revset.getstring(args[0],
                            "the argument to svnrev() must be a number")
     try:
 <at>  <at>  -360,7 +367,7  <at>  <at>  def revset_svnrev(repo, subset, x):
     except ValueError:
         raise error.ParseError("the argument to svnrev() must be a number")

-    meta = repo.svnmeta(skiperrorcheck=True)
+    meta = _getsvnmetaobject(repo)
     if not meta.revmapexists:
         raise hgutil.Abort("svn metadata is missing - "
                            "run 'hg svn rebuildmeta' to reconstruct it")

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Kostia Balytskyi | 22 Jun 18:52 2016

[PATCH] svnmeta: avoid recreating svnmeta object for every revision

# HG changeset patch
# User Kostia Balytskyi <ikostia@...>
# Date 1466613817 25200
#      Wed Jun 22 09:43:37 2016 -0700
# Node ID 8544755f2ce2467346eb0a179bde9b19f84a1310
# Parent  f21605bcda2489483ffdbdb32cbcbdd83ca88391
svnmeta: avoid recreating svnmeta object for every revision

Before this change svnmeta object had a reference to a repo, but not
vice-versa, so a new meta was constructed on every revision lookup,
which made commands like 'hg log -r "r1 + r2 + ... + r100"' read
./hg/svn/rev_map many times.

This change introduces a reference cycle between a repo object and a
svnmeta object. In my opinion this is ok because repo is a long-living
object anyway so we should be fine if it gets collected by mark-and-sweep
rather than reference counting. However,  <at> quark told me that hgsubversion
tries to avoid that, so if reviewers are not happy with this, I can
introduce a config option that will enable cycle creation and stay
disabled by default.

diff --git a/hgsubversion/util.py b/hgsubversion/util.py
--- a/hgsubversion/util.py
+++ b/hgsubversion/util.py
 <at>  <at>  -332,6 +332,14  <at>  <at>  def getsvnrev(ctx, defval=None):
     '''Extract SVN revision from commit metadata'''
     return ctx.extra().get('convert_revision', defval)

+def _getsvnmetaobject(repo):
+    try:
+        meta = repo._svnmeta
+    except AttributeError:
+        meta = repo.svnmeta(skiperrorcheck=True)
+    repo._svnmeta = meta
+    return meta
+
 def revset_fromsvn(repo, subset, x):
     '''``fromsvn()``
     Select changesets that originate from Subversion.
 <at>  <at>  -340,7 +348,7  <at>  <at>  def revset_fromsvn(repo, subset, x):

     rev = repo.changelog.rev
     bin = node.bin
-    meta = repo.svnmeta(skiperrorcheck=True)
+    meta = _getsvnmetaobject(repo)
     if not meta.revmapexists:
         raise hgutil.Abort("svn metadata is missing - "
                            "run 'hg svn rebuildmeta' to reconstruct it")
 <at>  <at>  -352,7 +360,6  <at>  <at>  def revset_svnrev(repo, subset, x):
     Select changesets that originate in the given Subversion revision.
     '''
     args = revset.getargs(x, 1, 1, "svnrev takes one argument")
-
     rev = revset.getstring(args[0],
                            "the argument to svnrev() must be a number")
     try:
 <at>  <at>  -360,7 +367,7  <at>  <at>  def revset_svnrev(repo, subset, x):
     except ValueError:
         raise error.ParseError("the argument to svnrev() must be a number")

-    meta = repo.svnmeta(skiperrorcheck=True)
+    meta = _getsvnmetaobject(repo)
     if not meta.revmapexists:
         raise hgutil.Abort("svn metadata is missing - "
                            "run 'hg svn rebuildmeta' to reconstruct it")

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Jun Wu | 16 Jun 07:24 2016

[PATCH] tests: drop unittest2

# HG changeset patch
# User Jun Wu <quark@...>
# Date 1466049291 -3600
#      Thu Jun 16 04:54:51 2016 +0100
# Node ID a8361216162875434379554e5ec297095e3dd53b
# Parent  019c3e194fba3d920a0c56f801c406bae8511481
# Available At https://bitbucket.org/quark-zju/hgsubversion-revmap
#              hg pull https://bitbucket.org/quark-zju/hgsubversion-revmap -r a83612161628
tests: drop unittest2

Mixed using unittest2.SkipTest and unittest.TestCase will cause skips result
in errors. We are probably not going to rewriting every "unittest.TestCase"
to "unittest2.TestCase", then unittest2 is causing more trouble with little
benefit. Let's drop it.

To remain support for Py 26 in run.py, a simple loader.discover is added.

diff --git a/tests/run.py b/tests/run.py
--- a/tests/run.py
+++ b/tests/run.py
 <at>  <at>  -3,11 +3,7  <at>  <at> 
 import optparse
 import os
 import sys
-
-if sys.version_info[:2] < (2, 7):
-    import unittest2 as unittest
-else:
-    import unittest
+import unittest

 if __name__ == '__main__':
     description = ("This script runs the hgsubversion tests. If no tests are "
 <at>  <at>  -63,8 +59,19  <at>  <at> 
     loader = unittest.TestLoader()
     suite = unittest.TestSuite()

+    if sys.version_info[:2] < (2, 7):
+        import glob
+        def discover(start_dir, pattern='test*.py', top_level_dir=None):
+            tests = []
+            sys.path.append(start_dir)
+            for path in glob.glob(os.path.join(start_dir, pattern)):
+                name = os.path.splitext(os.path.basename(path))[0]
+                tests.append(loader.loadTestsFromModule(__import__(name)))
+            return tests
+        loader.discover = discover
+
     if not args:
-        suite = loader.discover('.')
+        suite.addTests(loader.discover('.'))

         if options.comprehensive:
             suite.addTests(loader.discover('comprehensive',
diff --git a/tests/test_util.py b/tests/test_util.py
--- a/tests/test_util.py
+++ b/tests/test_util.py
 <at>  <at>  -39,13 +39,10  <at>  <at> 
 try:
     SkipTest = unittest.SkipTest
 except AttributeError:
-    try:
-        from unittest2 import SkipTest
-    except ImportError:
-        try:
-            from nose import SkipTest
-        except ImportError:
-            SkipTest = None
+    if 'nose' in sys.modules:
+        SkipTest = sys.modules['nose'].SkipTest
+    else:
+        SkipTest = None

 from hgsubversion import svnwrap
 from hgsubversion import util

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Jun Wu | 15 Jun 21:04 2016

[PATCH 1 of 9 V4] maps: implement sqlite revmap

# HG changeset patch
# User Jun Wu <quark@...>
# Date 1466011387 -3600
#      Wed Jun 15 18:23:07 2016 +0100
# Node ID 51f0e236c78e6755da391bd130f319587b619195
# Parent  019c3e194fba3d920a0c56f801c406bae8511481
# Available At https://bitbucket.org/quark-zju/hgsubversion-revmap
#              hg pull https://bitbucket.org/quark-zju/hgsubversion-revmap -r 51f0e236c78e
maps: implement sqlite revmap

This patch adds the SqliteRevMap, which has a same interface with RevMap
but is backed by a sqlite database.

It uses database indexes to accelerate all kinds of queries and disables
iteration to prevent slow code being written in the future.

In practise, it should be faster on large repos with millions of svn
revisions but slower on small repos due to the overhead introduced.

diff --git a/hgsubversion/maps.py b/hgsubversion/maps.py
--- a/hgsubversion/maps.py
+++ b/hgsubversion/maps.py
 <at>  <at>  -1,8 +1,13  <at>  <at> 
 ''' Module for self-contained maps. '''

+import collections
+import contextlib
 import errno
 import os
 import re
+import sqlite3
+import sys
+import weakref
 from mercurial import error
 from mercurial import util as hgutil
 from mercurial.node import bin, hex, nullid
 <at>  <at>  -455,6 +460,258  <at>  <at> 
             self._hashes[ha] = (revnum, branch)

 
+class SqliteRevMap(collections.MutableMapping):
+    """RevMap backed by sqlite3.
+
+    It tries to address performance issues for a very large rev map.
+    As such iteration is unavailable for both the map itself and the
+    reverse map (self.hashes).
+
+    It migrates from the old RevMap upon first use. Then it will bump the
+    version of revmap so RevMap no longer works. The real database is a
+    separated file which has a ".db" suffix.
+    """
+
+    VERSION = 2
+
+    TABLESCHEMA = [
+        '''CREATE TABLE IF NOT EXISTS revmap (
+               rev INTEGER NOT NULL,
+               branch TEXT NOT NULL DEFAULT '',
+               hash BLOB NOT NULL)''',
+    ]
+
+    INDEXSCHEMA = [
+        'CREATE UNIQUE INDEX IF NOT EXISTS revbranch ON revmap (rev,branch);',
+        'CREATE INDEX IF NOT EXISTS hash ON revmap (hash);',
+    ]
+
+    # "bytes" in Python 2 will get truncated at '\0' when storing as sqlite
+    # blobs. "buffer" does not have this issue. Python 3 does not have "buffer"
+    # but "bytes" won't get truncated.
+    sqlblobtype = bytes if sys.version_info >= (3, 0) else buffer
+
+    class ReverseRevMap(object):
+        # collections.Mapping is not suitable since we don't want 2/3 of
+        # its required interfaces: __iter__, __len__.
+        def __init__(self, revmap):
+            self.revmap = weakref.proxy(revmap)
+            self._cache = {}
+
+        def get(self, key, default=None):
+            if key not in self._cache:
+                result = None
+                for row in self.revmap._query(
+                    'SELECT rev, branch FROM revmap WHERE hash=?',
+                    (SqliteRevMap.sqlblobtype(key),)):
+                    result = (row[0], row[1] or None)
+                    break
+                self._cache[key] = result
+            return self._cache[key] or default
+
+        def __contains__(self, key):
+            return self.get(key) != None
+
+        def __getitem__(self, key):
+            dummy = self._cache
+            item = self.get(key, dummy)
+            if item == dummy:
+                raise KeyError(key)
+            else:
+                return item
+
+        def keys(self):
+            for row in self.revmap._query('SELECT hash FROM revmap'):
+                yield bytes(row[0])
+
+    lastpulled = util.fileproperty('_lastpulled', lambda x: x._lastpulledpath,
+                                   default=0, deserializer=int)
+
+    def __init__(self, revmap_path, lastpulled_path):
+        self._filepath = revmap_path
+        self._dbpath = revmap_path + '.db'
+        self._lastpulledpath = lastpulled_path
+
+        self._db = None
+        self._hashes = None
+        self._opendb()
+        # update firstpulled, lastpulled
+        self.firstpulled = 0
+        for row in self._query('SELECT MIN(rev), MAX(rev) FROM revmap'):
+            if row != (None, None):
+                self.firstpulled, lastpulled = row
+                if lastpulled > self.lastpulled:
+                    self.lastpulled = lastpulled
+        # __iter__ is expensive and thus disabled by default
+        # it should only be enabled for testing
+        self._allowiter = False
+
+    def hashes(self):
+        if self._hashes is None:
+            self._hashes = self.ReverseRevMap(self)
+        return self._hashes
+
+    def branchedits(self, branch, rev):
+        return [((r[0], r[1] or None), bytes(r[2])) for r in
+                self._query('SELECT rev, branch, hash FROM revmap ' +
+                                'WHERE rev < ? AND branch = ? ' +
+                                'ORDER BY rev DESC, branch DESC',
+                                (rev.revnum, branch or ''))]
+
+    def branchmaxrevnum(self, branch, maxrev):
+        for row in self._query('SELECT rev FROM revmap ' +
+                               'WHERE rev <= ? AND branch = ? ' +
+                               'ORDER By rev DESC LIMIT 1',
+                               (maxrev, branch or '')):
+            return row[0]
+        return 0
+
+     <at> property
+    def lasthash(self):
+        for row in self._query('SELECT hash FROM revmap ORDER BY rev DESC'):
+            return bytes(row[0])
+        return None
+
+    def revhashes(self, revnum):
+        for row in self._query('SELECT hash FROM revmap WHERE rev = ?',
+                               (revnum,)):
+            yield bytes(row[0])
+
+    def clear(self):
+        hgutil.unlinkpath(self._filepath, ignoremissing=True)
+        hgutil.unlinkpath(self._dbpath, ignoremissing=True)
+        self._db = None
+        self._hashes = None
+        self._firstpull = None
+        self._lastpull = None
+
+    def batchset(self, items, lastpulled):
+        with self._transaction():
+            self._insert(items)
+        self.lastpulled = lastpulled
+
+    def __getitem__(self, key):
+        for row in self._querybykey('SELECT hash', key):
+            return bytes(row[0])
+        raise KeyError(key)
+
+    def __iter__(self):
+        if not self._allowiter:
+            raise NotImplementedError(
+                'SqliteRevMap.__iter__ is not implemented intentionally ' +
+                'to avoid performance issues')
+        # collect result to avoid nested transaction issues
+        rows = []
+        for row in self._query('SELECT rev, branch FROM revmap'):
+            rows.append((row[0], row[1] or None))
+        return iter(rows)
+
+    def __len__(self):
+        # 'WHERE rev >= 0' hints sqlite to use the rev index
+        with self._transaction() as db:
+            return db.execute('SELECT COUNT(1) FROM revmap ' +
+                              'WHERE rev >= 0').fetchone()[0]
+
+    def __setitem__(self, key, binha):
+        revnum, branch = key
+        with self._transaction():
+            self._insert([(revnum, branch, binha)])
+        if revnum < self.firstpulled or not self.firstpulled:
+            self.firstpulled = revnum
+        if revnum > self.lastpulled or not self.lastpulled:
+            self.lastpulled = revnum
+        if self._hashes is not None:
+            self._hashes._cache[binha] = key
+
+    def __delitem__(self, key):
+        for row in self._querybykey('DELETE', key):
+            return
+        # For performance reason, self._hashes is not updated
+        raise KeyError(key)
+
+     <at> contextlib.contextmanager
+    def _transaction(self, mode='IMMEDIATE'):
+        if self._db is None:
+            self._opendb()
+        with self._db as db:
+            db.execute('BEGIN %s' % mode)
+            yield db
+
+    def _query(self, sql, params=()):
+        with self._transaction() as db:
+            cur = db.execute(sql, params)
+            try:
+                for row in cur:
+                    yield row
+            finally:
+                cur.close()
+
+    def _querybykey(self, prefix, key):
+        revnum, branch = key
+        return self._query(
+            '%s FROM revmap WHERE rev=? AND branch=?'
+            % prefix, (revnum, branch or ''))
+
+    def _insert(self, rows):
+        # convert to a safe type so '\0' does not truncate the blob
+        if rows and type(rows[0][-1]) is not self.sqlblobtype:
+            rows = [(r, b, self.sqlblobtype(h)) for r, b, h in rows]
+        self._db.executemany(
+            'INSERT OR REPLACE INTO revmap (rev, branch, hash) ' +
+            'VALUES (?, ?, ?)', rows)
+
+    def _opendb(self):
+        '''Open the database and make sure the table is created on demand.'''
+        version = None
+        try:
+            version = int(open(self._filepath).read(2))
+        except (ValueError, IOError):
+            pass
+        if version and version not in [RevMap.VERSION, self.VERSION]:
+            raise error.Abort('revmap too new -- please upgrade')
+
+        if self._db:
+            self._db.close()
+
+        # if version mismatch, the database is considered invalid
+        if version != self.VERSION:
+            hgutil.unlinkpath(self._dbpath, ignoremissing=True)
+
+        self._db = sqlite3.connect(self._dbpath)
+        self._db.text_factory = bytes
+
+        # disable auto-commit. everything is inside a transaction
+        self._db.isolation_level = 'DEFERRED'
+
+        with self._transaction('EXCLUSIVE'):
+            map(self._db.execute, self.TABLESCHEMA)
+            if version == RevMap.VERSION:
+                self._importrevmapv1()
+            # "bulk insert; then create index" is about 2.4x as fast as
+            # "create index; then bulk insert" on a large repo
+            map(self._db.execute, self.INDEXSCHEMA)
+
+        # write a dummy rev map file with just the revision number
+        if version != self.VERSION:
+            f = open(self._filepath, 'w')
+            f.write('%s\n' % self.VERSION)
+            f.close()
+
+     <at> util.gcdisable
+    def _importrevmapv1(self):
+        with open(self._filepath, 'r') as f:
+            # 1st line is version
+            assert(int(f.readline())) == RevMap.VERSION
+            data = {}
+            for line in f:
+                revnum, ha, branch = line[:-1].split(' ', 2)
+                # ignore malicious lines
+                if len(ha) != 40:
+                    continue
+                data[revnum, branch or None] = bin(ha)
+            self._insert([(r, b, h) for (r, b), h in data.iteritems()])
+
+
 class FileMap(object):

     VERSION = 1

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Picon
Gravatar

[PATCH 1 of 9] svnwrap: allow committing directory copies

# HG changeset patch
# User Dan Villiom Podlaski Christiansen <danchr@...>
# Date 1376056872 -7200
#      Fri Aug 09 16:01:12 2013 +0200
# Node ID 24b0c34bdeb7128f4c50a3b72f77ef6e98c43e26
# Parent  019c3e194fba3d920a0c56f801c406bae8511481
svnwrap: allow committing directory copies

We've never needed to record a directory copy, since Mercurial doesn't
track directories, and thus can't represent copying them. However, in
order to push a tag, we need to store the source as the copy source.

diff --git a/hgsubversion/svnwrap/subvertpy_wrapper.py b/hgsubversion/svnwrap/subvertpy_wrapper.py
--- a/hgsubversion/svnwrap/subvertpy_wrapper.py
+++ b/hgsubversion/svnwrap/subvertpy_wrapper.py
 <at>  <at>  -465,7 +465,11  <at>  <at>  class SubversionRepo(object):
                 else:
                     # visiting a directory
                     if path in addeddirs:
-                        direditor = editor.add_directory(path)
+                        frompath, fromrev = copies.get(path, (None, -1))
+                        if frompath:
+                            frompath = self.path2url(frompath)
+                        direditor = editor.add_directory(path, frompath, fromrev)
+
                     elif path in deleteddirs:
                         direditor = editor.delete_entry(path, base_revision)
                         continue
diff --git a/hgsubversion/svnwrap/svn_swig_wrapper.py b/hgsubversion/svnwrap/svn_swig_wrapper.py
--- a/hgsubversion/svnwrap/svn_swig_wrapper.py
+++ b/hgsubversion/svnwrap/svn_swig_wrapper.py
 <at>  <at>  -420,7 +420,10  <at>  <at>  class SubversionRepo(object):
                 return bat
             if path not in file_data:
                 if path in addeddirs:
-                    bat = editor.add_directory(path, parent, None, -1, pool)
+                    frompath, fromrev = copies.get(path, (None, -1))
+                    if frompath:
+                        frompath = self.path2url(frompath)
+                    bat = editor.add_directory(path, parent, frompath, fromrev, pool)
                 else:
                     bat = editor.open_directory(path, parent, base_revision, pool)
                 batons.append(bat)

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Jun Wu | 13 Jun 18:04 2016

[PATCH 1 of 2] util: add a fileproperty helper method

# HG changeset patch
# User Jun Wu <quark@...>
# Date 1465833591 -3600
#      Mon Jun 13 16:59:51 2016 +0100
# Node ID 4a578aa1f82f153edd25747bd0953d50568c55d2
# Parent  945700dac23713ee2e2971781bebdd70eb158d5a
# Available At https://bitbucket.org/quark-zju/hgsubversion-revmap
#              hg pull https://bitbucket.org/quark-zju/hgsubversion-revmap -r 4a578aa1f82f
util: add a fileproperty helper method

The code base uses the pattern that syncs a property with disk frequently.
Let's have a helper method for declaring this kind of property.

diff --git a/hgsubversion/util.py b/hgsubversion/util.py
--- a/hgsubversion/util.py
+++ b/hgsubversion/util.py
 <at>  <at>  -44,6 +44,27  <at>  <at> 
     path = ui.config('hgsubversion', name)
     return path and hgutil.expandpath(path)

+def fileproperty(fname, pathfunc, default=None,
+                 serializer=str, deserializer=str):
+    """define a property that is backed by a file"""
+    def fget(self):
+        if not hgutil.safehasattr(self, fname):
+            path = pathfunc(self)
+            if os.path.exists(path):
+                with open(path, 'r') as f:
+                    setattr(self, fname, deserializer(f.read()))
+            else:
+                setattr(self, fname, default)
+        return getattr(self, fname)
+
+    def fset(self, value):
+        setattr(self, fname, value)
+        path = pathfunc(self)
+        with open(path, 'w') as f:
+            f.write(serializer(value))
+
+    return property(fget, fset)
+
 def filterdiff(diff, oldrev, newrev):
     diff = newfile_devnull_re.sub(r'--- \1\t(revision 0)' '\n'
                                   r'+++ \1\t(working copy)',

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Picon
Gravatar

[PATCH] tests: drop hard-coded list of tests

# HG changeset patch
# User Dan Villiom Podlaski Christiansen <danchr@...>
# Date 1465736157 -7200
#      Sun Jun 12 14:55:57 2016 +0200
# Node ID 992b112dd2af5e8bcb09da9ce4e47bdc68257519
# Parent  945700dac23713ee2e2971781bebdd70eb158d5a
tests: drop hard-coded list of tests

The list of out of date, missing 'test_helpers' and
'comprehensive/test_custom_layout'. Instead, use the discover
functionality introduced in Python 2.7, and available for Python 2.6
and earlier from the 'unittest2' backport.

Tested by invoking 'run.py' both with and without '-A' in Python 2.6 &
2.7, and ensuring that passing comprehensive tests as arguments
continues to work.

As a minor (but welcome) side-effect, this should restore the ability to
test hgsubversion under demandimport; previously, test_util was
imported before we enabled demandimport, so it didn't affect most of
Mercurial. Since unittest2 (and unittest) do define SkipTest, we can
remove the earlier import, restoring the likely originally intended
testing mode.

diff --git a/tests/run.py b/tests/run.py
--- a/tests/run.py
+++ b/tests/run.py
 <at>  <at>  -3,54 +3,11  <at>  <at> 
 import optparse
 import os
 import sys
-import unittest
-
-import test_util
-test_util.SkipTest = None

-def tests():
-    import test_binaryfiles
-    import test_diff
-    import test_externals
-    import test_fetch_branches
-    import test_fetch_command
-    import test_fetch_command_regexes
-    import test_fetch_exec
-    import test_fetch_mappings
-    import test_fetch_renames
-    import test_fetch_symlinks
-    import test_fetch_truncated
-    import test_hooks
-    import test_svn_pre_commit_hooks
-    import test_pull
-    import test_pull_fallback
-    import test_push_command
-    import test_push_renames
-    import test_push_dirs
-    import test_push_eol
-    import test_push_autoprops
-    import test_single_dir_clone
-    import test_single_dir_push
-    import test_svnwrap
-    import test_tags
-    import test_template_keywords
-    import test_utility_commands
-    import test_unaffected_core
-    import test_urls
-
-    sys.path.append(os.path.dirname(__file__))
-    sys.path.append(os.path.join(os.path.dirname(__file__), 'comprehensive'))
-
-    import test_rebuildmeta
-    import test_stupid_pull
-    import test_updatemeta
-    import test_verify_and_startrev
-
-    return locals()
-
-def comprehensive(mod):
-    dir = os.path.basename(os.path.dirname(mod.__file__))
-    return dir == 'comprehensive'
+if sys.version_info[:2] < (2, 7):
+    import unittest2 as unittest
+else:
+    import unittest

 if __name__ == '__main__':
     description = ("This script runs the hgsubversion tests. If no tests are "
 <at>  <at>  -100,26 +57,22  <at>  <at>  if __name__ == '__main__':
         import tempfile
         sys.stdout = tempfile.TemporaryFile()

-    all_tests = tests()
-
-    args = [i.split('.py')[0].replace('-', '_') for i in args]
+    args = [os.path.basename(os.path.splitext(arg)[0]).replace('-', '_')
+            for arg in args]

     loader = unittest.TestLoader()
     suite = unittest.TestSuite()

     if not args:
-        check = lambda x: options.comprehensive or not comprehensive(x)
-        suite.addTests(loader.loadTestsFromModule(m)
-                       for (n, m) in sorted(all_tests.iteritems())
-                       if check(m))
+        suite = loader.discover('.')
+
+        if options.comprehensive:
+            suite.addTests(loader.discover('comprehensive',
+                                           top_level_dir='comprehensive'))
     else:
-        for arg in args:
-            if arg == 'test_util':
-                continue
-            elif arg not in all_tests:
-                print >> sys.stderr, 'test module %s not available' % arg
-            else:
-                suite.addTest(loader.loadTestsFromModule(all_tests[arg]))
+        sys.path.append(os.path.join(os.path.dirname(__file__), 'comprehensive'))
+
+        suite.addTests(loader.loadTestsFromNames(args))

     runner = unittest.TextTestRunner(**testargs)
     result = runner.run(suite)

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.

Jun Wu | 12 Jun 16:42 2016

Re: [PATCH] tests: drop hard-coded list of tests

I like the idea. But consider Mercurial still supports 2.6, it may be a bit
earlier to drop its support here.

Since we only have single test files, I think it's considerable to just list
them and call __import__:

    def discover(dir, pattern='test_*.py'):
        sys.path.append(dir)
        for path in glob.glob(os.path.join(dir, pattern))
            __import__(os.path.splitext(os.path.basename(path))[0])

The code is still simple and supports Py 2.6.

Excerpts from Dan Villiom Podlaski Christiansen's message of 2016-06-12 15:14:23 +0200:
> # HG changeset patch
> # User Dan Villiom Podlaski Christiansen <danchr@...>
> # Date 1465736157 -7200
> #      Sun Jun 12 14:55:57 2016 +0200
> # Node ID 8a755163ae4c76e6153f861a56ad40cc00344e3c
> # Parent  1d4b2c46b77640e0fd3422df07e111910b1b6662
> tests: drop hard-coded list of tests
> 
> The list of out of date, missing 'test_helpers' and
> 'comprehensive/test_custom_layout'. Instead, use the discover
> functionality available in Python 2.7.

--

-- 
You received this message because you are subscribed to the Google Groups "hgsubversion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hgsubversion+unsubscribe@...
To post to this group, send email to hgsubversion@...
Visit this group at https://groups.google.com/group/hgsubversion.
For more options, visit https://groups.google.com/d/optout.


Gmane