Feature list for 1.9

Now that we've released a 1.8 candidate, I thought it would be a good
time to share with the rest of dev <at>  what we five subversives at WANdisco
have been discussing for 1.9. This is only a very high-level list as I
don't want to make this mail too long ... I'm sure we'll have enough
time in Berlin to argue about specifics.

A. Server-side rename tracking

  * probably involves implementing EV2 (with shims) which includes
    rename operation
  * change the way copy-id is generated to avoid possible unique-key
    collisions:
      o curently rename is copy+delete: (node-id,copy-id,txn-id) ->
        (node-id,new-copy-id,new-txn-id)
      o a proper rename does: (node-id,copy-id,txn-id) ->
        (node-id,copy-id,new-txn-id)
      o the latter conflicts with the current way that copy-ids are
        "inherited" in the tree hierarchy during lazy copying
  * server (and possibly client?) should support simple heuristics for
    converting copy+delete records from older wire protocol into move
    records (possibly part of EV2 shim layer?)
  * metadata structures for rename support added by "svnadmin upgrade",
    no dump+reload required

B. Merge enhancements

  * general merge algorithm improvements (common algo for currently
    different cherrypick vs. whole-hawg merges, etc.)
  * merge tracks renames
  * possibly: redefinition of mergeinfo architecture

[svnbench] Revision: 1484368 compiled May 20 2013, 00:21:58 on x86_64-unknown-linux-gnu

1.7.0 <at> 1181106 vs. trunk <at> 1484356
Started at Mon May 20 00:26:14 UTC 2013

*DISCLAIMER* - This tests only file://-URL access on a GNU/Linux VM.
This is intended to measure changes in performance of the local working
copy layer, *only*. These results are *not* generally true for everyone.

Charts of this data are available at http://svn-qavm.apache.org/charts/

Averaged-total results across all runs:
---------------------------------------

Compare trunk <at> 1484356 to 1.7.0
       N        avg         operation
     21/9    0.56|-32.898   TOTAL RUN
   1K/530    0.88| -0.002   add
    42/18    0.79| -0.180   checkout
   168/72    0.64| -0.722   commit
     21/9    0.69| -0.008   copy
     21/9    0.79| -0.061   delete
   105/45    0.18| -3.572   info
    42/18    0.52| -1.035   merge
   1K/516    0.85| -0.002   mkdir
    56/21    0.81| -0.002   propdel
   15K/6K    0.78| -0.002   proplist
   16K/6K    0.80| -0.002   propset
   1K/591    0.78| -0.002   ps
    42/18    1.88| +0.009   resolve
    42/18    0.83| -0.033   resolved
  294/126    0.69| -0.055   status

Review of invoke-diff-cmd-feature branch

Review in diff form:

Index: subversion/include/svn_client.h
===================================================================
--- subversion/include/svn_client.h	(revision 1484305)
+++ subversion/include/svn_client.h	(working copy)
 <at>  <at>  -2988,8 +2988,8  <at>  <at>  svn_client_blame(const char *path_or_url,
  *
  *  <at> a invoke_diff_cmd is used to call an external diff program but may
  * not be  <at> c NULL.  The command line invocation will override the
- * invoke-diff-cmd invocation entry(if any) in the Subversion
- * configuration file.
+ * invoke-diff-cmd invocation entry(if any) in  <at> c ctx->config.
+ * ### "May not be NULL" !?  log-cmd.c and deprecated.c pass NULL for it.
  *
  * Generated headers are encoded using  <at> a header_encoding.
  *
 <at>  <at>  -3047,7 +3047,8  <at>  <at>  svn_client_diff7(const apr_array_header_t *options
                  svn_client_ctx_t *ctx,
                  apr_pool_t *pool);

-/** Similar to svn_client_diff7(), but with  <at> a invoke_diff_cmd.
+/** Similar to svn_client_diff7(), but with  <at> a invoke_diff_cmd
+ * set to  <at> c NULL.
  *
  *  <at> deprecated Provided for backward compatibility with the 1.8 API.
  *  <at> since New in 1.8.
 <at>  <at>  -3245,6 +3246,7  <at>  <at>  svn_client_diff_peg7(const apr_array_header_t *dif
 /** Similar to svn_client_peg5(), but with  <at> a no_diff_added set to
  *  FALSE,  <at> a ignore_properties set to FALSE and  <at> a properties_only

All the fuzz about properties and hashes

Apache Subversion 1.8.0-rc2 Released

I'm happy to announce the release of Apache Subversion 1.8.0-rc2, the
first public release candidate of Subversion 1.8.0.
Please choose the mirror closest to you by visiting:

    http://subversion.apache.org/download/#pre-releases

The SHA1 checksums are:

    272a32237cf297f9cca5f511e600e46ac19ab052 subversion-1.8.0-rc2.tar.bz2
    9b8d319f6b556f856348a72563fd2d5005a6fa31 subversion-1.8.0-rc2.tar.gz
    bab03d90dc8cc27877372ba875009c5b878c9785 subversion-1.8.0-rc2.zip

PGP Signatures are available at:

    http://www.apache.org/dist/subversion/subversion-1.8.0-rc2.tar.bz2.asc
    http://www.apache.org/dist/subversion/subversion-1.8.0-rc2.tar.gz.asc
    http://www.apache.org/dist/subversion/subversion-1.8.0-rc2.zip.asc

For this release, the following people have provided PGP signatures:

   Ben Reser [4096R/16A0DE01] with fingerprint:
    19BB CAEF 7B19 B280 A0E2  175E 62D4 8FAD 16A0 DE01
   Branko Čibej [2048R/C8628501] with fingerprint:
    8769 28CD 4954 EA74 87B6  B96C 29B8 92D0 C862 8501
   C. Michael Pilato [4096R/FE681333] with fingerprint:
    753B 2F9D F717 FA23 A43E  E7C3 F5E0 F001 FE68 1333
   Ivan Zhakov [4096R/F6AD8147] with fingerprint:
    4829 8F0F E47F 4B8A 43FD  6525 919F 6F61 F6AD 8147
   Mark Phippard [1024D/035A96A9] with fingerprint:
    D315 89DB E1C1 E9BA D218  39FD 265D F8A0 035A 96A9
   Paul T. Burba [4096R/56F3D7BC] with fingerprint:
    1A0F E7C6 B3C5 F8D4 D0C4  A20B 64DD C071 56F3 D7BC
   Philip Martin [2048R/ED1A599C] with fingerprint:
    A844 790F B574 3606 EE95  9207 76D7 88E1 ED1A 599C
   Stefan Sperling [2048R/9A59B973] with fingerprint:
    8BC4 DAE0 C5A4 D65F 4044  0107 4F7D BAA9 9A59 B973

This is a release candidate for what will eventually become Apache
Subversion 1.8.0.  It is thought to be free of blocking issues, and if
none are found will become the final release.  For this reason, we
encourage thorough testing in as many environments as possible.  This
release candidate begins the four-week "soak" period to allow for
further testing, and barring show-stopping bugs, the final 1.8.0
release can be expected on or near June 14th.

A pre-release means the Subversion developers feel that this release
is ready for widespread testing by the community.  There are known issues
(and unknown ones!), so please use it at your own risk, though we do
encourage people to test this release thoroughly.  Of particular note, please
remember than persistent data, such as the working copy or repository
formats may change before the final release, and there may not be an
upgrade path from the pre-releases to the final.

As a note to operating system distro packagers: while we wish to have this
release candidate widely tested, we do not feel that it is ready for packaging
and providing to end-users through a distro package system.  Packaging a
release candidate poses many problems, the biggest being that our policy lets
us break compatibility between the release candidate and the final release, if
we find something serious enough.  Having many users depending on a release
candidate through their distro would cause no end of pain and frustration that
we do not want to have to deal with.  However, if your distro has a branch that
is clearly labeled as containing experimental and often broken software, and
explicitly destined to consenting developers and integrators only, then we're
okay with packaging the release candidate there.  Just don't let it near the
end users please.

Release notes for the 1.8.x release series may be found at:

    http://subversion.apache.org/docs/release-notes/1.8.html

You can find the list of changes between 1.8.0-rc2 and earlier versions at:

    http://svn.apache.org/repos/asf/subversion/tags/1.8.0-rc2/CHANGES

Questions, comments, and bug reports to users <at> subversion.apache.org.

Thanks,
- The Subversion Team

Uncaught tree conflicts

Hi dev,

what output do you expect for these two almost similar cases?

[[[
# make a simple dir with file
mkdir srcdir
echo x > srcdir/x
svn add srcdir
svn ci -m1

# make a locally added dir
mkdir branchdir
echo yy > branchdir/x
svn add branchdir

# create a branch behind the locally added dir
svn cp -m2 ^/srcdir ^/branchdir

# case (2) adds only this line:
#rm -rf branchdir

svn up --accept=postpone
svn st
]]]

The output you expect -- a tree conflict, right? -- you're not getting
it.

Case (1) gives a *text* conflict:
[[[
+ svn up --accept=postpone
Updating '.':
C    branchdir/x
E    branchdir
Updated to revision 2.
Summary of conflicts:
  Text conflicts: 1
]]]

Case (2) goes completely nuts:
[[[
+ svn up --accept=postpone
Updating '.':
subversion/svn/update-cmd.c:172,
subversion/libsvn_client/update.c:706,
subversion/libsvn_client/update.c:614,
subversion/libsvn_client/update.c:466,
subversion/libsvn_wc/adm_crawler.c:845,
subversion/libsvn_repos/reporter.c:1542,
subversion/libsvn_repos/reporter.c:1453,
subversion/libsvn_repos/reporter.c:1445,
subversion/libsvn_repos/reporter.c:1383,
subversion/libsvn_repos/reporter.c:1321,
subversion/libsvn_repos/reporter.c:1040,
subversion/libsvn_repos/reporter.c:1321,
subversion/libsvn_repos/reporter.c:1074,
subversion/libsvn_wc/update_editor.c:4374,
subversion/libsvn_wc/update_editor.c:4071,
subversion/libsvn_wc/update_editor.c:3911,
subversion/libsvn_wc/merge.c:1176,
subversion/libsvn_wc/merge.c:869,
subversion/libsvn_wc/merge.c:414,
subversion/libsvn_diff/diff_file.c:1332,
subversion/libsvn_diff/diff3.c:277,
subversion/libsvn_diff/diff_file.c:788,
subversion/libsvn_subr/io.c:3271: (apr_err=ENOENT)
svn: E000002: Can't open file '/tmp/trunk.wvg/wc/branchdir/x': No such
file or directory
]]]

I'll add two issues if I get plussed.

~Neels

[PATCH] SVN_UNALIGNED_ACCESS_IS_OK for ppc

[[[
* subversion/include/svn_types.h:
   Set SVN_UNALIGNED_ACCESS_IS_OK for PowerPC.
]]]

Index: subversion/include/svn_types.h
===================================================================
--- subversion/include/svn_types.h	(revision 1483563)
+++ subversion/include/svn_types.h	(working copy)
 <at>  <at>  -75,7 +75,8  <at>  <at> 
  *  <at> since New in 1.7.
  */
 #ifndef SVN_UNALIGNED_ACCESS_IS_OK
-# if defined(_M_IX86) || defined(_M_X64) || defined(i386) || defined(__x86_64)
+# if defined(_M_IX86) || defined(_M_X64) || defined(i386) || defined(__x86_64) \
+     || defined __powerpc__ || defined __ppc__
 #  define SVN_UNALIGNED_ACCESS_IS_OK 1
 # else
 #  define SVN_UNALIGNED_ACCESS_IS_OK 0

1.8.0-rc2 up for testing/signing

Here it is: the second Release Candidate for Subversion 1.8.0.  You can
fetch the proposed tarballs from here:
  https://dist.apache.org/repos/dist/dev/subversion

The magic rev is r1482456

This is a release candidate, meaning if all goes well it (or something
very much like it) will become the final release.  Please test this as
if it were the final release, understanding that there will be bugs in
any software we ship.  <insert lecture about perfect being enemy of
the good here>

To all distributors and binary packagers: This isn't the final
release.  It *won't* be the final release for a few weeks yet, so
please don't ship it as such.  In any case, don't distribute any
releases based upon these tarballs until we complete our signature
gathering process and officially announce the release.

Reminder that the STATUS file is still for 1.8.0, please be careful
about moving changes to the approved section and only move changes
that appropriate to be merged at this point (i.e. changes that would not
restart our soak).  If we have changes we want to put into 1.8.1, it's fine
to put them in the STATUS file and vote on them, just please do not move
them to the approved section.  It may be helpful to refer to this section of
our documentation when determining what can done to the 1.8.x branch
during our soak period:
http://subversion.apache.org/docs/community-guide/releasing.html#release-stabilization

FSFS Repository corruption on high load on Windows (was Re: svn commit: r1082451 - /subversion/trunk/subversion/libsvn_subr/io.c)

On Thu, Mar 17, 2011 at 4:21 PM,  <rhuijben <at> apache.org> wrote:
> Author: rhuijben
> Date: Thu Mar 17 12:21:29 2011
> New Revision: 1082451
>
> URL: http://svn.apache.org/viewvc?rev=1082451&view=rev
> Log:
> On Windows avoid a file flush that is taking over 15% of the checkout time.
>
> * subversion/libsvn_subr/io.c
>   (svn_io_write_unique): Avoid a very costly file flush on Windows.
>

[...]

> --- subversion/trunk/subversion/libsvn_subr/io.c (original)
> +++ subversion/trunk/subversion/libsvn_subr/io.c Thu Mar 17 12:21:29 2011
>  <at>  <at>  -3199,8 +3199,17  <at>  <at>  svn_io_write_unique(const char **tmp_pat
>
>    err = svn_io_file_write_full(new_file, buf, nbytes, NULL, pool);
>
> -  if (!err)
> +  /* ### BH: Windows doesn't have the race condition between the write and the
> +     ###     rename that other operating systems might have. So allow windows
> +     ###     to decide when it wants to perform the disk synchronization using
> +     ###     the normal file locking and journaling filesystem rules.
> +
> +     ### Note that this function doesn't handle the rename, so we aren't even
> +     ### sure that we really have to sync. */
> +#ifndef WIN32
> +  if (!err && nbytes > 0)
>      err = svn_io_file_flush_to_disk(new_file, pool);
> +#endif
>
Hi Bert,

How did you come to conclusion that Windows doesn't have race
condition between write and rename?

It seems Windows actually have race condition between write and move
in case of unexpected computer shutdown. In this case move operation
completes after restart, but file contains zero data. I understand
that you intention was optimize client side operation but
svn_io_write_unique() function actively used in FSFS and lead
repository data corruption.

I've created small program to demonstrate this problem (see
attachment). On start it create four threads actively writing data to
disk and one thread that uses write + move to update file atomically.
If you run this program on virtual machine and then reset then with
very good chance you'll get 'current' file with zeros.

--

-- 
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com

With 1.8.0-rc1 DOA, shall we accept more approved backports?

With 1.8.0-rc1 DOA, should we roll more of the 1.8.x/STATUS items into what
will be rc2?  There look to be quite a few that have "(1.8.1)" decorators on
them, I assume because they aren't considered valuable enough to restart the
soak process.  But the soak is already restarted, so...

Ben, can you give us a new (very near-term, please) deadline for backport
approvals based on when you plan to cut rc2?

--

-- 
C. Michael Pilato <cmpilato <at> collab.net>
CollabNet   <>   www.collab.net   <>   Enterprise Cloud Development

Nominating your r1481848 and r1481870

Hi Brane,

Just saw the veto on my compiler warning commits for 1.8.x.

I nominated your two commits on top of my commits (to avoid merge conflicts), 
which should resolve the veto, but I left it in the vetoed section to be safe. 
Pinging you here so you can take a look.

Blair