Re: Delta source ended unexpectedly [500, #200003] using Subversion 1.6.17 with Apache http server 2.2.21

On Sat, Feb 11, 2012 at 04:36:46PM +0530, Ravi Roy wrote:
> I found this under
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=193527
> in the mail archive of dated back to 2002 and this seems to be fixed as per
> the bug report.
> 
> Can somebody please comment on the issue?

So you're running a master/slave setup with a write-through proxy
configuration, yes? You don't mention whether you're quoting logs
from the master or from the slave server.

Is the slave repository properly synced with the master?

Does 'svnadmin verify' pass on both sides or is there repository
corruption involved?

Could this problem be relevant?
http://subversion.tigris.org/issues/show_bug.cgi?id=3445

> On Sat, Feb 11, 2012 at 2:40 PM, Ravi Roy <ravi.aroy <at> gmail.com> wrote:
> 
> > Hello community,
> >
> > I have compiled subversion 1.6.17 and Apache 2.2.21 from source on CentOS
> > 5.6 and serving my repository using Apache httpd server with the following
> > details :
> >
> > Server version: Apache/2.2.21 (Unix)
> > Server loaded:  APR 1.4.5, APR-Util 1.3.12

Delta source ended unexpectedly [500, #200003] using Subversion 1.6.17 with Apache http server 2.2.21

Re: multiple svn front-ends, single SAN repo volume

Re: multiple svn front-ends, single SAN repo volume

On Fri, Feb 10, 2012 at 04:47:31PM -0600, Ryan Schmidt wrote:
> So thinking all this through, I agree svnsync does not make sense if
> you are hosting a repository on a SAN and trying to connect multiple
> svn servers to it. But it sounds like it would work fine, if you
> simply don't use svnsync. Configure one server to be the master (let
> it accept write requests). Configure the other servers to be slaves
> (read-only, and proxy any incoming write requests to the master). All
> servers point to the same repository data on the SAN and it can't get
> corrupted because only one server is writing to it. Sound ok?

Ah, I see what you mean.

Well, I suppose this would work, yes. You are essentially using
the write-through proxy feature to implement load balancing for
incoming TCP connections.

But it isn't necessary because the SAN should support file locking
so multiple concurrent servers writing to the same repository
synchronise write operations anyway.

E.g. consider offering both http:// and svn:// access to the repository.
This is rarely done but it is a supported use case. In your scheme, 
only http:// works because the write-through proxy feature is specific
to mod_dav_svn.

Re: multiple svn front-ends, single SAN repo volume

On Feb 10, 2012, at 16:34, Stefan Sperling wrote:

> On Fri, Feb 10, 2012 at 04:20:02PM -0600, Ryan Schmidt wrote:
>>> While replicating commits, svnsync performs the exact same kinds of
>>> write operations against the slave servers that happen on the master
>>> repository when a client makes a commit.
>> 
>> So when using svnsync one should always use a separate and preferably
>> local copy of the repository(ies) on each server, yes?
> 
> Yes. That's the entire idea. Revisions are replicated by performing
> a commit against the slave. The only difference to a normal commit
> is that data originates from an existing revision of a different
> repository, rather than a working copy.

Here you're talking about a commit that happened on the master, and is being replicated to the slaves. I
understand how that works.

I was talking about a commit that is done against the slave, and is proxied to the master. The book says:

http://svnbook.red-bean.com/en/1.7/svn.serverconfig.httpd.html#svn.serverconfig.httpd.extra.writethruproxy

"All read requests go to their local slave. Write requests get automatically routed to the single master
server. When the commit completes, the master then automatically “pushes” the new revision to each
slave server using the svnsync replication tool."

So thinking all this through, I agree svnsync does not make sense if you are hosting a repository on a SAN and
trying to connect multiple svn servers to it. But it sounds like it would work fine, if you simply don't use
svnsync. Configure one server to be the master (let it accept write requests). Configure the other
servers to be slaves (read-only, and proxy any incoming write requests to the master). All servers point
to the same repository data on the SAN and it can't get corrupted because only one server is writing to it.
Sound ok?

Re: multiple svn front-ends, single SAN repo volume

On Fri, Feb 10, 2012 at 04:20:02PM -0600, Ryan Schmidt wrote:
> > While replicating commits, svnsync performs the exact same kinds of
> > write operations against the slave servers that happen on the master
> > repository when a client makes a commit.
> 
> So when using svnsync one should always use a separate and preferably
> local copy of the repository(ies) on each server, yes?

Yes. That's the entire idea. Revisions are replicated by performing
a commit against the slave. The only difference to a normal commit
is that data originates from an existing revision of a different
repository, rather than a working copy.

This design allowed svnsync to reuse functionality already implemented
for normal svn clients (e.g. network support). The slave server cannot 
tell the difference between an svn client and svnsync. Well, it could
by looking for revision properties of revision 0 which svnsync uses
to store meta data. But it cannot as far as the commit process is concerned.

You probably assumed that svnsync transferred underlying revision data
files across the network? That is not how svnsync operates.
But 'svnadmin hotcopy' works this way (minus the network support ;)
This is why 'svnadmin hotcopy' is faster when making a copy of a
complete repository. It performs a direct disk-to-disk copy,
and does not replay commits like svnsync does.

Re: multiple svn front-ends, single SAN repo volume

On Feb 10, 2012, at 16:16, Stefan Sperling wrote:
> On Fri, Feb 10, 2012 at 04:09:45PM -0600, Ryan Schmidt wrote:
>> you could consider having any
>> number of read-only slave servers, which would each proxy their write
>> requests back to the single master server that Subversion supports.
>> This way read operations would be accelerated, while write operations
>> would be securely limited to just the single master. The slave servers
>> could keep individual copies of the repository(ies) synchronized with
>> the master using svnsync
> 
> This is misleading. A write-through proxy setup does not eliminate
> write operations on slave servers.

Oh I see! That is not how I had envisioned it happening, so thank you for the clarification.

> While replicating commits, svnsync performs the exact same kinds of
> write operations against the slave servers that happen on the master
> repository when a client makes a commit.

So when using svnsync one should always use a separate and preferably local copy of the repository(ies) on
each server, yes?

Re: multiple svn front-ends, single SAN repo volume

On Fri, Feb 10, 2012 at 04:09:45PM -0600, Ryan Schmidt wrote:
> Have you verified that a single server will not be fast enough?

Good point. It might very well be fast enough.

> If so, you could consider having any
> number of read-only slave servers, which would each proxy their write
> requests back to the single master server that Subversion supports.
> This way read operations would be accelerated, while write operations
> would be securely limited to just the single master. The slave servers
> could keep individual copies of the repository(ies) synchronized with
> the master using svnsync

This is misleading. A write-through proxy setup does not eliminate
write operations on slave servers.

While replicating commits, svnsync performs the exact same kinds of
write operations against the slave servers that happen on the master
repository when a client makes a commit.

In fact, in the corrupted rep-cache.db case I mentioned earlier,
the write operation to the CIFS share was performed by svnsync
(luckily, only the slave was storing its repositories on CIFS :)

Re: multiple svn front-ends, single SAN repo volume

On Feb 10, 2012, at 15:00, Nico Kadel-Garcia wrote:

> On Fri, Feb 10, 2012 at 1:21 PM, Bruce Lysik wrote:
>> 
>> I'm considering deploying 3 front-ends, all mounting the same SAN volume for
>> repo.  (The SAN handle flock() and fnctl() correctly.)  These 3 FEs would be
>> load balanced by a Citrix Netscaler.  (At least for http(s).)
>> 
>> Would there be any problems with this configuration?
> 
> Potentially. Read operations, I wouldn't expect to be a big problem,
> but commit operations need to atomic, and the software wasn't
> *written* to behave well with network mounted back end filesystems
> across multiple servers. So I wouldn't know, off hand, what phase
> delays between two front ends writing revisions at the same time might
> create for genuine adventures on the back end.
> 
> If you need 3 front ends, you might really consider Wandisco's
> commercial package, which is designed for multiple front end use.

My understanding is that WANdisco's strength is for multi-site deployments, but the OP's mention of a load
balancer implies to me that all servers will be located at a single site. Bruce, if that's so, are three
servers really needed? Have you verified that a single server will not be fast enough? If so, you could
consider having any number of read-only slave servers, which would each proxy their write requests back
to the single master server that Subversion supports. This way read operations would be accelerated,
while write operations would be securely limited to just the single master. The slave servers could keep
individual copies of the repository(ies) synchronized with the master using svnsync, or possibly
keeping a single copy of the data on a SAN that all the servers access would be ok, though as Nico said that's
not the usual (and well-tested) configuration.

Re: multiple svn front-ends, single SAN repo volume

On Fri, Feb 10, 2012 at 04:00:02PM -0500, Nico Kadel-Garcia wrote:
> On Fri, Feb 10, 2012 at 1:21 PM, Bruce Lysik <blysik <at> yahoo.com> wrote:
> > Hi,
> >
> > I'm considering deploying 3 front-ends, all mounting the same SAN volume for
> > repo.  (The SAN handle flock() and fnctl() correctly.)  These 3 FEs would be
> > load balanced by a Citrix Netscaler.  (At least for http(s).)
> >
> > Would there be any problems with this configuration?
> 
> Potentially. Read operations, I wouldn't expect to be a big problem,
> but commit operations need to atomic, and the software wasn't
> *written* to behave well with network mounted back end filesystems
> across multiple servers. So I wouldn't know, off hand, what phase
> delays between two front ends writing revisions at the same time might
> create for genuine adventures on the back end.

Subversion was designed to allow multiple concurrent server processes
accessing the same repository.

And, generally, yes, there is a lot less risk of curruption if no
network i/o is involved when data is written to the repository
by a server process. After all, you're adding yet another complex
layer where something can go wrong.

But assuming locking via fcntl() works correctly, there shouldn't be
a problem with FSFS repositories ("svnadmin create --fs-type=fsfs",
which is default). FSFS was specifically designed for use with NFS.
This was stated in the release announcement of Subversion 1.1 which
was the first release to support FSFS
(see http://subversion.apache.org/docs/release-notes/1.1.html).
Technical details are available at
http://svn.apache.org/repos/asf/subversion/trunk/notes/fsfs

SAN storage usually appears as a local disk, so things should work fine.
I know of setups that run virtualised servers which access repositories
on SAN storage without issues.

It would be far from the truth to claim that problems have never been
seen on network filesystems, though. For example, I know people who,
after putting FSFS repositories on a CIFS share, ended up with a corrupt
rep-cache.db. This is an sqlite database added to FSFS in Subversion 1.6.
Sqlite requires the same locking primitives that Subversion requires
(see http://sqlite.org/faq.html#q5). This problem happened even with
just a single server instance writing to the repository. However, the
most likely cause is flawed or misconfigured file locking support in
the CIFS implementation. I could not examine the failure in detail.
But it involved a huge commit that took a long time, with possibly
concurrent commits. The rep-cache.db file is opened by every commit
operation so a file locking race is quite likely to hit here (not
discounting other possible races such as two commits writing to the
same revision file at the same time, they're just less likely).
The rep cache could be disabled in fsfs.conf without harm to recover
from this problem. Disabling this cache only increases the odds that
future revisions store redundant content but has no effect on correctness.
AFAIK repositories were moved off the share and the problem has not
occured since.

Re: Subversion reports error.

To all who helped me.

Thanks a lot for your kind helps.

Finally, I found that it was my environment issue. There was "usr/bin/ssh"
in my "Users/account/.ssh/config" file. It couldn't seen from Mac's Finder
as its name starts with ".".

I used Terminal.app to correct it to "/usr/bin/ssh" and succeeded to connect
to Subversion via Terminal.app.

But, it still failed connecting from Dreamweaver. Even after I changed the
Subversion setup with site in DW, it still tried to search for old server.

I couldn't find the misconfiguration so I create a new folder and copied
visible sources from Finder in order to not to include the files starts
with ".".

And re-create the site setting in DW and it works correctly now.

Thanks again for all your helps. This issue is closed.

Kindest regards,
Masaru

On 2012/02/10, at 8:23, Ryan Schmidt wrote:
> On Feb 9, 2012, at 16:09, Alagazam.net Subversion wrote:
>> On 2012-02-09 00:23, Ryan Schmidt wrote:
>>> On Feb 8, 2012, at 15:41, Alagazam.net Subversion wrote:
>>> 
>>>> On Feb 7, 2012, at 15:44, Masaru Kitajima wrote:
>>>> 
>>>>> On 2012/02/08, at 6:36, Daniel Shahaf wrote:
>>>>> 
>>>>>> What is the output of
>>>>>> 
>>>>>> % ssh 
>>>>>> section-9 <at> section-9.sakura.ne.jp
>>>>>> svnserve -t
>>>>>> 
>>>>> It is as below:
>>>>> 
>>>>> ( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops
partial-replay ) ) ) 
>>>>> 
>>>>> and then stops. A prompt is not shown.
>>>>> 
>>>> This symptom of not getting any prompt back reminds me of a totally
>>>> non-svn related "bug" but a network error I encountered some time ago.
>>>> 
>>> Not seeing a prompt in this case is not a bug; it's expected behavior. svnserve is not an interactive
program that has a prompt. It's a Subversion server; the above test demonstrated that svnserve is running
correctly and is waiting for a Subversion client to connect to it.
>> 
>> Maybe I was reading the threads wrong [snip]
> 
> Yes; here are the two original messages in question:
> 
> http://mail-archives.apache.org/mod_mbox/subversion-users/201202.mbox/%3c3B642053-4FF0-4BD0-A0DE-7EA55D053666 <at> gmail.com%3e
> 
> http://mail-archives.apache.org/mod_mbox/subversion-users/201202.mbox/%3cE130C27C-FB32-453A-8F8A-830D079EAAD2 <at> gmail.com%3e
> 
> 

<><><><><><><><><><><><><><><><><><><><><><><><><><>
Manager / Photographer / Lecturer / Writer
Masaru Kitajima

E-mail:tachi.silver <at> gmail.com
blog:http://www.section-9.jp/blog/bluez/

<><><><><><><><><><><><><><><><><><><><><><><><><><>