headers
Miquel van Smoorenburg | 1 Jun 2005 23:50
Picon

PATCH: Diablo-style hashfeeds

I'm using INN to (help) feed a bunch of Diablo boxes. In the
diablo world, it is common to split one full feed over several
boxes using something called a "hashfeed". This patch adds
that functionality to innd.

( Russ, did you look at the innfeed-refcount-arthandle patch
  I sent some time ago yet ? )

inn-2.4.2-hashfeed.patch

diff -ruN inn-2.4.2/doc/pod/newsfeeds.pod inn-2.4.2-new/doc/pod/newsfeeds.pod
--- inn-2.4.2/doc/pod/newsfeeds.pod	2004-12-22 05:21:19.000000000 +0100
+++ inn-2.4.2-new/doc/pod/newsfeeds.pod	2005-05-03 23:54:12.000000000 +0200
 <at>  <at>  -382,6 +382,29  <at>  <at> 
 new process will run with.  This flag can be used to raise the priority to
 normal if you're using the I<nicekids> parameter in F<inn.conf>.

+=item B<Q> I<hashfeed>
+
+The hashfeed value for this site. The message-id of the article is
+hashed using a quick (and dirty) hash method (31-bits sum of the value
+of all characters). The hashfeed value must be given as value/mod
+or as start-end/mod. If the message-id hash modulus "mod" plus one
+equals "number" or is between "start" and "end", the article will be
+fed to the site.
+
+Example:
+
+    Q1/2     Feeds about 50% of the messages to this site
+    Q2/2     Feeds the other 50% of the messages
(Continue reading)

Permalink | Reply | 0 comments |
headers
Chris Caputo | 2 Jun 2005 06:45

PATCH: fix unnecessary flushes in hisv6

My expire process was running slow.  I ran an strace while it was running
and noticed that it was doing ~8k read()s and 48 byte write()s.  A write()  
call for each line in the history file is bad, especially during an
expire.

Turns out hisv6_writeline() calls ftello() and ftello() in glibc was
causing buffered writes to flush.

hisv6 has its own flush behavior, based on things like icdsynccount or
HIS_INCORE flag, but these were being superceded by ftello()'s
implementation.

The following patch makes it so ftello() isn't used, but rather an
internal offset is maintained.

Now when I run expire, the write()s are ~4k at a time, rather than 48
bytes at a time.  Net gain is 1/85th the number of write() system calls
and no llseek() system calls.  This appears to have shaved a minute or two
off my expire, while lowering CPU usage, but I have not done extensive
benchmarking.

I've tested it with my system and it appears to work.  I'd appreciate if
others would test it.  Russ please feel free to integrate this if it looks
good.

Patch at http://www.caputo.com/foss/inn-2.4.2-hisv6_syncfix.patch or
below.

Thanks,
Chris
(Continue reading)

Permalink | Reply | 1 comment |
headers
Chris Caputo | 2 Jun 2005 23:21

PATCH: fix bug in 2.4.2 dbz.c:getcore()

Hi.  getcore() in inn-2.4.2's lib/dbz.c calls madvise() with an incorrect
length.  In addition to defeating the purpose of madvise(), on Linux this
results in extra virtual memory being used since madvise() creates a
second map when a length does not match an existing map.

This patch fixes this and cleans up the function a little by adding a
variable for a common calculation.

Patch is below and also at:

  http://www.caputo.com/foss/inn-2.4.2-dbz_getcorefix.patch

Thanks,
Chris

--- inn-2.4.2-stock/lib/dbz.c	2004-12-22 04:21:19.000000000 +0000
+++ inn-2.4.2/lib/dbz.c	2005-06-02 20:55:29.676186007 +0000
 <at>  <at>  -1238,9 +1238,9  <at>  <at> 
     return ret;
 }

-/* getcore - try to set up an in-core copy of .pag file
+/* getcore - try to set up an in-core copy of file
  *
- * Returns: pointer to copy of .pag or NULL on errror
+ * Returns: pointer to copy of file or NULL on errror
  */
 static bool
 getcore(hash_table *tab)
 <at>  <at>  -1249,6 +1249,7  <at>  <at>
(Continue reading)

Permalink | Reply | 1 comment |
headers
Christoph Biedl | 5 Jun 2005 22:07
Picon
Picon

Re: tinyleaf miniature transit news server

Russ Allbery wrote...

> Christoph Biedl <cbiedl <at> gmx.de> writes:

> > Is it possible to backport tinyleaf to 2.4.2?
(...)

> It's probably easier to just build the binary from CURRENT; it doesn't use
> inn.conf or anything and is completely stand-alone.

Ups, how simple. Static binaries. I didn't know such things still happen.

> Sigh, I had a bunch of stupid mistakes left over from having changed the
> API on myself a few times.  Now fixed in CVS.

Now I finally found the time to play with tinyleaf again and ran into
another issue: An innfeed cannot transmit articles:

| innfeed[23498]: tinyleaf:0 cxnsleep response format: \
|                 tinyfeed: starting 200 tinyfeed ready

The greeting is garbled due to
|    notice("starting");
[ l.251 ]
which is printed to stderr according to strace. Adding an "2>/dev/null"
in the right place solves the problem - but do things work as intended?

    Christoph
(Continue reading)

Permalink | Reply | 2 comments |
headers
Russ Allbery | 5 Jun 2005 23:41
Picon
Favicon
Gravatar

Re: tinyleaf miniature transit news server

Christoph Biedl <cbiedl <at> gmx.de> writes:

> The greeting is garbled due to
> |    notice("starting");
> [ l.251 ]
> which is printed to stderr according to strace. Adding an "2>/dev/null"
> in the right place solves the problem - but do things work as intended?

That smells like debugging code that I left in.  Will fix.

--

-- 
Russ Allbery (rra <at> stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.
Permalink | Reply | 1 comment |
headers
Russ Allbery | 6 Jun 2005 05:04
Picon
Favicon
Gravatar

Re: tinyleaf miniature transit news server

Russ Allbery <rra <at> stanford.edu> writes:
> Christoph Biedl <cbiedl <at> gmx.de> writes:

>> The greeting is garbled due to
>> |    notice("starting");
>> [ l.251 ]
>> which is printed to stderr according to strace. Adding an "2>/dev/null"
>> in the right place solves the problem - but do things work as intended?

> That smells like debugging code that I left in.  Will fix.

Ah, I see.  No, I was assuming tcpserver behavior of not sending stderr to
the network, rather than inetd behavior that does.  Really, that message
is extraneous and the other message logged via notice should go to syslog.
I'll make that change.

That will be a good test to be sure the new Subversion repository is
sending commit messages to the right place and they'll go through
properly.

--

-- 
Russ Allbery (rra <at> stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.
Permalink | Reply | 0 comments |
headers
Russ Allbery | 6 Jun 2005 05:34
Picon
Favicon
Gravatar

Re: "make check" fails

Russ Allbery <rra <at> stanford.edu> writes:

> It's a bug in the test suite, or rather a test that wasn't updated
> properly when we decided on a change in behavior.  Sorry about that.
> I'll try to get the test suite updated for 2.4.3 (most of the work on
> the test suite has been going into CURRENT instead).

The test suite should pass again now in Subversion.  (I'll send out
details about the new repository in just a moment.)

--

-- 
Russ Allbery (rra <at> stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.
Permalink | Reply | 0 comments |
headers
Russ Allbery | 6 Jun 2005 06:13
Picon
Favicon
Gravatar

Subversion repository

nyheter is still up and down, and it was down this afternoon at the time
when I wanted to start working on INN, so I took that as a sign and did
the Subversion conversion.

DNS is likely still propagating, but if it's caught up with you, you can
get to the Subversion repository via:

    <http://inn.eyrie.org/svn/>
    <http://inn.eyrie.org/viewcvs>

The latter is for human viewing; the former is suitable for pointing a
Subversion client at.  In other words, you can now do:

    svn co http://inn.eyrie.org/svn/trunk inn

to check out a copy of the trunk.  The repository accessed by those two
URLs is a public read-only copy of the actual repository and is updated
hourly (so you won't see new commits immediately).

Right now, I'm hosting the repository on one of my machines.  I'll talk to
Peter about whether it makes sense to leave it there or move it back to
ISC; I'd rather do the latter at some point so that I don't have to worry
about backups and the like, but it was easier for me to do the initial
conversion and tool setup on my own system.

Please note that the INN repository as converted by cvs2svn is more than
7,000 revisions, so don't go pointing svk at it as is.  I'm going to set
up something to generate periodic repository dumps that you can use to
seed an svk repository.  The repository follows the normal Subversion
trunk, branches, tags layout.  If you're not familiar with Subversion, the
(Continue reading)

Permalink | Reply | 0 comments |
headers
Russ Allbery | 6 Jun 2005 06:27
Picon
Favicon
Gravatar

Re: innfeed: cxnsleep can't write command: Bad address [PATCH]

"Miquel van Smoorenburg" <list-inn-workers <at> news.cistron.nl> writes:

> I've seen this message regularly in my logs:

> May  2 01:40:47 quantum innfeed[10062]: transit.news.xs4all.nl:6 cxnsleep can't write command: Bad address
> May  2 01:40:47 quantum innfeed[10062]: transit.news.xs4all.nl:6 final seconds 26 offered 249
accepted 59 refused 181 rejected 6 accsize 21955221 rejsize 2292445
> .. but hadn't payed much attention to it. Now that I am using
> innd/innfeed to feed much more diablo boxes a full feed, I'm
> seeing it more and more.

> It turns out that Diablo can return an "article rejected" status
> before the entire article has been transfered. But by then,
> hostArticleRejected has called delArticle and SMfreearticle()
> or munmap() has been called, and the nntpBuffers point to
> data no longer there. Which means that writev() will EFAULT.

> The correct solution is to put arthandle/mMapping in a seperate
> struct and refcount it, only releasing it when the last buffer
> that references it is deleted.

I'm not at all sure that I like this.  It adds another layer of reference
counting to the layer that we already have, and as near as I can tell, is
doing so because the existing reference counting doesn't work properly.
Shouldn't we instead fix the reference counting on articles so that
delArticle won't free the allocated data while it's still referenced
elsewhere?

It seems like the truly correct thing to do here is to teach
hostArticleRejected how to dig the article out of the NNTP buffers for
(Continue reading)

Permalink | Reply | 2 comments |
headers
Russ Allbery | 6 Jun 2005 06:33
Picon
Favicon
Gravatar

Re: Openssl 0.9.8

The Doctor <doctor <at> doctor.nl2k.ab.ca> writes:

> Looks as if Openssl 0.9.8 will have new stuff in it.
> Can this please be looked st for the --with-openssl option?

Is there anything in particular that you're seeing?  In a quick skim of
<http://www.openssl.org/news/news.html>, I didn't see anything that jumped
out as being particularly applicable to INN.

--

-- 
Russ Allbery (rra <at> stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.
Permalink | Reply | 1 comment |

Gmane