headers
bill davidsen | 1 Dec 2003 05:58

Any reason not to split perl filtering?

Single threaded innd is simply not keeping up with the incoming volume,
and the obvious trouble spot is the perl filter. I have one CPU 98% used
and the rest idle, is there any reason not to move filtering into a
separate process to use some extra power. Looks cheaper than buying a
new system :-(

Has anyone done this, and if so are there records of how it worked or
why it didn't?

The actual reading of the articles is the next target, but the perl
filter takes way too much CPU!
--

-- 
bill davidsen <davidsen <at> tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
Permalink | Reply | 5 comments |
headers
Russ Allbery | 1 Dec 2003 18:49
Picon
Favicon
Gravatar

Re: Any reason not to split perl filtering?

bill davidsen <davidsen <at> tmr.com> writes:

> Single threaded innd is simply not keeping up with the incoming volume,
> and the obvious trouble spot is the perl filter. I have one CPU 98% used
> and the rest idle, is there any reason not to move filtering into a
> separate process to use some extra power. Looks cheaper than buying a
> new system :-(

> Has anyone done this, and if so are there records of how it worked or
> why it didn't?

> The actual reading of the articles is the next target, but the perl
> filter takes way too much CPU!

You take a fairly sizable I/O performance hit by doing this since you have
to start talking back and forth with a separate process (hence going
through the kernel).  Whether that's better or not depends a lot on the
system configuration.

The other obvious problem is that no one's submitted the code for it yet.

I have plans at some point to redo the filtering setup to provide a
cleaner API, which would make this much easier, but I have no idea when
I'll find the time.

--

-- 
Russ Allbery (rra <at> stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.
(Continue reading)

Permalink | Reply | 4 comments |
headers
Martin Mewes | 1 Dec 2003 19:29
Picon

[mail2news + mailpost]

Howdy folks,
didn't find something similar in the archive so I kindly ask my 
question:

I have recently installed an INN Version 2.3.2 on my Debian Woody.
I sucessfully can gat news with externeal hosts back and forth via 
newsfeed etc and I am able to gate MailingLists back and forth as 
well using mailpost/moderators.

Some of my users are annoyed by the fact that mailpost puts a 
user-defined "Reply-to:" from a Mail into a "X-Reply-to:" within the 
newsgroup.

So I tried to alter the "mailpost"-script in order not to touch this 
header to leave it alone. This was not a success ...

I googled a bit and found this diff

http://batleth.sapienti-sat.org/projects/news2mail/mailpost.diff

which seems to to the job, but I am unable to take action, because:
I do not know how to diff ;-)

Can anyone help me out or has a link or a ready altered "mailpost" I 
can use?

bis dahin / kind regards

Martin Mewes
(Continue reading)

Permalink | Reply | 2 comments |
headers
Martin Mewes | 1 Dec 2003 20:23
Picon

Re: [mail2news + mailpost]

Howdy folks,
Am Montag, 1. Dezember 2003 19:29 schrieb Martin Mewes:

> I do not know how to diff ;-)

Not necessary anymore :-)

	patch oldfile <diff

worked out perfectly :-D

bis dahin / kind regards

Martin Mewes

--

-- 
Kölsch fuer SysAdmins #005
Dat alles kom bloß durch en Hääd Bugs em OpenSSL. Do han ich mich
natörlich gefrog: patche oder bugfixe?
#######################################################################

-- Attached file included as plaintext by Ecartis --
-- Desc: signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQA/y5VQFZiIIrV6w6gRAp8nAJoDuACvPKMnY/uYOqa9hEwuITQ+9wCeOrE5
U6sbnegjiC4dI0zeP+jIToQ=
=CL3H
(Continue reading)

Permalink | Reply | 0 comments |
headers
bill davidsen | 2 Dec 2003 03:51

Re: Any reason not to split perl filtering?

In article <8765h08m3d.fsf <at> windlord.stanford.edu>,
Russ Allbery  <rra <at> stanford.edu> wrote:
| bill davidsen <davidsen <at> tmr.com> writes:
| 
| > Single threaded innd is simply not keeping up with the incoming volume,
| > and the obvious trouble spot is the perl filter. I have one CPU 98% used
| > and the rest idle, is there any reason not to move filtering into a
| > separate process to use some extra power. Looks cheaper than buying a
| > new system :-(
| 
| > Has anyone done this, and if so are there records of how it worked or
| > why it didn't?
| 
| > The actual reading of the articles is the next target, but the perl
| > filter takes way too much CPU!
| 
| You take a fairly sizable I/O performance hit by doing this since you have
| to start talking back and forth with a separate process (hence going
| through the kernel).  Whether that's better or not depends a lot on the
| system configuration.

Other news programs do this, but I was actually going to go pthreads. As
long as I use a lock to keep from running more than one at a time and
exploring how thread-safe the perl stuff might be, I think I can do what
I need.

My fallback is to use two processes sharing a memory segment large
enough to hold the max size article. That would cut the overhead down to
one memcopy, although not desirable.
|
(Continue reading)

Permalink | Reply | 3 comments |
headers
bill davidsen | 2 Dec 2003 04:01

Re: help needed in posted message administration

In article <20031128115901.48222.qmail <at> web13705.mail.yahoo.com>,
sidharth  <sidharth_bora <at> yahoo.com> wrote:
| sir,
|     Iam faced with the following probs any pointers
| will be greatly appreciated and obliging
| 
| the problem is the articles posted from various users
| has to be deleted say immediately i.e it shoud not be
| viewable from any reader.

My solution for this was to set the postinghost to another machine,
which does nothing but bless or curse the local posts and then submit
them back to the main machine. If you have a devious mind you can do
that on a single server running two copies of inn. I did it as proof of
concept, with the "posting host" listening on the loopback and another
port, but it adds complexity to save the need for a very small machine.

Anyway, that give you an alternative to putting the filtering in nnrpd
(a valid thing to do, of course). Of course if you just don't want to
see user posts you could block posting ;-)

Hint: it's much easier not to really do the post from a user until you
mean it than to try to undo the process later. The cancel process should
be considered advisory.
--

-- 
bill davidsen <davidsen <at> tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Russ Allbery | 2 Dec 2003 04:14
Picon
Favicon
Gravatar

Re: Any reason not to split perl filtering?

bill davidsen <davidsen <at> tmr.com> writes:
> Russ Allbery  <rra <at> stanford.edu> wrote:

> | You take a fairly sizable I/O performance hit by doing this since you
> | have to start talking back and forth with a separate process (hence
> | going through the kernel).  Whether that's better or not depends a lot
> | on the system configuration.

> Other news programs do this,

Yup, I know.  They take a sizable performance hit by doing so.  :)

> but I was actually going to go pthreads. As long as I use a lock to keep
> from running more than one at a time and exploring how thread-safe the
> perl stuff might be, I think I can do what I need.

Threading and Perl does not mix at all well.  Some people have used it,
but be prepared for a lot of instability.

> My fallback is to use two processes sharing a memory segment large
> enough to hold the max size article. That would cut the overhead down to
> one memcopy, although not desirable.

Actually, I think writing to a pipe is going to be of comparable speed to
using shared memory, if not faster.  But my information may be dated.

--

-- 
Russ Allbery (rra <at> stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Greg Andruk | 2 Dec 2003 05:17

Re: Any reason not to split perl filtering?

On Dec 2, 2003, at 2:51 AM, bill davidsen wrote:

> Other news programs do this, but I was actually going to go pthreads. 
> As
> long as I use a lock to keep from running more than one at a time and
> exploring how thread-safe the perl stuff might be, I think I can do 
> what
> I need.

Maybe I'm missing something, but unless the filter is spending a lot of 
time counting things between articles, isn't this still going to end up 
running pretty much sequentially?
Permalink | Reply | 1 comment |
headers
bill davidsen | 2 Dec 2003 21:52

Re: Any reason not to split perl filtering?

In article <6F100264-247E-11D8-A872-000A9589374A <at> meowing.net>,
Greg Andruk  <gja <at> meowing.net> wrote:
| On Dec 2, 2003, at 2:51 AM, bill davidsen wrote:
| 
| > Other news programs do this, but I was actually going to go pthreads. 
| > As
| > long as I use a lock to keep from running more than one at a time and
| > exploring how thread-safe the perl stuff might be, I think I can do 
| > what
| > I need.
| 
| Maybe I'm missing something, but unless the filter is spending a lot of 
| time counting things between articles, isn't this still going to end up 
| running pretty much sequentially?

If I can't make this machine run faster I have to disable filtering
completely. Given that as a choice, it's time to at least try to get
something running in parallel. The single threaded small memory design
is showing its age, going to faster and faster CPUs to use a single
thread design just doesn't scale. And my next target will be the input
logic, big selects suck and the throughput per sockets is poor as well.

I have one more idea, but that one is so odd I'm not ready to even
admit to having such a thought ;-)

--

-- 
bill davidsen <davidsen <at> tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Alex Moen | 2 Dec 2003 23:27
Favicon

Custom logging...

OK.  Here's what I want outta my logging:

1. I do not want to see my users accessing the newsgroups.  What I don't
know cannot hurt me.

2. I do want to see my users posting to groups.  Might need this info some
day.

Any ideas?  I have looked through the docs and have come up with nothing.

Thanks,

Alex
Permalink | Reply | 2 comments |

Gmane