headers
Steinar Bang | 2 Sep 1997 14:01
Picon

opening speed for nnml groups

I haven't done any timing of it yet, but it seems like opening nnml
groups get noticably slower, as the groups get bigger (with more
ticked or read articles in them).

I was sort of idly wondering where the speed hit was?  Is it in the
parsing of the .overview file?

I remember that slow exit on big or sparse nnml group, was a problem
when you did expiry on exiting a group, so I went for demonical
expiry, as well as demonical fetch and sort.

If the reading and parsing of the .overview file is the problem, would
it be an idea to cache the parsed info between visits? And update it
during fetch and sort operations?

Or is that too expensive memory-wise?
Permalink | Reply | 15 comments |
headers
Niels Olof Bouvin | 2 Sep 1997 15:04
Picon
Picon

Caps?


How are things progressing on the Caps front? Summer has now
turned to autumn in Denmark, and I would love something to
protect me from the rain.

Cheers
	Niels Olof

--

-- 
Niels Olof Bouvin http://www.daimi.aau.dk/~bouvin/ Voice: (+45)89423274 \ /
In omnibus requiem quaesivi, et nusquam inveni nisi in angulo cum libro (")
MSc Academic Associate  <at>  Dep. of Computer Science Aarhus University Denmark
Permalink | Reply | 2 comments |
headers
Lars Balker Rasmussen | 2 Sep 1997 15:37
Picon
Favicon

Re: Caps?

Niels Olof Bouvin <bouvin <at> daimi.aau.dk> writes:
> How are things progressing on the Caps front? Summer has now
> turned to autumn in Denmark, 

(Yesterday, at noon, as scheduled.)

> and I would love something to protect me from the rain.

Personally, I hate getting snow behind my glasses when I ride my bike,
so I just hope we get it before winter sets in.
--

-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr <at> mjolner.dk
Permalink | Reply | 1 comment |
headers
Danny Siu | 2 Sep 1997 18:53
X-Face
Picon
Favicon

Re: Caps?

Lars Balker Rasmussen writes:

  Lars> Niels Olof Bouvin <bouvin <at> daimi.aau.dk> writes:
  >> How are things progressing on the Caps front? Summer has now turned to
  >> autumn in Denmark,

  Lars> (Yesterday, at noon, as scheduled.)

  >> and I would love something to protect me from the rain.

  Lars> Personally, I hate getting snow behind my glasses when I ride my
  Lars> bike, so I just hope we get it before winter sets in.  -- Lars
  Lars> Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
  Lars> lbr <at> mjolner.dk

I don't think anyone will ever see Lars again after he grabbed all of our
$$$.  Well, maybe someone will bump into him in Maui enjoying his vacation.

Remember he specifically wanted cash for the caps?  He must have planed this
a long time ago.  The previous T-shirt and sweat-shirt sale were to gain our
trust....  I bet this time there must be lots of us paid for the caps....

:-)

--

-- 
                                                                ^..^
 '/..\"                                                        ( oo )  )~
m( oo )m                                                         ,,  ,,
/=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\
|  Danny Dick-Fung Siu                http://www-leland.stanford.edu/~dsiu |
(Continue reading)

Permalink | Reply | 0 comments |
headers
Steve Folkes | 3 Sep 1997 15:05
Picon
Picon

Re: opening speed for nnml groups

Steinar Bang <sb <at> metis.no> writes:

>>> I haven't done any timing of it yet, but it seems like opening nnml
>>> groups get noticably slower, as the groups get bigger (with more
>>> ticked or read articles in them).

I've noticed this to.

>>> I was sort of idly wondering where the speed hit was?  Is it in the
>>> parsing of the .overview file?

It may be in the Unix filesystem itself.  I had a problem
with our newsfeed a few years ago that put about 10,000
files in one directory; it took over a week to remove them
all (using "rm -rf").

Some of the NNML groups that I use for mailing lists have
become quite large (over 1,000 messages) and have started to
slow down.  They are also slow when I use "ls" on the
directory that stores them though.

I started to look at changing the file naming code for the
NNML method so that it would attempt to spread files across
several directories, but I did not have the time to get
anywhere with it.

			- Steve.
--

-- 
Steve Folkes, N137, DERA Malvern, St Andrews Rd, Malvern, Worcs. WR14 3PS. UK.
   Tel: +44 1684 895649; Fax: +44 1684 894303; Email: smf <at> rivers.dra.hmg.gb
(Continue reading)

Permalink | Reply | 13 comments |
headers
Lars Balker Rasmussen | 3 Sep 1997 17:01
Picon
Favicon

Re: opening speed for nnml groups

Steve Folkes <smf <at> rivers.dra.hmg.gb> writes:
> It may be in the Unix filesystem itself.  I had a problem
> with our newsfeed a few years ago that put about 10,000
> files in one directory; it took over a week to remove them
> all (using "rm -rf").

What Unix would that be?

% mkdir test
% perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
% time rm -rf test
0.60u 11.18s 1:55.40 10.2%
% uname -a
SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc

> Some of the NNML groups that I use for mailing lists have
> become quite large (over 1,000 messages) and have started to
> slow down.  They are also slow when I use "ls" on the
> directory that stores them though.

ls is primarily slow because it has to sort the files.

But of course, the Unix filesystem gets slower as more files are in a
directory, but it's not _that_ bad.
--

-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr <at> mjolner.dk
Permalink | Reply | 12 comments |
headers
Matt Simmons | 3 Sep 1997 20:18
Picon
Favicon

Random signatures

I didn't see anything about being able to do random signatures in the
gnus manual, so I wrote something to do it.  It's rather short, so
I'll append it below.

This is my first crack at elisp coding - any comments, critiques, "you 
shouldn't use this function - use this one instead"s, etc would be
appeciated.

Thanks

Matt

--

-- 
     Matt Simmons  -  simmonmt <at> acm.org  -  http://www.netcom.com/~simmonmt
      One of the greatest labour-saving inventions of today is tomorrow. 
                                                       --Vincent T. Foss
Attachment (random-sig.el): application/octet-stream, 2774 bytes
Permalink | Reply | 0 comments |
headers
Russ Allbery | 3 Sep 1997 23:26
Picon
Favicon
Gravatar

Re: opening speed for nnml groups

Lars Balker Rasmussen <lbr <at> mjolner.dk> writes:
> Steve Folkes <smf <at> rivers.dra.hmg.gb> writes:

>> It may be in the Unix filesystem itself.  I had a problem with our
>> newsfeed a few years ago that put about 10,000 files in one directory;
>> it took over a week to remove them all (using "rm -rf").

> What Unix would that be?

> % mkdir test
> % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
> % time rm -rf test
> 0.60u 11.18s 1:55.40 10.2%
> % uname -a
> SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc

A week is a bit long, but when I cut back on the expire time for
control.cancel on our news server from ten days to half a day, it took
fastrm (which is a good bit faster than rm -rf under most circumstances)
the majority of a day to remove the files in the directory, and that was
an UltraSPARC 2 on a RAID array.  The directory itself, not including its
contents, was over 1MB.

> But of course, the Unix filesystem gets slower as more files are in a
> directory, but it's not _that_ bad.

Unless I'm mistaken, Unix stores a directory as a linked list of files.
Each time you open or delete a file, it has to traverse the linked list to
find it, and each time you create a file, it has to traverse the linked
list to find a hole.  It should quickly become clear from that why large
(Continue reading)

Permalink | Reply | 0 comments |
headers
Ken Raeburn | 3 Sep 1997 22:33
Picon

Re: opening speed for nnml groups

Steinar Bang <sb <at> metis.no> writes:

> I haven't done any timing of it yet, but it seems like opening nnml
> groups get noticably slower, as the groups get bigger (with more
> ticked or read articles in them).
> 
> I was sort of idly wondering where the speed hit was?  Is it in the
> parsing of the .overview file?

(I forget who pointed this out to me...)
If you use the message date in the summary line, that's probably a big
piece of it.  See my messages of some time ago about regexp ordering
in timezone-parse-date.  Getting the order right (since all email I
tried matched one pattern, and it wasn't the first one) saves a large
fraction of the time spent in that routine.

In a test I just did with ELP and a mail group of some 7800 messages,
timezone-parse-date (probably without fixed regexp ordering, I've been
lame) took 46.9 seconds out of the total of 194.3.

gnus-summary-read-group-1 took 194.3 seconds
  gnus-select-newsgroup took 32
    gnus-get-newsgroup-headers-xover took 29.6 (nov-parse-line done inline)
  gnus-summary-prepare took 152
    gnus-make-threads took 4
    gnus-sort-threads took 12.8
    gnus-summary-prepare-threads took 135.6
      formatting and inserting summary line took ...??

Since my line-format-spec uses the date, let's be generous and say
(Continue reading)

Permalink | Reply | 0 comments |
headers
Mark Eichin | 3 Sep 1997 21:45
Picon

Re: opening speed for nnml groups

Indeed, 
+% mkdir test
+% time perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
1.490u 72.550s 1:31.65 80.7%    0+0k 0+0io 287pf+0w
+% time sync
0.000u 0.040s 0:00.21 19.0%     0+0k 0+0io 21pf+0w
+% time rm -rf test
0.200u 8.120s 0:09.26 89.8%     0+0k 0+0io 86pf+0w

But that's a p133 running linux, run of the mill 2G SCSI disks.

> Some of the NNML groups that I use for mailing lists have
> become quite large (over 1,000 messages) and have started to

Heh -- my mail.misc has 16000 files, mail.personal has 6500... and I
usually enter them with 33 or 99 (whatever is easiest to type) to read
new stuff, or maybe 300 or so if I want to search back a bit.  *That*
isn't slow... but when I hit $ to refile some spam, from mail.misc to
mail.spam (which only has 600 messages at the moment) it takes as long
as 5 to 10 seconds to recover.  I'd guess most of that has to do with
rewriting the 3.5M .overview file in mail.misc -- having 64M of RAM
does help, especially with linux aggressive caching...

Needless to say I'm pushing gnus quite a bit here.  All I really need
to do is write some scripts to refile all of mail.misc during 1996
into archive.1996.mail.misc, or something like that...
Permalink | Reply | 0 comments |

Gmane