headers
Dave Warren | 1 Feb 2004 01:09

Re: Re: Another Overload problem Update

Jeff Snavely wrote:
> --- In bnr1 <at> yahoogroups.com, "Helen McGee" <helwitch <at> v...> wrote:
>> On this note, Jeff, will there be the ability to set a default
>> queue sort order anytime soon? You did add this ability for the
>> groups, would it be difficult to add it for the queue? Because I
>> like my queue sorted score(1) data(2). And it bugs me no end when
>> it gets out of that order, which it does often.
>
> I'm not sure you realize what you are asking.  The queue is different
> from a Message-List page because it is constantly changing.  If you
> notice, each time you add something to the queue, it puts the new
> items at the bottom and forgets the current sort.  In order to keep
> the queue arranged in order, it would have to resort the queue each
> time it adds a new item.  This includes all of the individual
> commands involved in a header refresh.  If you remember the way BNR1
> used to go berserk hopping around all over the list and forgetting
> your selection when you tried to mark things for download while it
> was still downloading headers, the queue would become even more like
> that than it already is.  Are you sure you want this?

How about resorting it whenever the queue tab is selected, but not while
you're actually using the window?

--

-- 
Dave Warren,
 Email Address:  dave.warren <at> devilsplayground.net
 Cell: (403) 371-3470         Fax: (403) 371-3471
 Toll free: (888) 371-3470 Vonage: (817) 886-0860
 ICQ: 17848192  AIM: devilspgd  Yahoo!: devilspgd
 MSN/PASSPORT:   dave.warren <at> devilsplayground.net
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jeff Snavely | 1 Feb 2004 02:14
Picon
Favicon

Re: Suggishness + interesting server setup method

--- In bnr1 <at> yahoogroups.com, "Dave Warren" <dave.warren <at> d...> wrote:
> How about if a thread doesn't find any work, it won't look again
> for another 2-5 seconds.  If a thread just completed a job, then it
> will look for new work ASAP (as is the current system)

The scan is done hierarchically by priority.  There might be ways to reduce the frequency of the scans when
every connection is either idle or already busy doing something, but trying to have individual wait
different amounts of time will probably break the logic and end up with things like lower priority server
downloading articles during the 2-5 seconds that the higher priority server is being skipped because it
didn't have anything to do during the last scan.

> 
> Another consideration, if a server has multiple connections, do you
> need to scan the queue for every connection?
> 
> If a server has multiple connections, the algorthim could check the
> previous connections and if they're idle, then assume there is no
> work to do.  If #2 is idle, it would check #1 and if #1 is idle, it
> wouldn't do anything, if #1 is active then it would scan the
> queue.  If #3 is idle, it would check #1 and #2 and if EITHER are
> idle it wouldn't do anything, if #1 and #2 are both active then it
> would scan the queue.

It's already smart enough to something like that.  It does the scan from the server level, not from each
individual connection in that server.  It involves factors such as whether or not the folder containing
that server is already using it's max number of connections, whether or ot this server is already using
it's max number of connections, how many connections are already busy, etc.  It scans just enough of the
queue to decide whether connecting another connection to this server would be able to do something at that
moment.  Also, if this server has any idle connections, whether disconnecting one of them will allow a
connection to another server to do something.
(Continue reading)

Permalink | Reply | 3 comments |
headers
frovingslosh | 1 Feb 2004 02:57
Favicon

Re: Suggishness + interesting server setup method

--- In bnr1 <at> yahoogroups.com, "Jeff Snavely" <jsnavely.geo <at> y...> wrote:
>...will probably break the logic and end up 
with things like lower priority server downloading articles during
the 2-5 seconds that the higher priority server is being skipped 
because it didn't have anything to do during the last scan.

That really doesn't sound like a real problem. If the higher priority 
server did pause for a few seconds then it apparently had nothing to 
do. If a lower priority process did find something to do at that
point it likely was a valid find. Again, marking articles as wanted 
could wake up the thread if needed, but it doesn't seem like it really
would be. As long as the higher priority connection is connected and 
valid, a lower priority connection wouldn't take stuff targeted for 
"any", would it?

Another way to look at this is that even scanning every .5 seconds is 
a fair size window in computer time, but we never see lower priority 
connections grab stuff they shouldn't in that 1/2 second, do we? So
if  1/2 second is safe, the a few seconds should be. And if it's not, 
then it can be made safe.

Another approach one might want to take is to let the delta t between 
scans change on some huristic basis. As long as the scan finds 
something to do, the delta t is set to .5 seconds. If it makes a pass 
and finds nothing, it's delta t grows by some value (perhaps 1 
second), but never to more than one half of the disconnect timer. In 
this way active connections stay active, those that just went
inactive check back frequently, but those that are not being used only 
check if there is new work for them on an infrequent basis.
(Continue reading)

Permalink | Reply | 2 comments |
headers
Todd Vierling | 1 Feb 2004 03:17
Gravatar

Re: Re: Suggishness + interesting server setup method

On Sat, 31 Jan 2004, Jeff Snavely wrote:

: It does start over at the head of the queue for each connection that it
: evaluates and will likely stay that way because it is way too complex as
: it is to have it try to handle all connections at once.  Also, I don't
: think it will save much,

Then I'd like to know where all my CPU has gone.

: and could make matters a lot worse.

Aw, c'mon, this is a first-year CS student problem.  8-)  To wit:

Let's describe my real-world scenario more accurately.  Let's say there are
*five* servers, all with differing priorities.  I'll name them by their
priority to make it easier to read.  There are articles in the queue such
that three of the connections should be awakened.

Connection 125: can handle first item in queue, so skips rest of items.
Connection 50: can handle item 101; skips items after that.
Connection 0: scans entire queue and finds nothing it can do.
Connection -50: can handle item 6001; skips items after that.
Connection -100: scans entire queue and finds nothing it can do.

Inner loop: 26,103 iterations

: Your Suggestion: (loop over queue items, then over connections)
: Queue item 1: first connection can handle this item, second cannot
: Queue item 2: first connection can handle this item, second cannot
(Continue reading)

Permalink | Reply | 4 comments |
headers
Jeff Snavely | 1 Feb 2004 03:41
Picon
Favicon

Re: Suggishness + interesting server setup method

--- In bnr1 <at> yahoogroups.com, "frovingslosh" <fooyahoo.19.frothingslosh <at> s...> wrote:
> That really doesn't sound like a real problem. If the higher
> priority server did pause for a few seconds then it apparently had
> nothing to do. If a lower priority process did find something to do
> at that point it likely was a valid find. Again, marking articles
> as wanted could wake up the thread if needed, but it doesn't seem
> like it really would be. As long as the higher priority connection
> is connected and valid, a lower priority connection wouldn't take
> stuff targeted for "any", would it?

If someone marked something to download during the time that the server was waiting, or another higher
priority connection failed to download something, or the user changes priorities, or a HD filled up, or
other reasons cause the server to suddenly be able to do something when it wasn't able to before.  The lower
priority server shouldn't start downloading if the higher priority is actually connected, but it might
if the higher priority connection was not connected yet.  Probably a better example for a break in the logic
would be something like...

t+0: 1 item in queue gets assigned to Server A, Server B is not-connected the connection manager decides to
wait 5 seconds before checking again.  

t+1: Server A still downloading, Server B skipped (4 seconds before next scan)

t+2: Server A finishes downloading article, nothing to do so decides to wait 5 seconds before checking
again, Server B skipped (3 seconds before next scan)

t+3: User queues 1 more article, Server A skipped (4 seconds), Server B skipped (2 seconds)

t+4: 1 item in queue, Server A skipped (3 seconds), Server B skipped (1 seconds)

t+5: 1 item in queue, Server A skipped (2 seconds), Server B finally checks again, has same priority as
(Continue reading)

Permalink | Reply | 1 comment |
headers
Jeff Snavely | 1 Feb 2004 03:51
Picon
Favicon

Re: Suggishness + interesting server setup method

--- In bnr1 <at> yahoogroups.com, Todd Vierling <tv <at> d...> wrote:
> Connection 125: can handle first item in queue, so skips rest of
> items.
> Connection 50: can handle item 101; skips items after that.
> Connection 0: scans entire queue and finds nothing it can do.
> Connection -50: can handle item 6001; skips items after that.
> Connection -100: scans entire queue and finds nothing it can do.
> 
> Inner loop: 26,103 iterations
> 
> : Your Suggestion: (loop over queue items, then over connections)
> : Queue item 1: first connection can handle this item, second cannot
> : Queue item 2: first connection can handle this item, second cannot
> 
> No, that description is not right.  Rather:
> 
> List of idle connections: { 125, 50, 0, -50, -100 }
> Single Pass:
> 
> Queue item 1: can be handled by Connection 125.  Kick off
> Connection 125, and remove it from the list.  List is now { 50, 0, 
> -50, -100 }
> 
> Queue items 2-100: can be handled by some Connection > 50; skip.
> 
> Queue item 101: can be handled by Connection 50.  Kick off
> Connection 50, and remove it from the list.  List is now { 0, -50, 
> -100 }
> 
> Queue items 102-6000: can be handled by some Connection >0; skip.
(Continue reading)

Permalink | Reply | 3 comments |
headers
Chris | 1 Feb 2004 04:03
Picon
Favicon

Disappearing Newsgroup List

I've been able to duplicate this several times with 12.7.  When a server is 
removed the whole newsgroup list disappears.  Below is the error.

01/31/2004 08:55:32 PM (0.12.7.Linux): An unhandled exception  "List index 
out of bounds (5)" was caught!
StackDump not supported on Linux!

Chris

-- 
  Regards
  Chris
  A 100% Microsoft free computer
  Registered Linux User 283774 http://counter.li.org
  8:58pm  up 6 days, 21:16,  4 users,  load average: 0.48, 0.61, 1.02

Important Addresses:
Post message: bnr1 <at> yahoogroups.com
Subscribe:  bnr1-subscribe <at> yahoogroups.com
Unsubscribe:  bnr1-unsubscribe <at> yahoogroups.com
List owner:  bnr1-owner <at> yahoogroups.com
Group homepage: http://groups.yahoo.com/group/bnr1
BNR homepage: http://www.bnr2.org/ 

Yahoo! Groups Links

To visit your group on the web, go to:
 http://groups.yahoo.com/group/bnr1/

To unsubscribe from this group, send an email to:
(Continue reading)

Permalink | Reply | 1 comment |
headers
Todd Vierling | 1 Feb 2004 04:05
Gravatar

Re: Re: Suggishness + interesting server setup method

On Sun, 1 Feb 2004, Jeff Snavely wrote:

: You are assuming that the queue happens to be sorted based on the highest
: priority server that can handle it.

No, I'm not.  How do you come about that conclusion?  Here's an even more
detailed flow description:

: > Queue item 1: can be handled by Connection 125.  Kick off
: > Connection 125, and remove it from the list.  List is now { 50, 0,
: > -50, -100 }

This is determined by looking at the article availability for item 1.
Since the highest priority server capable of handling this article is in the
list of odle connections, kick off its thread.  This connection is removed
from the idle list since it will now do its own scan to pick an article to
download.

: > Queue items 2-100: can be handled by some Connection > 50; skip.

So, when looking at these articles, we see that the higest priority server
capable of handling it is 125, which is not in our idle list.  So we skip
these articles.

: > Queue item 101: can be handled by Connection 50.  Kick off
: > Connection 50, and remove it from the list.  List is now { 0, -50,
: > -100 }

But this article's highest priority server is Connection 50.  Handle this
like we did for article 1 -- kick off the thread and remove it from the
(Continue reading)

Permalink | Reply | 2 comments |
headers
Jeff Snavely | 1 Feb 2004 05:10
Picon
Favicon

Re: Suggishness + interesting server setup method

--- In bnr1 <at> yahoogroups.com, Todd Vierling <tv <at> d...> wrote:
> On Sun, 1 Feb 2004, Jeff Snavely wrote:
> 
> : You are assuming that the queue happens to be sorted based on the
> : highest priority server that can handle it.
> 
> No, I'm not.  How do you come about that conclusion?  Here's an
> even more detailed flow description:

That was how it looked from the way example was arranged to just happen to match each priority server in the
order by priority.  You probably should have mixed it up a bit, such as having the item 3 only be available on
the lowest priority server.

Also, your iteration counts you gave is just for the outer loop (iterating over the queue items).  For each
queue item, you need not only check against anywhere from 1 to all idle connections to determine the
highest priority idle connection that can handle this item, if any, but you also have to check against
anywhere from 1 to all non-idle higher priority connections to determine if any of those could possibly
handle this item, and therefore should not use the lower priority idle server.  Therefore, the inner loop
is not a flat 1 iteraton per each iteration of the outer loop.  If you have many servers and every queue item is
only available on the lowest priority server than it would have many iterations to process to determine that.

for example...
> : > Queue item 6001: can be handled by Connection -50.  Kick off
> : > Conenction -50, and remove it from the list.  List is now { 0,
> : > -100 }
> 
> This one's highest priority capable server is -50 (it's not even
> available on Connection 0).  So again, we kick off the thread and
> remove -50 from the idle connection list.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Todd Vierling | 1 Feb 2004 06:26
Gravatar

Re: Re: Suggishness + interesting server setup method

On Sun, 1 Feb 2004, Jeff Snavely wrote:

: The only way that you would know that the higher priority server that can
: handle item 6001 is by first checking ALL of the higher priority servers
: (both idle and busy) and discovering that this article is not available on
: any of those servers.

OK, so how is it done today?

You must determine somehow that a server is or isn't the best for each queue
item.  This would have to be done by evaluating the server article pointers
for that item, and doing one of:

1. skipping the article if a higher priority server is found in the pointer
   list (walking only a subset of the pointers),

2. skipping the article if the server is not found in the pointer list (by
   walking the entire set of the pointers),

3. kicking off the download if the current server *is* the best one in the
   pointer list (by walking the entire set of the pointers).

The only non-worst-case option is 1.  Both 2 and 3 require exhausting the
entire server article pointer list for a given article, and 2 is likely to
happen on most articles nearly every time through the loop, if there is a
large number of lower-priority idle servers.

If I've missed something here, please do correct me.  (I'm using the term
"server article pointers" to refer to the article->articlenumber*server
tuple mappings for each article in the queue.)
(Continue reading)

Permalink | Reply | 0 comments |

Gmane