Re: self-paging

On Tue, 2005-11-29 at 20:54 +0100, ness wrote:

> I actually wanted to express: every visible resource revokation on an 
> application A that might be caused by an application B where B isn't 
> allowed to talk to A seems to be covert channel.
> 
> > [...]

This is correct.

Re: self-paging

I apologize for my delay in responding on this thread -- I've been
dealing with impending final exams and some Coyotos contract issues.

Concerning self-paging, I think we need to break the discussion down
into several distinct issues:

  1. Hard vs. soft page frame commitments
  2. Application control over pageout selection
  3. Notifying the application about changes in contract

The first point is simple. I just wanted to point out that we are
talking here about negotiated, changing contracts, so we are discussing
only "soft" contracts.

The point about application pageout control is that this is separate
from the question of notification. There is a paper design in EROS for
dealing with this that can be summarized as follows:

  1. There is a first class "frame pool" abstraction. A frame
     pool defines some number of frames that have been guaranteed
     to the application.
  2. Given a frame pool, an application can associate regions
     of the virtual address space with that pool.
  3. Whenever a page fault occurs, the page frame is allocated
     against the quota imposed by the frame pool.
  4. If the frame pool is fully subscribed, some existing frame
     within that frame pool is evicted (policy for this not yet
     defined, but conceptually: FIFO)

The point: you don't need to notify the application of anything in order

Coyotos, EROS, and Hurd

Greetings all.  I have been lurking on the coyotos email list for a
while, and before that I was a lurker on the eros list.  All quiet -
and then I checked l4-hurd .

I have read some, but by no means all of the vast and fascinating
discussions of the last couple of months.  I just wanted to add two
remarks:

a)  I was and am tremendously excited by the prospect of a Hurd tuned
to work with/on Coyotos.  I have long been impressed by the new and yet
practical ideas in both projects, and would be very excited to see the
practical results of such a system.

b)  I was very much impressed by Dr. Shapiro's comment on the approach
being taken with Coyotos being a design first and then write approach,
and the contrast between that and normal open source development
methodology.  It caught my eye because I am involved, in a small way,
with the Axiom computer algebra system project.  That project is also
applying a different methodology that shifts the focus away from
immediate coding - literate programming.  In essense, the goal is to
write a document describing the idea, design considerations, and the
implementation, with the code then following naturally from the design.
 I doubt Coyotos will be a literate program in the Knuth sense, but I
am impressed by Dr. Shapiro's insight on the fundamental nature of
development methodologies and their potential consequences.

I guess, in a way, I would like to suggest the following:  Linux and
FreeBSD have come to dominate the open source OS scene.  Because of the
momentum behind these systems, the wide experience and massive testing
history behind the code, and the corporate inertia that has formed

Re: self-paging

On Thu, Dec 01, 2005 at 07:30:28PM -0500, Jonathan S. Shapiro wrote:
> I apologize for my delay in responding on this thread -- I've been
> dealing with impending final exams and some Coyotos contract issues.

I missed you.   I hope everything went well with the exams and the
contracts.

> Concerning self-paging, I think we need to break the discussion down
> into several distinct issues:
> 
>   1. Hard vs. soft page frame commitments
>   2. Application control over pageout selection
>   3. Notifying the application about changes in contract
> 
> The first point is simple. I just wanted to point out that we are
> talking here about negotiated, changing contracts, so we are discussing
> only "soft" contracts.

Right.

> The point: you don't need to notify the application of anything in order
> to let the application define residency requirements.

Well, it's all fine when the quota decreases.  A page will be swapped out, and
the process will get a page fault at some point.  In response to that it can
reoptimise its strategy (how much cache to have, for example).

However, this doesn't work when the quota increases.  Assume a process which
wants as much cache as possible, but only in physical memory.  When it knows
how much it can use, it will allocate that and use it.  As long as it isn't

Re: self-paging

On Fri, 2005-12-02 at 15:52 +0100, Bas Wijnen wrote:
> > The point: you don't need to notify the application of anything in order
> > to let the application define residency requirements.
> 
> Well, it's all fine when the quota decreases.  A page will be swapped out, and
> the process will get a page fault at some point.  In response to that it can
> reoptimise its strategy (how much cache to have, for example).
> 
> However, this doesn't work when the quota increases.  Assume a process which
> wants as much cache as possible, but only in physical memory.  When it knows
> how much it can use, it will allocate that and use it.  As long as it isn't
> notified, it doesn't start using more, even if that would be more efficient.
> Of course it could poll for its quota every now and then, but that doesn't
> seem right.  It just adds work to the process while all the information to
> notify it is available.

I think I made my statement badly. What I meant to say was: you don't
need to notify the application in order to let the application specify
how pages compete for residency.

That is: expressing preference about what to page out doesn't require
notification. You don't need notification until your application wants
to do adaptive overall residency.

> We always need to consider channels, except with hard page frame commitments.
> 
> >   2. Soft reservations are adjusted only in response to user action.
> >      For example, when your window ceases to be in front, we can tell
> >      you safely that your CPU contract and memory contract are reduced.
> 

Re: self-paging

On Fri, Dec 02, 2005 at 01:41:53PM -0500, Jonathan S. Shapiro wrote:
> I think I made my statement badly. What I meant to say was: you don't
> need to notify the application in order to let the application specify
> how pages compete for residency.
> 
> That is: expressing preference about what to page out doesn't require
> notification. You don't need notification until your application wants
> to do adaptive overall residency.

Ok, but it seems nice to allow that.  Therefore I think the notifications are
a good idea.

> > >   2. Soft reservations are adjusted only in response to user action.
> > >      For example, when your window ceases to be in front, we can tell
> > >      you safely that your CPU contract and memory contract are reduced.
> > 
> > While this would not be an acceptable policy for me, I agree that the idea
> > can work.  I'm not sure if I'd like it though.  I think I prefer to have
> > settings for the quota which can be bound by the user (so dynamic values,
> > but limited to a range).
> 
> Two examples to consider:
> 
> 1. Video. Front window closes, resources get rotated so that the new
> front video window now gets "frontmost" resources.
> 
> 2. Tuning knobs: app gives jaggy video, user turns a quality knob.

What I mean by settings for the user really is 2, except that 2 seems very
window-oriented.  The point I'm making is that the user may want something to

Re: self-paging

On Sat, 2005-12-03 at 09:30 +0100, Bas Wijnen wrote:
> On Fri, Dec 02, 2005 at 01:41:53PM -0500, Jonathan S. Shapiro wrote:
> > 
> > 2. Tuning knobs: app gives jaggy video, user turns a quality knob.
> 
> What I mean by settings for the user really is 2, except that 2 seems very
> window-oriented.  The point I'm making is that the user may want something to
> get resources which isn't in the foreground.  For real-time video this is
> obviously not the case, but computationally intensive tasks are often
> uninteresting until they're done.  Consider a fractal generator.  If you want
> generation to be fast, you minimize the window so it doesn't waste time on
> drawing intermediate states.  It is not desirable if that means the generation
> is even slower because much of its resources are revoked.

I don't agree. First, there is no reason to imagine that tuning knobs
are limited to visible windows, so I don't really agree with that
concern. The user has X amount of real-time slice to divide up and give
out, and has an interface that lets them decide who gets it.

Second, I don't agree with the background fractal example. If the user
puts the fractal generator in the background and then starts a movie,
they are saying "my attention is on the movie". I think that the
principle here is that real time guarantees have to do with responding
in a timely fashion where the focus of attention is. The background
fractal problem may be a motivator for a high priority, but it is *not*
a motivator for a real time guarantee.

Yes, I can see that we may construct other scenarios that will make
real-time background schedules motivated. Fair-share scheduling comes
immediately to mind, but note that fair-share scheduling was in

Re: self-paging

On reflection, There may be a way to give Bas what he seems to want.

We've agreed that notification is a separate problem, so I'm not talking
about that right now.

W.r.t. residency pools, we can potentially separate two things:

   1. Specifying the group of pages that compete with each other. 
      This is the part that the EROS frame pool deals with well.

   2. Specifying which page to shoot down when the pool is full and 
      a new page must come in.

I provisionally believe that if we want to use any sort of LRU, the
gathering of age data must be performed by the kernel. Since the kernel
needs to collect this data anyway, I think that *one* of the policies we
should have is "LRU w.r.t. the rest of the pool" and that this should be
directly implemented by the kernel.

But it is also possible to have a protocol in which the kernel turns to
a "keeper" associated with the pool. Basically, the kernel upcalls the
keeper (very much like fault handling) and says "make some room in this
pool".

The keeper can then shoot down member frames explicitly by naming their
virtual address, using a call of the form

	pool->unbind(va);

One part of this differs from what Bas seems to be proposing: we want to

Re: self-paging

On Sun, Dec 04, 2005 at 01:49:34PM -0500, Jonathan S. Shapiro wrote:
> On Sat, 2005-12-03 at 09:30 +0100, Bas Wijnen wrote:
> > On Fri, Dec 02, 2005 at 01:41:53PM -0500, Jonathan S. Shapiro wrote:
> > > 
> > > 2. Tuning knobs: app gives jaggy video, user turns a quality knob.
> > 
> > What I mean by settings for the user really is 2, except that 2 seems very
> > window-oriented.  The point I'm making is that the user may want something
> > to get resources which isn't in the foreground.  For real-time video this
> > is obviously not the case, but computationally intensive tasks are often
> > uninteresting until they're done.  Consider a fractal generator.  If you
> > want generation to be fast, you minimize the window so it doesn't waste
> > time on drawing intermediate states.  It is not desirable if that means
> > the generation is even slower because much of its resources are revoked.
> 
> I don't agree. First, there is no reason to imagine that tuning knobs are
> limited to visible windows, so I don't really agree with that concern.

Ok, in that case my "settings" really is your option 2.

> Second, I don't agree with the background fractal example. If the user puts
> the fractal generator in the background and then starts a movie, they are
> saying "my attention is on the movie". I think that the principle here is
> that real time guarantees have to do with responding in a timely fashion
> where the focus of attention is. The background fractal problem may be a
> motivator for a high priority, but it is *not* a motivator for a real time
> guarantee.

Oh, I wasn't talking about real-time guarantees.  I was just saying that the
user should have the option to specify that one process is the most important

How come DRM would provide anything?

Hi,

I have just a simple question that hit me when reading the discussions
about TC implementation decision: some of us try to design µ-kernels or
OSes in a reliable way; we are taking great care to design Hurd in a way
that it's properties would be based on more or less formal proofs, or at
least scientific results.

But as we talk about what TC could add in the field of privacy to Hurd
or any other system, did we just forget that TC chips are black boxes?
How could we use TC chips to achieve privacy and security in any way
although we have no clue about what is in the inside?

Curiously,
Nowhere man
--

-- 
nowhere.man <at> levallois.eu.org
OpenPGP 0xD9D50D8A

_______________________________________________
L4-hurd mailing list
L4-hurd <at> gnu.org
http://lists.gnu.org/mailman/listinfo/l4-hurd