headers
Thomas Leonard | 2 May 2005 10:06
Picon
Favicon

Re: Hashdrop

On Fri, Apr 29, 2005 at 05:02:19PM +0200, Joachim Kupke wrote:
> Thomas Leonard:
> 
> > One thing that has been bothering me is that it's not easy to see where
> > something in the hash directory came from. Specifically, it's useful to
> > know which user(s) require it, and have each user have a way to find out
> > why the need it (and whether they still do).
> > 
> > I was thinking of some scheme combining quotas with reference counting.
> > The system should be able to remove directories when no user needs them.
[...]
> Besides, would a quota pertain to disk usage only (with storage costs
> becoming almost negligible...)? Or would it pertain to network
> downloads? The latter could still seem unfair when two users know pretty
> well that both of them need some package, but one of them has to pay for
> its download.

I don't think network use needs to have a quota. In the injector's case,
users download the archives themselves, and then submit the unpacked
archive to hashdrop, so there's no need for the system to do anything
special.

I'm mainly concerned about the case on a shared system where a single
user has used up all the disk space. The sysadmin should at least be
able to find out who's responsible and ask them to remove their stuff.

> > You'll need something extra to avoid conflicts (TIMESTAMP-SEQUENCE?),
> > but that's easy to do.
> 
> You mean for multiple installations per second? I'd prefer TIMESTAMP-PID
(Continue reading)

Permalink | Reply | 14 comments |
headers
Thomas Leonard | 2 May 2005 10:09
Picon
Favicon

Re: Two suggestions

[ splitting into two threads ]

On Fri, Apr 29, 2005 at 05:02:19PM +0200, Joachim Kupke wrote:
[...]
> To make myself clearer; what I was talking about is this: We could
> "mount -t autofs" the /uri/0install directory; the existing autofs/amd
> infrastructure on Linux and most other unices will then give us a file
> handle where lookup requests for /uri/0install/anything will be
> reported. When we receive such a request, we could download (all of) the
> "anything" site, extract it to /var/cache/zero-install/anything
> (including symlinks, excluding "..." files), and report back to the
> kernel that it should "mount --bind" the
> /var/cache/zero-install/anything/ directory to /uri/0install/anything/.

OK, but you'll need to rearrange things. Downloading a whole site isn't
useful at the moment, because you get all versions of each program.

However, if a site contains nothing but little scripts to run 0launch,
then this should work just fine.

--

-- 
Dr Thomas Leonard		http://rox.sourceforge.net
GPG: 9242 9807 C985 3C07 44A6  8B9A AE07 8280 59A5 3CC1

-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
(Continue reading)

Permalink | Reply | 1 comment |
headers
Joachim Kupke | 2 May 2005 15:55
Picon

Re: Two suggestions

Thomas Leonard:

> [ splitting into two threads ]

Good idea.

> On Fri, Apr 29, 2005 at 05:02:19PM +0200, Joachim Kupke wrote:
> [...]
> > To make myself clearer; what I was talking about is this: We could
> > "mount -t autofs" the /uri/0install directory; the existing autofs/amd
> > infrastructure on Linux and most other unices will then give us a file
> > handle where lookup requests for /uri/0install/anything will be
> > reported. When we receive such a request, we could download (all of) the
> > "anything" site, extract it to /var/cache/zero-install/anything
> > (including symlinks, excluding "..." files), and report back to the
> > kernel that it should "mount --bind" the
> > /var/cache/zero-install/anything/ directory to /uri/0install/anything/.
> 
> OK, but you'll need to rearrange things. Downloading a whole site isn't
> useful at the moment, because you get all versions of each program.

And that's a scalability issue. One that's relatively easy to avoid if
distinct versions of one package become distinct sub-sites in one sense
or the other.

> However, if a site contains nothing but little scripts to run 0launch,
> then this should work just fine.

That's another extreme. The site may pretty well contain a big
application. zero install would just need to know ahead of time that
(Continue reading)

Permalink | Reply | 0 comments |
headers
Joachim Kupke | 2 May 2005 15:55
Picon

Re: Hashdrop

Excellent idea to split this into two threads. Possibly, we may need
even more. Allow me to restructure a bit...

Thomas Leonard:

[Quotas]

> I don't think network use needs to have a quota. In the injector's case,
> users download the archives themselves, and then submit the unpacked
> archive to hashdrop, so there's no need for the system to do anything
> special.

Unless, of course, users argued that traditionally, their sys admin
would have supplied them with the software they required. If network
usage is (relatively) expensive, they may want to be reimbursed or
something.

> I'm mainly concerned about the case on a shared system where a single
> user has used up all the disk space. The sysadmin should at least be
> able to find out who's responsible and ask them to remove their stuff.

Well, then the idea is that although this is a shared system, most
packages are used by single users. So we could provide a simple
mechanism that records who was the one who installed each package.
Again, this only becomes inaccurate when users begin to use other users'
packages, which we even encourage them to do. When it comes to excessive
disk usage, however, this strategy would yield good hints as to whom to
blame.

[Naming convention; conflicts]
(Continue reading)

Permalink | Reply | 2 comments |
headers
Andrew Flegg | 2 May 2005 20:34
Gravatar

Re: Hashdrop

On Mon, May 02, 2005 at 03:55:43PM +0200, Joachim Kupke wrote:
[snip]
> 
> Not that I disliked using the environment to patch together whatever
> resources are available. In fact, I once suggested envfs or some similar
> mechanism to make /proc/≤pid>/env/LIBFOO_DIR be a symlink to whatever
> the contents of $LIBFOO_DIR...

Indeed:

    http://www.wimpos.org/envfs-0.0.1.tar.gz

Development on the rest of WIMP OS stalled whilst I had a lack of a
development platform but could conceivably restart soon(ish).

I've already got a console-based system running solely from appdirs
using envfs to provide relocatability.

Obviously, envfs still requires root's co-operation on non-WIMP OS
systems; but has the potential to be useful beyond 0install and so
*could* be widespread (yeah, right ;-))

Cheers,

Andrew

--

-- 
Andrew Flegg -- mailto:andrew <at> bleb.org  |  http://www.bleb.org/

-------------------------------------------------------
(Continue reading)

Permalink | Reply | 1 comment |
headers
Joachim Kupke | 3 May 2005 11:12
Picon

Re: Hashdrop

Andrew Flegg:

> On Mon, May 02, 2005 at 03:55:43PM +0200, Joachim Kupke wrote:
> [snip]
> > 
> > Not that I disliked using the environment to patch together whatever
> > resources are available. In fact, I once suggested envfs or some similar
> > mechanism to make /proc/≤pid>/env/LIBFOO_DIR be a symlink to whatever
> > the contents of $LIBFOO_DIR...
> 
> Indeed:
> 
>     http://www.wimpos.org/envfs-0.0.1.tar.gz
> 
> Development on the rest of WIMP OS stalled whilst I had a lack of a
> development platform but could conceivably restart soon(ish).
> 
> I've already got a console-based system running solely from appdirs
> using envfs to provide relocatability.
> 
> Obviously, envfs still requires root's co-operation on non-WIMP OS
> systems; but has the potential to be useful beyond 0install and so
> *could* be widespread (yeah, right ;-))

Actually, I had no idea your envfs existed. I had once hacked something
together for 2.4.x kernels that also allowed to inspect the environment
of other processes (that you own).

Although it would have to be standardized, an environment f/s definitely
deserves a wide spread, but yes, we are far from that.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Mike Hearn | 4 May 2005 15:51

Re: Hashdrop

On Mon, 02 May 2005 09:06:45 +0100, Thomas Leonard wrote:
>> It depends. We should distinguish static and dynamic relocation.
>> binreloc provides dynamic relocation, and---erm---reading from
>> /proc/self/maps is an incredible kludge... ;-)  I don't want to become
>> off-topic here, and this might belong to the autopackage list, but I'd
>> be worried most about the fact that /proc might not be mounted. This
>> could be the case, for example, when you are chroot(2)ed somewhere.

If /proc isn't mounted then stuff will randomly break, end of story.
There's no way to dynamically locate your own binary without using it on
Linux. 

Actually binreloc has recently been completely rewritten. It's now a code
generator and the "basic" form now only uses /proc/self/exe (I think).

> Actually, I suppose we want something similar to it, but not quite the
> same, since we can get the locations from the environment variables set
> by the injector. Eg, you can do:
> 
>     data = fopen(expand("$LIBFOO_DIR/data.dat"));

We have had macros to do this in binreloc, eg:

  data = fopen(LIBDIR("/data.dat"))

the problem being that the memory management was unintuitive (it would be
freed next time the macro was called). Simply keeping them all in a hash
table which is never freed would probably work better.

In the rewrite the macros apparently disappeared. They should be added
(Continue reading)

Permalink | Reply | 10 comments |
headers
Joachim Kupke | 4 May 2005 16:41
Picon

Re: Re: Hashdrop

Mike Hearn:

> >> It depends. We should distinguish static and dynamic relocation.
> >> binreloc provides dynamic relocation, and---erm---reading from
> >> /proc/self/maps is an incredible kludge... ;-)  I don't want to become
> >> off-topic here, and this might belong to the autopackage list, but I'd
> >> be worried most about the fact that /proc might not be mounted. This
> >> could be the case, for example, when you are chroot(2)ed somewhere.
> 
> Actually binreloc has recently been completely rewritten. It's now a code
> generator and the "basic" form now only uses /proc/self/exe (I think).

This is from binreloc's README:

| For executables, you can still find your full location by resolving the
| symlink /proc/self/exe, but that won't work for libraries.

And br_locate() in prefix.c clearly reads from /proc/self/maps, which it
apparently has to so libraries can find their own prefixes.

> If /proc isn't mounted then stuff will randomly break, end of story.

What if:

  mount --bind /proc /some/where/proc
  chroot /some/where

> There's no way to dynamically locate your own binary without using it on
> Linux.
(Continue reading)

Permalink | Reply | 9 comments |
headers
Guido Schimmels | 4 May 2005 20:06
Picon

Re: Re: Hashdrop

On Wed, 4 May 2005 16:41:30 +0200
Joachim Kupke <joachim.kupke <at> inf.ethz.ch> wrote:

> More to the point, still, if userland zero installable applications
> adopted these techniques in order to find themselves, it would still be
> unclear how dependencies would be found. Likely enough, by means of
> environment variables (introducing problems already described in
> <20050503091233.GA23637 <at> inf.ethz.ch>). Which, of course, raises the
> question why we don't use these in the first place to tell applications
> about their prefixes.

The only reason I can imagine for bin-reloc to exist, is the broken/non-existant Gnome resource location API.
For KDE you simply add the app PREFIX to KDEDIRS in the launcher script and you are done!
I wager Gnome hackers hate launcher scripts as they largely still hold on to the CLI as a poor-man's IPC.
gaim-remote/gimp-remote - give me a fucking break, it's not the eighties anymore!

Further note on the use of /proc for this purpose - haven't the BSD's given that crappy interface the
deserved kick in the butt?
Unfortunately, the addition of /sys to the Linux kernel indicates, there is no hope for Linux in this
respect. Hardly any kernel developers with less concern for and a clue about userland interfacing to be found.

A final question, does the 
-z origin
linker flag support relative pathes like "../lib"?
That would at least make it possible to avoid setting LD_LIBRARY_PATH for shared libraries included in the package.

-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
(Continue reading)

Permalink | Reply | 0 comments |
headers
Mike Hearn | 5 May 2005 00:04

Re: Re: Hashdrop

On Wed, 04 May 2005 16:41:30 +0200, Joachim Kupke wrote:
> | For executables, you can still find your full location by resolving the
> | symlink /proc/self/exe, but that won't work for libraries.
> 
> And br_locate() in prefix.c clearly reads from /proc/self/maps, which it
> apparently has to so libraries can find their own prefixes.

Yes, that's why it's done. That can still happen if you generate the
complete version instead of the basic version of the code. But most libs
have no data files anyway.

> For example, /usr/bin/dumpscore is a stripped version of a binary and
> for some unknown reason, dumps core. I still have
> /tmp/pkgsrc/bin/dumpscore which is unstripped so it can be debugged.
> 
> If dumpscore just has its prefix of "/usr" hard-coded, then I can simply
> proceed and need not worry about getting the contents of
> /tmp/pkgsrc/etc, etc. right.

I don't understand. When this happens to me I just copy the unstripped
version to /usr/bin/dumpscore.unstripped - that's the easiest way to do
this.

> Also, hard-linking a binary to /tmp (and then running it from there)
> will break.

Well, I'm really not convinced this is a use case that's common.

> Don't get me wrong: I am very appreciative of your efforts to teach
> Linux applications to dynamically locate their binaries. It's just not
(Continue reading)

Permalink | Reply | 7 comments |

Gmane