headers
Roman Divacky | 1 Mar 2008 13:17
Picon
Favicon

Re: Updating vmware3 (and missing "avail_end" from /usr/include/machine/pmap.h)

On Fri, Feb 29, 2008 at 01:29:23PM -0800, fchang <at> cs.ubc.ca wrote:
> Hi all:
> 
> I'm trying to get vmware3 working on FreeBSD RELEASE 7.0 for i386
> I've resolved every compilation problem except one:
> the driver.c in vmware3 requires the "avail_end" variable
> in 6.2's /usr/include/machine/pmap.h
> but is missing from 7.0's /usr/include/machine/pmap.h
> 
> The vmware3 code goes like this:
> 
>   r = malloc(sizeof *r, M_DEVBUF, M_WAITOK);
>   if (r == NULL) return ENOMEM;
>   if (avail_end > ISA_DMA_BOUNCE_THRESHOLD)
>      high = trunc_page(ISA_DMA_BOUNCE_THRESHOLD);
>   else
>      high = trunc_page(avail_end);
> 
> It turns out "avail_end" was removed from /usr/include/machine/pmap.h
> by revision 1.125, whose commit message reads:
> 
>    The global variable avail_end is redundant and only used once.
>    Eliminate it.  Make avail_start static to the pmap on amd64.
>    (It no longer exists on other architectures.)
> 
> So it appears that we can still get the value represented by
> the old avail_end variable, but I don't know anything about the kernel
> and I don't understand that comment.   Could some kind soul help me
> decipher that so I can try to get vmware3 compilable again?
(Continue reading)

Permalink | Reply | 2 comments |
headers
Roman Divacky | 1 Mar 2008 13:20
Picon
Favicon

Re: Updating vmware3 (and missing "avail_end" from /usr/include/machine/pmap.h)

On Sat, Mar 01, 2008 at 01:17:01PM +0100, Roman Divacky wrote:
> On Fri, Feb 29, 2008 at 01:29:23PM -0800, fchang <at> cs.ubc.ca wrote:
> > Hi all:
> > 
> > I'm trying to get vmware3 working on FreeBSD RELEASE 7.0 for i386
> > I've resolved every compilation problem except one:
> > the driver.c in vmware3 requires the "avail_end" variable
> > in 6.2's /usr/include/machine/pmap.h
> > but is missing from 7.0's /usr/include/machine/pmap.h
> > 
> > The vmware3 code goes like this:
> > 
> >   r = malloc(sizeof *r, M_DEVBUF, M_WAITOK);
> >   if (r == NULL) return ENOMEM;
> >   if (avail_end > ISA_DMA_BOUNCE_THRESHOLD)
> >      high = trunc_page(ISA_DMA_BOUNCE_THRESHOLD);
> >   else
> >      high = trunc_page(avail_end);
> > 
> > It turns out "avail_end" was removed from /usr/include/machine/pmap.h
> > by revision 1.125, whose commit message reads:
> > 
> >    The global variable avail_end is redundant and only used once.
> >    Eliminate it.  Make avail_start static to the pmap on amd64.
> >    (It no longer exists on other architectures.)
> > 
> > So it appears that we can still get the value represented by
> > the old avail_end variable, but I don't know anything about the kernel
> > and I don't understand that comment.   Could some kind soul help me
> > decipher that so I can try to get vmware3 compilable again?
(Continue reading)

Permalink | Reply | 1 comment |
headers
Martin Laabs | 1 Mar 2008 14:38
Picon

SIGPIPE propagation

Hello,

as you probably know I want to expand dump in such
a way it enableds multivolume dumps also when the
output data is written to stdout/a pipe.
Now I ran into a serious problem.
Dump counts the written bytes to remember where
it has to proceede after the volumen change.
Now the SIGPIPE signal does not propagate fast
enought through the system. Thus dump writes
some time after the pipe has got broken data into
the pipe while it doesn't exist anymore. This short
time it does not receive any error about that.
(Until the signal propagates to dump.)
I wrote a small test progam to demonstrate this
effect (and get clear about it by myself). It just
write data to stdout and counts it until the pipe
is broken (and the write call will return -1) This
is the output of a few successively runs:

$ ./a.out |asdf
$ asdf: command not found
cought broken pipe
i: 12 could only wrote -1 bytes. total wrote: 61440

$ ./a.out |asdf
$ asdf: command not found
cought broken pipe
i: 1 could only wrote -1 bytes. total wrote: 5120
(Continue reading)

Permalink | Reply | 2 comments |
headers
fchang | 1 Mar 2008 16:07
Picon
Picon
Favicon

Re: Updating vmware3 (and missing "avail_end" from /usr/include/machine/pmap.h)

On Sat, 1 Mar 2008, Roman Divacky wrote:
> I think you can use "ptoa(Maxmem) - round_page(MSGBUF_SIZE)",
> the ptoa(Maxmem)
> erm.. what I meant to say what
> that hw.realmem - round_page(MSGBUF_SIZE) should work

Hi:

Before I saw your message, I continued my search and found an January 2007
mailing list message by Kip Macy on how to compute avail_end:
http://groups.google.com/group/mailing.freebsd.current/msg/7039973c1586a8a7

   static vm_paddr_t
   get_avail_end(void)
    {
     vm_paddr_t avail_end;
     int i;
     avail_end = phys_avail[1];
     for (i = 0; phys_avail[i + 1]; i += 2) {
       if (phys_avail[i + 1] > avail_end)
         avail_end = phys_avail[i + 1];
       }
     return avail_end;
    }

Are the two methods equivalent?

Also, how do I use hw.realmem like you suggested?
I tried the following, but the file wouldn't compile:
(Continue reading)

Permalink | Reply | 0 comments |
headers
Pieter de Goeje | 1 Mar 2008 16:40
Picon
Gravatar

Re: SIGPIPE propagation

On Saturday 01 March 2008, Martin Laabs wrote:
> Hello,
>
> as you probably know I want to expand dump in such
> a way it enableds multivolume dumps also when the
> output data is written to stdout/a pipe.
> Now I ran into a serious problem.
> Dump counts the written bytes to remember where
> it has to proceede after the volumen change.
> Now the SIGPIPE signal does not propagate fast
> enought through the system. Thus dump writes
> some time after the pipe has got broken data into
> the pipe while it doesn't exist anymore. This short
> time it does not receive any error about that.
> (Until the signal propagates to dump.)
> I wrote a small test progam to demonstrate this
> effect (and get clear about it by myself). It just
> write data to stdout and counts it until the pipe
> is broken (and the write call will return -1) This
> is the output of a few successively runs:

This problem cannot be solved using sigpipe. You will always have a race 
condition because the reader's only way of notifying the writer that it 
shouldn't write anymore is through closing the pipe, after which the reader 
cannot read any remaining data that was just written to the pipe buffer.

Given ./a.out | asdf

Consider the fact that when you create a pipe, both ends are open. At this 
point it is possible to write a small amount of data to the pipe without
(Continue reading)

Permalink | Reply | 0 comments |
headers
Martin Laabs | 1 Mar 2008 21:11
Picon

multi volume dump with gzip/crypt - solved

Hi,

I solved the dump problem in a not very
clean way.
First I changed dumps behaviour to ignore
the SIGPIPE signal when the -P and -a option
is given. The problem with occured than was
the loss of data because of the race condition
between "write date to pipe" and "receive SIGPIPE".

I solved that problem with a script that
takes the data from the pipe of dump, compress
it and sends a dedicated SIGPIPE to the three
dump processes when the output volume is nealy full.
After that it flushs the rest out of the pipe to
avoid data loss.

Greetings,
  Martin L.
_______________________________________________
freebsd-hackers <at> freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe <at> freebsd.org"
Permalink | Reply | 1 comment |
headers
Robert Watson | 2 Mar 2008 12:36
Picon
Favicon

Re: SIGPIPE propagation


On Sat, 1 Mar 2008, Martin Laabs wrote:

> as you probably know I want to expand dump in such a way it enableds 
> multivolume dumps also when the output data is written to stdout/a pipe. Now 
> I ran into a serious problem. Dump counts the written bytes to remember 
> where it has to proceede after the volumen change. Now the SIGPIPE signal 
> does not propagate fast enought through the system. Thus dump writes some 
> time after the pipe has got broken data into the pipe while it doesn't exist 
> anymore. This short time it does not receive any error about that. (Until 
> the signal propagates to dump.) I wrote a small test progam to demonstrate 
> this effect (and get clear about it by myself). It just write data to stdout 
> and counts it until the pipe is broken (and the write call will return -1) 
> This is the output of a few successively runs:

Here's how SIGPIPE works: when one endpoint of a pipe or socketpair is closed, 
the other endpoint will be flagged to return EPIPE for any further system 
calls to write to the pipe or socket.  The interesting case involves what to 
do if there is already an in-progress system call for a thread writing to it. 
There are a few cases from the perspective of an application writer:

(1) The write is issued "before" the close and can be fully satisfied without
     sleeping, in which case it returns the number of bytes written.

(2) The write is issued "after" the close and returns EPIPE and/or SIGPIPE
     depending on signal disposition, socket options, etc.

(3) The write is issued "during" the close, and in particular, in a case where
     the write was blocking waiting on the reader to make more space available
     to hold all of the written data.  The kernel will return a positive
(Continue reading)

Permalink | Reply | 0 comments |
headers
Robert Watson | 2 Mar 2008 12:44
Picon
Favicon

Re: multi volume dump with gzip/crypt - solved


On Sat, 1 Mar 2008, Martin Laabs wrote:

> I solved the dump problem in a not very clean way. First I changed dumps 
> behaviour to ignore the SIGPIPE signal when the -P and -a option is given. 
> The problem with occured than was the loss of data because of the race 
> condition between "write date to pipe" and "receive SIGPIPE".
>
> I solved that problem with a script that takes the data from the pipe of 
> dump, compress it and sends a dedicated SIGPIPE to the three dump processes 
> when the output volume is nealy full. After that it flushs the rest out of 
> the pipe to avoid data loss.

SIGPIPE occurs when a sending process sends data that the recipient doesn't 
expect, as the recipient has (presumably) closed the socket because it doesn't 
expect further data.  Stream sockets are intended to provided buffered, 
reliable communication only when open, so generally protocols running over 
stream sockets (be it TCP, UNIX domain sockets, etc) have an explicit protocol 
close sequence so that the two endpoints reach mutual agreement that it's time 
to close the sockets and avoid data loss associated with unexpected assymetric 
close.  If the recipient closed the socket while the sender is still writing 
to the socket, you may "lose" up to the size of the receive socket buffer, 
which from the perspective of the sender may have been reported as 
successfully delivered.  The receive socket buffer, FYI, is effectively the 
amount of data that the recipient has agreed can be locally buffered before 
causing the sender to block.

Robert N M Watson
Computer Laboratory
University of Cambridge
(Continue reading)

Permalink | Reply | 0 comments |
headers
carlos neira | 3 Mar 2008 02:18
Picon

readahead(2) - Linux

Greetings :

is there an equivalent of  readahead syscall in linux , for  freebsd ?.
i was looking at  http://preload.sourceforge.net/ ,  and it needs this .
_______________________________________________
freebsd-hackers <at> freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe <at> freebsd.org"
Permalink | Reply | 7 comments |
headers
Ed Schouten | 3 Mar 2008 09:10
Picon
Gravatar

Re: readahead(2) - Linux

* carlos neira <cneirabustos <at> gmail.com> wrote:
> is there an equivalent of  readahead syscall in linux , for  freebsd ?.
> i was looking at  http://preload.sourceforge.net/ ,  and it needs this .

Maybe a mmap(), followed by a madvise()?

--

-- 
 Ed Schouten <ed <at> fxq.nl>
 WWW: http://g-rave.nl/
Permalink | Reply | 6 comments |

Gmane