headers
Shriramana Sharma | 7 Oct 2007 12:34
Picon

Linux kernel, drivers, shims

Dear list,

Recently I have been looking through some of the Linux kernel - binary 
drivers - GPL issues and came across what people mention about shims -- 
interfaces between GPL-ed kernel and incompatible-licensed-drivers. And 
about the thing where totally independently developed fs-s cannot be 
considered derivative works (at least according to Torvalds).

I am not hacking out the legal parts of GPL (that only apparently leads 
to endless flamewars) but I am asking about the programming part of it.

1)

I don't understand how two separate programs can work in the same memory 
space -- if I have understood properly the part where closed-source 
drivers are "loaded in kernel space" -- if they do not have knowledge of 
how to talk to each other in the level of pushing a set of parameters to 
the stack and making a function call. At that low level, when the fs is 
talking to the kernel, how can it be developed without using the 
kernel's headers?

2)

How is it also possible to just throw in an independently developed fs 
module into the kernel and make it a perfect fit so they communicate 
with each other?

3)

How can a shim be written so as to circumvent legal issues? I mean, a
(Continue reading)

Permalink | Reply | 1 comment |
headers
Glynn Clements | 7 Oct 2007 16:12

Re: Linux kernel, drivers, shims


Shriramana Sharma wrote:

> Recently I have been looking through some of the Linux kernel - binary 
> drivers - GPL issues and came across what people mention about shims -- 
> interfaces between GPL-ed kernel and incompatible-licensed-drivers. And 
> about the thing where totally independently developed fs-s cannot be 
> considered derivative works (at least according to Torvalds).
> 
> I am not hacking out the legal parts of GPL (that only apparently leads 
> to endless flamewars) but I am asking about the programming part of it.
> 
> 1)
> 
> I don't understand how two separate programs can work in the same memory 
> space -- if I have understood properly the part where closed-source 
> drivers are "loaded in kernel space" -- if they do not have knowledge of 
> how to talk to each other in the level of pushing a set of parameters to 
> the stack and making a function call. At that low level, when the fs is 
> talking to the kernel, how can it be developed without using the 
> kernel's headers?
> 
> 2)
> 
> How is it also possible to just throw in an independently developed fs 
> module into the kernel and make it a perfect fit so they communicate 
> with each other?
> 
> 3)
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Dennis Heuer | 9 Oct 2007 17:54

how to stay acid-compliant when using mmap'ed files

hello,

i don't really understand the implications of using mmap. for example,
will linux write out changes to an mmap'ed file as is or as part of a
full page-write? if the latter is true, what happens if the program
reads from mmap'ed pages but writes directly to the file? as far as i
see it, linux will catch the writing and divert it to the mmap'ed page.
this implies that only full page-writes will reach the file.

i ask about this because if i want to write a transaction-safe layer
for a database in a file and linux always affects more bytes in the
file than the program actually commanded, there's no way, after a crash,
to know about what area was actually affected and possibly crippled up.

am i right or is there something i miss? how is it with common file
accesses (via write or fwrite). are they paged automatically by the os
too?

regards,
dennis heuer
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Permalink | Reply | 1 comment |
headers
Glynn Clements | 10 Oct 2007 00:50

Re: how to stay acid-compliant when using mmap'ed files


Dennis Heuer wrote:

> i don't really understand the implications of using mmap. for example,
> will linux write out changes to an mmap'ed file as is or as part of a
> full page-write? if the latter is true, what happens if the program
> reads from mmap'ed pages but writes directly to the file? as far as i
> see it, linux will catch the writing and divert it to the mmap'ed page.
> this implies that only full page-writes will reach the file.

You seem to misunderstand how files and mmap() work on Unix.

All file I/O (actually, all block-device I/O) goes through the buffer
cache. The kernel only ever transfers whole blocks between memory and
disk.

If you open() a file then read() e.g. 237 bytes from the file, the
kernel reads a (typically) 4K block into the buffer cache, then copies
237 bytes from the buffer cache into the process address space.

If you write() data to a file, the kernel ensures that the affected
blocks are in the buffer cache, then overwrites the specified portions
with the data supplied by the process.

If multiple processes have the same file open simultaneously, all
processes share the same cache; IOW, the in-memory copy is
"definitive".

When a process mmap()s a file, any memory pages which cache blocks of
that file are mapped directly into the process' address space using
(Continue reading)

Permalink | Reply | 0 comments |
headers
Dennis Heuer | 13 Oct 2007 00:01

how to best map/buffer a file to memory by being able to lock sequences thread-safe

hello

i tried to write a memory cache for a file and keep it in sync with the
file while threads or foreign processes are writing safely to it. then
i found mmap and lockf but the both don't work together. i studied all
the other alternatives i found, like fcntl or flockfile, but they
don't support the one or the other action. for example, fcntl locks are
not based on a per-thread base. flockfile only works on the full FILE
object, etc. also, if i don't use mmap, as far as i can see, i have to
map the file to a buffer myself, which causes sync-problems especially
with foreign processes.

how can i reach the same level of features and comfort like with lockf
and mmap without one of the both and by staying posix-compliant (at
least)?

by the way: can i lock sections of a plain memory buffer (mmap'ed or
allocated with malloc or whatever) like i can do with fcntl or lockf
to have threads not running over eachother?

regards,
dennis heuer
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Permalink | Reply | 6 comments |
headers
Glynn Clements | 13 Oct 2007 18:00

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe


Dennis Heuer wrote:

> i tried to write a memory cache for a file and keep it in sync with the
> file while threads or foreign processes are writing safely to it. then
> i found mmap and lockf but the both don't work together. i studied all
> the other alternatives i found, like fcntl or flockfile, but they
> don't support the one or the other action. for example, fcntl locks are
> not based on a per-thread base. flockfile only works on the full FILE
> object, etc. also, if i don't use mmap, as far as i can see, i have to
> map the file to a buffer myself, which causes sync-problems especially
> with foreign processes.
> 
> how can i reach the same level of features and comfort like with lockf
> and mmap without one of the both and by staying posix-compliant (at
> least)?

One option is to create another file of the appropriate size, and
obtain locks on that file instead of the mapped file. If the file is
sparse (i.e. is enlarged using ftruncate rather than by having data
written to it), it won't use significant disk space.

> by the way: can i lock sections of a plain memory buffer (mmap'ed or
> allocated with malloc or whatever) like i can do with fcntl or lockf
> to have threads not running over eachother?

You can implement locks for anything using semaphore sets, condition
variables, mutexes etc.

The only way that file locks are really any different is that they let
(Continue reading)

Permalink | Reply | 0 comments |
headers
Dennis Heuer | 15 Oct 2007 17:46

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

i did not try obstacks yet, will have a look at them. what bothers me
at the moment is that mremap() is not a standard and that it operates
strangely to me:

ENOMEM
    The region is private writable, and insufficient virtual memory is
available to extend it. Also, this error will occur if MREMAP_MAYMOVE
is not given and the extension would collide with another mapped region.

i thought that mmap'ed space is only virtually aligned. possibly it's
better to write buffers oneself? the posix interfaces look strange to
me. they seem halfhearted and weirdly incompatible. consider a file
that is growing. how to best remap it?

regards,
dennis heuer
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Permalink | Reply | 4 comments |
headers
Dennis Heuer | 15 Oct 2007 20:05

strange message received after posting to here

to my last message (which has no bcc or whatever untypical in the
header) i received the following answer. do you know about them?

Return-path: <root <at> ctf.com.br>
Envelope-to: dh <at> triple-media.com
Delivery-date: Mon, 15 Oct 2007 17:47:53 +0200
Received: from mail.ctf.com.br ([201.63.119.87] helo=ctf.com.br)
	by srvh02.vc-server.de with esmtps (TLSv1:AES256-SHA:256)
	(Exim 4.68)
	(envelope-from <root <at> ctf.com.br>)
	id 1IhSAa-0005nj-Gq
	for dh <at> triple-media.com; Mon, 15 Oct 2007 17:47:53 +0200
BrmaOutput: root <at> localhost
Received: (from root <at> localhost)
	by ctf.com.br (8.12.11.20060308/8.12.11) id l9FGsYGl022991;
	Mon, 15 Oct 2007 14:54:34 -0200
Date: Mon, 15 Oct 2007 14:54:34 -0200
Message-Id: <200710151654.l9FGsYGl022991 <at> ctf.com.br>
From: postmaster <at> ctf.com.br
To: dennis heuer <dh <at> triple-media.com>
Subject: (BRMA) Mensagem n_o autorizada

Mensagem n_o autorizada
----------------------------------------
Palavra (FF) proibida no campo Subject
----------------------------------------
Para:  linux-c-programming <at> vger.kernel.org
Assunto:  Re: how to best map/buffer a file to memory by being able to
lock
-
(Continue reading)

Permalink | Reply | 4 comments |
headers
Steve Graegert | 15 Oct 2007 22:17
Picon

Re: strange message received after posting to here

Hi Dennis,

Although my Portuguese is pathetic I'd say that the mail has been
blocked for an user's account that is being subscribed to this list.

The reason for the delivery failure remains unknown.

	\Steve

On 10/15/07, Dennis Heuer <dh <at> triple-media.com> wrote:
> to my last message (which has no bcc or whatever untypical in the
> header) i received the following answer. do you know about them?
>
> Return-path: <root <at> ctf.com.br>
> Envelope-to: dh <at> triple-media.com
> Delivery-date: Mon, 15 Oct 2007 17:47:53 +0200
> Received: from mail.ctf.com.br ([201.63.119.87] helo=ctf.com.br)
>         by srvh02.vc-server.de with esmtps (TLSv1:AES256-SHA:256)
>         (Exim 4.68)
>         (envelope-from <root <at> ctf.com.br>)
>         id 1IhSAa-0005nj-Gq
>         for dh <at> triple-media.com; Mon, 15 Oct 2007 17:47:53 +0200
> BrmaOutput: root <at> localhost
> Received: (from root <at> localhost)
>         by ctf.com.br (8.12.11.20060308/8.12.11) id l9FGsYGl022991;
>         Mon, 15 Oct 2007 14:54:34 -0200
> Date: Mon, 15 Oct 2007 14:54:34 -0200
> Message-Id: <200710151654.l9FGsYGl022991 <at> ctf.com.br>
> From: postmaster <at> ctf.com.br
> To: dennis heuer <dh <at> triple-media.com>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Glynn Clements | 16 Oct 2007 12:54

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe


Dennis Heuer wrote:

> i did not try obstacks yet, will have a look at them. what bothers me
> at the moment is that mremap() is not a standard and that it operates
> strangely to me:
> 
> ENOMEM
>     The region is private writable, and insufficient virtual memory is
> available to extend it. Also, this error will occur if MREMAP_MAYMOVE
> is not given and the extension would collide with another mapped region.
> 
> i thought that mmap'ed space is only virtually aligned. possibly it's
> better to write buffers oneself? the posix interfaces look strange to
> me. they seem halfhearted and weirdly incompatible.

I don't understand what you're saying in any of the preceding
paragraph.

> consider a file that is growing. how to best remap it?

If you don't need portability, use mremap() with MREMAP_MAYMOVE. If
you need portability, use munmap() and mmap().

The only advantage of mremap() (other than efficiency) is that it can
be used to remap private or anonymous mappings, while mmap/munmap will
only work if changes are written back to the file.

--

-- 
Glynn Clements <glynn <at> gclements.plus.com>
(Continue reading)

Permalink | Reply | 3 comments |

Gmane