Brills Peng | 1 Apr 08:09 2011

[GSoC] Proposal posted: Improve dsched interfaces and implement BFQ disk scheduling policy

After a week-long discussion with Alex Hornung, I finally fill out my
proposal for GSoC project. Here is the link:

Thank everyone on the mailing list who helped me, and I wish to read your further
suggestions! :)

Brills Peng

Naohiro Aota | 1 Apr 11:46 2011

Re: [GSoC] HAMMER compression and new unionfs

Michael Neumann <mneumann <at>> writes:

> Am Dienstag, den 29.03.2011, 00:48 +0900 schrieb Naohiro Aota:
>> Hi,
>> I'm Naohiro Aota, undergraduate student at Osaka University, Japan.
>> Last year I've participated GSoC with Gentoo and worked on porting
>> Gentoo system to DragonFly. Since then I'm so interested in DragnFly
>> kernel, so I'd like to take part in GSoC with some DragnFly kernel work
>> this year. I've read the project page and get interested these two
>> ideas: HAMMER compression and new unionfs. (yes, I like filesystem ;))
>> I have some question about the ideas.
>> about HAMMER compression:
>> - "compression could be turned on a per-file" may support all files
>>   under "/foo" get compressed?
> Individual blocks of data will be compressed, so that it could happen
> that a file contains uncompressed and compressed data blocks. You only
> have to record a flag whether a given block is compressed (or not) and
> uncompress/compress it transparently before passing it to/from the
> buffer cache. The decision whether to compress a block when writing a
> file can be many-fold: Either a filesystem-wide flag (all files created
> within this filesystem will by default be compressed), a recursivly
> inherited per-directory flag (a new file that gets created inside this
> directory will be compressed), or what is also feasible is that the
> compression is done by the reblocker, i.e. as a background process, so
> that you will never directly write compressed data "online" (this could
> be a starting point).

so if I have a fully uncompressed file like this ("|" indicate block

file Foo: |ABC|DEF|GH|

then it get partly compressed, it become:

file Foo: |<compressed 1>|EFG|H|

finally when all blocks compressed, it become:

file Foo: |<compressed 1>|<compressed 2>|

Is this right?

Implementation process would be:

- Implement userland tool to set compression to a file (hammer set-compress <file> ?)
- Implement systemcalls or such to be used by the tool
- Implement userland tool to search and compress blocks (hammer compress ?)
- Implement the ioctl or such
- Check if blocks are really compressed

- Improve: implement per-directory flag
- Improve: implement per-filesystem flag

- (documentat the feature and the implementation)

Anything to do otherwise?

> As we keep historical data for a longer period of time (this is how
> HAMMER works and we like it), compression could increase the amount of
> historical data that we can store. As most of the historical data is
> only very infrequently accessed (they mainly serve as backup), the
> decompression must not be hyper-performant (IMHO), but of course an
> acceptable performance is desirable (due to slow disk reads, compression
> could even lead to faster access).
>> - file size measurement commands, such as "df", "du" and "ls", also need
>>   to change? (actual disk space size and file size may differ if compressed)
> I think is will be enough to display the uncompressed file size, not the
> compressed one, so no changes should be required. Note that we also have
> deduplication and that "du" and "ls" will not show IMHO the actual disk
> space used.

I see.


Naohiro Aota | 1 Apr 11:50 2011

Re: [GSoC] HAMMER compression and new unionfs

Matthew Dillon <dillon <at>> writes:

> :src/sys/sys/namecache.h or sys/kern/vfs_cache.c?  These are guesses on my
> :part, cause I'm not familiar with it.  Matt did a lot of work on the
> :namecache back in 2003/2004; I linked to some of it on the Digest:
> :
> :
> :
> :Material there may lead you to more information, I hope.
>     The unionfs code hasn't worked in years, and I don't think any of
>     it is salvageable.  Also, the original unionfs code (and probably
>     FreeBSD's current code too) depends on whiteouts which HAMMER does
>     not support.  They never worked very well even in FreeBSD.
>     A new unionfs would require a complete rewrite from scratch and a
>     new approach to recording file deletions.

hm, seems difficult for me.. To have "whiteout database file" on the top
of writable directory, would be another implementation? Then how can I
avoid reading from/writing to the database file?


Chris Turner | 1 Apr 19:20 2011

Re: [GSoC] HAMMER compression and new unionfs

On 04/01/11 04:46, Naohiro Aota wrote:

> - Implement userland tool to set compression to a file (hammer set-compress<file>  ?)

nit: better to add a chflags(1) option

Matthew Dillon | 2 Apr 01:39 2011

Re: [GSoC] HAMMER compression and new unionfs

:On 04/01/11 04:46, Naohiro Aota wrote:
:> - Implement userland tool to set compression to a file (hammer set-compress<file>  ?)
:nit: better to add a chflags(1) option

    chflags flags are a bit annoying to add but there is already
    infrastructure in place from the swapcache work for the 'cache' flag
    that could be reviewed to determine how best to add a new flag.

    The 'cache' flag only had to be set in a higher level directory and
    would get inherited by the run-time system recursively.  A compression
    flag could be made to work the same way.  i.e. so you would only have
    to set it on /usr/src and not on every single file inside /usr/src.

					Matthew Dillon 
					<dillon <at>>

Matthew Dillon | 2 Apr 01:43 2011

Re: [GSoC] HAMMER compression and new unionfs

:hm, seems difficult for me.. To have "whiteout database file" on the top
:of writable directory, would be another implementation? Then how can I
:avoid reading from/writing to the database file?

   The alternatives are expensive.  For example, you could create a
   special <filename>.<magicnumber> file or something like that
   to represent the deleted file.  Any directory entry or stat or open
   or other operation would then have to check whether the magic file
   exists or not when looking up <filename> successfully, and act as
   if <filename> didn't exist if found.

   I'm not sure I'd like a unionfs having to do that but it would be
   fairly easy to implement.

   Using whiteouts is a non-starter, they never worked well even when
   unionfs was being used with UFS because the only way to backup a
   UFS filesystem using whiteouts was to use dump/restore.

					Matthew Dillon 
					<dillon <at>>

Naohiro Aota | 3 Apr 14:02 2011

[GSoC][RFC] HAMMER compression


I wrote my GSoC project proposal for "HAMMER compression". Could you
take a look at it and provide me some comments or suggestions?


Michael Neumann | 4 Apr 12:08 2011

Re: [GSoC][RFC] HAMMER compression

Am Sonntag, den 03.04.2011, 21:02 +0900 schrieb Naohiro Aota:
> Hi,
> I wrote my GSoC project proposal for "HAMMER compression". Could you
> take a look at it and provide me some comments or suggestions?

I would not concentrate too much to the chflags stuff and the user land
utilities other than the "hammer reblocker" [1]. 

1) Imagine there would be compressed blocks stored in a HAMMER
filesystem. How can they be distinguished from non-compressed blocks and
how can they transparently be decompressed and returned to the user.
This is a major milestone! If you would start with that major task,
write yourself an ioctl syscall that generates for you a compressed file
to experiment with.

2) A "hammer compress" user-level utility that scans the filesystem
(similar to the reblocker and dedup) for uncompressed blocks. Start with
compressing each block, then think about "compressing only historical
blocks" or similar policies).

That's how I would do it. I think setting compression for each
individual file is not that important as we would have one PFS
for /usr/src for example (where we want compression) and /usr/obj (where
we don't want compression) so the hammer reblocker would be the ideal
choice for that.



[1]: For example

Kedar Soparkar | 4 Apr 19:47 2011

[GSOC] i386 ABI

Hi there,

I've submitted my GSOC proposal for the i386 ABI implementation in the x86_64 kernel.

Please read it at:
Requesting any pertinent feedback on the same,

Kedar Soparkar.

Irina Preșa | 4 Apr 21:47 2011

[GSOC] Make vkernels checkpointable


My name is Irina Presa and I am a final year undergraduate student from
University Politehnica of Bucharest, Romania, Computer Science and
Engineering Department.

After reading the article "Virtual Kernel Peek" [1], I found vkernel
architecture quite impressing. I've also noticed the "Make vkernels
checkpointable" project on GSOC page and I would be really happy to work
on it.
I'm a beginner in kernel programming, so I would like to know if you
think I might be suitable for this project. This semester in school, I
am attending my first kernel architecture course and I find it
fascinating. As for further OS knowledge, last year I had a course of
system programming (mostly operating systems concepts applied from
userland using POSIX or WIN API). I've also followed Assembly and
Compilers courses.

About this project, after browsing the sources and the mailing list I
came to the following conclusions:

Basically, the save and restore of segment registers (especially the TLS
support) seems to be the most "sensitive" part. I am not very familiar
with this subject (besides the theoretical part I've learned in school)
and I would like some advices in this direction. From the mailing list I
got to this implementation[2]. Apparently, a starting point would be
adding another field in the elf core dump (prsavetls_t tls), that keeps
the tls info and stores it from each thread
bcopy(&lp->lwp_thread->td_tls, tls, sizeof *tls)). I also found this
linux patch[3] which might make a good reference since it implements
thread saving and restore successfully. Wouldn't be necessary to also
load the tls in the Global Descriptor Table?

As for the recreation of the vmspaces, I read here[4] that "it should be
possible to do without any modification of the real kernel", allowing
the vkernel to save the vpagetable information. Is that a valid solution
or we will go by the one described in the gsoc page? Of course, using
the host kernel's checkpoint facility in order to save the VPAGETABLES
offset and map type shouldn't be a problem, but I thought it would be
better to isolate the vkernel as much as possible. As for the vkernel's
part, I'm guessing that some cpu_vmspace_alloc calls (that will call the
corresponding syscall in order to ask host kernel to do the mappings)
using the checkpointed offsets should be enough.

And for the last part the dump/load of the current state of network and
console drivers I browsed the source code and noticed that:
For the network drivers I might need to save the
interface/address/netmask/bridge(maybe reading them with ioctl calls)
and on restore to reopen each interface (netif_open_tap) and reset it's
parameters (netif_init_tap) as the methods from init.c do. The console
code would do something similar using termios getattr and setattr.

I would be glad to know your opinion on the above, and if you can
provide me a starting point - some further reading or even some kernel
tests that I could try.

Best regards,

[3] <at>