Francois Tigeot | 6 May 2013 12:43

Testers with sili(4) hardware needed

Hi,

I'm trying to make sure the Silicon Image SATA chipsets can properly
handle I/O requests of 256KB (their apparent theoretical limit). 

If you use this hardware, could you please apply the attached patch and
report your findings ?

Running this command as root will generate 256KB I/O requests:

    dd if=/dev/da0 of=/dev/null bs=1m count=10k

I'm also interested if there is any issue in the course of regular
operation.

Thanks!

--

-- 
Francois Tigeot
diff --git a/sys/cpu/i386/include/param.h b/sys/cpu/i386/include/param.h
index d6547ce..2f48017 100644
--- a/sys/cpu/i386/include/param.h
+++ b/sys/cpu/i386/include/param.h
 <at>  <at>  -121,7 +121,7  <at>  <at> 
 #define BLKDEV_IOSIZE	PAGE_SIZE	/* default block device I/O size */
 #endif
 #define DFLTPHYS	(64 * 1024)	/* default max raw I/O transfer size */
-#define MAXPHYS		(128 * 1024)	/* max raw I/O transfer size */
(Continue reading)

tripun goel | 3 May 2013 14:32
Picon

Deadlock while switching from bfq scheduler to another policy

Hi all ,
it is a known bug  documented in the code too that   deadlock can occur when  teardown and helper thread is on the same  cpu
[1] Referring to bfq_teardown() in bfq.c
I found a possible cause and have an amateur solution for it .
it happens because  when teardown sends a kill message [2] to helper thread , helper_msg_kill is  called to add the kill message in queue with other messages serialized by lwkt and return to teardown function .The helper thread will only receive kill once it has executed all other messages before it .
After making this  call teardown destroy the message cache and the helper thread  never receives the kill message which continues executing and hence a deadlock.

Solution :
Either we can directly kill the helper thread ( using a global variable which is usually a bad idea) without adding the message to lwkt
ex 
if(kill==1)// where kill is global variable initialized zero and is set to 1 by teardown
break;

or 
a spinlock which makes teardown wait for the helper thread to complete reading its messages and release it .

Comments please..
Cheers
Tripun
Chris Turner | 2 May 2013 15:07

Re: DragonFly 3.4 released!

On 05/01/13 14:05, Justin Sherrill wrote:
> People are so used to BSD ports == building from source that it's
> going to take a while to disabuse them of the notion, I bet.  The
> Slashdot post on the 3.4 release (for instance) had a number of people
> blithely going on about building from source and so on.
...
> On Wed, May 1, 2013 at 12:42 PM, John Marino <dragonflybsd <at> marino.st> wrote:
>> On 4/30/2013 06:24, Justin Sherrill wrote:
...
>> I've been reading a lot of the response to DPorts on the various outlets,
>> and there's a common misconception that users are expected to build software
>> from source.
...
>> In fact, building from source is the exception.

Meh. Building from source is the true unix way.

Overreliance on binaries gradually devolves until you end up
with a situation (like linux 'distrows') where noone
knows how the system as a whole is built except for a
small 'caste' of professionals who are abstracted away into
some insular and poorly-documented 'developer team', and a
large and ignorant user base who takes a slavish attitude
to their binary servility and is behaviorally
conditioned to be afraid to 'peek under the hood'
and 'do' anything.

That being said, binaries do have their place and should
be made available.

A chef, does not serve microwave pizzas, and doesn't complain
about cooking from scratch - but at the same time doesn't
mind going to a restaurant periodically. And we all cook
meals from time to time. So too it should be, imho, with
binaries.

Favicon
Gravatar

Hi Matt

Hi Matt,

Sorry if this is my second e-mail. I think i lost it in my sentmail
folder.

I am currently looking on BigData optimisations using DragonFly's
designs.

Regards,
Muhammad Nuzaihan Bin Kamal Luddin

--

-- 
Taqi Systems
269-J Jalan Panji 
Kampung Chempaka, Kota Bharu, Kelantan 16100

pub   4096R/4C77F88C 2013-04-06 [expires: 2015-04-06]
      Key fingerprint = 2FE1 87FA E775 2E05 CC0F  B3F6 3CB7 C65F 4C77
F88C
uid                  Muhammad Nuzaihan Bin Kamal Luddin

Tobias Weingartner | 2 May 2013 00:48
Favicon

Fwd: Selection of roadmap for i386 platform End-of-Life (EOL)

---------- Forwarded message ----------
From: Tobias Weingartner <weingart <at> tepid.org>
Date: Wed, May 1, 2013 at 3:47 PM
Subject: Re: Selection of roadmap for i386 platform End-of-Life (EOL)
To: "B. Estrade" <estrabd <at> gmail.com>

On Wed, May 1, 2013 at 11:55 AM, B. Estrade <estrabd <at> gmail.com> wrote:
>
> FWIW, FreeBSD and NetBSD support PAE (physical address extensions) which
> allow processes to take advantage of 32 bit hardware that can support
> greater than 4 GB of memory. FWIW, OpenBSD appears to not.
>
> If PAE is not part of the direction of DragonflyBSD, then it might be moot
> as to whether 32 bit is supported or not. You might be better of using
> FreeBSD, NetBSD, Linux, or even Windows.

PAE does not really allow a single process to grow beyond 4GB of
memory.  It is really
just a way to have the OS access more than 4GB of physical ram, and
map the extra
ram beyond 4GB into max 4GB address spaces for user processes to use.

One nice thing that PAE provides (at least today) is the NX bit, which
means that you
can do per-page no-execute, and don't need to use segment protections
to do this part.
So for that reason, I'd say that dropping i386 support for i386
without PAE support might
be a good idea.  However, it would not solve the "limited" VM space
problem, and it might
end up opening other issues, since a physaddr would be 64 bits, but a
vmaddr would remain
at 32 bits.  *shrug*  YMMWV.

-Toby.

Just a Normal Person | 1 May 2013 19:50
Picon

Selection of roadmap for i386 platform End-of-Life (EOL)

I don't think there is a really good reason to remove support for
32-bit x86 machines in my opinion. As it was pointed out before, it is
not like they are not capable of running DragonFly well; granted, for
the project running on machines with less than 256 megabytes of RAM
may not be a priority, but there are a lot of machines with >= 256 MB
of memory and they're 32-bit x86.
I have run the latest DragonFly release on a Pentium 4 machine from
2001 with 512 MB of RDRAM, and it ran perfectly. And we're talking
about a machine from 2001; newer ones are going to run it even faster.

I think that the thought of thinking that a machine is not useful just
because it is old is a naive and dangerous one: people do not usually
test lesser known operating systems such as DragonFly on new systems.
 And if they do not test it, they're not going to be introduced to
DragonFly, and the already small userbase will shrink even more.
Another thing to remember: not everyone in the world has access to the
greatest and latest machines!
Other points are that the maintenance burden is really small, and that
i386 machines are all over the place; there's like a huge supply of
them, every flea market one goes to, for instance, is bound to have at
least some.

At the end I think that the only wise choice for the DragonFly project
is to keep supporting the i386 architecture, at most what could be
done instead of removing support is moving the i386 architecture to a
"second tier" place. That way you would show that i386 support is not
your focus and that people better expect to make up for the lack of
focus themselves (like building the ports themselves, instead of using
premade precompiled packages).
You never see projects like NetBSD and OpenBSD removing support for
architectures, just because they think they're not as used as they
once were; when they discontinue architectures, it's because they
really have problems at maintenance.
In fact, the second tier approach I suggested is just like what NetBSD did.

Removing i386 support? For me it is a sure no-no and it can only hurt
DragonFly at the end.
In before I get replies like "go use another *BSD flavor": that's
pretty amusing.

Giuseppe Gatta

Justin Sherrill | 30 Apr 2013 06:24
Favicon
Gravatar

DragonFly 3.4 released!

Version 3.4 of DragonFly is officially out.

The release ISO/IMG files are all available at the usual mirrors:

http://www.dragonflybsd.org/mirrors/

The release notes have details on all the changes:

http://www.dragonflybsd.org/release34/

If you are planning to try the new dports system for installing
third-party software, check the DPorts Howto page:

http://www.dragonflybsd.org/docs/howtos/HowToDPorts/

If you have an installed DragonFly 3.2 system and you are looking to
upgrade, these (not directly tested) steps should work, as root:

cd /usr/src
git fetch origin
git branch DragonFly_RELEASE_3_4 origin/DragonFly_RELEASE_3_4
git checkout DragonFly_RELEASE_3_4

... And then go through the normal buildworld/buildkernel process
found in /usr/src/UPDATING.  If you are running a generic kernel, that
can be as simple as

make buildworld && make buildkernel && make installkernel && make
installworld && make upgrade

(and then reboot)

If you encounter problems, please report them at
bugs.dragonflybsd.org.  I get better at testing for each release, but
I also get better at discovering new problems just after release.

Daniel Flores | 23 Apr 2013 21:14
Picon

[GSoC] HAMMER2 block compression feature

Hello.

My name is Daniel Flores and I’m a student from Spain. I study in Barcelona School of Informatics, which is a faculty of Polytechnic University of Catalonia, and it’s my last year before I graduate.

I sent a mail with this proposal to Matthew 4 days ago, but I didn't get a reply yet, so I decided to send this to the mailing list, as it is suggested on Dragonfly's GSoC page. Before submitting any official application, I would like to ask for your opinion and criticism regarding my proposal and whether I have the required skill and adequate planning to successfully finish this project.

I’m interested in implementing the block compression feature in HAMMER2, which is described on Dragonfly’s GSoC page [1]. The feature consists in that any written block should be compressed, if possible, to 50% of its size or less (in powers of 2 – 32 KB, 16 KB and so on, the initial size of block being 64 KB). It should be possible to set that compression mode on any directory, so that new contents written in it and its subdirectories would be compressed.

I believe that I can succeed because I have programming experience in C/C++, since we use those languages for most practices in our university. I also completed courses centered on algorithms and their implementation in C or C++. I also have an interest in operating systems, their concepts and their development. I completed a course on Operating Systems development where I and a partner implemented a simple linux-based kernel called “ZeOS” (using C and, at times, x86 assembler). It’s a very simple kernel that has a task scheduler, system calls, exceptions, interruptions, memory management and a file system. We were able to successfully run it on bochs emulator and it passed all tests needed to successfully finish the course.

If you wish, you may take a look at it [2].

Sadly, it’s documented in Spanish, but I just want to provide it as an example of code that I can produce.

In spite of it being a kernel implementation, I think that my skills and knowledge aren’t yet the best suitable for kernel development projects, so instead I prefer to do something moderately difficult, yet related to operating systems, like HAMMER2 block compression feature.

My initial outline for project planning is the following:

1. (Starting May 27 or before – June 17) During community bonding period I would get myself familiarized with the community, Dragonfly BSD system/code and “hammer2” branch code. I would also refresh my knowledge on compression algorithms and would complete a design for the future implementation, like choosing the appropriate algorithm and having defined everything needed to be done. I have read Matthew's design document for HAMMER2 [3] and, as I understand, the algorithm #2 is not yet determined, so I believe this should be decided before the actual coding period will start. Also, more detailed planning for coding period would be done, with clear milestones and deliverables.

The deliverables for this phase would be in the form of documents that would describe the design, the chosen algorithm(s) and the calendar with milestones.

2. (June 17 – July 29) I would start by coding a prototype application that would implement the defined design, consisting of algorithms #0, #1 and #2. I would need it to insure that the algorithm is implemented correctly and to do all necessary debugging and refining before integrating it in HAMMER2. I would create a series of tests to prove that the algorithm works well. If time allows, I would also try to study its behavior statistically and to improve it, if possible.

The deliverables would be code of the prototype as well as any supporting applications (to generate tests, for example) and tests themselves.

3. (July 29 – September 16) After confirming that the prototype works well, I would start actually integrating it into HAMMER2 filesystem. At this point, the task would be gradually adding prototype’s code without any conceptual modification into HAMMER2 and testing if it works correctly in real-life. A series of testing applications with appropriate system calls would be created and the results checked for correctness. Also, the appropriate hammer2 utility command to set the compression feature to directories would be created and tested.

The deliverables would be the code of implemented feature and application tests for it.

4. (September 16 – End of GSoC 2013) Hopefully, all known bugs would be corrected at this point, so I would take the time to do final refinements to documentation, to write a detailed description of what was implemented for future developers and to finish everything needed to start submitting my code to Google.

Generally, my philosophy is to start “as soon as possible”, so if some phase is done before the deadline, I would immediately go to the next phase. If project goes smoothly and there is some time left, I would be happy to do small refinements or bugs fixes on general system or HAMMER2.

During the coding period (June 17 – September 16) I don’t have any commitments (like courses or trips) at all, so I should be able to easily dedicate 40 hours per week or more to this project. Prior to June 17 I have my studies and some exams, but generally it should not prevent me from getting familiarized with code and getting the design/algorithm decisions done, because my workload is not very big.

I don’t have any past experience with Open Source or generally big projects, but I believe that I can accomplish this task since it is generally related to C programming and algorithms, and I believe that I have sufficient knowledge of those fields from my university courses.

Currently I am getting myself familiarized with Dragonfly 3.2 (for i386) running it on Oracle VM Virtual Box 4.2.12, but I’m able to install it on physical machine, if necessary. I’m also trying to learn as much as possible about BSD in general and particularly Dragonfly.

If you think that my set of skills is more appropriate to some other project related to HAMMER2 or Dragonfly, I would be grateful if you pointed at them. I also would consider working on HAMMER2 block encryption feature, since it is similar in requirements to my current proposal, if there is already a strong candidate to implement the block compression feature.

Thank you.

[1] http://www.dragonflybsd.org/docs/developer/gsocprojectspage/
[2] https://dl.dropboxusercontent.com/u/56646280/zeos.zip
[3] http://leaf.dragonflybsd.org/mailarchive/users/2012-02/msg00020.html


Daniel
grigore larisa | 23 Apr 2013 19:02
Picon

[GSOC] Userland System V Shared Memory / Semaphore / Message Queue implementation

Hello, 
 
My name is Larisa Grigore and I am student in the second year at a master program "Internet  
Engineering" from Polytechnics University of Bucharest, Romania, Computer Science and  
Engineering Department.  
 
Currently I am working at a research project for master program. The project is about "High Availability".  We try to keep a system, that runs over a microkernel 
from L4 family, alive no matter what happens with its components. The detected faults are segmentation faults, deadlocks and killed threads. From the point of view of recovery, two possible options are provided: restarting an address space or restoring it from some checkpoint. The user can order classic and incremental checkpoint and use those two types in correlation with fork mechanism. In order to build the HA module, memory management and fork with copy-on-write mechanisms were implemented. 
High Availability features, previously developed for native applications running on top of microkernel,  were extended for Linux operating system. A kernel module was implemented to support fault detection (deadlocks, segmentation faults and killed processes) and recovery. The process can be restarted from zero or from a specific checkpoint. 
The project aims to cover cases where Android fails to restore properly an application. After studding frequent android application failures, we concluded that High Availability support can be useful for some classes of applications too.  
 
From July 2013, I am Associated Teaching Assistant in University Polytechnics of Bucharest. I am teaching labs and develop lab material for Operating Systems classes. 
 
I am interested in "Userland System V Shared Memory / Semaphore / Message Queue implementation" project on  
GSOC page. Here are few ideas after some research in System V area:
- daemon 
  - it will take care for the management of system v resources; all operations as creation/destruction will be implemented through messages sent  
  to it; 
- communication with daemon 
  - first step (registration)  
    - using a known named pipe to tell the daemon that the process wants to open a communication 
  - second step (communication) 
    - both client process and daemon will open another named pipe, based on client pid and will use it to talk each other 
 
- shared memory 
  - daemon will create files in order to be mapped in the processes address space and will keep other information related 
 
- semaphores 
  - an implementation similar to Posix Unnamed semaphores (memory-based semaphores) [1] 
  - client will ask for a semaphore and the daemon will return a file and an offset for the semaphore 
  - acquiring and releasing a semaphore will be done as sem_wait and sem_post posix implementation; atomic functions will be used in order  
  to test the semaphore value; umtx_sleep(2)/umtx_wakeup(2) will be used in case one process must block 
  - the daemon is responsible for telling the clients where to find a semaphore 
  - there are two approaches:  
    1. for each semaphore, a process must map a file with the size of PAGE_SIZE; the problem  
    with this approach is the big amount of memory used in case of multiple semaphores 
    2. more semaphores corresponding to a file; here some security issues appear because an application may have access to  
    semaphores not opened by it
 
- message queues 
  - client will send a message to the daemon, asking for a message queue; the daemon will respond with a file to map in its address space 
  - the queue size will depend on file size 
  - the file will contain, beside messages written by processes, some control information (related to number of messages, the first message offset, etc)

Any feedback on this is welcome.

Pedro J. Ruiz López | 22 Apr 2013 20:48
Picon

GSoC - Making DragonflyBSD multiboot capable

Hello.

My name is Pedro. I am interested in working on DragonflyBSD as a GSoC 
student and the idea of making it multiboot capable is very interesting 
to me.

I have a BSc in Computer Science and I am studying MSc in Computer 
Engineering at University of Malaga (Spain). I use GNU/Linux systems on 
a daily basis and have used FreeBSD for several years. My main 
contribution to free software community is the GCC backend for PIC18 
target CPUs. I carried out such project as final BSc thesis. I also have 
been co-maintaining GNU Idutils since 2007. Apart from this, I have 
professional experience developing software in C language and know other 
languages as Perl and Bourne shell scripting language.

Am I on time for discussing the work to do and preparing my application?

Regards.

Mihai Carabas | 21 Apr 2013 23:42
Picon

[GSOC] Implement hardware nested page table support for vkernels

Hello,

My name is Mihai Carabas and I am a second year student at a master program from Politehnica University of Bucharest, Romania, Computer Science and Engineering Department. 

I was envolved last year in the GSoC program with the DragonFLY BSD scheduler ("SMT/HT awareness to DragonFlyBSD scheduler"). The ones who aren't familiar with what I had accomplished last year, here [1] is a summary with the work and results. Meanwhile Matthew did a refactoring to the scheduler and together with another improvements, the results got much better.

More about me you can find on the last year proposal [2]. In the past half a year I worked on virtualizing Android on top of the L4 microkernel. The goal is to have two Androids running on a Galaxy Nexus. My role in this project was to port (virtualize) a flavour of linux kernel 3.0.8 from Samsung (tuna/maguro) on top of the microkernel. Most problems came from the fact that were three layer of addresses (physical addresses used by the microkernel, microkernel virtual addresses used by the kernel and linux virtual addresses used by the user space linux programs). The linux kernel was running in a virtual space (the microkernel address space) and one type of problem was when allocating "physical" memory for devices (ex. GPU) and passing the device the address to read/write from there. We had to take care to pass the actual physical address, not the virtual one, otherwise we would get and interconnect error which is hard to trace.

After browsing through all this year's projects and due to the fact that I have been working a lot with memory mapping and address translations, I would like to work this year on the "Implementing hardware nested page table support for vkernels".

Before I began I must get a strong understading of how the current virtual page table is implemented starting from the vmspace implementation [3]. Another point where to see how the vmspace is working/implemented is the "Page Faults" section from the "Virtual Kernel Peek" [4]. Another point to look is the Matthew Dillon's article from here [1]. It's worth mentioning that last year I worked a little with the vkernels: I implemented the CPU topology for them (basicly you can create any kind of topology you want).

I started documenting on how nested page tables work. It's good to mention that both AMD and Intel have implemented this virtualization extension but under different names: NPT (nested page tables) for AMD and EPT(extended page tables) for Intel. As far as I have read, they differ in some important details (for example EPT doesn't support accessed/dirty bits - I have to see how this would influence the implementation).

A brief description of how NPT works can be found in the System Programming manual from AMD [5] at page 491. Basicly, instead of using only the CR3 register which indicates the place where the page tables are, there are two registers: gCR3 (guest CR3) which points to the guest page tables (mapping the virtual guest pages with the physical guest ones) and nCR3 (nested CR3) which points to the host page table (mapping the guest physical pages with the physical memory). In the TLB are kept the direct mappings (guest virtual pages with physical memory pages and the guests can be differentiated by an Address Space ID - ASID). A more extended description can be found in this paper published by AMD [7].

My plan is as following:

1) Create a mechanism to detect what virtualization extension supports the CPU. For this I can use the cpuid instruction. For example for AMD: check the CPUID Fn8000_0001_ECX[SVM] to see if supports virtualization at all and if so, check the CPUID Fn8000_000A_EDX[NP] to see if the NPT extension is available [8].

2) While exposing the info's discovered at 1), document on the flow of calls regarding the virtual memory allocation/creation when a vkernel starts (and more when a process in the vkernel is created).

3) Documenting on how the virtual page tables are walked through and propose a design for a hardware implementation using NPT/EPT.

4) Peek a platform (probabl an Intel core-i3) and write a stub implementation for activating/using the virtualization extension (this can be done by looking at the normal implementation with one CR3 register). Here we have to take care to leave the current implementation if no NPT/EPT is present.

5) After having a stable vkernel with the NPT/EPT begin testing to see what is the gain with the virtualization extension enabled. Here we must peak programs that are allocating/freeing and accesing a lot of pages to invalidate mappings and create new ones. This way we force of page table walks.

6) Extending the implementation on all platforms (32/64 - AMD/Intel).

I would be glad to know your opinion on the above.

Best regards,
Mihai


Gmane