headers
Andrew Hill | 1 May 2008 10:54
Favicon

ZFS docs / info

Ivan Voras wrote:
 > Do you know about http://wiki.freebsd.org/ZFS ?

Yes, that was my starting point as I learnt about ZFS. I simply wanted  
to offer documentation aimed at a different level of user.

I found that the documentation on that wiki and the docs it links to  
tended to fit into one of three categories
1. it provided a very high level listing of features of the whole  
system, without talking about specific components, what each one is  
responsible for and how they fit together (e.g. is the zpool or the  
zfs responsible for checksumming, compression, redundancy, etc) -  
great for convincing people of the worth of ZFS
2. it assumes the reader has full knowledge of how the zfs pieces fit  
together (i.e. they what they want to create and when) and was simply  
there to document the syntax of the zpool and zfs commands - a good  
quick-reference guide for those familiar with zfs
3. it provided very detailed information about commands, which must of  
course include how to use every single component available to ZFS, a  
lot of which is far beyond what a typical 'home' bsd user would want,  
and perhaps confusing due to the level of detail - but perfect for an  
engineer or administrator

Obviously the right documentation for a specific user really depends  
on their background knowledge, and I felt that the first category was  
great for convincing someone to use ZFS, but if they knew nothing of  
how the pieces fit together then 2 and 3 were a very deep pool to dive  
into. So I've tried to summarise the info I found from all three into  
a simpler document aimed somewhere in between high-level-overview and  
detailed-man-pages, containing what I found most useful from the
(Continue reading)

Permalink | Reply | 0 comments |
headers
Eric Anderson | 2 May 2008 22:40
Picon
Favicon
Gravatar

Re: Consistent inodes between distinct machines

On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote:

> Hello,
>
> I have several NFS servers, where the service must be available  
> 0-24. The servers are mounted read only on the clients and I've  
> solved the problem of maintaining consistent inodes between them by  
> rsyncing an UFS image and mounting it via md on the NFS servers.
> The machines have a common IP address with CARP, so if one of them  
> falls out, the other(s) can take over.
>
> This works nice, but rsyncing multi gigabyte files are becoming more  
> and more annoying, so I've wondered whether it would be possible to  
> get constant inodes between machines via alternative ways.

Why not avoid syncing multi-gigabyte files by splitting your huge FS  
image into many smaller say 512MB files, then use md and geom concat/ 
stripe/etc to make them all one image that you mount?

Eric

_______________________________________________
freebsd-fs <at> freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe <at> freebsd.org"
Permalink | Reply | 5 comments |
headers
Bernd Walter | 3 May 2008 14:50
Picon

Re: Consistent inodes between distinct machines

On Fri, May 02, 2008 at 03:40:11PM -0500, Eric Anderson wrote:
> On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote:
> 
> >Hello,
> >
> >I have several NFS servers, where the service must be available  
> >0-24. The servers are mounted read only on the clients and I've  
> >solved the problem of maintaining consistent inodes between them by  
> >rsyncing an UFS image and mounting it via md on the NFS servers.
> >The machines have a common IP address with CARP, so if one of them  
> >falls out, the other(s) can take over.
> >
> >This works nice, but rsyncing multi gigabyte files are becoming more  
> >and more annoying, so I've wondered whether it would be possible to  
> >get constant inodes between machines via alternative ways.
> 
> 
> Why not avoid syncing multi-gigabyte files by splitting your huge FS  
> image into many smaller say 512MB files, then use md and geom concat/ 
> stripe/etc to make them all one image that you mount?

Where would be the positive effect by doing this?
FFS distributes data over the media, so all the small files changes
in almost every case and you have to checksum-compare the whole virtual
disk anyway.
With multiple files the syncing is more complex. For example a normal
rsync run can garantie that you get a complete file synced or none
at all, but this doesn't work out of the box with multiple files, so
you risk half updated data.
(Continue reading)

Permalink | Reply | 4 comments |
headers
Ruslan Kovtun | 3 May 2008 17:55
Picon

Re: Choppy performance.

Sorry, maybe I miss something.
What "memory allocation errors in rtorrent" do you mean?

> But if it isn't really using that much memory how come I get
> memory allocation errors in rtorrent if there's more memory
> avaliable?

One week ago was observed problem with write speed on ZFS pool with following 
configuration on i386:
vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"
KVA_PAGES=512
Write speed in 8 disks (raidz) is 40 Mb/sec and very choppy. 
If I change to vm.kmem_size_max="999M", write speed increase in 4 times 
(160Mb/sec). I think this is bug. 
What is yours configuration? 

--

-- 
________________
С уважением
Ковтун Руслан mailto <yalur <at> mail.ru>
_______________________________________________
freebsd-fs <at> freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe <at> freebsd.org"
Permalink | Reply | 4 comments |
headers
Attila Nagy | 3 May 2008 20:09
Picon

Re: Consistent inodes between distinct machines

Hello,

On 2008.05.03. 14:50, Bernd Walter wrote:
> On Fri, May 02, 2008 at 03:40:11PM -0500, Eric Anderson wrote:
>   
>> On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote:
>>
>>     
>>> Hello,
>>>
>>> I have several NFS servers, where the service must be available  
>>> 0-24. The servers are mounted read only on the clients and I've  
>>> solved the problem of maintaining consistent inodes between them by  
>>> rsyncing an UFS image and mounting it via md on the NFS servers.
>>> The machines have a common IP address with CARP, so if one of them  
>>> falls out, the other(s) can take over.
>>>
>>> This works nice, but rsyncing multi gigabyte files are becoming more  
>>> and more annoying, so I've wondered whether it would be possible to  
>>> get constant inodes between machines via alternative ways.
>>>       
>> Why not avoid syncing multi-gigabyte files by splitting your huge FS  
>> image into many smaller say 512MB files, then use md and geom concat/ 
>> stripe/etc to make them all one image that you mount?
>>     
>
> Where would be the positive effect by doing this?
> FFS distributes data over the media, so all the small files changes
> in almost every case and you have to checksum-compare the whole virtual
> disk anyway.
(Continue reading)

Permalink | Reply | 2 comments |
headers
Bernd Walter | 3 May 2008 20:51
Picon

Re: Consistent inodes between distinct machines

On Sat, May 03, 2008 at 08:09:25PM +0200, Attila Nagy wrote:
> Hello,
> 
> On 2008.05.03. 14:50, Bernd Walter wrote:
> >On Fri, May 02, 2008 at 03:40:11PM -0500, Eric Anderson wrote:
> >  
> >>On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote:
> >Nevertheless I think that the UFS/NFS combo is not very good for this
> >problem.
> >  
> I don't think so. I need a stable system and UFS/NFS is in that state in 
> FreeBSD.

ZFS is pretty stable as well, although it has some points you need
to care and tune about.

> >With ZFS send/receive however inode numbers are consistent.
> >  
> Yes, they are, but the filesystem IDs are not, so you cannot have CARP 
> failover for the NFS servers, because all clients will have ESTALE 
> errors on everything.

Havn't though about this.
Of course this is a real problem.
Have you tried the following:
Setup Server A with all required ZFS filesystems.
Replicate everything to Server B using dd.
Then the filesystem ID should be the same on both systems.
This will not work for newly created filesystems however and you may
need to take extra care about not accidently change disks between the
(Continue reading)

Permalink | Reply | 1 comment |
headers
Attila Nagy | 3 May 2008 21:53
Picon

Re: Consistent inodes between distinct machines

On 2008.05.03. 20:51, Bernd Walter wrote:
> On Sat, May 03, 2008 at 08:09:25PM +0200, Attila Nagy wrote:
>   
>> Hello,
>>
>> On 2008.05.03. 14:50, Bernd Walter wrote:
>>     
>>> On Fri, May 02, 2008 at 03:40:11PM -0500, Eric Anderson wrote:
>>>  
>>>       
>>>> On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote:
>>>>         
>>> Nevertheless I think that the UFS/NFS combo is not very good for this
>>> problem.
>>>  
>>>       
>> I don't think so. I need a stable system and UFS/NFS is in that state in 
>> FreeBSD.
>>     
>
> ZFS is pretty stable as well, although it has some points you need
> to care and tune about.
>   
I have (had, switched back one to UFS) two machines with ZFS. One i386 
and one amd64. Both kept crashing or freezing, so I don't consider ZFS 
pretty stable ATM. :(

> Havn't though about this.
> Of course this is a real problem.
> Have you tried the following:
(Continue reading)

Permalink | Reply | 0 comments |
headers
Daniel Andersson | 4 May 2008 11:03
Picon

Re: Choppy performance.

> Sorry, maybe I miss something.
>  What "memory allocation errors in rtorrent" do you mean?

"Storage error: [File chunk write error: Cannot allocate memory]"
"Storage error: [File chunk read error: Cannot allocate memory]"

It happens with big torrents ~40gb+. It probably happens
because I have:
send_buffer_size=1M
receive_buffer_size=1M

1MB send/receive buffers plus big torrent chunks eats the memory
pretty fast. I have it at 1M to lessen the disk load. But maybe I
don't need it that high anymore with two zfs disks?

>
>
>  > But if it isn't really using that much memory how come I get
>  > memory allocation errors in rtorrent if there's more memory
>  > avaliable?
>
>  One week ago was observed problem with write speed on ZFS pool with following
>  configuration on i386:
>
> vm.kmem_size_max="1073741824"
>  vm.kmem_size="1073741824"
>  KVA_PAGES=512
>  Write speed in 8 disks (raidz) is 40 Mb/sec and very choppy.
>  If I change to vm.kmem_size_max="999M", write speed increase in 4 times
>  (160Mb/sec). I think this is bug.
(Continue reading)

Permalink | Reply | 3 comments |
headers
Gary Jennejohn | 4 May 2008 16:08
Picon
Favicon

Re: Choppy performance.

On Sun, 4 May 2008 11:03:30 +0200
"Daniel Andersson" <engywook <at> gmail.com> wrote:

> > vm.kmem_size_max="1073741824"
> >  vm.kmem_size="1073741824"
> >  KVA_PAGES=512
> >  Write speed in 8 disks (raidz) is 40 Mb/sec and very choppy.
> >  If I change to vm.kmem_size_max="999M", write speed increase in 4 times
> >  (160Mb/sec). I think this is bug.
> >  What is yours configuration?
> >
> My loader.conf(AMD64):
> 
> vfs.zfs.prefetch_disable=1
> vm.kmem_size_max="1073741824"
> vm.kmem_size="1073741824"
> 
> I'll try setting it to 999M instead, thanks!
> 

I just tried setting it to 999M for kicks (amd64) and saw no significant
speed up doing a ``make -j4 buildworld''.  I only saw a time difference
of 20 seconds between 1024M and 999M, which is just in the noise.

---
Gary Jennejohn
_______________________________________________
freebsd-fs <at> freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe <at> freebsd.org"
(Continue reading)

Permalink | Reply | 2 comments |
headers
Daniel Andersson | 4 May 2008 16:57
Picon

Re: Choppy performance.

>  I just tried setting it to 999M for kicks (amd64) and saw no significant
>  speed up doing a ``make -j4 buildworld''.  I only saw a time difference
>  of 20 seconds between 1024M and 999M, which is just in the noise.
>
>  ---
>  Gary Jennejohn

What kind of speeds do you get when/if you ftp stuff? I seem to have
trouble getting above 30MB/s. What kind of setup do you have?
How many disks? raidz?

Cheers;
Daniel
_______________________________________________
freebsd-fs <at> freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe <at> freebsd.org"
Permalink | Reply | 0 comments |

Gmane