Chris Murphy | 20 Apr 22:18 2014

btrfs fi df "unknown"

kernel 3.15.0-0.rc1.git0.1.fc21.x86_64
btrfs-progs v3.14

One 80GB virtual disk, formatted btrfs by installer and Fedora Rawhide installed to it. Post-install I see:

[root <at> localhost ~]# btrfs fi show
Label: 'fedora'  uuid: d372e5d1-386f-460c-b036-611469e0155e
	Total devices 1 FS bytes used 4.00GiB
	devid    1 size 79.31GiB used 6.04GiB path /dev/sda3

Btrfs v3.14
[root <at> localhost ~]# btrfs fi df /
Data, single: total=4.01GiB, used=3.79GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=1.00GiB, used=215.69MiB
Metadata, single: total=8.00MiB, used=0.00
unknown, single: total=80.00MiB, used=0.00

What is unknown?

Chris Murphy--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Marc MERLIN | 20 Apr 21:59 2014

Do quota groups cost noticeable performance in 3.14?

I was looking at using qgroups for my backup server, which will be
filled with millions of files in subvolumes with snapshots.

I read a warning that quota groups had performance issues, at least in
the past.

Is it still true? 
If there is a performance issue, is it as simple as just turning off
quota support and then things go back to normal?

Thanks,
Marc
--

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Marc MERLIN | 20 Apr 21:46 2014

Planning for subvolumes of subvolumes and btrfs send/receive

Can you help me design this right?

Long story short, I'm wondering if I can use btrfs send to copy sub
subvolumes (by snapshotting a parent subvolume, and hopefully getting
all the children underneath). My reading so far, says no.

So, Ideally I would have:

/mnt/btrfs_pool/backup: backup is a subvolume of btrfs_pool

/mnt/btrfs_pool/backup/debian/machine1: 
/mnt/btrfs_pool/backup/debian/machine2: 
/mnt/btrfs_pool/backup/debian/machine3: 
machinex would be a subvolume ideally speaking for quota counting purposes

/mnt/btrfs_pool/backup/debian/machine1/current
/mnt/btrfs_pool/backup/debian/machine1/201401
/mnt/btrfs_pool/backup/debian/machine1/201308
...

current would be a subvolume, that would be snapshotted to 201401 with
an rsync of diffs on top
Same thing for 201308

In other words, they'd also be subvolumes, and I'm using snapshots so
that I can collapse the identical data (I'm currently using hardlinks
for that).

This would all work fine from what I can see, but from here, if I try to
do 
(Continue reading)

Adam Brenner | 20 Apr 19:27 2014

Slow Write Performance w/ No Cache Enabled and Different Size Drives

Howdy,

I recently setup a new BTRFS filesystem based on BTRFS version 3.12 on
Linux kernel 3.13-1 running Debian Jessie.

The BTRFS volume spans 3x 4TB disks, two of which are using the entire
raw block device, and one of them is using a partition (OS disks). The
setup is like so:

     root <at> gra-dfs:/data/tmp# btrfs filesystem show
     Label: none  uuid: 63d51c9b-f851-404f-b0f2-bf84d07df163
	    Total devices 3 FS bytes used 3.03TiB
	    devid    1 size 3.61TiB used 1.01TiB path /dev/sda3
	    devid    2 size 3.64TiB used 1.04TiB path /dev/sdb
	    devid    3 size 3.64TiB used 1.04TiB path /dev/sdc

     Btrfs v3.12
     root <at> gra-dfs:/data/tmp# btrfs filesystem df /data
     Data, single: total=3.07TiB, used=3.03TiB
     System, RAID1: total=8.00MiB, used=352.00KiB
     System, single: total=4.00MiB, used=0.00
     Metadata, RAID1: total=5.00GiB, used=3.60GiB
     Metadata, single: total=8.00MiB, used=0.00
     root <at> gra-dfs:/data/tmp#
     root <at> gra-dfs:/home# mount | grep /data
     /dev/sda3 on /data type btrfs (rw,noatime,space_cache)
     root <at> gra-dfs:/home#

The setup is supposed to be "RAID-0 like" but with different size drives
within the volume, I created the BTRFS filesystem using the following
(Continue reading)

Rakesh Pandit | 20 Apr 15:17 2014

[PATCH] Btrfs-progs: fsck: fix double free memory crash

Fix double free of memory if btrfs_open_devices fails:
*** Error in `btrfs': double free or corruption (fasttop): 0x000000000066e020 ***

Crash happened because when open failed on device inside
btrfs_open_devices it freed all memory by calling btrfs_close_devices but
inside disk-io.c we call btrfs_close_again it again.

Signed-off-by: Rakesh Pandit <rakesh <at> tuxera.com>
---
 disk-io.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/disk-io.c b/disk-io.c
index 19b95a7..8db0335 100644
--- a/disk-io.c
+++ b/disk-io.c
 <at>  <at>  -1091,8 +1091,7  <at>  <at>  static struct btrfs_fs_info *__open_ctree_fd(int fp, const char *path,

 	ret = btrfs_open_devices(fs_devices, oflags);
 	if (ret)
-		goto out_devices;
-
+		goto out;

 	disk_super = fs_info->super_copy;
 	if (!(flags & OPEN_CTREE_RECOVER_SUPER))
--

-- 
1.8.5.3

--
(Continue reading)

Filipe David Borba Manana | 20 Apr 16:07 2014
Picon

[PATCH 6/6] Btrfs: add send_stream_version attribute to sysfs

So that applications can find out what's the highest send stream
version supported/implemented by the running kernel:

    $ cat /sys/fs/btrfs/send_stream_version
    2

Signed-off-by: Filipe David Borba Manana <fdmanana <at> gmail.com>
---
 fs/btrfs/send.h  |  1 +
 fs/btrfs/sysfs.c | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h
index 987936c..047fd6d 100644
--- a/fs/btrfs/send.h
+++ b/fs/btrfs/send.h
 <at>  <at>  -22,6 +22,7  <at>  <at> 
 #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream"
 #define BTRFS_SEND_STREAM_VERSION_1 1
 #define BTRFS_SEND_STREAM_VERSION_2 2
+#define BTRFS_SEND_STREAM_VERSION_LATEST BTRFS_SEND_STREAM_VERSION_2

 #define BTRFS_SEND_BUF_SIZE (1024 * 64)
 #define BTRFS_SEND_READ_SIZE (1024 * 48)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 58a1dd1..2f8fff6 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
 <at>  <at>  -31,6 +31,7  <at>  <at> 
 #include "transaction.h"
(Continue reading)

Filipe David Borba Manana | 20 Apr 16:06 2014
Picon

[PATCH 5/6] Btrfs: add missing cleanup on sysfs init failure

If we failed during initialization of sysfs, we weren't unregistering the
top level btrfs sysfs entry nor the debugfs stuff.
Not unregistering the top level sysfs entry makes future attempts to reload
the btrfs module impossible and the following is reported in dmesg:

[ 2246.451296] WARNING: CPU: 3 PID: 10999 at fs/sysfs/dir.c:486 sysfs_warn_dup+0x91/0xb0()
[ 2246.451298] sysfs: cannot create duplicate filename '/fs/btrfs'
[ 2246.451298] Modules linked in: btrfs(+) raid6_pq xor bnep rfcomm bluetooth binfmt_misc nfsd
auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc parport_pc parport psmouse serio_raw
pcspkr evbug i2c_piix4 e1000 floppy [last unloaded: btrfs]
[ 2246.451310] CPU: 3 PID: 10999 Comm: modprobe Tainted: G        W    3.13.0-fdm-btrfs-next-24+ #7
[ 2246.451311] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 2246.451312]  0000000000000009 ffff8800d353fa08 ffffffff816f1da6 0000000000000410
[ 2246.451314]  ffff8800d353fa58 ffff8800d353fa48 ffffffff8104a32c ffff88020821a290
[ 2246.451316]  ffff88020821a290 ffff88020821a290 ffff8802148f0000 ffff8800d353fb80
[ 2246.451318] Call Trace:
[ 2246.451322]  [<ffffffff816f1da6>] dump_stack+0x4e/0x68
[ 2246.451324]  [<ffffffff8104a32c>] warn_slowpath_common+0x8c/0xc0
[ 2246.451325]  [<ffffffff8104a416>] warn_slowpath_fmt+0x46/0x50
[ 2246.451328]  [<ffffffff81367dc5>] ? strlcat+0x65/0x90
(....)

Signed-off-by: Filipe David Borba Manana <fdmanana <at> gmail.com>
---
 fs/btrfs/sysfs.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index c5eb214..58a1dd1 100644
--- a/fs/btrfs/sysfs.c
(Continue reading)

Filipe David Borba Manana | 20 Apr 16:05 2014
Picon

[PATCH 4/6 v4] Btrfs: send, use fallocate command to allocate extents

The send stream version 2 adds the fallocate command, which can be used to
allocate extents for a file or punch holes in a file. Previously we were
ignoring file prealloc extents or treating them as extents filled with 0
bytes and sending a regular write command to the stream.

After this change, together with my previous change titled:

    "Btrfs: send, use fallocate command to punch holes"

an incremental send preserves the hole and data structure of files, which can
be seen via calls to lseek with the whence parameter set to SEEK_DATA or SEEK_HOLE,
as the example below shows:

    mkfs.btrfs -f /dev/sdc
    mount /dev/sdc /mnt
    xfs_io -f -c "pwrite -S 0x01 -b 300000 0 300000" /mnt/foo
    btrfs subvolume snapshot -r /mnt /mnt/mysnap1

    xfs_io -c "fpunch 100000 50000" /mnt/foo
    xfs_io -c "falloc 100000 50000" /mnt/foo
    xfs_io -c "pwrite -S 0xff -b 1000 120000 1000" /mnt/foo
    xfs_io -c "fpunch 250000 20000" /mnt/foo

    # prealloc extents that start beyond the inode's size
    xfs_io -c "falloc -k 300000 1000000" /mnt/foo
    xfs_io -c "falloc -k 9000000 2000000" /mnt/foo

    btrfs subvolume snapshot -r /mnt /mnt/mysnap2

    btrfs send /mnt/mysnap1 -f /tmp/1.snap
(Continue reading)

Filipe David Borba Manana | 20 Apr 16:04 2014
Picon

[PATCH 3/6 v4] Btrfs: send, use fallocate command to punch holes

Instead of sending a write command with a data buffer filled with 0 value bytes,
use the fallocate command, introduced in the send stream version 2, to tell the
receiver to punch a file hole using the fallocate system call.

Signed-off-by: Filipe David Borba Manana <fdmanana <at> gmail.com>
---

V2: A v2 stream is now only produced if the send ioctl caller passes in one of
    the new flags (BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE | BTRFS_SEND_FLAG_SUPPORT_FALLOCATE)
    to avoid breaking old clients.
V3: Added missing path allocation, messed up rebase.
V4: Removed BTRFS_SEND_FLAG_SUPPORT_FALLOCATE and added BTRFS_SEND_FLAG_STREAM_V2,
    added commands for inode set flags and otime.

 fs/btrfs/send.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
 fs/btrfs/send.h |  4 ++++
 2 files changed, 56 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 2a52cc9..e57000b 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
 <at>  <at>  -564,6 +564,7  <at>  <at>  static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len)
 		return tlv_put(sctx, attr, &__tmp, sizeof(__tmp));	\
 	}

+TLV_PUT_DEFINE_INT(32)
 TLV_PUT_DEFINE_INT(64)

 static int tlv_put_string(struct send_ctx *sctx, u16 attr,
(Continue reading)

Filipe David Borba Manana | 20 Apr 16:03 2014
Picon

[PATCH 2/6 v3] Btrfs: send, implement total data size command to allow for progress estimation

This new send flag makes send calculate first the amount of new file data (in bytes)
the send root has relatively to the parent root, or for the case of a non-incremental
send, the total amount of file data the stream will create (including holes and prealloc
extents). In other words, it computes the sum of the lengths of all write, clone and
fallocate operations that will be sent through the send stream.

This data size value is sent in a new command, named BTRFS_SEND_C_TOTAL_DATA_SIZE, that
immediately follows a BTRFS_SEND_C_SUBVOL or BTRFS_SEND_C_SNAPSHOT command, and precedes
any command that changes a file or the filesystem hierarchy. Upon receiving a write, clone
or fallocate command, the receiving end can increment a counter by the data length of that
command and therefore report progress by comparing the counter's value with the data size
value received in the BTRFS_SEND_C_TOTAL_DATA_SIZE command.

The approach is simple, before the normal operation of send, do a scan in the file system
tree for new inodes and new/changed file extent items, just like in send's normal operation,
and keep incrementing a counter with new inodes' size and the size of file extents (and file
holes)  that are going to be written, cloned or fallocated. This is actually a simpler and
more lightweight tree scan/processing than the one we do when sending the changes, as it
doesn't process inode references nor does any lookups in the extent tree for example.

After modifying btrfs-progs to understand this new command and report progress, here's an
example (the -o flag tells btrfs send to pass the new flag to the kernel's send ioctl):

    $ btrfs send -s --stream-version 2 /mnt/sdd/snap_base | btrfs receive /mnt/sdc
    At subvol /mnt/sdd/snap_base
    At subvol snap_base
    About to receive 9212392667 bytes
    Subvolume /mnt/sdc//snap_base, 4059722426 / 9212392667 bytes received, 44.07%, 40.32MB/s

    $ btrfs send -s --stream-version 2 -p /mnt/sdd/snap_base /mnt/sdd/snap_incr | btrfs receive /mnt/sdc
(Continue reading)

Michael Welsh Duggan | 20 Apr 00:00 2014

snapshot send with parent question

Assume the following scenario:
There exists a read-only snapshot called a.
A read-write snapshot called b is created from a, and is then modified.
A read-only snapshot of b is created, called c.
A btrfs send is done for c, with a marked as its parent.

Will the send data only contain the differences between a and c?  My
experiments seem to indicate no, but I have no confidence that I am not
doing something else correctly.

Also, when a btrfs receive gets a stream containing the differences
between a (parent) and c, does it only look at the relative pathname
differences between a and c in order to determine the matching parent on
the receiving side?

--

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Gmane