Pino Toscano | 31 Mar 17:32 2015
Picon

[PATCH] WIP: New virt-dib tool

virt-dib is a new tool to run the elements of diskimage-builder using
libguestfs.
---

Work in progress (debug stuff here and there), although I've submitting
it for initial review, for being included later when polished for good.

TODO items open:
 - move Uname from builder to mllib
 - improve the documentation
 - review
 - getting more testing (although it has been successfully tested for
   a while, not only by myself)
 - polish


 .gitignore               |   6 +
 Makefile.am              |   1 +
 appliance/packagelist.in |   4 +
 configure.ac             |   3 +
 dib/Makefile.am          | 148 ++++++++
 dib/cmdline.ml           | 247 +++++++++++++
 dib/dib.ml               | 930 +++++++++++++++++++++++++++++++++++++++++++++++
 dib/elements.ml          | 219 +++++++++++
 dib/link.sh.in           |  22 ++
 dib/uname-c.c            |  57 +++
 dib/uname.ml             |  27 ++
 dib/uname.mli            |  28 ++
 dib/utils.ml             | 148 ++++++++
 dib/virt-dib.pod         | 541 +++++++++++++++++++++++++++
(Continue reading)

Chen Hanxiao | 31 Mar 04:15 2015

[PATCH] actions.ml: fix a typo

s/inclding/including

Signed-off-by: Chen Hanxiao <chenhanxiao <at> cn.fujitsu.com>
---
 generator/actions.ml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/generator/actions.ml b/generator/actions.ml
index bd654a3..11ea436 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
 <at>  <at>  -12262,7 +12262,7  <at>  <at>  Destroy a quota group." };
     optional = Some "btrfs"; camel_name = "BTRFSQgroupShow";
     shortdesc = "show subvolume quota groups";
     longdesc = "\
-Show all subvolume quota groups in a btrfs filesystem, inclding their
+Show all subvolume quota groups in a btrfs filesystem, including their
 usages." };

   { defaults with
--

-- 
2.1.0
Chen Hanxiao | 30 Mar 11:39 2015

[PATCH RFC] resize: add p_mbr_p_type as member of type partition

Add p_mbr_p_type as member of type partition
to describe mbr partition type.

Currently we use:
List.filter (fun p -> parttype <> MBR || p.G.part_num <= 4_l)
to filter out logical partitions.

Commit 0c396a4bce578486dfc4a38e1f8c47fd5c2836ea
introduce API part_get_mbr_part_type,
we could use this to know the part_type.
Furthermore, we could also use p_mbr_p_type for resizing
logical partitions.

Signed-off-by: Chen Hanxiao <chenhanxiao <at> cn.fujitsu.com>
---
 resize/resize.ml | 47 +++++++++++++++++++++++++++++++----------------
 1 file changed, 31 insertions(+), 16 deletions(-)

diff --git a/resize/resize.ml b/resize/resize.ml
index 84fd6d4..284d0e3 100644
--- a/resize/resize.ml
+++ b/resize/resize.ml
 <at>  <at>  -57,6 +57,7  <at>  <at>  type partition = {
   p_target_partnum : int;        (* TARGET partition number. *)
   p_target_start : int64;        (* TARGET partition start (sector num). *)
   p_target_end : int64;          (* TARGET partition end (sector num). *)
+  p_mbr_p_type : partition_type  (* Partiton Type (master/extended/logical) *)
 }
 and partition_content =
   | ContentUnknown               (* undetermined *)
(Continue reading)

Maros Zatko | 26 Mar 16:47 2015
Picon

[PATCH v3] customize: fix --upload to FAT partition (RHBZ#1196101)

Some filesystems, such as FAT doesn't support file
ownership, so show warning instead of error on EPERM.

Maros Zatko (1):
  customize: fix --upload to FAT partition (RHBZ#1196101)

 customize/customize_run.ml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

--

-- 
1.9.3
Maros Zatko | 25 Mar 18:07 2015
Picon

[PACTH v2] customize: fix --upload to FAT partition

FAT doesn't support file ownership, so show warning instead
of error.

Fixes RHBZ#1196101

Maros Zatko (1):
  customize: fix --upload to FAT partition (RHBZ#1196101)

 customize/customize_run.ml | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

--

-- 
1.9.3
Maros Zatko | 25 Mar 15:41 2015
Picon

[PATCH] customize: fix --upload to FAT partition

FAT doesn't support file ownership, so show warning instead
of error.

Fixes RHBZ#1196101

Maros Zatko (1):
  customize: fix --upload to FAT partition (RHBZ#1196101)

 customize/customize_run.ml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

--

-- 
1.9.3
Chen Hanxiao | 25 Mar 10:04 2015

[PATCH] New API: ls_i

Signed-off-by: Hu Tao <hutao <at> cn.fujitsu.com>
Signed-off-by: Chen Hanxiao <chenhanxiao <at> cn.fujitsu.com>
---
 daemon/ls.c          | 24 ++++++++++++++++++++++++
 generator/actions.ml |  7 +++++++
 src/MAX_PROC_NR      |  2 +-
 3 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/daemon/ls.c b/daemon/ls.c
index d3689cd..fdacdaf 100644
--- a/daemon/ls.c
+++ b/daemon/ls.c
 <at>  <at>  -148,3 +148,27  <at>  <at>  do_llz (const char *path)

   return out;			/* caller frees */
 }
+
+char *
+do_ls_i (const char *dir)
+{
+  int r;
+  char *out;
+  CLEANUP_FREE char *err = NULL;
+  CLEANUP_FREE char *sdir = NULL;
+
+  sdir = sysroot_path (dir);
+  if (!sdir) {
+    reply_with_perror ("malloc");
+    return NULL;
+  }
(Continue reading)

Chen Hanxiao | 24 Mar 12:20 2015

[PATCH 0/2] New API: part_get_part_type


Chen Hanxiao (2):
  parted: introduce enum for whether parted has option -m
  New API: part_get_part_type for showing partition type

 daemon/parted.c      | 136 ++++++++++++++++++++++++++++++++++++++++++++++-----
 generator/actions.ml |  18 +++++++
 src/MAX_PROC_NR      |   2 +-
 3 files changed, 143 insertions(+), 13 deletions(-)

--

-- 
2.1.0
NoxDaFox | 24 Mar 09:54 2015
Picon

Fwd: Inspection of disk snapshots

I was sure I was doing something wrong as I'm not yet fully aware of QCOW2 snapshot feature and how it interacts with libguestfs.

I'll try to explain better the scenario:

I have several hosts running lots of VMs which are generated from few base images, say A, B, C the base images (backing file) and A1, A2, A*, B1, B2, B* clones on top of which the newly spawned VMs are running.
I need to collect the disk states of A*, B*, C* machines and see what has been written there. I don't care about the whole content as the base images content A, B, C are well known to me, only thing it matters are the deltas of the new clones.

One more piece in the puzzle is that the inspection does not happen on the hosts running the VMs but on a dedicated server.

My idea was to collect those "snapshots" (generic term not the QEMU one) from the hosts and send them to my inspection server. As A, B and C are accessible from that server only thing I need is to rebase those snapshot to correctly inspect them through libguestfs, and it proved to work (I'm using readonly mode as I only care about reading the disks). I'm not really interested in having consistent point-in-time state of the disks as the operation is done several times a day so I can cope with semi-consistent data as it can be easily re-constructed.

My real problem comes when I try to inspect the disk snapshot: libguestfs will, of course, let me see the whole content of the disks, which means A + A*. Apart from the waste of CPU time spend on looking at files I already know the state (the ones contained in A), it generates a lot of noise. A Linux base image with some library installed consists in 20+ K files, installing something extra (Apache server for example) just brings some hundreds new files and I'm interested only in those ones.

So my real question is: is there a way to distinguish the files contained in the two different disk images (A and A1) or shall I think about a totally different approach?

Thank you.


2015-03-24 0:43 GMT+02:00 Richard W.M. Jones <rjones <at> redhat.com>:
On Mon, Mar 23, 2015 at 10:41:01PM +0000, Richard W.M. Jones wrote:
> On Mon, Mar 23, 2015 at 04:34:21PM +0200, NoxDaFox wrote:
> > Greetings,
> >
> > I have the following typical scenario: given one or more qcow2 base images
> > I clone them with COW and start the VMs.
> >
> > At a certain point I'd like to inspect them in order to see their evolution
> > compared to the known base images. To do so I was thinking about taking a
> > disk snapshot of each VM and inspect its content through libguestfs (using
> > it's Python bindings).
> >
> > Obviously I need the base image in order for libguestfs to correctly guess
> > the OS, the FS structure etc.. Problem is that that point when I inspect
> > the disk I get the whole disk state including the base image content (30K+
> > files and directories).
> >
> > This is not an issue but it's a very heavy operation considering that some
> > of the snapshots are few megabytes while the base images are several
> > gigabytes.
> >
> > Is there a way to programmatically instruct libguestfs to limit the
> > inspection to the sole snapshot?
> > Would it work as well with other disk format (vmdk linked clones for
> > example)?
> >
> > For better comprehension I'll show the sequence I'm doing (I might do it
> > wrong of course):
> >
> > virsh -c "qemu:///system" snapshot-create --disk-only <domain-ID>
> >
> > I get the snapshot location from its XML description and then:
> >
> > qemu-img convert -f qcow2 -O qcow2 base_image.qcow2 snapshot.qcow2
>
> This makes a copy of the whole disk image.  It's also not a consistent
> (point in time) copy.

Oh I see that you're copying the _snapshot_ that you created with
libvirt; it's not a whole disk copy.  There's still not any point in
doing this, and what I said below stands.

> > At that point I mount it through libguestfs and inspect its content.
>
> As long as you use the 'readonly=1' flag (which is really *essential*,
> else you'll get disk corruption), you can just point libguestfs at the
> base image:
>
>   g = guestfs.GuestFS (python_return_dict=True)
>   g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1)
>
> That also doesn't get you a consistent snapshot, but it'll work most
> of the time, and give you a clear error in libguestfs when it doesn't
> (and won't corrupt your base disk or anything like that, provided
> you're using readonly=1).
>
> The effect of the readonly=1 flag is to create an external snapshot.
> It is roughly the equivalent of doing:
>
>   qemu-img create -f qcow2 -b base_image snapshot.qcow2
>   < point libguestfs at snapshot.qcow2 >
>
> If you want lightweight, consistent, point-in-time snapshots (which it
> sounds like you do), qemu has recently been adding this capability.
> See the 'drive-backup' monitor command.  I've not tried using that and
> I don't know if it is wired up through libvirt, but libguestfs should
> be able to consume it since it's just an NBD source.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top


_______________________________________________
Libguestfs mailing list
Libguestfs <at> redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
Chen Hanxiao | 24 Mar 03:51 2015

[PATCH v3] New API: btrfs-image

Signed-off-by: Chen Hanxiao <chenhanxiao <at> cn.fujitsu.com>
---
v3: remove optional paramter numthreads(-t)
v2: add optargs_bitmask check

 daemon/btrfs.c       | 43 +++++++++++++++++++++++++++++++++++++++++++
 generator/actions.ml | 22 ++++++++++++++++++++++
 src/MAX_PROC_NR      |  2 +-
 3 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/daemon/btrfs.c b/daemon/btrfs.c
index fabb00b..39392f7 100644
--- a/daemon/btrfs.c
+++ b/daemon/btrfs.c
 <at>  <at>  -37,6 +37,7  <at>  <at>  GUESTFSD_EXT_CMD(str_btrfstune, btrfstune);
 GUESTFSD_EXT_CMD(str_btrfsck, btrfsck);
 GUESTFSD_EXT_CMD(str_mkfs_btrfs, mkfs.btrfs);
 GUESTFSD_EXT_CMD(str_umount, umount);
+GUESTFSD_EXT_CMD(str_btrfsimage, btrfs-image);

 int
 optgroup_btrfs_available (void)
 <at>  <at>  -2040,3 +2041,45  <at>  <at>  do_btrfstune_enable_skinny_metadata_extent_refs (const char *device)

   return 0;
 }
+
+int
+do_btrfs_image (char *const *sources, const char *image,
+	int compresslevel)
+{
+  size_t nr_sources =  count_strings (sources);
+  const size_t MAX_ARGS = 64 + nr_sources;
+  const char *argv[MAX_ARGS];
+  size_t i = 0, j;
+  CLEANUP_FREE char *err = NULL;
+  CLEANUP_FREE char *out = NULL;
+  char compresslevel_s[64];
+  int r;
+
+  if (nr_sources == 0) {
+      reply_with_error ("list of sources must be non-empty");
+      return -1;
+  }
+
+  ADD_ARG (argv, i, str_btrfsimage);
+
+  if ((optargs_bitmask & GUESTFS_BTRFS_IMAGE_COMPRESSLEVEL_BITMASK)
+    && compresslevel >= 0) {
+    snprintf (compresslevel_s, sizeof compresslevel_s, "%d", compresslevel);
+    ADD_ARG (argv, i, "-c");
+    ADD_ARG (argv, i, compresslevel_s);
+  }
+
+  for (j = 0; j < nr_sources; ++j)
+    ADD_ARG (argv, i, sources[j]);
+
+  ADD_ARG (argv, i, image);
+  ADD_ARG (argv, i, NULL);
+
+  r = commandv (&out, &err, argv);
+  if (r == -1) {
+    reply_with_error ("%s %s: %s", sources[0], image, err);
+    return -1;
+  }
+
+  return 0;
+}
diff --git a/generator/actions.ml b/generator/actions.ml
index 786b9a2..5e462a8 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
 <at>  <at>  -12527,6 +12527,28  <at>  <at>  This will Enable extended inode refs." };
     longdesc = "\
 This enable skinny metadata extent refs." };

+  { defaults with
+    name = "btrfs_image";
+    style = RErr, [DeviceList "source"; Pathname "image"], [OInt "compresslevel"];
+    proc_nr = Some 453;
+    optional = Some "btrfs"; camel_name = "BTRFSImage";
+    tests = [
+      InitPartition, Always, TestRun (
+        [["part_init"; "/dev/sda"; "mbr"];
+         ["part_add"; "/dev/sda"; "p"; "64"; "204799"];
+         ["part_add"; "/dev/sda"; "p"; "204800"; "409599"];
+         ["mkfs_btrfs"; "/dev/sda1"; ""; ""; "NOARG"; ""; "NOARG"; "NOARG"; ""; ""];
+         ["mkfs_btrfs"; "/dev/sda2"; ""; ""; "NOARG"; ""; "NOARG"; "NOARG"; ""; ""];
+         ["mount"; "/dev/sda1"; "/"];
+         ["btrfs_image"; "/dev/sda2"; "/1.img"; ""];
+         ["btrfs_image"; "/dev/sda2"; "/2.img"; "2"]]), []
+    ];
+
+    shortdesc = "create an image of a btrfs filesystem";
+    longdesc = "\
+This is used to create an image of a btrfs filesystem.
+All data will be zeroed, but metadata and the like is preserved." };
+
 ]

 (* Non-API meta-commands available only in guestfish.
diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR
index 8670c73..534b992 100644
--- a/src/MAX_PROC_NR
+++ b/src/MAX_PROC_NR
 <at>  <at>  -1 +1  <at>  <at> 
-452
+453
--

-- 
2.1.0
Maros Zatko | 23 Mar 19:47 2015
Picon

[PATCH] customize: add --move

From: Maros Zatko <hacxman <at> gmail.com>

This adds --move SOURCE:DEST, equivalent of calling g#mv src dst.

RFE: RHBZ#1203817

Maros Zatko (1):
  customize: add --move

 builder/cmdline.ml         |  2 +-
 customize/customize_run.ml |  4 ++++
 generator/customize.ml     | 10 ++++++++++
 3 files changed, 15 insertions(+), 1 deletion(-)

--

-- 
1.9.3

Gmane