Kristen Eisenberg | 5 Nov 14:06 2011
Picon

High CPU Utilization When Copying to Ext4

Sorry if this is not the correct mailing list for ext4 questions.

Kristen Eisenberg
Billige Flüge
Marketing GmbH
Emanuelstr. 3,
10317 Berlin
Deutschland
Telefon: +49 (33)
5310967
Email:
utebachmeier at
gmail.com
Site:
http://flug.airego.de - Billige Flüge vergleichen

_______________________________________________
Ext3-users mailing list
Ext3-users <at> redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
Ted Ts'o | 5 Nov 18:21 2011
Picon
Picon

Re: High CPU Utilization When Copying to Ext4

On Sat, Nov 05, 2011 at 06:06:57AM -0700, Kristen Eisenberg wrote:
> Sorry if this is not the correct mailing list for ext4 questions.

You might get a better discussion going on the
linux-ext4 <at> vger.kernel.org developer's list, but I do listen in on
ext3-users.  What do you consider high CPU utilization, and how are
you writing to ext4?  And what kernel version are you using?

Regards,

						- Ted
Markus Feldmann | 26 Nov 15:15 2011
Picon
Picon

damaged encrypted LUKS device

Hi people,

i had created encrypted device with cryptsetup/LUKS which i setup with 
an ext4 filesystem. This device is an external USB harddisk. When i 
plugin this device it will be automatically mounted by my gnome3 system 
(Debian Wheezey/Testing), but this week i got an error. I did ask the 
ask the LUKS developers what is the problem and they told me that this 
is an ext4 problem.

The error message from gnome is:
> Einhängen von AIRY_1TB nicht möglich
>
> Error mounting: mount: wrong fs type, bad option, bad superblock on /dev/mapper/udisks-luks-uuid-70a2aedf-ce7e-4e12-8f0b-ec7974ebdbd4-uid1000,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so

And from <dmesg> i got the message:
> [ 3094.394043] EXT4-fs (dm-0): ext4_check_descriptors: Inode bitmap for group 3328 not in group (block 1235654634)!
> [ 3094.394056] EXT4-fs (dm-0): group descriptors corrupted!

Further on, when i try to <fsck -N /dev/sdb> i get:
> fsck from util-linux 2.19.1
> fsck: fsck.crypto_LUKS: not found
> fsck: Error 2 while executing fsck.crypto_LUKS for /dev/sdb

But cryptsetup is installed, only lvm2 is missing, but i dont have a LVM 
device and as i researched at "debian.org" there is no package which 
provides the command <fsck.crypto_LUKS>.

Any hints/instructions for me?

regards Markus

_______________________________________________
Ext3-users mailing list
Ext3-users <at> redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
Matija Nalis | 26 Nov 17:40 2011
Picon

Re: damaged encrypted LUKS device


On Sat, Nov 26, 2011 at 03:15:42PM +0100, Markus Feldmann wrote:
> And from <dmesg> i got the message:
> >[ 3094.394043] EXT4-fs (dm-0): ext4_check_descriptors: Inode bitmap for group 3328 not in group
(block 1235654634)!
> >[ 3094.394056] EXT4-fs (dm-0): group descriptors corrupted!
> 
> Further on, when i try to <fsck -N /dev/sdb> i get:
> >fsck from util-linux 2.19.1
> >fsck: fsck.crypto_LUKS: not found
> >fsck: Error 2 while executing fsck.crypto_LUKS for /dev/sdb
> 
> But cryptsetup is installed, only lvm2 is missing, but i dont have a
> LVM device and as i researched at "debian.org" there is no package
> which provides the command <fsck.crypto_LUKS>.
> 
> Any hints/instructions for me?

try "e2fsck -C0 /dev/dm-0" (assuming /dev/dm-0 is your block device 
after decryption).

And do full block-by-block backup of your external USB disk first! 
(using dd(1))

--

-- 
Opinions above are GNU-copylefted.
Markus Peuhkuri | 28 Nov 11:17 2011
Picon
Picon

Re: damaged encrypted LUKS device

On 26.11.2011 18:40, Matija Nalis wrote:
> And do full block-by-block backup of your external USB disk first! 
> (using dd(1)) 
And if it happens to abort on error, try http://safecopy.sourceforge.net 
(available on package in recent distros) that makes quite good work on 
reading disks with read errors.  Worked better than any 
dd{_recover,rescue}.  After doing the 3-stage copy, I was able copy 
files from ext4 partition inside LVM that was in qcow2 virtual machine 
image that was on ext3 partition on LVM on the failed disk.

t. Markus
Markus Feldmann | 28 Nov 15:42 2011
Picon
Picon

Re: damaged encrypted LUKS device

Am 26.11.2011 17:40, schrieb Matija Nalis:
>
> try "e2fsck -C0 /dev/dm-0" (assuming /dev/dm-0 is your block device
> after decryption).

How can i be sure, that my cryptsetup/LUKS harddisk is still decrypted?

<cryptsetup status /dev/dm-1> gives me:
> root <at> feld-bert:/home/markus# cryptsetup status /dev/dm-1
> /dev/mapper//dev/dm-1 is active.
>   type:    LUKS1
>   cipher:  aes-xts-plain
>   keysize: 256 bits
>   device:  /dev/sdb
>   offset:  4096 sectors
>   size:    1953521072 sectors
>   mode:    read/write
Bodo Thiesen | 28 Nov 16:58 2011
Picon
Picon

Re: damaged encrypted LUKS device

Hello Markus.

* Markus Feldmann <feldmann_markus <at> gmx.de> hat geschrieben:
>* Am 26.11.2011 17:40, schrieb Matija Nalis:
>> try "e2fsck -C0 /dev/dm-0" (assuming /dev/dm-0 is your block device
>> after decryption).
> How can i be sure, that my cryptsetup/LUKS harddisk is still decrypted?
> <cryptsetup status /dev/dm-1> gives me:
>> root <at> feld-bert:/home/markus# cryptsetup status /dev/dm-1
>> /dev/mapper//dev/dm-1 is active.

That line tells you ;)

So, try a e2fsck -n -C0 /dev/dm-1 (the -n causes no changes to be made
to the file system - errors will be reported only). If it's not too much,
but you're unsure what to do, maybe you want to post them here (use
command

	LANG=C e2fsck -n /dev/dm-1 > some-file.txt

for that and post the contents of some-file.txt). If it's too much maybe
you want to pastebin them and post the link here. In both cases, remember
to strip path names and similar data unless you don't care to publish them.
(However - why are you encrypting them then in the first place?)

Additionally, I suggest following Markus Peuhkuri's tip to first create a
block by block copy before doing anything else. I suggest copying the
encrypted version (i.e. /dev/sdb) not the decrypted one (i.e. /dev/dm-1).

Regards, Bodo
Markus Feldmann | 28 Nov 18:09 2011
Picon
Picon

Re: damaged encrypted LUKS device

Here comes my new output from <LANG=C e2fsck -n -C0 /dev/dm-1>

http://pastebin.com/raw.php?i=x0siBmrj

I will buy another 1 TByte harddisk these days and then make a 
byte-to-byte backup.

regards Markus
Bodo Thiesen | 28 Nov 19:26 2011
Picon
Picon

Re: damaged encrypted LUKS device

* Markus Feldmann <feldmann_markus <at> gmx.de> hat geschrieben:

> Here comes my new output from <LANG=C e2fsck -n -C0 /dev/dm-1>
> 
> http://pastebin.com/raw.php?i=x0siBmrj

Ok, the summary of that paste is:
| root <at> feld-bert:/home/markus# LANG=C e2fsck -n -C0 /dev/dm-1
| e2fsck 1.42-WIP (16-Oct-2011)
| e2fsck: Group descriptors look bad... trying backup blocks...
| Error writing block 1 (Attempt to write block to filesystem
|     resulted in short write). Ignore error? no
(more lines of this kind)
| e2fsck: Device or resource busy while trying to open /dev/dm-1
| Filesystem mounted or opened exclusively by another program?

And the bug report is here:
https://sourceforge.net/tracker/?func=detail&aid=3444351&group_id=2406&atid=102406

| 1. e2fsck should not try to write when using -n.
| 2. ignore error is not a question to be answered with "no" when
|    using -n but with "yes" instead.
| 3. I do not know what that "Device or resource busy" error is about
|    (someone else reported that problem on ext3-users <at> redhat.com, I'm
|    merely relaying that bug report), but it doesn't matter either: -n
|    means don't write and reading worked obviously.

> I will buy another 1 TByte harddisk these days and then make a
> byte-to-byte backup.

Do that. But before you start the copy, reboot your system, maybe there is
some processes waiting in disk-io on the problem disk (task state D in
process lists). At least, that's what
| e2fsck: Device or resource busy while trying to open /dev/dm-1
| Filesystem mounted or opened exclusively by another program?
suggest. However, that message may as well be caused by the other bug in
e2fsck.

Maybe it could be useful to init 1, wait for the X server and other
stuff to stop and then log in on the text console and do the copy there,
so that all that normally fancy but for this problem nasty auto mounting
stuff is not active when you connect the disk(s) to your system.

Regards, Bodo
Theodore Ts'o | 30 Nov 00:54 2011
Picon
Picon

E2fsprogs 1.42 is released!


I'm happy to announce the release of e2fsprogs 1.42.

You can download the tarball from here:

    http://prdownloads.sourceforge.net/e2fsprogs/e2fsprogs-1.42.tar.gz

Or you can download it from the git repository on git.kernel.org.

    git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git.

Or, you can wait for your favorite distribution to package it.  :-)

The release notes are attached below.

Many thanks to all of the contributors to e2fsprogs, whose employers
include Canonical, Ctera Networks, Fujitsu, Google, IBM, Red Hat, SuSE,
Tao Bao, Whamcloud, and of course to all of the users of e2fsprogs.
Many thanks for your support, bug reports, code contributions, and
translations over the years.

Regards,

				- Ted

E2fsprogs 1.42 (November 29, 2011)
==================================

This release of e2fsprogs has support for file systems > 16TB.  Online
resize requires kernel support which will hopefully be in Linux
version 3.2.  Offline support is not yet available for > 16TB file
systems, but will be coming.

This release of e2fsprogs has support for clustered allocation.  This
reduces the number of block (now cluster) bitmaps by allocating and
deallocating space in contiguous power-of-2 collections of blocks,
which are called clustered.  This is a file system level feature,
called 'bigalloc', which must be enabled when the file system is
initially formatted.  It is not backwards compatible with older
kernels.

Added support for the Multi-Mount Protection (MMP) feature.

E2fsck more efficiently uses scratch files for really big file
systems.  (This is a feature that has to be turned on explicitly; see
[scratch_files] in the e2fsck.conf man page.)

Fix a bug in e2fsck where if the free blocks and inodes counts are
incorrect, e2fsck would fix them without printing an error message.
This would cause a "*** FILE SYSTEM WAS MODIFIED ***" message without
any explanation of what was fixed.

E2fsck will no longer attempt to clone an extended attribute block in
pass1b handling if the file system does not support extended
attributes.

E2fsck will be more careful accidentally asking the user to continue
if the file system is mounted, so that an escape sequence won't cause
a false positive.  (Addresses Debian Bug: #619859)

E2fsck now uses less cpu time in pass 5 when large portions of the
bitmaps are uninitialized.

E2fsck will no longer segault when a corrupted file system has a bad
extent, and removing it leads to a block needing to be deallocated.
(Addresses SourceForge Bug: #2971800)

E2fsck will catch termination signals (segfaults, bus errors, sigfpe)
and print debugging information to make it easier to find potential
problems.

E2fsck will check to see if the bad block inode looks insane, and will
skip trying to use if it certain fields which should be zero look
non-zero.  This is to avoid a corrupted bad block inode causing e2fsck
to get confused and do more harm than good.

If e2fsck modifies certain superblock fields which the kernel doesn't
look at, it will now mark the superblock as dirty without marking the
file system as changed.  This avoids signalling the init scripts that
a reboot is necessary, since the kernel isn't going to look at those
fields, so it won't care if they have been changed.

Fixed a bug in the libext2fs library (in the binary search routine of
the icount abstraction) that could (very, very rarely) cause e2fsck to
die in the middle of pass 1 or pass 2 processing.

E2fsck will not try to do a discard operation if the -n option was
specified on the command line.

E2fsck now supports an extended "discard" option which will cause
e2fsck to attempt discard all unused blocks after a full, successful
file system check.

E2fsck will check for the bad block inode to make sure it looks sane
before trusting it, to avoid causing more harm than good to the file
system.

E2fsck now returns additional status bits in its exit code if it
aborts early in the e2fsck run.

E2fsck now correctly calculates the maximum file size in the case of
the huge_file file system feature enabled without extents.

The mke2fs and e2fsck programs now tries to use the punch hole command
as a "discard" when operating on normal files.

The e2image program now supports the qcow2 format, which is a more
efficient way of capturing file system dumps.

Mke2fs now supports the [devices] stanza in mke2fs.conf which allows
per-device defaults to be specified in the configuration file.

Mke2fs now supports the reserved_ratio relation in the [defaults] and
[fs_types] section in mke2fs.conf.

Mke2fs now creates extent-mapped directories for the root and
lost+found directories.

Mke2fs will skip zero'ing the journal if the extended option
"lazy_journal_init" is specified.  This can save a lot of time, but it
does add a small amount of risk if the system crashes before the
journal is overwritten entirely once.  It is epsecially useful for
testing.

Mke2fs will now create file systems that enable user namespace
extended attributes and with time- and mount count-based file
system checks disabled.

Mke2fs will not set a stride or strip size of one block based on block
device attributes obtained from sysfs.

Mke2fs now displays a progress report during the discard process.

Mke2fs now handles extreme file system parameters correctly which
previously caused the inodes per group to drop below 8, leading to a
segfault.  (The inodes per group must be a multiple of 8, but the code
didn't correctly deal with an inodes per group count less than 8.)

Mke2fs and tune2fs previously would give an error if the user tried
setting the stride and stripe-width parameters to zero; but this is
necessary to disable the stride and stripe-width settings.  So allow
setting these superblock fields to zero.  (Addresses Google Bug:
#4988557)

Mke2fs now gives a warning if the auto-detected block size exceeds the
system's page size.

If the enable_periodic_fsck option is false in /etc/mke2fs.conf (which
is the default), mke2fs will now set the s_max_mnt_count superblock
field to -1, instead of 0.  Kernels older then 3.0 will print a
spurious message on each mount then they see a s_max_mnt_count set to
0, which will annoy users.  (Addresses Debian Bug: #632637)

The default mke2fs.conf now has entries for "big" and "huge", which
are needed for very big file systems.

The resize2fs program now has support for a new online resize ioctl
that can support file systems > 16TB, once it arrives in v3.x kernels.

Fixed bug which caused resize2fs to fail when shrinking an empty file
system down to its minimal size.  (Addresses Sourceforge Bug #3404051)

Fixed tune2fs's mount options parsing.  (Addresses Debian Bug: #641667)

Allow tune2fs to remove external journals if the device can not be
found.

Debugfs's icheck will now correctly find inodes which use the
searched-for block as an extended attribute block.

Debugfs now has a new "punch" command which remove blocks from the
middle of an inode.

Debugfs now has a new "e2freefrag" command which analyzes the free
space fragmentation of the file system, using the same code as the
e2freefrag program.

Debugfs now has a "filefrag" command which displays information about
a file's fragmentation.

Add support to build a metadata-only, read-only, stripped-down version
of debugfs called rdebugfs.

Fixed a potential stack overrun bug in debugfs.

The badblocks program now correctly recovers from I/O errors when
direct I/O is being used.  The badblocks command now also supports a
-B option which forces the use of buffered I/O, and the -v option will
provide a more detailed breakdown of read, write, and failed
comparison errors.

Added e4defrag tool which uses the EXT4_IOC_MOVE_EXT ioctl.

Added support for journals larger than 2GB.

Support using both hard links and symlinks when installing e2fsprogs.

Add overflow checking to tune2fs -i's fsck interval, which must fit in
a 32-bit field.

The debugfs command now has a new 'blocks' command which prints out
data blocks of a particular inode in a format which is useful for
scripting.

Filefrag will report 0 extents correctly in verbose mode.  (Addresses
RedHat Bugzilla: #653234)

Filefrag has been fixed so its -v report prints the correct expected
block number (previously there had been an off-by-one error).  In
addition, it will now display the number of contiguous extents when -v
is not specified.  This makes it consistent with the number of extents
printed when the -v option was specified.  In addition, the number of
contiguous extents is far more interesting/useful than the number of
physical extents for very large files.  (Addresses Debian Bug:
#631498, #644792)

Logsave's usage message has been fixed.  (Addresses Debian Bug:
#619788)

Avoid an infinite loop in ext2fs_find_block_device() if there are
symlink loops in /dev caused by a buggy udev.

Added a useful "fallocate" program to the contrib directory.

Fixed an ABI compatibility problem in libext2fs which broke the dump
program.  Also added back some macros which dump needed so it could
compile against the latest version of ext2_fs.h (Addresses Debian Bug:
#636418)

Fixed parsing of MNTOPT_ options for tune2fs and debugfs (Addresses
Debian Bug: #641667)

Added internationalization support for libcom_err error table strings.

Fixed various spelling mistakes found in various output strings found
by I18N translators.

Update translations: French, Chinese, Germany, Indonesian, Swedish,
Vietnamese, Polish, Dutch, Czech.  (Addresses Debian Bugs: #520985,
#620659)

Fixed various Debian Packaging issues.  (Addresses Debian Bugs:
#614662, #632169, #641838, #627535, #629355)

Updated/clarified man pages.  (Addresses Debian Bugs: #639411,
#642193, #634883)

Programming notes
-----------------

Initial support for quota as an integrated feature, where the quota
files are hidden system files that are automatically maintained by
e2fsck is present, although disabled by default.  It must be enabled
by using the configure option --enable-quota.

Reserved the on-disk fields for the metadata checksum and snapshot
features, which are currently in development.

The ext2fs library now has the new functions ext2fs_punch(),
ext2fs_get_memzero() and ext2fs_file_get_inode().

The ext2fs library now has support for calculating the crc32c checksum
(via the new functions ext2fs_crc32c_be and ext2fs_crc32c_le).

The I/O manager now supports the discard operation.

Reserved file system code points for new 1st class quota feature.

Shortened the compilation lines by moving the autoconf defines to
a config.h header file.

Fixed a potential free of an unitialized pointer in
ext2fs_update_bb_inode().

Fixed miscellaneous gcc -Wall and coverity warnings.

Fixed portability issues for Mac OS X, Hurd, and FreeBSD.

Fixed a build failure when OMIT_COM_ERR is defined.

Improved error checking and fixed memory leaks caused by error return
paths.

Add ext2fs_flush2() and ext2fs_close2() which takes a flag to allow
the fsync() to be skipped.

Added a test for extent-mapped journals by mke2fs and tune2fs.

Added a test for creating a large (over 4GB) journal using mke2fs.

Added a test to make sure the inode size remains constant.

The regression test script now prints the list of which tests failed.

The regression test system now uses /tmp for its scratch files, which
is often a tmpfs mounted file system and hence much faster.

The i_e2image test was fixed so it works with valgrind.

Tests that rely on debugfs are now skipped if debugfs wasn't built.

Fixed the dependencies for "make check" so all required dependencies
are built before running the gression tests.

A link to com_err.h is installed in $(includedir) during a "make install".

The po/*.gmo files are automatically rebuilt if they are missing or
out of date.  This helps out Debian packaging.

Allow ext2fs_get_memalign() to compile on systems that don't have
posix_memalign().

Fixed a namespace leak in libext2fs (tdb_null).

E2fsprogs 1.41.14 (December 22, 2010)
=====================================

Fix spurious complaint in mke2fs where it would complain if the file
system type "default" is not defined in mke2fs.conf.

The resize2fs program will no longer clear the resize_inode feature
when the number reserved group descriptor blocks reaches zero.  This
allows for subsequent shrinks of the file system to work cleanly for
flex_bg file systems.

The resize2fs program now handles devices which are exactly 16T;
previously it would give an error saying that the file system was too
big.

E2fsck (and the libext2fs library) will not use the extended rec_len
encoding for file systems whose block size is less than 64k, for
consistency with the kernel.

Programming notes
-----------------

E2fsprogs 1.41.13 would not compile on big-endian systems.  This has
been fixed.  (Addresses Sourceforge Bug: #3138115)

The ext2fs_block_iterator2() function passed an incorrect ref_offset
to its callback function in the case of sparse files.  (Addresses
Sourceforge Bug: #3081087)

Fix some type-punning warnings generated by newer versions of gcc.

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Gmane