[RFC/PATCH] repoman: unroll escaped lines so we can check the entirety of it

Sometimes people wrap long lines in their ebuilds to make it easier to
read, but this causes us issues when doing line-by-line checking.  So
automatically unroll those lines before passing the full content down
to our checkers.

This seems to work, but maybe someone can suggest something simpler.

Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
 pym/repoman/checks.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/pym/repoman/checks.py b/pym/repoman/checks.py
index 65e7136..67f2b0a 100644
--- a/pym/repoman/checks.py
+++ b/pym/repoman/checks.py
@@ -750,11 +750,46 @@ _ignore_comment_re = re.compile(r'^\s*#')
 def run_checks(contents, pkg):
 	checks = _constant_checks
 	here_doc_delim = None
+	multiline = None

 	for lc in checks:
 		lc.new(pkg)
 	for num, line in enumerate(contents):

+		# Unroll multiline escaped strings so that we can check things:
+		#		inherit foo bar \
+		#			moo \
+		#			cow

[PATCH] repoman: add a mini framework for checking eclasses, and fill it out

Rather than copying & pasting the same behavior for the different eclass
checks, add a common class for them to extend.  This makes adding more
eclass checks trivial, and keeps down bitrot.

This does abuse the checking interface slightly -- the eclass will change
its category between unused and missing based on the checks.

URL: https://bugs.gentoo.org/417159
URL: https://bugs.gentoo.org/417231
Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
 bin/repoman           |    6 +-
 pym/repoman/checks.py |  143 ++++++++++++++++++++++++++++++++-----------------
 pym/repoman/errors.py |    1 -
 3 files changed, 97 insertions(+), 53 deletions(-)

diff --git a/bin/repoman b/bin/repoman
index 3697403..d7ffcdd 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -315,8 +315,9 @@ qahelp={
 	"file.size.fatal":"Files in the files directory must be under 60 KiB",
 	"file.name":"File/dir name must be composed of only the following chars: %s " % allowed_filename_chars,
 	"file.UTF8":"File is not UTF8 compliant",
-	"inherit.autotools":"Ebuild inherits autotools but does not call eautomake, eautoconf or eautoreconf",
 	"inherit.deprecated":"Ebuild inherits a deprecated eclass",
+	"inherit.missing":"Ebuild uses functions from an eclass but does not inherit it",
+	"inherit.unused":"Ebuild inherits an eclass but does not use it",
 	"java.eclassesnotused":"With virtual/jdk in DEPEND you must inherit a java eclass",
 	"wxwidgets.eclassnotused":"Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass",

About some settings to auto-replace with dispatch-conf

Hello

I recently installed Gentoo on my uncle's laptop and he was a bit
annoyed about needing to run "dispatch-conf" and merge a lot of changes
on files nobody ever touched.

Looking to /etc/dispatch-conf.conf I noticed options to improve this
situation exist, but they are disabled by default. I would want to
confirm if they are safe enough or could cause problems. Options are:

# Automerge files comprising only whitespace and/or comments
# (yes or no)
replace-wscomments=no

# Automerge files that the user hasn't modified
# (yes or no)
replace-unmodified=no

Looks really surprising to me that "replace-wscomments" is not enabled
by default as merging that changes shouldn't hurt at all. About
"replace-unmodified", if it works as intended, it should also be safer
to get it enabled by default as would prevent breakage if people forgets
to run dispatch-conf, reboot and, for example, sees some init.d script
fail to start.

Thanks a lot for the info

[RFC/PATCH] prepstrip/ecompressdir: parallelize operations

Stealing some ideas from ferringb, add a new API for doing parallel
processing in bash, and then deploy this with the stripping and
compressing stages.

For stripping coreutils which has about 100 ELFs, this brings time
to strip down from ~7 seconds to ~0.7 seconds on my system.

Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
note: i'm not terribly happy with the name "helper-functions.sh", so any
	better suggestions would be good.  i didn't want to use "ebuild-helpers.sh"
	as that messes up tab completion ;).

 bin/ebuild-helpers/ecompressdir |   30 ++++++++++++++++---
 bin/ebuild-helpers/prepstrip    |   20 ++++++++++---
 bin/helper-functions.sh         |   62 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 104 insertions(+), 8 deletions(-)
 create mode 100644 bin/helper-functions.sh

diff --git a/bin/ebuild-helpers/ecompressdir b/bin/ebuild-helpers/ecompressdir
index 17ecd80..a2c9e52 100755
--- a/bin/ebuild-helpers/ecompressdir
+++ b/bin/ebuild-helpers/ecompressdir
@@ -2,7 +2,7 @@
 # Copyright 1999-2011 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2

-source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/isolated-functions.sh
+source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/helper-functions.sh

[PATCH] Ignore some unowned files during collision check

Hello,

I've added a new variable to portage named
COLLISION_UNOWNED_IGNORE_PATTERNS. It works similar to COLLISION_IGNORE, it
specifies shell patterns for unowned files that can be overwritten (will not
cause collision-protect or protect-owned to go off).

I'm not convinced about the name - if anyone has better idea I'd be more than
happy to change it :)

Attached patch adds code to vartree, documents new variable in manual page and
adds a default to make.globals.

--

-- 
Krzysztof Pawlik  <nelchael at gentoo.org>  key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

diff -r 216c6f63069e cnf/make.globals
--- a/cnf/make.globals	Wed May 02 18:32:33 2012 -0700
+++ b/cnf/make.globals	Thu May 03 09:59:04 2012 +0200
@@ -61,6 +61,11 @@ FEATURES="assume-digests binpkg-logs dis
 # FEATURES=protect-owned to operate smoothly in all cases.
 COLLISION_IGNORE="/lib/modules"

+# Ignore file collisions for unowned *.pyo and *.pyc files, this helps during
+# transition from compiling python modules in live file system to compiling
+# them in src_install() function.
+COLLISION_UNOWNED_IGNORE_PATTERNS="*.pyc *.pyo"
+

GSoC project proposal

Hi all,

I am working on a proposal for GSoC 2012 and I'd appreciate any input that you can have towards my proposal,
which I'm enclosing below:

Project Name: Package statistics reporting tool

Abstract

I am proposing to work on improving the Package statistics reporting tool that was started last as part of
GCoC 2011 for Gentoo, Gentoostats (http://wiki.gentoo.org/wiki/Gentoostats). 
The plan is to
continue working on the user end program to upload anonymous information about installed packages on
users’ machines to a database that package maintainers and developers have access to, as well as
offline support.
Additional package information statistics will be aggregated and a more robust web application will be
written to allow for third party data exports, as well as graphical analysis graphs and tools will be incorporated.

Objective

The objective of this project will improve both the data being collected, stored and presented by last
year’s iteration of this project, Gentoostats (http://wiki.gentoo.org/wiki/Gentoostats). 
The anonymous data being uploaded being analyzed and we will improve the data submission model to support
additional fields for future updates via a more general package build description file metadata that is
more easily extensible.
We will also extend the application and the web site itself to allow for offline submission of data
statistics. 
At the core of the project will be creating a rich web 2.0 application with aggregated package information
statistics, third party export to CSV and other formats, data manipulation via API calls, as well as
graphical analysis graphs and tools will be incorporated.

[PATCH] portageq: add "colormap" helper

Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
 bin/isolated-functions.sh              |    2 +-
 bin/portageq                           |    8 ++++++++
 pym/portage/output.py                  |    6 ++++++
 pym/portage/package/ebuild/doebuild.py |    8 ++------
 4 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index 9321ad5..98be41e 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -431,8 +431,8 @@ set_colors() {
 		BAD=$'\e[31;01m'
 		HILITE=$'\e[36;01m'
 		BRACKET=$'\e[34;01m'
+		NORMAL=$'\e[0m'
 	fi
-	NORMAL=$'\e[0m'
 }

 RC_ENDCOL="yes"
diff --git a/bin/portageq b/bin/portageq
index 5ecbb21..fcdb9d9 100755
--- a/bin/portageq
+++ b/bin/portageq
@@ -44,6 +44,7 @@ del pym_path
 from portage import os
 from portage.eapi import eapi_has_repo_deps
 from portage.util import writemsg, writemsg_stdout

[PATCH] dispatch-conf: do regex matching ourselves

This avoids having to pipe through multiple greps, as well as running
diff multiple times on the same set of files.

Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
 bin/dispatch-conf            |   37 ++++++++++++++++++++++++-------------
 pym/portage/dispatch_conf.py |    8 ++++----
 2 files changed, 28 insertions(+), 17 deletions(-)

diff --git a/bin/dispatch-conf b/bin/dispatch-conf
index a778118..f6d7499 100755
--- a/bin/dispatch-conf
+++ b/bin/dispatch-conf
@@ -27,13 +27,11 @@ except ImportError:
 from portage import os
 from portage import dispatch_conf
 from portage import _unicode_decode
-from portage.dispatch_conf import diffstatusoutput_len
+from portage.dispatch_conf import diffstatusoutput
 from portage.process import find_binary

 FIND_EXTANT_CONFIGS  = "find '%s' %s -name '._cfg????_%s' ! -name '.*~' ! -iname '.*.bak' -print"
 DIFF_CONTENTS        = "diff -Nu '%s' '%s'"
-DIFF_CVS_INTERP      = "diff -Nu '%s' '%s' | grep '^[+-][^+-]' | grep -v '# .Header:.*'"
-DIFF_WSCOMMENTS      = "diff -Nu '%s' '%s' | grep '^[+-][^+-]' | grep -v '^[-+]#' | grep -v '^[-+][[:space:]]*$'"

 # We need a secure scratch dir and python does silly verbose errors on the use of tempnam
 oldmask = os.umask(0o077)
@@ -62,7 +60,7 @@ def cleanup(mydir=SCRATCH_DIR):
     shutil.rmtree(mydir)

Rerunning src_test with modifications

When I run into test failures in www-client/chromium, the normal
procedure is to file a bug and filter the failing tests until the
problem can be fixed (usually upstream).

Ideally, I would like to be able to do the following:

1. Start building chromium using emerge or ebuild.

2. If src_test dies, modify the ebuild to filter the failing tests.

3. Re-run the test phase without rebuilding the source code.

If I do not run ebuild clean, portage re-uses the environment from the
previous run, which does not include my changes to the ebuild.

Is there an easy way to have portage source the modified ebuild
instead of just using the previous environment? Wiping out the
environment file isn't really an option because we set some global
variables in pkg_setup.

[PATCH] add support for compressing debug sections to save space

Since binutils/gdb can compress the .debug* sections with zlib into
.zdebug*, we should be able to save quite a bit of space -- on my
system, I saw /usr/lib/debug/ go from ~20GB to ~7.5GB.

This requires binutils/gdb to be built with USE=zlib which is why
this defaults to off.

Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
 bin/ebuild-helpers/prepstrip |    6 ++++--
 man/make.conf.5              |   10 +++++++++-
 pym/_emerge/EbuildPhase.py   |    6 ++++--
 pym/portage/const.py         |    2 +-
 4 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip
index ee547f2..daaa252 100755
--- a/bin/ebuild-helpers/prepstrip
+++ b/bin/ebuild-helpers/prepstrip
@@ -15,7 +15,7 @@ exp_tf() {
 		eval ${var}_${flag}=$(tf has ${flag} ${!var})
 	done
 }
-exp_tf FEATURES installsources nostrip splitdebug
+exp_tf FEATURES compressdebug installsources nostrip splitdebug
 exp_tf RESTRICT binchecks installsources strip

 [[ " ${FEATURES} " == *" force-prefix "* ]] || \
@@ -121,7 +121,9 @@ save_elf_debug() {
 		if [[ -e ${T}/prepstrip.split.debug ]] ; then

[PATCH] prepstrip: add writable workaround for everyone

The writable issue shows up when using `ebuild` as non-root users
in non-prefix setups.  So always do it.

Signed-off-by: Mike Frysinger <vapier <at> gentoo.org>
---
 bin/ebuild-helpers/prepstrip |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip
index 1d7b3d2..ee547f2 100755
--- a/bin/ebuild-helpers/prepstrip
+++ b/bin/ebuild-helpers/prepstrip
@@ -224,7 +224,7 @@ do
 	# unwritable objects.  Make them temporarily writable for the
 	# stripping.
 	was_not_writable=false
-	if [[ -n ${EPREFIX} && ! -w ${x} ]] ; then
+	if [[ ! -w ${x} ]] ; then
 		was_not_writable=true
 		chmod u+w "${x}"
 	fi
--

-- 
1.7.8.3