Python 3 support

Hi all-

First, thanks for the great tool. I've been using Gentoo for about 10 
years, and I use Catalyst to build custom Gentoo live CDs that contain 
extra software (e.g. netcat6, rsync, ddrescue, etc.).

At some point in the past the stage3 tarballs only contained Python 3, 
though it looks like the 20130516 amd64 stage3 contains both 2.7 and 
3.2. It's reasonable to assume that any current Gentoo system has a 
Python 3.2+ interpreter, so is there any objection to making Catalyst 
work on Python 3? The fewer things in the world that depend on Python 2, 
the better (in my opinion). Portage has supported Python 3 for some time 
and it makes sense for Catalyst to follow suit.

I use Python 3 for all of my own work and have a reasonable amount of 
experience porting projects to Python 3. This typically involves a lot 
of code churn even when there aren't any logic changes -- every print 
statement has to be converted to calling the print function, for 
instance. Explicit conversion between bytes and str can require a bit 
more thought but isn't difficult.

This conversion might require other miscellaneous cleanup, including 
some whitespace-only commits. Python 3 does not allow mixing tabs and 
spaces for indentation, and (e.g.) catalyst_support.py is a *mess* with 
indentation. PyCharm also notices things like unnecessary "\" line 
continuation characters, unused variables, mutable default arguments 
(e.g. catalyst_support.spawn's "env"), equality comparisons against 
None, and unreachable code (lines 183 - 220 of catalyst_lock.py).

It looks like the project has mostly standardized on using tabs for 

Support for custom memory settings in AUFS

On the original livedvd which I initially implemented AUFS support to
allow users to use of emerge, I implemented the ability to append
"aufs_mem=X" into the kernel since the default was to use tmpfs without
"-o size" option. Attached is a patch that will reintroduce this to
catalyst.

--

-- 
Thank you,
Fernando Orocu
GPG BDD75DD7
Mission Accomplish, Inc.
Tel: 7187100008
Cell: 3479275477

Catalyst Python reimplementation

Hello everyone!

My name is Dmitry and I'm writing here because I've found a proposal to rewrite Catalyst in Gentoo GSoC-2013 Ideas page[1]. I know that application period hasn't started yet, but I already think that I'd like to take up this job. However I've found that before posting application I must perform some actions:

1. Use the tools that you will use in your project to make changes to code.

2. Participate in our development community.

3. Give us your contact info and working hours.

I'd like to begin from the second question, i.e. I'd like to introduce myself. I'm Russian student, who studies programming for three years already, though it's rather hobby than education. My speciality is classical philology and now I'm studying colloquial Latin and read classical authors in Italy. My interest to programming began when I've started to use Linux. It happened more than three years ago; I've begun with bash and Python, now I'm trying to explore C++, though I've begun to study it only two or three months ago and didn't have explored it so deep to tell that I know C++ (though I certainly already know some basics).

Last year I've first time participated in GSoC with FSF and rewritten gnulib-tool huge shell script in Python to make it more portable and flexible[2]. I've successfully finished this work and decided to help gnulib-developers even in the future[3]. However, since I was missed to Italy for a long time to study Latin, I couldn't continue work so intensively like before (though even in Italy I've really never stoped working). Recently I've published a new version of gnulib-tool, which seems to be more easy to use and also better organized than the older one[4]. I've finished it some days ago (it also probably was a good exercise in Python) and will now develop this version (thus I've created a separated repository). There is a small (yet?) discussion in gnulib-tool list[5] about this update, though my mentor (Bruno Haible) didn't respond yet (but other gnulib developer has already written an answer). I'm planning to continue this work.

Some other things were written before gnulib reimplementation. Some time before I've written a small Russian-Latin dictionary (which in fact was my first work in Python) for my cathedra and all people studying classical philology[6]. However, now I think that it needs reimplementation too (since I see that it could have been implemented better and also see some things which I could add like morphological parser from Perseus site[7] to explain word forms). For some time I've been creating some portability layers[8] between Python and Python3, but now I don't think it is necessary since I feel that the time to migrate to Python3 has probably already come. The last thing that I was working around is flex/yacc parsers generation and implementation for some classes in C++ (this work was initially begun to understand C++ better, but I think that it can be in future something more than just exercise); this work is also located in my github repositories[9].

If you think that my candidature may be considered, I'd like to ask some questions.

As you've already understand, I've been used git for some time already. During the previous GSoC I intensively used git to work with code, though I can't say that I'm really experienced person (i.e. git push, git pull, diff, patch were enough for my previous project). However, I'm not sure it is enough. The other question is that it is really hard to find some project which uses python and emit patch to it to pass requirements. Could you advice a good bug, please?

If you think I have a chance, I can even answer the third question (about work hours, contacts, etc.).

Looking forward to your answers!

P.S. Thanks for being so patience to read such a long letter till the end!

[1] http://wiki.gentoo.org/wiki/Google_Summer_of_Code/2013/Ideas#Catalyst

[2] http://www.gnu.org/software/soc-projects/ideas-2012.html#gnulib

[3] https://github.com/ghostmansd/gnulib-python

[4] https://github.com/ghostmansd/pygnulib

[5] http://lists.gnu.org/archive/html/bug-gnulib/2013-04/msg00037.html

[6] http://sourceforge.net/projects/quirinus

[7] http://www.perseus.tufts.edu/hopper/morph

[8] https://github.com/ghostmansd/gnupython

[9] https://github.com/ghostmansd/quirinus

--

With best regards,

Dmitry Selyutin

E-mail: ghostmansd-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

Phone: +7(985)334-07-70

Re: [PATCH 0/2] Blacklisting binary packages

On Tue, Apr 16, 2013 at 03:28:04PM -0700, Zac Medico wrote:
> If you can't get the people maintaining the ebuilds in gentoo-x86 to
> cooperate with migration to EAPI 5, then I think your best option is
> to start an eapi-5 overlay which contains EAPI 5 ports of those
> ebuilds.

Good idea.  I'll look into that after I hit a wall on sub-slotting
udev [1].

Cheers,
Trevor

[1]: https://bugs.gentoo.org/show_bug.cgi?id=466162

--

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

Re: make catalyst call emerge --deep

On 04/10/2013 01:35 PM, Matt Turner wrote:
> On Wed, Apr 10, 2013 at 8:54 AM, Burak Arslan
> <burak.arslan@...> wrote:
>> Hi,
>>
>> In a build (a desktop stage 4) I'm trying to run, I'm getting multiple udevs
>> (=197-r1 and >=197-r2) pulled in (among other errors, see below). My
>> impression is that if I were able to pass --deep to emerge command, it'd
>> just go through. Is there a non-hacky way of doing this?
> 
> Modify catalyst and try it. We should confirm that --deep actually
> would fix this problem before deciding it's the solution.

I have no idea, I've never hit this problem as --deep is in my profile
as a default option.
> 
> If it is, I'm all for adding it.
> 
> 

-ZC

make catalyst call emerge --deep

Hi,

In a build (a desktop stage 4) I'm trying to run, I'm getting multiple 
udevs (=197-r1 and >=197-r2) pulled in (among other errors, see below). 
My impression is that if I were able to pass --deep to emerge command, 
it'd just go through. Is there a non-hacky way of doing this?

Best regards,
Burak

======================

[blocks B     ] <sys-apps/sandbox-2.6-r1 ("<sys-apps/sandbox-2.6-r1" is 
blocking dev-lang/python-3.3.0-r1)

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

virtual/udev:0

   (virtual/udev-197-r1::gentoo, installed) pulled in by
     =virtual/udev-197-r1 required by (virtual/dev-manager-0::gentoo, 
installed)
     (and 18 more with the same problem)

   (virtual/udev-197-r2::gentoo, ebuild scheduled for merge) pulled in by
     >=virtual/udev-197[gudev,hwdb] required by 
(sys-fs/udisks-2.1.0::gentoo, ebuild scheduled for merge)
     >=virtual/udev-197[gudev] required by 
(sys-power/upower-0.9.20-r2::gentoo, ebuild scheduled for merge)

[PATCH] Add support for m68k

---
 arch/m68k.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 arch/m68k.py

diff --git a/arch/m68k.py b/arch/m68k.py
new file mode 100644
index 0000000..37b26a4
--- /dev/null
+++ b/arch/m68k.py
 <at>  <at>  -0,0 +1,22  <at>  <at> 
+
+import builder,os
+from catalyst_support import *
+
+class generic_m68k(builder.generic):
+	"abstract base class for all m68k builders"
+	def __init__(self,myspec):
+		builder.generic.__init__(self,myspec)
+		self.settings["CHROOT"]="chroot"
+		self.settings["CFLAGS"]=" -pipe"
+
+class arch_m68k(generic_m68k):
+	"builder class for generic m68k"
+	def __init__(self,myspec):
+		generic_m68k.__init__(self,myspec)
+		self.settings["CFLAGS"]+=" -O2"
+		self.settings["CHOST"]="m68k-unknown-linux-gnu"
+
+def register():
+	"Inform main catalyst program of the contents of this plugin."
+	return ({ "m68k":arch_m68k },
+	("m68k", ))

[PATCH] Make the use of preserved_libs optional by setting the options variable.

From: "Jorge Manuel B. S. Vicetto (jmbsvicetto)" <jmbsvicetto@...>

---
 catalyst                   | 3 +++
 files/catalyst.conf        | 2 ++
 targets/support/unmerge.sh | 3 ++-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/catalyst b/catalyst
index 9f8dd30..f885dd4 100755
--- a/catalyst
+++ b/catalyst
 <at>  <at>  -147,6 +147,9  <at>  <at>  def parse_config(myconfig):
 	if "pkgcache" in string.split(conf_values["options"]):
 		print "Package cache support enabled."
 		conf_values["PKGCACHE"]="1"
+	if "preserve_libs" in string.split(conf_values["options"]);
+		print "Preserving libs during unmerge."
+		conf_values["PRESERVE_LIBS"]="1"

 	if "purge" in string.split(conf_values["options"]):
 		print "Purge support enabled."
diff --git a/files/catalyst.conf b/files/catalyst.conf
index f88f105..2983acc 100644
--- a/files/catalyst.conf
+++ b/files/catalyst.conf
 <at>  <at>  -59,6 +59,8  <at>  <at>  hash_function="crc32"
 #	build stops in livecd-stage2)
 # pkgcache = keeps a tbz2 of every built package (useful if your build stops
 #	prematurely)
+# preserve_libs = enables portage to preserve used libs when unmerging packages
+#   (used on installcd-stage2 and stage4 targets)
 # seedcache = use the build output of a previous target if it exists to speed up
 #	the copy
 # snapcache = cache the snapshot so that it can be bind-mounted into the chroot.
diff --git a/targets/support/unmerge.sh b/targets/support/unmerge.sh
index 907113e..0c0c037 100644
--- a/targets/support/unmerge.sh
+++ b/targets/support/unmerge.sh
 <at>  <at>  -2,7 +2,8  <at>  <at> 

 source /tmp/chroot-functions.sh

-FEATURES="${clst_myfeatures} preserve-libs"
+# If the user enabled PRESERVE_LIBS in options, tell portage to preserve them.
+[ -n "${clst_PRESERVE_LIBS}" ] && FEATURES="${clst_myfeatures} preserve-libs"
 run_merge -C ${clst_packages}

 exit 0

[PATCH 1/2] Fix update_seed use by not using nor building binary packages during the seed update.

From: "Jorge Manuel B. S. Vicetto (jmbsvicetto)" <jmbsvicetto@...>

---
 targets/stage1/stage1-chroot.sh     | 5 ++++-
 targets/support/chroot-functions.sh | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
index 3f628c2..ed83f38 100644
--- a/targets/stage1/stage1-chroot.sh
+++ b/targets/stage1/stage1-chroot.sh
 <at>  <at>  -30,12 +30,15  <at>  <at>  if [ -n "${clst_update_seed}" ]; then
 		if [ -n "${clst_update_seed_command}" ]; then
 			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
 		else
-			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
+			clst_root_path=/ run_merge "--update --deep --newuse --complete-graph --rebuild-if-new-ver gcc"
 		fi
 	elif [ "${clst_update_seed}" != "no" ]; then
 		echo "Invalid setting for update_seed: ${clst_update_seed}"
 		exit 1
 	fi
+
+	# reset emerge options for the target
+	clst_update_seed=no setup_myemergeopts
 else
 	echo "Skipping seed stage update..."
 fi
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index 35ffbaa..0705b5b 100644
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
 <at>  <at>  -133,7 +133,7  <at>  <at>  setup_myemergeopts(){
 	then
 		export bootstrap_opts="${bootstrap_opts} -f"
 		export clst_myemergeopts="${clst_myemergeopts} -f"
-	elif [ -n "${clst_PKGCACHE}" ]
+	elif [ -n "${clst_PKGCACHE}" -a -z "${clst_update_seed}" ]
 	then
 		export clst_myemergeopts="${clst_myemergeopts} --usepkg --buildpkg --newuse"
 		export bootstrap_opts="${bootstrap_opts} -r"

sem_open inconsistency in installed Python versions

I've been bumping into failures installing Matplotlib 1.2.0-r2:

  # cat /tmp/livecd-stage1.log
  …
  /build/lib/matplotlib/tests/baseline_images/test_delaunay
  Traceback (most recent call last):
    File "/usr/lib/python3.2/multiprocessing/synchronize.py", line 54, in <module>
      from _multiprocessing import SemLock
  ImportError: cannot import name SemLock
  …
  ImportError: This platform lacks a functioning sem_open implementation,
   therefore, the required synchronization primitives needed will not function,
   see issue 3770.
  …

Looking through my logs, it seems that some versions of Python 3.2.3
found a sem_open implementation, but some did not:

  $ cd /var/log/catalyst
  $ grep sem_open dev-lang\:python-3.2.3\:*
  dev-lang:python-3.2.3:20130310-213508.log:checking for sem_open... yes
  dev-lang:python-3.2.3:20130310-222606.log:checking for sem_open... no
  dev-lang:python-3.2.3:20130310-234410.log:checking for sem_open... yes

I cleaned out my logs before my last run, so we can match those build
times up with stages:

  $ grep dev-lang/python-3.2.3 /tmp/*.log
  /tmp/stage1.log:>>> Emerging (14 of 17) dev-lang/python-3.2.3
  /tmp/stage1.log:>>> Installing (14 of 17) dev-lang/python-3.2.3
  /tmp/stage1.log:>>> Emerging (72 of 75) dev-lang/python-3.2.3 for /tmp/stage1root/
  /tmp/stage1.log:>>> Installing (72 of 75) dev-lang/python-3.2.3 to /tmp/stage1root/
  /tmp/stage3.log:>>> Emerging (99 of 189) dev-lang/python-3.2.3
  /tmp/stage3.log:>>> Installing (99 of 189) dev-lang/python-3.2.3

So the stage1 seed_update emerge finds sem_open, but the stage1root
setup does not.  I'm not sure why this would happen, but since the
stage3 emerge *does* find sem_open, I'm not sure why the Matplotlib
emerge is failing in livecd-stage1.  Does anyone want to take a guess?

Thanks,
Trevor

--

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[PATCH] livecdfs-update.sh: Escape ampersands in STARTX sed expression

From: "W. Trevor King" <wking@...>

From sed(1):

  s/regexp/replacement/
    Attempt to match regexp against the pattern space.  If successful,
    replace that portion matched with replacement.  The replacement
    may contain the special character & to refer to that portion of
    the pattern space which matched, and the special escapes \1
    through \9 to refer to the corresponding matching sub-expressions
    in the regexp.

This means that the old expression (with unescaped ampersands) lead
to:

  source /etc/profile ##STARTX##STARTX su - ${first_user} -c startx

when we want:

  source /etc/profile && su - ${first_user} -c startx

with ${first_user} expanded in both cases.
---
 targets/support/livecdfs-update.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/targets/support/livecdfs-update.sh b/targets/support/livecdfs-update.sh
index 77d694e..fda3e36 100644
--- a/targets/support/livecdfs-update.sh
+++ b/targets/support/livecdfs-update.sh
 <at>  <at>  -389,7 +389,7  <at>  <at>  esac
 if [ -e /etc/startx ]
 then
 	sed -i \
-		"s:##STARTX:source /etc/profile && su - ${first_user} -c startx:" \
+		"s:##STARTX:source /etc/profile \&\& su - ${first_user} -c startx:" \
 		/root/.bashrc
 fi