createrepo/__init__.py createrepo/utils.py createrepo/yumbased.py genpkgmetadata.py Makefile worker.py

 Makefile               |    3 
 createrepo/__init__.py |  232 +++++++++++++++++++++++++++++--------------------
 createrepo/utils.py    |    9 +
 createrepo/yumbased.py |    9 +
 genpkgmetadata.py      |    5 +
 worker.py              |   76 ++++++++++++++++
 6 files changed, 239 insertions(+), 95 deletions(-)

New commits:
commit b0b85d5c6cdb9b7b97147ce31bda8b4711d3a67c
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Thu Sep 9 17:07:06 2010 -0400

    create a worker script for createrepo so createrepo can
    fork off N processes to handle the md gathering from pkgs.
    This should speed up results on systems which have been cpubound
    on the createrepo process.

    If you're io bound it won't help you at all, and MAY make it worse.

    many misc issues to iron out here - not the least of which is the
    callback output and gathering stdout/stderr from the workers

diff --git a/Makefile b/Makefile
index 6b907d8..60bb9db 100644
--- a/Makefile
+++ b/Makefile
 <at>  <at>  -47,7 +47,8  <at>  <at>  RM              = rm -f

 MODULES = $(srcdir)/genpkgmetadata.py \

createrepo/__init__.py worker.py

 createrepo/__init__.py |  141 +++++++++++++++++++++++++++++--------------------
 worker.py              |   29 +++++++++-
 2 files changed, 111 insertions(+), 59 deletions(-)

New commits:
commit e4c674f2851f2ca71f744e86972e1c3acaf1e8f2
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Fri Sep 10 16:06:17 2010 -0400

    - make sure we handle remote_url pkgs correctly until we get the worker hooked up to handle them
    - if there are no pkgs to handle, don't launch workers with nothing to do.
    - give better output from the workers and have them obey -v/-q
    - everyone loves callbacks!

diff --git a/createrepo/__init__.py b/createrepo/__init__.py
index b2937c9..a2ae8b0 100644
--- a/createrepo/__init__.py
+++ b/createrepo/__init__.py
 <at>  <at>  -580,77 +580,106  <at>  <at>  class MetaDataGenerator:
         # filter out those pkgs which are not files - but are pkgobjects
         pkgfiles = []
         for pkg in newpkgs:
+            po = None
             if isinstance(pkg, YumAvailablePackage):
                 po = pkg
                 self.read_pkgs.append(po.localpath)
+
+            # if we're dealing with remote pkgs - pitch it over to doing
+            # them one at a time, for now. 
+            elif pkg.find('://') != -1:

Update 0.4.11-0.9.8: AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'

Guys,

	I updated my createrepo install to the current 0.9.8 package on my Arch Linux 
server. I have 4 openSUSE repos I maintain. All updates with createrepo worked 
fine with 0.4.11. After updating to the following:

beecrypt-4.2.1-2-x86_64.pkg.tar.xz
kid-0.9.6-4-x86_64.pkg.tar.xz
python-iniparse-0.4-1-x86_64.pkg.tar.xz
repoview-0.6.5-1-any.pkg.tar.xz
rpm-5.1.9-1-x86_64.pkg.tar.xz
yum-3.2.28-1-x86_64.pkg.tar.xz
yum-createrepo-0.9.8-3-x86_64.pkg.tar.xz
yum-metadata-parser-1.1.4-2-x86_64.pkg.tar.xz

     I receive the following error on my opensuse_11.0 repo. (~6900 rpms). 
However, my opensuse_11.3 repo updates just fine (~1200 rpms). The error I get is:

[13:55 nirvana:/home/backup/rpms] # createrepo --update -d openSUSE_11.0/
1149/6790 - i586/libopencore-amrwb0-0.1.2-0.pm.1.1.i586.rpm 
Traceback (most recent call last):
   File "/usr/share/createrepo/genpkgmetadata.py", line 249, in <module>
     main(sys.argv[1:])
   File "/usr/share/createrepo/genpkgmetadata.py", line 223, in main
     mdgen.doPkgMetadata()
   File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 367, in 
doPkgMetadata
     self.writeMetadataDocs(packages)
   File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 531, in 
writeMetadataDocs

Re: Update 0.4.11-0.9.8: AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'

err... only suse uses PreReq nowadays. that might be your problem.
See openSUSETools on build.o.o for the package createrepo and see what
patches they used.

On Tue, Sep 14, 2010 at 17:18, David C. Rankin
<drankinatty <at> suddenlinkmail.com> wrote:
> Guys,
>
>        I updated my createrepo install to the current 0.9.8 package on my
> Arch Linux server. I have 4 openSUSE repos I maintain. All updates with
> createrepo worked fine with 0.4.11. After updating to the following:
>
> beecrypt-4.2.1-2-x86_64.pkg.tar.xz
> kid-0.9.6-4-x86_64.pkg.tar.xz
> python-iniparse-0.4-1-x86_64.pkg.tar.xz
> repoview-0.6.5-1-any.pkg.tar.xz
> rpm-5.1.9-1-x86_64.pkg.tar.xz
> yum-3.2.28-1-x86_64.pkg.tar.xz
> yum-createrepo-0.9.8-3-x86_64.pkg.tar.xz
> yum-metadata-parser-1.1.4-2-x86_64.pkg.tar.xz
>
>    I receive the following error on my opensuse_11.0 repo. (~6900 rpms).
> However, my opensuse_11.3 repo updates just fine (~1200 rpms). The error I
> get is:
>
> [13:55 nirvana:/home/backup/rpms] # createrepo --update -d openSUSE_11.0/
> 1149/6790 - i586/libopencore-amrwb0-0.1.2-0.pm.1.1.i586.rpm Traceback (most
> recent call last):
>  File "/usr/share/createrepo/genpkgmetadata.py", line 249, in <module>
>    main(sys.argv[1:])

Re: Update 0.4.11-0.9.8: AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'

David C. Rankin wrote:

> 	I updated my createrepo install to the current 0.9.8 package on my  
> Arch Linux server. I have 4 openSUSE repos I maintain. All updates  
> with createrepo worked fine with 0.4.11. After updating to the  
> following:
>
> [...]
>  File "/usr/lib/python2.6/site-packages/yum/packages.py", line  
> 1359, in _is_pre_req
>     rpm.RPMSENSE_SCRIPT_PRE |
> AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'
>
>     I know very little about python, so I need a bit of help on  
> where I should start trying to figure out what is going on. Any ideas?

Yum needs a patch to work with 4.4.3 and beyond (such as your 5.1.9)

Easiest is just adding the enum values, rather than conditionals...

--anders

_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata

Re: Update 0.4.11-0.9.8: AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'

On Tue, 2010-09-14 at 23:40 +0200, Anders F Björklund wrote:
> David C. Rankin wrote:
> 
> > 	I updated my createrepo install to the current 0.9.8 package on my  
> > Arch Linux server. I have 4 openSUSE repos I maintain. All updates  
> > with createrepo worked fine with 0.4.11. After updating to the  
> > following:
> >
> > [...]
> >  File "/usr/lib/python2.6/site-packages/yum/packages.py", line  
> > 1359, in _is_pre_req
> >     rpm.RPMSENSE_SCRIPT_PRE |
> > AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'
> >
> >     I know very little about python, so I need a bit of help on  
> > where I should start trying to figure out what is going on. Any ideas?
> 
> Yum needs a patch to work with 4.4.3 and beyond (such as your 5.1.9)
> 
> Easiest is just adding the enum values, rather than conditionals...

rpm 4.7.X and rpm 4.8.x work fine with yum 3.2.2X

-sv

_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata

Re: Update 0.4.11-0.9.8: AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'

On 09/14/2010 05:04 PM, seth vidal wrote:
> rpm 4.7.X and rpm 4.8.x work fine with yum 3.2.2X
>
> -sv
>

Thanks all, I'll try the patch and report back. What makes this situation a bit 
different is my opensuse repositories are hosted on my Arch Linux server not on 
openSUSE. So I end up building everything from source (well actually from the 
Arch AUR PKGBUILD scripts -- that, shall we say, need a little TLC to handle the 
0.4x to 0.9x jump.

Thank you again. I'll see how far I can get, but I may need to beg for a bit 
more help if I get stuck :p

--

-- 
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com

Re: Update 0.4.11-0.9.8: AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'

On 09/14/2010 04:40 PM, Anders F Björklund wrote:
> David C. Rankin wrote:
>
>> I updated my createrepo install to the current 0.9.8 package on my Arch Linux
>> server. I have 4 openSUSE repos I maintain. All updates with createrepo worked
>> fine with 0.4.11. After updating to the following:
>>
>> [...]
>> File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1359, in
>> _is_pre_req
>> rpm.RPMSENSE_SCRIPT_PRE |
>> AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'
>>
>> I know very little about python, so I need a bit of help on where I should
>> start trying to figure out what is going on. Any ideas?
>
> Yum needs a patch to work with 4.4.3 and beyond (such as your 5.1.9)
>
> Easiest is just adding the enum values, rather than conditionals...
>
> --anders
>
>

Anders,

	You-da-man! Applied the patch and then re-ran the update against my opensuse 
11.0 repo and it worked just fine:

[17:28 nirvana:/home/backup/rpms] # createrepo --update -d openSUSE_11.0/

Createrepo in delta form.

_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata

Re: Createrepo in delta form.

On Wed, 2010-09-15 at 20:38 -0700, Edward Peschko wrote:
> All,
> 
> We maintain a series of yum repo using an old version of 2.6.1, and
> are running into performance issues when it comes to indexing. What we
> would like do is exchange our createrepo (which AFAICT re-calculates
> all the metadata for the indexing each time is run) with a createrepo
> that only calculates deltas since the last time it was run.
> 
> So I had a couple of questions:
> 
> 1. is there a flag for createrepo (that I’m not aware of) which allows
> for any rpm commands that are run in the past to be cached, and only
> calculates deltas to create the repo files? 
> 2. can a newer version of createrepo be used with the older version of
> yum listed above?
> 3. if so, does this newer version do deltas in the way that I’ve just
> described?
> 
> Thanks much for any help..
> 

a. new yum and createrepo have lots of new options - but that'll depend
mostly on what distro and ver of rpm/python you're using.

b. yum and createrepo versions  are getting tighter and tighter together

c. deltarpms are supported in new versions of createrepo (0.9.8) and yum
- but that may not be what you mean - do you want delta rpms or delta
metadata? Or maybe you just want the --update option to createrepo to
not reparse all the rpms each time it runs?

Hope this helps.

-sv

_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata