Jon Wallace | 2 Sep 08:04 2011
Picon

Re: mergerepo weird behavior

On 8/24/2011 5:07 PM, James Antill wrote:
>   Ok ... got some free time to test this and it turned out to be a bug in
> yum (in theory it affects more just groups/pkgtags in mergerepo, but in
> practice probably not). Fix is simple enough:
>
> commit 581d161bbefc9b940289f15a93c3fdef6ad8c843
> Author: James Antill<james <at> and.org>
> Date:   Wed Aug 24 17:03:58 2011 -0400
>
>      Fix mergerepo on generated data (groups/pkgtag/etc.)
>
> diff --git a/yum/yumRepo.py b/yum/yumRepo.py
> index 91b7dde..a681370 100644
> --- a/yum/yumRepo.py
> +++ b/yum/yumRepo.py
>  <at>  <at>  -1256,6 +1256,11  <at>  <at>  Insufficient space in download directory %s
>
>               # if we have a 'fresh' repomd.xml then update the cookie
>               self.setMetadataCookie()
> +            #  Also remove any files that are generated ... this can be slightl
> +            # non-optimal, in the case where we have to revert, but it's just
> +            # sooo much easier to deal with it here.
> +            for fname in misc.getFileList(self.cachedir + '/gen', '', []):
> +                misc.unlink_f(fname)
>
>           if self._repoXML is None:
>               self._repoXML = self._parseRepoXML(result)
>
>
> ...feel free to confirm.
(Continue reading)

Dennis Gregorovic | 7 Sep 18:18 2011
Picon

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
> -    def getNodes(self, relpath):
> -        """Return base, filelist, and other nodes for file, if they exist
>  
> -        Returns a tuple of nodes, or None if not found
> +    def getNodes(self, relpath):
> +        """return a package object based on relative path of pkg
>          """
> -        bnode = self.basenodes.get(relpath,None)
> -        if bnode is None:
> -            return None
> -        pkgid = self.pkg_ids.get(relpath,None)
> -        if pkgid is None:
> -            print _("No pkgid found for: %s") % relpath
> -            return None
> -        fnode = self.filesnodes.get(pkgid,None)
> -        if fnode is None:
> -            return None
> -        onode = self.othernodes.get(pkgid,None)
> -        if onode is None:
> -            return None
> -        return bnode, fnode, onode
> -
> -    def freeNodes(self,relpath):
> -        #causing problems
> -        """Free up nodes corresponding to file, if possible"""
> -        bnode = self.basenodes.get(relpath,None)
> -        if bnode is None:
> -            print "Missing node for %s" % relpath
> -            return
(Continue reading)

seth vidal | 7 Sep 18:58 2011

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Wed, 2011-09-07 at 12:18 -0400, Dennis Gregorovic wrote:
> On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
> > -    def getNodes(self, relpath):
> > -        """Return base, filelist, and other nodes for file, if they exist
> >  
> > -        Returns a tuple of nodes, or None if not found
> > +    def getNodes(self, relpath):
> > +        """return a package object based on relative path of pkg
> >          """
> > -        bnode = self.basenodes.get(relpath,None)
> > -        if bnode is None:
> > -            return None
> > -        pkgid = self.pkg_ids.get(relpath,None)
> > -        if pkgid is None:
> > -            print _("No pkgid found for: %s") % relpath
> > -            return None
> > -        fnode = self.filesnodes.get(pkgid,None)
> > -        if fnode is None:
> > -            return None
> > -        onode = self.othernodes.get(pkgid,None)
> > -        if onode is None:
> > -            return None
> > -        return bnode, fnode, onode
> > -
> > -    def freeNodes(self,relpath):
> > -        #causing problems
> > -        """Free up nodes corresponding to file, if possible"""
> > -        bnode = self.basenodes.get(relpath,None)
> > -        if bnode is None:
> > -            print "Missing node for %s" % relpath
(Continue reading)

skvidal | 7 Sep 19:41 2011

createrepo/readMetadata.py

 createrepo/readMetadata.py |    4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

New commits:
commit a8b1a9d7d8878d5e4bc2194511befe8ab7949fee
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Wed Sep 7 13:40:34 2011 -0400

    remove a bogon error message.

diff --git a/createrepo/readMetadata.py b/createrepo/readMetadata.py
index ac89bf7..54863cb 100644
--- a/createrepo/readMetadata.py
+++ b/createrepo/readMetadata.py
 <at>  <at>  -107,9 +107,7  <at>  <at>  class MetadataIndex(object):
         if relpath in self.pkg_tups_by_path:
             pkgtup = self.pkg_tups_by_path[relpath]
             return self._repo.sack.searchPkgTuple(pkgtup)[0]
-        else:
-            print _("No pkg found for: %s") % relpath
-            return None
+        return None
Dennis Gregorovic | 7 Sep 19:53 2011
Picon

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Wed, 2011-09-07 at 12:58 -0400, seth vidal wrote:
> On Wed, 2011-09-07 at 12:18 -0400, Dennis Gregorovic wrote:
> > On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
> > > -    def getNodes(self, relpath):
> > > -        """Return base, filelist, and other nodes for file, if they exist
> > >  
> > > -        Returns a tuple of nodes, or None if not found
> > > +    def getNodes(self, relpath):
> > > +        """return a package object based on relative path of pkg
> > >          """
> > > -        bnode = self.basenodes.get(relpath,None)
> > > -        if bnode is None:
> > > -            return None
> > > -        pkgid = self.pkg_ids.get(relpath,None)
> > > -        if pkgid is None:
> > > -            print _("No pkgid found for: %s") % relpath
> > > -            return None
> > > -        fnode = self.filesnodes.get(pkgid,None)
> > > -        if fnode is None:
> > > -            return None
> > > -        onode = self.othernodes.get(pkgid,None)
> > > -        if onode is None:
> > > -            return None
> > > -        return bnode, fnode, onode
> > > -
> > > -    def freeNodes(self,relpath):
> > > -        #causing problems
> > > -        """Free up nodes corresponding to file, if possible"""
> > > -        bnode = self.basenodes.get(relpath,None)
> > > -        if bnode is None:
(Continue reading)

seth vidal | 7 Sep 21:29 2011

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Wed, 2011-09-07 at 13:53 -0400, Dennis Gregorovic wrote:
> On Wed, 2011-09-07 at 12:58 -0400, seth vidal wrote:
> > On Wed, 2011-09-07 at 12:18 -0400, Dennis Gregorovic wrote:
> > > On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
> > > > -    def getNodes(self, relpath):
> > > > -        """Return base, filelist, and other nodes for file, if they exist
> > > >  
> > > > -        Returns a tuple of nodes, or None if not found
> > > > +    def getNodes(self, relpath):
> > > > +        """return a package object based on relative path of pkg
> > > >          """
> > > > -        bnode = self.basenodes.get(relpath,None)
> > > > -        if bnode is None:
> > > > -            return None
> > > > -        pkgid = self.pkg_ids.get(relpath,None)
> > > > -        if pkgid is None:
> > > > -            print _("No pkgid found for: %s") % relpath
> > > > -            return None
> > > > -        fnode = self.filesnodes.get(pkgid,None)
> > > > -        if fnode is None:
> > > > -            return None
> > > > -        onode = self.othernodes.get(pkgid,None)
> > > > -        if onode is None:
> > > > -            return None
> > > > -        return bnode, fnode, onode
> > > > -
> > > > -    def freeNodes(self,relpath):
> > > > -        #causing problems
> > > > -        """Free up nodes corresponding to file, if possible"""
> > > > -        bnode = self.basenodes.get(relpath,None)
(Continue reading)

skvidal | 9 Sep 23:12 2011

createrepo/__init__.py createrepo/utils.py docs/createrepo.8 genpkgmetadata.py mergerepo.py

 createrepo/__init__.py |   44 +++++++++++++++++++++++++++++---------------
 createrepo/utils.py    |   22 ++++++++++++++++++++++
 docs/createrepo.8      |   10 +++++++++-
 genpkgmetadata.py      |    4 +++-
 mergerepo.py           |    5 +++++
 5 files changed, 68 insertions(+), 17 deletions(-)

New commits:
commit 773b3dd40ee7ec77b74ea3269cb0b27deafe29c0
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Fri Sep 9 17:11:27 2011 -0400

    add xz support for making repodata - only use it on sqlitedbs and non primary/filelist/other xml data
    leave primary/filelists/other xml files as gz for compatibility
    add xz support to mergerepo

diff --git a/createrepo/__init__.py b/createrepo/__init__.py
index 0fbbff3..7d147a9 100644
--- a/createrepo/__init__.py
+++ b/createrepo/__init__.py
 <at>  <at>  -34,7 +34,7  <at>  <at>  from yum.packageSack import MetaSack
 from yum.packages import YumAvailablePackage

 import rpmUtils.transaction
-from utils import _, errorprint, MDError
+from utils import _, errorprint, MDError, lzma
 import readMetadata
 try:
     import sqlite3 as sqlite
 <at>  <at>  -46,7 +46,7  <at>  <at>  try:
(Continue reading)

skvidal | 9 Sep 23:13 2011

createrepo.spec

 createrepo.spec |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

New commits:
commit 242b042118a74d76b172d17bec857a8c0ca9472f
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Fri Sep 9 17:13:01 2011 -0400

    add lzma require to spec

diff --git a/createrepo.spec b/createrepo.spec
index 1e491cd..eea7092 100644
--- a/createrepo.spec
+++ b/createrepo.spec
 <at>  <at>  -11,7 +11,7  <at>  <at>  URL: http://createrepo.baseurl.org/
 BuildRoot: %{_tmppath}/%{name}-%{version}root
 BuildArchitectures: noarch
 Requires: python >= 2.1, rpm-python, rpm >= 0:4.1.1, libxml2-python
-Requires: yum-metadata-parser, yum >= 3.2.29, python-deltarpm
+Requires: yum-metadata-parser, yum >= 3.2.29, python-deltarpm, pyliblzma

 %description
 This utility will generate a common metadata repository from a directory of
 <at>  <at>  -43,6 +43,9  <at>  <at>  make DESTDIR=$RPM_BUILD_ROOT sysconfdir=%{_sysconfdir} install
 %{python_sitelib}/createrepo

 %changelog
+* Fri Sep  9 2011 Seth Vidal <skvidal at fedoraproject.org>
+- add lzma dep
+
(Continue reading)

Ville Skyttä | 10 Sep 07:32 2011

createrepo.bash

 createrepo.bash |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

New commits:
commit 8c35f58d29cd487ece15c27e1247d155f3a3b65f
Author: Ville Skyttä <ville.skytta <at> iki.fi>
Date:   Sat Sep 10 08:32:09 2011 +0300

    Add --xz to createrepo and mergerepo bash completions.

diff --git a/createrepo.bash b/createrepo.bash
index 0ee7a04..5a32a01 100644
--- a/createrepo.bash
+++ b/createrepo.bash
 <at>  <at>  -48,7 +48,7  <at>  <at>  _cr_createrepo()
             --skip-symlinks --changelog-limit --unique-md-filenames
             --simple-md-filenames --retain-old-md --distro --content --repo
             --revision --deltas --oldpackagedirs --num-deltas --read-pkgs-list
-            --max-delta-rpm-size --workers' -- "$2" ) )
+            --max-delta-rpm-size --workers --xz' -- "$2" ) )
     else
         COMPREPLY=( $( compgen -d -- "$2" ) )
     fi
 <at>  <at>  -70,7 +70,7  <at>  <at>  _cr_mergerepo()
     esac

     COMPREPLY=( $( compgen -W '--version --help --repo --archlist --no-database
-        --outputdir --nogroups --noupdateinfo' -- "$2" ) )
+        --outputdir --nogroups --noupdateinfo --xz' -- "$2" ) )
 } &&
(Continue reading)

James Antill | 14 Sep 17:21 2011

Re: createrepo/__init__.py createrepo/utils.py docs/createrepo.8 genpkgmetadata.py mergerepo.py

On Fri, 2011-09-09 at 21:12 +0000, skvidal <at> osuosl.org wrote:
> createrepo/__init__.py |   44 +++++++++++++++++++++++++++++---------------
>  createrepo/utils.py    |   22 ++++++++++++++++++++++
>  docs/createrepo.8      |   10 +++++++++-
>  genpkgmetadata.py      |    4 +++-
>  mergerepo.py           |    5 +++++
>  5 files changed, 68 insertions(+), 17 deletions(-)
> 
> New commits:
> commit 773b3dd40ee7ec77b74ea3269cb0b27deafe29c0
> Author: Seth Vidal <skvidal <at> fedoraproject.org>
> Date:   Fri Sep 9 17:11:27 2011 -0400
> 
>     add xz support for making repodata - only use it on sqlitedbs and non primary/filelist/other xml data
>     leave primary/filelists/other xml files as gz for compatibility
>     add xz support to mergerepo

 All looks awesome, to me. Couple of minor comments:

> diff --git a/createrepo/__init__.py b/createrepo/__init__.py
> index 0fbbff3..7d147a9 100644
> --- a/createrepo/__init__.py
> +++ b/createrepo/__init__.py

>  <at>  <at>  -1038,7 +1051,7  <at>  <at>  class MetaDataGenerator:
>  
>          if self.conf.additional_metadata:
>              for md_type, md_file in self.conf.additional_metadata.items():
> -                mdcontent = self._createRepoDataObject(md_file, md_type)
> +                mdcontent = self._createRepoDataObject(md_file, md_type, compress_type='xz')
(Continue reading)


Gmane