Discussion of the data format for repositories of rpms ()

Seth Vidal | 15 Jul 2011 23:53

[PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

---
 createrepo/__init__.py     |   40 ++------
 createrepo/readMetadata.py |  240 +++++++++++++------------------------------
 2 files changed, 83 insertions(+), 197 deletions(-)

diff --git a/createrepo/__init__.py b/createrepo/__init__.py
index 44035cc..8549188 100644
--- a/createrepo/__init__.py
+++ b/createrepo/__init__.py
 <at>  <at>  -530,39 +530,19  <at>  <at>  class MetaDataGenerator:
                 old_pkg = pkg
                 if pkg.find("://") != -1:
                     old_pkg = os.path.basename(pkg)
-                nodes = self.oldData.getNodes(old_pkg)
-                if nodes is not None: # we have a match in the old metadata
+                old_po = self.oldData.getNodes(old_pkg)
+                if old_po: # we have a match in the old metadata
                     if self.conf.verbose:
                         self.callback.log(_("Using data from old metadata for %s")
                                             % pkg)
-                    (primarynode, filenode, othernode) = nodes
-
-                    for node, outfile in ((primarynode, self.primaryfile),
-                                          (filenode, self.flfile),
-                                          (othernode, self.otherfile)):
-                        if node is None:
-                            break
-
-                        if self.conf.baseurl:
-                            anode = node.children

(Continue reading)

tim.lauridsen@gmail.com | 17 Jul 2011 11:20

Picon

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Fri, Jul 15, 2011 at 11:53 PM, Seth Vidal <skvidal <at> fedoraproject.org> wrote:
> ---
>  createrepo/__init__.py     |   40 ++------
>  createrepo/readMetadata.py |  240 +++++++++++++------------------------------
>  2 files changed, 83 insertions(+), 197 deletions(-)
>
> diff --git a/createrepo/__init__.py b/createrepo/__init__.py
> index 44035cc..8549188 100644
> --- a/createrepo/__init__.py
> +++ b/createrepo/__init__.py
>  <at>  <at>  -530,39 +530,19  <at>  <at>  class MetaDataGenerator:
>                 old_pkg = pkg
>                 if pkg.find("://") != -1:
>                     old_pkg = os.path.basename(pkg)
> -                nodes = self.oldData.getNodes(old_pkg)
> -                if nodes is not None: # we have a match in the old metadata
> +                old_po = self.oldData.getNodes(old_pkg)
> +                if old_po: # we have a match in the old metadata
>                     if self.conf.verbose:
>                         self.callback.log(_("Using data from old metadata for %s")
>                                             % pkg)
> -                    (primarynode, filenode, othernode) = nodes
> -
> -                    for node, outfile in ((primarynode, self.primaryfile),
> -                                          (filenode, self.flfile),
> -                                          (othernode, self.otherfile)):
> -                        if node is None:
> -                            break
> -
> -                        if self.conf.baseurl:

(Continue reading)

seth vidal | 18 Jul 2011 07:35

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Sun, 2011-07-17 at 11:20 +0200, tim.lauridsen <at> gmail.com wrote:
> > _______________________________________________
> > Rpm-metadata mailing list
> > Rpm-metadata <at> lists.baseurl.org
> > http://lists.baseurl.org/mailman/listinfo/rpm-metadata
> >
> 
> ACK, Looks good to me
> 

I'm going to do some more testing on it today to make sure the memory
footprint is sane before I commit it.

-sv

tim.lauridsen@gmail.com | 18 Jul 2011 17:14

Picon

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Mon, Jul 18, 2011 at 7:35 AM, seth vidal <skvidal <at> fedoraproject.org> wrote:
> On Sun, 2011-07-17 at 11:20 +0200, tim.lauridsen <at> gmail.com wrote:
>> > _______________________________________________
>> > Rpm-metadata mailing list
>> > Rpm-metadata <at> lists.baseurl.org
>> > http://lists.baseurl.org/mailman/listinfo/rpm-metadata
>> >
>>
>> ACK, Looks good to me
>>
>
> I'm going to do some more testing on it today to make sure the memory
> footprint is sane before I commit it.
>

Sound like a good idea :)

Tim

seth vidal | 18 Jul 2011 21:59

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
> ---
>  createrepo/__init__.py     |   40 ++------
>  createrepo/readMetadata.py |  240 +++++++++++++------------------------------
>  2 files changed, 83 insertions(+), 197 deletions(-)

Tested locally on repodata of 9000 pkgs.

Goes from 1.8-> 2GB of  memory in use with the old createrepo code to
325MB of memory in use - same operation - performance-wise it is not
considerably different. More testing will bear that out, though.

I think I'll push this

-sv

skvidal | 18 Jul 2011 22:30

createrepo/init.py createrepo/readMetadata.py

 createrepo/__init__.py     |   40 +------
 createrepo/readMetadata.py |  240 +++++++++++++--------------------------------
 2 files changed, 83 insertions(+), 197 deletions(-)

New commits:
commit 0a67bc57a9eda626735513a4015d8087f3f4bb29
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Fri Jul 15 17:50:48 2011 -0400

    switch --update to use the sqlite dbs instead of the xml files. Should massively impact
    memory footprint and hopefully only marginally impact performance.

diff --git a/createrepo/__init__.py b/createrepo/__init__.py
index 44035cc..8549188 100644
--- a/createrepo/__init__.py
+++ b/createrepo/__init__.py
 <at>  <at>  -530,39 +530,19  <at>  <at>  class MetaDataGenerator:
                 old_pkg = pkg
                 if pkg.find("://") != -1:
                     old_pkg = os.path.basename(pkg)
-                nodes = self.oldData.getNodes(old_pkg)
-                if nodes is not None: # we have a match in the old metadata
+                old_po = self.oldData.getNodes(old_pkg)
+                if old_po: # we have a match in the old metadata
                     if self.conf.verbose:
                         self.callback.log(_("Using data from old metadata for %s")
                                             % pkg)
-                    (primarynode, filenode, othernode) = nodes
-
-                    for node, outfile in ((primarynode, self.primaryfile),

(Continue reading)

tim.lauridsen@gmail.com | 19 Jul 2011 07:13

Picon

Re: [PATCH] switch --update to use the sqlite dbs instead of the xml files. Should massively impact memory footprint and hopefully only marginally impact performance.

On Mon, Jul 18, 2011 at 9:59 PM, seth vidal <skvidal <at> fedoraproject.org> wrote:
> On Fri, 2011-07-15 at 17:53 -0400, Seth Vidal wrote:
>> ---
>>  createrepo/__init__.py     |   40 ++------
>>  createrepo/readMetadata.py |  240 +++++++++++++------------------------------
>>  2 files changed, 83 insertions(+), 197 deletions(-)
>
>
> Tested locally on repodata of 9000 pkgs.
>
> Goes from 1.8-> 2GB of  memory in use with the old createrepo code to
> 325MB of memory in use - same operation - performance-wise it is not
> considerably different. More testing will bear that out, though.

1.8 GB -> 328 MB sound like a winner :)

Tim
_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata

skvidal | 20 Jul 2011 23:03

createrepo/readMetadata.py

 createrepo/readMetadata.py |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

New commits:
commit 7e24b1ba2c939cff52f789b599d907c8fdac9747
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Wed Jul 20 17:02:42 2011 -0400

    changes to make this more robust:
    1. when making a new repo - use mkdtemp() so we don't run the possibility of a race to the repo cache
    2. try/except when we setup the old repodata so if the dir is not there or the repo is broken we don't traceback

diff --git a/createrepo/readMetadata.py b/createrepo/readMetadata.py
index a449e68..88e5d95 100644
--- a/createrepo/readMetadata.py
+++ b/createrepo/readMetadata.py
 <at>  <at>  -21,8 +21,8  <at>  <at>  from utils import errorprint, _

 import yum
 from yum import misc
-
-
+from yum.Errors import YumBaseError
+import tempfile
 class CreaterepoPkgOld(yum.sqlitesack.YumAvailablePackageSqlite):
     # special for special people like us.
     def _return_remote_location(self):
 <at>  <at>  -48,13 +48,16  <at>  <at>  class MetadataIndex(object):
         repodatadir = self.outputdir + '/repodata'
         self._repo = yum.yumRepo.YumRepository('garbageid')

(Continue reading)

skvidal | 28 Jul 2011 00:08

2 commits - createrepo/init.py genpkgmetadata.py

 createrepo/__init__.py |   40 +++++++++++++++++++++++++++++++++-------
 genpkgmetadata.py      |    2 ++
 2 files changed, 35 insertions(+), 7 deletions(-)

New commits:
commit 79b7871aaba432e267a8e15a4070118a9168f9e4
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Wed Jul 27 18:07:35 2011 -0400

    reset the worker cmd location, whoops

diff --git a/createrepo/__init__.py b/createrepo/__init__.py
index 769defa..8cce31a 100644
--- a/createrepo/__init__.py
+++ b/createrepo/__init__.py
 <at>  <at>  -107,9 +107,9  <at>  <at>  class MetaDataConfig(object):
                                    # read in this run of createrepo
         self.collapse_glibc_requires = True
         self.workers = 1 # number of workers to fork off to grab metadata from the pkgs
-        #self.worker_cmd = '/usr/share/createrepo/worker.py'
+        self.worker_cmd = '/usr/share/createrepo/worker.py'

-        self.worker_cmd = './worker.py' # helpful when testing
+        #self.worker_cmd = './worker.py' # helpful when testing
         self.retain_old_md = 0

 class SimpleMDCallBack(object):
commit 180e3042bf7337edba2b80dd867f3f7953622725
Author: Seth Vidal <skvidal <at> fedoraproject.org>
Date:   Wed Jul 27 18:05:17 2011 -0400

(Continue reading)

Ville Skyttä | 28 Jul 2011 20:03

createrepo.bash

 createrepo.bash |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

New commits:
commit 377318ac5de5a402f8ce537b4ac9d5c64e6286af
Author: Ville SkyttÃ¤ <ville.skytta <at> iki.fi>
Date:   Thu Jul 28 21:01:23 2011 +0300

    Add --retain-old-md bash completion.

diff --git a/createrepo.bash b/createrepo.bash
index 54ac8b2..4222fa0 100644
--- a/createrepo.bash
+++ b/createrepo.bash
 <at>  <at>  -30,6 +30,10  <at>  <at>  _cr_createrepo()
             COMPREPLY=( $( compgen -f -o plusdirs -X '!*.rpm' -- "$2" ) )
             return 0
             ;;
+        --retain-old-md)
+            COMPREPLY=( $( compgen -W '0 1 2 3 4 5 6 7 8 9' -- "$2" ) )
+            return 0
+            ;;
         --num-deltas)
             COMPREPLY=( $( compgen -W '1 2 3 4 5 6 7 8 9' -- "$2" ) )
             return 0
 <at>  <at>  -42,8 +46,8  <at>  <at>  _cr_createrepo()
             --cachedir --checkts --no-database --update --update-md-path
             --skip-stat --split --pkglist --includepkg --outputdir
             --skip-symlinks --changelog-limit --unique-md-filenames
-            --simple-md-filenames --distro --content --repo --revision --deltas

(Continue reading)

Group

gmane.linux.rpm.metadata.

Project Web Page

Discussion of the data format for repositories of rpms ()

Search Archive

Page

1 | 2

Articles in period: 16

July 2011

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane

Recent entries

createrepo.spec (17 Apr 2013)
modifyrepo.py (16 Apr 2013)
docs/createrepo.8 genpkgmetadata.py (16 Apr 2013)
createrepo.bash (13 Apr 2013)
modifyrepo.py (11 Apr 2013)
[PATCH] modifyrepo: fix --compress option bug. BZ 950724 (11 Apr 2013)
3 commits - createrepo.spec genpkgmetadata.py Makefile modifyrepo.py (5 Apr 2013)
[PATCH 1/2] Get rid of useless opts.simple_md_filenames option, update (5 Apr 2013)
2 commits - createrepo.bash (5 Apr 2013)
[PATCH] move bash-completion scripts to /usr/share/ BZ 923001 (20 Mar 2013)
Changes to 'refs/tags/createrepo-0-9-9' (6 Mar 2013)
modifyrepo.py (6 Mar 2013)
URL-encoded hrefs (4 Mar 2013)
worker.py (1 Mar 2013)
createrepo/__init__.py (17 Dec 2012)
3 commits - createrepo/__init__.py createrepo/utils.py worker.py (29 Nov 2012)
2 commits - createrepo/__init__.py (26 Nov 2012)
createrepo not generate deltarpm in case of update (20 Nov 2012)
createrepo.bash (20 Nov 2012)
2 commits - docs/createrepo.8 modifyrepo.py (19 Nov 2012)
createrepo/__init__.py (12 Nov 2012)

Archives

10	April 2013
12	March 2013
5	December 2012
10	November 2012
2	October 2012
3	September 2012
1	August 2012
3	July 2012
1	March 2012
8	February 2012
3	January 2012
1	December 2011
14	November 2011
3	October 2011
25	September 2011
12	August 2011
16	July 2011
2	June 2011
1	April 2011
4	March 2011
7	February 2011
11	January 2011
6	November 2010
5	October 2010
10	September 2010
118	August 2010
8	June 2010
9	May 2010
13	April 2010
2	March 2010
10	February 2010
9	January 2010
8	November 2009
3	October 2009
1	September 2009
15	August 2009
3	July 2009
4	June 2009
9	May 2009
47	April 2009
3	March 2009
3	February 2009
16	January 2009
7	December 2008
2	November 2008
28	October 2008
19	September 2008
2	August 2008
3	July 2008
2	June 2008
6	May 2008
15	April 2008
5	March 2008
20	February 2008
72	January 2008
17	December 2007
21	November 2007
15	October 2007
8	August 2007
9	June 2007
9	May 2007
9	March 2007
4	February 2007
13	January 2007
6	November 2006
10	October 2006
16	September 2006
32	August 2006
14	July 2006
22	June 2006
6	May 2006
13	April 2006
8	March 2006
7	February 2006
9	January 2006
3	December 2005
41	November 2005
10	October 2005
2	September 2005
8	July 2005
1	June 2005
8	May 2005
10	April 2005
7	March 2005
23	February 2005
29	January 2005
9	December 2004
3	November 2004
11	October 2004
15	September 2004
13	August 2004
11	July 2004
29	June 2004
8	May 2004
7	April 2004
7	February 2004
28	January 2004
16	December 2003
65	November 2003
183	October 2003
15	September 2003
31	August 2003

Design

Zawodny | Right Menu | Left Menu

Your Own Design

Paste the URL of your CSS below. Download CSS template