James Antill | 16 Feb 17:26 2012

createrepo/utils.py

 createrepo/utils.py |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

New commits:
commit 96571b42608e3b45ce56b93f1f1851576bed916d
Author: James Antill <james <at> and.org>
Date:   Thu Feb 16 11:17:12 2012 -0500

    For for split problems due to floating point math (Eg. 9 workers and 5 pkgs).

diff --git a/createrepo/utils.py b/createrepo/utils.py
index 84a43ba..d66b115 100644
--- a/createrepo/utils.py
+++ b/createrepo/utils.py
 <at>  <at>  -183,12 +183,17  <at>  <at>  def encodefiletypelist(filetypelist):
     return result

 def split_list_into_equal_chunks(seq, num_chunks):
+    if num_chunks <= 1:
+        return [seq[:]]
     avg = len(seq) / float(num_chunks)
     out = []
     last = 0.0
-    while last < len(seq):
+    #  Due to floating point math, we do one less than the number of chunks
+    # and then the rest. Eg. range(1,6), 9
+    while len(out) < (num_chunks - 1):
         out.append(seq[int(last):int(last + avg)])
         last += avg
+    out.append(seq[int(last):])
(Continue reading)

James Antill | 16 Feb 17:29 2012

genpkgmetadata.py

 genpkgmetadata.py |    3 +++
 1 file changed, 3 insertions(+)

New commits:
commit 6a3315bfe7704d277c43a8ec5dd3733350105e9e
Author: James Antill <james <at> and.org>
Date:   Thu Feb 16 11:29:26 2012 -0500

    Limit number of workers to 128 (should be enough for anyone :).

diff --git a/genpkgmetadata.py b/genpkgmetadata.py
index ce8e452..9bf8c8d 100755
--- a/genpkgmetadata.py
+++ b/genpkgmetadata.py
 <at>  <at>  -145,6 +145,9  <at>  <at>  def parse_args(args, conf):
     else:
         directories = argsleft

+    if opts.workers >= 128:
+        errorprint(_('Warning: More than 128 workers is a lot. Limiting.'))
+        opts.workers = 128
     if opts.sumtype == 'sha1':
         errorprint(_('Warning: It is more compatible to use sha instead of sha1'))
Ville Skyttä | 18 Feb 09:29 2012
Picon
Picon

Re: genpkgmetadata.py

On 2012-02-16 18:29, James Antill wrote:

>     Limit number of workers to 128 (should be enough for anyone :).

Sure, but out of curiosity, why?
seth vidal | 18 Feb 16:47 2012

Re: genpkgmetadata.py

On Sat, 18 Feb 2012 10:29:46 +0200
Ville Skyttä <ville.skytta <at> iki.fi> wrote:

> On 2012-02-16 18:29, James Antill wrote:
> 
> >     Limit number of workers to 128 (should be enough for anyone :).
> 
> Sure, but out of curiosity, why?

Number of workers is bound pretty tightly to number of processors/cores
you have.

-sv
_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
Ville Skyttä | 18 Feb 18:28 2012
Picon
Picon

Re: genpkgmetadata.py

On 2012-02-18 17:47, seth vidal wrote:
> On Sat, 18 Feb 2012 10:29:46 +0200
> Ville Skyttä <ville.skytta <at> iki.fi> wrote:
> 
>> On 2012-02-16 18:29, James Antill wrote:
>>
>>>     Limit number of workers to 128 (should be enough for anyone :).
>>
>> Sure, but out of curiosity, why?
> 
> Number of workers is bound pretty tightly to number of processors/cores
> you have.

That still doesn't answer the question why this particular hunk of code
was added, but ... meh, nevermind.
_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
James Antill | 22 Feb 17:16 2012

Re: genpkgmetadata.py

On Sat, 2012-02-18 at 10:29 +0200, Ville Skyttä wrote:
> On 2012-02-16 18:29, James Antill wrote:
> 
> >     Limit number of workers to 128 (should be enough for anyone :).
> 
> Sure, but out of curiosity, why?

 Well if the user does --workers=666666 ... I think we can both agree
bad things will probably happen. You think the limit should be higher? I
picked 128 kind of randomly, although --workers=0 will still do it as
number of cores even if you have 1024 cores.

_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
Ville Skyttä | 22 Feb 18:06 2012
Picon
Picon

Re: genpkgmetadata.py

On 2012-02-22 18:16, James Antill wrote:

>  Well if the user does --workers=666666 ... I think we can both agree
> bad things will probably happen. You think the limit should be higher?

No, I think "don't do that, then" is perfectly adequate advice for cases
like that; no code needed for that nor for every other "bad"
option/argument users might end up inventing (doesn't seem that
createrepo in particular necessarily has any others, but I mean in general).
Panu Matilainen | 23 Feb 09:40 2012

Re: genpkgmetadata.py

On 02/22/2012 06:16 PM, James Antill wrote:
> On Sat, 2012-02-18 at 10:29 +0200, Ville Skyttä wrote:
>> On 2012-02-16 18:29, James Antill wrote:
>>
>>>      Limit number of workers to 128 (should be enough for anyone :).
>>
>> Sure, but out of curiosity, why?
>
>   Well if the user does --workers=666666 ... I think we can both agree
> bad things will probably happen. You think the limit should be higher? I
> picked 128 kind of randomly, although --workers=0 will still do it as
> number of cores even if you have 1024 cores.

Since the point is to protect the innocent... I suspect 128 workers 
would quite effectively kill a lowly single-core system anyway. Wouldn't 
it be more useful to cap it based on the actual number of cores on the 
system? Say, num_cpus_online() times two or four or thereabouts, to 
allow testing multiple workers on non-SMP system but with a reasonably 
safe upper limit.

Not that I actually care, just seems like a nice subject for bikeshedding :P

	- Panu -
_______________________________________________
Rpm-metadata mailing list
Rpm-metadata <at> lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/rpm-metadata

Gmane