headers
Michael Bishop | 30 Apr 21:00

Overloading the divide symbol on String to facilitate path concatenation?

Hello fellow rake developers,

I started using this construction in my rake files and I wanted to get your opinion on it (though if there is a
better forum, please direct me there).

Problem
-------

In my rake scripts, I like to assign directory paths to variables so I can use the variables later in my
program. The problem is, when concatenating the variables, I end up with ugly combinations like:

  File.join(BUILD_D, "bin", "x86")
  File.join(CACHE_D, "scripts", SCRIPT_TEMP_NAME)

Possible Solution
-----------------

I find this hard to read so I tried something:

  class String

    def /(path)
      if !path
        self
      else
        File.join(self, path)
      end
    end

  end
(Continue reading)

Permalink | Reply | 0 comments |
headers
Michael Bishop | 27 Apr 15:54

ANN: (yet another) version of -j implementation (includes -m for multitask)

Hello Everyone,

My last implementation of -j had a fatal flaw which could have led to stack overflow for large amounts of
tasks. I discovered this after prototyping a version with drake's feature of turning tasks into multitasks.

I have a new version with a rewritten MultiTask implementation which removes that flaw, is smaller, and
easily fits into Rake code. In addition, I've added an optional "--multitask -m" flag to turn every task
into a multitask (in direct homage to drake).

   https://github.com/michaeljbishop/rake

I've again added a pull-request for the inclusion of the change to the master branch (as always allowing for
further changes to match the style and simplicity of the original)

  https://github.com/jimweirich/rake/pull/113

Questions and comments as always, are welcome.

Sincerely,

Michael Bishop

---

## PROBLEM SUMMARY (THE CASE FOR -j and -m)

Rake can be unusable for builds invoking large numbers of concurrent external processes.

## PROBLEM DESCRIPTION
(Continue reading)

Permalink | Reply | 0 comments |
headers
Michael Bishop | 19 Apr 13:36

Announcement: Alternate -j implementation

Hello Everyone,

I've recently finished an implementation of -j <max_concurrent_jobs> and would be delighted if the members of this list would take a detailed look at it

It's in this rake branch on github:

   https://github.com/michaeljbishop/rake/commit/295c7a4d6d58b3e10c27b940a8259dd3e01c52f0
 or
In addition, I've added a pull-request for the inclusion of the change to the master branch (allowing of course for further changes to match the style and simplicity of the original)

 https://github.com/jimweirich/rake/pull/112

My apologies if this has been hashed out many times on this list. I'm hoping to offer a fresh look at the problem. Thank you for your consideration.

Sincerely,

Michael Bishop


---


SUMMARY
-------

USER INTERFACE:
The user-interface is simply a new -j flag that specifies the maximum number of tasks that can execute simultaneously (discussed here before as I've seen). If the -j flag is omitted, the old behavior of unlimited concurrent tasks is retained.


IMPLEMENTATION:
The implementation is inspired by Apple's Grand Central Dispatch model. The core of the problem is solved via changes to the file "multi-task.rb".

The current implementation of MultiTask#invoke_prerequisites spawns a new thread for each prerequisite and then waits to return until all the threads have completed.

In the alternate implementation, MultiTask#invoke_prerequisites creates a list of blocks inside which each prerequisite is called. Each block is then added to a thread-safe Queue for processing. A class-member ThreadPool is then expanded to include enough threads to consume and call the blocks, but only up to the limit as passed to -j.

Then, rather than MultiTask#invoke_prerequisites sleeping by joining to the newly spawned threads, it participates in the processing of the queued blocks while it waits. This prevents deadlock situations should more threads call MultiTask#invoke_prerequisites.

How then does MultiTask#invoke_prerequisites know when its prerequisites have finished? Before enqueing the prerequisite blocks, it surrounds the original prerequisite block with *another* block which maintains bookkeeping tasks as to when the prerequisite is completed and which thread is working on it. That block is what is added to the queue.

Two conditions need to be met for MultiTask#invoke_prerequisites to return:

  1 - It notices that its prerequisites have all been processed
  2 - It notices there are no more blocks on the queue but its prerequisites are not yet finished. In this case, it joins those threads that are still executing its prerequisites.

What is attractive to me about this implementation is that the flow retains the simplicity of the original: MultiTask#invoke_prerequisites sends all the prerequisites to be executed then waits until they are done.



THE CASE FOR -J
---------------

(Quoted from the pull-request)

PROBLEM SUMMARY:

Rake can be unusable for builds invoking large numbers of concurrent external processes.

PROBLEM DESCRIPTION:

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

SOLUTION SUMMARY:

This request includes the code to add support for a "--jobs NUMBER (-j)" command-line option to specify the number of simultaneous tasks to execute.

SOLUTION:

The solution creates a work queue to which blocks calling the task-prerequisites are added and a thread pool to process them. To prevent deadlock, the task that added the pre-requisites processes items on the queue (alongside the thread pool) until its prerequisites have been processed.

To maintain backward compatibility, not passing -j reverts to the old behavior of unlimited concurrent tasks.

REQUIREMENTS:

The Ruby version requirements remain the same. "multi-task.rb" adds two new requirements: 'thread' and 'set' 

_______________________________________________
Rake-devel mailing list
Rake-devel <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/rake-devel
Permalink | Reply | 11 comments |
headers
Jos Backus | 17 Sep 07:39

Merging drake and rake?

Hi,

Are there any plans to merge drake's features back into rake?

Jos
--

-- 
Jos Backus
jos at catnook.com
Permalink | Reply | 1 comment |
headers
Luis Lavena | 16 Aug 19:03
Picon
Gravatar

No longer able to run Rake tests due fork() usage

Hello,

I guess the subject explain it completely, but I'm not able to run
Rake tests anymore due fork usage in them.

This not only affects Windows but also JRuby.

I know this is related to session gem removal, but was wondering if
there isn't another workaround?

Thank you.
--

-- 
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Permalink | Reply | 0 comments |
headers
Jim Weirich | 6 Aug 05:05
Picon
Gravatar

Beta 1 for Rake 0.9.3 is out

I've pushed a beta version of 0.9.3 out to Rubygems.org.  

Here is the change list.

* The rake test loader now removes arguments it has processed.  Issue #51
* Rake::TaskArguments now responds to #values_at
* RakeFileUtils.verbose_flag = nil silences output the same as 0.8.7
* Rake tests are now directory-independent
* Rake tests are no longer require flexmock
* Commands constant is no longer polluting top level namespace.
* Show only the interesting portion of the backtrace by default (James M. Lawrence).
* Added --reduce-compat optiont to remove backward compatible DSL hacks (James M. Lawrence).

The rake tests task pattern issue isn't explicitly mentioned here, but it seems to be resolved.  Eric, I'm
assuming you did something to fix that (cf. https://github.com/jimweirich/rake/issues/51)

Anyways, grab the beta and give it a spin and report back if there are issues.

Thanks.

--

-- 
-- Jim Weirich
-- jim.weirich <at> gmail.com
Permalink | Reply | 0 comments |
headers
Luis Lavena | 24 Jun 01:37
Picon
Gravatar

3 Failures at master (a774ef8)

Hello,

I'm getting the following failures running against 1.8.7, 1.9.2 and trunk:

  1) Failure:
test_directory_win32(TestRakeDirectoryTask)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_directory_task.rb:32]:
Expected Rake::FileCreationTask, not Rake::FileTask.

  2) Failure:
test_egrep_with_output(TestRakeFileList)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_file_list.rb:345]:
Expected "xyzzy.txt:2:XYZZY", not "xyzzy.txt:2:XYZZY\r\n".

  3) Failure:
test_egrep_with_block(TestRakeFileList)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_file_list.rb:356]:
Expected ["xyzzy.txt", 2, "XYZZY\n"], not ["xyzzy.txt", 2, "XYZZY\r\n"].

Is worth mentioning that I'm using Windows :-)

Seems to me that #2 and #3 are related to newlines output in the
console, perhaps there is another way to verify the output
independently of LF/CR?

--

-- 
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Permalink | Reply | 0 comments |
headers
Jim Weirich | 14 Mar 02:24
Picon
Gravatar

Rake 0.9.0.beta.5 is out

No big functionality changes in this one, just cleaning up some of the cruft leftover from the namespacing
experiment. I expect this version to become 0.9.0 final unless someone discovers a big problem.

--

-- 
-- Jim Weirich
-- jim.weirich <at> gmail.com
Permalink | Reply | 0 comments |
headers
James M. Lawrence | 2 Mar 04:55
Picon
Gravatar

Environment changes in the beta

% cat mystuff.rb
module MyStuff
  CONST_B = 44
end

% cat Rakefile
require 'rake'
require './mystuff'

module MyStuff
  CONST_A = 33
end

p MyStuff::CONST_A
p MyStuff::CONST_B
# => uninitialized constant Rake::Environment::MyStuff::CONST_B

task :default

% rake --version
rake, version 0.9.0.beta.1

% rake
33
rake aborted!
uninitialized constant Rake::Environment::MyStuff::CONST_B

Writing ::MyStuff::CONST_B instead works, and conversely
::MyStuff::CONST_A is an error.

It seems that Rake::DSL already provides the necessary partitioning.
If I wanted to keep the global scope clean, I would do

module MyStuff
  extend Rake::DSL

  task :default do
    # ...
  end
end

Could an example like that in the README replace the need for
environment.rb?
Permalink | Reply | 12 comments |
headers
Jim Weirich | 28 Feb 13:46
Picon
Gravatar

New Beta Version of Rake

A new beta version of Rake is out (0.9.0.beta.1) and is ready for folks to kick the tires.  It's been a while
since the last release and with Eric's help we've been whipping this into shape.

Install with:   gem install rake --pre

The CHANGES file should have all the details.

-- Jim Weirich
Permalink | Reply | 0 comments |
headers
James M. Lawrence | 28 Feb 05:10
Picon
Gravatar

About that new magic comment ...

Assume the following tasks were written for Rake 0.8.x.

  # This comment describes a non-toplevel-task.
  task :complex_task do
    # ...
  end

  # For internal use only -- washes laundry, takes out trash, other
  # crap.
  task :house do
    # ...
  end

  #
  # I prefer this comment style.
  #
  task :style do
    # ...
  end

#### output of rake --tasks:

  rake complex_task  # This comment describes a non-toplevel-task.
  rake house         # crap.

Fortunately my own commenting style avoids this problem. Others will
likely encounter some annoyance (small or possibly great).

One could make the comment-grabber smarter, but the question is: How
should a programmer document a top-level task? There is already an API
method for doing so: desc. The new comment-to-desc feature does not
solve an actual problem or deficiency.

There could hundreds of these new magic-comment 'desc's cropping up,
making the --tasks output useless. Even one newly-public comment could
be an issue (think of a programmer making a veiled complaint about a
co-worker). A programmer may have good reason to use '#' instead of
'desc'. Why should a new version of Rake suddenly dismiss that reason?

Surely there is someone out there who wrote

  (0...1000).each do |i|
    # blah blah ...
    task "blah#{i}" do
      # ...
    end
  end

#### rake --tasks output:

  rake blah0         # blah blah ...
  rake blah1         # blah blah ...
  rake blah10        # blah blah ...
  rake blah100       # blah blah ...
  rake blah101       # blah blah ...
  rake blah102       # blah blah ...
  rake blah103       # blah blah ...
  rake blah104       # blah blah ...
  rake blah105       # blah blah ...
  rake blah106       # blah blah ...
  rake blah107       # blah blah ...
  rake blah108       # blah blah ...
  rake blah109       # blah blah ...
  rake blah11        # blah blah ...
  rake blah110       # blah blah ...
  rake blah111       # blah blah ...
  rake blah112       # blah blah ...
  ...

In exchange for annoying that programmer, what do we get in return?
Usually compatibility is purposefully broken in order to fix a flaw.
What is the flaw in 'desc'? Why is '#' better than 'desc'?

The worst-case scenario is an auto-generated web page of the --tasks
output. With the new Rake, a password in the comments gets exposed.

Previously '#' meant internal documentation for a non-top-level task.
Now it means public documentation for a top-level task, an effective
alias of 'desc'. For internal docs, leave a space between the comment
and the task definition. That is natural for me, but requiring all
programmers to do that in all Rakefiles seems rather unnatural.
Permalink | Reply | 3 comments |

Gmane