Discuss the build tool Jam with other users

headers Ludvig Svenonius | 8 Sep 2008 14:15

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

Hi.

Are you sure you gave us all the relevant lines from -dm? I get the 
following:

make    --        source\Common\PythonBinding\AdfPy_wrap.cxx
time    --        source\Common\PythonBinding\AdfPy_wrap.cxx: missing
make    --         <.>source\Common\PythonBinding\AdfPy_wrap.cxx
bind    --         <.>source\Common\PythonBinding\AdfPy_wrap.cxx:
source\Common\PythonBinding\AdfPy_wrap.cxx
time    --         <.>source\Common\PythonBinding\AdfPy_wrap.cxx: missing
make    --          <.>source\Common\Adf\AdfPy.i
bind    --          <.>source\Common\Adf\AdfPy.i:
source\Common\Adf\AdfPy.i
time    --          <.>source\Common\Adf\AdfPy.i: Tue Sep 09 11:33:29 2008
made*   newer       <.>source\Common\Adf\AdfPy.i
made+   missing    <.>source\Common\PythonBinding\AdfPy_wrap.cxx
make    --         source\Common\Adf\AdfPy.i
time    --         source\Common\Adf\AdfPy.i: Tue Sep 09 11:33:29 2008
made*   newer      source\Common\Adf\AdfPy.i
made+   update    source\Common\PythonBinding\AdfPy_wrap.cxx
made    update   all
made+   update  <jam>command-line

In an isolated test, mine builds the missing target (cxx). When you say 
it refuses, that suggests to me that you depend on the <.> versions? 
What is the line just before the output you gave us, i.e. what is the 
target that depends on <.>cxx ?

I think you should explain your problem that led you to introduce the 
alternate gristed targets as this appears to be the problem to me. 
Perhaps I can provide another solution.

Craig.

Ludvig Svenonius wrote:
> jam -dm yields the following output:
> 
>  
> 
> make    --          <.>source/Common/PythonBinding/AdfPy_wrap.cxx
> 
> bind      --          <.>source/Common/PythonBinding/AdfPy_wrap.cxx: 
> source/Common/PythonBinding/AdfPy_wrap.cxx
> 
> time      --          <.>source/Common/PythonBinding/AdfPy_wrap.cxx: missing
> 
> make    --          <.>source/Common/Adf/AdfPy.i
> 
> bind      --          <.>source/Common/Adf/AdfPy.i: 
> source/Common/Adf/AdfPy.i
> 
> time      --          <.>source/Common/Adf/AdfPy.i: Tue Sep  2 15:17:37 2008
> 
> made*   newer    <.>source/Common/Adf/AdfPy.i
> 
> made+  missing <.>source/Common/PythonBinding/AdfPy_wrap.cxx
> 
>  
> 
> Here are the relevant portions of the Jamfile:
> 
>  
> 
> rule Swig
> 
> {
> 
>     Depends all : $(1) ;
> 
>     Depends $(1) : $(1:G=.) ;
> 
>     Depends $(1) : $(2) ;
> 
>     Depends $(1:G=.) : $(2:G=.) ;
> 
>     Clean clean : $(1) ;
> 
>     if $(PLATFORM) = Linux || $(PLATFORM) = PS3
> 
>     {
> 
>         SWIG_BIN on $(1) = $(SWIG_PATH)/Linux/bin/swig ;
> 
>     }
> 
>     else
> 
>     {
> 
>         SWIG_BIN on $(1) = $(SWIG_PATH)\\Win32\\bin\\swig.exe ;
> 
>     }
> 
>     SWIG_FLAGS on $(1) = $(4) ;
> 
>     SWIG_OUT_DIR on $(1) = $(3) ;
> 
> }
> 
>  
> 
> actions Swig
> 
> {
> 
>      $(SWIG_BIN) -c++ -python -noexcept $(SWIG_FLAGS) -outdir 
> $(SWIG_OUT_DIR) -o $(1) $(2)
> 
> }
> 
>  
> 
> if $(PLATFORM) = Linux || $(PLATFORM) = Win32
> 
> {
> 
>     Swig [ FDirName source Common PythonBinding AdfPy_wrap.cxx ] : [ 
> FDirName source Common Adf AdfPy.i ]  : common : "-D$(PLATFORM:U) 
> -Isource/Common -Iinclude" ;
> 
> }
> 
>  
> 
> (The added <.> grist and extra dependency relations are a workaround for 
> another problem. Source files without this added grist used to not be 
> found at all by Jam, causing builds to fail.)
> 
>  
> 
> Obviously from the -dm log one can tell that there is a proper 
> dependency between the targets 
> <.>source/Common/PythonBinding/AdfPy_wrap.cxx and 
> <.>source/Common/Adf/AdfPy.i, but even though Jam detects this and finds 
> that the former file is missing, whereas the second one is not, it 
> simply refuses to rebuild the dependant target, proceeding instead with 
> a cryptic “made+ missing …” entry in the -dm log. Does anyone have an 
> inkling what may be the cause of this strange behaviour?
> 
>  
> 
> I should perhaps also mention that this only happens sometimes. The very 
> same code build behaved just fine just a few hours ago, and I’m not sure 
> what may have changed on my local workstation since then to break it.
> 
>  
> 
> Thankful for any assistance,
> 
>  
> 
> Ludvig Svenonius
> 
> Build engineer
> 
> Avalanche Studios
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> jamming mailing list  -  jamming <at> perforce.com
> http://maillist.perforce.com/mailman/listinfo/jamming
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

> Hi all,
> 
> 
> 
> Just wrapping my head around Jam and would love help understanding a
> problem a having.
> 
> 
> 
> I'm seeing the grist show up in the shell command of an action:
> 
> e.g. cl /nologo /c /Fo<jobj!stub>accelerometer.generated.obj ...
> 
> 
> 
> My understanding is that the grist should be resolved into a path
> automatically before the
> 
> action is run. Any ideas why that isn't happening?

Actually no. The grist is used for generating unique names for the targets,
so jam don't mix up different targets with the same names. For grist
name you can use anything and (some) practice is to use file path, but
that value has nothing to do with the real file path.

To instruct jam about the real paths, you can point SEARCH_SOURCE or
LOCATE_SOURCE variables to the directories where sources are placed.

> 
> Thanks,
> 
> Kris

Best,
--

-- 
Sanel
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

How is the Objects rule in Jambase supposed to work? Here's what I'm
seeing:
1. Objects foo.cpp
(Objects uses FGristFiles before calling Object.)
2. Object <src!bar>foo.obj <src!bar>foo.cpp
3. C++ <src!bar>foo.obj <src!bar>foo.cpp
4. The C++ action runs on <src!bar>foo.obj and src\bar\foo.cpp.

There are two problems. One is that FGristFiles applies SOURCE_GRIST to
both the source and object files, instead of applying LOCATE_GRIST to
the object file. 

The second is that while <src!bar>foo.cpp somehow ends up as
src\bar\foo.cpp in the action, the target is unchanged and therefore an
invalid path.

Am I using the Jambase rules incorrectly?

Thanks,
Kris

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged
material (including material protected by the solicitor-client or other applicable privileges), or
constitute non-public information. Any use of this information by anyone other than the intended
recipient is prohibited. If you have received this transmission in error, please immediately reply to
the sender and delete this information from your system. Use, dissemination, distribution, or
reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

Hi Kris,

Can you show your jamfile? Does it have SubDir at the top? Jam does
not use LOCATE_GRIST.

What you have is correct but I think you do not have LOCATE_TARGET set
(done by SubDir), thus no path is set for your targets.

Craig.

On Thu, Sep 18, 2008 at 5:06 AM, Kris Braun <kbraun <at> rim.com> wrote:
> How is the Objects rule in Jambase supposed to work? Here's what I'm
> seeing:
> 1. Objects foo.cpp
> (Objects uses FGristFiles before calling Object.)
> 2. Object <src!bar>foo.obj <src!bar>foo.cpp
> 3. C++ <src!bar>foo.obj <src!bar>foo.cpp
> 4. The C++ action runs on <src!bar>foo.obj and src\bar\foo.cpp.
>
> There are two problems. One is that FGristFiles applies SOURCE_GRIST to
> both the source and object files, instead of applying LOCATE_GRIST to
> the object file.
>
> The second is that while <src!bar>foo.cpp somehow ends up as
> src\bar\foo.cpp in the action, the target is unchanged and therefore an
> invalid path.
>
> Am I using the Jambase rules incorrectly?
>
> Thanks,
> Kris
>
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential information, privileged
material (including material protected by the solicitor-client or other applicable privileges), or
constitute non-public information. Any use of this information by anyone other than the intended
recipient is prohibited. If you have received this transmission in error, please immediately reply to
the sender and delete this information from your system. Use, dissemination, distribution, or
reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
>
> _______________________________________________
> jamming mailing list  -  jamming <at> perforce.com
> http://maillist.perforce.com/mailman/listinfo/jamming
>
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

Hi all,

I've had a theory for some time and have now confirmed it with a
simple profiling test. In a typical setup I use daily, with a rather
large project, jam takes about 5 seconds to do nothing. I thought that
there is no way that just checking file dates takes so long, so I've
added a few timers to it and the results are like this:

initialization: 0.00s
rule execution: 3.98s
make: 1.45s
cleanup: 0.00s

Just as I expected, ~73% time is used for rule execution. Which (if I
neither added any new files to the project, nor changed any header
inclusions) yields always the same results - a dependency tree and
per-target actions.

I believe that it would be possible to cache the results of the rule
execution pass in something that would look like a makefile - just
which targets depend on which sources, and which actions to execute if
something is not up to date. This would more than triple the speed for
my example.

Btw, this is actually how make works, if you don't have many ifs in
the makefile. I actually got a push to think in this direction from
this quote: "Have you considered generating GNU makefiles out of
whatever GUI you're using, and build using GNU make?"
(http://lists.midnightryder.com/private.cgi/sweng-gamedev-
midnightryder.com/2008-September/008537.html)

Just that I don't think generating makefiles is needed, since the
jam's dependency engine is good. All that is needed is to cache the
intermediate results between parse_file() and make().

Even though this example is contrived, because you always build
something, it still makes sense even if I change some .cpp file:

initialization: 0.01s
rule execution: 4.02s
updating 6 target(s)...
updated 6 target(s)...
make: 2.89s
cleanup: 0.00s

In this case, an ideal caching scheme (0s rule execution) would more
than double the overall speed.

Has anyone done any work in this direction, so I don't reinvent the
wheel? Does anyone see a roadblock I haven't noticed? Any comments
appreciated!

Thanks,
Alen

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

David wrote:
> how can you guarantee that your cached results correspond to a new invokation ?
> You do realize that any change in the filesystem can invalidate your cached dependencies ?

Why do you think that _any_ change in the filesystem would invalidate
the dependencies? I believe that only specific changes will have such
effect and that we can detect those. I was thinking along these lines:

- Store MD5s of the jamfile(s) that were executed.
- Store MD5s of envvars (either all, or just those that were actually
read by the rules).
- If GLOB rules are used, remember which GLOBs were used and which
results they yielded. (Again, can use MD5 on the returned list of
filenames.)

This should be enough to cache the pre-hrdscan tree. The results of
hdrscan can be cached as well, but... I'd do this one thing at a time.

Am I missing something important here?

Thanks,
Alen

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

Mon	Tue	Wed	Thu	Fri	Sat	Sun

1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

5	October 2012
10	September 2012
1	June 2012
3	July 2011
19	June 2011
13	May 2011
3	March 2011
4	February 2011
4	December 2010
3	November 2010
2	October 2010
9	April 2010
1	January 2010
4	November 2009
3	October 2009
19	September 2009
15	August 2009
7	June 2009
12	May 2009
31	April 2009
15	January 2009
34	December 2008
1	November 2008
8	October 2008
12	September 2008
8	August 2008
6	June 2008
5	February 2008
2	January 2008
5	December 2007
5	November 2007
1	September 2007
22	August 2007
18	July 2007
4	June 2007
17	May 2007
2	April 2007
47	March 2007
6	February 2007
9	January 2007
1	December 2006
2	November 2006
2	October 2006
4	September 2006
15	August 2006
7	July 2006
12	June 2006
11	May 2006
3	April 2006
4	March 2006
2	February 2006
3	January 2006
2	December 2005
3	November 2005
1	September 2005
1	June 2005
1	April 2005
4	January 2005
10	December 2004
13	November 2004
11	October 2004
9	September 2004
9	August 2004
5	July 2004
10	June 2004
29	May 2004
22	April 2004
1	March 2004
17	February 2004
34	January 2004
15	December 2003
19	November 2003
36	October 2003
57	September 2003
10	August 2003
11	July 2003
12	June 2003
15	May 2003
42	April 2003
31	March 2003
87	February 2003
68	January 2003
38	December 2002
35	November 2002
21	October 2002
14	September 2002
7	August 2002
16	July 2002
12	June 2002
31	May 2002
37	April 2002
40	March 2002
40	February 2002
57	January 2002
43	December 2001
54	November 2001
12	October 2001
14	September 2001
61	August 2001
37	July 2001
88	June 2001
39	May 2001
14	April 2001
24	March 2001
12	February 2001
28	January 2001
2	December 2000
5	November 2000
6	October 2000
37	September 2000
18	August 2000
38	July 2000
14	June 2000
20	May 2000
27	April 2000
41	March 2000
33	February 2000
7	January 2000
8	December 1999
8	November 1999
7	October 1999
26	September 1999
35	August 1999
2	July 1999

Jam refuses to rebuild missing target

Re: Jam refuses to rebuild missing target

Girst appearing in an action

Re: Girst appearing in an action

Re: Girst appearing in an action

Re: Girst appearing in an action

Optimizing rule execution time

Re: Optimizing rule execution time

Re: Optimizing rule execution time

Re: Optimizing rule execution time