8 Sep 2008 14:15

### Jam refuses to rebuild missing target

jam -dm yields the following output:

Here are the relevant portions of the Jamfile:

rule Swig

{

Depends all : $(1) ; Depends$(1) : $(1:G=.) ; Depends$(1) : $(2) ; Depends$(1:G=.) : $(2:G=.) ; Clean clean :$(1) ;

if $(PLATFORM) = Linux ||$(PLATFORM) = PS3

{

SWIG_BIN on $(1) =$(SWIG_PATH)/Linux/bin/swig ;

}

else

{

SWIG_BIN on $(1) =$(SWIG_PATH)\\Win32\\bin\\swig.exe ;

}

SWIG_FLAGS on $(1) =$(4) ;

SWIG_OUT_DIR on $(1) =$(3) ;

}

actions Swig

{

$(SWIG_BIN) -c++ -python -noexcept$(SWIG_FLAGS) -outdir $(SWIG_OUT_DIR) -o$(1) $(2) } if$(PLATFORM) = Linux || $(PLATFORM) = Win32 { Swig [ FDirName source Common PythonBinding AdfPy_wrap.cxx ] : [ FDirName source Common Adf AdfPy.i ] : common : "-D$(PLATFORM:U) -Isource/Common -Iinclude" ;

}

(The added <.> grist and extra dependency relations are a workaround for another problem. Source files without this added grist used to not be found at all by Jam, causing builds to fail.)

Obviously from the -dm log one can tell that there is a proper dependency between the targets <.>source/Common/PythonBinding/AdfPy_wrap.cxx and <.>source/Common/Adf/AdfPy.i, but even though Jam detects this and finds that the former file is missing, whereas the second one is not, it simply refuses to rebuild the dependant target, proceeding instead with a cryptic “made+ missing …” entry in the -dm log. Does anyone have an inkling what may be the cause of this strange behaviour?

I should perhaps also mention that this only happens sometimes. The very same code build behaved just fine just a few hours ago, and I’m not sure what may have changed on my local workstation since then to break it.

Thankful for any assistance,

Ludvig Svenonius

Build engineer

Avalanche Studios

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming
9 Sep 2008 03:55

### Re: Jam refuses to rebuild missing target

Hi.

Are you sure you gave us all the relevant lines from -dm? I get the
following:

In an isolated test, mine builds the missing target (cxx). When you say
it refuses, that suggests to me that you depend on the <.> versions?
What is the line just before the output you gave us, i.e. what is the
target that depends on <.>cxx ?

I think you should explain your problem that led you to introduce the
alternate gristed targets as this appears to be the problem to me.
Perhaps I can provide another solution.

Craig.

Ludvig Svenonius wrote:
> jam -dm yields the following output:
>
>
>
>
>
>
>
>
>
>
>
>
>
> Here are the relevant portions of the Jamfile:
>
>
>
> rule Swig
>
> {
>
>     Depends all : $(1) ; > > Depends$(1) : $(1:G=.) ; > > Depends$(1) : $(2) ; > > Depends$(1:G=.) : $(2:G=.) ; > > Clean clean :$(1) ;
>
>     if $(PLATFORM) = Linux ||$(PLATFORM) = PS3
>
>     {
>
>         SWIG_BIN on $(1) =$(SWIG_PATH)/Linux/bin/swig ;
>
>     }
>
>     else
>
>     {
>
>         SWIG_BIN on $(1) =$(SWIG_PATH)\\Win32\\bin\\swig.exe ;
>
>     }
>
>     SWIG_FLAGS on $(1) =$(4) ;
>
>     SWIG_OUT_DIR on $(1) =$(3) ;
>
> }
>
>
>
> actions Swig
>
> {
>
>      $(SWIG_BIN) -c++ -python -noexcept$(SWIG_FLAGS) -outdir
> $(SWIG_OUT_DIR) -o$(1) $(2) > > } > > > > if$(PLATFORM) = Linux || $(PLATFORM) = Win32 > > { > > Swig [ FDirName source Common PythonBinding AdfPy_wrap.cxx ] : [ > FDirName source Common Adf AdfPy.i ] : common : "-D$(PLATFORM:U)
> -Isource/Common -Iinclude" ;
>
> }
>
>
>
> (The added <.> grist and extra dependency relations are a workaround for
> another problem. Source files without this added grist used to not be
> found at all by Jam, causing builds to fail.)
>
>
>
> Obviously from the -dm log one can tell that there is a proper
> dependency between the targets
> that the former file is missing, whereas the second one is not, it
> simply refuses to rebuild the dependant target, proceeding instead with
> a cryptic “made+ missing …” entry in the -dm log. Does anyone have an
> inkling what may be the cause of this strange behaviour?
>
>
>
> I should perhaps also mention that this only happens sometimes. The very
> same code build behaved just fine just a few hours ago, and I’m not sure
> what may have changed on my local workstation since then to break it.
>
>
>
> Thankful for any assistance,
>
>
>
> Ludvig Svenonius
>
> Build engineer
>
> Avalanche Studios
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> jamming mailing list  -  jamming <at> perforce.com
> http://maillist.perforce.com/mailman/listinfo/jamming
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

17 Sep 2008 17:40

### Girst appearing in an action

Hi all,

Just wrapping my head around Jam and would love help understanding a problem a having.

I’m seeing the grist show up in the shell command of an action:

e.g. cl /nologo /c /Fo<jobj!stub>accelerometer.generated.obj …

My understanding is that the grist should be resolved into a path automatically before the

action is run. Any ideas why that isn’t happening?

Thanks,

Kris

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming
17 Sep 2008 18:12

### Re: Girst appearing in an action

> Hi all,
>
>
>
> Just wrapping my head around Jam and would love help understanding a
> problem a having.
>
>
>
> I'm seeing the grist show up in the shell command of an action:
>
> e.g. cl /nologo /c /Fo<jobj!stub>accelerometer.generated.obj ...
>
>
>
> My understanding is that the grist should be resolved into a path
> automatically before the
>
> action is run. Any ideas why that isn't happening?

Actually no. The grist is used for generating unique names for the targets,
so jam don't mix up different targets with the same names. For grist
name you can use anything and (some) practice is to use file path, but
that value has nothing to do with the real file path.

To instruct jam about the real paths, you can point SEARCH_SOURCE or
LOCATE_SOURCE variables to the directories where sources are placed.

>
> Thanks,
>
> Kris

Best,
--

--
Sanel
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

17 Sep 2008 21:06

### Re: Girst appearing in an action

How is the Objects rule in Jambase supposed to work? Here's what I'm
seeing:
1. Objects foo.cpp
(Objects uses FGristFiles before calling Object.)
2. Object <src!bar>foo.obj <src!bar>foo.cpp
3. C++ <src!bar>foo.obj <src!bar>foo.cpp
4. The C++ action runs on <src!bar>foo.obj and src\bar\foo.cpp.

There are two problems. One is that FGristFiles applies SOURCE_GRIST to
both the source and object files, instead of applying LOCATE_GRIST to
the object file.

The second is that while <src!bar>foo.cpp somehow ends up as
src\bar\foo.cpp in the action, the target is unchanged and therefore an
invalid path.

Am I using the Jambase rules incorrectly?

Thanks,
Kris

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged
material (including material protected by the solicitor-client or other applicable privileges), or
constitute non-public information. Any use of this information by anyone other than the intended
the sender and delete this information from your system. Use, dissemination, distribution, or
reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

18 Sep 2008 00:04

### Re: Girst appearing in an action

Hi Kris,

Can you show your jamfile? Does it have SubDir at the top? Jam does
not use LOCATE_GRIST.

What you have is correct but I think you do not have LOCATE_TARGET set
(done by SubDir), thus no path is set for your targets.

Craig.

On Thu, Sep 18, 2008 at 5:06 AM, Kris Braun <kbraun <at> rim.com> wrote:
> How is the Objects rule in Jambase supposed to work? Here's what I'm
> seeing:
> 1. Objects foo.cpp
> (Objects uses FGristFiles before calling Object.)
> 2. Object <src!bar>foo.obj <src!bar>foo.cpp
> 3. C++ <src!bar>foo.obj <src!bar>foo.cpp
> 4. The C++ action runs on <src!bar>foo.obj and src\bar\foo.cpp.
>
> There are two problems. One is that FGristFiles applies SOURCE_GRIST to
> both the source and object files, instead of applying LOCATE_GRIST to
> the object file.
>
> The second is that while <src!bar>foo.cpp somehow ends up as
> src\bar\foo.cpp in the action, the target is unchanged and therefore an
> invalid path.
>
> Am I using the Jambase rules incorrectly?
>
> Thanks,
> Kris
>
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential information, privileged
material (including material protected by the solicitor-client or other applicable privileges), or
constitute non-public information. Any use of this information by anyone other than the intended
the sender and delete this information from your system. Use, dissemination, distribution, or
reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
>
> _______________________________________________
> jamming mailing list  -  jamming <at> perforce.com
> http://maillist.perforce.com/mailman/listinfo/jamming
>
_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

26 Sep 2008 11:10

### Optimizing rule execution time

Hi all,

I've had a theory for some time and have now confirmed it with a
simple profiling test. In a typical setup I use daily, with a rather
large project, jam takes about 5 seconds to do nothing. I thought that
there is no way that just checking file dates takes so long, so I've
added a few timers to it and the results are like this:

initialization: 0.00s
rule execution: 3.98s
make: 1.45s
cleanup: 0.00s

Just as I expected, ~73% time is used for rule execution. Which (if I
neither added any new files to the project, nor changed any header
inclusions) yields always the same results - a dependency tree and
per-target actions.

I believe that it would be possible to cache the results of the rule
execution pass in something that would look like a makefile - just
which targets depend on which sources, and which actions to execute if
something is not up to date. This would more than triple the speed for
my example.

Btw, this is actually how make works, if you don't have many ifs in
the makefile. I actually got a push to think in this direction from
this quote: "Have you considered generating GNU makefiles out of
whatever GUI you're using, and build using GNU make?"
(http://lists.midnightryder.com/private.cgi/sweng-gamedev-
midnightryder.com/2008-September/008537.html)

Just that I don't think generating makefiles is needed, since the
jam's dependency engine is good. All that is needed is to cache the
intermediate results between parse_file() and make().

Even though this example is contrived, because you always build
something, it still makes sense even if I change some .cpp file:

initialization: 0.01s
rule execution: 4.02s
updating 6 target(s)...
updated 6 target(s)...
make: 2.89s
cleanup: 0.00s

In this case, an ideal caching scheme (0s rule execution) would more
than double the overall speed.

Has anyone done any work in this direction, so I don't reinvent the
appreciated!

Thanks,
Alen

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

26 Sep 2008 12:27

### Re: Optimizing rule execution time

Hello Alen,

how can you guarantee that your cached results correspond to a new invokation ?
You do realize that any change in the filesystem can invalidate your cached dependencies ?

regards,

- David

Hi all,

I've had a theory for some time and have now confirmed it with a
simple profiling test. In a typical setup I use daily, with a rather
large project, jam takes about 5 seconds to do nothing. I thought that
there is no way that just checking file dates takes so long, so I've
added a few timers to it and the results are like this:

initialization: 0.00s
rule execution: 3.98s
make: 1.45s
cleanup: 0.00s

Just as I expected, ~73% time is used for rule execution. Which (if I
neither added any new files to the project, nor changed any header
inclusions) yields always the same results - a dependency tree and
per-target actions.

I believe that it would be possible to cache the results of the rule
execution pass in something that would look like a makefile - just
which targets depend on which sources, and which actions to execute if
something is not up to date. This would more than triple the speed for
my example.

Btw, this is actually how make works, if you don't have many ifs in
the makefile. I actually got a push to think in this direction from
this quote: "Have you considered generating GNU makefiles out of
whatever GUI you're using, and build using GNU make?"
(http://lists.midnightryder.com/private.cgi/sweng-gamedev-
midnightryder.com/2008-September/008537.html)

Just that I don't think generating makefiles is needed, since the
jam's dependency engine is good. All that is needed is to cache the
intermediate results between parse_file() and make().

Even though this example is contrived, because you always build
something, it still makes sense even if I change some .cpp file:

initialization: 0.01s
rule execution: 4.02s
updating 6 target(s)...
updated 6 target(s)...
make: 2.89s
cleanup: 0.00s

In this case, an ideal caching scheme (0s rule execution) would more
than double the overall speed.

Has anyone done any work in this direction, so I don't reinvent the
appreciated!

Thanks,
Alen

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming
26 Sep 2008 16:48

### Re: Optimizing rule execution time

David wrote:
> how can you guarantee that your cached results correspond to a new invokation ?
> You do realize that any change in the filesystem can invalidate your cached dependencies ?

Why do you think that _any_ change in the filesystem would invalidate
the dependencies? I believe that only specific changes will have such
effect and that we can detect those. I was thinking along these lines:

- Store MD5s of the jamfile(s) that were executed.
- Store MD5s of envvars (either all, or just those that were actually
- If GLOB rules are used, remember which GLOBs were used and which
results they yielded. (Again, can use MD5 on the returned list of
filenames.)

This should be enough to cache the pre-hrdscan tree. The results of
hdrscan can be cached as well, but... I'd do this one thing at a time.

Am I missing something important here?

Thanks,
Alen

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

26 Sep 2008 18:57

### Re: Optimizing rule execution time

Header-scan caching is already available. It's been so long, I've forgotten now who provided it, but it should be easy to find. Try searching the archives for HCACHEFILE, or "header cache", something along those lines.

Diane

On Fri, Sep 26, 2008 at 7:48 AM, Alen Ladavac wrote:
David wrote:
> how can you guarantee that your cached results correspond to a new invokation ?
> You do realize that any change in the filesystem can invalidate your cached dependencies ?

Why do you think that _any_ change in the filesystem would invalidate
the dependencies? I believe that only specific changes will have such
effect and that we can detect those. I was thinking along these lines:

- Store MD5s of the jamfile(s) that were executed.
- Store MD5s of envvars (either all, or just those that were actually
- If GLOB rules are used, remember which GLOBs were used and which
results they yielded. (Again, can use MD5 on the returned list of
filenames.)

This should be enough to cache the pre-hrdscan tree. The results of
hdrscan can be cached as well, but... I'd do this one thing at a time.

Am I missing something important here?

Thanks,
Alen

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

_______________________________________________
jamming mailing list  -  jamming <at> perforce.com
http://maillist.perforce.com/mailman/listinfo/jamming

Gmane