headers
Kay Röpke | 2 Aug 2008 17:15

Problem with LL1Analyzer

Hi!

FYI,

I've ran into a problem with the LL1Analyzer:
Upon adding a syntactic predicate to resolve an ambiguous function  
prototype in SQL both 3.1b1 and b2 crashed:

classdump:sqlparser kroepke$ antlr312 MySQL51.g
ANTLR Parser Generator  Version 3.1b2 (July 17, 2008)  1989-2008
error(10):  internal error: MySQL51.g :  
java.lang.IndexOutOfBoundsException: Index: 32, Size: 32
java.util.ArrayList.RangeCheck(ArrayList.java:546)
java.util.ArrayList.get(ArrayList.java:321)
org.antlr.misc.IntervalSet.addAll(IntervalSet.java:197)
org.antlr.analysis.LookaheadSet.orInPlace(LookaheadSet.java:56)
org.antlr.analysis.LL1Analyzer._FIRST(LL1Analyzer.java:270)
org.antlr.analysis.LL1Analyzer._FIRST(LL1Analyzer.java:268)
org.antlr.analysis.LL1Analyzer._FIRST(LL1Analyzer.java:239)
org.antlr.analysis.LL1Analyzer.FIRST(LL1Analyzer.java:150)
org.antlr.tool.Grammar.FIRST(Grammar.java:2924)
org.antlr.codegen.CodeGenerator.generateLocalFOLLOW(CodeGenerator.java: 
572)
org.antlr.codegen.CodeGenTreeWalker.atom(CodeGenTreeWalker.java:2075)
org.antlr.codegen.CodeGenTreeWalker.element(CodeGenTreeWalker.java:1684)
org.antlr.codegen.CodeGenTreeWalker.alternative(CodeGenTreeWalker.java: 
1288)
org.antlr.codegen.CodeGenTreeWalker.block(CodeGenTreeWalker.java:1063)
org.antlr.codegen.CodeGenTreeWalker.rule(CodeGenTreeWalker.java:779)
org.antlr.codegen.CodeGenTreeWalker.rules(CodeGenTreeWalker.java:570)
(Continue reading)

Permalink | Reply | 0 comments |
headers
Terence Parr | 2 Aug 2008 20:39
Gravatar

ANTLR 3.1 imminent

Hi Guys,

taking break from book to get 3.1 released into the wild...I will fix  
any really important bugs (hopefully not requiring changes on your  
part) and then push this week.

Ter
Permalink | Reply | 0 comments |
headers
Benjamin Niemann | 2 Aug 2008 21:45
Picon
Favicon

Bug in TokenRewriteStream.java

Hi Ter,

I'm just about to port the new TokenRewriteStream to Python. Something
smells fishy in the reduceToSingleOperationPerIndex method...
In lines 456, 464 and 493 you write
  rewrites.set(j, null);  // delete insert as it's a no-op.

j is the index into the list returned by getKindOfOps, but you use it
as the index into the rewrites list. In all testcases this seems to be
doing what you intend, but I can't believe this is correct.
E.g. if I modify the test2ReplaceMiddleIndex test to have a
tokens.insertBefore(0, "_") before the replaces, the numbers get
screwed up and the exception in line 517 is triggered.

I guess what you want is that j is the index within rewrites of those
ops filtered out by getKindOfOps.
My implementation of getKindOfOps now looks like

        for i, op in enumerate(rewrites[:before]):
            if op is None:
                # ignore deleted
                continue
            if op.__class__ == kind:
                yield i, op

i.e. for each matching op it returns a tuple (index_in_rewrites, op). Then I do

    for j, iop in self.getKindOfOps(rewrites, InsertBeforeOp, i):
        ...
(Continue reading)

Permalink | Reply | 1 comment |
headers
Terence Parr | 3 Aug 2008 02:03
Gravatar

Re: Bug in TokenRewriteStream.java

Wow!  You are SOooooo correct.  Fixed. Had to add another  
field...minor tweak, folks.  Excellent catch Benjamin...wow, this  
rewrite engine is a bitch!  added new test.

edit //depot/code/antlr/main/CHANGES.txt#80
edit //depot/code/antlr/main/runtime/Java/src/org/antlr/runtime/ 
TokenRewriteStream.java#11
edit //depot/code/antlr/main/src/org/antlr/test/ 
TestTokenRewriteStream.java#16
Change 5132 submitted.

Ter
On Aug 2, 2008, at 12:45 PM, Benjamin Niemann wrote:

> Hi Ter,
>
> I'm just about to port the new TokenRewriteStream to Python. Something
> smells fishy in the reduceToSingleOperationPerIndex method...
> In lines 456, 464 and 493 you write
>  rewrites.set(j, null);  // delete insert as it's a no-op.
>
> j is the index into the list returned by getKindOfOps, but you use it
> as the index into the rewrites list. In all testcases this seems to be
> doing what you intend, but I can't believe this is correct.
> E.g. if I modify the test2ReplaceMiddleIndex test to have a
> tokens.insertBefore(0, "_") before the replaces, the numbers get
> screwed up and the exception in line 517 is triggered.
>
> I guess what you want is that j is the index within rewrites of those
> ops filtered out by getKindOfOps.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Kunle Odutola | 3 Aug 2008 04:23
Picon
Favicon

Re: C# runtime license (AST output for Ruby)

Sorry about the late reply.

[SNIP]

> But the 4-clause license in the C# target doesn't include this text. In 
> effect, it is identical with the one used by Ter while adding this:
>
> Unless explicitly state otherwise, any contribution intentionally
> submitted for inclusion in this work to the copyright owner or licensor
> shall be under the terms and conditions of this license, without any
> additional terms or conditions.
>
> Barring the typo, this effectively says the same as the contributor 
> license does. It is actually redundant and can be removed.

It serves a similar purpose.

> It should be removed because the contribution license points out to a 
> particular BSD license - via a hypertext link to the website where the 3 
> clause license is detailed (Ter told me that this text never changed 
> beyound adjusting the years). And even if it isn't mentioned in the 
> license itself - one can't go around and choose a license himself.

Actually, it doesn't. The paper copy I received just said BSD licence (hence 
my initial comments about including the full licence text or URL in the 
agreement).

> IANAL but that Ter uses a particular one already makes the intent quite 
> clear which one he actually means. BTW, you didn't give each and every 
> file that new license header, so effectively those files are inherently
(Continue reading)

Permalink | Reply | 1 comment |
headers
Terence Parr | 3 Aug 2008 22:45
Gravatar

AST rewrites for tree parser fixes

edit //depot/code/antlr/main/CHANGES.txt#81
edit //depot/code/antlr/main/src/org/antlr/codegen/templates/Java/ 
ASTTreeParser.stg#10
edit //depot/code/antlr/main/src/org/antlr/codegen/templates/Java/ 
Java.stg#134
edit //depot/code/antlr/main/src/org/antlr/test/ 
TestTreeGrammarRewriteAST.java#10
Change 5140 submitted.

Fixes

http://www.antlr.org:8888/browse/ANTLR-294

August 3, 2008

* Was using RuleReturnScope not Rulename_return for list labels in tree
  parser.

* Didn't set _last in tree parser for rule ref track stuff  
(rewrite=true)

Ter
Permalink | Reply | 0 comments |
headers
Johannes Luber | 3 Aug 2008 22:50
Picon
Picon

Re: C# runtime license (AST output for Ruby)

Kunle Odutola schrieb:
> Sorry about the late reply.
> 
> [SNIP]
> 
>> But the 4-clause license in the C# target doesn't include this text. In 
>> effect, it is identical with the one used by Ter while adding this:
>>
>> Unless explicitly state otherwise, any contribution intentionally
>> submitted for inclusion in this work to the copyright owner or licensor
>> shall be under the terms and conditions of this license, without any
>> additional terms or conditions.
>>
>> Barring the typo, this effectively says the same as the contributor 
>> license does. It is actually redundant and can be removed.
> 
> It serves a similar purpose.

Why have it then there, if there is no real difference between the 4th 
clause and the agreement?
> 
>> It should be removed because the contribution license points out to a 
>> particular BSD license - via a hypertext link to the website where the 3 
>> clause license is detailed (Ter told me that this text never changed 
>> beyound adjusting the years). And even if it isn't mentioned in the 
>> license itself - one can't go around and choose a license himself.
> 
> Actually, it doesn't. The paper copy I received just said BSD licence (hence 
> my initial comments about including the full licence text or URL in the 
> agreement).
(Continue reading)

Permalink | Reply | 0 comments |
headers
Terence Parr | 4 Aug 2008 19:49
Gravatar

Altering static FOLLOW bitsets! fixed

Hi Guys, yes looks like a problem. I do a remove in 1 place that's  
incorrect:

	public boolean mismatchIsMissingToken(IntStream input, BitSet follow) {

		if ( follow.member(Token.EOR_TOKEN_TYPE) ) {
			if ( state._fsp>=0 ) { // remove EOR if we're not the start symbol
 >>>>>			follow.remove(Token.EOR_TOKEN_TYPE);
			}
			BitSet viableTokensFollowingThisRule =  
computeContextSensitiveRuleFOLLOW();
			follow = follow.or(viableTokensFollowingThisRule);
		}

Easiest thing is to just move the IF below the  
computeContextSensitiveRuleFOLLOW which calls combineFollows which does:

		BitSet followSet = new BitSet();

so it's a set we can modify.

So now it looks like:

		if ( follow.member(Token.EOR_TOKEN_TYPE) ) {
			BitSet viableTokensFollowingThisRule =  
computeContextSensitiveRuleFOLLOW();
			follow = follow.or(viableTokensFollowingThisRule);
             if ( state._fsp>=0 ) { // remove EOR if we're not the  
start symbol
                 follow.remove(Token.EOR_TOKEN_TYPE);
(Continue reading)

Permalink | Reply | 0 comments |
headers
Terence Parr | 4 Aug 2008 19:52
Gravatar

Re: Bug in DFAxx_SpecialStateTransition code

Added parens in Java.stg:

evalPredicate(pred,description) ::= "(<pred>)"

shows up as

                 if ( ((synpred1_T()&&( condition1 || condition2 ))) ) {

now for

parser grammar T;
rule1 : (rule2) => rule2 | A;
rule2 : { condition1 || condition2 }? => A ;

edit //depot/code/antlr/main/CHANGES.txt#83
edit //depot/code/antlr/main/src/org/antlr/codegen/templates/Java/ 
Java.stg#135
Change 5157 submitted.

Ter

On Jul 25, 2008, at 8:54 AM, Johannes Luber wrote:

> Sam Harwell schrieb:
>> The generated grammars need to parentheses around the expressions  
>> in a gated semantic predicate when used in the  
>> DFAxx_SpecialStateTransition functions. Here is an example failure  
>> case:
>> rule1
>> : (rule2) => rule2
(Continue reading)

Permalink | Reply | 0 comments |
headers
Terence Parr | 4 Aug 2008 20:11
Gravatar

extra init code removed for AST

Bug in rewriteCode as you say.  Made the !rewriteMode clause in an  
else of !TREE_PARSER:

Seems correct

<if(!TREE_PARSER)>
<prevRuleRootRef()>.tree = root_0;
<else>
<if(!rewriteMode)>
<prevRuleRootRef()>.tree = root_0;
<endif>
<endif>

edit //depot/code/antlr/main/CHANGES.txt#84
edit //depot/code/antlr/main/src/org/antlr/codegen/templates/Java/ 
AST.stg#50
edit //depot/code/antlr/main/src/org/antlr/test/ 
TestJavaCodeGeneration.java#8
Change 5159 submitted.

The TestJavaCO...one is unrelated...sorry.
Ter
Permalink | Reply | 0 comments |

Gmane