temp | 1 May 01:28
Favicon

Re: 2x: how to access token attributes

>> How token attributes can be accessed from lexer code like this:
>>
>> template <typename Lexer>
>> struct position_helper_tokens : lexer<Lexer>
>> {
>>   position_helper_tokens()
>>   {
>>     // define tokens and associate them with the lexer
>>     eol = "\n";
>>     any = "[^\n]+";
>>
>>     // associate tokens with the lexer
>>     this->self
>>       =   eol [HOW TO ACCESS ATTRIBUTES OF TOKENS GENERATED BY THIS
>> RULE???]
>>       |   any [AND THIS???]
>>     ;
>>   }
>>
>>   token_def<> any, eol;
>> };
>>
>> where token type looks like:
>>
>> typedef lexertl::token<base_iterator, boost::mpl::vector<position>,
>> boost::mpl::false_> token_type; //'position' is some struct type
>
>
> The short answer is: you can't.
>>From inside a lexer semantic action you can only access the
(Continue reading)

Hartmut Kaiser | 1 May 03:20
Picon
Gravatar

Re: 2x: how to access token attributes

> Thanks, I see now. Things I'm trying to do are:
> 
> 1. tokenize some input
> 2. than filter (remove some) and transform (update token attributes)
> tokens created in step #1
> 3. than use the tokens produced in step #2 in a grammar
> 
> How this can be achieved using the approach you suggested? The grammar
> 'some_grammar' from your suggestion maps to the step #2. It will use
> tokens produced in step #1 and consume them. My understanding is that
> it
> should produce a new token list that can be used in step #3. Is this
> the
> way to go?

I'd suggest to use a different approach. What if you write your own token
class (probably deriving from lex::lexertl::token)? The required interface
is documented in the comments in the file
boost/spirit/home/lex/lexertl/token.hpp.

This allows you to do whatever you think needs to be done (i.e. the
transformation). 

The filtering can be done using the skip grammar. If you define an empty
token_set and associate this with a separate lexer state, then this lexer
state could be used to identify the tokens to be filtered out. 

    token_set empty;
    this->self("TOSKIP") = empty;

(Continue reading)

temp | 1 May 03:34
Favicon

Re: 2x: token attributes from functor

I need to access token attribute from a functor used in the tokenize call
(similar to one from word_count_functor.c lex example). How to do this?
Let's say my tokens have attribute of type int. How to read/write it?
(I looked into token_def.hpp but it didn't help much. Class there does not
provide access to the attributes. Looks that such access is buried
somewhere deeper in the parent types)

struct token_filter
{
    typedef bool result_type;

    template <typename Token>
    bool operator()(Token const& t) const
    {
        switch( t.id() )
        {
        case ID_EOL:        // matched a newline character
            HOW TO ACCESS ATTRIBUTE OF 't' HERE? LETS SAY SET IT TO 5.
            break;
        case ID_ANY:       // matched something else
            break;
        }
        return true;        // always continue to tokenize
    }
};

...
// token type
typedef lexertl::token<base_iterator, boost::mpl::vector<int>,
boost::mpl::false_> token_type;
(Continue reading)

Hartmut Kaiser | 1 May 04:51
Picon
Gravatar

Re: 2x: token attributes from functor

> I need to access token attribute from a functor used in the tokenize
> call
> (similar to one from word_count_functor.c lex example). How to do this?
> Let's say my tokens have attribute of type int. How to read/write it?
> (I looked into token_def.hpp but it didn't help much. Class there does
> not
> provide access to the attributes. Looks that such access is buried
> somewhere deeper in the parent types)
> 
> struct token_filter
> {
>     typedef bool result_type;
> 
>     template <typename Token>
>     bool operator()(Token const& t) const
>     {
>         switch( t.id() )
>         {
>         case ID_EOL:        // matched a newline character
>             HOW TO ACCESS ATTRIBUTE OF 't' HERE? LETS SAY SET IT TO 5.
>             break;
>         case ID_ANY:       // matched something else
>             break;
>         }
>         return true;        // always continue to tokenize
>     }
> };
> 
> 
> ...
(Continue reading)

temp | 1 May 05:35
Favicon

Re: 2x: token attributes from functor

>> I need to access token attribute from a functor used in the tokenize
>> call
>> (similar to one from word_count_functor.c lex example). How to do this?
>> Let's say my tokens have attribute of type int. How to read/write it?
>> (I looked into token_def.hpp but it didn't help much. Class there does
>> not
>> provide access to the attributes. Looks that such access is buried
>> somewhere deeper in the parent types)
>>
>> struct token_filter
>> {
>>     typedef bool result_type;
>>
>>     template <typename Token>
>>     bool operator()(Token const& t) const
>>     {
>>         switch( t.id() )
>>         {
>>         case ID_EOL:        // matched a newline character
>>             HOW TO ACCESS ATTRIBUTE OF 't' HERE? LETS SAY SET IT TO 5.
>>             break;
>>         case ID_ANY:       // matched something else
>>             break;
>>         }
>>         return true;        // always continue to tokenize
>>     }
>> };
>>
>>
>> ...
(Continue reading)

Hartmut Kaiser | 1 May 13:35
Picon
Gravatar

Re: 2x: token attributes from functor

> Sorry but this does not compile:
> 
> struct token_filter
> {
>     typedef bool result_type;
> 
>     template <typename Token>
>     bool operator()(Token & t) const
>     {
>         switch( t.id() )
>         {
>         case ID_EOL: {
>             int val = 5;
>        	    construct(val, t);
>             } break;
>         case ID_ANY:       // matched something else
>             break;
>         }
>         return true;        // always continue to tokenize
>     }
> };
> 
> I get the error "../../../../boost/spirit/home/qi/parse.hpp:58: error:
> 'struct boost::spirit::tag::int_' has no member named 'parse'"

Lex uses the Qi parsers to do the conversion. So you need to include the
appropriate header files:

#include <boost/spirit/home/qi_numeric.hpp>

(Continue reading)

Unknown | 1 May 15:55

longest_d in spirit2?

Hi,

I'm trying to parse something like this: int_ | double_.
If I put the int first it will consume the first part of the double
followed by a parse error.
If I put the double first it will also match ints.
Is there something like longest_d (from spirit1) in spirit2 to solve my
problem? Or can you think of any other solutions?

Cheers,
Philipp

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations 
Conference from O'Reilly Media. Velocity features a full day of 
expert-led, hands-on workshops and two days of sessions from industry 
leaders in dedicated Performance & Operations tracks. Use code vel09scf 
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Hartmut Kaiser | 1 May 16:22
Picon
Gravatar

Re: longest_d in spirit2?

> I'm trying to parse something like this: int_ | double_.
> If I put the int first it will consume the first part of the double
> followed by a parse error.
> If I put the double first it will also match ints.
> Is there something like longest_d (from spirit1) in spirit2 to solve my
> problem? Or can you think of any other solutions?

You need to define:

    qi::real_parser<double, qi::strict_real_policies<double> >
strict_double;

and then you can use it as:

    r = strict_double | int_;

strict_doule will consume floating point numbers only, falling back to int_
if that fails.

HTH
Regards Hartmut

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations 
Conference from O'Reilly Media. Velocity features a full day of 
expert-led, hands-on workshops and two days of sessions from industry 
leaders in dedicated Performance & Operations tracks. Use code vel09scf 
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Joel de Guzman | 2 May 00:50
Picon
Gravatar

Re: longest_d in spirit2?

Hartmut Kaiser wrote:
>> I'm trying to parse something like this: int_ | double_.
>> If I put the int first it will consume the first part of the double
>> followed by a parse error.
>> If I put the double first it will also match ints.
>> Is there something like longest_d (from spirit1) in spirit2 to solve my
>> problem? Or can you think of any other solutions?
> 
> You need to define:
> 
>     qi::real_parser<double, qi::strict_real_policies<double> >
> strict_double;
> 
> and then you can use it as:
> 
>     r = strict_double | int_;
> 
> strict_doule will consume floating point numbers only, falling back to int_
> if that fails.

longest will no longer be supported. Its use can be avoided with
better usage constructs such as that provided by Hartmut. It is
slow and promotes ambiguous grammars, which in general can be
avoided.

Regards,
--

-- 
Joel de Guzman
http://www.boostpro.com
http://spirit.sf.net
(Continue reading)

Picon
Favicon

Confusion about diverting BOOST_DEBUG_OUT information to file

Question: How to redirect BOOST_SPIRIT_DEBUG messages to a set file (e.g. foo.log)
 
In 2004 the following code was suggested to redirect the debugging output from the std::cout to a file:
 
>struct MyDebugOut {
>     bool Output;
>     ofstream* where_to;
>     MyDebug() : Output(true) {};
>};
>
>template MyDebugOut& operator<<(MyDebugOut& _mdo,
>const T& _data)
>{
>     if(Output)
>         where_to << _data;
>
>     return *this;
>};
>
>MyDebugOut mdo;
>#DEFINE BOOST_SPIRIT_DEBUG_OUT mdo
 
I do not believe this example would work today. I get messages from Visual Studio that there is a missing type specifier for the generic template function. Here is a sample grammar I figured we could work from for this discussion. What I would like to do is discover the proper technique for redirecting output to a file and add this as example code to the Boost Spirit debug page documentation.
 
Stephen
 
----------- CODE -----------------
 

#include

<string>

#define

BOOST_SPIRIT_DEBUG

 

#include

<boost/spirit/core.hpp>

 

 

namespace

mystuff

{

class grammar : public boost::spirit::grammar< grammar >

{

public:

 

// ---[ THE ACTUAL GRAMMAR DEFINITION ]-------------------------------------------------------------------

template <typename SCANNER>

class definition

{

// Grammar rules

boost::spirit::rule< SCANNER > m_equation;

boost::spirit::rule< SCANNER > m_number;

 

public:

boost::spirit::rule< SCANNER >

const & start() const { return m_equation; }

 

// - -[ create the definition ] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

definition(grammar

const & self)

{

using namespace boost::spirit;

 

// 1: an object is an empty set or set of members...

m_equation

= m_number >> ch_p(

'+') >> m_number;

 

m_number

= strict_real_p

| int_p

;

 

BOOST_SPIRIT_DEBUG_RULE(m_equation);

BOOST_SPIRIT_DEBUG_RULE(m_number);

 

}

};

};

 

// ==========================================================================================================

// === T H E F I N A L P A R S I N G R O U T I N E ===

// ==========================================================================================================

}

int

main (int, char**)

{

// 1. parse the input

mystuff::grammar gr;

std::string input =

"3+3";

if ( boost::spirit::parse(input.begin(), input.end(), gr, boost::spirit::space_p).full)

{

std::cout <<

"parsing succeeded" << std::endl;

}

else

{

std::cout <<

"parsing failed" << std::endl;

}

}

Attachment (smime.p7s): application/x-pkcs7-signature, 2886 bytes
------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
Spirit-general mailing list
Spirit-general <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/spirit-general

Gmane