headers
protobuf | 1 Mar 2012 01:00

Re: Issue 256 in protobuf: Error Converting Message to array of Bytes (When negative Enum) - Java


Comment #8 on issue 256 by jhan... <at> gmail.com: Error Converting Message to  
array of Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256

Any update on this? I'm running into this exception running version 2.4.1

Is there any way from the public API to set the underlying buffer size of  
the CodedOutputStream? We are not creating that directly. We are calling  
writeDelimitedTo()

Any help is appreciated.

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
Jeremiah Jordan | 1 Mar 2012 03:14
Picon
Gravatar

Re: Segfault in python using CPP implementation

The issue is here:

       uir = UIR() 
        uir.ParseFromString(base64.decodestring(b64_user_blob)) 
        self.pb = uir.up 

The python cpp protobuf implementation doesn't suport keeping references to internal attributes when the base attribute leaves scope.  So what happens is that when uir eventually gets garbage collected accessing self.pb.s1 will crash.

You need to keep the top level uir around if you want to use the cpp implementation.

On Tuesday, February 21, 2012 4:45:25 PM UTC-6, shamer wrote:
I have found a segfault when using the CPP implementation from python
in 2.4.1. I can reproduce it in two different environments with a
small number of files.

The segfault is happening in google/protobuf/internal/cpp_message.py
in the ScalarProperty getter. There seems to be some interplay between
iterating through one repeated field and accessing a scalar property
in another message.

I have reduced the reproduction to a small set of .proto and py files.
I have bundled the files and uploaded the whole set here: (http://
dl.dropbox.com/u/24148866/py_cpp_pbcrash.tgz).  To reproduce the
segfault run the crashtest.py script with the cpp protocol buffer
implementation:
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp PYTHONPATH=./ python
crashtest.py

Simply accessing the self.pb.s1 property multiple times isn't enough
to cause a segfault, nor is iterating over repeated astr field.

I have made a change to the python_generator.cc file to include the
package name in the python imports. I have included the file in the
tarball and the diff is below. This modification was made to the 2.4.1
codebase. I believe that this should only change where python packages
are imported from.

I have tested this on two separate systems. Their respective
configurations are below. They both experience the segfault after a
different number of iterations through the "for a in uar.str:" loop.
Adding imports like "from datetime import date" to the top of the
script also changes the number of iterations through the loop before
segfaulting.

Any thoughts on what might be causing this or things I can do to help
narrow down the root cause?

Cheers,
Stephen


crashtest.py:
from api_pb.ui_pb2 import UIR
from api_pb.ua_pb2 import UAR
from datetime import date

import base64

activities_blob = ... large b64 blob ...

class Container(object):
    def __init__(self):
        b64_user_blob =
'''EnwKDFN0ZXBoZW5IYW1lchIHU3RlcGhlbhoFSGFtZXIyTmh0dHA6Ly9hMS50d2ltZy5jb20vc3RpY2t5L2RlZmF1bHRfcHJvZmlsZV9pbWFnZXMvZGVmYXVsdF9wcm9maWxlXzBfbm9ybWFsLnBuZzi9gafrBEIASgBSAGgB'''
        uir = UIR()
        uir.ParseFromString(base64.decodestring(b64_user_blob))
        self.pb = uir.up

    def iterate_on_astr(self):
        uar = UAR()
        uar.ParseFromString(base64.decodestring(activities_blob))
        for a in uar.astr:
            self.pb.s1

container = Container()
container.iterate_on_astr()



Tested environments:
$ uname -a
Linux isengard 3.1-pf #1 SMP PREEMPT Mon Jan 9 02:15:02 EST 2012
x86_64 Intel(R) Core(TM) i7 CPU 950 <at> 3.07GHz GenuineIntel GNU/Linux
$ python2 --version
Python 2.7.2

And

shamer <at> prod5:~$ uname -a
Linux prod5.upverter.com 2.6.35-24-virtual #42-Ubuntu SMP Thu Dec 2
05:15:26 UTC 2010 x86_64 GNU/Linux
shamer <at> prod5:~$ python --version
Python 2.6.6


Changes to python_generator.cc
79,80c79,80
< string ModuleName(const string& filename) {
<   string basename = StripProto(filename);
---
> string ModuleName(const FileDescriptor *file) {
>   string basename = StripProto(file->name());
83c83,89
<   return basename + "_pb2";
---
>
>   string package = file->package();
>   if (package.length() > 0) {
>     return package + "." + basename + "_pb2";
>   } else {
>     return basename + "_pb2";
>   }
245c251
<   string module_name = ModuleName(file->name());
---
>   string module_name = ModuleName(file);
286c292
<     string module_name = ModuleName(file_->dependency(i)->name());
---
>     string module_name = ModuleName(file_->dependency(i));
950c956
<     name = ModuleName(descriptor.file()->name()) + "." + name;
---
>     name = ModuleName(descriptor.file()) + "." + name;
962c968
<     name = ModuleName(descriptor.file()->name()) + "." + name;
---
>     name = ModuleName(descriptor.file()) + "." + name;
975c981
<     name = ModuleName(descriptor.file()->name()) + "." + name;
---
>     name = ModuleName(descriptor.file()) + "." + name;

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/sApBblFDezsJ.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
Jeremiah Jordan | 1 Mar 2012 03:21
Picon
Gravatar

Re: protobuf TextFormat question: Parsing TextFormat into message [fixed]

Are you actually reading these values from a file?  If not, why don't you just:

my_msg = my_envelope_pb2.MyEnvelope() 
my_msg.InnerMessage.value1 = 100
my_msg.InnerMessage.value2 = 200

You are correct that ParseFromString just reads back the output of SerializeToString


On Monday, February 6, 2012 6:14:34 PM UTC-6, Kevin wrote:
...previous message didn't have example code sanitized and probably
doesn't make sense. Please delete/ignore previous and use this one. :)

Hard to find much high-level documentation on using TextFormat, but
for some Python protobuf work I'm doing, the text format is much
easier to read (and easier to organize into test vector files) in the
protobuf text format, rather than manipulating the python objects
manually.

(I've changed message types/contents to generalize, please forgive any
typos, this isn't the real code)

my_envelope.proto (this file gets sent through protoc to generate
Python code):
message MyEnvelope {
  optional InnerMessage innerMessage = 1;
}

and I have a test message that I'd like to take from TextFormat and
put into a protobuf message structure:
innerMessage {
  value1: 100
  value2: 200
}

I have found that I can get it to work in python using

from google.protobuf import text_format
import my_envelope_pb2
my_msg = my_envelope_pb2.MyEnvelope()
text_format.Merge("""
    innerMessage {
        value1: 100
        value2: 200
    }
""", my_msg)

But I guess my question is this- is there a preferred way to do this?
I had originally expected that ParseFromString would figure out that
this was an ASCII representation and call Merge appropriately.
i.e. I had expected the following would work
import my_envelope_pb2
my_msg = my_envelope_pb2.MyEnvelope()
my_msg.ParseFromString("""
    inner {
        value1: 100
        value2: 200
    }
""")
but from the errors it appeared that ParseFromString only deals with
string containers for binary format. Is this the case? Just trying to
make sure I'm not calling Merge at an innapropriate/strange layer when
some high-level call exists. As I mentioned, documentation on text
format is somewhat thin, so this is the best I could piece together
from API info. Thanks!

On Monday, February 6, 2012 6:14:34 PM UTC-6, Kevin wrote:
...previous message didn't have example code sanitized and probably
doesn't make sense. Please delete/ignore previous and use this one. :)

Hard to find much high-level documentation on using TextFormat, but
for some Python protobuf work I'm doing, the text format is much
easier to read (and easier to organize into test vector files) in the
protobuf text format, rather than manipulating the python objects
manually.

(I've changed message types/contents to generalize, please forgive any
typos, this isn't the real code)

my_envelope.proto (this file gets sent through protoc to generate
Python code):
message MyEnvelope {
  optional InnerMessage innerMessage = 1;
}

and I have a test message that I'd like to take from TextFormat and
put into a protobuf message structure:
innerMessage {
  value1: 100
  value2: 200
}

I have found that I can get it to work in python using

from google.protobuf import text_format
import my_envelope_pb2
my_msg = my_envelope_pb2.MyEnvelope()
text_format.Merge("""
    innerMessage {
        value1: 100
        value2: 200
    }
""", my_msg)

But I guess my question is this- is there a preferred way to do this?
I had originally expected that ParseFromString would figure out that
this was an ASCII representation and call Merge appropriately.
i.e. I had expected the following would work
import my_envelope_pb2
my_msg = my_envelope_pb2.MyEnvelope()
my_msg.ParseFromString("""
    inner {
        value1: 100
        value2: 200
    }
""")
but from the errors it appeared that ParseFromString only deals with
string containers for binary format. Is this the case? Just trying to
make sure I'm not calling Merge at an innapropriate/strange layer when
some high-level call exists. As I mentioned, documentation on text
format is somewhat thin, so this is the best I could piece together
from API info. Thanks!

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/h1xKMrdMlsAJ.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
Robert Abadi | 2 Mar 2012 16:20
Picon

Re: sending a message over TCP from a C++ client to a Java server

Paul <mjpablo23 <at> gmail.com> writes:

> 
> ok thanks a lot!  I ended up finding that my TCP send function was not
> set up correctly.  Once that was fixed, it worked fine.  Thanks!
> 
> On Oct 13, 2:20 pm, Evan Jones <ev... <at> MIT.EDU> wrote:
> > On Oct 13, 2010, at 16:49 , Paul wrote:
> >
> > > Thanks for the suggestion.  However, I am already prepending the
> > > message size on the C++ side in the line:
> > > coded_output->WriteVarint64(snap1.ByteSize());
> >
> > You may want to verify that the exact bytes that come out of  
> > msg.SerializeToString (or related) are coming out the other end and  
> > getting passed into parseDelimited. It might be helpful if you sent a  
> > snippet of code where you are sending and receiving the messages, but  
> > I can't think of anything off the top of my head.
> >
> > Evan
> >
> > --
> > Evan Joneshttp://evanjones.ca/
> 

I am new in java networking programming and also in protocol buffers.
If the client side is also run in Java.
How to send the message?
thank you for your help.

Robert Abadi

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
protobuf | 2 Mar 2012 19:30

Re: Issue 351 in protobuf: Make protobuf_lite proto files not create any static initializers

Updates:
	Owner: pli... <at> google.com

Comment #8 on issue 351 by liuj... <at> google.com: Make protobuf_lite proto  
files not create any static initializers
http://code.google.com/p/protobuf/issues/detail?id=351

(No comment was entered for this change.)

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
Julien PONNOU | 1 Mar 2012 15:41
Picon

Package Mandatory ?

Hello,

When I include several couples of .h/.cc in my project, I got the
following errors during the compilation:

error C2086: 'google::protobuf::ProtobufOnceType `anonymous-
namespace'::protobuf_AssignDescriptors_once_' : redefinition
see declaration of '`anonymous-
namespace'::protobuf_AssignDescriptors_once_'
error C2084: function 'void `anonymous-
namespace'::protobuf_AssignDescriptorsOnce(void)' already has a body
see previous definition of 'protobuf_AssignDescriptorsOnce'
error C2084: function 'void `anonymous-
namespace'::protobuf_RegisterTypes(const std::string &)' already has a
body
see previous definition of 'protobuf_RegisterTypes'

Basically, it's telling me these functions are declared in all couples
of .h/.cc. Using a different package for each couple of .h/.cc
corrects the problem, but I don't want to do so. Without using any
package, can someone explain me how to solve this error ?

Best regards,

Julien.

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 1 comment |
headers
Jan Bezget | 1 Mar 2012 21:56
Picon

Re: I am new in Protocol Buffers

When I first started using protobuf in my application, I relied solely on http://code.google.com/apis/protocolbuffers/docs/overview.html. It's a great resource of documentation and provides helpful examples.


There's a cool plugin for Eclipse! You can get it at http://code.google.com/p/protobuf-dt/wiki/Installing.

Protoc is the protocol buffer compiler. Once you've defined your protocol in the .proto file, you must compile it to a language of your choice (Python, C++, Java) so you can begin using it in your project.

On Monday, February 13, 2012 2:38:43 PM UTC+1, Lyrya wrote:
Hi All,

I don't know how to add protocol buffers library for eclipse.
and what is the use of protoc?
Is there any tutorial that can help me understanding about protocol
buffers?

Thank you.....

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/6w52mYXzW3cJ.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 2 comments |
headers
Julien PONNOU | 1 Mar 2012 15:48
Picon

Re: protobuf_AssignDescriptors_once_ redefined?

Hello,

I'm getting the exact same issue and I'd like to know how did you resolve it ?

Julien.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/jbVLBPO0J-0J.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
protobuf | 6 Mar 2012 23:34

Re: Issue 256 in protobuf: Error Converting Message to array of Bytes (When negative Enum) - Java

Updates:
	Labels: FixedIn-2.4.1

Comment #9 on issue 256 by jas... <at> google.com: Error Converting Message to  
array of Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256

2.4.1 should have contained this fix. Can you post a reproduction? If your  
serialization does not have a negative enum value, then please open a  
separate issue - the exception can occur generally when the ByteSize  
computation does not match the size of data written by the serializer.

There is no message level interface, but you can just create the  
CodedOutputStream around your OutputStream
directly:
http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/CodedOutputStream.html#newInstance(java.io.OutputStream,

int)
However, I think you should only be getting the OutOfSpaceException if  
you're using the byte[] version of the methods, in which case the data is  
written directly to the array. You need to make sure that the byte array  
you provide is sufficiently large.

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 0 comments |
headers
Evan Jones | 7 Mar 2012 00:08
Picon
Picon
Favicon

Re: How to read continuous stream of messages from TCP

On Feb 27, 2012, at 17:27 , waynix wrote:
> 1. Is this still the way to do it? Seems quite cumbersome (to lazy me ;-).  Is there  a wrapper built in to do this?

Yes. Sadly there is no wrapper included in the library.

> 2. If I understand Jason's suggestion riht, the length is really not
> part of the message, and the sender has to explcitly set it, instead
> of having protobuf encode it in. Which means a generic third party
> sender using my .proto file would not be sufficient.  Plus how would
> they know the length before encoding the message proper? Filling it in
> after the fact would change the length again?     or I am totally
> missing it.

As long as both sides encode the length in the same way , just having the right .proto will do the trick. 

> 3. A related quesiton is in general do I have to manage reading of the
> socket, or for that matter any istream, and spoon feed the protobuf
> parser until it says OK, that's a whole message?

Basically yes. There is a sketch of some example code here:

https://groups.google.com/forum/?fromgroups#!searchin/protobuf/sequence/protobuf/pLwqN4jTVvY/60PBaEadW5IJ

Good luck,

Evan

--
http://evanjones.ca/

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to protobuf <at> googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Permalink | Reply | 2 comments |

Gmane