Arun Chandrasekaran | 28 Jan 14:27 2015
Picon

Unable to get protobuf 2.6.1 working with python2.7

Hi,

I am on Ubuntu 14.10 and I'm using protobuf 2.6.1, built from source.

When I try to run the python protobuf tests, I get the below error.

18:44:33 ~/Setup/protobuf-2.6.1/python
$ python setup.py google_test
running google_test
running egg_info
creating protobuf.egg-info
writing requirements to protobuf.egg-info/requires.txt
writing protobuf.egg-info/PKG-INFO
writing namespace_packages to protobuf.egg-info/namespace_packages.txt
writing top-level names to protobuf.egg-info/top_level.txt
writing dependency_links to protobuf.egg-info/dependency_links.txt
writing manifest file 'protobuf.egg-info/SOURCES.txt'
reading manifest file 'protobuf.egg-info/SOURCES.txt'
writing manifest file 'protobuf.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 200, in <module>
    "Protocol Buffers are Google's data interchange format.",
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/test.py", line 142, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/test.py", line 122, in with_project_on_sys_path
    func()
  File "/home/arun/Setup/protobuf-2.6.1/python/.eggs/google_apputils-0.4.1-py2.7.egg/google/apputils/setup_command.py", line 157, in run_tests
    ok &= self._RunTestModule(file_path)
  File "/home/arun/Setup/protobuf-2.6.1/python/.eggs/google_apputils-0.4.1-py2.7.egg/google/apputils/setup_command.py", line 123, in _RunTestModule
    module = imp.load_module(module_name, *import_tuple)
  File "google/protobuf/internal/symbol_database_test.py", line 36, in <module>
    from google.protobuf import unittest_pb2
ImportError: cannot import name unittest_pb2

Ignoring the above test failure, I did a sudo python setup.py install.

After this, I'm not able to import the generated proto files. Please see below.

msg.proto
message msg {
    required int64 value = 1;
}

$ protoc --version
libprotoc 2.6.1

$ PYTHONPATH=gen python                                                              
Python 2.7.8 (default, Oct 20 2014, 15:05:19) 
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import msg_pb2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arun/prototest/gen/msg_pb2.py", line 9, in <module>
    from google.protobuf import symbol_database as _symbol_database
ImportError: cannot import name symbol_database
>>> 

Any idea where I am going wrong?

Thanks,
Arun

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Kavali Naresh | 28 Jan 15:43 2015
Picon

Protocol Buffer - Static Initialization of array's

Hi All,
 
Our project is build using VC++(MFC) and few files have structures declared, few arrays of integers and char's and they are initialized statically. We have few static member variables initialized with data as below,
char* class::m_psz[] = { "COL", "FAC", "MAS", "NOL", "SET" }; 
TYPE_20 class::m_Defaults[] = { {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0} }; 
And have few typedefs as below,
typedef unsigned char TYPE_20[20];
 
My requirement is to use Protocol Buffer concept and achieve the above. So How can i write the *.proto file (or the schema) to declare the static array of integers and initialize values to it. And finally i need to generate the *.cs (cshar file) out of it.
 
Below are my Basic knowledge on PB:
1) I created the classes, used "optional", "repeated" concepts.
2) created the code to declare 2d array.
3) how do I initialize the array at declaration time as it is done in C++?
Below is code of PB schema for 2d array without initialization(at static time)
 
repeated DefaultsForTYPE_20 m_DefaultsForTYPE_20 = 1; message DefaultsForTYPE_20{ optional bytes m_DefaultsForTYPE_20 = 1; }but how can i initialize data to it here?
 
Please provide me your suggestions, it's kind of Urgent for me.
 
Thanks in Advance

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
uma rengasamy | 28 Jan 12:01 2015
Picon

ImportError: cannot import name symbol_database

Trying to use protobuf 2.6.1 in python 2.6.8 and getting the below error 

 py.test test_spoc.py
========================================================================= test session starts ==========================================================================
platform linux2 -- Python 2.6.8 -- py-1.4.23 -- pytest-2.6.1
collected 0 items / 1 errors

================================================================================ ERRORS ================================================================================
____________________________________________________________________ ERROR collecting test_spoc.py _____________________________________________________________________
test_spoc.py:2: in <module>
    import Spoc_pb2
Spoc_pb2.py:10: in <module>
    from google.protobuf import symbol_database as _symbol_database
E   ImportError: cannot import name symbol_database

how can we import this symbol_database in python 2.6.8

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
uma rengasamy | 28 Jan 12:04 2015
Picon

ImportError: cannot import name symbol_database

trying to use the compiled protobuf  (using 2.6.1) o/p in  python 2.6.8 and getting the below error 
py.test test_spoc.py
========================================================================= test session starts ==========================================================================
platform linux2 -- Python 2.6.8 -- py-1.4.23 -- pytest-2.6.1
collected 0 items / 1 errors

================================================================================ ERRORS ================================================================================
____________________________________________________________________ ERROR collecting test_spoc.py _____________________________________________________________________
test_spoc.py:2: in <module>
    import Spoc_pb2
Spoc_pb2.py:10: in <module>
    from google.protobuf import symbol_database as _symbol_database
E   ImportError: cannot import name symbol_database
======================================================================= 1 error in 0.16 seconds ========================================================================

how can it be included 

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Jon Emerson | 27 Jan 01:31 2015
Picon

How to construct a Builder from a FieldDescriptor with JavaType.MESSAGE

Hi There,

I'm writing a utility to convert data stored in MongoDB back to protocol buffer format.  My strategy is to look for each field in the protocol buffer definition, and if there's a matching field from my MongoDB object, then pull that data and set it on my protocol buffer builder.  This works great, except for embedded messages.  I can't figure out how to create a Builder for an embedded message.

Here's my first try:

          switch (fieldDescriptor.getJavaType()) {
            ...
            case MESSAGE:
              Class<T> type = (Class<T>) fieldDescriptor.getDefaultValue().getClass(); // getDefaultValue throws.
              messageBuilder.addRepeatedField(fieldDescriptor,
                  fromDBObject((BasicBSONObject) object.get(fieldName), type));
              break;

But the noted line throws with the error, "java.lang.UnsupportedOperationException: FieldDescriptor.getDefaultValue() called on an embedded message field."

Here's my second try:
          switch (fieldDescriptor.getJavaType()) {
            ...
            case MESSAGE:
              Message.Builder embeddedMessageBuilder = messageBuilder.getFieldBuilder(fieldDescriptor); // getFieldBuilder throws.
              Class<T> type = (Class<T>) embeddedMessageBuilder.getDefaultInstanceForType().getClass();
              messageBuilder.addRepeatedField(fieldDescriptor,
                  fromDBObject((BasicBSONObject) object.get(fieldName), type));
              break;

Here, the noted line throws with the error, "java.lang.UnsupportedOperationException: getFieldBuilder() called on a non-Message type."

I'm not sure what else to try.  My question boils down to a very simple one: Given a FieldDescriptor that describes an embedded repeated MESSAGE field, how can you create a Builder for it?

Thanks!
jon

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Jana Baran | 24 Jan 00:19 2015
Picon

getting crash from MessageLite::AppendPartialToString. Potential overflow problem all around the google protobuf code.

Potential overflow problem occurs all around the protobuf code.
For example see message_lite.cc

"
bool MessageLite::AppendPartialToString(string* output) const {
  int old_size = output->size();
  int byte_size = ByteSize();
  STLStringResizeUninitialized(output, old_size + byte_size);
  uint8* start = reinterpret_cast<uint8*>(string_as_array(output) + old_size);
  uint8* end = SerializeWithCachedSizesToArray(start);
  if (end - start != byte_size) {
    ByteSizeConsistencyError(byte_size, ByteSize(), end - start);
  }
  return true;
}
"

First two lines, size_t (unsigned 64b on a 64b architecture) is cast to int (signed 32b). 

Third line problem is 'old_size + byte_size' which can overflow. There should be a check before the summation like:

size_t new_size = 0;
if (std::numeric_limits<size_t>::max() - old_size < byte_size) {
 // we have a problem
} else {
 new_size = old_size + byte_size;
}

STLStringResizeUninitialized correctly takes size_t as an input but you need to cast before the summation and not after, like "(size_t)old_size + byte_size" 

Next two lines: 'start' and 'end' is of type uint8 which is unsigned 8bit integer. On next line this uint8 is compared with byte_size (1B variable compared with 4B variable) and ByteSizeConsistencyError is called when we are not happy with the comparison.

Now funny part is the comment above MessageLite::ByteSizeConsistencyError method which states:

"...If serialization produces a different number of bytes than expected, we call this function, which crashes. The problem could be due to a bug in the protobuf implementation..." 

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Paweł Zatorski | 23 Jan 13:45 2015
Picon

Handling parse errors

Hi,

In my project i am using protocol buffers in two modes: text and binary. Sometimes it happend that same data once serialized in text format is deserialized correctly and when using binary format parsing method returns false. I tried to created some parsing error handling code and i came to such solution, for text format:
        std::fstream input( path, std::ios::in | std::ios::binary );
        google::protobuf::io::IstreamInputStream inputt( & input );
google::protobuf::TextFormat::Parser parser;
        parser.RecordErrorsTo(&err); //err - custom class implementing google::protobuf::io::ErrorCollector
        result = parser.Parse(&inputt, &data);

My question is, is it possible to create similar code for binary format ( i mean error recording ). For now my parsing code in binary format looks like:
        std::fstream input( path, std::ios::in | std::ios::binary );
        result = data.ParseFromIstream( & input );

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
protobuf | 22 Jan 00:58 2015

Issue 686 in protobuf: Protoc does not provide a dependency manifest output file for build tools like make

Status: New
Owner: liujisi <at> google.com
Labels: Type-Defect Priority-Medium

New issue 686 by richardg... <at> gmail.com: Protoc does not provide a  
dependency manifest output file for build tools like make
https://code.google.com/p/protobuf/issues/detail?id=686

Protobuf files can have import statements and include directories, so the  
full dependency tree is not known by the build system before executing  
protoc. This makes it hard for the build system to know when to schedule  
any given protoc command.
Make & other build systems can use a dependency file to trigger a rebuild  
when a dependency changes. GCC and other compilers expose this with the -MF  
flag, which outputs a file in the format

output_filename: input_file \
  input_file2 \
  input_file3
...

Protobuf knows the transitive set of input files, but doesn't expose this  
information in this format.

Patch to follow.

-- 
You received this message because this project is configured to send all  
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

--

-- 
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Chris Johns | 21 Jan 07:04 2015
Picon

Building for RTEMS ARM

Hello,

I am building Protocol Buffers from github for RTEMS (http://www.rtems.org/) to run on a Xilinx Zynq (ARM CortexA9) and I am making some progress but I have some questions.

I am using the current development head for RTEMS which is called 4.11 and it is stable and almost about to be released. The gcc is:

$ arm-rtems4.11-gcc -v
Using built-in specs.
COLLECT_GCC
=/opt/work/rtems/4.11/bin/arm-rtems4.11-gcc
COLLECT_LTO_WRAPPER
=/opt/work/rtems/4.11/libexec/gcc/arm-rtems4.11/4.9.2/lto-wrapper
Target: arm-rtems4.11
Configured with: ../gcc-4.9.2/configure --prefix=/opt/work/rtems/4.11 --bindir=/opt/work/rtems/4.11/bin --exec_prefix=/opt/work/rtems/4.11 --includedir=/opt/work/rtems/4.11/include --libdir=/opt/work/rtems/4.11/lib --libexecdir=/opt/work/rtems/4.11/libexec --mandir=/opt/work/rtems/4.11/share/man --infodir=/opt/work/rtems/4.11/share/info --datadir=/opt/work/rtems/4.11/share --build=x86_64-freebsd10.0 --host=x86_64-freebsd10.0 --target=arm-rtems4.11 --disable-libstdcxx-pch --with-gnu-as --with-gnu-ld --verbose --with-newlib --with-system-zlib --disable-nls --without-included-gettext --disable-win32-registry --enable-version-specific-runtime-libs --disable-lto --enable-newlib-io-c99-formats --enable-newlib-iconv --enable-newlib-iconv-encodings=big5,cp775,cp850,cp852,cp855,cp866,euc_jp,euc_kr,euc_tw,iso_8859_1,iso_8859_10,iso_8859_11,iso_8859_13,iso_8859_14,iso_8859_15,iso_8859_2,iso_8859_3,iso_8859_4,iso_8859_5,iso_8859_6,iso_8859_7,iso_8859_8,iso_8859_9,iso_ir_111,koi8_r,koi8_ru,koi8_u,koi8_uni,ucs_2,ucs_2_internal,ucs_2be,ucs_2le,ucs_4,ucs_4_internal,ucs_4be,ucs_4le,us_ascii,utf_16,utf_16be,utf_16le,utf_8,win_1250,win_1251,win_1252,win_1253,win_1254,win_1255,win_1256,win_1257,win_1258 --enable-threads --disable-plugin --enable-obsolete --enable-languages=c,c++
Thread model: rtems
gcc version
4.9.2 20141030 (RTEMS 4.11, RSB e7cbf74fe2c94f0a233b8a7efcac3e75c239333c, Newlib de616601501c4f82968683e80c112604a2d40222) (GCC)


The first are errors related to Atromics. We are adding -fpermissive to make then warning. Here are a few of the many warnings we are seeing:

                 from ../../protobuf.git/src/google/protobuf/arenastring.h:40,
                 
from ../../protobuf.git/src/google/protobuf/descriptor.pb.h:23,
                 
from ../../protobuf.git/src/google/protobuf/compiler/ruby/ruby_generator.cc:36:
../../protobuf.git/src/google/protobuf/stubs/once.h: In function 'void google::protobuf::GoogleOnceInit(google::protobuf::ProtobufOnceType*, void (*)())':
../../protobuf.git/src/google/protobuf/stubs/once.h:125:34: warning: invalid conversion from 'google::protobuf::ProtobufOnceType* {aka int*}' to 'const volatile Atomic32* {aka const volatile long int*}' [-fpermissive]
 
if (internal::Acquire_Load(once) != ONCE_STATE_DONE) {
                                 
^

In file included from ../../protobuf.git/src/google/protobuf/stubs/atomicops.h:205:0,
                 
from ../../protobuf.git/src/google/protobuf/stubs/atomic_sequence_num.h:33,
                 
from ../../protobuf.git/src/google/protobuf/arena.h:38,
                 
from ../../protobuf.git/src/google/protobuf/descriptor.pb.h:22,
                 
from ../../protobuf.git/src/google/protobuf/compiler/ruby/ruby_generator.cc:36:
../../protobuf.git/src/google/protobuf/stubs/atomicops_internals_arm_gcc.h:136:17: note: initializing argument 1 of 'google::protobuf::internal::Atomic32 google::protobuf::internal::Acquire_Load(const volatile Atomic32*)'


The other issue is RTEMS is not self hosting and so always cross-compiled. Is there a way to disable running tests ? It makes no sense to run the tests when the build and host are not the same.

Thanks

Chris

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Ji Wan | 20 Jan 02:41 2015

Can protobuf work with C++ templates?

Suppose I have two message types `DoubleMatrix` and `FloatMatrix`, and a template class `Matrix`:


message DoubleMatrix {
  required uint32 rows
= 1;
  required uint32 cols
= 2;
  repeated
double data = 3 [packed=true];
}


message
FloatMatrix {
  required uint32 rows
= 1;
  required uint32 cols
= 2;
  repeated
float data = 3 [packed=true];
}


template<typename DType>
class Matrix {
   
MSTType mat_data_;
};



Is it possible to make `MSTType` as `FloatMatrix` if `DType` is `float`, as `DoubleMatrix` if `DType` is `double`?

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
王柏龍 | 20 Jan 08:41 2015
Picon

Protobuf message deserialize error

Hi, group

     In the function, I use a protobuf A serialized but there is a column stored : data type is string, the content of data is protobuf B serialized result.

The problem is after I deserialize  protobuf A message is fine, but I could not deserialize the column protobuf B after convert string to byte[]. The exception is Invalid wire-type.

so the question I want to ask is 

First, the serialize process is code C++ then store in ZooKeeper, C# application get the message to deserialize. In two code-base to transfer might be the reason for the problem? (such like string format)

Second, the protobuf B contain the jpg data. Is that a reason for the problem?

Thank you guys.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe <at> googlegroups.com.
To post to this group, send email to protobuf <at> googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Gmane