headers
SourceForge.net | 2 Apr 2009 05:41
Picon
Favicon

[ expat-Bugs-2723522 ] expat memory consumption issue - advise needed

Bugs item #2723522, was opened at 2009-03-31 18:50
Message generated for change (Comment added) made by alexmanovbg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=2723522&group_id=10127

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: Not a Bug
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Manov (alexmanovbg)
Assigned to: Nobody/Anonymous (nobody)
Summary: expat memory consumption issue - advise needed

Initial Comment:

We have a application which uses expat to convert a xml data file into a binary version of the file. The file at
the moment is about 600M but will grow.
We encountered a blocking problem - while parsing the file the application starts using a huge amount of
memory it needs 4G of RAM to finish successfuly a 600MB file.
Our engineers explained that this is due to block memory management in expat when it builds the xml tree.
They explained that our xml has alot of tags which in turn requires separate 4K memory pages for even 3 bytes
of actual data.

Is there any way to improve this? Could anyone suggest how we can optimize this process? Is there any
settings which we can use to make it work?
(Continue reading)

Permalink | Reply | 0 comments |
headers
SourceForge.net | 2 Apr 2009 05:44
Picon
Favicon

[ expat-Bugs-2723522 ] expat memory consumption issue - advise needed

Bugs item #2723522, was opened at 2009-03-31 18:50
Message generated for change (Comment added) made by alexmanovbg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=2723522&group_id=10127

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: Not a Bug
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Manov (alexmanovbg)
Assigned to: Nobody/Anonymous (nobody)
Summary: expat memory consumption issue - advise needed

Initial Comment:

We have a application which uses expat to convert a xml data file into a binary version of the file. The file at
the moment is about 600M but will grow.
We encountered a blocking problem - while parsing the file the application starts using a huge amount of
memory it needs 4G of RAM to finish successfuly a 600MB file.
Our engineers explained that this is due to block memory management in expat when it builds the xml tree.
They explained that our xml has alot of tags which in turn requires separate 4K memory pages for even 3 bytes
of actual data.

Is there any way to improve this? Could anyone suggest how we can optimize this process? Is there any
settings which we can use to make it work?
(Continue reading)

Permalink | Reply | 0 comments |
headers
SourceForge.net | 2 Apr 2009 05:51
Picon
Favicon

[ expat-Bugs-2723522 ] expat memory consumption issue - advise needed

Bugs item #2723522, was opened at 2009-03-31 18:50
Message generated for change (Comment added) made by alexmanovbg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=2723522&group_id=10127

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: Not a Bug
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Manov (alexmanovbg)
Assigned to: Nobody/Anonymous (nobody)
Summary: expat memory consumption issue - advise needed

Initial Comment:

We have a application which uses expat to convert a xml data file into a binary version of the file. The file at
the moment is about 600M but will grow.
We encountered a blocking problem - while parsing the file the application starts using a huge amount of
memory it needs 4G of RAM to finish successfuly a 600MB file.
Our engineers explained that this is due to block memory management in expat when it builds the xml tree.
They explained that our xml has alot of tags which in turn requires separate 4K memory pages for even 3 bytes
of actual data.

Is there any way to improve this? Could anyone suggest how we can optimize this process? Is there any
settings which we can use to make it work?
(Continue reading)

Permalink | Reply | 0 comments |
headers
SourceForge.net | 2 Apr 2009 14:07
Picon
Favicon

[ expat-Bugs-2723522 ] expat memory consumption issue - advise needed

Bugs item #2723522, was opened at 2009-03-31 11:50
Message generated for change (Comment added) made by kwaclaw
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=2723522&group_id=10127

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: Not a Bug
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Alex Manov (alexmanovbg)
Assigned to: Nobody/Anonymous (nobody)
Summary: expat memory consumption issue - advise needed

Initial Comment:

We have a application which uses expat to convert a xml data file into a binary version of the file. The file at
the moment is about 600M but will grow.
We encountered a blocking problem - while parsing the file the application starts using a huge amount of
memory it needs 4G of RAM to finish successfuly a 600MB file.
Our engineers explained that this is due to block memory management in expat when it builds the xml tree.
They explained that our xml has alot of tags which in turn requires separate 4K memory pages for even 3 bytes
of actual data.

Is there any way to improve this? Could anyone suggest how we can optimize this process? Is there any
settings which we can use to make it work?
(Continue reading)

Permalink | Reply | 1 comment |
headers
Jody Haynes | 3 Apr 2009 14:00
Picon

expat-2.0.1 will not compile on AIX 6.1 due to a problem with fcntl.h

Is there any workaround for this issue on AIX 6.1.

This same source compiles on AIX 5.3 with no issues.

 
[6100-02-03-0909]  /tmp/expat-2.0.1 -> ./configure
checking build system type... rs6000-ibm-aix
checking host system type... rs6000-ibm-aix
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking for a sed that does not truncate output... /usr/bin/sed
checking for egrep... grep -E
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... no
checking for /usr/bin/ld option to reload object files... -r
checking for BSD-compatible nm... /usr/bin/nm -B
checking whether ln -s works... yes
checking how to recognise dependent libraries... unknown
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... no
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking for g77... g77
checking whether we are using the GNU Fortran 77 compiler... yes
checking whether g77 accepts -g... yes
checking the maximum length of command line arguments... 262144
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for objdir... .libs
checking for ar... ar
checking for ranlib... ranlib
checking for strip... strip
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC...
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... no
checking dynamic linker characteristics... no
checking how to hardcode library paths into programs... unsupported
checking whether stripping libraries is possible... no
checking if libtool supports shared libraries... no
checking whether to build shared libraries... no
checking whether to build static libraries... yes
configure: creating libtool
appending configuration tag "CXX" to libtool
checking for ld used by g++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... no
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... no
checking for g++ option to produce PIC...
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... no
checking dynamic linker characteristics... no
checking how to hardcode library paths into programs... unsupported
appending configuration tag "F77" to libtool
checking if libtool supports shared libraries... no
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking for g77 option to produce PIC...
checking if g77 static flag -static works... no
checking if g77 supports -c -o file.o... yes
checking whether the g77 linker (/usr/bin/ld) supports shared libraries... no
checking dynamic linker characteristics... no
checking how to hardcode library paths into programs... unsupported
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ANSI C... (cached) none needed
checking for a BSD-compatible install... conftools/install-sh -c
checking whether gcc accepts -fexceptions... yes
checking for ANSI C header files... (cached) yes
checking whether byte ordering is bigendian... yes
checking for an ANSI C-conforming const... yes
checking for size_t... yes
checking for memmove... yes
checking for bcopy... yes
checking fcntl.h usability... no
checking fcntl.h presence... yes
configure: WARNING: fcntl.h: present but cannot be compiled
configure: WARNING: fcntl.h:     check for missing prerequisite headers?
configure: WARNING: fcntl.h: see the Autoconf documentation
configure: WARNING: fcntl.h:     section "Present But Cannot Be Compiled"
configure: WARNING: fcntl.h: proceeding with the preprocessor's result
configure: WARNING: fcntl.h: in the future, the compiler will take precedence
configure: WARNING:     ## -------------------------------------- ##
configure: WARNING:     ## Report this to expat-bugs <at> libexpat.org ##
configure: WARNING:     ## -------------------------------------- ##
checking for fcntl.h... yes
checking for unistd.h... (cached) no
checking for off_t... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) no
checking for getpagesize... yes
checking for working mmap... no
checking for an ANSI C99-conforming __func__... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating expat_config.h

[6100-02-03-0909]  /tmp/expat-2.0.1 -> make
        /bin/sh ./libtool --silent --mode=compile gcc -I./lib -I. -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -fexceptions  -DHAVE_EXPAT_CONFIG_H -o lib/xmlparse.lo -c lib/xmlparse.c
        /bin/sh ./libtool --silent --mode=compile gcc -I./lib -I. -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -fexceptions  -DHAVE_EXPAT_CONFIG_H -o lib/xmltok.lo -c lib/xmltok.c
        /bin/sh ./libtool --silent --mode=compile gcc -I./lib -I. -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -fexceptions  -DHAVE_EXPAT_CONFIG_H -o lib/xmlrole.lo -c lib/xmlrole.c
        /bin/sh ./libtool --silent --mode=link gcc -I./lib -I. -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -fexceptions  -DHAVE_EXPAT_CONFIG_H -no-undefined -version-info 6:2:5 -rpath /usr/local/lib  -o libexpat.la lib/xmlparse.lo lib/xmltok.lo lib/xmlrole.lo
        gcc -I./lib -I. -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -fexceptions  -DHAVE_EXPAT_CONFIG_H -o xmlwf/xmlwf.o -c xmlwf/xmlwf.c
        gcc -I./lib -I. -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -fexceptions  -DHAVE_EXPAT_CONFIG_H -o xmlwf/xmlfile.o -c xmlwf/xmlfile.c
In file included from /usr/include/fcntl.h:188,
                 from xmlwf/xmlfile.c:9:
/usr/include/unistd.h:923: parse error before '[' token
/usr/include/unistd.h:924: parse error before "rid_t"
make: The error code from the last command is 1.


Stop.
--

Thanks,


----

Jody Haynes
AIX SAN Host Engineering
-----------------------------------------------------------------
Network Appliance, Inc.
7301 Kit Creek Road
Research Triangle Park, NC 27709

Email:  Jody.Haynes <at> netapp.com
Phone:  919.476.5714
-----------------------------------------------------------------

 
_______________________________________________
Expat-bugs mailing list
Expat-bugs <at> libexpat.org
http://mail.libexpat.org/mailman/listinfo/expat-bugs
Permalink | Reply | 0 comments |

Gmane