Scott Crosby | 3 Jan 05:08 2011

PBF version 1.1, a PBF conformance file, and changes to Osmosis's PBF support

I am releasing version 1.1 and have just pushed several changes to
osmosis trunk. These changes don't effect the format, but may require
minor changes to software that uses the *.proto definitions.

   Although it has been in trunk for a while, this is the first
release where the generated C/C++ protobuf code is in package OSMPBF.

   I am renaming from 'osmbin.jar' to 'osmpbf.jar'.

   I am licensing the osmpbf source code under the LGPL.

   As previously discussed, I am removing bzip2 compression and
marking lzma compression as a proposed extension, this may require
minor changes to existing software.

   I am renaming the protocol buffer fileformat.BlockHeader to
fileformat.BlobHeader (It has a name confusingly similar to
osmformat.HeaderBlock.). Software will have to be patched to compile
against the new name for this object.

   I have a conformance file. This file exercises all of the features
required to decode the geographic data in PBF files, including offset
and granularity. Note that this test is not exhaustive, in particular,
it does not currently test the metadata storage (timestamps, version
numbers, changeset numbers, etc.) of the PBF format. It is located at
the main PBF repository (https://github.com/scrosby/OSM-binary). As it
is still a work in progress, invoke crosby.binary.test.BuildTestFile
to generate the PBF testfiles. The reference XML output can be
generated with osmosis "" osmosis --read-pbf TestFile1-deflate.osm.pbf
--write-xml file=- "". To generate the corresponding XML output, you
(Continue reading)

Peter Schrammel | 4 Jan 08:19 2011
Picon
Picon

Extacting some areas

Hi,

I'm new to this whole openstretmap/(Post)GIS/osmosis stuff so please 
forgive my naive questions ;-)

The question is simple: Can I extract the organisational areas out of a 
planet file with osmosis?

I just want to have country, state, ... city shapes (evtl. zip-code 
areas). I'm not interested in any othe information like POI, streets, 
lakes,..... Is this possible with osmosis? Has anybody done this? Do you 
have any numbers how big the file/DB will be?

THX
Peter
Brett Henderson | 7 Jan 16:54 2011

Re: Source for JAR archives in osmosis source

I've finally checked this in.  There's now a new directory in the Osmosis source tree called hstore-jdbc.  It is a project called osmosis-hstore which creates the osmosis-hstore.jar file.  Note that the licence is BSD unlike the rest of Osmosis because it is a file obtained from the PostgreSQL forum that will (hopefully) eventually become part of the PostgreSQL JDBC driver.

On Fri, Dec 31, 2010 at 9:07 AM, Brett Henderson <brett <at> bretth.com> wrote:
On 24/12/2010, at 10:26, Brett Henderson <brett-ylI/TL/YVhnQT0dZR+AlfA@public.gmane.org> wrote:

On Fri, Dec 24, 2010 at 9:18 AM, Giovanni Mascellani <mascellani-1ggtkMrCeZFoG1tgCFlPok6FHaWFtxp8@public.gmane.org> wrote:
Hi.

On 23/12/2010 22:27, Brett Henderson wrote:
> Hi Giovanni,

Thanks for the quick reply. :-)

> The source for the osmbin jar is stored in a git repo on github. I
> don't have the URL handy (I'm on my phone) but I think a search for
> PBF or similar should find it on the osm wiki ...

Ok, found. Anyway, no license indication is present in the repository,
and this is a problem for including the package in Debian, because
technically it means that no one can do anything with it.

Since this project is strictly tied with osmosis, I suppose that it's
its author intention to release it in public domain like osmosis. Could
this bit be made explicit in some file into the repository, something
like the copying.txt present in osmosis main repository?

(BTW, Scott Crosby reads this mailing list, doesn't he?)

Yes, he should do.

I'm glad you've asked the licence question actually.  I should have sorted that out sooner but it slipped my mind.
 

> The source for osmosis-hstore is in the binary jar I think. I didn't
> bother setting up a proper project for it because it is relatively
> tiny.

No, they're not. I can just see the .class files.

Oops.  Not sure what I've done there.  I'll take a look when I get home, perhaps it's just sitting on my home PC.  I need to get that checked in somewhere.

Sorry for delay. I have found the source. Will try to get it checked in over the next few days.

It's a single class I found on a forum somewhere. Relatively simple so I hope licensing isn't an issue.

Brett

_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
david.remer | 18 Jan 14:33 2011
Picon

tf accept-nodes natural

Hi everyone,
I want to extract the border and all natural-tags. i tried it this way:

 "C:\map\osmosis-0.37\bin\osmosis.bat" 
--read-xml file="c:\map\3.osm"  
--tf accept-ways admin_level=1,2,3,4,5
--tf accept-nodes natural=*
--tf reject-relations  
--used-node   
--write-xml output2.osm

but it doesnt work :(
does someone knows how to accept "natural"?

thanks alot.
___________________________________________________________
Empfehlen Sie WEB.DE DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.web.de
Dima Batenkov | 23 Jan 00:25 2011
Picon

Merging two entity streams fails on Windows

Hi all,

I have osmosis 0.38 on Windows XP 32-bit.
I am trying to run the following command:

osmosis -q \
--rx israel.osm --tf accept-ways boundary=* --tf accept-relations boundary=* --used-node --sort \
--rx israel.osm --tf accept-ways natural=coastline --tf reject-relations --used-node --sort \
--merge --wx admin.osm

The command fails with the output (attached).

I can split the above command into three separate commands which run successfully:

osmosis -q --rx israel.osm --tf accept-ways boundary=* --tf accept-relations boundary=* --used-node --sort --wx b1.osm
osmosis -q --rx israel.osm --tf accept-ways natural=coastline --tf reject-relations --used-node --sort --wx b2.osm
osmosis -q --rx b1.osm --rx b2.osm --merge --wx admin.osm

However (this is another unrelated problem) I cannot put the three above commands into one batch file because the batch exits after completing just the first line.

Thanks,
Dmitry

Jan 23, 2011 1:20:37 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-rx failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error has occurred, aborting.
	at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:76)
	at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:127)
	at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:182)
	at org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:34)
	at org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:58)
	at org.openstreetmap.osmosis.core.sort.v0_6.EntitySorter.complete(EntitySorter.java:62)
	at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.complete(UsedNodeFilter.java:143)
	at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
	at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
	at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:110)
	at java.lang.Thread.run(Unknown Source)
Jan 23, 2011 1:20:37 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 6-rx failed
java.lang.LinkageError: loader (instance of  org/codehaus/plexus/classworlds/realm/ClassRealm):
attempted  duplicate class definition for name: "org/apache/xerces/jaxp/datatype/DatatypeFactoryImpl"
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClassCond(Unknown Source)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at java.security.SecureClassLoader.defineClass(Unknown Source)
	at java.net.URLClassLoader.defineClass(Unknown Source)
	at java.net.URLClassLoader.access$000(Unknown Source)
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(Unknown Source)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:386)
	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230)
	at javax.xml.datatype.FactoryFinder.getProviderClass(Unknown Source)
	at javax.xml.datatype.FactoryFinder.newInstance(Unknown Source)
	at javax.xml.datatype.FactoryFinder.findJarServiceProvider(Unknown Source)
	at javax.xml.datatype.FactoryFinder.find(Unknown Source)
	at javax.xml.datatype.DatatypeFactory.newInstance(Unknown Source)
	at org.openstreetmap.osmosis.core.time.DateParser.<init>(DateParser.java:36)
	at org.openstreetmap.osmosis.xml.common.XmlTimestampFormat.parseTimestamp(XmlTimestampFormat.java:58)
	at org.openstreetmap.osmosis.core.domain.common.UnparsedTimestampContainer.getTimestamp(UnparsedTimestampContainer.java:65)
	at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.getTimestamp(CommonEntityData.java:293)
	at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.store(CommonEntityData.java:188)
	at org.openstreetmap.osmosis.core.domain.v0_6.Entity.store(Entity.java:138)
	at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:184)
	at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.store(NodeContainer.java:49)
	at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63)
	at org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:106)
	at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:78)
	at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
	at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:61)
	at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:133)
	at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:133)
	at org.openstreetmap.osmosis.xml.v0_6.impl.NodeElementProcessor.end(NodeElementProcessor.java:117)
	at org.openstreetmap.osmosis.xml.v0_6.impl.OsmHandler.endElement(OsmHandler.java:107)
	at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
	at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
	at javax.xml.parsers.SAXParser.parse(Unknown Source)
	at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:108)
	at java.lang.Thread.run(Unknown Source)
Jan 23, 2011 1:20:37 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 11-merge failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An input error has occurred, aborting.
	at org.openstreetmap.osmosis.core.store.DataPostbox.checkForInputErrors(DataPostbox.java:88)
	at org.openstreetmap.osmosis.core.store.DataPostbox.consumeCentralQueue(DataPostbox.java:158)
	at org.openstreetmap.osmosis.core.store.DataPostbox.hasNext(DataPostbox.java:235)
	at org.openstreetmap.osmosis.set.v0_6.EntityMerger.run(EntityMerger.java:107)
	at java.lang.Thread.run(Unknown Source)
Jan 23, 2011 1:20:37 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:31)
_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
Peter Körner | 23 Jan 19:07 2011
Picon

Re: Merging two entity streams fails on Windows

Dima Batenkov schrieb:
> However (this is another unrelated problem) I cannot put the three above
> commands into one batch file because the batch exits after completing
> just the first line.

This is because osmosis is launched as a batch script (osmosis.bat) 
which uses exit to terminate. Instead it use a jump-mark (:eof) at the 
end of the batch-file and jump there using goto :eof when it wishes to 
terminate.

Patches welcome.

Peter
Dima Batenkov | 23 Jan 19:51 2011
Picon

Re: Merging two entity streams fails on Windows

On 23 January 2011 20:07, Peter Körner <osm-lists-ttMXgdK/SSfIJoCZWJrliQ@public.gmane.org> wrote:
Dima Batenkov schrieb:

However (this is another unrelated problem) I cannot put the three above
commands into one batch file because the batch exits after completing
just the first line.

This is because osmosis is launched as a batch script (osmosis.bat) which uses exit to terminate. Instead it use a jump-mark (:eof) at the end of the batch-file and jump there using goto :eof when it wishes to terminate.

Patches welcome.

Peter

I am not familiar with the inner workings of batch files, but is seems to me that there is no  "exit" used in osmosis.bat. The actual call is at the end of the batch, with the lines

SET EXEC=%JAVACMD% %JAVACMD_OPTIONS% -cp "%PLEXUS_CP%" -Dapp.home="%MYAPP_HOME%" -Dclassworlds.conf="%MYAPP_HOME%\config\plexus.conf" %MAINCLASS%  %OSMOSIS_OPTIONS% %*

%EXEC%

Dmitry
_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
Scott Crosby | 26 Jan 06:37 2011

osmpbf v1.1.1 jar released. It should fix the PBF-on-windows issues.

Over the last few months, several bugs have been reported where the
pbf reader seems to sometimes not work correctly on Windows. The bug
was caused by a 32-bit cleanness bug in the shared osmpbf library,
where the reader would erroneously sense the end of a file and cease
reading. For whatever reason, linux doesn't trigger the bug. This
version of the library should fix those issues on windows and has been
committed to osmosis trunk. I recommend replacing any other existing
installs of osmbin.jar or osmpbf.jar with the new library.

I would like to thank Stephan Knauss for helping to identify the root
cause of this bug and testing the fixed library. If there are any
other issues with this release, please send an email.

Scott
Chris66 | 26 Jan 21:27 2011
Picon
Picon

Re: osmpbf v1.1.1 jar released. It should fix the PBF-on-windows issues.

Am 26.01.2011 06:37, schrieb Scott Crosby:

> This version of the library should fix those issues on windows and has been
> committed to osmosis trunk. 

Hi,
when will this be released to osmosis-latest?

Tried to build on Win7 without success.

I used:

svn co "http://svn.openstreetmap.org/applications/utils/osmosis/trunk/"
osmosis

cd osmosis

ant publish

Chris
Scott Crosby | 27 Jan 03:31 2011

Re: osmpbf v1.1.1 jar released. It should fix the PBF-on-windows issues.

On Tue, Jan 25, 2011 at 11:37 PM, Scott Crosby <scott@...> wrote:
> Over the last few months, several bugs have been reported where the
> pbf reader seems to sometimes not work correctly on Windows. The bug
> was caused by a 32-bit cleanness bug in the shared osmpbf library,
> where the reader would erroneously sense the end of a file and cease
> reading. For whatever reason, linux doesn't trigger the bug. This
> version of the library should fix those issues on windows and has been
> committed to osmosis trunk. I recommend replacing any other existing
> installs of osmbin.jar or osmpbf.jar with the new library.
>
> I would like to thank Stephan Knauss for helping to identify the root
> cause of this bug and testing the fixed library. If there are any
> other issues with this release, please send an email.

Actually it would help if I actually pushed the new release to SVN. I
prepared it, but didn't actually push. My apologies.

I have just pushed it now to both SVN and git and it has passed the
hudson build.

Scott

Gmane