Vivek Nallur | 3 Jul 17:06 2014
Picon
Picon

osmosis --tag-transform not actually transforming

Hi all,

I'm trying to modify an OSM file using Osmosis, using the following
commandline:

> osmosis --read-xml ./dublin-m50.osm --tag-transform no_highway_transform.xml --write-xml dublin-m50-modified.osm

Unfortunately, no errors are reported and a new file is created. But when I 
open the file, the required tag hasn't been transformed :(

I created a simple transformation file like so:

=========
<translation>
        <name>Do not use highways</name>
        <description>Changes the value of the maxspeed key to be negative</description>
        <match mode="and" type="way">
                <tag k="highway" v="secondary"/>
                <tag k="maxspeed" match_id="speed" v=".*"/>
        </match>
        <output>
                <copy-all />
                <tag from_match="speed" v="-1" />
        </output>
</translation>

===========

I'm unable to diagnose what I'm doing wrong. I tried to write stats to a file, but 
all I get is the name of the transformation file inside it.
(Continue reading)

Simon Nuttall | 27 Jun 13:38 2014
Picon

Unable to parse 41.75901END into a double precision number.

What should I do about this one?

26-Jun-2014 15:37:53
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
SEVERE: Thread for task 48-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to
parse 41.75901END into a double precision number.

Simon
Paul Norman | 19 Jun 21:16 2014
Picon
Gravatar

pgsnapshot memory options

I'm doing an a pgsnapshot --wpd on a machine with a lot of RAM, and ran 
across a garbage collector memory error.

When running, I had the options -Xmx60g -Xms8g and the osmosis command 
line --rb --lp --wpd enableLinestringBuilder=yes 
nodeLocationStoreType=yes. About 1.8b nodes in, I ran into a 
java.lang.OutOfMemoryError: GC overhead limit exceeded. This is 
apparently to stop applications from spending too much time garbage 
collecting because the heap is too small. At 60GB, I hope the heap is 
not too small.

I've tried making a few changes and am running again

* -server (doh, should of had this before)
* -Xms24g (We know it's going to need lots of heap)
* -XX:+UseParallelGC (Apparently higher throughput, and latency or 
pauses isn't an issue on a bulk ETL job)
* -XX:-UseGCOverheadLimit (Disables the overhead limit check)

My thought is that there may be some point in the process where it does 
a *lot* of garbage collection at once, and this is triggering the limit 
check, but periodically doing a lot of GC isn't a huge problem on an ETL 
task which runs overnight, the total time spend on GC is more important. 
Does anyone have any suggestions for improvement?

Something else that's come up is that the node store for --wpd is very 
inefficient. osm2pgsql shows that you can store all the nodes in the 
planet in ~20GB with its dense/sparse cache, or in ~23GB as a simple 
flat array (--flat-nodes). Osmosis needed about twice that the last time 
I checked.
(Continue reading)

Simon Nuttall | 14 Jun 08:28 2014
Picon

Osmosis crash: OsmosisRuntimeException: An output error has occurred

Just a quick note to say that the redefined user table at:

https://lists.openstreetmap.org/pipermail/osmosis-dev/2014-June/001641.html

has successfully allowed my use of osmosis to complete.
Simon Nuttall | 10 Jun 11:35 2014
Picon

Osmosis crash: OsmosisRuntimeException: An output error has occurred

I'm getting this report from running the osmosis command below. The
timings indicate that it crashes after 75 minutes. If I run the
osmosis for only the British Isles that takes two hours - that may be
a clue to where the crashing is occuring. But I do not know what to do
to debug this problem.

Simon

Output:

09-Jun-2014 16:55:11
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
FINE: Waiting for task 1-read-pbf to complete.

09-Jun-2014 18:10:23
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
SEVERE: Thread for task 1-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output
error has occurred, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:160)
        at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:216)
        at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:303)
        at org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:45)
        at org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:68)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)
(Continue reading)

Jochen Topf | 28 Mar 18:33 2014

Building/Debian Package

Hi!

The Debian package of Osmosis is really old and doesn't work on current
unstable any more. Since then Osmosis has switched build systems from ant
to gradle, so we basically have to set up the Debian package build from
scratch. I think it is important we get Osmosis in a current version back
into Debian and Ubuntu and want to help get this going.

Debian doesn't allow getting any binaries from external sources during
the building process. Build has to be done from scratch and using only
other Debian packages as dependencies. I have already figured out that
I can use the Debian "gradle" package and call the gradle from it instead
of the "gradlew" script. But it downloads a bunch of files from Maven:

Download http://repo1.maven.org/maven2/com/puppycrawl/tools/checkstyle/5.6/checkstyle-5.6.pom
Download http://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-parent-5.pom
Download http://repo1.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom
...

I don't know gradle at all.  Is there a list of those dependencies somewhere?
Any suggestions how to disable all this and tell gradle where to search for its
dependecies locally?

Jochen
--

-- 
Jochen Topf  jochen@...  http://www.jochentopf.com/  +49-721-388298
Martijn van Exel | 18 Mar 20:29 2014
Gravatar

Example code for PBF reading

Hi all,

Are there any examples known to you that use the osmosis code in
another project? I am specifically interested in using the PBF reading
code. Or is there perhaps an easier way to add OSM PBF reading to my
own (Java) project?

Thx
Martijn
--

-- 
Martijn van Exel
http://oegeo.wordpress.com/
http://openstreetmap.us/
jotpe | 23 Dec 00:51 2013
Picon

writing a custom plugin - unkown plugin version

Hello List,

i would like to build a custom plugin for osmosis, but the wiki informations are to less for me.

Is there anywhere a tutorial available?

My current problem is an unknown or incompatible version between my plugin dependency and the  org.openstreetmap.osmosis.core.plugin.Core

<?xml version="1.0" ?> <!DOCTYPE plugin PUBLIC "-//JPF//Java Plug-in Manifest 1.0" "http://jpf.sourceforge.net/plugin_1_0.dtd"> <plugin id="OsmosisImporter" version="0.1"> <requires> <import plugin-id="org.openstreetmap.osmosis.core.plugin.Core" plugin-version="0.43.1" reverse-lookup="false"/> </requires> <runtime> <library id="mycode" path="/" type="code"/> </runtime> <extension plugin-id="org.openstreetmap.osmosis.core.plugin.Core" point-id="Task" id="OsmosismImporter"> <parameter id="name" value="importer"/> <parameter id="class" value="de.regioosm.osmosis.plugin.PluginLoader" /> </extension> </plugin>

This command causes 
osmosis --read-xml city1.osm --importer



Caused by: org.java.plugin.PluginLifecycleException: plug-in OsmosisImporter requires plug-in org.openstreetmap.osmosis.core.plugin.Core which is unknown or has incompatible version
at org.java.plugin.standard.StandardPluginManager.checkPrerequisites(StandardPluginManager.java:460)
at org.java.plugin.standard.StandardPluginManager.activatePlugin(StandardPluginManager.java:397)


My osmosis version is 0.43.1 and the maven dependecies of the plugin project is 0.43.1 too. What can i do? Regards Johannes
_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
https://lists.openstreetmap.org/listinfo/osmosis-dev
Walter Nordmann | 16 Nov 02:21 2013
Picon

Re: Streaming Replication

Hi Brett,

is Streaming replication running or is it dead?

The first (and last) state i get is that:

and when i do an "osmosis  --receive-replication
host=planet.openstreetmap.org port=80 pathPrefix=replication/streaming
--replication-to-change --write-xml-change -"
i get

any hints?

Regards
walter 

* sent this mail a second time because of some techical problems with
nabble.

-----
[url=http://osm.wno-edv-service.de/residentials] Missing Residentials Map 1.17[/url]
[url=http://osm.wno-edv-service.de/plz] Postcode Map 2.0.2[/url]
--
View this message in context: http://gis.19327.n5.nabble.com/osmosis-dev-Streaming-Replication-tp5730318p5785642.html
Sent from the Osmosis Development mailing list archive at Nabble.com.
Nuno Miguel Lourenço | 25 Oct 19:11 2013
Picon

Error when updating OSM database

Hi all,

 

I’ve built an ANT process to populate an OSM database with a full PLANET file.

Afterwards I built another ANT process to update the given OSM database.

 

I’m getting an error on the update process.

Basically what I’ve done:

First:

<path to osmosis> --read-replication-interval-init workingDirectory=.

Second:

<path to osmosis> -v --read-replication-interval workingDirectory=. --write-pgsql-change host=host:port user=user database=database password=password

 

I’m getting stuck because I cannot make it work…

I’m getting this exceptions:

update_database:

     [echo] UPDATING OSM DB...

     [echo] JAVACMD_OPTIONS -> -server -Xms2000m -Xmx4000m -Djava.io.tmpdir=D:/Temp/tests/osmosis/rep/osmosis/tmp/...

     [exec] The command attribute is deprecated.

     [exec] Please use the executable attribute and nested arg elements.

     [exec] Out 25, 2013 6:10:19 PM org.openstreetmap.osmosis.core.Osmosis run

     [exec] INFO: Osmosis Version 0.43.1

     [exec] Out 25, 2013 6:10:19 PM org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins

     [exec] FINE: Searching for JPF plugins.

     [exec] Out 25, 2013 6:10:19 PM org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins

     [exec] FINE: Registering the core plugin.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins

     [exec] FINE: Registering the extension plugins.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.Osmosis run

     [exec] INFO: Preparing pipeline.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline prepare

     [exec] FINE: Building tasks.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks

     [exec] FINE: Created task "1-read-replication-interval"

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks

     [exec] FINE: Created task "2-sort-change"

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks

     [exec] FINE: Created task "3-write-pgsql-change"

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline prepare

     [exec] FINE: Connecting tasks.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask

     [exec] FINE: Task "1-read-replication-interval" produced unnamed pipe stored at level 1 in the default pipe stack.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks

     [exec] FINE: Connected task "1-read-replication-interval"

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask

     [exec] FINE: Task "2-sort-change" consumed unnamed pipe stored at level 1 in the default pipe stack.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask

     [exec] FINE: Task "2-sort-change" produced unnamed pipe stored at level 1 in the default pipe stack.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks

     [exec] FINE: Connected task "2-sort-change"

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask

     [exec] FINE: Task "3-write-pgsql-change" consumed unnamed pipe stored at level 1 in the default pipe stack.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks

     [exec] FINE: Connected task "3-write-pgsql-change"

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.Osmosis run

     [exec] INFO: Launching pipeline execution.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager execute

     [exec] FINE: Launching task 1-read-replication-interval in a new thread.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager execute

     [exec] FINE: Task 2-sort-change is passive, no execution required.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager execute

     [exec] FINE: Task 3-write-pgsql-change is passive, no execution required.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.Osmosis run

     [exec] INFO: Pipeline executing, waiting for completion.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion

     [exec] FINE: Waiting for task 1-read-replication-interval to complete.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader runImpl

     [exec] FINE: Reading current server state.

     [exec] Out 25, 2013 6:10:20 PM sun.net.www.protocol.http.HttpURLConnection writeRequests

     [exec] FINE: sun.net.www.MessageHeader <at> 591686f75 pairs: {GET /replication/hour/state.txt HTTP/1.1: null}{User-Agent: Java/1.7.0_40}{Host: planet.openstreetmap.org}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*;

q=.2}{Connection: keep-alive}

     [exec] Out 25, 2013 6:10:20 PM sun.net.www.protocol.http.HttpURLConnection getInputStream

     [exec] FINE: sun.net.www.MessageHeader <at> a18047511 pairs: {null: HTTP/1.1 200 OK}{Date: Fri, 25 Oct 2013 17:10:10 GMT}{Server: Apache/2.2.22 (Ubuntu)}{Last-Modified: Fri, 25 Oct 2013 17:02:07 GMT}{ETag: "5f2983d-53-4e993b5

d8f0f4"}{Accept-Ranges: bytes}{Content-Length: 83}{Vary: Accept-Encoding}{Keep-Alive: timeout=5, max=100}{Connection: Keep-Alive}{Content-Type: text/plain; charset=utf-8}

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader download

     [exec] FINE: The maximum timestamp to be downloaded is Wed Oct 09 01:00:00 BST 2013.

     [exec] Out 25, 2013 6:10:20 PM sun.net.www.protocol.http.HttpURLConnection writeRequests

     [exec] FINE: sun.net.www.MessageHeader <at> 1a2b53fb5 pairs: {GET /replication/hour/000/009/401.state.txt HTTP/1.1: null}{User-Agent: Java/1.7.0_40}{Host: planet.openstreetmap.org}{Accept: text/html, image/gif, image/jpeg, *;

q=.2, */*; q=.2}{Connection: keep-alive}

     [exec] Out 25, 2013 6:10:20 PM sun.net.www.protocol.http.HttpURLConnection getInputStream

     [exec] FINE: sun.net.www.MessageHeader <at> 1296912911 pairs: {null: HTTP/1.1 200 OK}{Date: Fri, 25 Oct 2013 17:10:10 GMT}{Server: Apache/2.2.22 (Ubuntu)}{Last-Modified: Wed, 09 Oct 2013 00:02:05 GMT}{ETag: "5f98325-53-4e8439

87f67dc"}{Accept-Ranges: bytes}{Content-Length: 83}{Vary: Accept-Encoding}{Keep-Alive: timeout=5, max=99}{Connection: Keep-Alive}{Content-Type: text/plain; charset=utf-8}

     [exec] Out 25, 2013 6:10:20 PM sun.net.www.protocol.http.HttpURLConnection writeRequests

     [exec] FINE: sun.net.www.MessageHeader <at> 1d8330fd5 pairs: {GET /replication/hour/000/009/401.osc.gz HTTP/1.1: null}{User-Agent: Java/1.7.0_40}{Host: planet.openstreetmap.org}{Accept: text/html, image/gif, image/jpeg, *; q=

.2, */*; q=.2}{Connection: keep-alive}

     [exec] Out 25, 2013 6:10:20 PM sun.net.www.protocol.http.HttpURLConnection getInputStream

     [exec] FINE: sun.net.www.MessageHeader <at> 1005412e10 pairs: {null: HTTP/1.1 200 OK}{Date: Fri, 25 Oct 2013 17:10:10 GMT}{Server: Apache/2.2.22 (Ubuntu)}{Last-Modified: Wed, 09 Oct 2013 00:02:05 GMT}{ETag: "5f98324-babfc-4e8

43987f67dc"}{Accept-Ranges: bytes}{Content-Length: 764924}{Keep-Alive: timeout=5, max=98}{Connection: Keep-Alive}{Content-Type: application/x-gzip}

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion

     [exec] SEVERE: Thread for task 1-read-replication-interval failed

     [exec] org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read the changeset file 000/009/401.osc.gz from the server.

     [exec]     at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.downloadReplicationFile(BaseReplicationDownloader.java:133)

     [exec]     at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.download(BaseReplicationDownloader.java:262)

     [exec]     at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:302)

     [exec]     at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:381)

     [exec]     at java.lang.Thread.run(Unknown Source)

     [exec] Caused by: java.io.IOException: The system cannot find the path specified

     [exec]     at java.io.WinNTFileSystem.createFileExclusively(Native Method)

     [exec]     at java.io.File.createNewFile(Unknown Source)

     [exec]     at java.io.File.createTempFile(Unknown Source)

     [exec]     at java.io.File.createTempFile(Unknown Source)

     [exec]     at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.downloadReplicationFile(BaseReplicationDownloader.java:111)

     [exec]     ... 4 more

     [exec]

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager waitForCompletion

     [exec] FINE: Task 2-sort-change is passive, no completion wait required.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager waitForCompletion

     [exec] FINE: Task 3-write-pgsql-change is passive, no completion wait required.

     [exec] Out 25, 2013 6:10:20 PM org.openstreetmap.osmosis.core.Osmosis main

     [exec] SEVERE: Execution aborted.

     [exec] org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.

     [exec]     at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)

     [exec]     at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)

     [exec]     at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)

     [exec]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     [exec]     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

     [exec]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

     [exec]     at java.lang.reflect.Method.invoke(Unknown Source)

     [exec]     at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)

     [exec]     at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)

     [exec]     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)

     [exec]     at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

     [exec]     at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

     [exec]

 

BUILD FAILED

 

Can anyone point me to a correction direction.

 

Thanks to all.

Nuno Lourenço

_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
https://lists.openstreetmap.org/listinfo/osmosis-dev
NopMap | 21 Oct 19:45 2013
Picon
Picon

Exception in PluginManager


Hi!

On a particular machine, osmosis is throwing the following exception:

Okt 20, 2013 2:30:36 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.39
Okt 20, 2013 2:30:36 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
java.lang.NullPointerException
at
org.java.plugin.standard.StandardPluginManager.publishPlugins(StandardPluginManager.java:181)
at
org.openstreetmap.osmosis.core.TaskRegistrar.registerJpfPlugins(TaskRegistrar.java:239)
at
org.openstreetmap.osmosis.core.TaskRegistrar.loadJPFPlugins(TaskRegistrar.java:167)
at
org.openstreetmap.osmosis.core.TaskRegistrar.initialize(TaskRegistrar.java:88)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:81)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)

It's a prepared setup that has been working flawlessly on many machines for
a long time.

Any ideas what could be possible reasons for this exception?

bye, Nop

--
View this message in context: http://gis.19327.n5.nabble.com/Exception-in-PluginManager-tp5782406.html
Sent from the Osmosis Development mailing list archive at Nabble.com.

Gmane