Christoph Heer | 11 Oct 02:57 2014
Picon

Will write-pgsql-change update lines from LinestringBuilder?

Hi everyone,

first of all I would like to say thanks for this great tool. I did my import with osmosis in a pgsnapshot schema
with '--write-pgsql-dump’ and used the enableLinestringBuilder feature. Now I want keep my database
up-to-date with change files over '--write-pgsql-change’ but will this also updates the linestring
in the ways table? I didn’t found any comment or note to this topic. I hope someone can answer my question. =)

Best regards
Christoph
_______________________________________________
osmosis-dev mailing list
osmosis-dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Justin Palmer | 10 Oct 21:05 2014
Picon

Supplying postgres port to osmosis commands

Hi,

I have Postgres running on a non-default port and unfortunately it doesn't look like osmosis will accept a custom port according to the detailed usage wiki page.  Is this a correct assumption or is there a way to supply the postgres port to osmosis?

Here is the error I get while running osmosis without supplying a port:  https://gist.github.com/Caged/0c701eddfaa4b93ee172#file-osmosis-log

Here's the error I get when attempting to supply a port:


Thanks,
  -Justin
_______________________________________________
osmosis-dev mailing list
osmosis-dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Jörg Frings-Fürst | 28 Aug 11:29 2014
Picon

timeout error

Hi,

in the last time I get a lot of time out errors with osmosis.

Is there a option to increase the value of

 - Keep-Alive: timeout=5, max=1

or to set a bandwidth limit?

Thanks

CU
Jörg

####
AM FEINSTEN: Proxy used: DIRECT
Aug 28, 2014 10:51:35 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
FEIN: sun.net.www.MessageHeader <at> 2d6a66055 pairs: {GET
/replication/minute/001/022/015.state.txt HTTP/1.1: null}{User-Agent: Java/1.7.0_65}{Host:
planet.openstreetmap.org}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*;
q=.2}{Connection: keep-alive}
Aug 28, 2014 10:51:36 AM sun.net.www.protocol.http.HttpURLConnection getInputStream
FEIN: sun.net.www.MessageHeader <at> 244fe98e11 pairs: {null: HTTP/1.1 200 OK}{Date: Thu, 28 Aug 2014
08:51:35 GMT}{Server: Apache/2.2.22 (Ubuntu)}{Last-Modified: Wed, 27 Aug 2014 13:51:04 GMT}{ETag:
"5b8d0a6-b9-5019cb607409b"}{Accept-Ranges: bytes}{Content-Length: 185}{Vary:
Accept-Encoding}{Keep-Alive: timeout=5, max=1}{Connection: Keep-Alive}{Content-Type:
text/plain; charset=utf-8}
Aug 28, 2014 10:51:36 AM sun.net.www.protocol.http.HttpURLConnection plainConnect
AM FEINSTEN: ProxySelector Request for http://planet.openstreetmap.org/replication/minute/001/022/015.osc.gz
Aug 28, 2014 10:53:43 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SCHWERWIEGEND: Thread for task 1-rri failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read the changeset file
001/022/015.osc.gz from the server.
####

-- 
Jörg Frings-Fürst
OSM privat
D-54526 Landscheid

Threema: SYR8SJXB

GPG Fingerprint: 13E3 4D4A 3228 D138 8511 EA5A 08AC AF02 3C6D 750A
Full GPG key: hkp://pool.sks-keyservers.net
CAcert Serialnr.: 0D:9A:23
SHA1-Fingerprint: CA:36:4D:44:D1:71:4A:78:C8:6C:C2:CC:94:F3:6E:42:38:BA:CE:4E
http://cacert.org

_______________________________________________
osmosis-dev mailing list
osmosis-dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Nuno Miguel Lourenço | 17 Jul 13:08 2014
Picon

Plugin: Changing default schema for pgsnapshot_schema

Hi all,

How can I add the capability of changing the default schema for pgsnapshot_schema schema?
Instead of the tables being in "public" schema in PortgreSQL, changing it for example "osm" schema?

Thank you all.

Nuno Lourenço 
DTS - Sapo Technology Department 
GIS/Maps
Vivek Nallur | 3 Jul 17:06 2014
Picon
Picon

osmosis --tag-transform not actually transforming

Hi all,

I'm trying to modify an OSM file using Osmosis, using the following
commandline:

> osmosis --read-xml ./dublin-m50.osm --tag-transform no_highway_transform.xml --write-xml dublin-m50-modified.osm

Unfortunately, no errors are reported and a new file is created. But when I 
open the file, the required tag hasn't been transformed :(

I created a simple transformation file like so:

=========
<translation>
        <name>Do not use highways</name>
        <description>Changes the value of the maxspeed key to be negative</description>
        <match mode="and" type="way">
                <tag k="highway" v="secondary"/>
                <tag k="maxspeed" match_id="speed" v=".*"/>
        </match>
        <output>
                <copy-all />
                <tag from_match="speed" v="-1" />
        </output>
</translation>

===========

I'm unable to diagnose what I'm doing wrong. I tried to write stats to a file, but 
all I get is the name of the transformation file inside it.

I'm using Osmosis Version 0.43.1, so I'm assuming the tag-transform plugin doesn't 
need to be built separately.

Any ideas/suggestions would be appreciated. 

Note: I'm aware that highways could be removed using "reject-ways highway=motorway,motorway_link"
but let's just assume that I actually want to update a tag's value.

regs
Vivek
Simon Nuttall | 27 Jun 13:38 2014
Picon

Unable to parse 41.75901END into a double precision number.

What should I do about this one?

26-Jun-2014 15:37:53
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
SEVERE: Thread for task 48-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to
parse 41.75901END into a double precision number.

Simon
Paul Norman | 19 Jun 21:16 2014
Picon
Gravatar

pgsnapshot memory options

I'm doing an a pgsnapshot --wpd on a machine with a lot of RAM, and ran 
across a garbage collector memory error.

When running, I had the options -Xmx60g -Xms8g and the osmosis command 
line --rb --lp --wpd enableLinestringBuilder=yes 
nodeLocationStoreType=yes. About 1.8b nodes in, I ran into a 
java.lang.OutOfMemoryError: GC overhead limit exceeded. This is 
apparently to stop applications from spending too much time garbage 
collecting because the heap is too small. At 60GB, I hope the heap is 
not too small.

I've tried making a few changes and am running again

* -server (doh, should of had this before)
* -Xms24g (We know it's going to need lots of heap)
* -XX:+UseParallelGC (Apparently higher throughput, and latency or 
pauses isn't an issue on a bulk ETL job)
* -XX:-UseGCOverheadLimit (Disables the overhead limit check)

My thought is that there may be some point in the process where it does 
a *lot* of garbage collection at once, and this is triggering the limit 
check, but periodically doing a lot of GC isn't a huge problem on an ETL 
task which runs overnight, the total time spend on GC is more important. 
Does anyone have any suggestions for improvement?

Something else that's come up is that the node store for --wpd is very 
inefficient. osm2pgsql shows that you can store all the nodes in the 
planet in ~20GB with its dense/sparse cache, or in ~23GB as a simple 
flat array (--flat-nodes). Osmosis needed about twice that the last time 
I checked.
Simon Nuttall | 14 Jun 08:28 2014
Picon

Osmosis crash: OsmosisRuntimeException: An output error has occurred

Just a quick note to say that the redefined user table at:

https://lists.openstreetmap.org/pipermail/osmosis-dev/2014-June/001641.html

has successfully allowed my use of osmosis to complete.
Simon Nuttall | 10 Jun 11:35 2014
Picon

Osmosis crash: OsmosisRuntimeException: An output error has occurred

I'm getting this report from running the osmosis command below. The
timings indicate that it crashes after 75 minutes. If I run the
osmosis for only the British Isles that takes two hours - that may be
a clue to where the crashing is occuring. But I do not know what to do
to debug this problem.

Simon

Output:

09-Jun-2014 16:55:11
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
FINE: Waiting for task 1-read-pbf to complete.

09-Jun-2014 18:10:23
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
SEVERE: Thread for task 1-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output
error has occurred, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:160)
        at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:216)
        at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:303)
        at org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:45)
        at org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:68)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)
        at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)
        at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
        at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
        at java.lang.Thread.run(Thread.java:701)

Command:

[09-Jun-2014 15:55:10 UTC] reconnectExec About to execute
/usr/local/bin/osmosis -v 100 --read-pbf
/websites/www/content/data/osm/downloads/europe/british-isles-latest.os
m.pbf  --tag-filter accept-ways highway=* leisure=* aerialway=*
route=ferry man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --read-pbf /website
s/www/content/data/osm/downloads/europe/denmark-latest.osm.pbf
--tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reje
ct-ways highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/estonia-latest.osm.pbf
 --tag-filter accept-ways highway=* leisur
e=* aerialway=* route=ferry man_made=pier,jetty --tag-filter
reject-ways highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/
germany-latest.osm.pbf  --tag-filter accept-ways highway=* leisure=*
aerialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --
merge --read-pbf
/websites/www/content/data/osm/downloads/europe/czech-republic-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_m
ade=pier,jetty --tag-filter reject-ways highway=motorway,construction
--merge --read-pbf
/websites/www/content/data/osm/downloads/europe/croatia-latest.osm.pbf
 --tag-fil
ter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge --read-pbf /websites/www/co
ntent/data/osm/downloads/europe/lithuania-latest.osm.pbf  --tag-filter
accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-way
s highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/iceland-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* ae
rialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/norway
-latest.osm.pbf  --tag-filter accept-ways highway=* leisure=*
aerialway=* route=ferry man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge -
-read-pbf /websites/www/content/data/osm/downloads/europe/sweden-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty
--tag-filter reject-ways highway=motorway,construction --merge
--read-pbf /websites/www/content/data/osm/downloads/europe/netherlands-latest.osm.pbf
 --tag-filter accept-
ways highway=* leisure=* aerialway=* route=ferry man_made=pier,jetty
--tag-filter reject-ways highway=motorway,construction --merge
--read-pbf /websites/www/content/data/
osm/downloads/europe/belgium-latest.osm.pbf  --tag-filter accept-ways
highway=* leisure=* aerialway=* route=ferry man_made=pier,jetty
--tag-filter reject-ways highway=mot
orway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/luxembourg-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=*
route=ferry man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/france-latest.os
m.pbf  --tag-filter accept-ways highway=* leisure=* aerialway=*
route=ferry man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/switzerland-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge --read-pbf
/websites/www/content/data/osm/downloads/north-america/us/new-york-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --bounding-polygon
file="import/fz_ny_cm.txt" --merge --read-pbf
/websites/www/content/data/osm/downloads/north-america/us/new-jersey-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --bounding-polygon
file="import/fz_nj_cm.txt" --merge --read-pbf
/websites/www/content/data/osm/downloads/europe/spain-latest.osm.pbf
--tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --bounding-polygon
file="import/fz_spain_cm.txt" --merge --read-pbf
/websites/www/content/data/osm/downloads/north-america/us/massachusetts-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --bounding-polygon
file="import/fz_boston_cm.txt" --merge --read-pbf
/websites/www/content/data/osm/downloads/north-america/us/district-of-columbia-latest.osm.pbf
 --tag-filter accept-ways highway=* leisure=* aerialway=* route=ferry
man_made=pier,jetty --tag-filter reject-ways
highway=motorway,construction --merge  --buffer --write-apidb
dbType=mysql populateCurrentTables=no host=localhost
database=planetExtractOSM140609 user=import password=
validateSchemaVersion=no
Jochen Topf | 28 Mar 18:33 2014

Building/Debian Package

Hi!

The Debian package of Osmosis is really old and doesn't work on current
unstable any more. Since then Osmosis has switched build systems from ant
to gradle, so we basically have to set up the Debian package build from
scratch. I think it is important we get Osmosis in a current version back
into Debian and Ubuntu and want to help get this going.

Debian doesn't allow getting any binaries from external sources during
the building process. Build has to be done from scratch and using only
other Debian packages as dependencies. I have already figured out that
I can use the Debian "gradle" package and call the gradle from it instead
of the "gradlew" script. But it downloads a bunch of files from Maven:

Download http://repo1.maven.org/maven2/com/puppycrawl/tools/checkstyle/5.6/checkstyle-5.6.pom
Download http://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-parent-5.pom
Download http://repo1.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom
...

I don't know gradle at all.  Is there a list of those dependencies somewhere?
Any suggestions how to disable all this and tell gradle where to search for its
dependecies locally?

Jochen
--

-- 
Jochen Topf  jochen@...  http://www.jochentopf.com/  +49-721-388298
Martijn van Exel | 18 Mar 20:29 2014
Gravatar

Example code for PBF reading

Hi all,

Are there any examples known to you that use the osmosis code in
another project? I am specifically interested in using the PBF reading
code. Or is there perhaps an easier way to add OSM PBF reading to my
own (Java) project?

Thx
Martijn
--

-- 
Martijn van Exel
http://oegeo.wordpress.com/
http://openstreetmap.us/

Gmane