fin | 23 Apr 15:20 2013

feature request: used-relations

Hi there,

i'd like to request a feature: please allow users to _include_
relations which are referenced by relations, much like --used-way and
--used-node filter out ways and nodes that are not referenced.

my use case is simple: i'm trying to extract boundary data for one
admin_level in austria and some of those boundaries reference
subareas:
http://wiki.openstreetmap.org/wiki/Relation:boundary

i can't seem to filter them out and i can't have osmosis include them,
so most tools i've tried end up barfing on that data.

including all entities - either by default or as an option, since that
would add complexity & computing time - that are referenced seems like
the most reasonable course of action.

thanks!
-fin
Simon Nuttall | 22 Apr 23:52 2013
Picon

Password contains a percent character

I'm calling osmosis from php, and when the password contains a %
character it fails to login.

E.g when the last character of the password is % I get this:

java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing
escape (%) pattern

and if the % char is in the middle of the password:

java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters
in escape (%) pattern - For input string: "es"

Could this be due to lack of percent encoding in any GET request that
is being made?
Martin Schafran | 22 Apr 11:29 2013

wrong intermediary timestamp

hi,

I noticed that there is a wrong timestamp in the intermediary diff file when 
populating diffs from api db to simple db via a diff.osm file.

at the end it does not impact the snapshot but only the diff.osm file

steps to reproduce:

make a change at 10:00:00Z and upload to api db (josm)
api db has the correct time 10Z

make diff file:  --write-xml-change intervalBegin=10:00:00Z
diff file has the wrong time 08:00:00Z

import diff file to snapshot or simple schema
snapshot db has the correct time 10Z

my local machine has UTC+2 time.

martin
Paul Norman | 19 Apr 14:14 2013
Picon

pgsnapshot alternative query optimizations: ways using nodes

In part 1 of a N+1 part series, I'm intending to outline some of the
benchmarking I have done into different queries and indexes with a
pgsnapshot database.

Meta: Where on the wiki should this type of stuff go?

All benchmarking times are with a hot cache, but similar trends were
observed with a cold cache. I tested with two areas: the large area
-87.75,41.80,-87.65,41.85 (322k nodes, 58.1k ways) and the small area
-111.92,33.82,-111.91,33.83 (159 nodes, 30 ways)

My testing machine is a 6 core AMD 1090T with all databases on a 6 x 7200
RPM drive array, 32GB ram.

The traditional way of running a query to get all ways which contain a set
of nodes as their children is to do two joins. This type of query is used in
extracting a bbox from the database.

First you need a set of nodes. This could be generated by this query for a
bbox

CREATE TEMPORARY TABLE bbox_nodes AS 
  SELECT * FROM nodes 	
    WHERE geom &&
ST_SetSRID(ST_MakeBox2D(ST_Point(-87.75,41.80),ST_Point(-87.65,41.85)),4326)
;
	
The exact query doesn't matter much - it could be a polygon and
ST_Intersects(geom, area_geom) and everything below should still apply. I am
assuming that some kind of geometry query is used. A random list of nodes
(Continue reading)

Paul Norman | 18 Apr 09:36 2013
Picon

Wrapper scripts for streaming replication

I'm considering making use of streaming replication to feed change data to a
process, but I'm not quite sure how to script it to handle interruptions
like power outages and such correctly without getting in advance of itself.
Does anyone have any wrapper scripts that would help with this?

I'd rather not reinvent the wheel so I can focus on the logic of
interpreting the osc data and putting it into my database.

Alternately, I could re-implement a streaming replication reader in python,
which is what I intend to be writing my osc to DB code in.
Brett Henderson | 16 Apr 13:50 2013

Re: FW: question

On 15 April 2013 00:28, mahmoud ghareebh <m.soliman_2010-PkbjNfxxIARBDgjK7y7TUQ@public.gmane.org> wrote:
yes an old version , do i need to remove it?

The plugin is failing to load, so yes I'd remove it.

Again, do you really need to compile Osmosis yourself?
 



Hi Mahmoud,

Is there a reason you're not using the pre-compiled version?
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip

The errors you're receiving are unusual.  I see the following message in your logs.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot load JPF-plugin 'org.mapsforge.MapFileWriter' for extensionpoint 'Task'
Do you have some custom Osmosis plugins installed somewhere?

Brett
 
_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
Paul Norman | 16 Apr 10:31 2013
Picon

Bad PostgreSQL plans with pgsnapshot

I ran across a PostgreSQL query planner bug today when benchmarking
pgsnapshot. This may affect others so I'm documenting it.

A common task is to extract a bounding box or polygon from the database,
equivalent to a map? call.

The way I was benchmarking was a multi-step process, but there's two that
are relevant for this.

The first is getting all the nodes in the area. The query for this is

CREATE TEMPORARY TABLE bbox_nodes AS 
  SELECT * FROM nodes 
    WHERE geom &&
ST_SetSRID(ST_MakeBox2D(ST_Point(-87.75,41.80),ST_Point(-87.65,41.85)),4326)
;
ANALYZE bbox_nodes;

The next is to get the ways with any of those nodes as members.

I started this with 

SELECT DISTINCT ... FROM ways
  JOIN way_nodes ON (ways.id=way_nodes.way_id)
  JOIN bbox_nodes ON (way_nodes.node_id=bbox_nodes.id);

This was basically the query I intended to benchmark, but I was getting
horrible query plans and run times so I split the query up, arriving at
this: 

SELECT DISTINCT way_nodes.way_id
  FROM way_nodes 
  JOIN bbox_nodes ON (way_nodes.node_id=bbox_nodes.id);

I was still getting bad query plans involving a sequential scan of
way_nodes, a 2.2 billion row table.

After much help from RhodiumToad from #postgresql the cause was tracked down
to a bug in the query planner where a fixed fudge factor was accounting for
almost all of the cost of fetching the random pages. The cost estimate was
about 356 of which 348 was from the fudge factor. This was causing it to
vastly overestimate the cost of the index lookups, forcing it to do a
sequential scan.

This bug has already been fixed in 9.3
(http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;f=src/backend/u
tils/adt/selfuncs.c;h=31f38f28b00cbe2b9267205359e3cf7bafa1cb97) but not
backported to earlier versions.

The workaround was to increase cpu_tuple_cost to 0.1. This makes it use the
indexes, resulting in a 50s query for a 350k node 60k way bbox instead of a
query aborted at 10 minutes.

If you are running a query against a small bbox (50k nodes) this bug does
not matter as the incorrect fudge factor is not significant enough.

If you are running it against a very large bbox (~10-100 million nodes?) it
also does not matter, a sequential scan will be faster there.

I do not know if this bug impacts the osmosis --dataset-bounding-box task,
jxapi or pyxapi. 

I do not know if set enable_seqscan=false is an appropriate solution. If you
have an index on bbox_nodes (id) it is not as it will force index usage on
that table when it needs to fetch 100% of the table anyways.
mahmoud ghareebh | 13 Apr 12:43 2013
Picon

FW: question

hi 
i`m trying to install osmosis on windows 7 , i have jdk 6 , and i follow this instructions :
./gradlew assemble
success
Perform a complete build including unit tests:
./gradlew build
failure , and give me errors i`m attach the report , could u help me?
<!-- .ExternalClass .ecxhmmessage P { padding:0px; } .ExternalClass body.ecxhmmessage { font-size:12pt; font-family:Calibri; } -->
Attachment (tests.zip): application/zip, 36 KiB
_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
Paul Norman | 7 Apr 19:44 2013
Picon

Re: Problem with Windows 7

I find no mention of a –mapfile-writer task in the usage page. Are you using a plugin?

 

From: Malky [mailto:malky-D0VgoRl4D1jNcuzrQHMqQ9BPR1lH4CV8@public.gmane.org]
Sent: Sunday, April 07, 2013 5:48 AM
To: osmosis-dev-3+rWM/WnaLOn4i5uJCXUsti2O/JbrIOy@public.gmane.org
Subject: [osmosis-dev] Problem with Windows 7

 

When I try to create a map, I get a "Files was unexpected at this time" error!

 

I know it must be a variable error with the path name Program Files as I had something similar getting osmosis to run, but I can' figure out what or where?

 

I tried this osmosis

 

--read-xml file="C:\bulgaria.osm" --mapfile-writer file="C:\varna.map" bbox=42.185,27.063,43.463,28.103

 

before getting the error.

 

Any ideas are more than welcome as it's driving me mad.

 

Malky

_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
Malky | 7 Apr 14:47 2013

Problem with Windows 7

When I try to create a map, I get a "Files was unexpected at this time" error!

 

I know it must be a variable error with the path name Program Files as I had something similar getting osmosis to run, but I can' figure out what or where?

 

I tried this osmosis

 

--read-xml file="C:\bulgaria.osm" --mapfile-writer file="C:\varna.map" bbox=42.185,27.063,43.463,28.103

 

before getting the error.

 

Any ideas are more than welcome as it's driving me mad.

 

Malky

_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev
Brett Henderson | 5 Apr 00:51 2013

Osmosis 0.43 Released

Hi All,

I've just released Osmosis 0.43.
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.43-RELEASE.tgz
http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.43-RELEASE.zip

From changes.txt:
  • Update command line help to point to the correct wiki location.
  • Remove idTrackerType arguments to filtering tasks, only the Dynamic option is now available.
  • Fix the --write-apidb task to use 64-bit ids.
  • Upgrade to version 1.4 of the Gradle build tool.
  • Enhance the build scripts to support publishing to Sonatype OSS Repository and Maven Central.
  • Rename all projects to use an "osmosis-" prefix.
  • Included a copy of the PBF OSM Binary project in the source tree, and eliminated the pre-compiled osmbin.jar.
  • Remove pbfmarshall project due to the same functionality being provided by the osm-binary project.
  • Removed the internal Ivy repository due to all dependencies now being available on Maven Central.
  • Rename --read-empty short name --re to --rem to avoid clash with --report-entity.
  • Rename --read-empty-change short name --rec to --remc for consistency with --read-empty.

A significant change in this release is that the artefacts have also been published to Maven Central.  This will make it easier for people to include Osmosis libraries in their own projects.  One relatively minor downside is that I can no longer perform the build from the Jenkins server because I am signing builds with my personal GPG key.  The build has been performed off my own local desktop.  I don't expect this to cause any noticeable changes.
http://search.maven.org/#search|ga|1|g%3A%22org.openstreetmap.osmosis%22

Let me know if you see any issues.

Brett

_______________________________________________
osmosis-dev mailing list
osmosis-dev@...
http://lists.openstreetmap.org/listinfo/osmosis-dev

Gmane