headers
Daniel Behr | 1 Aug 2011 11:16
Picon

[OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

Hello,
my first email to the OSM list and I hope someone had this problem 
before (and solved it). I have OSM Europe data in postgres 8.4.7/postgis 
1.5.2 and render tiles with mapnik0.7.1. this actually works fine, until 
I apply the hourly diffs from 
http://planet.openstreetmap.org/hour-replicate in a bunch once per day 
with osmosis [1]. It usually takes about 4-5 hours to finish. After a 
few days of updates I get errors in the rendered images. It seems to 
affect only line features (roads, borders, railways) in a way that 
random gaps appear [2]. I've also seen that some linenodes jumped for 
several hundred kilometers [3].

I already re-imported europe.osm and its still the same after some days 
of updates.
I dont necessary need the hourly diffs, I just dont want to fall months 
behind with my map. So if there is no easy fix for this problem, I would 
rather re-import europe.osm monthly or bi-monthly and render a new set 
of tiles then. This means I would have to cache all 20 zoomlevels to be 
able to provide tiles while doing the re-import, which would be a bit of 
a drawback... (level 17 still rendering after 4 days. 18? 19?)

I am no postgres ace and I only tried reindexing tables and vacuum DB, 
without any result. I dont even know if the problem is with osmosis or 
the database setup (or other)?

will be happy about any pointers.

cheers
Daniel
(Continue reading)

Permalink | Reply | 6 comments |
headers
Peter Körner | 1 Aug 2011 11:49
Picon

Re: [OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

Am 01.08.2011 11:16, schrieb Daniel Behr:
> osmosis --rri workingDirectory=/var/lib/pgsql/.osmosis --simc
> --write-xml-change "-" |\
> /opt/osm2pgsql/osm2pgsql -s -a -b "-3500000,3720000,5080000,11940000"\
> --tablespace-main-data europe --tablespace-main-index europe
> --tablespace-slim-data europe --tablespace-slim-index europe\
> -d osm -C 4096 -S /opt/osm2pgsql/default.style -

Have you tried it without the bbox argument? I'm not sure if bbox works 
with updates, because of problems when nodes get moved out (or into) the 
bbox.

Peter
Permalink | Reply | 3 comments |
headers
Daniel Behr | 1 Aug 2011 12:00
Picon

Re: [OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

thanks Peter.
nope, not yet. I thought when I set the BBOX larger than my dataset this 
would be no problem and I read it somewhere that it takes that argument 
for updates, too.

I'll try that on the next run, but how does it behave outside the BBOX 
then? It doesnt matter if the map gets messed up there, but what about 
the database size, doesnt it increase drastically?

Daniel

Am 01.08.2011 11:49, schrieb Peter Körner:
> Am 01.08.2011 11:16, schrieb Daniel Behr:
>> osmosis --rri workingDirectory=/var/lib/pgsql/.osmosis --simc
>> --write-xml-change "-" |\
>> /opt/osm2pgsql/osm2pgsql -s -a -b "-3500000,3720000,5080000,11940000"\
>> --tablespace-main-data europe --tablespace-main-index europe
>> --tablespace-slim-data europe --tablespace-slim-index europe\
>> -d osm -C 4096 -S /opt/osm2pgsql/default.style -
>
> Have you tried it without the bbox argument? I'm not sure if bbox works
> with updates, because of problems when nodes get moved out (or into) the
> bbox.
>
> Peter

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
(Continue reading)

Permalink | Reply | 2 comments |
headers
Peter Körner | 1 Aug 2011 12:03
Picon

Re: [OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

Am 01.08.2011 12:00, schrieb Daniel Behr:
> thanks Peter.
> nope, not yet. I thought when I set the BBOX larger than my dataset this
> would be no problem and I read it somewhere that it takes that argument
> for updates, too.
>
> I'll try that on the next run, but how does it behave outside the BBOX
> then? It doesnt matter if the map gets messed up there, but what about
> the database size, doesnt it increase drastically?

I think so, but it would help to know if this is the cause of your 
problem. Maybe you just have to make the BBOX bigger..

Some people try to reduce the size by running a DELETE once a week just 
before the VACUUM.

Peter
Permalink | Reply | 1 comment |
headers
Jon Burgess | 1 Aug 2011 12:06
Picon

Re: [OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

On Mon, 2011-08-01 at 11:16 +0200, Daniel Behr wrote:
> Hello,
> my first email to the OSM list and I hope someone had this problem 
> before (and solved it). I have OSM Europe data in postgres 8.4.7/postgis 
> 1.5.2 and render tiles with mapnik0.7.1. this actually works fine, until 
> I apply the hourly diffs from 
> http://planet.openstreetmap.org/hour-replicate in a bunch once per day 
> with osmosis [1]. It usually takes about 4-5 hours to finish. After a 
> few days of updates I get errors in the rendered images. It seems to 
> affect only line features (roads, borders, railways) in a way that 
> random gaps appear [2]. I've also seen that some linenodes jumped for 
> several hundred kilometers [3].

> [1]
> osmosis --rri workingDirectory=/var/lib/pgsql/.osmosis --simc 
> --write-xml-change "-" |\
>   /opt/osm2pgsql/osm2pgsql -s -a -b "-3500000,3720000,5080000,11940000"\
>   --tablespace-main-data europe --tablespace-main-index europe 
> --tablespace-slim-data europe --tablespace-slim-index europe\
>   -d osm -C 4096 -S /opt/osm2pgsql/default.style -

The bounding box should be specified in degrees. It looks like you have
used the projected Mercator co-ordinate. With your current settings you
will be dropping all nodes in the diffs since none will be in the
latitude range specified. 

   Jon
Permalink | Reply | 1 comment |
headers
Daniel Behr | 1 Aug 2011 12:29
Picon

Re: [OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

thanks Jon and Peter

okay, that might well be the problem. I catched it somewhere using 
projected coords in meters for BBOX is okay, but osm2pgsql's help says 
its not. my bad.
Maybe I should even use it the other way round; give the BBOX param to 
osmosis and not to osm2pgsql?! Will try it with different setups, but I 
guess I already ruined my DB again with the last 6 updates :(

Daniel

Am 01.08.2011 12:06, schrieb Jon Burgess:
> On Mon, 2011-08-01 at 11:16 +0200, Daniel Behr wrote:
>> Hello,
>> my first email to the OSM list and I hope someone had this problem
>> before (and solved it). I have OSM Europe data in postgres 8.4.7/postgis
>> 1.5.2 and render tiles with mapnik0.7.1. this actually works fine, until
>> I apply the hourly diffs from
>> http://planet.openstreetmap.org/hour-replicate in a bunch once per day
>> with osmosis [1]. It usually takes about 4-5 hours to finish. After a
>> few days of updates I get errors in the rendered images. It seems to
>> affect only line features (roads, borders, railways) in a way that
>> random gaps appear [2]. I've also seen that some linenodes jumped for
>> several hundred kilometers [3].
>
>> [1]
>> osmosis --rri workingDirectory=/var/lib/pgsql/.osmosis --simc
>> --write-xml-change "-" |\
>>    /opt/osm2pgsql/osm2pgsql -s -a -b "-3500000,3720000,5080000,11940000"\
>>    --tablespace-main-data europe --tablespace-main-index europe
(Continue reading)

Permalink | Reply | 0 comments |
headers
Stephan Knauss | 1 Aug 2011 12:59
Picon

Re: [OSM-dev] applying hourly OSM diffs with osmosis gradually messes up database

Peter Körner writes: 

> Some people try to reduce the size by running a DELETE once a week just 
> before the VACUUM.

problem is that ways and relations don't have a geometry. So osm2pgsql 
would need to do asdditional lookups when processing a bounding box. 

I thought about fixing it but found that a regular cleanup is a lot cheaper. 

See here:
http://wiki.openstreetmap.org/wiki/User:Stephankn/knowledgebase#Cleanup_of_w
ays_outside_the_bounding_box 

Stephan

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Permalink | Reply | 0 comments |
headers
Jason Lee | 1 Aug 2011 14:03
Picon

[OSM-dev] Mod_Tile render and cache based on URL value (source x style combo)

Hi,
I am trying to work out if I can get Mod_Tile to dynamically render and cache maps dynamically based on parameter values read from the incoming URL tile request.

I know that it's possible to increase the no. of layers (XML stylesheet configs) from the default max of 10 by updating the setting in "render_config.h" and recompiling. But what I would like is the ability to "mix and match" map sources and map styles.

A simple example, I have 10 thematic maps of an area, one for property value bands (PROP) in shades of Red, another for crime rates (CRIM) in shades of Blue, another for pollution index (POLL) in shades of Green etc. So, initially I have 10 layers with their styles and data sources. I then want Mod_Tile to render the CRIM source in the same style as the PROP layer (Shades of Red instead of Blue) and any of the other 9 existing defined styles.

If I wanted the ability to render every combination as a defined layer (XML file), I would end up with 100 (10x10) stylesheets which is a bit too much. It would be better to read in a "Style" parameter like "myMap.html?source=PROP&style=CRIM", so that Mod_Tile could recognize the "Style" paramter and access the PROP.xml stylesheet and apply the Red styles BUT select data from the CRIM table source instead of PROP (some inconsistency relating to normalising values between different sources, but this is just an example).

This would also mean multiple folders for caching of different combinations PROP/CRIM, PROP/POLL, CRIM/PROP... etc.

I'd like to know if this is easily possible? 
Any help or advice much appreciated. Thanks.

Jason
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Permalink | Reply | 2 comments |
headers
Andy Allan | 2 Aug 2011 20:18
Picon
Gravatar

Re: [OSM-dev] Mod_Tile render and cache based on URL value (source x style combo)

On Mon, Aug 1, 2011 at 1:03 PM, Jason Lee <jaslee.st <at> gmail.com> wrote:

> I'd like to know if this is easily possible?

It's not easily possible with the current way that mod_tile works.

> Any help or advice much appreciated. Thanks.

I'd suggest having a look into TileStache and making a custom
"Provider" that will read your URL parameters and twiddle things
appropriately. I'm not sure if there's already a provider that works
in the same way, but if not it would be useful to set up a working
example for it!

http://tilestache.org

Cheers,
Andy
Permalink | Reply | 1 comment |
headers
Mayeul Kauffmann | 2 Aug 2011 20:43
Picon
Favicon

Re: [OSM-dev] Which OSM to PostGreSQL converter shall I use?

Hi,
(sorry for late reply)
You can use osm2postgresql (different from osm2pgsql) and run pgrouting
on top of it. See:
http://wiki.openstreetmap.org/wiki/Osm2postgresql

It has built-in reprojection. If you need a simpler tool or sql-only
code, you can look at the osm2postgresql code (bash file), you will find
a lot of examples.
Regards,
Mayeul

Le lundi 11 juillet 2011 à 09:01 -0700, jchen a écrit :
> Hello,
> 
> I need a postgresql database to do following jobs:
> 1. A lot of shortest path tree constructions given an root node.
> 2. A huge amount of queries for length of roads. ( any suggestion for
> converting the coordinates to Cartesian system?)
> 3. A few queries for nearby nodes given a point.
> 
> Actually, I need these jobs for my map matching application.
> 
> I tried to use osm2pgsql to generate a database. But it seems not
> simple/efficient to accomplish job 1. I haven't tried others. And I hope
> that I can get some useful information here.
> 
> Thank you in advance!
> 
> Best regards.
> 
> Jingmin
> 
> --
> View this message in context: http://gis.638310.n2.nabble.com/Which-OSM-to-PostGreSQL-converter-shall-I-use-tp6571476p6571476.html
> Sent from the Developer Discussion mailing list archive at Nabble.com.
> 
> _______________________________________________
> dev mailing list
> dev <at> openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Permalink | Reply | 1 comment |

Gmane