Jonathan-David SCHRODER | 3 Feb 2010 01:09
Picon

Re: [OSM-dev] mod_tile svn's project variables 1)hardcoded 2)in many files

Hi,
for the render_expired.c program (;-) )..
maximum zoom values all equal 18 and are hardcoded
this namely also concerns the twopow[18] array

thus the following macros give a segfault if all values 18 are replaced by something > 18 ; but it works ok for values <= 18 (tested with 16)
#define TILE_REQUESTED(z,x,y) \
   (tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))]>>(((x)*twopow[z]+(y))%(8*sizeof(int))))&0x01
#define SET_TILE_REQUESTED(z,x,y) \
   tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))] |= (0x01 << (((x)*twopow[z]+(y))%(8*sizeof(int))));

this expression : tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))] alone has been tested to be enough to trigger a segfault when function pipe's parameter z=30-3=27

I apologize to not be able to dig further...

can someone help remove the segfault if all 18 values are replaced by MAX_ZOOM (<=define included from render_config.h) and MAZ_ZOOM is different from 18 ?

Thanks so much in advance !!

Jonathan-David Schröder

On Fri, Jan 22, 2010 at 5:18 PM, Lennard <ldp <at> xs4all.nl> wrote:
Jonathan-David SCHRODER wrote:

> for our project (Geopard) which does indoor & outdoor mapping, we need
> zoom levels up to 24 or so.
> mod_tile's svn's render_config.h defines a constand #define MAX_ZOOM 18
> (and also MIN_ZOOM)
> but this constant is used in only 2 or so files and the rest of max zoom
> values are hardcoded (just grep for 18 in the svn and you will find them)...

The only 2 hardcoded z18 references I could find for renderd/mod_tile
proper were in gen_tile.cpp. The other references are in the python
support scripts, none of which play a crucial role in normal renderd
operation.

Unless you use renderd.py instead of renderd, that is. There is another
hardcoded 18 in there.

I increased my renderd to z19 for haiti related tiles, and that was as
easy as changing those 2 hardcoded '18' occurences and the MAX_ZOOM
define to 19.

--
Lennard

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Frederik Ramm | 3 Feb 2010 01:45
Favicon

Re: [OSM-dev] mod_tile svn's project variables 1)hardcoded 2)in many files

Hi,

Jonathan-David SCHRODER wrote:
> for the render_expired.c program (;-) )..

Uh-oh, that's my nasty little secret I believe.

render_expired is more or less a version of render_list, but it 
processes an expiry list coming from osm2pgsql. Because osm2pgsql has no 
knowledge about meta tiles, the expire list might contain up to 64 tiles 
that fall onto the same meta tile. render_list tries to keep track of 
meta tiles already requested for rendering, to avoid requesting the same 
tile multiple times.

It does so in a bit array which is manipulated with these beastly things 
(which anyone is welcome to replace by something that is more en vogue):

> #define TILE_REQUESTED(z,x,y) \
> (tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))]>>(((x)*twopow[z]+(y))%(8*sizeof(int))))&0x01

> #define SET_TILE_REQUESTED(z,x,y) \
>    tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))] |= (0x01 << 
> (((x)*twopow[z]+(y))%(8*sizeof(int))));

You should be able to simply pre-compute the twopow array to something 
higher and it should work.

HOWEVER #1:

These macros are used only at the meta tile level, i.e. the "z" in these 
macros is always the real zoom level minus 3. If they break with z=19 
then that means that the script cannot handle tiles on zoom level 22. 
Are you *sure* you want to handle tiles on zoom level 22? On that zoom 
level, one pixel corresponds to 4 centimetres on the surface of the 
earth (at the equator). One tile corresponds to less than 100 square 
metres. You'll be rapidly approaching a situation where OSM's accuracy 
becomes a problem...

HOWEVER #2:

You will need roughly 2^(2*maxzoom-39) GB of RAM to run this program. So 
if you want to go down to zoom level 22, then that's 32 GB RAM.

For your application, you might want to use a different approach, one 
that keeps rendered tiles in a tree or list like osm2pgsql does and not 
one that makes an array like render_expired.

Bye
Frederik

--

-- 
Frederik Ramm  ##  eMail frederik <at> remote.org  ##  N49°00'09" E008°23'33"
Jonathan-David SCHRODER | 3 Feb 2010 02:16
Picon

Re: [OSM-dev] mod_tile svn's project variables 1)hardcoded 2)in many files

Hi !!

On Wed, Feb 3, 2010 at 1:45 AM, Frederik Ramm <frederik <at> remote.org> wrote:
Hi,


Jonathan-David SCHRODER wrote:
for the render_expired.c program (;-) )..

Uh-oh, that's my nasty little secret I believe.

render_expired is more or less a version of render_list, but it processes an expiry list coming from osm2pgsql. Because osm2pgsql has no knowledge about meta tiles, the expire list might contain up to 64 tiles that fall onto the same meta tile. render_list tries to keep track of meta tiles already requested for rendering, to avoid requesting the same tile multiple times.

It does so in a bit array which is manipulated with these beastly things (which anyone is welcome to replace by something that is more en vogue):

Yes I had understood the purpose of tile_requested by reading the c code and its comments.

#define TILE_REQUESTED(z,x,y) \
(tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))]>>(((x)*twopow[z]+(y))%(8*sizeof(int))))&0x01

#define SET_TILE_REQUESTED(z,x,y) \
  tile_requested[z][((x)*twopow[z]+(y))/(8*sizeof(int))] |= (0x01 << (((x)*twopow[z]+(y))%(8*sizeof(int))));

You should be able to simply pre-compute the twopow array to something higher and it should work.

HOWEVER #1:

These macros are used only at the meta tile level, i.e. the "z" in these macros is always the real zoom level minus 3. If they break with z=19 then that means that the script cannot handle tiles on zoom level 22. Are you *sure* you want to handle tiles on zoom level 22? On that zoom level, one pixel corresponds to 4 centimetres on the surface of the earth (at the equator). One tile corresponds to less than 100 square metres. You'll be rapidly approaching a situation where OSM's accuracy becomes a problem...

I want to do rendering for indoor stuff and the extent will be that of <5 buildings in height &width. So yes zoom = 22 is a zoom level I want. Even I could want more than 22, so as to see small things indoor (that depends on the project, but that could be at most a pile of books on a shelf for an amazon-like assets facility)... I'm just planning large. If I can pushing limits without doing hardcode C hacking (I'm a scripting guy), then I'm ok to try doing it.

As to precision issue, someone earlier on this list told (using the ruby code as a prof) that the rails port would allow to store vector data with integer precision up to 7cm without modifying anything to the rails port (and for JOSM I don't know).
Do you know if mapnik loses rendering precision up to some zoom level ?


HOWEVER #2:

You will need roughly 2^(2*maxzoom-39) GB of RAM to run this program. So if you want to go down to zoom level 22, then that's 32 GB RAM.
yes.... for zoommax = 17, your code states that we'd need 2GB RAM. 2GB => 32GB is a big leap... and for example an Amazon EC2 32 bits instance has 1.7 GB memory...

For your application, you might want to use a different approach, one that keeps rendered tiles in a tree or list like osm2pgsql does and not one that makes an array like render_expired.
There's an issue in using mod_tile as is for what I'm trying to achieve.
the tiles expiry itself that osm2pgsql provides doesn't consider as to which indoor level has changed (osm2pgsql would need to read level= attributes of changed tags to guess it...)
... and I'll want to do tiles compositing with mapnik xml stylesheets changed online before being handed to mapnik...

As Dane Springmeyer had warned/advised me earlier on IRC, for my indoor rendering thing, I don't want to use mod_tile (even though I could use a tree as you say instead of this 2-dimensions array... while I could just use bash's sort --unique option on osm2pgsql's expiry list file before calling render_expired)...

Now I've got to understand mod_tile and render_expired better thanks !


Bye
Frederik

--
Frederik Ramm  ##  eMail frederik <at> remote.org  ##  N49°00'09" E008°23'33"

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Frederik Ramm | 3 Feb 2010 02:52
Favicon

Re: [OSM-dev] mod_tile svn's project variables 1)hardcoded 2)in many files

Hi,

Jonathan-David SCHRODER wrote:
> As to precision issue, someone earlier on this list told (using the ruby 
> code as a prof) that the rails port would allow to store vector data 
> with integer precision up to 7cm without modifying anything to the rails 
> port (and for JOSM I don't know).

I think that's true. With one pixel being 4cm on z22, this means that 
already on z22 you have the issue that there will be some pixel 
positions where OSM doesn't let you place a node because of lack of 
precision and rounding errors. Not a big deal but at some point zooming 
in further will just give weird results ;-)

> As Dane Springmeyer had warned/advised me earlier on IRC, for my indoor 
> rendering thing, I don't want to use mod_tile (even though I could use a 
> tree as you say instead of this 2-dimensions array... while I could just 
> use bash's sort --unique option on osm2pgsql's expiry list file before 
> calling render_expired)...

No, you would have to use a little more logic than that because you want 
to replace each x and y value by int(n/8)*8 so that you only have one 
line per meta tile.

Bye
Frederik

--

-- 
Frederik Ramm  ##  eMail frederik <at> remote.org  ##  N49°00'09" E008°23'33"
Jukka Rahkonen | 4 Feb 2010 11:34
Picon

[OSM-dev] Ways and tags to separate tables?

Hi,

Osm2pgsql is fine and fast utility for importing OSM data into PostGIS, but it
is mainly made for Mapnik rendering. For that it is OK to import just a subset
of tags and make a wide schema with an own column for each tag.  However,
sometimes it would nice to get all the tags available in some database.

Has anybody made yet a tool which separates ways and tags to different tables? 
The way tables could contain just the geometries and OSM_IDs, perhaps separated
to points, lines and polygons, and tag tables would hold the tags by using
OSM_ID as a foreign key. Perhaps relations should be pushed to a fourth table
which accepts all kind of things, including geometry collections. Spatialite
would be an ideal output for me but I an not sure if is could take in very big
datasets. I have had troubles in creating bigger that 5 gigabyte spatialite
databases so perhaps PostGIS would be more safe alternative.

-Jukka Rahkonen-
Frederik Ramm | 4 Feb 2010 11:45
Favicon

Re: [OSM-dev] Ways and tags to separate tables?

Hi,

Jukka Rahkonen wrote:
> Has anybody made yet a tool which separates ways and tags to different tables?

Yes, Brett Henderson has, it's called Osmosis ;-) Osmosis can generate a 
database schema that has all the data plus geometry columns.

Bye
Frederik
Gravatar

Re: [OSM-dev] Ways and tags to separate tables?

you could run a tagscanner tool like I have posted elsewhere and then
just generate a stylesheet.

It would be possible to modify the osm2pgsql tool, I have patches for
it as well. Right now I am a bit busy, but when I have nothing to do;
could look into it.

mike

On Thu, Feb 4, 2010 at 11:34 AM, Jukka Rahkonen
<jukka.rahkonen <at> mmmtike.fi> wrote:
> Hi,
>
> Osm2pgsql is fine and fast utility for importing OSM data into PostGIS, but it
> is mainly made for Mapnik rendering. For that it is OK to import just a subset
> of tags and make a wide schema with an own column for each tag.  However,
> sometimes it would nice to get all the tags available in some database.
>
> Has anybody made yet a tool which separates ways and tags to different tables?
> The way tables could contain just the geometries and OSM_IDs, perhaps separated
> to points, lines and polygons, and tag tables would hold the tags by using
> OSM_ID as a foreign key. Perhaps relations should be pushed to a fourth table
> which accepts all kind of things, including geometry collections. Spatialite
> would be an ideal output for me but I an not sure if is could take in very big
> datasets. I have had troubles in creating bigger that 5 gigabyte spatialite
> databases so perhaps PostGIS would be more safe alternative.
>
> -Jukka Rahkonen-
>
>
> _______________________________________________
> dev mailing list
> dev <at> openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
Jukka Rahkonen | 4 Feb 2010 12:00
Picon

Re: [OSM-dev] Ways and tags to separate tables?

Frederik Ramm <frederik <at> remote.org> writes:

> Hi,
> 
> Jukka Rahkonen wrote:
> > Has anybody made yet a tool which separates ways and tags to different
> > tables?
> 
> Yes, Brett Henderson has, it's called Osmosis  Osmosis can generate a 
> database schema that has all the data plus geometry columns.

Hi,

Apologies, I had thought that Osmosis is just for OSM specialists. Does it
really create spatial tables which I can use as simply as the ones created by
osm2pgsql with QGis, uDig or OpenJUMP, and which I can use for rendering with
Mapnik, Mapserver, Geoserver etc.?

-Jukka-
Emilie Laffray | 4 Feb 2010 16:00
Picon

Re: [OSM-dev] Ways and tags to separate tables?



On 4 February 2010 11:00, Jukka Rahkonen <jukka.rahkonen <at> mmmtike.fi> wrote:
Frederik Ramm <frederik <at> remote.org> writes:

> Hi,
>
> Jukka Rahkonen wrote:
> > Has anybody made yet a tool which separates ways and tags to different
> > tables?
>
> Yes, Brett Henderson has, it's called Osmosis  Osmosis can generate a
> database schema that has all the data plus geometry columns.

Hi,

Apologies, I had thought that Osmosis is just for OSM specialists. Does it
really create spatial tables which I can use as simply as the ones created by
osm2pgsql with QGis, uDig or OpenJUMP, and which I can use for rendering with
Mapnik, Mapserver, Geoserver etc.?


The only thing that Osmosis doesn't do is polygons. Points and ways are rendered as POINT and LINESTRING respectively.

Emilie Laffray
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Christian H. Bruhn | 5 Feb 2010 21:55
Picon
Favicon

Re: [OSM-dev] Problems with own Mapnik-installation

Hello Jon,

Wednesday, January 27, 2010, 9:49:21 PM, you wrote:

> You don't really need a new copy of the executable, the default.style
> rules file gets updated fairly frequently and can be downloaded from:

> http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/default.style?format=raw

Finally, everything works. Only if I use the generate_tiles.py with a
NUM_THREADS greater than 1 python crashes after a few seconds. (I have
an Intel I7 so that at least 4 should work.) There will be a window
"phyton.exe" that only says "python.exe doesn't work anymore". The
number of connections of the database is -1, I also tried 2 or more.

How often are boundaries and coastlines
(world_boundaries-spherical.tgz, processed_p.tar.bz2,
shoreline_300.tar.bz2) updated? How can I see from which date they
are?

Christian

Gmane