Paul Norman | 23 Jul 03:51 2014
Picon

[OSM-dev] Upcoming openstreetmap-carto changes

v2.17.0 of the openstreetmap-carto stylesheet has been released, though 
not yet deployed on tile.osm.org.

Significant changes include

* Rendering other shop values with a generic icon 
(https://github.com/gravitystorm/openstreetmap-carto/pull/604)
* Rendering wider road shields, and converting the road shields to SVGs 
(https://github.com/gravitystorm/openstreetmap-carto/pull/694)
* Cleaning up path rendering on low zooms 
(https://github.com/gravitystorm/openstreetmap-carto/pull/747)
* Improving highway=track name rendering 
(https://github.com/gravitystorm/openstreetmap-carto/pull/748)
* Rendering access=permissive the same as access=yes (or no access) 
(https://github.com/gravitystorm/openstreetmap-carto/pull/742)
* Waterway cleanups 
(https://github.com/gravitystorm/openstreetmap-carto/pull/722, and others)
* Assorted code cleanups

https://github.com/gravitystorm/openstreetmap-carto/compare/v2.16.0...v2.17.0 
has a full list of commits.

We are still very much cleaning up a lot of messy CartoCSS brought over 
from the Mapnik XML

Some interesting stuff under discussion

* Multi-line road shield rendering: 
https://github.com/gravitystorm/openstreetmap-carto/pull/750
* Rendering ref from route relationships: 
(Continue reading)

Christian Quest | 20 Jul 12:10 2014
Picon

[OSM-dev] recursion added to render_list

I've talked about this idea with some of you at last SOTM and SOTM-EU.

The idea is to have render_list taking advantage of disk cache by rendering tiles in a more "pyramidal" way than the actual full horizontal.

1) better use of disk cache to reduce disk I/O needs

Up to now, render_list renders a tile at zoom N but when getting to zoom N+1, most of the needed data are usually the same but not in the disk cache anymore causing a lot of I/O when rendering several zoom level.

With my changes, render_list immediately enqueue the 4 underlying N+1 tiles (at the top of the queue). This allows to reduce disk I/O by taking advantage of the caches still containing most of the needed data.

That's fine but where should be stop recursion ?


2- pre-render only what's worth prerendering...

For this I've added a rendering time limit.

If a metatile takes less than this limit, it is not worth pre-rendering it as it will be quickly generated if requested.
This allows to limit recursion and at the same time pre-render what can be done fast enough upon request.


To use this new feature, I'v only added on option to render_list to control the rendering time limit (in ms). With it, its behavior is unchanged.


Example:
render_list -m osm -a -f -z 9 -Z 9 --max-time 5000


On my todo list:
- when rendering several zoom levels (example: -z 9 -Z 12), check if a metatile has not been already recursively rendered by comparing its timestamp to the render_list start time

- add a maximum zoom limit (for the moment recursion can go down to zoom 18)

My commit is on github, comments are welcome... before my first mod_tile pull request

--
Christian Quest - OpenStreetMap France
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Paul Norman | 17 Jul 17:03 2014
Picon

Re: [OSM-dev] Tuning render_list: beware of hyperthreading and RAID cards latency...


On 2014-07-17 6:39 AM, Christian Quest wrote:
I'm currently doing some rendering benchmark to tune a new tile server we're about to put online.

A strange thing I've found last night: render_list with 16 thread takes more time than with 8 .

The server has 2 quad-core Xeon (X5570), this means 8 physical cores. It looks like hyperthreading adds additionnal context switches or something similar.
With "-z 9 -Z 9 -n 16 -l 32" it takes 3016s, and with "-z 9 -Z 9 -n 8 -l 12" 2647s, 10% less.

I'll disable HT on my next run in the server bios and see if it is consistent.

Is there a chance you're running into IO bottlenecks?

Another thing I found... do not put your SSD behind a RAID card.
The RAID card (Dell Perc 6i) add too much latency.
On the same render_list, the iowait threads are increasing a lot and iops decreases a lot.
The Perc 6 is an older lower-end RAID card using a LSI 10xx chip. It doesn't surprise me it doesn't work ideally with SSDs. I'm also not sure if it supports all the SATA commands a SSD uses, which can be a serious issue for consistency, the bane of SSDs.

A more modern card would probably perform better, but if you're not RAIDing SSDs, you should really be using any card you have in HBA mode, unless you have a battery-backed cache and can therefor tune write barriers and related settings better.
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Christian Quest | 17 Jul 15:39 2014
Picon

[OSM-dev] Tuning render_list: beware of hyperthreading and RAID cards latency...

I'm currently doing some rendering benchmark to tune a new tile server we're about to put online.

A strange thing I've found last night: render_list with 16 thread takes more time than with 8 .

The server has 2 quad-core Xeon (X5570), this means 8 physical cores. It looks like hyperthreading adds additionnal context switches or something similar.
With "-z 9 -Z 9 -n 16 -l 32" it takes 3016s, and with "-z 9 -Z 9 -n 8 -l 12" 2647s, 10% less.

I'll disable HT on my next run in the server bios and see if it is consistent.


Another thing I found... do not put your SSD behind a RAID card.
The RAID card (Dell Perc 6i) add too much latency.
On the same render_list, the iowait threads are increasing a lot and iops decreases a lot.

Here is a munin graph clearly showing it: http://cl.ly/image/3V2A1J261o30

Before midnight, the SSD (Samsung 840 EVO 1TB) is on a PCIe/SATA adapter (Apricorn Velocity Solo x1). After 1:00 it moved to the builtin SAS/SATA enclosures driven by the RAID card.

The server is a Dell R710 with 48GB RAM running ubuntu server 14.04.
I'm using mapnik 2.3.x with async postgis queries enabled.

--
Christian Quest - OpenStreetMap France
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Ilya Zverev | 15 Jul 08:59 2014
Picon

[OSM-dev] MapBox imagery in the offset database

Hi! Recently someone changed MapBox Imagery URL in JOSM, which resulted 
in different keys for the same imagery in imagery offset database. This 
prompted me to create a distinct id for mapbox imagery. The regular 
expression for matching URL is 
"tiles\.mapbox\.com/v[3-9]/openstreetmap\.map". All mapbox imagery 
offsets in the database were converted to use this id, JOSM plugin from 
this moment also uses it. If you actively use offsets plugin, please 
update it now to access your old stored offsets.

I really hope this is the same layer, with same offsets. If not, please 
tell me earlier, so I can undo the database change.

If you maintain a different database of imagery, please update it 
accordingly.

IZ
Grant Slater | 7 Jul 18:44 2014

[OSM-dev] Full notes export (alpha)

Hi OSM Devs,

A test OSM notes dump has been created:
http://grant.dev.openstreetmap.org/tmp/planet-notes-dump-testing/

The dump contains all notes (open/closed) + all comments. The XML
format will likely change during this alpha phase. Once finalised the
weekly dumps/exports will be made available on
http://planet.openstreetmap.org/

Feedback / comments / patches:
https://github.com/iandees/planet-notes-dump

Many thanks to Ian for writing the code.

Kind regards
Grant
Part of the OSM sysadmin team.
Nick Whitelegg | 7 Jul 13:27 2014
Picon

[OSM-dev] Using osmdroid for an Android university course

Hi,

(not sure if this belongs in talk or dev)

Next year I am running a university course on Android development.

I am planning to include a section on location-aware apps and mapping, and naturally I want to use OSM. ;-)

Mapsforge is one option, however IMV the new 0.4 API has a little too much complexity for what will be complete newcomers to Android development (they have Java experience though).

So I'd like to use osmdroid, however I'm aware that apps that use raster tile sources can place a strain on tileservers.

It looks like osmdroid will now talk to the MapQuest tileservers though.

Will it be acceptable - particularly if I use MapQuest - to use osmdroid for this course? There will be 20 students max.

Thanks,
Nick
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Stephan Knauss | 6 Jul 23:15 2014
Picon

[OSM-dev] strange error message from mod_tile

Hi,

I've set up a new machine with a mod_tile / Tirex rendering stack.

I see some strange errors in the log file which I don't understand. Any 
help would be appreciated.

I can reproduce it when requesting a tile image considered dirty.
The rendering itself looks fine. For example loading the tile as /dirty 
will trigger a rendering which is visible in tirex-status.

Requesting the image as image will not trigger a rendering. It produces 
strange log lines.

Here are the log lines (shortened for readability):

[tile:debug] ./src/mod_tile.c(1381): tile_translate: op(tile_serve) 
xml(osm) mime(image/png) z(14) x(12612) y(7238)
[tile:info] tile_storage_hook: handler(tile_serve), 
uri(/osm/14/12612/7238.png)
[tile:debug] ./src/mod_tile.c(365): tile_state: determined state of osm 
12612 7238 14 on store 7fbe01e40a60: Tile size: 7124, expired: 1 
created: 1404246259
[tile:debug] ./src/mod_tile.c(166): Connecting to renderd on Unix socket 
/var/lib/tirex/modtile.sock
22:03:21.814464 [tile:info] Requesting style(osm) z(14) x(12612) y(7238) 
from renderer with priority 7
22:03:21.815113 [tile:warn] request_tile: Failed to read response from 
rendering socket No such file or directory

So mod_tile was able to open the socket.
The select() call returned immediately.

But then an error is reported.

Can I trust errno in this situation? I only know that recv() did not 
return the expected number of bytes.

Having tirex-master running in debug mode also gives no clue what's 
happening. I see the request being received.
tirex-master[32229]: Listening for commands on socket 
/var/run/tirex/master.sock
tirex-master[32229]: Listening for mod_tile connections on 
/var/lib/tirex/modtile.sock (UNIX)
tirex-master[32229]: Listening for backend responses
tirex-master[32229]: connection from mod_tile accepted
tirex-master[32229]: read request from mod_tile: ver=2 cmd=7 x=12612 
y=7238 z=14 map=osm

Stephan
Andreas Goss | 4 Jul 20:30 2014
Picon

[OSM-dev] Wiki Pageview Stats

Not sure if this is the right place to ask, but are there and stats 
available for the page views of the OSM Wiki?

Found this: http://wiki.openstreetmap.org/wiki/Stats "showing the access 
statistics etc for the wiki pages."

But this Link is dead: 
http://stats.wiki.openstreetmap.org/awstats/awstats.pl?config=wiki.openstreetmap.org
__________
openstreetmap.org/user/AndiG88
wiki.openstreetmap.org/wiki/User:AndiG88‎

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Michael Maier | 4 Jul 13:09 2014
Picon

[OSM-dev] Changesets

Hello list,

a friend of mine pointed me to the number of changesets increasing
extraordinary in the last days (from ~14k to more than 100k), I ran a
little investigation on the daily/hourly diffs:

I filtered out changeset numbers in the osc files, sorted and unique'd
them. The actual number of changesets appearing in the diffs did not
increase. But there were a lot of numbers missing in the diffs, so there
must be a lot of empty changesets.

And I found a user with a LOT of empty changesets:
http://www.openstreetmap.org/user/ettlohw

A Mapper registered a month ago, but with 607,556 changesets - and
increasing.
When picking a random changeset of this user, it shows to be empty:
http://www.openstreetmap.org/changeset/23430671
(changeset tags: created_by=Potlatch 2, build=2.3-605-gbec0ea7, version=2.3)

Is this a bug in potlatch?
Why does the API accept empty changesets?

Should I contact the user what the hell he is doing?

Best regards,
Michael

--

-- 
Michael Maier, Student of Telematics  <at>  Graz University of Technology
OpenStreetMap Graz                         http://osm.org/go/0Iz <at> paV
http://wiki.osm.org/Graz
http://wiki.osm.org/Graz/Stammtisch

_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
Josh Huffman | 3 Jul 04:36 2014
Picon

[OSM-dev] Contributing to OSM/Overpass

Hi all, 

I am doing an independent study as part of my Computer Science degree at the University I attend and am interested in contributing to OpenStreeMap.  For the last month I have been researching different aspects of OSM and getting familiar with all the working parts.  I am particularly interested in working on clickable POI's and ran across the Overpass API.  My question is in your opinion (those of you involved with Overpass) where would be a good place to start as far as jumping into source code?  I browsed through the github page and didn't see a lot of commented up code, is there some more documentation somewhere?  I have read through the wiki.  Anyhow, I'm wanting to code in C++ and get some practice with a larger project such as this, so any advice would be greatly appreciated.  Thanks.

-Josh
_______________________________________________
dev mailing list
dev <at> openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Gmane