RE: [GRASS-user] 64 bit or parallel processing
Roger Bivand <Roger.Bivand <at> nhh.no>
2007-03-01 09:56:57 GMT
By the way, there is a forthcoming paper in Computers and Geosciences at:
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V7D-4MY0TX9-8&_user=10&_coverDate=01%2F30%2F2007&_alid=543285067&_rdoc=1&_fmt=summary&_orig=search&_cdi=5840&_sort=d&_docanchor=&view=c&_ct=6&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=898ebe84b51937827e5e0b86a348340c
(we only see the abstract without paying):
Implementation of a parallel high-performance visualization technique in
GRASS GIS by Alexandre Sorokine - the scripts are at:
http://www.ornl.gov/sci/gist/software/grass/
The approach involves tiling a region for visualization on a big wall from
a head node to rendering nodes.
Roger
On Wed, 28 Feb 2007, Jonathan Greenberg wrote:
> I'm now working at UCD on a project with > 65 (!) hyperspectral images, and
> a LOT of the slow downs stem from I/O intensive processes -- with that said,
> there are plenty of "standard" raster processing routines that start
> becoming processor intensive (e.g. linear spectral unmixing) that, with a
> decent i/o setup (we have a lot of cheap-o 2-drive RAID0s that we back up to
> a permanent location, this nearly doubles our I/O speed). Not having any
> idea of how grass's raster access backbone functions, it does seem you could
> have a generic tiling routine that could be called from all of the raster
> commands that would be highly parallizable (I note that RSI ENVI does this
> out of the box now, it detects the # of processors and simply starts forking
> a single image across all of the processors, I assume one line/tile per
> processor). Mapcalc and related processes would be a good start, and keep
> in mind *spatial* analyses (e.g. texture windows) as you design it.
>
> My suggestion to the GRASS programmers: if you decide this is useful, I'd
> HIGHLY recommend trying to do this in an out-of-the-box approach, e.g.
> having an install-time "mp" capability -- if it requires being a) experts in
> raster processing, b) experts in GRASS and c) experts in parallel coding,
> you'll end up with 3 people doing SMP :)
>
> --j
>
> -----Original Message-----
> From: grassuser-bounces <at> grass.itc.it [mailto:grassuser-bounces <at> grass.itc.it]
> On Behalf Of Hamish
> Sent: Wednesday, February 28, 2007 4:14 PM
> To: Glynn Clements
> Cc: grassuser <at> grass.itc.it
> Subject: Re: [GRASS-user] 64 bit or parallel processing
>
> Glynn Clements wrote:
> > It would be more feasible to modify individual modules; e.g. a
> > multi-threaded version of r.mapcalc might be feasible. But unless the
> > raster I/O code was made thread-safe, you would still be limited by
> > the rate at which map data could be read and written by a single
> > thread, so it would only be worthwhile for cases where the bulk of the
> > overhead was in the actual calculations.
>
> Right, so working on individual modules which can typically take 10+
> hours to run (*.rst, watersheds, viewsheds (incl. r.sun), indices from
> RS data, etc) seems like a much better focus of effort for the greatest
> gain:work ratio. And as seen in the MPI add-ons this is exactly where
> the previous ad-hoc effort has gone.
>
> IIRC Helena said that the RST modules have changed substantially sice
> GRASS 5 and the s.surf.rst.mpi work would have to be largely redone.
>
> Any thoughts on gains that could be made by MPI'ing the segmentation
> lib? Do do modules using that usually do so for memory needs not
> processing speed?
>
>
> Hamish
>
> _______________________________________________
> grassuser mailing list
> grassuser <at> grass.itc.it
> http://grass.itc.it/mailman/listinfo/grassuser
>
>
> _______________________________________________
> grassuser mailing list
> grassuser <at> grass.itc.it
> http://grass.itc.it/mailman/listinfo/grassuser
>
--
--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand <at> nhh.no
_______________________________________________
grassuser mailing list
grassuser <at> grass.itc.it
http://grass.itc.it/mailman/listinfo/grassuser
RSS Feed