headers
Mitchell Wisidagamage | 1 Feb 2007 02:33
Picon

Re: massive parallel processing application required


> is this "e-science" term more popular where you are?  I don't really
> hear it here (Canada, probably NA in general.)  many (most) branches of 
> science are so dependent on computers that it seems redundant and archaic.
> 

"e-science" is huge here, in universities and scientific research . It's
associated with grid computing in the science area.

>     - how long will it transfer this 50GB blob of data?
> 
>     - if I provide neurophysiology data from my lab, can I
>     be sued if the subjects' privacy is violated?
> 
>     - do I have time to spend explaining how the data is encoded?
> 

and
  - hassle of going up the ladder to get permission from bosses
  - explain the formulas and how to process the data.
Permalink | Reply | 11 comments |
headers
Mitchell Wisidagamage | 1 Feb 2007 02:51
Picon

Re: massive parallel processing application required

lenoxx <at> tempemusic.com wrote:
> You can try:
> CMAQ
> www.cmascenter.org
> MM5
> http://www.mmm.ucar.edu/mm5/
> WRF
> http://www.wrf-model.org/index.php
> 
> 
> Any of the above could be used and input data can be gathered from many 
> sources. If you want some WRF inputs, let me know. I don't currently 
> have any CMAQ or MM5 inputs though.
> 
> 
> 
Thank you for the links. Wonder how I didn't find them before. I spend 
lots of time searching for sites like that.

Now I'm not sure what to do with these data sets. I should program my 
own application. But how should I be processing them?...without the 
algorithms for processing I'm lost.  :o)
Permalink | Reply | 3 comments |
headers
Gerry Creager | 1 Feb 2007 03:44
Picon
Favicon

Re: massive parallel processing application required

No... "e-Science" has been co-opted by "grid" as their own. 
Collaborative computational science has been around for quite some time. 
  Determining the appropriate degree of distributed computational 
interprocess communication requires more than trivial examination of the 
codes in use.

Please don't fall into the trap of thinking "e-Science" requires a tie 
to the Globus Toolkit to be valid.

gerry

Mitchell Wisidagamage wrote:
> 
>> is this "e-science" term more popular where you are?  I don't really
>> hear it here (Canada, probably NA in general.)  many (most) branches 
>> of science are so dependent on computers that it seems redundant and 
>> archaic.
>>
> 
> "e-science" is huge here, in universities and scientific research . It's
> associated with grid computing in the science area.
> 
> 
>>     - how long will it transfer this 50GB blob of data?
>>
>>     - if I provide neurophysiology data from my lab, can I
>>     be sued if the subjects' privacy is violated?
>>
>>     - do I have time to spend explaining how the data is encoded?
>>
(Continue reading)

Permalink | Reply | 10 comments |
headers
Gerry Creager | 1 Feb 2007 03:49
Picon
Favicon

Re: massive parallel processing application required

Don't forget the Weather Research and  Forecasting model at 
http://wrf-model.org/users/users.php

Mitchell Wisidagamage wrote:
> lenoxx <at> tempemusic.com wrote:
>> You can try:
>> CMAQ
>> www.cmascenter.org
>> MM5
>> http://www.mmm.ucar.edu/mm5/
>> WRF
>> http://www.wrf-model.org/index.php
>>
>>
>> Any of the above could be used and input data can be gathered from 
>> many sources. If you want some WRF inputs, let me know. I don't 
>> currently have any CMAQ or MM5 inputs though.
>>
>>
>>
> Thank you for the links. Wonder how I didn't find them before. I spend 
> lots of time searching for sites like that.
> 
> Now I'm not sure what to do with these data sets. I should program my 
> own application. But how should I be processing them?...without the 
> algorithms for processing I'm lost.  :o)
> 
> 
> 
> _______________________________________________
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jim Lux | 1 Feb 2007 06:18
Picon
Picon
Favicon

Re: massive parallel processing application required

At 03:17 AM 1/31/2007, Mitchell Wisidagamage wrote:
>Thank you very much for the fire dynamics idea. I will have a look at it.
>
>I did try to contact many e-science projects including some 
>researchers at Oxford. But I got no reply. Then I went to get some 
>contacts from a tutor who worked at a e-science project himself. He 
>told me people, especially scientists are "very jealous" of their 
>data. And not replying is a kind way of saying "no". And there's the 
>problem of "who's this guy wanting my data", "what will he do with it?".
>
>I have given up the e-science idea. Now looking for other real world 
>applications.

Optimum path routing of ships and/or airplanes, taking into account 
the winds, currents, sea state, temperatures, etc.

Large realtime and climatological databases are available.
The path optimization algorithms are simple and fairly well known (A 
and A-star are two to start with).  The challenge is in suitable 
heuristics to prune the search space.

You can optimize for minimum time in transit, or minimum fuel cost, 
or minimum probability of delay, etc.

You can burn a lot of compute cycles even doing a fairly simple route 
(say, Los Angeles to Yokohama by ship or New York to Los Angeles by 
air), because the search space is quite dense (probably don't want to 
change course too often, but that's still hundreds of waypoints).. 
and then, after you've found the route, you should (either by looking 
at what you calculated during route finding, or as a post process
(Continue reading)

Permalink | Reply | 1 comment |
headers
Jim Lux | 1 Feb 2007 06:29
Picon
Picon
Favicon

Re: massive parallel processing application required

At 02:03 PM 1/31/2007, Robert G. Brown wrote:
>On Wed, 31 Jan 2007, Mitchell Wisidagamage wrote:
>
>>Thank you very much for the fire dynamics idea. I will have a look at it.
>>
>>I did try to contact many e-science projects including some 
>>researchers at Oxford. But I got no reply. Then I went to get some 
>>contacts from a tutor who worked at a e-science project himself. He 
>>told me people, especially scientists are "very jealous" of their 
>>data. And not replying is a kind way of saying "no". And there's 
>>the problem of "who's this guy wanting my data", "what will he do with it?".
>>
>>I have given up the e-science idea. Now looking for other real 
>>world applications.
>
>Remember, NASA puts all (or at least a lot) of its e.g. weather data
>online.

Well.. not exactly NASA.. operational "weather" data is the province 
of NOAA.  NASA does research, not operational, data, so there's 
typically a time lag, especially for processed and calibrated data.

By and large, most environmental data collected by NASA winds up in 
DAACs (Distributed Active Archiving Centers). Physical Oceanography 
data, for instance, winds up at PO-DAAC... 
http://www-podaac.jpl.nasa.gov/ which has data for sea surface 
temperature, sea surface topography, and ocean vector winds acquired 
by NASA instruments.  This whole process is very well documented, and 
the data moves through the various levels of processing and into the 
archives in a regular and stately fashion.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Jim Lux | 1 Feb 2007 06:37
Picon
Picon
Favicon

Re: massive parallel processing application required

At 02:37 PM 1/31/2007, Mark Hahn wrote:
>>worked at a e-science project himself. He told me people, especially
>
>is this "e-science" term more popular where you are?  I don't really
>hear it here (Canada, probably NA in general.)  many (most) branches 
>of science are so dependent on computers that it seems redundant and archaic.
>
>>scientists are "very jealous" of their data. And not replying is a 
>>kind way of saying "no". And there's the problem of "who's this guy 
>>wanting my data", "what will he do with it?".
>
>sure - academia is all about publishing, so you don't want someone 
>else to scoop you.

This is an interesting aspect.  All the latest Announcements of 
Opportunity for space research (think cameras taking pictures of 
Mars, or analyzing rocks etc.) have fairly fast time lines(weeks, not 
months or years) for relase of data to general public, and you have 
to put your plans and budgets for public distribution in the 
proposal.  No more holding onto the data, "recalibrating and 
reprocessing",  until everyone gets their dissertations done.

The Dead Sea Scrolls are probably a pretty notorious example of 
"we're still working on it.. when we're done, we'll release 
it"...Amazing what a computer and a concordance can do.

>   but often concerns are much more mundane, like:
>
>         - how long will it transfer this 50GB blob of data?
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jim Lux | 1 Feb 2007 06:45
Picon
Picon
Favicon

Re: massive parallel processing application required

At 04:17 PM 1/29/2007, Mitchell Wisidagamage wrote:
>Hi all,
>  As part of my dissertation, I'm looking for "raw data" which will 
> be used for massive parallel processing using Beuwulf cluster (with 
> the use of PVM or MPI). I tried looking for e-science raw data (and 
> the computations required on it) such as bioinformatics, fluid 
> dynamics, etc. but without any luck.

An interesting computationally intensive task would be constructing a 
3-D model of a pot from multiple 2-d images of pot shards, especially 
if there are missing or extra pieces.

This would be quite a boon for archaeologists, who currently do it by 
hand, as a 3D jigsaw puzzle.

And, of course, it has almost no commercial value.. there are 
commercial archaeologists, but they don't get paid to reassemble broken pots.

A similar application might be to take a 3D tomographic image of a 
block of tar from the La Brea tarpits and extract all the bone 
images, and attempt to match them up.

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jim Lux | 1 Feb 2007 06:40
Picon
Picon
Favicon

Re: massive parallel processing application required

At 05:51 PM 1/31/2007, Mitchell Wisidagamage wrote:
lenoxx <at> tempemusic.com wrote:
You can try:
CMAQ
www.cmascenter.org
MM5
http://www.mmm.ucar.edu/mm5/
WRF
http://www.wrf-model.org/index.php

Any of the above could be used and input data can be gathered from many sources. If you want some WRF inputs, let me know. I don't currently have any CMAQ or MM5 inputs though.

Thank you for the links. Wonder how I didn't find them before. I spend lots of time searching for sites like that.

Now I'm not sure what to do with these data sets. I should program my own application. But how should I be processing them?...without the algorithms for processing I'm lost.  :o)


http://www.ocean-systems.com/VOSS.htm
www.weather.navy.mil/paoweb/starsams.ppt
http://realdistance.com/





_______________________________________________
Beowulf mailing list, Beowulf <at> beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875
<div>
At 05:51 PM 1/31/2007, Mitchell Wisidagamage wrote:<br><blockquote type="cite" class="cite" cite="">lenoxx <at> tempemusic.com
wrote:<br><blockquote type="cite" class="cite" cite="">You can try:<br>
CMAQ<br><a href="http://www.cmascenter.org" eudora="autourl">
www.cmascenter.org</a><br>
MM5<br><a href="http://www.mmm.ucar.edu/mm5/" eudora="autourl">
http://www.mmm.ucar.edu/mm5/</a><br>
WRF<br><a href="http://www.wrf-model.org/index.php" eudora="autourl">
http://www.wrf-model.org/index.php</a><br><br>
Any of the above could be used and input data can be gathered from many
sources. If you want some WRF inputs, let me know. I don't currently have
any CMAQ or MM5 inputs though.<br><br>
</blockquote>Thank you for the links. Wonder how I didn't find them
before. I spend lots of time searching for sites like that.<br><br>
Now I'm not sure what to do with these data sets. I should program my own
application. But how should I be processing them?...without the
algorithms for processing I'm lost.&nbsp; :o)</blockquote>
<br><br><a href="http://www.ocean-systems.com/VOSS.htm" eudora="autourl">
http://www.ocean-systems.com/VOSS.htm</a><br>
<a href="http://www.weather.navy.mil/paoweb/starsams.ppt" eudora="autourl">
www.weather.navy.mil/paoweb/starsams.ppt</a> <br><a href="http://realdistance.com/" eudora="autourl">
http://realdistance.com/<br><br><br><br><br><br></a><blockquote type="cite" class="cite" cite="">
_______________________________________________<br>
Beowulf mailing list, Beowulf <at> beowulf.org<br>
To change your subscription (digest mode or unsubscribe) visit
<a href="http://www.beowulf.org/mailman/listinfo/beowulf" eudora="autourl">
http://www.beowulf.org/mailman/listinfo/beowulf</a>
</blockquote>
<p></p>
James Lux, P.E.<br>
Spacecraft Radio Frequency Subsystems Group<br>
Flight Communications Systems Section<br>
Jet Propulsion Laboratory, Mail Stop 161-213<br>
4800 Oak Grove Drive<br>
Pasadena CA 91109<br>
tel: (818)354-2075<br>
fax: (818)393-6875</div>
Permalink | Reply | 1 comment |
headers
Steve Heaton | 1 Feb 2007 07:29
Picon
Picon

Re: massive parallel processing application, required

G'day Jim and all

Two interesting space examples would be the original Viking lander pics. 
The story goes that they were in such a rush to get the images to the 
press conference that they 'made the sky blue and the ground red'.

Later, more careful analysis showed the sky a more dusty pink/brown and 
the surface a truer brown (stronger purple component).

At the other end of the scale, the general public 'scooped' the Huygen's 
team by processing the raw landing approach images using the likes of 
Photoshop and had them on the Web in advance of the official release.

I'm sure, Jim, you likely have more inside details than I ;)

On the topic of met models, if anyone's looking for some good 'toy' code 
to start from, have a look at PUMA:
http://www.mi.uni-hamburg.de/PUMA.215.0.html

I recommend GrADS for the viz: http://www.iges.org/grads/

The PSU/NCAR MM5 code scares the willys out of me! ;)

(There *is* a train of thought here... I've used PUMA many moons ago to 
build my own Martian and Titan models. A certain perverse pleasure in 
watching your model shake itself to pieces!)

Cheers
Stevo

=====

Date: Wed, 31 Jan 2007 21:37:23 -0800
From: Jim Lux <James.P.Lux <at> jpl.nasa.gov>
Subject: Re: [Beowulf] massive parallel processing application
	required
To: Mark Hahn <hahn <at> mcmaster.ca>,	Mitchell Wisidagamage
	<06002352 <at> brookes.ac.uk>
Cc: beowulf <at> beowulf.org
Message-ID: <6.2.3.4.2.20070131213007.0307aa18 <at> mail.jpl.nasa.gov>
Content-Type: text/plain; charset="us-ascii"; format=flowed

At 02:37 PM 1/31/2007, Mark Hahn wrote:

 > >sure - academia is all about publishing, so you don't want someone
 > >else to scoop you.

This is an interesting aspect.  All the latest Announcements of
Opportunity for space research (think cameras taking pictures of
Mars, or analyzing rocks etc.) have fairly fast time lines(weeks, not
months or years) for relase of data to general public, and you have
to put your plans and budgets for public distribution in the
proposal.  No more holding onto the data, "recalibrating and
reprocessing",  until everyone gets their dissertations done.
Permalink | Reply | 0 comments |

Gmane