McCalla, Mac | 3 Nov 19:50 2010

DRMAA use case from Hess Corp.

Hi Peter,

Sorry it has taken so long, but corporate approval was required to "publish" any kind of public statement.  You may place this statement as you like in the usage stories section of your website.

Statement for release:

---------------------------------------------------------------------

Hess Corporation: Use case for DRMAA.


Hess Corporation is a global integrated energy company engaged in the exploration and production of crude oil and natural gas, as well as the refining and marketing of petroleum products, natural gas and electricity.

As part of our exploration activities, seismic data is acquired and processed to create images of the earth's subsurface which can be analyzed by geoscientists for indications of hydrocarbon deposits.

In order to process the seismic data for a particular area, which can be several terabytes in size for a single seismic survey, we use a cluster based supercomputing facility, composed of several thousand CPU and GPU processors.  To help manage the tens of thousands of individual application tasks created as part of the data processing and image generation workflow, we use a commercially supported grid engine software package. 

This grid engine software provides a "C-Programming language callable" DRMAA library which is used by our application front-end to submit the batch job streams which contain the processing and image generation tasks to the compute cluster for execution.

---------------------------------------------------------------------


Best regards,

Mac McCalla
Geoscience Systems Development Advisor
Office - 713.609.5434
Cell - 832.202.4262
E-mail - macmccalla-A/OqKpFVqZs@public.gmane.org


This e-mail and any attachments are for the sole use of the intended recipient(s) and may contain information that is confidential. If you are not the intended recipient(s) and have received this e-mail in error, please immediately notify the sender by return e-mail and delete this e-mail from your computer. Any distribution, disclosure or the taking of any other action by anyone other than the intended recipient(s) is strictly prohibited.
<div>

<p>Hi Peter,
</p>

<p>Sorry it has taken so long, but corporate approval was required to "publish" any kind of public statement.&nbsp; You may place this statement as you like in the usage stories section of your website.</p>

<p>Statement for release:
</p>

<p>---------------------------------------------------------------------
</p>

<p>Hess Corporation: Use case for DRMAA.
</p>
<br><p>Hess Corporation is a global integrated energy company engaged in the exploration and production of crude oil and natural gas, as well as the refining and marketing of petroleum products, natural gas and electricity.</p>

<p>As part of our exploration activities, seismic data is acquired and processed to create images of the earth's subsurface which can be analyzed by geoscientists for indications of hydrocarbon deposits.</p>

<p>In order to process the seismic data for a particular area, which can be several terabytes in size for a single seismic survey, we use a cluster based supercomputing facility, composed of several thousand CPU and GPU processors.&nbsp; To help manage the tens of thousands of individual application tasks created as part of the data processing and image generation workflow, we use a commercially supported grid engine software package.&nbsp; </p>

<p>This grid engine software provides a "C-Programming language callable" DRMAA library which is used by our application front-end to submit the batch job streams which contain the processing and image generation tasks to the compute cluster for execution.</p>

<p>---------------------------------------------------------------------
</p>
<br><p>Best regards,
</p>

<p>Mac McCalla

<br>Geoscience Systems Development Advisor

<br>Office - 713.609.5434 

<br>Cell - 832.202.4262

<br>E-mail - macmccalla@...
</p>

<br>
This e-mail and any attachments are for the sole use of the intended recipient(s) and may contain information that is confidential.  If you are not the intended recipient(s) and have received this e-mail in error, please immediately notify the sender by return e-mail and delete this e-mail from your computer. Any distribution, disclosure or the taking of any other action by anyone other than the intended recipient(s) is strictly prohibited.<br>
</div>
Peter Tröger | 3 Nov 20:54 2010
Picon

Reminder: Phone calls are suspended

Dear all,

just a short reminder: The bi-weekly phone calls are suspended, since  
we currently have to focus on document finalization. If you expected a  
call today, please try instead to contribute to the document text:

http://wikis.sun.com/display/DRMAAv2/Home

Thanks,
Peter.

Peter Tröger | 4 Nov 12:01 2010
Picon

Re: DRMAA use case from Hess Corp.

Hi Mac,

thanks for the text, I will put it on our drmaa.org home page.

Best,
Peter.

Am 03.11.2010 um 19:50 schrieb McCalla, Mac:

Hi Peter,

Sorry it has taken so long, but corporate approval was required to "publish" any kind of public statement.  You may place this statement as you like in the usage stories section of your website.

Statement for release:

---------------------------------------------------------------------

Hess Corporation: Use case for DRMAA.


Hess Corporation is a global integrated energy company engaged in the exploration and production of crude oil and natural gas, as well as the refining and marketing of petroleum products, natural gas and electricity.

As part of our exploration activities, seismic data is acquired and processed to create images of the earth's subsurface which can be analyzed by geoscientists for indications of hydrocarbon deposits.

In order to process the seismic data for a particular area, which can be several terabytes in size for a single seismic survey, we use a cluster based supercomputing facility, composed of several thousand CPU and GPU processors.  To help manage the tens of thousands of individual application tasks created as part of the data processing and image generation workflow, we use a commercially supported grid engine software package. 

This grid engine software provides a "C-Programming language callable" DRMAA library which is used by our application front-end to submit the batch job streams which contain the processing and image generation tasks to the compute cluster for execution.

---------------------------------------------------------------------


Best regards,

Mac McCalla
Geoscience Systems Development Advisor
Office - 713.609.5434
Cell - 832.202.4262
E-mail - macmccalla-A/OqKpFVqZs@public.gmane.org


This e-mail and any attachments are for the sole use of the intended recipient(s) and may contain information that is confidential. If you are not the intended recipient(s) and have received this e-mail in error, please immediately notify the sender by return e-mail and delete this e-mail from your computer. Any distribution, disclosure or the taking of any other action by anyone other than the intended recipient(s) is strictly prohibited.

<div>Hi Mac,<div><br></div>
<div>thanks for the text, I will put it on our <a href="http://drmaa.org">drmaa.org</a> home page.</div>
<div><br></div>
<div>Best,</div>
<div>Peter.</div>
<div>
<br><div>
<div>Am 03.11.2010 um 19:50 schrieb McCalla, Mac:</div>
<br class="Apple-interchange-newline"><blockquote type="cite">
<div>
<p>Hi Peter,
</p>
<p>Sorry it has taken so long, but corporate approval was required to "publish" any kind of public statement.&nbsp; You may place this statement as you like in the usage stories section of your website.</p>
<p>Statement for release:
</p>
<p>---------------------------------------------------------------------
</p>
<p>Hess Corporation: Use case for DRMAA.
</p>
<br><p>Hess Corporation is a global integrated energy company engaged in the exploration and production of crude oil and natural gas, as well as the refining and marketing of petroleum products, natural gas and electricity.</p>
<p>As part of our exploration activities, seismic data is acquired and processed to create images of the earth's subsurface which can be analyzed by geoscientists for indications of hydrocarbon deposits.</p>
<p>In order to process the seismic data for a particular area, which can be several terabytes in size for a single seismic survey, we use a cluster based supercomputing facility, composed of several thousand CPU and GPU processors.&nbsp; To help manage the tens of thousands of individual application tasks created as part of the data processing and image generation workflow, we use a commercially supported grid engine software package.&nbsp; </p>
<p>This grid engine software provides a "C-Programming language callable" DRMAA library which is used by our application front-end to submit the batch job streams which contain the processing and image generation tasks to the compute cluster for execution.</p>
<p>---------------------------------------------------------------------
</p>
<br><p>Best regards,
</p>
<p>Mac McCalla

<br>Geoscience Systems Development Advisor

<br>Office - 713.609.5434 

<br>Cell - 832.202.4262

<br>E-mail - <a href="mailto:macmccalla@...">macmccalla@...</a>
</p>

<br>
This e-mail and any attachments are for the sole use of the intended recipient(s) and may contain information that is confidential.  If you are not the intended recipient(s) and have received this e-mail in error, please immediately notify the sender by return e-mail and delete this e-mail from your computer. Any distribution, disclosure or the taking of any other action by anyone other than the intended recipient(s) is strictly prohibited.<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
Daniel Gruber | 8 Nov 10:55 2010
Picon

MonitoringSession

Hi, 

in the MonitorinSession we have on machine level machineSockets and coresPerSocket. 
To be consequent we should also add threadsPerCore. At least OGE/SGE does 
support this. I added it into our spreadsheet. 
If this is not supported by a DRM/OS it could return 0 as value for unknown.

0 for coresPerSocket and machineSockets is not allowed since we should 
define coresPerSocket*machineSockets=="processors" in case a DRM or OS 
does not support this kind of architectural information. I suggest to leave
it open for the DRMAA implementation if it maps the "processors" information
to coresPerSocket or machineSockets in case of missing architectural 
details.

If there is no objection I'll take this as accepted. 

Cheers

Daniel
Andre Merzky | 8 Nov 14:44 2010

Re: MonitoringSession

+1

On Mon, Nov 8, 2010 at 10:55 AM, Daniel Gruber
<daniel.x.gruber@...> wrote:
> Hi,
>
> in the MonitorinSession we have on machine level machineSockets and coresPerSocket.
> To be consequent we should also add threadsPerCore. At least OGE/SGE does
> support this. I added it into our spreadsheet.
> If this is not supported by a DRM/OS it could return 0 as value for unknown.
>
> 0 for coresPerSocket and machineSockets is not allowed since we should
> define coresPerSocket*machineSockets=="processors" in case a DRM or OS
> does not support this kind of architectural information. I suggest to leave
> it open for the DRMAA implementation if it maps the "processors" information
> to coresPerSocket or machineSockets in case of missing architectural
> details.
>
> If there is no objection I'll take this as accepted.
>
> Cheers
>
> Daniel
> --
>  drmaa-wg mailing list
>  drmaa-wg@...
>  http://www.ogf.org/mailman/listinfo/drmaa-wg
>

--

-- 
Nothing is ever easy...
Peter Tröger | 8 Nov 15:14 2010
Picon

Re: MonitoringSession

Hi,

I can agree to the new "threadsPerCore" attribute, but would prefer to have "1" as default value. From our
understanding of a core, each one can always execute at least one thread. It would also allow to compute an
estimation of the number of parallel threads, without looking on the specific numbers.

Best,
Peter.

Am 08.11.2010 um 10:55 schrieb Daniel Gruber:

> Hi, 
> 
> in the MonitorinSession we have on machine level machineSockets and coresPerSocket. 
> To be consequent we should also add threadsPerCore. At least OGE/SGE does 
> support this. I added it into our spreadsheet. 
> If this is not supported by a DRM/OS it could return 0 as value for unknown.
> 
> 0 for coresPerSocket and machineSockets is not allowed since we should 
> define coresPerSocket*machineSockets=="processors" in case a DRM or OS 
> does not support this kind of architectural information. I suggest to leave
> it open for the DRMAA implementation if it maps the "processors" information
> to coresPerSocket or machineSockets in case of missing architectural 
> details.
> 
> If there is no objection I'll take this as accepted. 
> 
> Cheers
> 
> Daniel
> --
>  drmaa-wg mailing list
>  drmaa-wg@...
>  http://www.ogf.org/mailman/listinfo/drmaa-wg

Daniel Gruber | 8 Nov 15:27 2010
Picon

Re: MonitoringSession

Ok. For simplicity we take 1 as default value with the 
drawback that we loose information if the SMT value 
is available (and correct) or not. 

Regards, 

Daniel

On 11/08/10 15:14, Peter Tröger wrote:
> Hi,
> 
> I can agree to the new "threadsPerCore" attribute, but would prefer to have "1" as default value. From our
understanding of a core, each one can always execute at least one thread. It would also allow to compute an
estimation of the number of parallel threads, without looking on the specific numbers.
> 
> Best,
> Peter.
> 
> 
> Am 08.11.2010 um 10:55 schrieb Daniel Gruber:
> 
>> Hi, 
>>
>> in the MonitorinSession we have on machine level machineSockets and coresPerSocket. 
>> To be consequent we should also add threadsPerCore. At least OGE/SGE does 
>> support this. I added it into our spreadsheet. 
>> If this is not supported by a DRM/OS it could return 0 as value for unknown.
>>
>> 0 for coresPerSocket and machineSockets is not allowed since we should 
>> define coresPerSocket*machineSockets=="processors" in case a DRM or OS 
>> does not support this kind of architectural information. I suggest to leave
>> it open for the DRMAA implementation if it maps the "processors" information
>> to coresPerSocket or machineSockets in case of missing architectural 
>> details.
>>
>> If there is no objection I'll take this as accepted. 
>>
>> Cheers
>>
>> Daniel
>> --
>>  drmaa-wg mailing list
>>  drmaa-wg@...
>>  http://www.ogf.org/mailman/listinfo/drmaa-wg
> 

Mariusz Mamoński | 8 Nov 16:01 2010
Picon

Re: MonitoringSession

Hi all,

 If we are talking about the monitoring session... what do you think
about the idea of:

 1. creating a new data struct MachineInfo with all the predefined
machine attributes (e.g.: threadsPerCore) +  "readonly attribute
Dictionary drmsSpecific;" (an extension point) and providing one
method: "MachineInfo getMachineInfo(in string machineName) for
accessing all of them
 2. adding a new attribute "slotsCount", which denotes the maximum
number of single-process jobs that can run on given machine
concurrently (use case: system administrator may either choose
configuration where one process runs per physical core or hardware
thread or choose choose any number that is totally independent from
hardware configuration)

Cheers,

2010/11/8 Daniel Gruber <daniel.x.gruber <at> oracle.com>:
> Ok. For simplicity we take 1 as default value with the
> drawback that we loose information if the SMT value
> is available (and correct) or not.
>
> Regards,
>
> Daniel
>
> On 11/08/10 15:14, Peter Tröger wrote:
>> Hi,
>>
>> I can agree to the new "threadsPerCore" attribute, but would prefer to have "1" as default value. From our
understanding of a core, each one can always execute at least one thread. It would also allow to compute an
estimation of the number of parallel threads, without looking on the specific numbers.
>>
>> Best,
>> Peter.
>>
>>
>> Am 08.11.2010 um 10:55 schrieb Daniel Gruber:
>>
>>> Hi,
>>>
>>> in the MonitorinSession we have on machine level machineSockets and coresPerSocket.
>>> To be consequent we should also add threadsPerCore. At least OGE/SGE does
>>> support this. I added it into our spreadsheet.
>>> If this is not supported by a DRM/OS it could return 0 as value for unknown.
>>>
>>> 0 for coresPerSocket and machineSockets is not allowed since we should
>>> define coresPerSocket*machineSockets=="processors" in case a DRM or OS
>>> does not support this kind of architectural information. I suggest to leave
>>> it open for the DRMAA implementation if it maps the "processors" information
>>> to coresPerSocket or machineSockets in case of missing architectural
>>> details.
>>>
>>> If there is no objection I'll take this as accepted.
>>>
>>> Cheers
>>>
>>> Daniel
>>> --
>>>  drmaa-wg mailing list
>>>  drmaa-wg <at> ogf.org
>>>  http://www.ogf.org/mailman/listinfo/drmaa-wg
>>
>
> --
>  drmaa-wg mailing list
>  drmaa-wg <at> ogf.org
>  http://www.ogf.org/mailman/listinfo/drmaa-wg
>

--

-- 
Mariusz
--
  drmaa-wg mailing list
  drmaa-wg <at> ogf.org
  http://www.ogf.org/mailman/listinfo/drmaa-wg
Daniel Templeton | 8 Nov 16:05 2010
Picon

Re: MonitoringSession

The slotsCount attribute would be challenging for something like OGE, 
where the number of slots available on a machine is taken from the sum 
of the slots in all the enabled queue instances on that machine modulo 
the active resource quota sets modulo any host-level slots settings 
modulo the queue subordination rules.  It's also not very meaningful, 
because many of those things can and do change dynamically.

Daniel

On 11/ 8/10 07:01 AM, Mariusz Mamoński wrote:
> Hi all,
>
>   If we are talking about the monitoring session... what do you think
> about the idea of:
>
>   1. creating a new data struct MachineInfo with all the predefined
> machine attributes (e.g.: threadsPerCore) +  "readonly attribute
> Dictionary drmsSpecific;" (an extension point) and providing one
> method: "MachineInfo getMachineInfo(in string machineName) for
> accessing all of them
>   2. adding a new attribute "slotsCount", which denotes the maximum
> number of single-process jobs that can run on given machine
> concurrently (use case: system administrator may either choose
> configuration where one process runs per physical core or hardware
> thread or choose choose any number that is totally independent from
> hardware configuration)
>
> Cheers,
>
>
> 2010/11/8 Daniel Gruber<daniel.x.gruber <at> oracle.com>:
>> Ok. For simplicity we take 1 as default value with the
>> drawback that we loose information if the SMT value
>> is available (and correct) or not.
>>
>> Regards,
>>
>> Daniel
>>
>> On 11/08/10 15:14, Peter Tröger wrote:
>>> Hi,
>>>
>>> I can agree to the new "threadsPerCore" attribute, but would prefer to have "1" as default value. From
our understanding of a core, each one can always execute at least one thread. It would also allow to compute
an estimation of the number of parallel threads, without looking on the specific numbers.
>>>
>>> Best,
>>> Peter.
>>>
>>>
>>> Am 08.11.2010 um 10:55 schrieb Daniel Gruber:
>>>
>>>> Hi,
>>>>
>>>> in the MonitorinSession we have on machine level machineSockets and coresPerSocket.
>>>> To be consequent we should also add threadsPerCore. At least OGE/SGE does
>>>> support this. I added it into our spreadsheet.
>>>> If this is not supported by a DRM/OS it could return 0 as value for unknown.
>>>>
>>>> 0 for coresPerSocket and machineSockets is not allowed since we should
>>>> define coresPerSocket*machineSockets=="processors" in case a DRM or OS
>>>> does not support this kind of architectural information. I suggest to leave
>>>> it open for the DRMAA implementation if it maps the "processors" information
>>>> to coresPerSocket or machineSockets in case of missing architectural
>>>> details.
>>>>
>>>> If there is no objection I'll take this as accepted.
>>>>
>>>> Cheers
>>>>
>>>> Daniel
>>>> --
>>>>   drmaa-wg mailing list
>>>>   drmaa-wg <at> ogf.org
>>>>   http://www.ogf.org/mailman/listinfo/drmaa-wg
>>>
>>
>> --
>>   drmaa-wg mailing list
>>   drmaa-wg <at> ogf.org
>>   http://www.ogf.org/mailman/listinfo/drmaa-wg
>>
>
>
>
--
  drmaa-wg mailing list
  drmaa-wg <at> ogf.org
  http://www.ogf.org/mailman/listinfo/drmaa-wg
Daniel Gruber | 8 Nov 16:12 2010
Picon

Re: MonitoringSession

Adding slotsCount to machineInfo would destroy the separation 
between queue level and host level information. Different 
users could have different slotCount on the same machine. 
The information must be retrieved on queue level (we have 
the queueMaxSlotsAllowed()) method for that. 

Cheers

Daniel

On 11/08/10 16:01, Mariusz Mamoński wrote:
> Hi all,
> 
>  If we are talking about the monitoring session... what do you think
> about the idea of:
> 
>  1. creating a new data struct MachineInfo with all the predefined
> machine attributes (e.g.: threadsPerCore) +  "readonly attribute
> Dictionary drmsSpecific;" (an extension point) and providing one
> method: "MachineInfo getMachineInfo(in string machineName) for
> accessing all of them
>  2. adding a new attribute "slotsCount", which denotes the maximum
> number of single-process jobs that can run on given machine
> concurrently (use case: system administrator may either choose
> configuration where one process runs per physical core or hardware
> thread or choose choose any number that is totally independent from
> hardware configuration)
> 
> Cheers,
> 
> 
> 2010/11/8 Daniel Gruber <daniel.x.gruber <at> oracle.com>:
>> Ok. For simplicity we take 1 as default value with the
>> drawback that we loose information if the SMT value
>> is available (and correct) or not.
>>
>> Regards,
>>
>> Daniel
>>
>> On 11/08/10 15:14, Peter Tröger wrote:
>>> Hi,
>>>
>>> I can agree to the new "threadsPerCore" attribute, but would prefer to have "1" as default value. From
our understanding of a core, each one can always execute at least one thread. It would also allow to compute
an estimation of the number of parallel threads, without looking on the specific numbers.
>>>
>>> Best,
>>> Peter.
>>>
>>>
>>> Am 08.11.2010 um 10:55 schrieb Daniel Gruber:
>>>
>>>> Hi,
>>>>
>>>> in the MonitorinSession we have on machine level machineSockets and coresPerSocket.
>>>> To be consequent we should also add threadsPerCore. At least OGE/SGE does
>>>> support this. I added it into our spreadsheet.
>>>> If this is not supported by a DRM/OS it could return 0 as value for unknown.
>>>>
>>>> 0 for coresPerSocket and machineSockets is not allowed since we should
>>>> define coresPerSocket*machineSockets=="processors" in case a DRM or OS
>>>> does not support this kind of architectural information. I suggest to leave
>>>> it open for the DRMAA implementation if it maps the "processors" information
>>>> to coresPerSocket or machineSockets in case of missing architectural
>>>> details.
>>>>
>>>> If there is no objection I'll take this as accepted.
>>>>
>>>> Cheers
>>>>
>>>> Daniel
>>>> --
>>>>  drmaa-wg mailing list
>>>>  drmaa-wg <at> ogf.org
>>>>  http://www.ogf.org/mailman/listinfo/drmaa-wg
>> --
>>  drmaa-wg mailing list
>>  drmaa-wg <at> ogf.org
>>  http://www.ogf.org/mailman/listinfo/drmaa-wg
>>
> 
> 
> 

--
  drmaa-wg mailing list
  drmaa-wg <at> ogf.org
  http://www.ogf.org/mailman/listinfo/drmaa-wg

Gmane