Mike Driscoll | 2 Mar 22:26 2010
Picon

Getting Peak Commit from NtQuerySystemInformation

Hi,

I was tasked with finding a way to query our workstations for their Peak 
Commit Charge. According to the sysinternals forum:

"The only way I'm aware of that one can get this detail is from the 
uMmPeakCommitLimit member of the SYSTEM_PERFORMANCE_INFORMATION 
structure one passes to NtQuerySystemInformation when calling it with 
the SystemPerformanceInformation type." (see 
http://forum.sysinternals.com/forum_posts.asp?TID=15540&PID=75852)

I started with this question on the PyWin32 mailing list, but it turns 
out that they don't expose the functionality inside 
NtQuerySystemInformation so that's why I came here. Unfortunately, I'm a 
complete newb when it comes to ctypes. While I did find that this worked 
"ctypes.windll.ntdll.NtQuerySystemInformation", I don't know how to go 
on and actually query the object. Can someone give me a clue?

I'm running Python 2.5 on Windows XP. Thanks!

- Mike

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
Mark Tolonen | 3 Mar 10:02 2010
Picon

Re: Getting Peak Commit from NtQuerySystemInformation


"Mike Driscoll" <mdriscoll <at> co.marshall.ia.us> wrote in message 
news:4B8D8276.6000308 <at> co.marshall.ia.us...
> Hi,
>
> I was tasked with finding a way to query our workstations for their Peak
> Commit Charge. According to the sysinternals forum:
>
> "The only way I'm aware of that one can get this detail is from the
> uMmPeakCommitLimit member of the SYSTEM_PERFORMANCE_INFORMATION
> structure one passes to NtQuerySystemInformation when calling it with
> the SystemPerformanceInformation type." (see
> http://forum.sysinternals.com/forum_posts.asp?TID=15540&PID=75852)
>
> I started with this question on the PyWin32 mailing list, but it turns
> out that they don't expose the functionality inside
> NtQuerySystemInformation so that's why I came here. Unfortunately, I'm a
> complete newb when it comes to ctypes. While I did find that this worked
> "ctypes.windll.ntdll.NtQuerySystemInformation", I don't know how to go
> on and actually query the object. Can someone give me a clue?
>
> I'm running Python 2.5 on Windows XP. Thanks!
>
> - Mike

REALLY quick and dirty...

Notes:

   2 is the enumeration value of SystemPerformanceInformation out of 
(Continue reading)

Thomas Heller | 3 Mar 10:14 2010

Re: Getting Peak Commit from NtQuerySystemInformation

Mike Driscoll schrieb:
> Hi,
> 
> I was tasked with finding a way to query our workstations for their Peak 
> Commit Charge. According to the sysinternals forum:
> 
> "The only way I'm aware of that one can get this detail is from the 
> uMmPeakCommitLimit member of the SYSTEM_PERFORMANCE_INFORMATION 
> structure one passes to NtQuerySystemInformation when calling it with 
> the SystemPerformanceInformation type." (see 
> http://forum.sysinternals.com/forum_posts.asp?TID=15540&PID=75852)
> 
> I started with this question on the PyWin32 mailing list, but it turns 
> out that they don't expose the functionality inside 
> NtQuerySystemInformation so that's why I came here. Unfortunately, I'm a 
> complete newb when it comes to ctypes. While I did find that this worked 
> "ctypes.windll.ntdll.NtQuerySystemInformation", I don't know how to go 
> on and actually query the object. Can someone give me a clue?
> 
> I'm running Python 2.5 on Windows XP. Thanks!

Not really tested, but seems to work:

<snip>
from ctypes import *

SystemBasicInformation = 0
SystemPerformanceInformation = 2

class SYSTEM_BASIC_INFORMATION(Structure):
(Continue reading)

Mike Driscoll | 3 Mar 15:13 2010
Picon

Re: Getting Peak Commit from NtQuerySystemInformation

Hi Mark,

On 3/3/2010 3:02 AM, Mark Tolonen wrote:
> "Mike Driscoll"<mdriscoll <at> co.marshall.ia.us>  wrote in message
> news:4B8D8276.6000308 <at> co.marshall.ia.us...
>    
>> Hi,
>>
>> I was tasked with finding a way to query our workstations for their Peak
>> Commit Charge. According to the sysinternals forum:
>>
>> "The only way I'm aware of that one can get this detail is from the
>> uMmPeakCommitLimit member of the SYSTEM_PERFORMANCE_INFORMATION
>> structure one passes to NtQuerySystemInformation when calling it with
>> the SystemPerformanceInformation type." (see
>> http://forum.sysinternals.com/forum_posts.asp?TID=15540&PID=75852)
>>
>> I started with this question on the PyWin32 mailing list, but it turns
>> out that they don't expose the functionality inside
>> NtQuerySystemInformation so that's why I came here. Unfortunately, I'm a
>> complete newb when it comes to ctypes. While I did find that this worked
>> "ctypes.windll.ntdll.NtQuerySystemInformation", I don't know how to go
>> on and actually query the object. Can someone give me a clue?
>>
>> I'm running Python 2.5 on Windows XP. Thanks!
>>
>> - Mike
>>      
> REALLY quick and dirty...
>
(Continue reading)

Mike Driscoll | 3 Mar 15:16 2010
Picon

Re: Getting Peak Commit from NtQuerySystemInformation

Hi Thomas,

On 3/3/2010 3:14 AM, Thomas Heller wrote:
> Mike Driscoll schrieb:
>    
>> Hi,
>>
>> I was tasked with finding a way to query our workstations for their Peak
>> Commit Charge. According to the sysinternals forum:
>>
>> "The only way I'm aware of that one can get this detail is from the
>> uMmPeakCommitLimit member of the SYSTEM_PERFORMANCE_INFORMATION
>> structure one passes to NtQuerySystemInformation when calling it with
>> the SystemPerformanceInformation type." (see
>> http://forum.sysinternals.com/forum_posts.asp?TID=15540&PID=75852)
>>
>> I started with this question on the PyWin32 mailing list, but it turns
>> out that they don't expose the functionality inside
>> NtQuerySystemInformation so that's why I came here. Unfortunately, I'm a
>> complete newb when it comes to ctypes. While I did find that this worked
>> "ctypes.windll.ntdll.NtQuerySystemInformation", I don't know how to go
>> on and actually query the object. Can someone give me a clue?
>>
>> I'm running Python 2.5 on Windows XP. Thanks!
>>      
> Not really tested, but seems to work:
>
> <snip>
> from ctypes import *
>
(Continue reading)

Thomas Heller | 4 Mar 08:22 2010

Re: Getting Peak Commit from NtQuerySystemInformation

Mike Driscoll schrieb:
> Hi Thomas,
> 
> On 3/3/2010 3:14 AM, Thomas Heller wrote:
[...]
> 
> When I run this on my machine, it gives me 368456 as the result, whereas 
> Task Manager give me 1473824. Stupid question, but do I need to convert 
> the long to a kilobyte or something?  I tried, but that was way off, so 
> I must be doing something wrong. Thank you for your help!

Mike, I said this snippet is not really tested.
I created the definition of the SYSTEM_PERFORMANCE_INFORMATION
structure from some sample C++ code I found on the internet;
it may be wrong, it may be that I made a mistake or whatever.

You have to do some research:  Find out the real definition of this structure,
(I don't known if it is in some MS header file), you have to compare it to my code,
you have to find out what the field contents really means: kilobyte, pages, or whatever.

--

-- 
Thanks,
Thomas

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
(Continue reading)

Mike Driscoll | 4 Mar 15:03 2010
Picon

Re: Getting Peak Commit from NtQuerySystemInformation

On 3/4/2010 1:22 AM, Thomas Heller wrote:
Mike Driscoll schrieb:
Hi Thomas, On 3/3/2010 3:14 AM, Thomas Heller wrote:
[...]
When I run this on my machine, it gives me 368456 as the result, whereas Task Manager give me 1473824. Stupid question, but do I need to convert the long to a kilobyte or something? I tried, but that was way off, so I must be doing something wrong. Thank you for your help!
Mike, I said this snippet is not really tested. I created the definition of the SYSTEM_PERFORMANCE_INFORMATION structure from some sample C++ code I found on the internet; it may be wrong, it may be that I made a mistake or whatever. You have to do some research: Find out the real definition of this structure, (I don't known if it is in some MS header file), you have to compare it to my code, you have to find out what the field contents really means: kilobyte, pages, or whatever.

Sorry...I did do some research, but MSDN is confusing. According to this page, http://msdn.microsoft.com/en-us/library/ms724509%28VS.85%29.aspx, SystemPerformanceInformation "returns an opaque SYSTEM_PERFORMANCE_INFORMATION structure that can be used to generate an unpredictable seed for a random number generator. Use the CryptGenRandom function instead."

My understanding of that sentence is that it's just a randomization seed function that may or may not return the same number. The struct definition looks like this:

typedef struct _SYSTEM_PERFORMANCE_INFORMATION { BYTE Reserved1[312]; } SYSTEM_PERFORMANCE_INFORMATION;
Unfortunately, my C++ programming didn't go beyond anything more than bare bones crap in college, so I'm not sure what this is saying other than type definition and I think it might be reserving some space in memory. I've looked on several websites that purport to know about this class, but they all end up talking about listing open files or say that it's undocumented.

Thanks for your help. I apologize for bothering you.

- Mike
------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
John Dama (jdama | 4 Mar 16:54 2010
Picon

Py_XDECREF() causes segmentation fault in _call_function_pointer()

Hi. We are consistently getting a segmentation violation in _ctypes.so in _call_function_pointer() for a specific function.

The problem disappears if I comment out statement "Py_XDECREF(error_object)" in callproc.c _call_function_pointer().

error_object is NULL when seg fault occurs so Py_XDECREF(error_object) should effectively be doing nothing yet causes a seg fault.

 

** I'm not very familiar with mips assembler or backtraces and would like to understand *why* this problem occurred and find out if there is an "official" fix or workaround.

 

The problem function looks like:

int myfunc( tHandle handle, tNames name, void* pValue ) where "typedef void* tHandle" and "typedef enum { ... } tNames"

 

so the corresponding python/ctypes call looks like

mylib = ctypes.CDLL('/lib/mylib.so')

c_value = ctypes.c_uint16( 0x00 )

mylib.myfunc( ctypes.c_void_p( 1 ), c_int( 4 ), ctypes.cast( ctypes.byref( c_value ), ctypes.c_void_p ) )

<< note: if it looks strange to init a void* to a 1, this is done because parm type was is declared as "typedef void* tHandle" and this handle was effectively implemented as an integer, not a pointer to a structure as you might expect >>

 

Platform is embedded MIPS (big-endian). _ctypes is from Python-2.6.1.tar.bz2.

_call_function_pointer() in callproc.c seg faults at the "Py_XDECREF(error_object)" statement at line 834, specifically "lw" at offset 126e4 in objdump below.

The error_object pointer is NULL at the time "Py_XDECREF(error_object)" is called since "flags" parm is 0x1101, ie: neither FUNCFLAG_USE_ERRNO or FUNCFLAG_USE_LASTERROR are set.

I believe commenting it out doesn’t affect program logic in this case since error_object is NULL so nothing needs to happen: "#define Py_XDECREF(op) if ((op) == NULL) ; else Py_DECREF(op)".

 

  16:25:59 | /home/jdama/ccm_wa/atl_client_db/CTA5/Plat/oe/out/work/mips-linux/python-2.6.1-ml5/Python-2.6.1/Modules/_ctypes/callproc.c:832

  16:25:59 |    126d8:    ae040000    sw  a0,0(s0)

  16:25:59 | /home/jdama/ccm_wa/atl_client_db/CTA5/Plat/oe/out/work/mips-linux/python-2.6.1-ml5/Python-2.6.1/Modules/_ctypes/callproc.c:834

  16:25:59 |    126dc:    12400006    beqz    s2,126f8 <_CallProc+0x414>

  16:25:59 |    126e0:    00000000    nop

=>16:25:59 |    126e4:    8e420000    lw  v0,0(s2)

  16:25:59 |    126e8:    00000000    nop

  16:25:59 |    126ec:    2442ffff    addiu   v0,v0,-1

  16:25:59 |    126f0:    10400065    beqz    v0,12888 <_CallProc+0x5a4>

  16:25:59 |    126f4:    ae420000    sw  v0,0(s2)

  16:25:59 | /home/jdama/ccm_wa/atl_client_db/CTA5/Plat/oe/out/work/mips-linux/python-2.6.1-ml5/Python-2.6.1/Modules/_ctypes/callproc.c:836

  16:25:59 |    126f8:    16a0004f    bnez    s5,12838 <_CallProc+0x554>

  16:25:59 |    126fc:    00000000    nop

 

DIAG 0x0b: Cause: 0x0b - SIGSEGV: Segmentation violation

DIAG 0x0b: Reason: code 0x80, Sent by kernel

DIAG 0x0b: Thread: 5066720 - "", Priority: 5066552, AppID: 0

DIAG 0x0b: Backtrace: 2b2f66e4 _CallProc+0x400

                      2b2f687c _CallProc+0x598

                     

MMAP: 2b2e4000-2b303000 r-xp 00000000 00:0d 336528     /usr/lib/python2.6/lib-dynload/_ctypes.so

MMAP: 2b303000-2b342000 ---p 0001f000 00:0d 336528     /usr/lib/python2.6/lib-dynload/_ctypes.so

MMAP: 2b342000-2b345000 rw-p 0001e000 00:0d 336528     /usr/lib/python2.6/lib-dynload/_ctypes.so

 

2b2f66e4 - 2b2e4000 = 126E4

 

Thanks,

John Dama

 

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Thomas Heller | 4 Mar 18:53 2010

Re: Getting Peak Commit from NtQuerySystemInformation

Mike Driscoll schrieb:
> On 3/4/2010 1:22 AM, Thomas Heller wrote:
>> Mike Driscoll schrieb:
>>    
>>> Hi Thomas,
>>>
>>> On 3/3/2010 3:14 AM, Thomas Heller wrote:
>>>      
>> [...]
>>    
>>> When I run this on my machine, it gives me 368456 as the result, whereas
>>> Task Manager give me 1473824. Stupid question, but do I need to convert
>>> the long to a kilobyte or something?  I tried, but that was way off, so
>>> I must be doing something wrong. Thank you for your help!
>>>      
>> Mike, I said this snippet is not really tested.
>> I created the definition of the SYSTEM_PERFORMANCE_INFORMATION
>> structure from some sample C++ code I found on the internet;
>> it may be wrong, it may be that I made a mistake or whatever.
>>
>> You have to do some research:  Find out the real definition of this structure,
>> (I don't known if it is in some MS header file), you have to compare it to my code,
>> you have to find out what the field contents really means: kilobyte, pages, or whatever.
>>
>>    
> 
> Sorry...I did do some research, but MSDN is confusing. According to this 
> page, http://msdn.microsoft.com/en-us/library/ms724509%28VS.85%29.aspx, 
> SystemPerformanceInformation "returns an opaque 
> *SYSTEM_PERFORMANCE_INFORMATION* structure that can be used to generate 
> an unpredictable seed for a random number generator. Use the 
> *CryptGenRandom* 
> <http://msdn.microsoft.com/en-us/library/aa379942%28VS.85%29.aspx> 
> function instead."
> 
> My understanding of that sentence is that it's just a randomization seed 
> function that may or may not return the same number. The struct 
> definition looks like this:
> 
> |typedef struct _SYSTEM_PERFORMANCE_INFORMATION {
>      BYTE Reserved1[312];
> } SYSTEM_PERFORMANCE_INFORMATION;|
> 

It simply means that the structure exposes internal information which is not
published officially.

Well, I ran my script again, and changed the final print statement to this:

print spi.PeakCommitment * 4096 / 1024

Now it print '2310636', which is exactly the same value as sysinternals process monitor
shows in its 'System Information' dialog as 'Peak Commit Charge (K)'
so I assume it is correct: spi.PeakCommitment returns the peak number of committed
memory pages, each page (on the x86 at least) has 4096 bytes so we multiply by 4096,
then we divide by 1024 to get the value in KB.

Looks good, if you ask me.

> 
> Thanks for your help. I apologize for bothering you.

No problem - you're welcome.

--

-- 
Thanks,
Thomas

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
Thomas Heller | 4 Mar 19:05 2010

Re: Py_XDECREF() causes segmentation fault in _call_function_pointer()

John Dama (jdama) schrieb:
> Hi. We are consistently getting a segmentation violation in _ctypes.so
> in _call_function_pointer() for a specific function.
> 
> The problem disappears if I comment out statement
> "Py_XDECREF(error_object)" in callproc.c _call_function_pointer().
> 
> error_object is NULL when seg fault occurs so Py_XDECREF(error_object)
> should effectively be doing nothing yet causes a seg fault.

Ok, so it seems that the function call has damaged something (the stack,
local vars, something like that).
Which also means that commenting out the Py_XDECREF() call isn't a solution.

> The problem function looks like:
> 
> int myfunc( tHandle handle, tNames name, void* pValue ) where "typedef
> void* tHandle" and "typedef enum { ... } tNames"

What does the function do?  Will it return some information
(means: write into the memory location) pointed to by the third argument?

>  
> 
> so the corresponding python/ctypes call looks like
> 
> mylib = ctypes.CDLL('/lib/mylib.so')
> 
> c_value = ctypes.c_uint16( 0x00 )
> 
> mylib.myfunc( ctypes.c_void_p( 1 ), c_int( 4 ), ctypes.cast(
> ctypes.byref( c_value ), ctypes.c_void_p ) )

If the function will write something into the third argument then
you are calling it incorrectly.  The third argument may not be what
you mean.
Assuming you have to pass the address of an integer variable as third arg,
then you should use code like the following:

c_value = ctypes.c_uint16(0x00)
mylib.myfunc(ctypes.c_void_p(1), c_int(4), ctypes.byref(c_value))

Anyway, it is always safer to define argtypes (and possible restype) for
functions, especially on machines where sizeof(int) != sizeof(void *).
In your case, probably:

mylib.myfunc.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p]

# After that, the following function call does the same as the above example,
# but will automatically convert and check the arguments:
mylib.myfunc(1, 4, byref(c_value))

--

-- 
Thanks,
Thomas

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

Gmane