I did some tests with some programs that PUT/GET a large amount of messages with /opt and /var being relatime and strictatime (where atime is always updated), and I can't see any noticeable difference in the wall times.  Looking at an strace, it probably has to do with the fact that MQ was only doing a handful of read system calls during the tests.  I tried having most of the messages spill to a queue file (over 100M worth) to force MQ to do more i/o, but still there were hardly any read system calls when MQ went to get the messages from the queue file. 

Anyway, the relatime it is still worth noting and being aware of.  If your Linux server gets heavily i/o bound, then relatime would help for sure. 

Thanks,

Tim

Hi Musthafa,

I don't have any personal information, but I would think it would be noticeable.  Here was one of the quotes on the performance improvements of relatime from one of the Linux kernel developers (Ingo Molnar):

"I cannot over-emphasize how much of a deal it is in practice. Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_."

I am curious as well, so I will look into it and get back to you.  I have an MQ sandbox Linux environment where I can have the Linux administrators change the MQ filesystem mounts between strictatime and relatime, and then I can get some MQ performance numbers to compare.

Thanks,

Tim

Hello Tim,

Thanks for the info. Have you personally seen/witnessed any performance degradation on versions older than 2.6.30 or any significant performance improvements/learnings with versions 2.6.32 and above?

Appreciate any info you can share. Thanks again.

Sincerely,

Mir

MQ Admin | PepsiCo, Inc. | BIS

I think this information I am sharing about Linux relatime is about 5 years old and some of you are probably already aware of it, but just wanted to pass it on in case you were not and run MQ on Linux.

This has to do with the Unix/Linux file attribute of atime.  As a reminder, a Unix/Linux file has 3 time attributes called atime (access time), ctime (change time), and mtime (modify time).  Access time is the last time a read or write happened against the file (or used to be for Linux, I'll get to that later), change time is the last time file attributes like permissions were changed, and modify time was the last time the file contents were changed.  These 3 values are stored in the inode of the file which is a structure stored on the file system and potentially a copy in-memory, as well.

What the Linux kernel programmers realized a while ago is that atime is a significant performance issue.  This is because anytime a file does a read or write (even if the read was against a file cached in memory) an i/o write has to occur to update the atime for the inode structure on the filesystem.  So they eventually came up with a new default called relatime (relative atime).  relatime changes the atime behavior so it is only updated when it is older than the ctime or mtime, or I think also if it hasn't been updated in the past 24 hours.  I believe it is the default for Linux since around the 2.6.30 kernel.  I have verified that our 2.6.32 Linux SUSE kernels are using relatime, and are older Red Hat Linux kernels which are at 2.6.18 are not.  Of course, this means that Linux has completely changed the default historical definition of atime, but they did it to provide some significant Linux performance improvements.

I couldn't find any mention of relatime in the MQ manuals or with a google search of MQ and relatime, so I thought I would mention it.  If you are an MQ administrator and your Linux kernels are older than 2.6.30, you may want to consider upgrading to a later kernel as you will get some MQ performance improvement from this Linux default relatime change.

Thanks,

Tim Zielke

CICS/MQ Systems Programmer

Aon Hewitt.

List Archive - Manage Your List Settings - Unsubscribe

Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com

List Archive - Manage Your List Settings - Unsubscribe

Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com

I wanted to pass on a notable enhancement to the MQTrcStats program.

After some more investigation, I found that if you run an MQ api trace with just the application program being traced (i.e. strmqtrc -m qmgr -p program -t api), you can get pretty close to capturing the actual MQI response times.  Therefore, I have changed the MQTrcStats program to now parse an api trace.  That code has been updated at -> http://www.capitalware.biz/mq_code_c.html

Below are more details of what I have found in my testing, in case you are interested. 

NOTE:  It is important to be aware that running any type of MQ trace should be done with caution, as it will have an overall response time impact on the traced processes and produce a large amount of data in the MQ trace directory (i.e. /var/mqm/trace).  However, if done judiciously (i.e. run the trace for a 5 minute window), I have found that running a selective MQ api trace in conjunction with the MQTrcStats formatter program to be a very helpful tool for problem determination with MQ response time issues.  Now that it looks like you can also leverage it to have a window into the MQI response time actual for the programs that access your queue manager, it could be useful for other benefits, as well.

More Details:

An MQ api trace is capturing details of the before and after of most MQI calls.  For the PUT example, you will see trace records for the PutBefore and trace records for the PutAfter.  What I found is that if you take the timestamp for the last PutBefore and the first timestamp for the PutAfter, you get a response time that is in line with the actual MQI response time that an API exit can capture.  Here are some more specifics.

First, I took an API exit and had it capture in memory the response times for PUTs and GETs.  I then took a program that did 10,000 PUTs/GETs of a 30 byte message and got the following timings:

ApiTrace.17816.1.log:Average put elapsed time was 00000.000042

ApiTrace.17816.1.log:Average get elapsed time was 00000.000042

ApiTrace.17894.1.log:Average put elapsed time was 00000.000052

ApiTrace.17894.1.log:Average get elapsed time was 00000.000052

ApiTrace.17896.1.log:Average put elapsed time was 00000.000055

ApiTrace.17896.1.log:Average get elapsed time was 00000.000054

ApiTrace.17897.1.log:Average put elapsed time was 00000.000041

ApiTrace.17897.1.log:Average get elapsed time was 00000.000040

ApiTrace.17898.1.log:Average put elapsed time was 00000.000038

ApiTrace.17898.1.log:Average get elapsed time was 00000.000038

Then I ran the same program with an MQ trace with the options "-p program -t api", ran the MQTrcStats program on the resulting formatted trace, and got the following timings:

NOTE:  The "-p program" trace option is important as it removes the queue manager from being traced.  This is important if you want to get more accurate MQI response times with this approach.

MQGET QUEUE STATS

The number of MQGETs with bad formats was 0

Following rows have columns as follows:

Queue Name, Total Ok GET, Total Ok GET Elapsed Time, Avg Ok GET Elapsed Time, Total Failed GET, Total Failed GET Elapsed Time, Avg Failed GET Elapsed Time, Total GET Bytes

TCZ.TEST1                                        10000 00000.537344 00000.000054 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.434234 00000.000043 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.665437 00000.000067 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.362229 00000.000036 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.513352 00000.000051 0 00000.000000 00000.000000 300000

MQPUT QUEUE STATS

The number of MQPUTs with bad formats was 0

Following rows have columns as follows:

Queue Name, Total Ok PUT, Total Ok PUT Elapsed Time, Avg Ok PUT Elapsed Time, Total Failed PUT, Total Failed PUT Elapsed Time, Avg Failed PUT Elapsed Time, Total PUT Bytes

TCZ.TEST1                                        10000 00000.530070 00000.000053 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.429242 00000.000043 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.657231 00000.000066 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.357531 00000.000036 0 00000.000000 00000.000000 300000

TCZ.TEST1                                        10000 00000.505919 00000.000051 0 00000.000000 00000.000000 300000

You can see from the results that the api trace MQI response timings are in line with the MQI response timings from the API exit.

Thanks,

Tim

Hello,

I just wanted to pass along a C program that I wrote called MQTrcStats that takes a formatted MQ distributed trace and pulls out summarized statistical information for some of the more common MQIs commands like MQGET with total and average response time by queue name.  I have found this helpful for problem determination with Test and Production issues where distributed MQ performance is being questioned. 

Roger Lacroix has posted this program on the Capitalware web site (thanks, Roger!).  I can also pass it along, if anyone wants it directly.

http://www.capitalware.biz/mq_code_c.html

NOTE:  If you find the following information useful, please vote for this RFE which is requesting that MQ provides this MQI response time details with the MQ statistics.

RFE - http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=28409

Here is an example of some of the information it can produce with a use case example:

You are called into a Production support issue where the application team is saying that their distributed linux application that uses MQ is having response time issues and it is because MQ is slow (allegedly).  You get the program name which is called apps1 and then run a trace as follows (use -p if possible to limit the tracing to just the program) to get a few minutes of trace history for the program.

strmqtrc -m qmgrname -p apps1 -t all -t detail

Then you format the trace with dspmqtrc and then run mqtrcstats on that formatted trace as follows:

dspmqtrc AMQ12345.0.TRC  > AMQ12345.0.FMT

mqtrcstats AMQ12345.0.FMT

This gives you the following information:

MQGET QUEUE STATS

The number of MQGETs with bad formats was 0

No MQGETs

MQPUT QUEUE STATS

The number of MQPUTs with bad formats was 0

Following rows have columns as follows:

Queue Name, Total Ok PUT, Total Ok PUT Elapsed Time, Avg Ok PUT Elapsed Time, Total Failed PUT, Total Failed PUT Elapsed Time, Avg Failed PUT Elapsed Time, Total PUT Bytes

XYZP.00000.WEB.USEREXPR                          28 00000.011372 00000.000406 0 00000.000000 00000.000000 13154

XYZP.00000.WEB5.TBAACTNDW                        111 00000.115486 00000.001040 0 00000.000000 00000.000000 139941

XYZP.00000.WEB5.USAGEDW                          54 00000.055976 00000.001037 0 00000.000000 00000.000000 83307

XYZP.00000.WEB5.USAGE                            54 00000.051623 00000.000956 0 00000.000000 00000.000000 61838

XYZP.00000.WEB.TOOLELIG                          10 00000.008753 00000.000875 0 00000.000000 00000.000000 3033

XYZP.00000.WEB5.ERROR                            2 00000.002046 00000.001023 0 00000.000000 00000.000000 11273

                                                 11 00000.008752 00000.000796 0 00000.000000 00000.000000 12208

XYZP.00000.CALL.R20ITMESSAGEALIAS                9 00000.010753 00000.001195 0 00000.000000 00000.000000 8425

XYZP.00000.WEB5.BUSEVNTDW                        5 00000.004026 00000.000805 0 00000.000000 00000.000000 3029

With this information, you are able to quickly see that the MQI response time for MQPUTs is sub-second and are able to communicate that to the support call with specific details.  

Thanks,

Tim

Hi Musthafa,

I don't have any personal information, but I would think it would be noticeable.  Here was one of the quotes on the performance improvements of relatime from one of the Linux kernel developers (Ingo Molnar):

"I cannot over-emphasize how much of a deal it is in practice. Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_."

I am curious as well, so I will look into it and get back to you.  I have an MQ sandbox Linux environment where I can have the Linux administrators change the MQ filesystem mounts between strictatime and relatime, and then I can get some MQ performance numbers to compare.

Thanks,

Tim

Hello Tim,

Thanks for the info. Have you personally seen/witnessed any performance degradation on versions older than 2.6.30 or any significant performance improvements/learnings with versions 2.6.32 and above?

Appreciate any info you can share. Thanks again.

Sincerely,

Mir

MQ Admin | PepsiCo, Inc. | BIS

I think this information I am sharing about Linux relatime is about 5 years old and some of you are probably already aware of it, but just wanted to pass it on in case you were not and run MQ on Linux.

This has to do with the Unix/Linux file attribute of atime.  As a reminder, a Unix/Linux file has 3 time attributes called atime (access time), ctime (change time), and mtime (modify time).  Access time is the last time a read or write happened against the file (or used to be for Linux, I'll get to that later), change time is the last time file attributes like permissions were changed, and modify time was the last time the file contents were changed.  These 3 values are stored in the inode of the file which is a structure stored on the file system and potentially a copy in-memory, as well.

What the Linux kernel programmers realized a while ago is that atime is a significant performance issue.  This is because anytime a file does a read or write (even if the read was against a file cached in memory) an i/o write has to occur to update the atime for the inode structure on the filesystem.  So they eventually came up with a new default called relatime (relative atime).  relatime changes the atime behavior so it is only updated when it is older than the ctime or mtime, or I think also if it hasn't been updated in the past 24 hours.  I believe it is the default for Linux since around the 2.6.30 kernel.  I have verified that our 2.6.32 Linux SUSE kernels are using relatime, and are older Red Hat Linux kernels which are at 2.6.18 are not.  Of course, this means that Linux has completely changed the default historical definition of atime, but they did it to provide some significant Linux performance improvements.

I couldn't find any mention of relatime in the MQ manuals or with a google search of MQ and relatime, so I thought I would mention it.  If you are an MQ administrator and your Linux kernels are older than 2.6.30, you may want to consider upgrading to a later kernel as you will get some MQ performance improvement from this Linux default relatime change.

Thanks,

Tim Zielke

CICS/MQ Systems Programmer

Aon Hewitt.

List Archive - Manage Your List Settings - Unsubscribe

Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com

List Archive - Manage Your List Settings - Unsubscribe

Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com