headers
Oleksandr "Sanja" Byelkin | 3 Feb 2008 13:21
Picon
Favicon

Re: A heuristic for query cache memory allocation

Hi!

On Jan 23, 2008, at 5:21 PM, Oleksandr Sanja Byelkin wrote:
[skip]
> Just several days ago we discussed other ways to avoid overhead of  
> memory defragmentation and the defragmentation itself - other  
> memory allocation strategy. I think it will work better (As soon as  
> the description of the task will be publicly visible I'll inform you).

http://forge.mysql.com/worklog/task.php?id=4210

[skip]

-- 
    __  ___     ___ ____  __
   /  |/  /_ __/ __/ __ \/ /    Mr. Oleksandr Byelkin <sanja <at> mysql.com>
  / /|_/ / // /\ \/ /_/ / /__   MySQL AB, Full-Time Developer
/_/  /_/\_, /___/\___\_\___/   Lugansk, Ukraine
        <___/   www.mysql.com

--

-- 
MySQL Internals Mailing List
For list archives: http://lists.mysql.com/internals
To unsubscribe:    http://lists.mysql.com/internals?unsub=gcdmd-internals <at> m.gmane.org
Permalink | Reply | 2 comments |
headers
Baron Schwartz | 3 Feb 2008 14:42
Favicon
Gravatar

Re: A heuristic for query cache memory allocation

Hi,

On Feb 3, 2008 7:21 AM, Oleksandr Sanja Byelkin <sanja <at> mysql.com> wrote:
> Hi!
>
> On Jan 23, 2008, at 5:21 PM, Oleksandr Sanja Byelkin wrote:
> [skip]
> > Just several days ago we discussed other ways to avoid overhead of
> > memory defragmentation and the defragmentation itself - other
> > memory allocation strategy. I think it will work better (As soon as
> > the description of the task will be publicly visible I'll inform you).
>
> http://forge.mysql.com/worklog/task.php?id=4210
>

Thanks for the link.  I'm not all that familiar with memory allocation
algorithms but this does sound more modern than the current strategy,
which seems pretty simple to me.  The current algorithm is basically
buddy allocation, right?
http://en.wikipedia.org/wiki/Buddy_memory_allocation  It sounds like
the new idea will have some pretty clear benefits over the current
approach.  Is there a target for the release it'll go into?

I'm a little concerned about this: "* Implemented as pluggable modules
which can be used in differentiation."

I just hate to see MySQL going down that path.  I think this search
for differentiation will eventually harm MySQL :-(  But that's
off-topic.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Rick James | 4 Feb 2008 18:28
Picon
Favicon

RE: A heuristic for query cache memory allocation

A new issue to consider in mem alloc choice -- jumping around memory is
costly, both in pagination (which should not occur) and in on-chip cache
misses.  A cache miss costs as much as dozens, maybe hundreds, of
instructions. 

> -----Original Message-----
> From: baron.schwartz <at> gmail.com 
> [mailto:baron.schwartz <at> gmail.com] On Behalf Of Baron Schwartz
> Sent: Sunday, February 03, 2008 5:43 AM
> To: Oleksandr Sanja Byelkin
> Cc: internals <at> lists.mysql.com
> Subject: Re: A heuristic for query cache memory allocation
> 
> Hi,
> 
> On Feb 3, 2008 7:21 AM, Oleksandr Sanja Byelkin 
> <sanja <at> mysql.com> wrote:
> > Hi!
> >
> > On Jan 23, 2008, at 5:21 PM, Oleksandr Sanja Byelkin wrote:
> > [skip]
> > > Just several days ago we discussed other ways to avoid overhead of
> > > memory defragmentation and the defragmentation itself - other
> > > memory allocation strategy. I think it will work better 
> (As soon as
> > > the description of the task will be publicly visible I'll 
> inform you).
> >
> > http://forge.mysql.com/worklog/task.php?id=4210
> >
(Continue reading)

Permalink | Reply | 0 comments |
headers
Mariella Petrini | 4 Feb 2008 18:45
Picon
Favicon

mysql 5.1.22-rc and long_query_time and timing : question

Hi All,

I have been using MySQL 5.1.22-rc on Linux 2.6.23 with
Debian.
I have compiled MySQL with the following options:

"CC='gcc'  CFLAGS='-DBIG_JOINS=1 -O3'  CXX='g++' 
CXXFLAGS='-DBIG_JOINS=1 -felide-constructors
-fno-exceptions -fno-rtti -O3'  LDFLAGS='' 
ASFLAGS=''"
CONFIGURE_LINE="./configure 'CC=gcc'
'CFLAGS=-DBIG_JOINS=1 -O3' 'CXX=g++'
'CXXFLAGS=-DBIG_JOINS=1 -felide-constructors
-fno-exceptions -fno-rtti -O3'
'--prefix=...../mysql-5.1.22-rc'
'--localstatedir=....../mysql-5.1.22-rc/data'
'--libexecdir=....../mysql-5.1.22-rc/bin'
'--datadir=....../mysql-5.1.22-rc/data'
'-with-comment=Debian x86_64'
'--with-server-suffix=Debian x86_64' '--enable-shared'
'--enable-static' '--enable-thread-safe-client'
'--enable-assembler' '--enable-local-infile'
'--with-big-tables' '--with-raid'
'--with-mysqld-user=mysql' '--with-libwrap'
'--with-mysqld-ldflags=-all-static' '--with-vio'
'--with-bench' '--with-readline'
'--with-extra-charsets=all' '--with-innodb'
'--with-isam' '--with-archive-storage-engine'
'--with-csv-storage-engine'
'--with-federated-storage-engine'
(Continue reading)

Permalink | Reply | 12 comments |
headers
Rick James | 4 Feb 2008 22:22
Picon
Favicon

RE: mysql 5.1.22-rc and long_query_time and timing : question

100 clients doing the INSERTs at essentially the same time...
They will be stumbling over each other (MyISAM and, to a lesser extent,
InnoDB).

Some issues...
* Are you running out of open_tables?
* Are the INSERTs at the "end" of the data (relevant to MyISAM)
* What is the median time for your query?
* How many INDEXes on the table (they require updates, too)
* Is everything in Cache?
* What are the settings (if InnoDB)?
* RAID?  Which kind?  (Hopefully hardware, not --with-raid)

In other words, I am not surprised that _some_ INSERTs take 3 seconds,
regardless of MySql version.  Many factors enter into why some inserts
might be slow.

> -----Original Message-----
> From: Mariella Petrini [mailto:mariellapetrini <at> yahoo.com] 
> Sent: Monday, February 04, 2008 9:45 AM
> To: internals <at> lists.mysql.com
> Subject: mysql 5.1.22-rc and long_query_time and timing : question
> 
> Hi All,
> 
> 
> I have been using MySQL 5.1.22-rc on Linux 2.6.23 with
> Debian.
> I have compiled MySQL with the following options:
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Martin Friebe | 5 Feb 2008 00:00

Re: mysql 5.1.22-rc and long_query_time and timing : question

Hi Mariella,

As far as my understanding goes it isn't you, and there are extras 
included on the server side.

I know that simple select statements ("select a from tbl") return the 
result *while* they read it from the table. As for the insert it can 
return a result to the client (your app) as soon as the data is in the 
table.
After that there is still clean up work to be done, such as closing 
tables and other stuff.

looking at the 5.0 source:
- the slow log seems to be done at the end of dispatch_command ( in 
sql/sql_parse.cc)
- the result seems to be send back in mysql_insert (in sql/sql_insert.cc)

the call strucure is
dispatch_command -> mysql_parse  ->mysql_execute_command   (all in 
sql/sql_parse.cc)
-> mysql_insert

It seems particular long that the "other stuff" takes 9 seconds, if the 
update only takes 3 seconds.

You could try playing with things like:
- the max amount of open tables / cached open tables
- size of the query cache (smaller size may give better results)

Hope it helps
(Continue reading)

Permalink | Reply | 0 comments |
headers
Stewart Smith | 5 Feb 2008 08:25
Picon
Favicon

Re: libndbclient invalid select() call


On Mon, 2008-01-28 at 16:16 +0100, Jonas Oreland wrote: 
> > * are there any arguments not to change the select call to poll call
> >   which I believe would fix the problem? (portability?)
> 
> no, that sounds good and clever

It's certainly clever - and does raise portability issues.

Which basically means if we care about:
	- MacOS X 10.2 (we don't... well, i don't - and that's a good enough
reason to not support it)
	- MacOS X 10.0 and 10.1 (i don't care.. they had a poll, but a buggy
one)
	- QNX 4 (no idea... i don't really care.. but somebody may)
	- Windows (yes, we do care... so need some wrapper)

But you can pretty easily wrap poll with a select call... 
http://www.flamingspork.com/portawiki/index.php?title=Poll

says that openntpd should have a BSD licensed one.

On linux, epoll is probably a better option to use...

I for one welcome our new poll/epoll overlords and would love to see a
patch :)
--

-- 
Stewart Smith, Senior Software Engineer (MySQL Cluster)
MySQL AB, www.mysql.com
Office: +14082136540 Ext: 6616
(Continue reading)

Permalink | Reply | 2 comments |
headers
Jan Kneschke | 5 Feb 2008 10:10
Picon
Favicon

Re: libndbclient invalid select() call

Stewart Smith wrote:
> On Mon, 2008-01-28 at 16:16 +0100, Jonas Oreland wrote: 
>>> * are there any arguments not to change the select call to poll call
>>>   which I believe would fix the problem? (portability?)
>> no, that sounds good and clever
> 
> It's certainly clever - and does raise portability issues.
> 
> Which basically means if we care about:
> 	- MacOS X 10.2 (we don't... well, i don't - and that's a good enough
> reason to not support it)
> 	- MacOS X 10.0 and 10.1 (i don't care.. they had a poll, but a buggy
> one)
> 	- QNX 4 (no idea... i don't really care.. but somebody may)
> 	- Windows (yes, we do care... so need some wrapper)
> 
> But you can pretty easily wrap poll with a select call... 
> http://www.flamingspork.com/portawiki/index.php?title=Poll
> 
> says that openntpd should have a BSD licensed one.
> 
> On linux, epoll is probably a better option to use...
> 
> I for one welcome our new poll/epoll overlords and would love to see a
> patch :)

Please look at libevent (see MySQL 6.0). It _is_ wrapping all the
syscalls already and also knows about its bugs.

http://www.monkey.org/~provos/libevent/
(Continue reading)

Permalink | Reply | 1 comment |
headers
Sergei Golubchik | 5 Feb 2008 10:32
Picon
Favicon

Re: mysql 5.1.22-rc and long_query_time and timing : question

Hi!

On Feb 04, Mariella Petrini wrote:

> I have noticed the following:
> the response time calculated on the client side (for
> the same query) is lower
> that the one written in the mysql slow log.

Does it happen often ? I can imagine why this could've happen once or
twice (*), but if you see it all the time, we need to look for a better
explanation.

(*) prompted by a thoughtful Martin's reply, I checked the sources - the
most probable suspect is LOCK_open, which is held in
close_thread_tables(). If a table was used by many threads, never closed
but kept in the table cache, then closing (real closing) of such a table
could, technically, cause a lot of I/O, and that would stall other threads.
But such an unusual combination of circumstances cannot happen often.

Regards / Mit vielen Grüssen,
Sergei

--

-- 
   __  ___     ___ ____  __
  /  |/  /_ __/ __/ __ \/ /   Sergei Golubchik <serg <at> mysql.com>
 / /|_/ / // /\ \/ /_/ / /__  Principal Software Developer
/_/  /_/\_, /___/\___\_\___/  MySQL GmbH, Dachauer Str. 37, D-80335 München
       <___/                  Geschäftsführer: Kaj Arnö - HRB München 162140
(Continue reading)

Permalink | Reply | 9 comments |
headers
Łukasz Osipiuk / Gadu-Gadu | 5 Feb 2008 12:26
Picon
Favicon

Re: libndbclient invalid select() call

Monday 28 of January 2008 16:16:23 Jonas Oreland napisał(a):
> Łukasz Osipiuk / Gadu-Gadu wrote:
> > Hello,
> >
> > I have a question concerning MySQL Cluster client library
> > implementation (hope I reached correct mailing list :).
> >
> > We are using libndbclient.so (telco branch: 5.1.23-ndb-6.3.7 compiled
> > under Linux 2.6 amd64) in the application which creates many file
> > descriptors (i.e TCP connections).  We are experiencing crashes of
> > client application (SIGSEGV with stack corruption) if shutdown one of
> > cluster nodes (leaving a cluster, as a whole, operational).
> >
> > After analysis of libndbclient code we found out that it uses select
> > in the loop which tries to reestablish connection with shutdown node.
> > (e.g. code in storage/ndb/src/common/util/SocketClient.cpp)
> >
> > As soon as number of used file descriptors in our application exceeds
> > 1024, Cluster client library starts to make invalid calls to select()
> > function. The file descriptor it gets from socket() call is >= than
> > 1024 (FD_SETSIZE), and calling select with such fd crashes client app
> > (doc: http://linux.die.net/man/2/select).
> >
> > And here comes the question (more than one really):
> >
> > * is the problem known?
>
> i think you're the first to report it...
>
> > * if so were there any attempts to fix/change client app behavior?
(Continue reading)

Permalink | Reply | 0 comments |

Gmane