For users and developers of memcached.

柴俊 | 1 Feb 06:48

Re: how to update all values as a daily task with memcached efficiently?

Yeah, really just like use "flush" method. but the exactly method is "flushAll()" for Java MemCacheClient.

jar packages include java_memcached-release_2.6.3.jar

2012/1/31 dormando <dormando <at> rydia.net>

> hi all,the problem is like this:There are 10 millions of key/values in memcached,and it requires to update all the values at 00:00 every day.
> For example,it is a daily task,the value data can not be accumulated .so it should clear up the old data value at a fixed time every day.
>
> Is there any efficient method to handle this problem ?
>
> THX for your advise.

Call "flush" on all the servers at 00:00. all the values are immediately
invalidated.

--

headers

memcached | 1 Feb 07:37

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #8 on issue 183 by jhpark...@gmail.com: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

Flush can be done with delay option.
So. "settings.oldest_live <= current_time" condition must be added
in order to find the flushed items.

headers

memcached | 1 Feb 07:47

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #9 on issue 183 by dorma...@rydia.net: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

What codebase and lines and sections are you referring to here?

If it's a new bug, please open a new bug report with more details

headers

memcached | 1 Feb 08:09

Re: Issue 248 in memcached: undefined reference to `__sync_sub_and_fetch_2'

Updates:
	Status: Fixed

Comment #2 on issue 248 by dorma...@rydia.net: undefined reference to  
`__sync_sub_and_fetch_2'
http://code.google.com/p/memcached/issues/detail?id=248

closing. 1.4.12 will be out any moment now.

headers

memcached | 1 Feb 08:14

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #10 on issue 183 by jhpark...@gmail.com: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

I have memcached-1.4.10.
In the following do_item_alloc(),
the check condition for invalidated items should be changed to the new code.
Because, flush can be request with delay option.

item *do_item_alloc(char *key, const size_t nkey, const int flags, const  
rel_time_t exptime, const int nbytes) {

     ...

     rel_time_t oldest_live = settings.oldest_live;

     search = tails[id];
     if (search == NULL) {
         it = slabs_alloc(ntotal, id);
     } else if (search->refcount == 0) {
#if 1 // old code
         if ((search->time < oldest_live && oldest_live <= current_time) ||  
// dead by flush
          (search->exptime != 0 && search->exptime < current_time)) {
#else // old code
         if ((search->time < oldest_live) || // dead by flush
          (search->exptime != 0 && search->exptime < current_time)) {
#endif
             STATS_LOCK();
             stats.reclaimed++;
             STATS_UNLOCK();
             itemstats[id].reclaimed++;
             if ((search->it_flags & ITEM_FETCHED) == 0) {
                 STATS_LOCK();
                 stats.expired_unfetched++;
                 STATS_UNLOCK();
                 itemstats[id].expired_unfetched++;
             }
             it = search;
             it->refcount = 1;
             slabs_adjust_mem_requested(it->slabs_clsid, ITEM_ntotal(it),  
ntotal);
             do_item_unlink_nolock(it, hash(ITEM_key(it), it->nkey, 0));
             /* Initialize the item block: */
             it->slabs_clsid = 0;
             it->refcount = 0;
         }
     }

     ...

}

headers

memcached | 1 Feb 08:18

Re: Issue 244 in memcached: Extremely Large Get Requests Fail


Comment #11 on issue 244 by dorma...@rydia.net: Extremely Large Get  
Requests Fail
http://code.google.com/p/memcached/issues/detail?id=244

Any update? :)

headers

memcached | 1 Feb 08:22

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #11 on issue 183 by jhpark...@gmail.com: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

I have memcached-1.4.10.
In the following do_item_alloc(),
the check condition for invalidated items should be changed to the new code.
Because, flush can be requested with delay option.

item *do_item_alloc(char *key, const size_t nkey, const int flags, const  
rel_time_t exptime, const int nbytes) {

     ...

     rel_time_t oldest_live = settings.oldest_live;

     search = tails[id];
     if (search == NULL) {
         it = slabs_alloc(ntotal, id);
     } else if (search->refcount == 0) {
#if 1 // new code
         if ((search->time < oldest_live && oldest_live <= current_time) ||  
// dead by flush
          (search->exptime != 0 && search->exptime < current_time)) {
#else // old code
         if ((search->time < oldest_live) || // dead by flush
          (search->exptime != 0 && search->exptime < current_time)) {
#endif
             STATS_LOCK();
             stats.reclaimed++;
             STATS_UNLOCK();
             itemstats[id].reclaimed++;
             if ((search->it_flags & ITEM_FETCHED) == 0) {
                 STATS_LOCK();
                 stats.expired_unfetched++;
                 STATS_UNLOCK();
                 itemstats[id].expired_unfetched++;
             }
             it = search;
             it->refcount = 1;
             slabs_adjust_mem_requested(it->slabs_clsid, ITEM_ntotal(it),  
ntotal);
             do_item_unlink_nolock(it, hash(ITEM_key(it), it->nkey, 0));
             /* Initialize the item block: */
             it->slabs_clsid = 0;
             it->refcount = 0;
         }
     }

     ...

}

headers

memcached | 1 Feb 08:26

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #12 on issue 183 by dorma...@rydia.net: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

Have you looked at 1.4.11? I rewrote that section, and might have fixed the  
bug at the same time.

headers

memcached | 1 Feb 08:37

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #13 on issue 183 by jhpark...@gmail.com: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

Now, I have looked at 1.4.11 code.

The code still does not have "oldest_live <= current_time" condition.

item *do_item_alloc(char *key, const size_t nkey, const int flags, const  
rel_time_t exptime, const int nbytes) {
     ...
         if ((search->exptime != 0 && search->exptime < current_time)
             || (search->time < oldest_live)) {  // dead by flush
     ...
}

headers

memcached | 1 Feb 08:41

Re: Issue 183 in memcached: Reclaiming the space of flushed items is needed.


Comment #14 on issue 183 by jhpark...@gmail.com: Reclaiming the space of  
flushed items is needed.
http://code.google.com/p/memcached/issues/detail?id=183

One more minor thing..

In do_item_get() function,
following code is used to check whether the fetched item is valid.

     if (it != NULL) {
         if (settings.oldest_live != 0 && settings.oldest_live <=  
current_time &&
             it->time <= settings.oldest_live) {
             do_item_unlink(it, hv);
             do_item_remove(it);
             it = NULL;
             if (was_found) {
                 fprintf(stderr, " -nuked by flush");
             }
         } else if (it->exptime != 0 && it->exptime <= current_time) {
             do_item_unlink(it, hv);
             do_item_remove(it);
             it = NULL;
             if (was_found) {
                 fprintf(stderr, " -nuked by expire");
             }
         } else {
             it->it_flags |= ITEM_FETCHED;
             DEBUG_REFCNT(it, '+');
         }
     }

In the above code,
"it->time <= settings.oldest_live" is used
instead of "it->time < settings.oldest_live".
So, it might be better to use the same condition in the code of  
do_item_alloc().

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29

49	May 2012
108	April 2012
101	March 2012
131	February 2012
116	January 2012
55	December 2011
96	November 2011
125	October 2011
154	September 2011
181	August 2011
158	July 2011
87	June 2011
90	May 2011
177	April 2011
191	March 2011
181	February 2011
89	January 2011
75	December 2010
114	November 2010
170	October 2010
123	September 2010
124	August 2010
181	July 2010
99	June 2010
110	May 2010
201	April 2010
232	March 2010
185	February 2010
152	January 2010
145	December 2009
145	November 2009
284	October 2009
247	September 2009
272	August 2009
266	July 2009
204	June 2009
153	May 2009
273	April 2009
336	March 2009
188	February 2009
226	January 2009
199	December 2008
176	November 2008
201	October 2008
193	September 2008
163	August 2008
31	July 2008
291	June 2008
160	May 2008
158	April 2008
239	March 2008
330	February 2008
205	January 2008
258	December 2007
302	November 2007
239	October 2007
279	September 2007
197	August 2007
387	July 2007
152	June 2007
281	May 2007
291	April 2007
125	March 2007
150	February 2007
134	January 2007
171	December 2006
170	November 2006
213	October 2006
161	September 2006
84	August 2006
152	July 2006
166	June 2006
102	May 2006
59	April 2006
87	March 2006
87	February 2006
62	January 2006
88	December 2005
98	November 2005
36	October 2005
51	September 2005
62	August 2005
48	July 2005
61	June 2005
49	May 2005
89	April 2005
58	March 2005
62	February 2005
88	January 2005
170	December 2004
28	November 2004