Sunday, December 4, 2011

Setup for Linux Memcached (daemon) with PHP PECL Memcache and Drupal Memcache Module - with localized UNIX sockets

This information enhances some documentation for the Drupal Project Memcache -> http://drupal.org/project/memcache
Revision Notes PHP: I have tested both PECL Memcache and PECL Memcached versions, and for the moment see less 'bugs' with the use of PECL memcache.
Suggested 'priority' - is to get the latest [memcache] release at http://svn.php.net/viewvc/pecl/memcache/
It has several improved 'features' - including the option to use faster 'localized' Unix 'sockets'

Memcache - Guidelines: see items 1) to 3)

1) A typical Drupal / Pressflow 6.2x use of memcache - via settings.php is: (copied form my own Linux server + based on fast [local] sockets for memcache)
<?php/**
* Memcache:
*/
$conf += array(
 
'memcache_extension'           => 'memcache',
 
'show_memcache_statistics'     => 0,
 
'memcache_persistent'          => TRUE,
 
'memcache_stampede_protection' => TRUE,
 
'memcache_stampede_semaphore'  => 15,
 
'memcache_stampede_wait_time'  => 5,
 
'memcache_stampede_wait_limit' => 3,
 
'memcache_key_prefix'          => basename(realpath(conf_path())),
);
// We don't use chained memcached caching for sites cron, php-cli and install!if (isset($_SERVER['HTTP_USER_AGENT']) && !preg_match("/(?:cron\.php|install\.php)/", $_SERVER['REQUEST_URI'])) {
  if (isset(
$_SERVER['HTTP_HOST'])) {
   
$conf += array(
     
'cache_inc'        => './sites/all/modules/memcache/memcache.inc',
     
'session_inc'      => './sites/all/modules/memcache/memcache-session.inc',
     
'lock_inc'         => './sites/all/modules/memcache/memcache-lock.inc',
     
'memcache_servers' => array(
        
'unix:///tmp/memcached.socket0:0'  => 'default',
        
'unix:///tmp/memcached.socket1:0'  => 'block',
        
'unix:///tmp/memcached.socket2:0'  => 'content',
        
'unix:///tmp/memcached.socket3:0'  => 'filter',
        
'unix:///tmp/memcached.socket4:0'  => 'form',
        
'unix:///tmp/memcached.socket5:0'  => 'menu',
        
'unix:///tmp/memcached.socket6:0'  => 'page',
        
'unix:///tmp/memcached.socket7:0'  => 'update',
        
'unix:///tmp/memcached.socket8:0'  => 'views',
        
'unix:///tmp/memcached.socket9:0'  => 'session',
        
'unix:///tmp/memcached.socket10:0' => 'users'
     
),
     
'memcache_bins'    => array(
        
'cache'         => 'default',
        
'cache_block'   => 'block',
        
'cache_content' => 'content',
        
'cache_filter'  => 'filter',
        
'cache_form'    => 'form',
        
'cache_menu'    => 'menu',
        
'cache_page'    => 'page',
        
'cache_update'  => 'update',
        
'cache_views'   => 'views',
        
'session'       => 'session',
        
'users'         => 'users'
     
),
    );
  }
}
?>
2) Sample 'memcache php.ini' settings:
[memcache]
; Data will be transferred in chunks of this size
memcache.chunk_size = 32768
memcache.hash_strategy = consistent
memcache.default_port = 0
session.save_handler = memcache
session.save_path = "unix:///tmp/memcached.socket11:0?persistent=1&weight=1&timeout=1&retry_interval=15"
3) Sample Linux memcached daemon /etc/init.d/memcached script:
#! /bin/sh
#
# chkconfig: - 55 45
# description:  The memcached-multi daemon is a network memory cache service.
# processname: memcached-multi
# config: /etc/sysconfig/memcached
# pidfile: /var/run/memcached/memcached.*.pid


# Standard LSB functions
#. /lib/lsb/init-functions

# Source function library.
. /etc/init.d/functions

PORT=11211
UDP=0
SOCKET=/tmp/memcached.socket
VAR=0
USER=memcached
MAXCONN=300
CACHESIZE=64
OPTIONS=""

if [ -f /etc/sysconfig/memcached ];then
        . /etc/sysconfig/memcached
fi

# Check that networking is up.
. /etc/sysconfig/network

if [ "$NETWORKING" = "no" ]
then
        exit 0
fi

RETVAL=0
prog="memcached"

start_instance() {
        echo -n $"Starting $prog ($1): "
#       daemon --pidfile /var/run/memcached/memcached.$1.pid memcached -d -p $PORT           -u $USER -m $2 -c $MAXCONN -P /var/run/memcached/memcached.$1.pid $OPTIONS
        daemon --pidfile /var/run/memcached/memcached.$1.pid memcached -d -s $3 -a 766 -L -t 8 -u $USER -m $2 -c $MAXCONN -P /var/run/memcached/memcached.$1.pid $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcached.$1
}

stop_instance() {
        echo -n $"Stopping $prog ($1): "
        killproc -p /var/run/memcached/memcached.$1.pid /usr/bin/memcached
        RETVAL=$?
        echo
        if [ $RETVAL -eq 0 ] ; then
            rm -f /var/lock/subsys/memcached.$1
            rm -f /var/run/memcached.$1.pid
        fi
}


start() {
    # insure that /var/run/memcached has proper permissions
    if [ "`stat -c %U /var/run/memcached`" != "$USER" ]; then
        chown $USER /var/run/memcached
    fi
    # we start 12 socket streams for memcached
    start_instance default 64 /tmp/memcached.socket0;
    start_instance block 64 /tmp/memcached.socket1;
    start_instance content 64 /tmp/memcached.socket2;
    start_instance filter 64 /tmp/memcached.socket3;
    start_instance form 64 /tmp/memcached.socket4;
    start_instance menu 64 /tmp/memcached.socket5;
    start_instance page 64 /tmp/memcached.socket6;
    start_instance update 64 /tmp/memcached.socket7;
    start_instance views 64 /tmp/memcached.socket8;
    start_instance session 64 /tmp/memcached.socket9;
    start_instance users 64 /tmp/memcached.socket10;
    start_instance pbold 64 /tmp/memcached.socket11;
}

stop () {
    stop_instance default;
    stop_instance block;
    stop_instance content;
    stop_instance filter;
    stop_instance form;
    stop_instance menu;
    stop_instance page;
    stop_instance update;
    stop_instance views;
    stop_instance session;
    stop_instance users;
    stop_instance pbold;
}

restart () {
        stop
        start
}


# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status memcached
        ;;
  restart|reload|force-reload)
        restart
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload}"
        exit 1
esac

exit $?
Notes: Above examples taken from my own working Linux server (multi-hosts).
For the last option (Linux), I used "yum install memcached"
eg:
yum search memcached
>
memcached.x86_64 : High Performance, Distributed Memory Object Cache



Additional Notes: The PHP Memcache PECL can also be compiled fully 'static' into the php-5.3.x binary-by moving the [memcache - PECL] source into the php /ext folder and rebuilding the PHP 'buildconf' - then compile PHP 5.3.x with --enable-memcache (it is *not* essential to use PECL if you are after the best speed - and understand *static* PHP compile builds).
PHP 5.3.x Static Compile guidelines:

In the PHP source root folder, issue the following (Linux);
rm configure
PHP_AUTOCONF=autoconf-2.13 PHP_AUTOHEADER=autoheader-2.13 ./buildconf --force


Then compile PHP manually (or use rpmbuild or similar). I usually work direct with Fedora RPM sources, hence I used; rpmbuild -bb php.spec >> php.checklist 2>&1

Using Unix Sockets with the Drupal Memcache Project:

To use Linux sockets with the Drupal memcache module - please refer to: http://drupal.org/node/538426
This method works well for me on my multi-hosted Linux dedicated server.

Current Changelog for PHP PECL memcache

: from http://pecl.php.net/package-changelog.php?package=memcache&release=3.0.6
3.0.6 - Fixed PECL Bug #16672 (memcache.php doesn't support unix socket)
- Fixed PECL Bug #19374 (memcache.php throws Notice: Undefined index: VALUE when viewing expired items)
- Fixed PECL Bug #17518 (Strange behavior in increment on non integer and after)
- Fixed potential segfault in memcache queue.
- Fixed various tests




Revison #1 Doc's by peter bowey http://drupal.org/user/564804

Peter Bowey Computer Solutions
http://www.pbcomp.com.au/

Memcache in Drupal

The Memcache contributed project provides memcache integration, and support for Drupal's caching, session, and lock backends.

Requirements

There are two different PECL packages supported by the 1.5+ release of the memcache module. If using the older memcache PECL package, 2.2.1 or higher is required to avoid a deletion bug. If using the more modern memcached PECL package, version 1.0 or higher is required.

Installation


These are the broad steps you need to take in order to use this software on Drupal 6.x.Information pertaining to the Drupal 7.x version is also found in the README.txt file from the module distribution.
For 6.x (order is important):
  1. Install the memcached binaries on your server. See How to install Memcache on Debian Etch or How to install Memcache on OSX
  2. Install the PECL memcache extension for PHP.
  3. In php.ini set memcache.hash_strategy="consistent".
  4. Put your site into offline mode.
  5. Download and install the memcache module.
  6. If you have previously been running the memcache module, run update.php.
  7. Start at least one instance of memcached on your server.
  8. Edit settings.php to configure the servers, clusters and bins that memcache is supposed to use. (see code snippet below)
  9. Edit settings.php to include memcache.inc. For example, $conf['cache_inc'] ='sites/all/modules/memcache/memcache.inc';
  10. Bring your site back online.
For 7.x (order is important):
  1. Install the memcached binaries on your server. See How to install Memcache on Debian Etch or How to install Memcache on OSX
  2. Install the PECL memcache extension for PHP. This must be version 2.2.1 or

    higher or you will experience errors.
  3. Put your site into offline mode.
  4. Download and install the memcache module.
  5. If you have previously been running the memcache module, run update.php.
  6. Start at least one instance of memcached on your server.
  7. Edit settings.php to mke memcache the default cache class, for example:
         $conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
         $conf['cache_default_class'] = 'MemCacheDrupal';
  8. Bring your site back online.
For instructions on 1 and 2 above, please see the INSTALLATION.txt file that comes with the memcache module download.

Servers

NOTE: As of 6.15, it is possible to efficiently run memcache in one daemon, shared by all bins. You can safely ignore advice found elsewhere on the internet that advises one memcached daemon per bin. In terms of reporting, there are still some advantages to having more daemons.
If you want the simple version, you can start one default memcache instance on your web server like this: memcached -m 24 -p 11211 -d If that is enough to meet your needs, there is no more configuration needed. If you want to utilize this module's sophisticated clustering feature and spread your cache over several machines, or if your cache is found on a machine other than your web server, read on.
You should probably lock down the memcache server so that it only listens for connections from the hosts that need to be served, as the default is that memcache listens to connections from all addresses. So, to close that hole, edit /etc/sysconfig/memcached with:
OPTIONS="-l ${HOSTIP}"
For example:
OPTIONS="-l 127.0.0.1"
The available memcached servers are specified in $conf in settings.php. If you do not specify any servers, memcache.inc assumes that you have a memcached instance running on 127.0.0.1:11211. If this is true, and it is the only memcached instance you wish to use, no further configuration is required.
If you have more than one memcached instance running, you need to add two arrays to $conf; memcache_servers and memcache_bins. The arrays follow this pattern:
<?php'memcache_servers' => array(host1:port => cluster, host2:port => cluster, hostN:port => cluster)'memcache_bins' => array(bin1 => cluster, bin2 => cluster, binN => cluster)?>
WARNING: Avoid the use of "localhost" and instead use real IP addresses or hostnames that will remain consistent across the network.
The bin/cluster/server model can be described as follows:
  • Servers are memcached instances identified by host:port.
  • Bins are groups of data that get cached together and map 1:1 to the $table param in cache_set(). Examples from Drupal core are cache_filter, cache_menu. The default is 'cache'.
  • Clusters are groups of servers that act as a memory pool.
  • Many bins can be assigned to a cluster.
  • The default cluster is 'default'.
Here is a simple setup that has two memcached instances, both running on 10.1.1.1. The 11212 instance belongs to the 'pages' cluster and the table cache_page is mapped to the 'pages' cluster. Thus everything that gets cached, with the exception of the page cache (cache_page), will be put into 'default', or the 11211 instance. The page cache will be in 11212.
<?php
$conf
= array(
 
'memcache_servers' => array('10.1.1.1:11211' => 'default',
                             
'10.1.1.1:11212' => 'pages'),
 
'memcache_bins' => array('cache_page' => 'pages'),
);
?>
?>
Here is an example configuration that has two clusters, 'default' and 'cluster2'. Five memcached instances are divided up between the two clusters. 'cache_filter' and 'cache_menu' bins go to 'cluster2'. All other bins go to 'default'.
D6:
<?php
$conf
= array(
 
'cache_inc' => './sites/all/modules/memcache/memcache.inc',
 
'memcache_servers' => array('10.1.1.1:11211' => 'default',
                             
'10.1.1.1:11212' => 'default',
                             
'10.1.1.2:11211' => 'default',
                             
'10.1.1.3:11211' => 'cluster2',
                             
'10.1.1.4:11211' => 'cluster2'),

 
'memcache_bins' => array('cache' => 'default',
                          
'cache_filter' => 'cluster2',
                          
'cache_menu' => 'cluster2'),
);
?>


?>
D7:
<?php
$conf
['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';$conf['cache_default_class'] = 'MemCacheDrupal';$conf = array(
 
'cache_default_class' = 'MemCacheDrupal',
 
'memcache_servers' => array('10.1.1.1:11211' => 'default',
                             
'10.1.1.1:11212' => 'default',
                             
'10.1.1.2:11211' => 'default',
                             
'10.1.1.3:11211' => 'cluster2',
                             
'10.1.1.4:11211' => 'cluster2'),

 
'memcache_bins' => array('cache' => 'default',
                          
'cache_filter' => 'cluster2',
                          
'cache_menu' => 'cluster2'),
);
?>

Prefixing

If you want to have multiple Drupal installations share memcached instances, you need to include a unique prefix for each Drupal installation in the $conf array of settings.php:
<?php
$conf
= array(
 
'memcache_key_prefix' => 'something_unique',
);
?>

Sessions

NOTE: Session.inc is not yet ported to Drupal 7.
Here is a sample config that uses memcache for sessions. Note you MUST have

a session and a users server set up for memcached sessions to work.
<?phpinclude_once('./includes/cache.inc');
include_once(
'./sites/all/modules/memcache/memcache.inc');$conf = array(
 
'cache_default_class' = 'MemCacheDrupal',
 
'session_inc' => './sites/all/modules/memcache/memcache-session.inc',
 
'memcache_servers' => array(
   
'localhost:11211' => 'default',
   
'localhost:11212' => 'filter',
   
'localhost:11213' => 'menu',
   
'localhost:11214' => 'page',
   
'localhost:11215' => 'session',
   
'localhost:11216' => 'users',
  ),
 
'memcache_bins' => array(
   
'cache' => 'default',
   
'cache_filter' => 'filter',
   
'cache_menu' => 'menu',
   
'cache_page' => 'page',
   
'session' => 'session',
   
'users' => 'users',
  ),
);
?>

Troubleshooting

PROBLEM: Error: Failed to set key: Failed to set key: cache_page-......

SOLUTION: Upgrade your PECL library to PECL package (2.2.1) (or higher).
PROBLEM: WARNING: Zlib compression at the php.ini level and Memcache conflict.

SOLUTION: See http://drupal.org/node/273824

Memcache Admin

A module offering a UI for memcache is included. It provides stats, a

way to clear the cache, and an interface to organize servers, bins, and clusters.

Memcached PECL Extension Support

We also now support the Memcached PECL extension. If you install this extension,

it will be used by default. This new extension backends to libmemcached and

allows you to use some of the newer advanced features in memcached 1.4.
NOTE: It is important to realize that the memcache php.ini options do not impact

the memcached extension, this new extension doesn't read in options that way.

Instead, it takes options directly from Drupal. Because of this, you must

configure memcached in settings.php. Please look here for possible options:
http://us2.php.net/manual/en/memcached.constants.php
An example configuration block is below, this block also illustrates our

default options. These will be set unless overridden in settings.php.
<?php
$conf
['memcache_options'] = array(
 
Memcached::OPT_COMPRESSION => FALSE,
 
Memcached::OPT_DISTRIBUTION => Memcached::DISTRIBUTION_CONSISTENT,
);
?>
These are as follows:
  • Turn off compression, as this takes more CPU cycles than its worth for most

    users
  • Turn on consistent distribution, which allows you to add/remove servers

    easily
If you are using memcached 1.4 or above, you should enable the binary protocol,

which is more advanced and faster, by adding the following to settings.php:
<?php
$conf
['memcache_options'] = array(
 
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);
?>


Setup for Linux Memcached (daemon) with PHP PECL Memcache and Drupal Memcache Module - with localized UNIX sockets

ICANN using Drupal

ICANN (Internet Committee for Assigned Names and Numbers), the non-profit organization that oversees the use of Internet domains is using Drupal at http://public.icann.org. They are using Mollom too!