Phusion Passenger and memcache-client revisited

March 23, 2009 by Michael

The last Passenger update brought some good explanation off the problems regarding Passenger and memcache-client (see here).

Smart spawning of Passenger processes creates shared file descriptors. As the connections to memcached are sockets they are shared as well so data on them gets corrupted which is explained very nicely in the Passenger documentation: Example 1: Memcached connection sharing (harmful).

The solution presented there works like a charm. The reestablish_connection_to_memcached line is actually not more than CACHE.reset where CACHE is the reference to the memcache connection.

After that change, spawning methods smart-lv2 and smart will work in connection with environment.rb configured memcache connections.

Edit: As requested in the comments, a little example:

memcache_options = {
  :c_threshold => 10000,
  :compression => true,
  :debug => false,
  :namespace => 'some_ns',
  :readonly => false,
  :urlencode => false
}
 
CACHE = MemCache.new memcache_options
CACHE.servers = '127.0.0.1:11211'
begin
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     if forked
       # We're in smart spawning mode, so...
       # Close duplicated memcached connections - they will open themselves
       CACHE.reset
     end
   end
# In case you're not running under Passenger (i.e. devmode with mongrel)
rescue NameError => error
end

In this case, CACHE is the global constant that i use to access my memcache-client.

I guess you’ll need to do the same with the global Rails.cache object, but i’m not sure. Anyway, the above solution works for me.

32 comments

  1. Matt Darby wrote:

    Would you mind sharing the relevant snippet of your environment.rb? I slapped the Phusion-provided code into my app and I was greeted with an internal server error.

    Posted on March 25, 2009 at 4:07 PM | Permalink
  2. Michael wrote:

    Hey Matt, no, not at all. Have a look at the edited post.

    Edit: I looked at your blogpost regarding CacheMoney. The variable in question you need to reset should be $memcache.

    Posted on March 25, 2009 at 4:18 PM | Permalink
  3. Matt Darby wrote:

    Added the the begin/rescue block to my environment.rb (and updated the var you mentioned), and all is working. Hopefully this will finally kill those damned exception notices!

    Thanks for your help!

    Posted on March 25, 2009 at 6:22 PM | Permalink
  4. Matt Darby wrote:

    Hrm, unfortunately I’m still getting the execution expired messages (though, perhaps with less frequency). Just posting back for the next guy 😉

    Posted on March 31, 2009 at 6:05 PM | Permalink
  5. chris wrote:

    Hey Matt, we are getting those “execution expired” messages too, even after Michael’s patch.

    Did you get this resolved?

    Posted on April 8, 2009 at 8:39 PM | Permalink
  6. Matt Darby wrote:

    Unfortunately, I didn’t. In fact, I removed cache-money from my app just this morning. Too many (like 10+/day) exceptions, and it was actually starting to mess with the application itself. Certain portions of the domain logic would get tripped up and half execute as the cache-money gem just threw a wrench in the works.

    Posted on April 8, 2009 at 8:44 PM | Permalink
  7. Michael wrote:

    Hey Matt & Chris,

    the thing that i experienced was total corruption of commands to the memcache server as described in the passenger docs.

    Maybe it’s an internal problem with cache-money…

    Good luck finding a solution!

    Posted on April 8, 2009 at 8:44 PM | Permalink
  8. chris wrote:

    Ug. Did you mention this to the passenger group?
    http://groups.google.com/group/phusion-passenger/

    The list (especially Hongli) is very responsive.

    I am tinkering and will report of any success.

    -Chris

    Posted on April 8, 2009 at 8:46 PM | Permalink
  9. chris wrote:

    By the way, I am using cache_fu, not cache_money, with a recent version of the memcache-client.

    -Chris

    Posted on April 8, 2009 at 8:48 PM | Permalink
  10. Matt Darby wrote:

    @Chris: I haven’t but please feel free 😉
    @Michael: Ahh, I must have misread your OP. Have you not experienced this issue with your app?

    Posted on April 8, 2009 at 8:49 PM | Permalink
  11. Michael wrote:

    Matt: No, i don’t. I use a globally configured cache variable and what i got with phusion and smart spawn in the first iteration was lots of interessting mixed up commands (read/gets in betweens). The cache is heavily used as all requests for images (see http://dailyfratze.de) go through a controller and the cache and the site wasn’t working at all.

    Apart from that i don’t use any additional cache plugin. I have some special caching needs in my app that i implemented myself.
    Also i should say that the build in rails caching works well with memcache and passenger, at last the few fragments i can cache regarding authentication and authorisation.

    The proper event handling as descriped in the op solved at last my problems.

    Posted on April 8, 2009 at 8:54 PM | Permalink
  12. Art wrote:

    I’ve been trying endlessly to get memcached working, but I’m getting bad errors everytime. On my local machine, running just mongrels, it’s working perfectly, so the only difference is that we’re running Passenger and REE on production.

    The memcached logs show no errors, nor does rails, but apache is failing and giving me the following error:

    [Sun May 03 23:14:57 2009] [error] [client 76.239.166.13] Premature end of script headers: amazon, referer: [WEBSITEOMITTEDFORTHISCOMMENT]
    [ pid=5227 file=ext/apache2/Hooks.cpp:546 time=2009-05-03 23:14:57.657 ]:
    Backend process 5255 did not return a valid HTTP response. It returned no data.
    /opt/ruby-enterprise-1.8.6-20090201/lib/ruby/gems/1.8/gems/memcache-client-1.7.2/lib/memcache.rb:335: [BUG] non-initialized struct

    I’ve implemented the above code you’ve posted, but it hasn’t made a difference.

    I’m also using Rails.cache.fetch everywhere, and that’s been working fine. As soon as I switch to CACHE which I set, I get bad command errors in memcached on my local machine using mongrels. Doing that in production yields no different results.

    Is the issue I’m seeing the passenger one that you’re resolving? If so, is there anything special I need to do other than the code above? If the error seems to be a different issue, any ideas of where that can be resolved?

    Stats:
    REE, Rails 2.3.2, Passenger 2.2, memcached, memcache-client gem, SystemTimer gem

    production.rb:

    config.cache_classes = true
     
    # Enable threaded mode
    # config.threadsafe!
     
    # Use a different logger for distributed setups
    # config.logger = SyslogLogger.new
     
    # Full error reports are disabled and caching is turned on
    config.action_controller.consider_all_requests_local = false
    config.action_controller.perform_caching             = true
     
    # cache store
    config.cache_store = :mem_cache_store
    require 'memcache'
    CACHE = MemCache.new '127.0.0.1:11211'
    begin
       PhusionPassenger.on_event(:starting_worker_process) do |forked|
         if forked
           # We're in smart spawning mode, so...
           # Close duplicated memcached connections - they will open themselves
           CACHE.reset
         end
       end
    # In case you're not running under Passenger (i.e. devmode with mongrel)
    rescue NameError => error
    end
     
    # Enable serving of images, stylesheets, and javascripts from an asset server
    # config.action_controller.asset_host                  = "http://assets.example.com"
     
    # Disable delivery errors, bad email addresses will be ignored
    # config.action_mailer.raise_delivery_errors = false
    Posted on May 5, 2009 at 8:41 AM | Permalink
  13. Michael wrote:

    Art, you say that you get the errors in mongrel as well, or didn’t i understand you wrong?

    I once had a similar problem in a custom plugin: Active Record Objects with belongs_to relationships to other objects should be loaded from the cache but it failed. I don’t have the stacktrace anymore but my solution was to just reference the other objects just before loading them like so:

    class Comment < ActiveRecord::Base
    belongs_to :commentable, :polymorphic => true
    end
     
    # Reference possible commentable items
    Blog
    Picture
    # etc.
     
    # load stuff from memcache

    Maybe this helps. The non-initialized struct message has definitely something to do with unmarshalling.

    Posted on May 5, 2009 at 9:08 AM | Permalink
  14. Art wrote:

    Hi Michael,
    mongrels are not giving me any problems, in fact the caching works great when I’m running it locally with caching turned on and memcached.

    When I loaded the application onto my production server, it errored out on that line I mentioned.

    I also understand that problem you’re talking about, where associations are not cached unless they are eager loaded, which isn’t an issue here because the objects I’m caching do not have associations / nor am I referencing any.

    So it seems that using Rails.cache.fetch works great locally, but does not work on a Passenger / REE setup, or so I’ve come to think. The issue you talked about where Passenger is doing smart spawning, might also come into play, but not sure if my error is erroring out on the spawning? I’ve implemented your solution, but the issues have not changed.

    Posted on May 5, 2009 at 9:17 AM | Permalink
  15. Michael wrote:

    Then i misinterpreted your line “As soon as I switch to CACHE which I set, I get bad command errors in memcached on my local machine using mongrels.”

    To rule out smart spawning, try to disable it and use conservative mode like so.

    Apart from that: I have no clue. I wasn’t aware of Rails.cache at the beginning of my app and always used a custom CACHE var and apart from the unmarshalling and spawn problems i had no errors.

    Posted on May 5, 2009 at 9:26 AM | Permalink
  16. Art wrote:

    Hi Michael,

    Thanks for all these quick responses! You actually didn’t misinterpret me, I’m sorry I was just a bit confusing. So to clarify a bit, I have two situations:

    1. Local development
    2. Production hosted on a slice

    Local development:
    – Works great with Rails.cache.fetch, memcached
    – Does not like doing the same cache with CACHE.fetch when I have CACHE set to MemCache.new, gives me a bad command error
    – Locally using mongrels

    Production:
    – Does not work with any combination of the above, errors out with Passenger and I feel like is erroring before it gets to even the caching part

    There’s a few items I think I might be missing:
    1. Do I need to install any other gem other than the memcache-client, SystemTimer, the memcached itself?
    2. Do I need to set the server to something other than localhost / 127.0.0.1 on production, even though the rails, apache, and such are all on the same IP?
    3. config.threadsafe! is currently commented out, does that need to be enabled?
    4. What kind of objects can I store with MemCache? Using Rails.fetch I can store arrays and hashes, but with MemCache.new’s CACHE global, it errors out on those same arrays and hashes (this is when I try this locally).

    Thanks again! you are extremely helpful, I really appreciate it!

    Posted on May 5, 2009 at 9:37 AM | Permalink
  17. Michael wrote:

    Hey Art, thank you very much for your kind comments.

    My setup:
    development
    ruby 1.8.6 (The one from OS X)
    Mongrel
    SystemTimer (1.1.1)
    memcache-client (1.7.2)
    rails (2.3.2)

    productive
    ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux]
    Passenger 2.2.2
    SystemTimer (1.1.1)
    memcache-client (1.7.2)
    rails (2.3.2)

    memcached debian package 1.2.2

    Regarding your items:
    1. see above
    2. if you’re memcache server is on the same host, no. If it is on another host, sure.
    3. only if you use a multithreaded server (i.e. Glassfish with JRuby) i guess.
    4. I throw pretty much everything at it: Hashes, arrays and complex objects.

    I actually have no clue what type Rails.cache is. The CACHE var that i use is a MemCache client instance and it seems to me, it has a bunch of different methods.

    I used caching before rails 2.1, therefore i had my own CACHE var and i was too lazy to rewrite my app and never bothered about Rails.cache. The options i pass to the intrinsic rails cache config are the same that i use to init my own instance so i make sure that the same memcached proc is used.

    Maybe some of the other commenters has an idea what went wrong with your setup.

    Have a great day,
    Michael.

    Posted on May 5, 2009 at 9:53 AM | Permalink
  18. Art wrote:

    That’s interesting, it seems like your setup is almost identical to mine on production.

    I hope it’s ok to ask a few more questions?

    1. Did you have to do any configuration for Passenger / Apache for memcached?
    2. Are you doing a specific require ‘memcache’ call somewhere? If so where?
    3. Where in your environment.rb file are you setting this? I’m putting it in my production.rb file, but just curious if that makes a difference.

    Thanks!!

    Posted on May 5, 2009 at 10:22 AM | Permalink
  19. Art wrote:

    Locally when I tried to run things with CACHE instead of Rails.cache, this happens:

    <22 new client connection
    22 END
    22 END
    22 CLIENT_ERROR bad command line format

    this is followed by the hashed object I tried to store. All I did was change from the Rails.cache to the MemCache version. It’s very strange

    Posted on May 5, 2009 at 10:32 AM | Permalink
  20. Art wrote:

    I figured out why my client_error was happening. The Rails.cache.fetch accepts a hash of parameters, while CACHE.fetch requires just parameters in order.

    Rails.cache.fetch(‘keyname’, :expires_in => 5.seconds) { #block }

    while CACHE.fetch does:
    CACHE.fetch(‘keyname’, 5.seconds) { #block }

    That seemed to do it. Does that sound right?

    I’m still getting strange errors on production though… I will test tomorrow morning.

    Thank you again for your help! I will follow your posts further. I hope to get this production issue resolved soon too, and will post any relevant findings in the comment thread.

    Posted on May 5, 2009 at 10:52 AM | Permalink
  21. Michael wrote:

    Regarding your questions:
    1. No, not apart from that snippet in the original post.
    2. yeah, require ‘memcache’ in environment.rb
    3. I’m creating an option hash in environment.rb, pass it to active record:
    config.cache_store = :mem_cache_store, ‘127.0.0.1:11211’, memcache_options
    then with the same hash, i create my own memcache client like so:
    CACHE = MemCache.new memcache_options
    CACHE.servers = ‘127.0.0.1:11211’

    CACHE is like i said an instance of MemCache, Rails.cache is ActiveSupport::Cache::MemCacheStore, you’ll find the api here

    Posted on May 5, 2009 at 11:01 AM | Permalink
  22. Art wrote:

    Hi Mike,

    Finally got things working. It was a combination of a lot of problems, but the biggest thing was sometimes I was using a response object that was not serializable!

    It took me a long time to figure out, 3 whole days. I wish the error messages were clearer!

    Thank you again, I appreciate all the replies. I can now go and write a monit to take care of memcached.

    Posted on May 6, 2009 at 8:30 AM | Permalink
  23. Art wrote:

    Just wanted to mention a follow up post I made about the issues that we were talking about and how I resolved them: http://kineticac.posterous.com.....e-memcache

    Posted on May 6, 2009 at 9:36 AM | Permalink
  24. Michael wrote:

    Arthur: You’re welcome and i’m glad, your problem is solved. Thanks for nice comment on your blog!

    Cheers,
    Michael.

    Posted on May 6, 2009 at 3:17 PM | Permalink
  25. Alex Farrill wrote:

    Anyone have any insight into using this with Libmemcached as the cache store?
    Thanks,
    Alex

    Posted on July 7, 2009 at 9:48 PM | Permalink
  26. Alex Farrill wrote:

    Am using this:

    Rails.cache.instance_variable_get(‘@cache’).reset

    for my reestablish_connection_to_memcached method

    In case it is useful to anyone else
    peace,
    Alex

    Posted on July 8, 2009 at 9:38 PM | Permalink
  27. Michael wrote:

    Hey Alex,
    sorry that i had no answer for and thank you for your input!
    Cheers,
    Michael

    Posted on July 8, 2009 at 9:42 PM | Permalink
  28. Ian Neubert wrote:

    Thank you for this tutorial, it should be included in the Passenger docs!

    As a side question, does anyone know if you need to go through a similar process on the ActiveRecord connection to the database?

    Thanks!

    Posted on July 23, 2009 at 9:32 PM | Permalink
  29. Michael wrote:

    Hello Ian,
    thank you very much for that nice comment!

    Its actually in the passenger docs…

    Regarding your second question: Connection from the Passenger process to memcache is done via a socket which is actually a file descriptor. This descriptor is copied when forking a new process with shared memory and both write and read to and from the exact same socket and it goes *boom*

    As long as you connect to the database via tcp i’d say you’re safe. I guess that problems will arive, if you use a socket connection to mysql. I have no idea how sqlite connections are handled.

    Posted on July 23, 2009 at 9:37 PM | Permalink
  30. Clive wrote:

    For Rails 2.2.2,reestablish_connection_to_memcached method should be
    Rails.cache.instance_variable_get(’@data’).reset

    Posted on August 7, 2009 at 5:15 PM | Permalink
  31. So it would seem like both the Rails.cache MemCache object and the ActionController::Base.session_options[:cache] (if you’re using memcached) would need to be reset in the block, right?

    http://gist.github.com/174221

    Posted on August 24, 2009 at 11:37 PM | Permalink
  32. Michael wrote:

    Hey Aaron, i think that’s correct.

    Posted on August 25, 2009 at 7:20 AM | Permalink
3 Trackbacks/Pingbacks
  1. […] wait. I’ve written before about problems with memcache-client and Passenger and this problems needs to be addressed in a Rails as well as Rack […]

  2. […] cruise that Michael Simons answered my doubt around his blog post. In essence, this is a known problem that exists between Passenger and […]

  3. […] cruise that Michael Simons answered my doubt around his blog post. In essence, this is a known problem that exists between Passenger and […]

Post a Comment

Your email is never published nor shared. Required fields are marked *