The last Passenger update brought some good explanation off the problems regarding Passenger and memcache-client (see here).
Smart spawning of Passenger processes creates shared file descriptors. As the connections to memcached are sockets they are shared as well so data on them gets corrupted which is explained very nicely in the Passenger documentation: Example 1: Memcached connection sharing (harmful).
The solution presented there works like a charm. The reestablish_connection_to_memcached line is actually not more than CACHE.reset where CACHE is the reference to the memcache connection.
After that change, spawning methods smart-lv2 and smart will work in connection with environment.rb configured memcache connections.
Edit: As requested in the comments, a little example:
memcache_options = { :c_threshold => 10000, :compression => true, :debug => false, :namespace => 'some_ns', :readonly => false, :urlencode => false } CACHE = MemCache.new memcache_options CACHE.servers = '127.0.0.1:11211' begin PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked # We're in smart spawning mode, so... # Close duplicated memcached connections - they will open themselves CACHE.reset end end # In case you're not running under Passenger (i.e. devmode with mongrel) rescue NameError => error end |
In this case, CACHE is the global constant that i use to access my memcache-client.
I guess you’ll need to do the same with the global Rails.cache object, but i’m not sure. Anyway, the above solution works for me.
32 comments
Would you mind sharing the relevant snippet of your environment.rb? I slapped the Phusion-provided code into my app and I was greeted with an internal server error.
Hey Matt, no, not at all. Have a look at the edited post.
Edit: I looked at your blogpost regarding CacheMoney. The variable in question you need to reset should be $memcache.
Added the the begin/rescue block to my environment.rb (and updated the var you mentioned), and all is working. Hopefully this will finally kill those damned exception notices!
Thanks for your help!
Hrm, unfortunately I’m still getting the execution expired messages (though, perhaps with less frequency). Just posting back for the next guy 😉
Hey Matt, we are getting those “execution expired” messages too, even after Michael’s patch.
Did you get this resolved?
Unfortunately, I didn’t. In fact, I removed cache-money from my app just this morning. Too many (like 10+/day) exceptions, and it was actually starting to mess with the application itself. Certain portions of the domain logic would get tripped up and half execute as the cache-money gem just threw a wrench in the works.
Hey Matt & Chris,
the thing that i experienced was total corruption of commands to the memcache server as described in the passenger docs.
Maybe it’s an internal problem with cache-money…
Good luck finding a solution!
Ug. Did you mention this to the passenger group?
http://groups.google.com/group/phusion-passenger/
The list (especially Hongli) is very responsive.
I am tinkering and will report of any success.
-Chris
By the way, I am using cache_fu, not cache_money, with a recent version of the memcache-client.
-Chris
@Chris: I haven’t but please feel free 😉
@Michael: Ahh, I must have misread your OP. Have you not experienced this issue with your app?
Matt: No, i don’t. I use a globally configured cache variable and what i got with phusion and smart spawn in the first iteration was lots of interessting mixed up commands (read/gets in betweens). The cache is heavily used as all requests for images (see http://dailyfratze.de) go through a controller and the cache and the site wasn’t working at all.
Apart from that i don’t use any additional cache plugin. I have some special caching needs in my app that i implemented myself.
Also i should say that the build in rails caching works well with memcache and passenger, at last the few fragments i can cache regarding authentication and authorisation.
The proper event handling as descriped in the op solved at last my problems.
I’ve been trying endlessly to get memcached working, but I’m getting bad errors everytime. On my local machine, running just mongrels, it’s working perfectly, so the only difference is that we’re running Passenger and REE on production.
The memcached logs show no errors, nor does rails, but apache is failing and giving me the following error:
[Sun May 03 23:14:57 2009] [error] [client 76.239.166.13] Premature end of script headers: amazon, referer: [WEBSITEOMITTEDFORTHISCOMMENT]
[ pid=5227 file=ext/apache2/Hooks.cpp:546 time=2009-05-03 23:14:57.657 ]:
Backend process 5255 did not return a valid HTTP response. It returned no data.
/opt/ruby-enterprise-1.8.6-20090201/lib/ruby/gems/1.8/gems/memcache-client-1.7.2/lib/memcache.rb:335: [BUG] non-initialized struct
I’ve implemented the above code you’ve posted, but it hasn’t made a difference.
I’m also using Rails.cache.fetch everywhere, and that’s been working fine. As soon as I switch to CACHE which I set, I get bad command errors in memcached on my local machine using mongrels. Doing that in production yields no different results.
Is the issue I’m seeing the passenger one that you’re resolving? If so, is there anything special I need to do other than the code above? If the error seems to be a different issue, any ideas of where that can be resolved?
Stats:
REE, Rails 2.3.2, Passenger 2.2, memcached, memcache-client gem, SystemTimer gem
production.rb:
Art, you say that you get the errors in mongrel as well, or didn’t i understand you wrong?
I once had a similar problem in a custom plugin: Active Record Objects with belongs_to relationships to other objects should be loaded from the cache but it failed. I don’t have the stacktrace anymore but my solution was to just reference the other objects just before loading them like so:
Maybe this helps. The non-initialized struct message has definitely something to do with unmarshalling.
Hi Michael,
mongrels are not giving me any problems, in fact the caching works great when I’m running it locally with caching turned on and memcached.
When I loaded the application onto my production server, it errored out on that line I mentioned.
I also understand that problem you’re talking about, where associations are not cached unless they are eager loaded, which isn’t an issue here because the objects I’m caching do not have associations / nor am I referencing any.
So it seems that using Rails.cache.fetch works great locally, but does not work on a Passenger / REE setup, or so I’ve come to think. The issue you talked about where Passenger is doing smart spawning, might also come into play, but not sure if my error is erroring out on the spawning? I’ve implemented your solution, but the issues have not changed.
Then i misinterpreted your line “As soon as I switch to CACHE which I set, I get bad command errors in memcached on my local machine using mongrels.”
To rule out smart spawning, try to disable it and use conservative mode like so.
Apart from that: I have no clue. I wasn’t aware of Rails.cache at the beginning of my app and always used a custom CACHE var and apart from the unmarshalling and spawn problems i had no errors.
Hi Michael,
Thanks for all these quick responses! You actually didn’t misinterpret me, I’m sorry I was just a bit confusing. So to clarify a bit, I have two situations:
1. Local development
2. Production hosted on a slice
Local development:
– Works great with Rails.cache.fetch, memcached
– Does not like doing the same cache with CACHE.fetch when I have CACHE set to MemCache.new, gives me a bad command error
– Locally using mongrels
Production:
– Does not work with any combination of the above, errors out with Passenger and I feel like is erroring before it gets to even the caching part
There’s a few items I think I might be missing:
1. Do I need to install any other gem other than the memcache-client, SystemTimer, the memcached itself?
2. Do I need to set the server to something other than localhost / 127.0.0.1 on production, even though the rails, apache, and such are all on the same IP?
3. config.threadsafe! is currently commented out, does that need to be enabled?
4. What kind of objects can I store with MemCache? Using Rails.fetch I can store arrays and hashes, but with MemCache.new’s CACHE global, it errors out on those same arrays and hashes (this is when I try this locally).
Thanks again! you are extremely helpful, I really appreciate it!
Hey Art, thank you very much for your kind comments.
My setup:
development
ruby 1.8.6 (The one from OS X)
Mongrel
SystemTimer (1.1.1)
memcache-client (1.7.2)
rails (2.3.2)
productive
ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux]
Passenger 2.2.2
SystemTimer (1.1.1)
memcache-client (1.7.2)
rails (2.3.2)
memcached debian package 1.2.2
Regarding your items:
1. see above
2. if you’re memcache server is on the same host, no. If it is on another host, sure.
3. only if you use a multithreaded server (i.e. Glassfish with JRuby) i guess.
4. I throw pretty much everything at it: Hashes, arrays and complex objects.
I actually have no clue what type Rails.cache is. The CACHE var that i use is a MemCache client instance and it seems to me, it has a bunch of different methods.
I used caching before rails 2.1, therefore i had my own CACHE var and i was too lazy to rewrite my app and never bothered about Rails.cache. The options i pass to the intrinsic rails cache config are the same that i use to init my own instance so i make sure that the same memcached proc is used.
Maybe some of the other commenters has an idea what went wrong with your setup.
Have a great day,
Michael.
That’s interesting, it seems like your setup is almost identical to mine on production.
I hope it’s ok to ask a few more questions?
1. Did you have to do any configuration for Passenger / Apache for memcached?
2. Are you doing a specific require ‘memcache’ call somewhere? If so where?
3. Where in your environment.rb file are you setting this? I’m putting it in my production.rb file, but just curious if that makes a difference.
Thanks!!
Locally when I tried to run things with CACHE instead of Rails.cache, this happens:
<22 new client connection
22 END
22 END
22 CLIENT_ERROR bad command line format
this is followed by the hashed object I tried to store. All I did was change from the Rails.cache to the MemCache version. It’s very strange
I figured out why my client_error was happening. The Rails.cache.fetch accepts a hash of parameters, while CACHE.fetch requires just parameters in order.
Rails.cache.fetch(‘keyname’, :expires_in => 5.seconds) { #block }
while CACHE.fetch does:
CACHE.fetch(‘keyname’, 5.seconds) { #block }
That seemed to do it. Does that sound right?
I’m still getting strange errors on production though… I will test tomorrow morning.
Thank you again for your help! I will follow your posts further. I hope to get this production issue resolved soon too, and will post any relevant findings in the comment thread.
Regarding your questions:
1. No, not apart from that snippet in the original post.
2. yeah, require ‘memcache’ in environment.rb
3. I’m creating an option hash in environment.rb, pass it to active record:
config.cache_store = :mem_cache_store, ‘127.0.0.1:11211’, memcache_options
then with the same hash, i create my own memcache client like so:
CACHE = MemCache.new memcache_options
CACHE.servers = ‘127.0.0.1:11211’
CACHE is like i said an instance of MemCache, Rails.cache is ActiveSupport::Cache::MemCacheStore, you’ll find the api here
Hi Mike,
Finally got things working. It was a combination of a lot of problems, but the biggest thing was sometimes I was using a response object that was not serializable!
It took me a long time to figure out, 3 whole days. I wish the error messages were clearer!
Thank you again, I appreciate all the replies. I can now go and write a monit to take care of memcached.
Just wanted to mention a follow up post I made about the issues that we were talking about and how I resolved them: http://kineticac.posterous.com.....e-memcache
Arthur: You’re welcome and i’m glad, your problem is solved. Thanks for nice comment on your blog!
Cheers,
Michael.
Anyone have any insight into using this with Libmemcached as the cache store?
Thanks,
Alex
Am using this:
Rails.cache.instance_variable_get(‘@cache’).reset
for my reestablish_connection_to_memcached method
In case it is useful to anyone else
peace,
Alex
Hey Alex,
sorry that i had no answer for and thank you for your input!
Cheers,
Michael
Thank you for this tutorial, it should be included in the Passenger docs!
As a side question, does anyone know if you need to go through a similar process on the ActiveRecord connection to the database?
Thanks!
Hello Ian,
thank you very much for that nice comment!
Its actually in the passenger docs…
Regarding your second question: Connection from the Passenger process to memcache is done via a socket which is actually a file descriptor. This descriptor is copied when forking a new process with shared memory and both write and read to and from the exact same socket and it goes *boom*
As long as you connect to the database via tcp i’d say you’re safe. I guess that problems will arive, if you use a socket connection to mysql. I have no idea how sqlite connections are handled.
For Rails 2.2.2,reestablish_connection_to_memcached method should be
Rails.cache.instance_variable_get(’@data’).reset
So it would seem like both the Rails.cache MemCache object and the ActionController::Base.session_options[:cache] (if you’re using memcached) would need to be reset in the block, right?
http://gist.github.com/174221
Hey Aaron, i think that’s correct.
3 Trackbacks/Pingbacks
[…] wait. I’ve written before about problems with memcache-client and Passenger and this problems needs to be addressed in a Rails as well as Rack […]
[…] cruise that Michael Simons answered my doubt around his blog post. In essence, this is a known problem that exists between Passenger and […]
[…] cruise that Michael Simons answered my doubt around his blog post. In essence, this is a known problem that exists between Passenger and […]
Post a Comment