Dalli and Angry TypeErrors - Part 2
Continuation to Dalli and Angry TypeErrors - Part 1
Here is what the stack looks like at the top, without fail, each and every single time:
[GEM_ROOT]/gems/actionpack-3.0.9/lib/action_view/helpers/text_helper.rb:56:in `concat'
[GEM_ROOT]/gems/actionpack-3.0.9/lib/action_view/helpers/text_helper.rb:56:in `safe_concat'
[GEM_ROOT]/gems/actionpack-3.0.9/lib/action_view/helpers/cache_helper.rb:38:in `cache'
Something is returning something funky and its getting typecast wrong. Do some Google research, no one seems to be having this issue. Hmm.
Well, looking at the source of the above Rails 3.0.9 libraries, its clear that the output from the cache retriever method is the trouble.
The dropdead giveaway, check this little trick out:
NoMethodError: undefined method `name' for " <ul class=\"gradient-bg\">\n":String
The HTML snippet you see is the first line of my cache block.
Dalli, Did YOU Do This to Me?
Nope. Dalli is an excellent client. I did this to myself.
A swift modification to config/environments/staging.rb later and look at what I found in my logs:
Rendered /data/my_app/shared/bundled_gems/ruby/1.9.1/bundler/gems/spree-multi-domain-f7099879bef8/app/views/shared/_google_analytics.html.erb (0.0ms)
Rendered shared/_canonical.html.erb (0.1ms)
Rendered shared/_head.html.erb (2.4ms)
[...]
Unexpected exception in Dalli: NoMemoryError: failed to allocate memory
This is a bug in Dalli, please enter an issue in Github if it does not already exist.
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/socket.rb:29:in `kgio_read!'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/socket.rb:29:in `block in readfull'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/socket.rb:28:in `loop'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/socket.rb:28:in `readfull'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/server.rb:373:in `read'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/server.rb:319:in `generic_response'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/server.rb:145:in `get'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/server.rb:45:in `request'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/options.rb:18:in `block in request'
/usr/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/options.rb:17:in `request'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/client.rb:240:in `perform'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/dalli-1.0.5/lib/dalli/client.rb:60:in `get'
(eval):7:in `block in get_with_newrelic_trace'
/data/my_app/shared/bundled_gems/ruby/1.9.1/gems/newrelic_rpm-3.1.0/lib/new_relic/agent/method_tracer.rb:191:in `trace_execution_scoped'
(eval):4:in `get_with_newrelic_trace'
[...]
localhost:11211 is down
localhost:11211 is down: NoMemoryError failed to allocate memory
retrying request with new server
down_retry_delay not reached for localhost:11211 (1.000 seconds left)
[...]
Sending request to http://hoptoadapp.com/notifier_api/v2/notices/:
{?xml version="1.0" encoding="UTF-8"?}
{notice version="2.0"}
{api-key}87405808963579afc7e08a9e1f86cfd4{/api-key}
{notifier}{name}Hoptoad Notifier{/name}{version}2.4.11{/version}
{url}http://hoptoadapp.com{/url}{/notifier}{error}{class}TypeError{/class}{message}TypeError: can't convert Hash into String{/message}{backtrace}
[...]
{/notice}
Look at that. Notice the action continues to process and something completely off is being reported. Identical behavior confirmed with memcache-client. Turns out Dalli is not a part of this problem whatsoever.
Time to simply tweak monit settings down for mongrels and watch my server ram. Simple as that. Until next time.
