Denis 'jumph4x' Ivanov

BatchFactory - A Ruby Gem for Parsing Hashes from Excel Spreadsheets

Heyyo, Find the respository at https://github.com/jumph4x/batch-factory

Scenario

• You maintain a large catalog of products and need to keep prices current or
• You create records from a spreadsheet your boss hands you or
• You have an external service that identifies spammers and you need to get rid of many at a time

Solution

Use tabular input. I chose to support Excel 97-2000 format since there’s a wonderful gem canonically and appropriately called ‘spreadsheet’. Also because the whole world uses MS Office.

Anyways, check out the project README for some quick copypasta for your IRB.

Basically we use the first row as indexed headings\keys and iterate over the consecutive rows, forming a hash from each one. Columns with empty headings are ignored. Basic guard conditions are in place to make sure no empty keys are used and not garbage data is used.

Enjoy :)

Testing Spree 1.0.x Extensions W/ Other Extension Dependencies

With a store of any kind of serious size and functionality, one will indubitably find him\herself pulling in a lot of extension dependencies. At 50K products, we have a dozen or so extensions and there have occurred multiple points of overlap.

This means that Extension A decorated some part of Spree and then Extension B came along and further made things more complex. Much like a git conflict, but will not resolve itself though any kind of clever recursion.

This is where one usually finds him\herself creating internal decorators to combine the functionality of the two decorators.

Technically, the semantics of expected behavior of extensions might change depending on what’s been decorated so far, even if there is no codebase decoration overlap.

Consider: Google Base XML Generation

Very simply, when generating the XML feed for Google Base it is a straightforward process in store with a single externally facing entity. However, once we throw in multiple domains that lead to the same backend, we not might want to generate multiple feeds, each one only including the relevant products.

This is exactly the case when you add spree-google-base to a store running on spree-multi-domain.

Approach

I was tasked with updating both of these extensions to the new Spree 1.0.x patterns. I wanted maximum amount of test coverage within the actual extensions, so I chose to account for this behaviour expectation by writing conditional specs right within spree-google-base itself

example from master

Quick ProTip: conditional testing like this dictates that the dummy application actually bundle and bootstrap the said dependency extension, in our case spree-multi-store. The recipe is as follows:

Gemfile

[...]
gemspec
gem 'spree', '1.0.3'
gem 'spree_multi_domain', :git => 'git://github.com/spree/spree-multi-domain.git'

Also, we now need to copy over the relevant migrations into the dummy app, before it gets migrated:

Rakefile

puts 'Installing spree-multi-domain migrations [required for testing]'
dep_path = `bundle show spree_multi_domain`.chomp
migration_path = File.join(dep_path,'db','migrate')
dummy_path = File.join(FileUtils.pwd, 'spec', 'dummy', 'db')

FileUtils.mkdir_p(dummy_path)
FileUtils.cp_r(migration_path, dummy_path)

Rake::Task['common:test_app'].invoke

Rails Site w/ Old Google Index, Wrong Response Headers: application/rss+xml; charset=utf-8

So we polished up our latest Rails 3 release. It has awesome sitemaps, radical canonicalization, awesome internal link distribution, proper REST. Everything is shiny.

Pulled the trigger, but Google is slow to reindex our content. Hmm, wait 1 week, 2 weeks, 3 weeks. What the hell: indexes all pages but the most important ones - the product browsing pages (Products#index).

What gives?! Boss is getting antsy, pressure is on! Spent all day reading about indexation issues, to no avail. Finally boss calls me over. He has Screaming Frog spider crawler on his screen. The rest of the site responds correctly with ‘text/html; charset=utf-8’, these fuckers however coming up as ‘application/rss+xml; charset=utf-8’.

Bullshit. Open up rails console:

uri = URI('http://www.fcpeuro.com/Volvo-parts/')

request,response = nil

Net::HTTP.start(uri.host, uri.port) do |http|
  request = Net::HTTP::Get.new uri.request_uri
  response = http.request request # Net::HTTPResponse object
end

response.to_hash['content-type'] => 'application/rss+xml; charset=utf-8' # <<< OMFG

Someone is going to fire me. Opening up app/controllers/products_controller.rb#index

respond_to do |format|
  format.rss{ render :action => 'feed', :layout => false}
  format.html
end

See that? First one gets priority. Swap the two lines. Fire up rails console:

uri = URI('http://localhost:3000/Volvo-parts/')

request,response = nil

Net::HTTP.start(uri.host, uri.port) do |http|
  request = Net::HTTP::Get.new uri.request_uri
  response = http.request request # Net::HTTPResponse object
end

response.to_hash['content-type'] => 'text/html; charset=utf-8' # WHEW

Also keep in mind, you can specify a catch-all mime-type responder to prevent rails from firing off blanks with:

respond_to do |format|
  format.all
end

Good luck! And check out the full documentation here, don’t take my word on it: http://apidock.com/rails/ActionController/MimeResponds/InstanceMethods/respond_to

CEO basically saved the day. Not only did he identify the wrong response headers but also had to figure out how to analyze the HTML for invalid format. He ended up showing me a screen where he took two HTML outputs (one correct, one presumably incorrect) and had them broken up in excel and had diff formulas going to identify HTML differences.

TLDR: my boss re-invents diff and finds out about wrong response header. Fix is to order formats within respond_to in descending priority (and have your mimetypes straight to begin with). Issue can be easily confirm through Google Webmaster Tools. Go to ‘Fetch as Googlebot’ and later click on the ‘success’ link.

You will see response headers as well as body as seen by Googlebot.

Ensuring a Trailing Slash in Rails W/Out Dependencies

For some reason this is more difficult in rails than it should be.

The semantic value of the concept of having a trailing slash in the URL is basically one of representing a folder, a virtual directory. In RESTful terms, this represents a collection route, or an :index action. Exhibit A:

GET /products/

This page will list many products as links. In restful terms, this will lead to the :show action. In this instance, we are representing a final destination, a member route. This dictates no trailing slash, semantically speaking:

GET /products/some-permalink
GET /products/some--ther-permalink

So how to deal with this in Rails? Rails will naturally favor slash-less versions of all URLs, unless specified otherwise. Step one is to generate all internal links the correct way, to do this, simply pass the :trailing_slash => true key-value pair to your named route. You ARE using named routes like you should be, right? Good.

member_path(object, :trailing_slash => true) or
collection_path(:trailing_slash => true)

This will generate the correct link. Excellent. However, when you write your routing or functional tests, like you should be, testing the extreme input, you will notice that the slash-less routes work just fine and Rails acts as if everything is normal.

I expected to be able to find a way to deal with this cleanly, by accessing a hash of sorts in the Rails’ request object, but nothing came up. the params hash omits this. As does request.request_uri, request.path and even request.fullpath. Well, shiet, what’s a bro to do?

Well, take a look at my solution in the form of a pair of private methods in ApplicationController:

class ApplicationController < ActionController::Base

private
  def ensure_trailing_slash
    redirect_to url_for(params.merge(:trailing_slash => true)), :status => 301 unless trailing_slash?
  end

  def trailing_slash?
    request.env['REQUEST_URI'].match(/[^\?]+/).to_s.last == '/'
  end
end

Then, when we need to use this in any actual controller, we simple declare the following:

class ObjectController < ApplicationController
  before_filter :ensure_trailing_slash, :only => :index
  [...]
end

Notice I am using 301 redirects here, as a sound SEO measure.

Hope this helped, until next time.

False Positive Tests in Rails, ProTip

False positive tests are a special kind of evil. This is the kind of stuff that will cost you your annual raise.

One of the ways this has recently crept up and done me wrong is very intuitive, yet so hard to find if you look in all the wrong places: caching.

Specifically:

# app/controllers/your_controller.rb

Rails.cache.fetch(you_key) do
  # testable code
end

At this point, make sure your environment is set up to do what you want:

# environments/test.rb

config.action_controller.perform_caching = false

Also take a look at my new and improved test helper. I use TestUnit with Shoulda:

# test/test_helper.rb
class ActiveSupport::TestCase
  setup do      
    Rails.cache.clear
  end
end

class ActionController::TestCase
  setup do      
    Rails.cache.clear
  end
end

Good luck, hope you never trick yourself into false confidence inspiring tests.

Checking for Select.h… Rake Aborted! AKA Wrong Number of Arguments (3 for 2)

Was trying to build Webroar on our virtualized box for internal deployments and ran into this.

Required directories created successfully. Building executables...
checking for select.h... rake aborted!
wrong number of arguments (3 for 2)

An hour later…

This is a rake version mismatch. The stuff expects 0.8.7, I had it installed along with 0.9.2 which was where the executable symlink was pointing to.

gem remove rake -v=0.9.2
gem install rake -v=0.8.7

Doneski.