Thursday, April 12, 2012

Saving your Ass-et Pipeline

Getting your Ass-ets in Gear

The Rails 3.1+ Asset Pipeline is a thing of beauty. Not only does it greatly improve the ability to cache forever these often expensive assets (javascript, css, images), but it compresses and delivers them far more efficiently.

It can also be a pain in the ass-et to ensure that the world does not come crashing down when you push your newly upgraded Rails app to production.

One of the great distinctions between previous versions of Rails and 3.1 is the high likelihood that though your development environment and test environment run without error, when it arrives in a production environment, the same code will fail.

This can be mostly blamed on pre-compilation. Ryan Bates does a great job of discussing this in a recent Railscast. He explains how to ensure that your capistrano scripts include steps to run pre-compile on your production server, or run it locally and send the results up with the rest of your code.

But there are other issues beyond getting production to pre-compile. You must also ensure that all of the javascript and css files you reference are set to be pre-compiled.

By default, Rails will pre-compile all non-CSS, non-JSS files, and all application.js and application.css files. But let's say you have something like this in your code:

- content_for :scripts do
  = javascript_include_tag 'signup_click'

Signup_click.js will not be pre-compiled.  So what, you say. Well, when a request is made to serve the signup_click.js file, you will get a nasty "signup_click.js isn't precompiled" error. And you will get this for any file that isn't named application.* and that is referred to directly by your erb/haml templates. If you are migrating up from 3.0, this can be a real gotcha.

So what to do.

Certainly, you should take some time to review your code, and look for specific references to css and js files and, if you can, refer to them in the application.* manifests. Invariably, you will miss some (I sure did).

So to catch them, I first attempted to run my application in production mode locally (Ryan Bates demonstrates how to do this). If you have a large site, that you have been building for a while (say, since 2.3.5), this could take a while to go through page by page.

Instead, or in addition really, I set up my cucumber environment to run in a simulated production environment. This required  a couple steps:
  1. Make sure that running all cucumber features in test mode returned no errors
  2. I added ENV["RAILS_ENV"] = "cucumber" to the top of my features/support/env.rb file (actually, within my Spork.prefork section - for those using Spork)
  3. I created a cucumber.rb environment file, copying everything from production, with the following changes. 
    1. config.serve_static_assets = true
    2. config.consider_all_requests_local       = true  
    3. config.action_controller.perform_caching = false
    4. Don't forget to disable mail 
    5. Make sure your config.assets.precompile matches production's
  4. I ran 'rake assets:precompile"
When I ran my tests, I got several failures,  all related to assets that were not precompiled. It was easy for me to find, and fix. In some cases, they were javascript/stylesheet includes/links that were redundant to what was in my application manifests, so I could just remove the include/link statements.

In other cases, they were controller specific css or js that I did not want to put into my main application manifests. At first, I included them in the config.assets.precompile statement in my production.rb environment file:

config.assets.precompile += %w( signup_click.js )

But after starting to load up this statement, I wanted to come up with a better, more flexible way.


Do these pants make me look fat (Ass-et reorganization)

Here is what I wanted to do:
  1. Not have to worry about adding files to the pre-compile list (because I know I will forget, and may not realize it until I get a production error notification)
  2. Be able to include controller specific stylesheets and javascript whenever I wanted. 
Knowing that anything listed underneath the app/assets folder named application.* would be precompiled, gave me the idea.

For each controller that I wanted a special file included when those pages were displayed, I added a folder named for the controller, and an application.css (or css.scss, or js, or js.coffee, or js.coffee.erb...but you get the picture) within that folder. I believe rails scaffolding does some of this for you, but I use rbates Nifty Scaffold, which does not.

So my file structure could look like this:
app
  assets
    javascript
      application.js
      blogs
        application.js
    stylesheets
      application.css
      blogs
        application.css

In my layout file (after the link/include tags for the base application files), I added the following lines:
= stylesheet_link_tag "#{controller_name}/application"
= javascript_include_tag "#{controller_name}/application"

This seemed to work great in development and test, but when I got to production, it failed whenever it encountered a <controller_name>/application.(js|css) that did not exist. What to do.

It occurred to me that I don't want these tags to be present if there is not an accompanying file. So I used the following helper methods to check whether the file exists.

def asset_exists?(filename, extension)
    asset_pathname = "#{Rails.root}/app/assets/"
    asset_file_path = "#{asset_pathname}/#{filename}".split(".")[0]
    !Dir.glob("#{asset_file_path}.#{extension}*").empty?
  end
  
  def js_asset_exists?(filename)
    asset_exists?("javascripts/#{filename}", "js") 
  end
  
  def css_asset_exists?(filename)
    asset_exists?("stylesheets/#{filename}", "css")
  end

and changed my include/link tags to the following:

= stylesheet_link_tag "#{controller_name}/application" if css_asset_exists?("#{controller_name}/application.css")    
= javascript_include_tag "#{controller_name}/application" if js_asset_exists?("#{controller_name}/application.js")

This worked like a charm. Whenever I wanted, I could add controller specific css and js by adding the folder and the appropriate file. I could include other files in their manifests, and/or directly add code.

If need be, a similar approach could be used for action specific files (I haven't needed to, but certainly could see doing so).

I am sure there are more elegant approaches, but this works well for me.

Any who, hope you found this helpful. And as my mother used to say "An ass-et in the hand, is better than two in the bush"

Thursday, April 5, 2012

Gems/Services to explore

Paper_trail - Versioning for database models
Globalization 3 - Multi-language translations of models
CopyCopter - User editing of web page text
Refinery - CMS engine

http://www.clickdesk.com/ - Chat and phone support

http://ckrack.github.com/fbootstrapp/ - Face Book iframe page using bootstrap

http://digg.com/newsbar/topnews/going_all_in_how_to_run_a_company_on_21_apps_in_the_cloud - List of 21 cloud based apps to run your business by

https://github.com/pokonski/public_activity = For tracking and displaying user activity

awesome_nested_fields

Wednesday, April 4, 2012

From heaven to hell and back again with Rails, Ruby, Rack, Capybara and sundry others

A few days ago, I started getting an odd error message while running some cucumber features. In a nutshell, the param values returned from a multiple choice selection box arrive as an array of arrays, instead of just an array.

"category_ids"=>["[\"\"]"] vs "category_ids"=>[""]


I posted an issue on the capybara git site. After a brief back and forth, I was graciously informed that I was running older versions of rack (using 1.2.5, current version 1.4.1), and that was likely the problem.

So off I go to upgrade rack. Unfortunately, rack 1.2.5 is a dependency of rails 3.0.11, which is what we are currently running.

Perhaps it is time to upgrade to rails 3.2?

But that requires ruby 1.9.3, which fails to install with rvm on a mac. Apparently, I need an updated version of Xcode. Or, I could use OSX GCC Installer.

If I was running Lion, I could have used Apple's newly released Command Line Tools for Xcode as described by Kenneth Rietz, but alas, I have not yet made the upgrade.

But after installing OSX GCC Installer, and following the recommendation that I delete the old Xcode, I got the following error when attempting to run 'rvm install 1.9.3"

/System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:70:35: error: SecurityHI/SecurityHI.h: No such file or directory

So perhaps the OSX GCC Installer doesn't have SecurityHI support.

Adventures in moving from Rails 2.3.14 to 3.0: Controllers

Using textmate find and replace in project, changed 'integrate_views' to 'render_views'

Added a config.ru file with the following:

require ::File.expand_path('../config/environment',  __FILE__)
run MyApp::Application

Dealing with has_tag? undefined method error in controller specs
In Rspec 2, it is recommended to replace has_tag with has_selector? Unfortunately, this doesn't work with view specs.

I added the gem rspec-tag_matchers. In my spec_helper, I added a config.include(RSpec::TagMatchers) line. Has_tag? is now working

my .ackrc file. Adding Ruby on Rails specific files to ack

--type-add=ruby=.haml,.rake,.rsel,.feature
--type-add=css=.scss
--type-add=css=.less
--type-add=js=.coffee
--type-add=rack=.ru


mate ~/.ackrc to open the file

Monday, March 26, 2012

Converting CSS background images to use asset pipeline

In order to use the asset pipeline in Rails 3.1, not only do you have to move your assets (images, css, javascript, fonts) to the appropriate folders in the assets folder, but you also have to modify the css files so that background images point to the new location.

To make this easier, I used a couple of regex replaces to swap out the old formats for the new format.

For example, I had the following css class in one of my css files.
.standard_form form fieldset input.submit_button {
    background-image: url('../images/common/donate_now_new.png');
    width: 202px;
    height: 50px;
    border: none;
    color: white;
    font-size: 19px;
    text-shadow: 2px 2px 2px #666666;
    font-weight: bold;
    cursor: pointer;
}
I needed to convert the backround-image line to:

image-url('common/donate_now_new.png')

I used the following regex search and replace strings to do so:

Search: (url)\('(\.\.\/images\/)(.*)'\)
Replace: image-url('$3')


I also had background-image statements with the following format: url(../../images/common/input_bg2.png)

I used the following regex to make the replacement for these

Search: (url)\((\.\.\/\.\.\/images\/)(.*)\)
Replace: image-url('$3')

Once I had completed all of the replacements, I changed the file name, adding scss to the end to ensure it would be compiled and the image-url method would substitute the appropriate address for me. 

Wednesday, March 7, 2012

Adventures in moving from Rails 2.3.14 to 3.0: Controller Spec changes

I use integrate_views (or now render_views) when testing controllers and then, for the most part, don't do separate view tests.

A good portion of my tests check whether a particular element appears on the page. I use have_tag to do this. The rspec-html-matchers gem is a great replacement for the have_tag functionality that was removed from Rspec 2. I wish I had found it before changing many of my have_tag statement to have_selector.

In any case, there are some formatting differences between the new and the old have_tag matcher.

Using css matching
Using the CSS matching with have_tag did not previously require quotes around the value portion of an identifier. For example you could write  have_tag("div[id=my_id]") and it would work. Now, you have to place quotes around the my_id portion, or have_tag("div[id='my_id']"). 

To make this easier, I wrote a regular expression to search for quoteless have_tag matchers and insert the quotes. 

Regex: \[(id|class|href|type)=([^\[^'.]*)\]
Replace: [$1='$2']
Using Textmates find and replace, with the regular expressions checkbox checked, made it relatively pain free to do the replacements. 

Content within a tag
Have_tag used to except a string as a second parameter. The string represents content to search for within the tag. For example have_tag("a[href='people/4']", "John Doe") would search for a link to the 'people/4' path containing the text 'John Doe'.

Though this may still work in some instances, I found that a more consistent way of representing this is to use a :text qualifier: have_tag("a[href='people/4']",  :text => "John Doe"). Use regular expressions I was able to make the following replacement:

Regex: have_tag\((.*),(?: *)(?:"|'|/)(.*)(?:"|'|/)\)
Replace: have_tag($1, :text => /$2/)

This also then used regular expression for the match, instead of straight text.

Matching against response.body, not response
It used to be that you could match the have_tag against the response. But now that causes the following error:
  

     Failure/Error: response.should have_tag("div#group_list") do
     NoMethodError:
       undefined method `current_scope' for #

The have_tag matcher must be applied to the response.body, as in:

response.body.should have_tag("div#group_list")

Issues encountered when upgrading from rails 3.0.11 to 3.1.7

Here are a few things I encountered when making the upgrade.

Once I had changed the entry in gemfile to gem rails '3.1.7' and performed a 'bundle update rails', I ran a single cucumber feature, and got the following error:

/Users/user/.rvm/gems/ruby-1.9.2-p290@givecorps_rails_3_1/gems/activerecord-3.1.4/lib/active_record/attribute_methods/read.rb:91: syntax error, unexpected keyword_end
  /Users/user/.rvm/gems/ruby-1.9.2-p290@givecorps_rails_3_1/gems/activerecord-3.1.4/lib/active_record/attribute_methods/read.rb:93: syntax error, unexpected $end, expecting keyword_end (SyntaxError)
  ./features/step_definitions/model_steps.rb:9:in `block in build_models_from_table'
  ./features/step_definitions/model_steps.rb:8:in `each'
  ./features/step_definitions/model_steps.rb:8:in `build_models_from_table'
  ./features/step_definitions/model_steps.rb:17:in `/^the following ([^"]*):$/'
  features/admin_assigns_categories_to_causes.feature:8:in `Given the following cause:'

It turns out, this was caused by incompatibilities between rails 3.1 and the mysql2 gem. So I modified the gemfile with gem mysql2 '~> 0.3', and ran bundle update mysql2.

Running the feature again, I got:

Please install the mysql2 adapter: `gem install activerecord-mysql2-adapter` (can't activate mysql2 (~> 0.3.10), already activated mysql2-0.3.7. Make sure all dependencies are added to Gemfile.) (RuntimeError)

It turns out, when I had first run bundler to update mysql2, version 0.3.7 was installed. Rerunning bundler install got version 0.3.11, and the mysql2 adapter issue went away, and the feature ran.

I then started the server, and got the following error:

undefined method `debug_rjs=' for ActionView::Base:Class (NoMethodError)

I had to remove from development.rb the following:

config.action_view.debug_rjs             = true

I was then able to start the server.

Hitting the first page, however, I got the following error:

NoMethodError (undefined method `each' for nil:NilClass):
  

 Rendered /Users/user/.rvm/gems/ruby-1.9.2-p290@givecorps_rails_3_1/gems/actionpack-3.1.4/lib/action_dispatch/middleware/templates/rescues/_trace.erb (1.0ms)
 Rendered /Users/user/.rvm/gems/ruby-1.9.2-p290@givecorps_rails_3_1/gems/actionpack-3.1.4/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (18.0ms)
 Rendered /Users/user/.rvm/gems/ruby-1.9.2-p290@givecorps_rails_3_1/gems/actionpack-3.1.4/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (22.0ms)

This led to a discussion on rails and rack about ill-formed cookies that as of today, does not seem to be resolved.

So I cleared my browsers cookies, and was able to view the site. Hopefully, this won't be an issue on the production server.

Browsing the site, I got the following error:
undefined method `paginate' for #

I had updated to the rails 3 version of will_paginate. This required me to modify the syntax used in calling paginate, and to include the 'paginate/array' module since some of what I paginate are returned arrays (mostly to ensure uniqueness)

Friday, February 10, 2012

From file_column to paperclip

At the time ClassroomParent was started, the file_column plugin was the state of art for uploads. Not only did it integrate nicely with rmagick, but there was also a bridge for active_scaffold.

But file_column has been left behind by many excellent plugins/gems, most notably, paperclip.

In 2008, Mark Cornick wrote an excellent blog on how to migrate from file_column to paperclip. For the most part, I have followed his instructions, though I split his single migration into several. 


One major hurdle I had to overcome was that my file storage path is dynamically generated. This is so I can better segregate one school's assets from another. 


Looking at the paperclip documentation, they state that the attributes associated with has_attached_file can accept a lambda, so I tried this on the :path attribute:


  :path => lambda { |attachment| "/system/#{attachment.instance.sub_domain}/:class/:id/:basename.:extension"


This worked for the path value, but when I tried to access the url value, I got the following error:


NoMethodError: undefined method `gsub' for #<Proc:0x000001058f79a0>
    from /Users/user/.rvm/gems/ruby-1.9.2-p180@classroom_parent/gems/paperclip-2.5.2/lib/paperclip/interpolations.rb:33:in `block in interpolate'
    from /Users/user/.rvm/gems/ruby-1.9.2-p180@classroom_parent/gems/paperclip-2.5.2/lib/paperclip/interpolations.rb:32:in `each'



Looking at the paperclip code, I could see the offending gsub statement, and it did not look easy to work around it. 


However, there was a little note, that I don't believe is echoed in the paperclip gems readme, that states that a symbol can be passed to the path attribute that references an instance method. Moving my code from the lambda to a public private method, and using that method name as the value passed to the path attribute worked like a charm. 


  :path => :path_to_file


  def path_to_file
    return "/system/#{sub_domain}/:class/:id/:basename.:extension"
  end


private
  def sub_domain
    ...code to get to the sub_domain of the school
  end

   

European dates in ruby 1.9.2

Date parse in Ruby 1.9.2 expects that dates should be in the european format of dd/mm/yyyy. Unfortunately, this meant that my month and days became reversed.

Troyk posted a gist with a solution.

Create an initializer (ruby_date_parse_monkey_patch.rb) with the following:

# Date.parse() with Ruby 1.9 is now defaulting to the European date style where the format is DD/MM/YYYY, not MM/DD/YYYY
# patch it to use US format by default
class Date
  class << self
    alias :euro_parse :_parse
    def _parse(str,comp=false)
      str = str.to_s.strip
      if str == ''
        {}
      elsif str =~ /^(\d{1,2})[-\/](\d{1,2})[-\/](\d{2,4})/
        year,month,day = $3.to_i,$1,$2
        date,*rest = str.split(' ')
        year += (year < 35 ? 2000 : 1900) if year < 100
        euro_parse("#{year}-#{month}-#{day} #{rest.join(' ')}",comp)
      else
        euro_parse(str,comp)
      end  
    end
  end
end

Thursday, February 9, 2012

Adventures in moving from Rails 2.3.14 to 3.0: ActionMailer changes

So many deprecation notices, so many things to fix, when it comes to mailers.

After getting some of my model specs to run without issue, I decided to focus on the my mailer specs, as I new there were many things that had to change.

In the related mailer_spec

Assert_select and assert_select_email instead of have_tag, have_text
The first mailer spec I ran, everything blew up, because I had been using have_tag and have_text as matchers. I could not figure out how to get them to work in Rails 3, so switched to using assert_select and assert_select_email.

However, in rails 3.0, assert_select_email has a couple of problems. First, it assumes that emails will be multipart. This wasn't true in my case. It also assumes that email.body returns a string, which it does not, it returns a Body object.

There is a fix in rails 3.1 for this, so I borrowed that code and created an initializer with the following:

module ActionDispatch
  module Assertions
    module SelectorAssertions
      # this is fixed in Rails 3.1
      # in 3.0, if the email has no parts, it would always return true
      # also, the body needed to be converted to a string, as that is what HTML::Document.new expects
      def assert_select_email(email = nil, &block)
        deliveries = email ? [email] : ActionMailer::Base.deliveries
        assert !deliveries.empty?, "No e-mail in delivery list"

        for delivery in deliveries
          for part in (delivery.parts.empty? ? [delivery] : delivery.parts)
            if part["Content-Type"].to_s =~ /^text\/html\W/
              root = HTML::Document.new(part.body.to_s).root
              assert_select root, ":root", &block
            end
          end
        end
      end
    end
  end
end

Unlike the fix rails 3.1 fix (https://github.com/rails/rails/pull/2499), this also allows you to pass in an email object. Without this, if assert_select_email found more than one email in the deliveries list, and one of them didn't match your select criteria, this would fail, even if other emails in the deliveries list matched.

By passing in an email, you don't leave that to chance.

Changing all of the have_tag, with_tag, etc to assert_select statements was a bit of a pain, but it works well now. 

Change have_text to have_body_text

Move "deliver_" at the front of the mailer method call to ".deliver" at the end of the mailer method call

In the mailer.rb file

Change @body[:foo] = 'bar' assignments to @foo = 'bar'
To change the @body[:foo] = 'bar' assignment to @foo = 'bar', I used the following in TextMates Find in Project
Find: \@body\[:([a-z_1-9]*)\]
Replace: @$1
Use Regular Expressions: true

Adventures in moving from Rails 2.3.14 to 3.0: Has_many through


There have been many blogs about moving from Rails 2.3x to 3.0. Kir Maximov has a good one. And there is the Simone Carletti's blog which gives some pre-migration advice.

I am going to focus on the things that I encountered that I could not find mentioned elsewhere, and may be peculiar to my application.

Each day that I work on the migration, I am going to post what I encountered. Mostly, this will help me if I ever have to do another migration. And it keeps me off the streets.


Issue with has_many through
I have the following has_many through relationship
class Parent
  has_many :relatives
  has_many :students, :through => :relatives
end

class Relative
  belongs_to :parent
  belongs_to :student
end

In the parent class, I had the following named_scope:

named_scope :in_school_year, lambda{|s,y|
    {:include => {:students => {:homerooms => :classroom}}, 
    :conditions => ['classrooms.year = ? and classrooms.school_id = ?', y, s.id]}
  } 


In 2.3.x, this worked fine. But in 3.0, this created a query where the relatives table was not being joined to the students table, so was returning all students.

I changed the scope in the rails 3 version of my app to the following, and it worked fine.

scope :in_school_year, lambda{|s,y|
    {:include => {:relatives => {:student => {:homerooms => :classroom}}}, 
    :conditions => ['classrooms.year = ? and classrooms.school_id = ?', y, s.id]}
  } 

Perhaps it never should have worked in rails 2.3, but it did.