Just Enough Developed Infrastructure

Rails Syndrome: It works on my PC but I don't know for how long

Rails projects tend to suffer from the 'it works on my PC syndrome'. People seem to struggle with keeping their application environment under control so it can be reproduced on another system. One of the reasons is that rails and ruby evolve fast to new concepts and ideas. While this is clearly a good thing, it does require additional steps to keep a rails project under control. There exist a lot of documentation in books and blogs, but that also suffers from the same problem: the recipe working for version X of rails, might not work anymore in the new version.
This document will try to document how to control it, additionally listing older approaches and indicating that they have become deprecated.
NOTE:

  • We assume that you have a working version of ruby and ruby-gems installed
  • This document uses unix commands useful for linux or macosx variants, Windows users should use their equivalent
  • This is work in progress. But I found it more important to get feedback before finishing the whole project
STEP 0: Create an initial directory structure for our projects let's make the project structure we would like
projects

  • project1 ....
  • projectX projectX.ENV +gem (where our project gems go) +rails (where our rails code goes)
    export PROJ_NAME=projectX
    export PROJ_PATH=pwd/$PROJ_NAME
    cd $PROJ_NAME
    STEP 1: control your RUBY The first thing to decide is the ruby version you will use. This is usually a step to people tend to skip, they will use whatever ruby that is in the path Lets see where the ruby binary is installed
    RUBY_BINARY=which ruby
    echo $RUBY_BINARY
    RUBY_VERSION=ruby --version
    echo $RUBY_VERSION
    To fix the ruby version you want to use the environment variable RUBY_HOME If this variable is set, everything will be executed from this environment variable
    export RUBY_HOME='/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr'
    echo $RUBY_HOME
    export PATH=$RUBY_HOME/bin:$PATH
    Let's write this variable to a file so that we can set it again everytime we need it
    echo "export RUBY_HOME=$RUBY_HOME" >> ENV.$PROJ_NAME
    STEP 2: control your RAILS Just as we decided which version of ruby, we have to do the same with our rails version Again in most documents you will find a 'gem install rails' installing gems either in the system environment or the per user environment @REF: http://blog.divergentsoftware.com/2007/11/gemhome-is-where-partys-at.html @REF: https://we.riseup.net/rails/subversion-for-rails If you want to protect this project from other gem installs and gem updates we can dedicate a directory for this The path where gem install writes it's gems is in GEM_HOME
    export GEM_HOME=$PROJ_PATH/gem
    mkdir $GEM_HOME
    The path where the are read is in GEM_PATH. We set this to be excluvely to the gems in this directory. So no use of other f.i. system gems
    export GEM_PATH=$GEM_HOME
    Let's write this again to our ENV file
     echo "export GEM_HOME=$GEM_HOME" >> ENV.$PROJ_NAME
    echo "export GEM_PATH=$GEM_PATH" >> ENV.$PROJ_NAME
    Let get the version we want
    RAILS_VERSION="2.1"
    gem install rails -v $RAILS_VERSION
    let's check if the files are indeed installed in the right place
    ls $GEMHOME/
    To check the path of the installed versions
    gem list -d
    STEP 3: let's setup a subversion to check our code in @REF: https://wincent.com/wiki/Checking_a_new_Rails_application_into_an_existing_Subversion_repository Chances are you have one allready but, this should show the basic setup of setting up yourself The place where you store your repository = REPO_PATH
    REPO_PATH=$HOME/subversion-repository
    #REPO_URL="file://"$REPO_PATH/$PROJ_NAME
    REPO_URL="svn://localhost/"$PROJ_NAME
    Go within the repository path
    cd $REPO_PATH
    and create the svn repository for our project = $PROJ_NAME
    svnadmin create ./$PROJ_NAME
    Add some userT with password passwordT
    export SVN_USER="userT"
    export SVN_PASSWORD="passwordT"
    echo "$SVN_USER = $SVN_PASSWORD" >> $REPO_PATH/$PROJ_NAME/conf/passwd
    echo "auth-access = write" >> $REPO_PATH/$PROJ_NAME/conf/svnserve.conf
    echo "password-db = passwd" >> $REPO_PATH/$PROJ_NAME/conf/svnserve.conf
    echo "realm = Project $PROJ_NAME" >> $REPO_PATH/$PROJ_NAME/conf/svnserve.conf
    Start the listener to listen on the network
    svnserve -d -r $REPO_PATH
    This command will create a trunk, tags, branches directory and commit the version of the project
    svn --username=$SVN_USER --password=$SVN_PASSWORD mkdir --message="standard project start" "$REPO_URL/trunk" "$REPO_URL/branches" "$REPO_URL/tags"
    STEP 4: Create our first rails skeleton for the application Let's go to the our project directory we created in STEP 0
    cd $PROJ_PATH
    Remember to set your RUBY_HOME, GEM_HOME, GEM_PATH We can use our project ENV to set things up again
    . ENV.$PROJ_NAME
    Within our PROJ_PATH we will checkout the trunk to the rails directory Remember that REPO_URL includes PROJ_NAME (we assume that each project will have it's own trunk, tags, release)
    svn checkout "$REPO_URL/trunk" rails
    cd rails
    Set the database type (DB_TYPE) we would like to use
    DB_TYPE=mysql
    rails . -d $DB_TYPE
    Take the generated and add it to the files to check in
    svn add --force .
    BUT: we don't want everything to be checked in so we need to set a set of ignores @REF = Addison Wesley - The Rails Way, 2008 : page 811
    We don't want to have .DS_Store files to be checked in
    svn propset svn:ignore -R ".DS_Store" . --force
    We don't want to have the log files to be checked in
    svn revert log/
    svn propset svn:ignore ".log" log
    We don't want to have generated documentation to be checked in
    svn propset svn:ignore "" doc
    @REF = http://www.sepcot.com/blog/2008/04/svn-rake-tasks-1 We don't want to have generated documentation to be checked in
    svn propset svn:ignore "" doc/api
    svn propset svn:ignore "" doc/app
    @REF = Addison Wesley - The Rails Way, 2008 : page 811
    svn propset svn:ignore "" tmp
    svn propset svn:ignore "" tmp/sessions tmp/cache tmp/sockets
    svn propset svn:ignore ".q" db/ --force
    svn propset svn:ignore "*.db" db/ --force

    We don't want to have developers overwrite eachother or worse production databases definition

    Instead we create a template file that can be copied but it should never be checked in

    cp config/database.yml config/database.yml.template
    svn add config/database.yml.template
    svn revert config/database.yml
    svn propset svn:ignore "database.yml" config
    We can set some scripts as executable so that they can be started using GUI tools This goes for all dispatchers
    svn propset svn:executable "" public/dispatch.
    and for all scripts inside the script directory
    svn propset svn:executable "" find script -type f | grep -v '.svn'
    Make sure we set the correct native eol style
    @REF = Pragmatic Programmers - Deploying Rails Application : p 28
    svn propset svn:eol-style native public/dispatch.
    @REF = Pragmatic Programmers - Deploying Rails Application : p 28
    svn propset svn:ignore "engine_files" public
    @REF = http://blog.teksol.info/2006/03/09/subversion-primer-for-rails-projects We want to make sure that the default page is not shown, so that it never looks like a default installation
    svn revert public/index.html
    @REF = http://blog.teksol.info/2006/03/09/subversion-primer-for-rails-projects

    This URL proposes to ignore the schema.rb because it is generated but the next one opposes this view

    svn propset svn:ignore "schema.rb" db


    @REF = Professional Ruby on Rails : p.45 @REF = http://books.google.be/books?id=aB4B13xGEv4C&pg=PA45&lpg=PA45&dq=subversion+schema.rb&source=bl&ots=fec_xmHyMa&sig=6PGXK4irrfzNYHxplVIJ2syoXgQ&hl=en&ei=ke64SfThCsm4-QaRm6HRBA&sa=X&oi=book_result&resnum=8&ct=result#PPA50,M1 They propose to insert the schema.rb within subversion, developers can start without applying all migrations and it might help
    Let's see if we got all files marked and with correct status
    svn status
    svn commit -m "initial rails checkin with ignores"
    Step 5: Freezing rails To handle this, rails seem to have moved through different solutions Option 1 (DEPRECATED): define the vendor/rails directory in your project as an SVN External, but the rails repository moved to GIT and so the latest versions can't be used @REF = http://blog.teksol.info/2006/03/09/subversion-primer-for-rails-projects @REF = Wrox - Professional Ruby on Rails - page 55
    #svn propset svn:externals "rails http://dev.rubyonrails.org/svn/rails/branches/ 2-2-stable" vendor
    #svn update vendor
    #svn commit -m "vendor integration"
    Option 2 (BACKWARDS COMPATIBLE): to solve this, there exist a gem called 'Piston' that allows you to handle git repositories and integrate them into a subversion repository @REF = http://blog.logeek.fr/2008/1/4/how-to-use-piston-to-ease-your-upgrades @REF = http://piston.rubyforge.org/
    Option 3 (RECENT VERSIONS) : the rails seems to have a command option to tell it to freeze @REF = Wrox - Professonal Ruby on Rails - p 56
    #rails . -f
    Option 4 (RECENT VERSIONS): rails now includes a rake task to freeze it into your project directory This will download the rails tar.gz from the edge site and we can tell to use a specific version using RELEASE= @REF = http://support.tigertech.net/freeze-rails @REF = http://codeclimber.blogspot.com/2008/09/freeze-those-rails.html
    rake rails:freeze:edge RELEASE=$RAILS_VERSION
    svn add vendor/rails
    svn commit -m "frozen rails to version $RAILS_VERSION"
    Should we set RAILS_HOME after freezing? @REF = http://snippets.dzone.com/posts/show/4730

    PATH=pwd/vendor/rails/railties/bin:$PATH

    export PATH


    Step 6: Freezing rails gems Recent versions only: Together with the rake task for rails there comes a rake task to freeze gems.
    If you have frozen rails in Step 5, there is no additional need to do this. Use this only if you want to freeze the rails gems and NOT rails itself.
    rake rails:freeze:gems
    svn add vendor/gems
    svn commit -m "frozen rails gems $RAILS_VERSION"
    Step 7: Freezing non-rails gems Option 1 (DEPRECATED): the first solution that appeared was to unpack of a gem in vendor , copy the files to the correct place and remove the unpack
    This is a tedious job because you have to go into the structure and move files around
    @REF = http://www.vaporbase.com/postings/Using_Subversion_and_Rails
    # cd vendor

    gem unpack gemname

    cp -Rf gemname-n.n.n/lib/* .

    cp -Rf gemname-n.n.n/MIT-LICENSE LICENSE-gemname

    cp -Rf gemname-n.n.n/README README-gemname

    svn add aaaa bbbb cccc dddd.rb LICENSE-gemname README-gemname

    svn propset version "n.n.n (Gem)" gemname.rb

    rm -Rf gemname-n.n.n

    cd ..

    Option 2 (CUSTOM): instead of copying the files, the gems are left unpacked in the vendor directory and the then the config.load_paths is adapted to include dynamically include the libs of all subdirs @REF = Oreilly - Advanced Rails - Large Projects p. 312
    # cd vendor

    mkdir gems

    cd gems

    gem unpack gemname

    cat ../../config/environment.rb |\

    sed -e 's@# config.load_paths += %W( #{RAILS_ROOT}/extras )@config.load_paths + = Dir["#{RAILS_ROOT}/vendor/gems/**"].map do |dir|\n\tFile.directory?(lib = "#{dir}/lib") ? lib : dir\nend@' > ../../config/environment.rb

    cd ..

    cd ..

    Option 3 (CUSTOM) : do the same unpack of the gem in the vendors directory but use the GEM_PATH to include the paths where the gems are unpacked @REF = http://errtheblog.com/posts/50-vendor-everything
    # cd vendor

    mkdir gems

    cd gems

    gem unpack gemname

    cd ..

    cd ..

    echo "ENV['GEM_PATH'] = File.join(RAILS_ROOT, 'vendor', 'gems')" >> config/boot.rb

    Option 4 (BACKWARDS COMPATIBLE) : gemsonrails @REF = http://gemsonrails.rubyforge.org/
    Option 5 (RECENT) : similar to the rails freeze, there is now a rake task that allows you to unpack the gems your project needs The first step is to list the required gems and their version and optional their source to the config/environment.rb file @REF = http://www.pathf.com/blogs/2009/03/keeping-up-with-the-joneses-keeping-rails-and-its-extensions-up-to-date/
    export GEM_NAME="hpricot"
    export GEM_VERSION="0.6.0"
    cat config/environment.rb | sed -e 's@# config.gem "bj"@config.gem "$GEM_NAME", :lib => "$GEM_NAME", :version => "$GEM_VERSION"@' > config/environment.rb.$$
    mv config/environment.rb.$$ config/environment.rb
    To use the rake task unpack, we first need to install the gem then we can execute the gem installation. Note: because we still have the GEMS_PATH and GEM_HOME set, the install will be specific to our project
    rake gems:install
    Now that it is installed we can unpack the gems, this will go in the vendor/gems directory
    rake gems:unpack:dependencies
    svn add vendor/gems
    svn commit -m "frozen additional gem $GEM_NAME to version $GEM_VERSION"
    To list the gems state there is a rake task (this only seems to work ok in Rails version 2.3.1 (otherwise you seem to get random nil.dependencies errors)
    rake gems
    Option 6: warbler
    @REF = http://caldersphere.rubyforge.org/warbler/
    If you need to transfer your application to another system, another approach would be to create a war file of your application and run your rails application with jruby.
    Option 7: gems2rpm
    @REF =http://www.fooplanet.com/projects/gem2rpm/
    Instead of putting your gems inside your project, it might be more useful to package your gems as an rpm. This is exactly what the gem2rpm script does. It only works with non-native gems
    Step 8: Freezing native gems
    the procedures from Step 7 will work but for gems requiring native when you unpack a gem, it will only unpack the ruby code that does not need to be compiled @REF = http://www.theagiledeveloper.com/localizing-gems-with-native-extensions.html @REF = http://agilewebdevelopment.com/plugins/gems_with_architecture @REF = http://www.linuxtopia.org/online_books/programming_books/ruby_tutorial/Extending_Ruby_Creating_a_Makefile_with_extconf.rb.html
    Option 1: works but needs recompile on other architectures
    rake gems:build
    Option 2: puts the binaries in a place where you want to
    We prepare a directory called native-lib where we put all the compiled stuff from the native gems
    mkdir native-lib
    ARCHDIR=pwd/native-lib/ruby -e "puts RUBY_PLATFORM"
    export ARCHDIR
    Next we search these directories for extconf.rb Then execute ruby extconf.rb , which will create a makefile, we do a make Then we install this to the native-lib directory we created, otherwise it would install itself in the system ruby directory
    find vendor/gems/ -name "extconf.rb" | xargs dirname | xargs -I {}  ksh 'cd {} ; ruby extconf.rb; make; make RUBYARCHDIR=$ARCHDIR install'
    We add the native-lib directory to the load_paths in config.environment so that these can be found when executing
    cat config/environment.rb | sed -e 's@# config.load_paths += %W( #{RAILS_ROOT}/extras )@ config.load_paths += %W( #{RAILS_ROOT}/native-lib/#{RUBY_PLATFORM} )@' > config/environment.rb.$$
    mv config/environment.rb.$$ config/environment.rb
    Now you can argue that we file is generated , so we might not need to check in But on some systems there isn't a compiler available (on Windows, no standard C compiler is installed f.i. , the same goes for stripped production installations)
    svn add $ARCHDIR
    svn commit -m "commited native libs"
    Step 9: Checking it all out on another system
    • Create the project structure (see STEP 0)
    • Set the ENV's (RUBY_HOME, GEM_HOME, GEM_PATH) (see STEP1)
    • Check the project out in the rails directory (see STEP )
    • Copy the database template
    • We need to repeat the svn ignores stuff (see STEP )
    • Generate the cross compiled stuff (see STEP 8). Otherwise you get rails to ask you to install the gem.
    @REF = http://tenderlovemaking.com/2008/11/21/cross-compiling-ruby-gems-for-win32/
    You can create a custom rake task for doing all this stuff. If you create a file in PROJECTDIR/rails/lib/tasks
    namespace :svn do
      desc "Sets the correct ignores and other stuff for a correct checkin of rails project in an svn repository"
      task :propset do
              system 'svn propset svn:ignore -R ".DS_Store" . --force'
              system 'svn propset svn:ignore "*.log" log/'
              system 'svn propset svn:ignore "*.db" db/'
              system 'svn propset svn:ignore "*.sqlite3" db/'
              system 'svn propset svn:ignore "database.yml" config/'
              system 'svn propset svn:ignore "*" doc/'
              system 'svn propset svn:ignore "*" doc/api'
              system 'svn propset svn:ignore "*" doc/app'
              system 'svn propset svn:ignore "*" tmp/'
              system 'svn propset svn:ignore "*" tmp/sessions tmp/cache tmp/sockets'
              system 'svn propset svn:executable "*" public/dispatch.*'
              system 'svn propset svn:eol-style native public/dispatch.*'
              system 'svn propset svn:ignore "engine_files" public/ '
              system 'svn propset svn:ignore index.html public/'
      end
    
    end
    then you can repeat this tasks with:
    $ rake -T |grep svn
    rake svn:propset                          # Sets the correct ignores and other stuff for a correct checkin of rails project in an svn repository
    Step 10: Do you first scaffold. If you supply the -svn , the files generated will automatically be add for the next commit
    ./script/generate scaffold -svn
    svn commit -m "first scaffold"

    TODO : Freeze plugins http://www.vaporbase.com/postings/Using_Subversion_and_Rails

    TODO : how to update to a new version of rails : rake rails:unfreeze, rake rails:update (seems to delete the who vendor dir?)

    rake rails:update -v 2.2.2

    @REF = http://stackoverflow.com/questions/313176/migrate-from-rails-211-to-222

    TODO : how to update to a new version of a gem

    TODO : show how you need to set this to work in a production environment (f.i. mod_rails)


    TODO : describe piston and gemsonrails more in detail

    TODO : show how you need to set this to work in a production environment (f.i. mod_rails)