Skip to content
accelerando

Category Archives: English posts

Logging in Rails outside a controller

25-Aug-09

You can use

RAILS_DEFAULT_LOGGER.log "foobar"
# or
Rails.logger.log "blah"

outside a controller for logging.

Creating a self containing mvc application with Sinatra

29-Jul-09

I recently presented biking.michael-simons.eu, a Sinatra based application written in Ruby.

Its main purpose for me is to keep track of my milage in 2009.

Although the application is completely self containing it has some nice features:

Simple setup

The application is simple and i didn’t want to use a “big” database like PostgreSQL or MySQL, therefore i choose SQLite. Together with DataMapper i can use the following and have a running SQLite database in no time:

configure :development do
  DataMapper.setup(:default, "sqlite3://#{Dir.pwd}/biking.dev.sqlite3")
end
 
# A milage is stored for a bike at the beginning of the month.
# Its the milage of the bike at this point of time.
class Milage
  include DataMapper::Resource
  property :id,          Integer, :serial => true
  property :when,        Date, :nullable => false
  property :value,       BigDecimal, :nullable => false, :precision => 8, :scale => 2
  property :created_at,  DateTime
 
  belongs_to :bike
 
  validates_is_unique :when, :scope => :bike
 
  is :list, :scope => [:bike_id]
end
 
DataMapper.auto_upgrade!

Creating charts with the google_chart gem

Google has a nice api for creating charts, the Google Charts API. The charts are created through URL parameters and as we know, most browsers limits the length of a query string. And for that, the parameters must be encoded.

Nobody wants to do it themselves so there is gchartrb that i use:

  # Create the google chart
  # All will include all float values
  all = Array.new
  @gc = GoogleChart::BarChart.new('800x375', nil, :vertical, false) do |bc|
    @bikes.each do |b|
      # Help is an array of floats for all periods
      hlp = b.milage_report.collect{|mr| mr[:milage]}
      # and gets added to the chart for that bike
      bc.data b.name, hlp, b.color
      all += hlp
    end
 
    hlp = Array.new
    # Get all periods. It sure can be done via a group by but 
    # the app should be database agnostiv
    periods = Milage.all.collect{|m| m.when.strftime '%m.%Y'}.uniq
    # Get all other trips in this period and also add them to the chart
    periods.each {|period| hlp <<= (AssortedTrip.sum(:distance, :conditions => ['strftime("%m.%Y", "when") = ?', period]) || 0.0)}
    all += hlp
    bc.data "Assorted Trips", hlp[0,hlp.length-1], "003366"
 
    # Define the labels
    bc.axis :x, :labels => periods, :font_size => 16, :alignment => :center
    # and the range from 0 to the highest value
    bc.axis :y, :range => [0,all.max], :font_size => 16, :alignment => :center
    bc.show_legend = true
    bc.grid :x_step => 0, :y_step => (100.0/all.max)*25.0, :length_segment => 1, :length_blank => 0 if all.size > 0
  end if @bikes.size > 0

The gc object will be used as simple as

%img{:src => @gc.to_url, :alt => 'gc'}

Using libxml for parsing RSS feeds

It seems that i’m not going to write an application without including my daily faces project dailyfratze.de in some way.

So i decided not only to present numbers but also my biking pictures. The images are available through a Media RSS feed. The feed itself contains pointers to other pages of that very same feed.

One of the fastest ways to parse XML is LibXML and luckily, it’s available for ruby through LibXML Ruby.

Together with memcached it can be used to efficiently handle feeds like so:

def BikingPicture.random_url
  # Check if pictures are cached...
  biking_pictures = @@cache['biking_pictures']
  unless biking_pictures then
    # Start retrieving the feed
    url = @@uri.parse("http://dailyfratze.de/michael/tags/Thema/Radtour?format=rss&dir=d")
    next_page = false
 
    Net::HTTP.new(url.host).start do |http|
      # Its a media rss feed that defines previous and next feeds
      while url
        req = Net::HTTP::Get.new("#{url.path}?#{url.query}")
        xml = http.request(req).body
        # Parse the data
        doc = LibXML::XML::Parser.io(StringIO.new(xml)).parse
        # Unless this url hasn't been retrieved...          
        unless @@cache[url.to_s]            
          doc.find('/rss/channel/item').each do |item|
            # Get all pictures and store them if not already grapped
            biking_picture = BikingPicture.first :url => item['url']
            unless biking_picture
              biking_picture = BikingPicture.new({:url => item.find_first('media:thumbnail')['url'], :link => item.find_first('link').content })
              biking_picture.save
            end
          end
          # Mark this url as seen
          @@cache[url.to_s] = next_page
          next_page = true                          
        end
        # Check if there are more feeds...
        next_url = doc.find_first("/rss/channel/atom:link[@rel = 'next']")
        url = if next_url
          @@uri.parse next_url['href']
        else
          nil
        end
      end
    end
    # Get the data...
    biking_pictures = BikingPicture.all()
    # ...and store it
    @@cache.set 'biking_pictures', biking_pictures, 3600
  end
  biking_pictures.sort_by{rand}[0]
end

Sinatra, Passenger and Memcached

Speaking of being efficient, the application certainly runs through Phusion Passenger aka modrails.

Running a Sinatra app under passenger is as simple as this: Create a directory for a new vhost, setup a structure like

biking
\- public
 - tmp
 - config.ru
 - biking.rb

and let config.ru contain the following:

root_dir = File.dirname(__FILE__)
 
require 'biking.rb'
 
set :environment, ENV['RACK_ENV'].to_sym
set :root,        root_dir
set :app_file,    File.join(root_dir, 'biking.rb')
disable :run
 
run Sinatra::Application

The apache vhost is configured like so:

<VirtualHost *>
        DocumentRoot "/path/to/the/applications/public/folder"
        RackBaseURI /
</VirtualHost>

I have turned off the Rack and Rails Autodetect features (RailsAutoDetect off, RackAutoDetect off) so i need to explicitly turn them on for a vhost.

After that, the application is running and thats all there is.

But wait. I’ve written before about problems with memcache-client and Passenger and this problems needs to be addressed in a Rails as well as Rack application.

I handle them in Sinatra as follows:

configure do
  # Create a global memcache client instance...
  @@cache = MemCache.new({
    :c_threshold => 10000,
    :compression => true,
    :debug => false,
    :namespace => 'some_ns',
    :readonly => false,
    :urlencode => false
  })
  @@cache.servers = 'some_server:11211'
 
  if defined?(PhusionPassenger)
    PhusionPassenger.on_event(:starting_worker_process) do |forked|
      @@cache.reset if forked
    end
  end
end

If PhusionPassenger is available, install an event handler that resets the freshly forked memcached connection to avoid corruption.

Using Geonames.org

Google Maps is great but one thing that’s often forgotten is reverse geocoding. Google doesn’t offer such api (as far as i know) but GeoNames does.

I’ve written a mobile J2ME client around JSR 179 api that can push my current location to this server when i’m biking so my girlfriend can follow my rides on the map.

The client pushes latitude and longitude and the server replies with the coordinates in angles and the name of the place. The name is retrieved through the Ruby Geonames API like so:

#
# Adds a new location
#
post '/locations' do
  require_administrative_privileges
  location = Location.new params
  if location.save then
    begin
      country_subdivision  = Geonames::WebService.country_subdivision(location.latitude, location.longitude)
      places_nearby = Geonames::WebService.find_nearby_place_name(location.latitude, location.longitude).first
      (location.description = "#{places_nearby.name}, #{country_subdivision ? country_subdivision.admin_name_1 + ', ' : ''}#{places_nearby.country_name}"[0,2048].strip) and location.save if places_nearby
    rescue Exception => exc      
    end
  end
  "#{location.to_s}\n"
end

Inline templates, external resources

The thing started as a small application. PHP was out of the question but Rails was overkill, too. I just wanted one single file.

I can use the great Haml syntax as inline templates. This goes for creating HTML as well as for CSS through Sass.

To get some basics, i also include one of the W3 core styles.

The only resources that lives outside biking.rb is JQuery for easy creation of the autorefreshing code and some images, namely the cute green bikers from Greensmilies.

Summary

It’s absolutely possible to write an application with thee MVC Pattern in mind without one of the “big” frameworks in just one single script.

I would go much further with this app using just one script, but for it’s current purpose, it’s great, i think.

I hope you enjoyed reading my little annotations as i enjoyed writing them and the application. The full source code is available right through biking.michael-simons.eu and it’s published under the BSD license.

Javas String.replaceAll

21-Jul-09

The following statement

"foo baz".replaceAll("baz","$bar");

will present you an java.lang.IllegalArgumentException: Illegal group reference exception as the replacement string can contain backreferences to the search pattern as stated in Mather#replaceAll:

This method first resets this matcher. It then scans the input sequence looking for matches of the pattern. Characters that are not part of any match are appended directly to the result string; each match is replaced in the result by the replacement string. The replacement string may contain references to captured subsequences as in the appendReplacement method.

Note that backslashes (\) and dollar signs ($) in the replacement string may cause the results to be different than if it were being treated as a literal replacement string. Dollar signs may be treated as references to captured subsequences as described above, and backslashes are used to escape literal characters in the replacement string.

Luckily, there is an easy solution:

"foo baz".replaceAll("baz",Matcher.quoteReplacement("$bar"));

Turn ruby heredocs from double-quoted into sinqle-quoted strings

17-May-09

By default, Rubys heredocs are interpreted as double-quoted strings, that is #{something} is evaluated, \r turns into newline and so forth.

You can change this behaviour by single quoting the heredoc identifier like so:

s = <<-'SINGLE_QUOTED'
#{i'm not interpreted}
SINGLE_QUOTED

Batchconvert ascii to utf8

12-May-09

Next time i see umlauts in source, I’ll scream. Loud.

In the mean time I try this:

find . -iname "*.java" -exec sh -c 'iconv -f cp1252 -t utf-8 {} > {}.utf8' \;
for i in `(find . -name "*.utf8")`; do mv $i ${i/.utf8/}; done

Before you try this, make a backup of your files. It worked for me but i don’t guaranty that your files won’t vanish.

Understanding

08-May-09

I’d love to understand what makes people create abominations such as this:

if((foo ? !bar : true))
    if(!somethingelse)
        dosomething()

I don’t want to judge anyone, the thing actually worked, but i really want to understand why people don’t see that this is crap. I’m really aware of the fact, that things tends to organically grow, but how can it be achieved to raise awareness to this fact? I mean, i’m probably only a not-totally-bad engineer, but i see things that will break or are just ugly as hell and i want my colleagues to be more aware of pitfalls.

Would make life much easier and probably more conflict free.

Grails: Completely disable stacktrace.log file

24-Mar-09

Regarding my question on twitter about completely disabling the creation of stacktrace.log from a Grails application in production mode, here is my answer:

environments {
    production {
        log4j = {
            appenders {
               null name:'stacktrace'
            }
        }
    }
    development {
    }
}

Be careful: This overwrites all other log4j closures that are made outside the environment specific settings. I found no solution for that.

Background: One of my Grails 1.1 app fails to start on a Tomcat application container deployed on an Oracle Enterprise Linux server because it fails to create the stacktrace.log.

What annoyed me is that i needed to dig around in the Grails source code to find the answer to my question in “src/groovy/org/codehaus/groovy/grails/plugins/logging/Log4jConfig.groovy” within the method “createFullstackTraceAppender()”. Should be stated in the documentation (perfect place here).

Edit: If your Grails 1.1 still fails to start(*) look out for xercesImpl.jar and xml-apis.jar or similar somewhere in your Tomcat 5 directory. These are some kind of Java 1.4 compatibility layers which break things in Grails 1.1. Oracle puts these into common/endorsed. After removing them, everything went fine.

(*) Exception was:

java.lang.RuntimeException: XPathFactory#newInstance() failed to create an XPathFactory for the default object model:
http://java.sun.com/jaxp/xpath/dom with the XPathFactoryConfigurationException: javax.xml.xpath.XPathFactoryConfigurationException:
No XPathFctory implementation found for the object model: http://java.sun.com/jaxp/xpath/dom

Phusion Passenger and memcache-client revisited

23-Mar-09

The last Passenger update brought some good explanation off the problems regarding Passenger and memcache-client (see here).

Smart spawning of Passenger processes creates shared file descriptors. As the connections to memcached are sockets they are shared as well so data on them gets corrupted which is explained very nicely in the Passenger documentation: Example 1: Memcached connection sharing (harmful).

The solution presented there works like a charm. The reestablish_connection_to_memcached line is actually not more than CACHE.reset where CACHE is the reference to the memcache connection.

After that change, spawning methods smart-lv2 and smart will work in connection with environment.rb configured memcache connections.

Edit: As requested in the comments, a little example:

memcache_options = {
  :c_threshold => 10000,
  :compression => true,
  :debug => false,
  :namespace => 'some_ns',
  :readonly => false,
  :urlencode => false
}
 
CACHE = MemCache.new memcache_options
CACHE.servers = '127.0.0.1:11211'
begin
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     if forked
       # We're in smart spawning mode, so...
       # Close duplicated memcached connections - they will open themselves
       CACHE.reset
     end
   end
# In case you're not running under Passenger (i.e. devmode with mongrel)
rescue NameError => error
end

In this case, CACHE is the global constant that i use to access my memcache-client.

I guess you’ll need to do the same with the global Rails.cache object, but i’m not sure. Anyway, the above solution works for me.

Grails’ withFormat block

23-Mar-09

Some things are not really different in Rails and Grails world. The pendant to Rails’ respond_to method is Grails withFormat block.

Both are supposed to render a different content type as requested according to the accept header and and the format parameter.

And both fail to some extend with Internet Explorer 5.5 to 7.0. For a longer explanation see my post on respond_to linked above. In short: First visit always gave me the an Excel File, all subsequent visits the intended html page.

I used nearly the exact solution within Grails 1.0.4 as in Rails:

withFormat {
      xls {
        // Same crap with IE 6/7 as with rails, compared to
        // http://info.michael-simons.eu/2007/08/06/rails-respond_to-method/
        if(params.format == 'xls') {
          def df = new SimpleDateFormat("yyyy-MM-dd")
 
          def report = ExcelReport.findById(params.reportId)          
          if(report == null)
            report = ExcelReport.findByBezeichnung('Absatzprognose')          
 
          response.contentType = 'application/vnd.ms-excel'
			    response.setHeader("content-disposition", "attachment;filename=some_filenname.xls")
          excelService.runExcelReport(
            report.bezeichnung,
            "some_parameter",
            response.outputStream
          )
          return
        }
      }
    }

Grails 1.1 seems to have fixed some issues on this case and a default or empty html {} block in front of any other format like so will do the trick:

withFormat {
      html {
        // depending on your needs
      }
      xls {
        // funny excel stuff
      }
    }

Frustration

25-Feb-09

if you want something done right, do it yourself

If you happen to use the mint tracking tool like me, be careful when migrating from a 32bit to a 64bit server or vice versa. Mint saves ip addresses and a buttload of checksums as signed long values. That bites you right in the ass when the first visitors starts arriving at your site. All from 127.255.255.255, or at last visitors with an ip starting with > 127.

If you read this post before migrating, just add the following to your migration:

ALTER TABLE mint_visit 
  MODIFY COLUMN `ip_long` BIGINT NOT NULL,
  MODIFY COLUMN `referer_checksum` BIGINT NOT NULL,
  MODIFY COLUMN `domain_checksum` BIGINT NOT NULL,
  MODIFY COLUMN `resource_checksum` BIGINT NOT NULL,
  MODIFY COLUMN `session_checksum` BIGINT NOT NULL;
 
ALTER TABLE mint_debugger   
  MODIFY COLUMN `ip_long` BIGINT NOT NULL;
 
ALTER TABLE mint_geo  
  MODIFY COLUMN `ip` BIGINT NOT NULL;
 
ALTER TABLE mint_hostnames  
  MODIFY COLUMN `ip_long` BIGINT NOT NULL;

and you’re done.

If it’s too late, change your tables as well. You then also have to delete your entire mint_visit data, as the records are already corrupted. Great fail.

However, why on earth store an ip address as a long value?!? It’s an ip address and if i want to look at it, the program needs to go the other way round. To me: It’s just stupid fucking with datatypes and problems as that is what you get if you do so. It maybe makes sense for computation of networks, but not for a statistic tool. At least, i don’t see any sense in this.

Actually, the problem is known.

Close
E-mail It