Technology Engineered

Moving

2008-08-27T22:38:00.001-05:00

JRuby + JMonkeyEngine = Hello 3D World

2008-08-18T23:04:00.002-05:00

JRuby released version 1.1.3 last month, and I missed it because these days I'm all corporate, and I have limited access to the Internet from work, and little time to get online from home. Speaking of which, I'm on daddy-diaper-duty tonight, so I'll keep this brief so as not to get interrupted...

As Charlie Nutter posted recently, "There are a few areas that remain to be tackled... Refactoring the extensive logic governing how Ruby classes can extend abstract or concrete Java classes. Kresten Krab Thorup contributed this well over a year ago, and it's kinda been an island unto itself. It will take a considerable effort to rework."

Too true, that issue has been a thorn in my virtual side for a while now. I've been interested in transitioning into game programming as a career move, and a lot of the libraries that I've dabbled with use abstract classes that must be extended. At one point, JRuby was incapable of doing that, since a Java object was actually decended from JavaProxy, not the actual Java class being extended.

Not so any more! Tonight I pulled down trunk, and loaded up jirb. I had also built the 3D scene graph library JMonkeyEngine, by all accounts a good one for Java. Rumor has it that Sun's Project Wonderland will be using it instead of Java3D, and they're dropping development on the latter for the time being. Anywho, I was able to place the native libraries in my working directory, require the jars, and...

class MyGame < com.jme.app.SimpleGame
def simpleInitGame
end
end

game = MyGame.new
game.start

and just like that, I had a bunch of output from LWJGL and an empty window! But not just any empty window... an empty window into a virtual world, yet to be populated with 3D things, and a little message in the bottom corner telling me that I was getting about 2K FPS and nothing to display. Hit ESC, and the window went away and Java shut down. Alls well that ends well.

Now, to put something into the virtual world will require more work on JRuby's Java Integration, because SimpleGame uses a protected field called rootNode, and the JRuby subclass can't access that field yet. But, oh, once it can, then it's on!

Here's how you'd do it in Scala:


class MyGame extends com.jme.app.SimpleGame {
  def simpleInitGame() = { rootNode attachChild new com.jme.scene.shape.Sphere("my sphere",10,10,5) }
}
var game = new MyGame
game start

Works pretty well. My only gripe with Scala right now is the lack of Ruby-ish method aliasing ala obj.blah=x instead of obj.setBlah(x) in Java Integration land. Scala's val statement does a nice job of adding those setters and getters for pure Scala objects though.

Until JRuby gets protected field access working, this bit of code does the job:


f=javax.swing.JFrame.new 'Blah'
f.default_close_operation=0
b=javax.swing.JButton.new 'Quit'
game=com.jmex.game.StandardGame.new 'test game'
b.add_action_listener do |ev|
  game.shutdown
  sleep 2
  java.lang.System.exit 0
end
f.add b
game.start
f.pack
f.show
state = com.jmex.game.state.DebugGameState.new
sphere = com.jme.scene.shape.Sphere.new('my sphere',10,10,5)
sphere.set_random_colors
sphere.update_render_state
state.root_node.attach_child sphere
com.jmex.game.state.GameStateManager.instance.attach_child state
state.active=true

Now there's a cool prismatic sphere in there. The window closer button is for safety's sake, because the default behavior of DebugGameState is to immediately exit Java, which tends to happen in the middle of the OpenGL loop and booches the VM. StandardGame is a move away from SimpleGame into a more OO-ish paradigm, and root_node is a Ruby-ized alias to getRootNode(). But hey, at least it's a working proof-of-concept: JRuby's Java Integration has certainly matured, and you can do some pretty snazzy stuff with it.

Integrate a forum with a Rails site

2008-03-05T14:43:00.004-05:00

So you've got a fancy rails site where Users can log in and do stuff. And you'd like them to be able to host a community forum on the site, using your existing User model to authenticate users for posting, moderation, and administration. The Savage Beast 2.0 engine does just that. In typical Rails style, you just follow its conventions for determining the logged-in user and authenticating them, and you're good to go with a mini-Beast forum.

When I tried this out yesterday, I ran into tons of issues with the Engines plugin and the way that Rails development environment loads, unloads, and reloads classes. So here are a couple of things that I figured out:

First, I don't know thing one about Engines, other than they sound pretty useful. For some reason, after installing the engines plugin, I couldn't run my migrations. It looks like Engines tries to redefine the db:migrate task into a namespace that then migrates all of the plugins. But that didn't work with my version of Rake, so my workaround is just to migrate before installing Engines.

*NOTE: I was using Rake 0.7.3, but when I tried using Rake 0.8.1, I didn't have the same error.
Lots of issues with modules being unloaded in development. So again the solution is to avoid the problem, and run in production mode.

*NOTE: This is actually due to Rails Dependencies module and the way that Models and Controllers are loaded and unloaded in between requests, and it supposedly happens with lots of different plugins in development. Basically, if your plugin defines models or controllers, you have to call "unloadable" inside the class. Otherwise Rails doesn't unload the class at the end of the request, but does unload referenced modules in your main app. So when Savage Beast's ForumsController calls methods of AuthenticatedSystem included in ApplicationController, the AuthenticatedSystem module gets unloaded from ApplicationController but not ForumsController, and on the next request it won't be reloaded because it is still defined.
I also have the old 1.2.3 Rails on my work machine, since I'm maintaining old apps that I don't have time to migrate to Rails 2.0, so as you'll see, I'm explicitly freezing the 2.0.2 gems to vendor/rails.

Here's my install script. If you follow along (and none of the dependent plugin trunks have changed in a way that breaks things), you should be all set.

# steps to create a sample app for Savage Beast 2.0 using Restful Authentication as the User model
# requires: rails, svn
# note: better to run in Production environment than development, because development will cause
# Models and Controllers to unload in between requests, causing unexplained missing constants and methods
rails savage_beast_test
cd savage_beast_test
rake rails:freeze:edge TAG=rel_2-0-2
cd ..
ruby savage_beast_test/vendor/rails/railties/bin/rails savage_beast_test
# to overwrite all, type: a
cd savage_beast_test
ruby script/plugin install restful_authentication
ruby script/generate authenticated user sessions
cd vendor
cd plugins
svn checkout http://savage-beast-2.googlecode.com/svn/trunk/ savage_beast
cd ..
cd ..
cp vendor/plugins/savage_beast/db/migrate/001_create_savage_tables.rb db/migrate/002_create_savage_tables.rb
rake db:migrate
rake db:migrate RAILS_ENV=production
ruby script/plugin install white_list
ruby script/plugin install white_list_formatted_content
ruby script/plugin install acts_as_list
ruby script/plugin install svn://errtheblog.com/svn/plugins/gibberish
ruby script/plugin install svn://errtheblog.com/svn/plugins/will_paginate
ruby script/plugin install http://svn.rails-engines.org/engines/trunk

# *** app file modifications: ***

# config/environment.rb:
# Bootstrap the Rails environment, frameworks, and default configuration
require File.join(File.dirname(__FILE__), 'boot')
# Boot the Engines plugin
require File.join(File.dirname(__FILE__), '../vendor/plugins/engines/boot')
#...

# config/routes.rb:
ActionController::Routing::Routes.draw do |map|
map.from_plugin :savage_beast

map.resources :users

map.resource :session
#...

# app/models/user.rb:
include SavageBeast::UserInit
def display_name
login
end
def admin?
true
end
def currently_online
false
end
protected
#...

# app/controllers/application.rb:
class ApplicationController < ActionController::Base
helper :all # include all helpers, all the time
include AuthenticatedSystem
def admin?
logged_in?
end
#...

# *** minor fix to Savage Beast application_helper.rb:
# (note: restful_authentication#current_user returns :false instead of nil)
# vendor/plugins/savage_beast/app/helpers/application_helper.rb:
def beast_user_name
#(current_user ? current_user.display_name : "Guest" )
(logged_in? ? current_user.display_name : "Guest" )
end

ruby script/server -e production

# get: http://localhost:3000/forums
# - first hit causes an error, lowpro.js missing. Second hit works.
# get: http://localhost:3000/forums/new
# - redirects to /sessions/new
# get: http://localhost:3000/users/new
# create a new user... logs in and redirects to /forums/new with "Thanks for signing up!" flash
# create a new forum... redirects to /forums/1 with no topics
# click New Topic link, gets /forums/1/topics/new
# create new topic/first post, redirects to /forums/1/topics/1

Runnable Java Procs in JRuby

2008-01-14T11:00:00.000-05:00

One of the cool features of the JRuby Swing GUI builder library, Profligacy (by the ever awesome Zed Shaw of Mongrel fame) is the Proc#to_runnable method that converts procs into a java.lang.Runnable interface that can be passed into any Java method that accepts an instance of Runnable, such as java.lang.Thread.new(). Profligacy's implementation looks something like this:


class RunnableProc
  include java.lang.Runnable
  def initialize &block
    @block = block
  end
  def run
    @block.call
  end
end
class Proc
  def to_runnable
    RunnableProc.new &self
  end
end

I prefer doing this instead:


class Proc
include java.lang.Runnable
alias_method :run, :call
end
p=Proc.new do
puts 'running'
end

The only disadvantage to that is that you can't use the familiar proc { ... } and lambda {...}, but I like having a simple 4-liner. And the Runnable interface is really useful in JRuby for working around protected Java methods when creating JRuby subclasses: I can mock up and compile a simple Java subclass that accepts a Runnable, then instantiate it in JRuby with a Proc. For example, the java.util.TimerTask class has a protected constructor, so you can't instantiate it in JRuby, even as a subclass, because a subclass is really a subclass of JavaProxy with an instance variable of the Java class. So:


//RubyTimerTask.java
public class RubyTimerTask extends java.util.TimerTask
{
  public Runnable task;
  
  public RubyTimerTask(Runnable r){
    this.task = r;
  }
  public void run(){
    this.task.run();
  }
}

#timer_test.rb
include_class 'RubyTimerTask'
p = Proc.new { puts 'running' }
rtt = RubyTimerTask.new p
t=java.util.Timer.new
t.schedule rtt, 2000 #run in 2 seconds
t.cancel #join the Timer's thread

JRuby 1.0.3, Rubygems 0.9.5 TypeError

2007-12-21T09:50:00.000-05:00

Here's that error again: After updating Rubygems to 0.9.5 (yeah, I know 1.0.0 just came out, but that one does some weird stuff with bin script filenames that I'd rather not deal with right now), trying to install rails gives:

ERROR:  While executing gem ... (TypeError)
    can't convert nil into Array

Running gem with the -V and --backtrace options makes it seem that the problem comes up from checking a default Ruby version requirement. Gem::Requirement.default creates a new Gem::Requirement with version GTE 0, and when it compares that version 0 to the Ruby version 1.8.5, the 0 doesn't properly get turned into an array [0] and gets left as nil, which when compared to an Array causes the above TypeError. Patching Rubygems to make it compare to_ints to to_ints rather than @ints to #ints works around the problem, but doesn't address the real issue: why does the same code work in MRI but not JRuby?

So I fired up jirb and started checking it out, and found that Gem::Version#normalize and to_ints weren't being called when comparing to the version requirement from the gemspec. Comparing to the Gem::Requirement.default worked fine.

According to the response I got from Eric on the Rubygems patch I submitted, @ints gets set from Version#initialize, #yaml_initialize, and #marshal_load. So why isn't it getting set for the default in this case? All of those constructor methods call self.version=, and #version= calls normalize to turn the version string into an array. Initialize seemed to work, but a gemspec uses yaml, and I found that yaml_initialize isn't being called!


class Foo
def initialize
@foo=1
end
def yaml_initialize tag, val
@foo=5
end
end
foo=Foo.new
YAML.load YAML.dump(foo) # #

Doh! yaml_initialize is definitely NOT being called. So I submitted JRUBY-1786, and hopefully Ola Bini or someone else will know how to fix it in the java code. In the meantime, my rubygems/version.rb workaround still does the trick.

JRuby 1.0.3, Rails 2.0.2, Rubygems 0.9.5

2007-12-19T15:07:00.000-05:00

So here's the deal... The easiest way to install Rails is through Rubygems, which automatically creates the "rails command" that invokes the rails application skeleton generator, in your ruby bin directory, and sticks rails in the rubygems load path so that you can require 'rubygems' and then require 'activerecord' etc.

Problems start to pop up when you try installing Rails 2.0 (the current, production-ready Rails) in JRuby. As I mentioned earlier, the new gems were created with Rubygems 0.9.5 rather than Rubygems 0.9.4 which ships in the JRuby binary package. In C-Ruby, you can easily upgrade Rubygems by running gem update --system. That runs gem's update_command.rb, which pulls down the latest rubygems-update-XXX.gem file, unpacks it, and then runs this little ditty:


Dir.chdir update_dir do
          say "Installing RubyGems #{version_string}"
          setup_cmd = "#{Gem.ruby} setup.rb #{args.join ' '}"

          # Make sure old rubygems isn't loaded
          if Gem.win_platform? then
            system "set RUBYOPT= & #{setup_cmd}"
          else
            system "RUBYOPT=\"\" #{setup_cmd}"
          end
        end

Did you catch that? UnSet the environment variable RUBYOPT, and re-run the ruby command in the install directory, running setup.rb. Like I said, it works fine in C-Ruby, but here's how Gem.ruby is defined in Rubygems 0.9.4:


# Return the Ruby command to use to execute the Ruby interpreter.
    def ruby
      "ruby"
    end

And here it is in 0.9.5:


# Return the Ruby command to use to execute the Ruby interpreter.
    def ruby
      if @ruby.nil? then
        @ruby = File.join(Config::CONFIG['bindir'],
                          Config::CONFIG['ruby_install_name'])
        @ruby << Config::CONFIG['EXEEXT']
      end

      @ruby
    end

Which is correct. The old version just says, "the ruby interpreter command is 'ruby', and that's that." Which doesn't do a lick of good if ruby isn't in your path, and more importantly, if the command is jruby and not ruby! So running gem update --system won't do any good in JRuby. We can get around that by pulling down the gem and running setup.rb manually with JRuby:

bin\jruby -S gem install rubygems-update --no-rdoc --no-ri
cd lib\ruby\gems\1.8\gems\rubygems-update-0.9.5
..\..\..\..\..\..\bin\jruby setup.rb --no-rdoc --no-ri
cd ..\..\..\..\..\..

Now, there's one more issue with Rubygems to fix before we can get Rails installed. Fire up this bit of code in bin\jirb:


fname='lib/ruby/site_ruby/1.8/rubygems/version.rb'
lines = File.readlines fname
lines[126]

You should get back: " @ints <=> other.ints\n", and ints is an attr_accessor. Makes sense, but if you just try installing rails at this point, there's an issue where checking against version 0 does not properly initialize the ints array, which is just a major/minor version number split on dots. What we need to check is to_ints, which splits the version number and returns @ints, so try this:

lines[126].gsub! /@ints|(\.)ints/, "\\1to_ints"
File.open(fname,'w'){|f| lines.each{|line| f.write line } }

Now, you can run bin\jruby -S gem install rails --no-rdoc --no-ri and be good to go. Note that Rubygems 0.9.5 has -y (--include-dependencies) as the default option, and will nag you if you specify it.

Cheers!

More Rails2.0.2 : C-Ruby Win32

2007-12-19T09:24:00.001-05:00

In my never-ending quest to figure out what's the deal with Rails2.0 and JRuby, I'm trying the new rails out in Win32 C-Ruby with a clean install.

Ruby-186-25 One-Click Windows Installer, installed to c:\rubytest


echo %PATH%
c:\rubytest\bin;...

gemwhich rake
c:/rubytest/lib/ruby/gems/1.8/gems/rake-0.7.2/lib/rake.rb

So far, so good. Now, let's update rubygems and get rails.


gem update --system (gem update DASH_DASHsystem)

gem --version
0.9.5

gem install rails
Successfully installed rake-0.7.3
Successfully installed activesupport-2.0.2
Successfully installed activerecord-2.0.2
Successfully installed actionpack-2.0.2
Successfully installed actionmailer-2.0.2
Successfully installed activeresource-2.0.2
Successfully installed rails-2.0.2
7 gems installed

rails testapp
cd testapp
ruby script/generate scaffold blah name:string
gem install sqlite3-ruby

(download and install sqlite3.dll to c:\rubytest\bin)

rake db:migrate
(in C:/rubytest/testapp)
== 1 CreateBlahs: migrating ===============
-- create_table(:blahs)
   - 0.0620s
== 1 CreateBlahs: migrated (0.0620s) ======

ruby script/server

(in another console)

irb
require 'active_resource'
class Blah < ActiveResource::Base
self.site='http://localhost:3000/'
end
b=Blah.create :name => 'Bob'
=> #<Blah:0x335287c @prefix_options={}, @attributes={"name"=>"Bob", "updated_at"=>Wed Dec 19 14:58:45 UTC 2007, "id"=>1, "created_at"=>Wed Dec 19 14:58:45 UTC 2007}>

So that sets up a WEBrick server with a resource scaffold, routed as /blahs, and accesses it remotely using ActiveResource to post to /blahs.xml and return the new Blah object. You can also fire up the web browser and go to http://localhost:3000/blahs to see the effect.

So on a whim, I figured I'd retry installing rails with JRuby 1.0.3, but I forgot that I still had Rubygems 0.9.4 on that install, and never successfully got it upgraded, so that'll be the focus of my next post.

JRuby 1.0.3 on Rails 2.0.2 : Too sexy for your migrations

2007-12-18T09:39:00.000-05:00

Wow, what a week. First we get a new stable JRuby release, quickly followed by the big whopper Rails 2.0! And with an interesting twist: in the young Rails 2.0.2 update, SQLite is now the default database instead of MySQL, which means you can roll out much faster and more simply in a normal C Ruby environment. In JRuby however, the results might be unpredictable. As far as I can tell, JRuby doesn't support SQLite, but I hoped that may have changed in the meantime.

The short of it is, it hasn't changed. Even in 1.0.3, you won't succeed with jruby -S gem install sqlite3-ruby. It stinks, but oh well. You can get around it easily enough by installing activerecord-jdbc and Derby, but I was hoping for a simple "rails and roll" scenario. You can also get back to using mysql as the database with rails myapp -d mysql.

There's also the matter of getting Rails2.0.2 installed in the first place. A bunch of people reported problems with gem, which boiled down to a version incompatibility between Rubygems 0.9.4 (which most people with the problem had, and which ships with JRuby) and 0.9.5 (with which the Rails gems were built). And running gem update --system from JRuby doesn't give the expected result, because the update script attempts to exec "ruby" at some point. So now I'm not even sure if my existing C-Ruby install is stable or not, since it might have run the update there. We'll see once it comes time for me to update any production Rails code... Anyway, I had to check rel_2-0-2 out of SVN to get Rails installed. But once I did that I was able to run the rails generator out of railties with -d mysql, generate a scaffold resource, and migrate it.

JRuby + JNLP = headache

2007-11-16T15:58:00.000-05:00

So I've been working in the JRuby codebase for the last few weeks, trying to realize my vision of a JRuby network client that uses ActiveResource for object persistence to a remote web server, and uses the WebStart JNLP feature of Java to automatically download the appropriate Java Runtime and latest versions of extension libraries (including JRuby). So far it's been one roadblock after another.

ActiveResource comes as a RubyGems package, which is great for a normal JRuby install. But when running out of JNLP, you're basically adding jruby-complete.jar to your classpath and invoking org.jruby.Ruby yourself, and all of your Ruby core scripts are stored in the JAR file. Which means that in order to install any RubyGems, you first have to extract the complete JAR file into your home directory's .jruby directory.

Which wouldn't be such a big deal if it weren't for JNLP's extreme security model. Java code within a signed JAR file is allowed access to the filesystem, but other code is not. And JRuby classes get defined on-the-fly through bytecode and then loaded with a custom classloader, so they don't meet the "signed JAR" requirement and run into lots of security problems that don't happen when you run straight Java classes.

There's also another issue: ActiveResource requires net/https, which in turn requires openssl, which in JRuby is loaded through an extension JAR. And you guessed it, that has a security conflict with JNLP, and would require modifying the openssl extension in addition to JRuby. I can get around it by writing my own trimmed-down ActiveResource, which isn't all that hard. Just provide some method_missing and YAML marshalling magic and you're good to go. But then, when you try to open an HTTP connection to the server, WebStart pops up a security dialog asking if you want to allow the connection to be made. Making the same connection through the standard java.net.URL.openConnection of course works fine.

So for now, I'm going to add "automatic extension downloader/updater that doesn't have the security roadblocks of JNLP" to my wish list. In the meantime, I've made a couple of patches for JRuby that fix some of my issues. JRUBY-1574 allows extraction from a non-file: url, so you can download the jruby-complete.jar over http. Working on another to fix part of JNLP JRuby not working at all for Java Integration, where building the proxy class fails because the static Java class members are inaccessible due to the JNLP security.

Blog Action Day: The Environment

2007-10-15T13:50:00.000-05:00

Today's the day when everyone is supposed to blog about the same topic, and this year, it's the environment. Since this blog tends to focus on Ruby programming lately, that makes my focus easy. For those who don't know, the environment is accessible via the constant ENV. For example:

puts ENV

You'll get all your standard environment variables in that Hash, including PATH, which is probably the one you'd use the most.

JRuby on Rails 1.2.4

2007-10-05T12:30:00.000-05:00

It's official, Rails is now up to version 1.2.4 and JRuby on Rails on Windows is currently busted. See JRUBY-1401, my bug report and patch. Luckily you can fix this in a JRuby install by just applying the patch to pathname.rb in the lib/ruby/1.8 directory, but if you're using Goldspike to make a Rails WAR using jruby-complete-1.*.jar, I'm not sure what you can do. Building the jar from patched source should work, and you might even be able to pull pathname.rb out of the jar, patch it, then update the jar with the patched version.

Jruby on non-Windows platforms is probably fine. Just another reason to switch to Ubuntu or your Linux flavor of choice.

JRuby 1.0.1 on Edge Rails 1.2.3.7605

2007-09-24T07:25:00.000-05:00

There are a few good tutorials out there on installing stable Rails on top of JRuby 0.92, but I haven't seen any yet describing how to do Edge Rails (1.2.3.7605 at the moment) on JRuby 1.0.1. The newest JRuby fixes a bunch of issues and adds some new features, and Edge Rails has all sorts of improvements to RESTful web services. I had gone through the older JRuby on Rails tutorials to get rolling, then upgraded to Edge Rails and found that I could no longer WAR up my application, nor even run Rails script commands. I'm running JRuby on Windows XP, and the problem arose from some Edge Rails code that determines the application root directory conditionally depending on the Ruby Platform, with a special case for Windows - not accounting for JRuby on Windows, which it sees as "java." So, here I'll describe the process to get rolling and fix some of these issues.

First, visit http://jruby.codehaus.org and download the JRuby 1.0.1 binaries. I unpacked them into C:\jruby, so C:\jruby\jruby-1.0.1\ is my JRuby "home" directory, and C:\jruby\jruby-1.0.1\bin is the path to jruby.bat, jirb.bat, and gem.bat. Following Dominic Da Silva's instructions on developer.com, I set my path to include c:\jruby\jruby-1.0.1\bin and ran:
jruby -version
and got: ruby 1.8.5 (2007-08-23 rev 4201) [x86-jruby1.0.1]

So far, so good.

Please note: If you get an error like this, just close your command prompt and open another one, and remember to reset your PATH to point to jruby. (Thanks to Arun Gupta for pointing that out.) It's due to a JRuby bug:

The input line is too long.
:gotCP
was unexpected at this time.

Now let's get rails installed, using jruby -S to make sure that we're running JRuby and not MRI:

jruby -S gem install rails -y --no-rdoc --no-ri
Bulk updating Gem source index for: http://gems.rubyforge.org
Successfully installed rails-1.2.3
Successfully installed activesupport-1.4.2
Successfully installed activerecord-1.15.3
Successfully installed actionpack-1.13.3
Successfully installed actionmailer-1.3.3
Successfully installed actionwebservice-1.2.3

Note the lack of activeresource in the installed gems - that's because we've installed stable Rails from RubyForge, as opposed to Edge Rails from rubyonrails.org. Now let's make an application to test:

jruby -S rails railstest

This runs the rails generator, and all of its usual output about what directories and files it created for us. Now, cd into railstest, edit config/database.yml to point to your usual MySQL database (remember to set both development and production passwords, as you'll see later), then run jruby script/server to get running. Visit http://127.0.0.1:3000 to make sure that WEBrick is working, and for me it is. So now let's try something more complex. Terminate WEBrick, and then create a simple Hello controller with an index method that outputs the time:
jruby script/generate controller Hello
app/controllers/hello_controller.rb:

class HelloController < ApplicationController
  def index
    render :inline => "Hello, it is #{Time.now}"
  end
end

Fire that up, go to http://127.0.0.1:3000/hello and we now get greeted with the time.
Hello, it is Mon Sep 24 09:00:36 EDT 2007

So far, we're doing alright with stable Rails 1.2.3. Now let's kick it up a notch and upgrade to Edge Rails:

jruby -S gem install rails -y --no-rdoc --no-ri --source http://gems.rubyonrails.org
Bulk updating Gem source index for: http://gems.rubyonrails.org
Successfully installed rails-1.2.3.7605
Successfully installed activesupport-1.4.2.7605
Successfully installed activerecord-1.15.3.7605
Successfully installed actionpack-1.13.3.7605
Successfully installed actionmailer-1.3.3.7605
Successfully installed activeresource-0.9.0.7605

jruby script/server
C:/jruby/jruby-1.0.1/lib/ruby/1.8/pathname.rb:420:in `realpath_rec': No such file or directory - C:/railstest/C: (Errno::ENOENT)
        from C:/jruby/jruby-1.0.1/lib/ruby/1.8/pathname.rb:453:in `realpath'
        from C:/jruby/jruby-1.0.1/lib/ruby/gems/1.8/gems/rails-1.2.3.7605/lib/initializer.rb:494:in `set_root_path!'
        from C:/jruby/jruby-1.0.1/lib/ruby/gems/1.8/gems/rails-1.2.3.7605/lib/initializer.rb:459:in `initialize'
        from ./script/../config/boot.rb:44:in `new'
        from ./script/../config/boot.rb:44:in `run'
        from ./script/../config/boot.rb:44
        from :1:in `require'
        from :1

Ouch! Now let's check out the stack trace. The problem seems to be that C:/railstest/C: doesn't exist, which is correct. But why was it expected to exist? Let's work our way up the stack. In C:/jruby/jruby-1.0.1/lib/ruby/gems/1.8/gems/rails-1.2.3.7605/lib/initializer.rb, around line 494 is the method set_root_path!, defined as:


# Set the root_path to RAILS_ROOT and canonicalize it.
    def set_root_path!
      raise 'RAILS_ROOT is not set' unless defined?(::RAILS_ROOT)
      raise 'RAILS_ROOT is not a directory' unless File.directory?(::RAILS_ROOT)

      @root_path =
        # Pathname is incompatible with Windows, but Windows doesn't have
        # real symlinks so File.expand_path is safe.
        if RUBY_PLATFORM =~ /(:?mswin|mingw)/
          File.expand_path(::RAILS_ROOT)

        # Otherwise use Pathname#realpath which respects symlinks.
        else
          Pathname.new(::RAILS_ROOT).realpath.to_s
        end
      
      Object.const_set(:RELATIVE_RAILS_ROOT, ::RAILS_ROOT.dup) unless defined?(::RELATIVE_RAILS_ROOT)
      ::RAILS_ROOT.replace @root_path
    end

See the problem? In line 489, if RUBY_PLATFORM =~ /(:?mswin|mingw)/ doesn't return true if the RUBY_PLATFORM is 'java' on windows, which will be true when ENV['OS'] =~ /windows/i. So we just modify that line to: if RUBY_PLATFORM =~ /(:?mswin|mingw)/ || (RUBY_PLATFORM == 'java' && ENV['OS'] =~ /windows/i)

Now try jruby script/server again, and you'll get this error:

C:/jruby/jruby-1.0.1/lib/ruby/gems/1.8/gems/activesupport-1.4.2.7605/lib/active_
support/dependencies.rb:500:in `require': Could not find RubyGem jruby-openssl (
>= 0.0.0) (Gem::LoadError)

That's easy to fix, just jruby -S gem install jruby-openssl --no-rdoc --no-ri and you're good to go. I would describe where openssl joined the Edge Rails dependencies, but I can't seem to find that in the stack trace. If you're really interested, I'd suggest grepping through the code. So now, the WEBrick server actually works, and we should get the proper greeting and timestamp on /hello, but we don't! Instead, we get a 404 error in the browser, and the following stack trace in the server:


500 Internal Server Error
#<NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.each>
["C:/jruby/jruby-1.0.1/lib/ruby/1.8/webrick/httputils.rb:129:in `parse_header'",
 "C:/jruby/jruby-1.0.1/lib/ruby/gems/1.8/gems/rails-1.2.3.7605/lib/webrick_serve
r.rb:146:in `extract_header_and_body'", "C:/jruby/jruby-1.0.1/lib/ruby/gems/1.8/
gems/rails-1.2.3.7605/lib/webrick_server.rb:118:in `handle_dispatch'", "C:/jruby
/jruby-1.0.1/lib/ruby/gems/1.8/gems/rails-1.2.3.7605/lib/webrick_server.rb:78:in
 `service'", "C:/jruby/jruby-1.0.1/lib/ruby/1.8/webrick/httpserver.rb:104:in `se
rvice'", "C:/jruby/jruby-1.0.1/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'",
"C:/jruby/jruby-1.0.1/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'", "C:
/jruby/jruby-1.0.1/lib/ruby/1.8/webrick/server.rb:95:in `start'"]
[2007-09-24 09:27:56] ERROR `/hello' not found.
127.0.0.1 - - [24/Sep/2007:09:27:56 EDT] "GET /hello HTTP/1.1" 404 275

Since the stack didn't tell us anything useful, check log/development.log:

  Status: 500 Internal Server Error
  A secret is required to generate an integrity hash for cookie session data. Use config.action_controller.session = { :session_key => "_myapp_session", :secret
 => "some secret phrase" } in config/environment.rb

Aha! This session_key thing is new in Edge Rails, and you'll get the same error in MRI. I did a google search for "edge rails webrick parse_header" and found a ticket about it. So just do what they say, paste that code into config/environment.rb and try again. Make sure you don't just refresh the page, otherwise you'll get a "tampered with cookie" error.

So now we've got a regular JRuby on Rails setup working, let's see if we can turn it into a J2EE servlet using the Warbler gem.

jruby -S gem install warbler --no-rdoc --no-ri
jruby -S warble

That unpacks all of the Rails gems along with the GoldSpike Rails-Integration gem, and bundles everything into a WAR file for deployment. Then deploy with asadmin deploy railstest.war, and http://127.0.0.1:8080/railstest/ now gives the standard Welcome Aboard page, and http://127.0.0.1:8080/railstest/hello ... takes 20 seconds to load an error message. Ugh... Checking the server log in my default Sun J2EE SDK installation in C:\Sun\SDK\domains\domain1\logs\server.log, I see:

WebModule[/railstest] ServletContext.log():Failed to load Rails: No such file or directory - C:/Sun/SDK/domains/domain1/config/C: ... gems/rails-1.2.3.7605/lib/initializer.rb:494:in `set_root_path!

Which is the same Rails initializer bug that we previously fixed. Warbler unpacked the cached gem file rather than copying our modified code, so we need to apply the same fix to the upacked version in tmp/war/WEB-INF/gems, rerun jruby -S wable, redeploy the war file, and... still nothing. Warbler needs to know about jruby-openssl, so run jruby -S warble config, edit config/warble.rb and add config.gems = ["jruby-openssl","rails"], rewarble, redeploy, and after about 10 seconds of JRuby initialization time... SUCCESS!!!!

Now to be perfectly honest, I had a problem at this point. The first time I loaded /hello, I got this error:

Mysql::Error in HelloController#index
#28000Access denied for user 'root'@'localhost' (using password: NO)

Which meant that my production environment was missing its password in config/database.yml. Simple enough to fix, but it meant that I had to shutdown the J2EE server and restart it before I could redeploy the war file, since the JRuby interpreter was already running.

Anyway... Remember how the whole point of this exercise was to get RESTful resources working? Well, lets try a little something out. We won't make an ActiveRecord model, since that would require setting up a JDBC adapter, so rather we'll turn a Hash into XML.

class HelloController < ApplicationController
  def myrecords
    @myrecords = [{:a => 123, :b=> 234}]
    render :xml => @myrecords
  end
end

Rewarble, redeploy, and check out /hello/myrecords.xml . The fact that it returns type="array" in the root node means that ActiveResource's find :all method, as well as any methods that return multiple resource instances, will work correctly. Older versions of the to_xml method left that attribute out, causing the load method to fail.

It is now time to do a happy dance!

Web Programming, Hobo-style

2007-03-20T08:19:00.000-05:00

Today I embark on a great journey.

I've been reading about Ruby on Rails since my buddy Jeremiah recommended it. I really wish that I had spent some time up front getting into the Ruby language and the Rails framework, because it would have saved a huge amount of time on some of the projects that I've been working on over the last year. I finally decided that it was time to really dig in and learn the ropes, and started by watching the Ruby on Rails screencasts. In a word: Wow! All the great stuff that I had heard about Rails was true. It was just what I was looking for: a standardized system designed to handle all of the repetitive, tedious tasks that go along with building web applications, like creating data objects with dozens of properties to match your database columns.

I installed Ruby on my Windows XP box with no trouble using the one-click installer. But when it came time to install Rails through the gems package manager, I got an error at the end about how it couldn't auto-generate some bit of documentation. Then when I tried starting up a test WEBrick server, it bombed out with an error.

It turns out that there is a bug with the current Ruby Gems package. The instructions that I found didn't quite work, but here's what did:

gem env

In the output from "gem env," find the GEM PATH directory, go there, and delete the "source_cache" file from that directory.

gem update
If it asks you what to do with any specific gem package, just skip it and finish.

gem update --system
Although this would seem to be the logical step after deleting the source_cache, I found that it produced this error:

ERROR: While executing gem ... (Gem::GemNotFoundException)

Could not find rubygems-update (> 0) in any repository

But running "gem update" first, and then following with "gem update --system" fixed it.

gem install rails --include-dependencies
If you get an error that the rails gem is in an invalid format, try running "gem update rails" instead.

At this point, I got the message "Successfully installed rails-1.2.3" and I was good to go.

Booting WEBrick...

Rails application started on http://0.0.0.0:3000

Here's the rest of the Ruby on Windows installation process

Why to choose CSS over table-based layouts

2007-03-13T22:07:00.000-05:00

Not too long ago I was asked by a designer colleague, who was still figuring out the whole HTML design process, "So, why do we have to use all of this CSS positioning stuff to lay out this page? Why not just use tables?" The short answer that I gave her was basically that tables aren't accessible, and since we design sites for government agencies that are mandated to have accessible websites, we can't use tables. She then asked, "Why aren't tables accessible?"

I had to give that one a little bit of thought, but then I explained screen readers and linearization and all of that. How when an older screen reader looks at a page, it reads it from left to right, top to bottom, just like a human would. But it doesn't know that there's a table dividing the page elements, and so it reads across cells before continuing down into the next row of text, often combining parallel text from adjacent cells. This results in nonsense speech, which, as you can imagine, presents a problem.

So she went on her way, digging into the CSS positioning tutorials once again. A few weeks later I got to thinking: if an older screen reader reads text linearly, and a page is laid out using CSS positioning, then wouldn't there still be a problem? For example, if a navigation column is floated left of the body content, wouldn't a screen reader read left to right, seeing a navigation link and then a line of body text, and so on? This just didn't seem right, and it made me start thinking again about accessibility.

The answer that I came up with was pretty simple: Turn off the CSS to see the accessible version of a page. You can usually do this in your web browser, for instance in Firefox you can choose View : Page Style : No Style. Then, the browser falls back to rendering the HTML as the W3C intended. Everything comes out in block order, starting with the first tag in the body and continuing to the end of the body tag. Nothing gets laid out side by side, except for tables (which, of course, should only contain tabular data!) And if you're putting design graphics into background-image properties, they'd disappear too, leaving you with a nice, clean, accessible version of your page. Naturally, if you were to lay out a page with a table-based layout, and then look at it without CSS turned on, you'd still see your circa 1996 table layout in all its boxy glory.

If you have trouble getting your browser to disable CSS, try visiting CSS Zen Garden, check out how the page is laid out, and then click on the "Download HTML Source" link to see the unstyled HTML source. Looks quite a bit different, eh? Much more simple and to the point, but also without the emotion of the graphic design. Fortunately, screen readers don't do such a hot job of synthesizing speech with emotion, as far as I can imagine.

There is, of course, a standard objection to this motive: if a user disables CSS, then all of your effort designing the page is for nothing, but if you do it with tables and bgimage attributes then they're stuck viewing your genius, and who cares about accessibility anyway? There are lots of reasons to make your design accessible: more visitors can view your site, which means that most likely more visitors will view your site than would otherwise. Afterall, you don't want the pool of visitors who will link to you to be limited only to sighted people. Also, keep in mind that Google is visually impaired. Google doesn't care how interesting your design is. It cares more about the words on the page, and how difficult it is for it to get at those words, including how much non-content markup it needs to parse through in order to extract the content. By removing the table, row, and cell markup, you're vastly reducing the file size of your HTML and offloading that payload into a separate CSS file, making it easier for the search engines to access your content, which makes them as happy as an artificially-intelligent agent can reasonably be, and quite possibly improving your page rank. I should add that human visitors will also be happy with the reduced file size, since they only have to download the CSS file once and have it apply to every page on your site, whereas a table-based layout bloats every page.

Access + ASP = Bad

2007-01-03T12:20:00.000-05:00

So here's the dilemma: Windows is still the most widely-used platform out there, and Office holds the lion's share of users, due to the ease of acquisition. Most folks who buy their first machine buy it off the shelf at a retail store or an online retailer like Dell, and it comes with Windows pre-loaded. Since they figure that running a Microsoft product on a Microsoft operating system is probably a good idea, they either go ahead an purchase Office, or again get it pre-loaded. If they splurge for the extras, Access is in the bundle, and they start using it for their database work. When they start doing web programming, they naturally drop an Access .mdb file onto the web server, write their scripts in ASP since it's the easiest thing to run with IIS, and use ADODB within ASP to access the Access file. You can Google a ton of tutorials on how to put drive a web page with an Access database in this manner, which is a testament to the technique's popularity.

The problem starts to appear when you start programming more complicated databases, such as multiple tables and multiple-step drill-down queries. If you think your script through procedurally, you tend to come up with a process like:

Find all records of table A, and get foreign key bID that points to ID field of table B.
For each record in A:
"Select * from B where ID=" & A.bID

This works fine for small do-it-yourself websites that don't get many page hits, so a lot of web developers who are starting out tend to do it this way (guilty!). Especially if they either don't understand table joins, or would rather not pull repeated data for A in each record. After all, with joins you'll wind up getting data like this:

A.ID, A.stuff, A.otherStuff, A.bID, B.ID, B.stuff, B.moreStuff

So it looks like you're wasting data by getting the A data in each returned record. Whether or not you actually are is determined by the underlying database driver implementation, but that's a story for some other time, and possibly someone who knows more about the guts of databases.

Anywho, as you start building bigger sites that handle more traffic, you suddenly get calls from angry clients who demand to know why it's taking so long for thirty simultaneous requests to retrieve data from their 300MB Access database. At this point you start researching why Access is bad, and you may or may not come to find out the dirty little secret: Access was never intended to serve as the backend to web sites. In fact, it was never even intended to serve any multi-user role. Thus, this shared ADODB stuff is sort of like black magic. Sometimes it works, especially for under ten users and smaller amounts of data. But it doesn't scale well.

There are ways to save the day without having to totally move to a different DBMS. If you cut out those queries that loop through sub-queries, either by converting them to joins or by temporarily caching the sub IDs that you need to look up, then you're in better shape. But sometimes you're dealing with totally unrelated data tables, and you need to perform multiple queries, retain the records, and use them later. The ActiveRecord design pattern, which I'll talk about some other time, takes care of this by allowing you to create an object with data members for each database field, which can also access the database to select/insert/update/delete (CRUD). You can then stick those objects into containers for safe keeping. ASP.Net gives you a workaround with the DataSet class, which creates an in-memory version of the query result set.

But what if you need something that's faster and easier than creating a class, to get you through a pinch, like if it's gotta be done right now, but not necessarily the right way? Enter the Execute() function, which dynamically evaluates and executes a string, as though it were code. Most scripting languages have some means of doing this, so check out your language of choice - it'll usually be named something along the lines of execute, evaluate, or a shorter version like exec, eval, etc. The function takes the string and treats it exactly like real source code, but evaluates it at run time. So for example:

for each f in rs.Fields
tvar = f.name
tval = f.value
Execute(tvar&"=tval")
next

This will loop through each field in a recordset. For each field it sets a variable tvar equal to the field name and value, eg tvar = "ID", tval = 123. Since the value of tvar is "ID", the string being passed into Execute() is "ID=tval", which is the same as doing it in plain source code. The beauty of it is that if you have a lot of fields to account for, you don't have to code it all by hand, like:

ID=rs("ID")
stuff=rs("stuff")
moreStuff=rs("moreStuff")
'etc

So you get a lot of bang for your buck, especially when dealing with lots of database fields.

The most important lesson: when dealing with Access databases through ASP, the most important thing is to make sure that you're closing your ADODB.Connection and nulling it before opening a new connection, and only performing one query per connection. So rather than doing this:

set db=Server.CreateObject("ADODB.Connection")
db.open("DSN=mydb")
set rs=db.execute("SELECT ID, bID from A where x=" & x)
set rs2=db.execute("SELECT * from B where ID=" & rs("bID") )
...
rs.close:set rs=nothing
rs2.close:set rs2=nothing
db.close:set db=nothing

Make sure you do this instead:

set db=Server.CreateObject("ADODB.Connection")
db.open("DSN=mydb")
set rs=db.execute("SELECT ID, bID from A where x=" & x)
bID=rs("bID")
rs.close:set rs=nothing
db.close:set db=nothing
set db=Server.CreateObject("ADODB.Connection")
db.open("DSN=mydb")
set rs=db.execute("SELECT * from B where ID=" & bID )
...
rs.close:set rs=nothing
db.close:set db=nothing

It may seem redundant, even painful, to do so much opening and closing. The pain is an indication that it may, in fact, be time to switch to a more capable scripting system or database management system.

Discover the Hidden Ability to Post .Net Pages

2007-01-02T13:27:00.000-05:00

If you're using a web hosting provider that allows you to script classic ASP pages, and you're curious to see if you can also publish ASP.NET pages, here's a little secret. A lot of servers built of newer hardware, with newer versions of Windows operating them, can and do handle .Net pages even if the hosting company doesn't realize it and hasn't officially acknowledged it or supported it. So if you want to see if you can step up to .Net on your server, simply upload a file with the extension .aspx and load it in your browser. It'd probably help to put some actual .Net data binding code in there to make sure that you're seeing a script in action, or at least a page_load event. But anyway, there you go. Just because a host doesn't explicitly state that they have the capability to run .Net doesn't mean it isn't there.

The lowdown on Office Web Components

2006-12-21T10:01:00.000-05:00

I recently mentioned the possibility of exploring MS Office Web Components as a replacement for server-side Excel automation. It turns out that for my particular problem, it didn't present a solution.

The problem workflow:
Client sends an Excel file as an email attachment. The spreadsheet contains tabular data. Depending on the particular data domain, it could be a phone directory, a report describing the status of their various projects, or whatever. The important thing to note is that they store this stuff in Excel, and that's what they give us as source material. Our Perl scripts then need to create HTML/PHP based on the input data, or possibly build SQL update commands to populate their web database. Web pages and SQL command files are then copied onto our local server for testing, and once everything is proofed, it gets sent by FTP to their live server.

The key problem:
Extracting data from Excel on a web server is "difficult."

Specifically, the programmer who first wrote this suite of automated scripts used Excel's Automation COM objects to do the data extraction. These COM objects actually control Excel, open/run it on the server, hide the application's GUI, open workbooks, create data ranges (eg. A1:E256) and then get that range as an array of values. Once it's done, it closes Excel. Or does it?

When running automation through a web server, IIS manages its own garbage collection. So when you close Excel and delete the reference to the application object, that should be the end of it - the process should terminate. But IIS doesn't actually delete the object until it collects garbage, so you end up with a zombie Excel process which must either be killed by fancy sysinternals programming, or just reboot the server at that point. If I'm around, I kill all the Excel zombies; if I'm not around, and nobody else knows how to kill, then someone has to reboot the server.

So how would you do Excel-like things on a server? MS took care of half of the problem with Web Components, which give a lot of the same functionality as the office programs. So for instance, you can web script the Spreadsheet component and use it to populate a sheet with data and dynamic formulae, and then export it to an Excel file or XML spreadsheet. But you can't do the opposite, ie. open an existing sheet to get the data from it, which is what I need.

So I moved on and discovered a new third option, but it required making a small change to the workflow. Rather than directly uploading an Excel file to the server for processing, the account executive who receives the Excel file from the client would first have to save as a delimited text file. This way, Perl on the server could just open it up and read it line-by-line. It would cut out a lot of the problems we've had in the past with having to wrap everything in UsedRange sub calls, which prevented the script from looping beyond the part of the sheet that had been filled in and off into infinity whenever the client forgot to place their END token at the end of the data. But it introduced more tedious work and could easily be scripted.

Introducing: Windows Script Host. You may have heard of it and wondered what it was. It's basically MS's answer to AppleScript. It allows you to use VBscript or JScript, along with whatever ActiveX/COM objects that you have access to locally, to build quick and easy scripts. If you need an interactive GUI, you can build the scripts into an HTML Application, which gets a .hta extension, is double-click runnable from Explorer, and runs inside whatever Internet Explorer monstrosity you happen to have. Then you can do the usual HTML form scripting operations, button onClick methods, etc. The script itself is pretty easy: create an Excel.Application instance, use its Workbooks.Open(filepath) method to return a handle to the Excel .xls file, then call its SaveAs(destination, filetype) method and pass it the proper type number for tabbed text. Now the AE can upload the text file rather than the Excel file.

Possible improvements: If processes that could be done just as easily in VBScript or JScript are currently done in Perl on the server, then the whole process could be done on local hosts. But there might be regular expression parsing or things like that, that remain better done in Perl, in which case it might be better to move the local script into a button onClick event in the web form page, with special security permissions on the local hosts allowing the script to start up Excel locally, do the conversion to text, save it with the original Excel file, and switch the form's file upload input to reflect the change so that the text file gets uploaded. That'd be pretty sweet.

Either way, my team has plenty of refactoring to do in the new year. If you find yourself in a similar situation, where you have a system that "works" but could certainly be done better, and you can't risk breaking the whole thing and starting from scratch, then check out Martin Fowler's excellent book Refactoring: Improving the Design of Existing Code

UPDATE: I forgot to mention that the Office Web Components are going by the wayside, which was another reason to not get into using them, even if they had been able to do the job.

Why Excel OLE Automation through the web is Bad

2006-12-18T15:30:00.000-05:00

In the category of "sub-optimal things that I have to maintain," one of the ugly ones that rears its head from time to time is Excel automation. We have a client that sends us data tables in the format they're comfortable with, which happens to be MS Excel spreadsheets. We then have a legacy Perl script that uses a Win32::OLE connection to the Excel.Application COM object to open the Excel file, which first gets uploaded to the server through a plain old CGI multipart form attachment.

The issue at hand, as described in MS KB 257757, Considerations for server-side Automation of Office, is that running Excel server-side through IIS causes an instance of Excel to be spawned within the memory space of IIS, or something to that effect. Whatever is really going on at the low-level, the end effect is that an instance of Excel is left running on the server, even after you go through the normal procedure of calling the Quit() method and setting the application reference to nothing/null. And with the number of files that we process, we get a lot of zombies on our test server. Occasionally I get notified that an update process returned a blank page rather than a confirmation, and I check out the process monitor, and sure enough there are a bunch of Excels that aren't doing anything other than hogging resources and preventing Excel from actually running. Normally you can't even end those processes, since the IIS user spawned them, so you have to either reboot or use a fancy non-standard "kill" script.

If you ever find yourself in a similar situation, there are a few ways around it. I've found that running the script on the server via Scheduled Tasks allows Excel to quit. So you can set up a task that runs your script periodically, checking for the existence of the uploaded file. Your processing won't be instantaneous, but it'll be more reliable. There's also a possibility of using Office Web Components, which includes a COM object for a Spreadsheet with a similar object model to Excel Automation. I'm actually going to check that out to see if a) we have it already on the server, and b) if it can open Excel files on the server the same way that the Automation does, and if so, I'll convert our scripts to use that rather than Automation.

This Is What The Web Sounds Like

2006-12-05T23:08:00.000-05:00

"CLiCk, Speak" is an extension for the Firefox web browser that provides a simple text-to-speech interface for web pages, intended for sighted users. If you don't already have Firefox, you're missing out! Click the link toward the bottom of the right-side column of this page to download Firefox, then download the extension, and listen to the web.

It'll give you a nice feel for some of the accessibility issues that you'll run into when designing sites, as you can listen to a simplified version of what a real screen reader would speak, minus the cues about what HTML element is being spoken. It's also handy as a productivity tool, freeing your eyes to do other things while you listen to web pages being read to you.

I haven't posted much lately because work has been crazy with a capital K. I've been promoted to Manager of Programming and placed in charge of the Web Production team, where I still handle dynamic web site and application programming in addition to project management, and now dynamic print job compilation. In the last few months I've learned more about PDFs than I ever intended, as well as gotten my first exposure to using Quark XPress for production work. I've had to hire a new programmer/designer, and we're in the midst of hiring for a HTML coder position.

Since I've picked up the lead on all of our web projects, I've become aware of many issues that I hope to write about when time permits. Among them:

Why you shouldn't perform OLE Automation tasks through web scripts (eg uploading Excel files to a CGI script that grabs data ranges and processes them)
Using dynamic evaluation functions as a quick and dirty substitute for the ActiveRecord pattern
Why performing multiple queries against Access databases in classic ASP through ADODB doesn't scale well, and how to improve performance as much as possible (see above)
How to discover hidden .Net potential

More about CLiCk, Speak

The Importance of Cohesion

2006-10-11T09:42:00.000-05:00

I haven't gotten to post in a while, because I've been playing catch-up at work. We've lost a number of key people since this past summer, and the rest of us are rushing to pick up the loose ends. So I thought this would be a nice time to not only vent, but also write about the importance of cohesion.

Cohesion, as it applies to source code, is a measure of how well each individual command or statement works with related commands and statements, whether they be related by proximity or structurally within a method or procedure. Ideally, you want a natural flow of data from one statement to the next, with abrupt disconnects separated by method calls or other such syntax. Creating a variable that never gets used, or isn't used until much later in the code block, would be a bad example of cohesion.

Web scripting, usually being taught in the "Get 'er Done" methodology, is very susceptible to bad cohesion, particularly when implementing the beast known as the "Magic Scriptlet" anti-pattern, where the entirety of an application is coded in one file full of nested IF statements, POST variable switches, and function definitions. Concepts like includes help to reduce the complexity, but only when done properly. There is a tendency for programmers to discover modular programming concepts, and take them a bit overboard, to the point where following the execution of a script through multiple source files becomes a nightmare.

Cohesion also applies to teamwork. If one key member of a team hoards knowledge, whether it be project status, file locations, or design philosophies, it's not going to suit the overall performance of the team. It may feel like a personal job security measure, making yourself indispensable, but the downsides of having to keep track of your esoteric obfuscation methods will ultimately weigh down the project, and you end up shooting not only yourself, but all of your team members, in the foot.

XPath Trick: Count() Function

2006-09-28T08:26:00.000-05:00

For those who use XML to mark up their data, you're probably used to seeing statements such as the one below, that returns a set of nodes that each have a specified child node as per the criteria:
"/root/record[lastname]"
This would return all record nodes that are children of the root node, but only if they have a lastname child node. If there are any record nodes that don't have lastname children, they won't be returned in the resulting nodeset.

But what if rather than search for nodes that have a certain child node, you want to search for all nodes that don't have that child node? For instance, recently I worked on a project where a last-minute change was applied to the specs, where a certain record only applied to one set of data, but not the others.

The thing was, they wanted a side-by-side comparison between these data sets, and the design called for showing the record in the relevant set and showing blank data cells for the other sets. Also, each data set could be viewed individually, and they wanted it to not show in the other data sets.

So I had an interesting problem. I needed a way to extensibly tag that particular record to indicate that it only applied to the one data set. I knew that I could set up a node named something like "includethis" in every other record, but that seemed like overkill. It would be much simpler to do an XPath query similar to:
"/root/record[!excludethis]"
i.e. Return all nodes that don't have the excludethis child node. But that particular syntax doesn't exist in XPath, so I had to find another way. Luckily the answer turned out to be pretty simple.

The standard XPath functions include a Count(node-set) function, that returns an integer. So for instance, if you used something like:
"Count(/root)"
you should always get a result of 1, because you can only have one root node in your XML tree. So the answer to my exclusion problem is as such:
"/root/record[Count(excludethis)=0]"
Which does exactly what I want: only return record nodes where the number of excludethis children is 0, i.e. there aren't any. This way, I only need to tag the one record, and I can leave all of the others alone.

If you know any other snazzy XPath tricks, feel free to leave them in the Comments!

Measuring Time

2006-09-25T12:44:00.000-05:00

Mantra: If you expect to improve a process, you must first be able to measure its efficiency. Otherwise, you don't know (other than intuitively, ie just having a hunch) whether or not you're making progress.

As developers, one of the ways we can improve our work efficiency is by measuring the time we spend on a task. This gives us a number of benefits:

We can measure and compare times from one execution to the next, make observations about the methods we used, and determine what works better by comparing the methods used against the time spent.
We can use the statistical data to create better estimations of the time it will take to complete a task. This can be a huge benefit when preparing project bids.

To make it easier to keep track of time, you can use a tool like Rachota to automatically keep track of how long you spend on a task. You tell it what you're working on, do your work, click the Relax button if you take a break, and click Done when you're done. You can switch tasks in the middle, too. Once you've done your work, it gives you graphs of time spent so that you can easily compare the figures.

Remember the mantra: Measure, Improve.

How to Not Frame the Issue

2006-09-18T10:54:00.000-05:00

Remember when it was cool to have a frameset on your site? Me neither. Granted, I've been guilty of placing the things on my old free-hosted site back in the day, but I since learned the error of my ways. And now you can, too!

Why You Might Want To Use Framesets

Many developers want a consistent layout that's also easy to manage. When learning HTML and getting into greater numbers of files on your site, you reach a point where creating a new primary link from your navigation bar becomes a huge chore, because you have to edit each file on the site and plug the link into the proper place. Using Framesets lets you create one navigation file, and divide the rest of the site into content frame pages. That way, you only have to edit the one navigation file when a new link comes along.

There's also the keep-em-0n-site effect. If your site has outbound links to other sites, you'd normally want to use the target="_top" attribute on your link tag so that when a visitor clicks the link, their browser opens the linked page at the main browsing level, rather than opening it inside your content frame. But, what if you want to allow users to keep seeing your navigation frame even when they've clicked a link to another site? That way they can always click back into pages on your site, because it's in sight and in mind. And maybe you have some kind of advertising banner that you want to optimize by keeping it in view as much as possible.

Stop the Insanity! (Please)

Now, here's why those reasons to use framesets don't hold up, plus a few extra for good measure.

First, the issue of what to do when your site is so big (how big is it?) that you need to spend all day pasting links to new pages. Whatever into-to-HTML page or book you've been reading might not have mentioned it, or maybe your skills haven't gotten to that point, but there have been a few add-ons to HTML that will be a big help to you.

SSI allows you to grab the contents of one HTML file and embed it inside another file. This gives you the same effect of the navigation frameset, where you only have one file containing navigation links. The only differences are that each file on your site will explicitly include the navigation file, and there won't be a frameset. To do things the SSI way, you'll probably want to rename your file extensions from .htm(l) to .shtm(l), the S standing for SSI. That tells the web server, "Hey, this file is somewhat dynamic, and will require actual processing rather than just fetch/retrieve." Then, where you want to include another file, you use this comment code: or:
Most scripting languages also have some mechanism to do this, for instance in php, include("file.php");

You can also use a feature in Dreamweaver called Templates, where you create a template file, which has static and editable regions (parts of the file). Then you create your site files based on the template. Any time you edit the static parts of the template, Dreamweaver automatically rounds up all of the files based on that template and updates them to reflect the change. You can use this if your site doesn't support SSI or scripting.

Now for the other issue, regarding keeping users on your site: just don't! For one thing, it's annoying to your visitors. You may think you're doing them a kindness by allowing them to access other parts of your site even though they found a link to another site and followed it, and they can still instantly get back into your site. Well, think of it this way: if there was other content on your site that they were interested in, they would have bookmarked it. Or Digged it, or del.icio.used it, or just opened the link in a new tab/window, or viewed the other site and then hit the Back button or History to get back to your site. By leaving your frameset sitting there, you're cluttering their screen real estate, and forcing them to look at something they neither want nor expect to see. Not to mention, you could be violating the terms of use of the site to which you are linking: some of them have rules about not bringing their content into a frameset, and your site might end up on their list of banned referers. Your users probably don't want to click a link from your site, only to get a message that you are a naughty boy/girl.

Here's the best reason to drop the framesets: search engine optimization and accessibility. Think about it: in order to get at the actual content of your site, a user agent (user using a browser, or a search engine spider that is automatically following links and recording what it finds) has to make more connections to your site if you use a frameset. First, it requests the frameset file. (GET /index.html) When it reads that file, it finds your frameset tag, and the nested frame tags with their src attributes pointing to your navigation file and home page content file. So now, it has to make requests for both of those files. (GET /nav.html) (GET /home.html) This extra time spent getting the files reflects poorly on the value of your content, and thus, your search engine ranking will take a hit. Also, if a visitor is using a non-visual browser, then they're going to get a prompt asking them which frame they want to get information from. If they don't know which is navigation and which is content, they might make the wrong choice. And if your content pages don't link to the rest of the site, then they have to switch frames into the navigation frame in order to browse through your site. You want things to be simple for your visitors, to ensure that they refer you to others and keep coming back to your site. You don't want to turn them off by making things more difficult than they need to be.

If you've never worked with accessibility in mind, try downloading the Lynx browser, which is a text-only interface and does a pretty good job visualizing your accessible content. It's also a piece of history, and knowing how to use it will improve your geek-cred.

Making 1300 Sales Calls - 30 Day Trial

2006-09-13T09:13:00.000-05:00

Personal Development blogger Steve Pavlina has a great insight into tackling insurmountable daily to-dos, by approaching the endeavor as a 30-day trial. The rationale:

30 days is a short enough time to suffer through a new process. It's not like you have to do it every day forever, just for 30 days.
At the end of the trial, you'll know the costs and benefits of the process, so
You'll have a better idea of whether or not it's worth continuing for the long term
You'll have gained whatever benefits come from having executed the process

This can be a great way to evaluate a potentially big change in your life, because you're not making a permanent commitment to it. In particular, Steve points out this example of a daily habit that can help your web design business:

Make 25 sales calls every day to solicit new business. Professional speaker Mike Ferry did this five days a week for two years, even on days when he was giving seminars. He credits this habit with helping build his business to over $10 million in annual sales. If you make 1300 sales calls a year, youÃ’re going to get some decent business no matter how bad your sales skills are.

And making those 25 sales calls each day will be easier when you use the Web Site Cold Call Helper.

Label: Use the Right Tags

2006-09-11T08:39:00.000-05:00

The longer you surf the web, the more common design elements you notice. One of things that clients (and bosses) asked me to do repeatedly, was to format forms in a two-column table, with the left column containing bold tags that labeled the input in the right column, with each column aligned in the opposite direction so that they lined up nicely in the center. There are three problems with this approach:

Bold tags, while making it easier to tell that they are describing the input rather than just adding content to the page, don't translate well to non-visual presentation media. And...
Non-visual browsers can't easily relate the inputs back to their label text, so you have to keep moving back and forth to figure out what input you're filling in.
Bonus problem: A table is meant for holding tabular data, rather than for laying out text. In a two-column layout with only labels and text inputs, this isn't much of a problem, as we can just fill in the summary and caption with "for layout purposes." However, if you're trying to do things like placing tons of descriptive text into the label column and making it render as multi-line, then you run into problems with screen readers; they'll read the first line, then the input, then the rest of the label lines.

The issue of how to column-format forms is a whole other can of worms, so I'll address that another time. For now, lets focus on the benefits of label tags:

Non-visual browsers gain a way to easily tell the intent of an input field.
Users of visual browsers can click the label text and send focus to the input, making it easier to move to it by giving a larger clicking target.
You can use the label to convey an access key through visual styling like underlining, and allow users to instantly jump to an input without having to tab through all of the preceeding inputs.
When using scripts (server- and client-side) to inform the user of validation errors, you can apply distinctive styles to the label to catch the user's attention. For instance, make it red. Your script gains the same benefit as a non-visual browser in that you can more easily find the label text associated with the input.