Tektastic

Techtarget's Java ServerSide Symposium 2009 Impressions

2009-03-22T21:16:00.001-07:00

Just got back from Vegas where between Wednesday and Friday a small-ish java conference was taking place. This was my second one, the first being five years ago in 2004.

Of course it's hard not to compare, and I find myself doing it anyway... I can't help but feel that the excitement has left java server software. Remembering one of the very first java conferences in Sydney 1995, when it was still almost academic and less of an industry conference, I feel like we've come to an end of an era.

The tedious reality of maintaining enormous enterprise systems built over the the last decade has sinked in, and unfortunately there is nothing exciting about it, but everything difficult and complex. Those less fortune whose systems took advantage of EJB1 and EJB2s will suffer the most, as those are dying technologies and are rapidly being replaced by simpler more component friendly software.

The first set of simplification came with frameworks over the last five years - Spring and Hibernate being the most notable. Now the simplifications are being pushed out across language boundaries, because of the need to stay competitive against exploding Ruby on Rails community, and even smaller but still disproportionally more productive frameworks like Django, when compared to java.

Several talks focused on jRuby, and I think this is where the future power lies. Combining the two platforms, Ruby on Rails for web development, and server side java for multi-threaded, asynchronous, transactional behavior seems like a fantastic opportunity.

However nobody is yet talking about how this all works together in production. What does it mean to run a multi-app server distributed java application, with a RoR application hosted inside JVM and interacting with the service layer? What if service layer mixes ActiveRecord and Hibernate? Questions like this, while entirely logical in early evaluations of emerging technologies, were simply beyond the depth of TSSJS09.

But for what it's worth, there was a decent overview of a multitude of scripting languages: javascript, ruby and scala, and a very impressive talk on Ruby meta programming.

There were several exciting news for me personally, the first having to do with Flex and BlazeDS integration in Spring 3.0. Basically what this means is that it's now possible to build a typically modular and well understood server-side backend architecture in Spring, the client side in Flex, and have them directly converse over binary AMF protocol, which is highly compressed and optimized to send data back and force. I believe the entire stack is free, except Flex Builder Eclipse-based tool that's not free. My exposure to Flex is minimal, but I love the ability to build beautiful application UIs that run consistently within the browser, and expose your backend in a way of clearly defined REST/XML api. And the load time in demos appeared significantly faster compared to java RMI applications I've had experience working with.

I'll add part II in a couple of days.

Binary PostgreSQL installer broken when used on RedHat EL5.3 / Dell PowerEdge 2950: Hangs server reboot.

2009-02-06T14:46:00.000-08:00

This is a very quick post that would hopefully save someone else hours of hitting the head on the wall like I did. If you are installing RedHat Enterprise Server 5.2/5.2 x86_64 on Dell PowerEdge server, be aware of the following issue with PostgreSQL binary installer offered by EnterpriseDB.

After installation the server is unable to cleanly shutdown due to the fact that the binary installer does some voodoo with libtermcap - basically making this system library be used from inside postgres installation folder. Because of this, kernel can not umount /usr (or whenever your PostgreSQL is installed) and hangs the shutdown.

Solution:

Build PostgreSQL from the sources. They are ahead in minor version number anyway.

Thanks.

The Magic of Outdoor Raving

2008-08-19T17:28:00.000-07:00

The campout happened, and it was an enormous success, everyone seemed to have had particularly special time this year, and I have to join the ranks. Check out just the insanely intricate lighting show put up by the Radiant Atmospheres crew.

I was involved in two music sets this year - a Friday evening downtempo set by PolyGroovers (with live appearance by Telene and briefly Fabienne), and then Sunday morning solo live set. The PolyGroovers stuff had many new tracks from the upcoming album, so that set won't be available for download, but the morning set is below:

LeftCtrl Live at Friends and Family Music Campout XII @ Willits CA (109Mb, 74min)

The above set has two songs composed together with Kem (Filip Jach), and one with Joseph Lee of Progress/SF.

Please feel free to download and leave your feedback as comments!

PolyGroovers are back!

2008-07-24T12:31:00.000-07:00

Tonight we are packing up and going camping for the weekend, to a music campout in Willits, where we'll be performing live on Friday night. A full mixed set will be published here next week. Very exciting stuff, and as a bonus I've got a new song to share...

Funk In A Pond (11Mb, 8mins)

This is an extended mix, which is why it's 8 minutes long, and meant to be played live. The album version will be much shortened.

Please feel free to leave comments with your feedback. Much appreciated!

Tell Your Mana About Obama, and Obama Car Art

2008-07-17T12:26:00.001-07:00

Obama's Potential Presidency

There is plenty of excitement around Barack Obama's potential presidency, and I think there's a lot to be excited about. There's is an overwhelming feeling among people I talked to in San Francsisco, how Obama is "Our President" — not a poster boy for old money or a powerful family. The fact that the Internet fund-raising helped Obama to compete and win against the Clinton clan is a testament to the truly new era, a more democratic one, where if someone can fire up a lot of people they can rally up quckly and effectively. This efficiency is something to look forward to, if and when Obama becomes a President.

Whether or not all the people behind Obama will continue to support him, is another question. People tend to idealize their heroes way to quickly, and that's also dangerous. The recent backlash of liberal bloggers about Obama's compromise vote — is a great example. Politics is still politics, and nobody ever gets their way all the time. Everyone must compromise at some point.

But enough said, I think that overall Obama is a great candidate, and will hopefully bring a new fresh perspective into the Washington. So, how do we get there? Well, donating money is, for once, a very easy thing to do thanks to http://barackobama.com. But now there are other ways.

Car Art Contest

Infectious.com (the company I work for) is running an open contest for Car Art Submissions around Obama. There are some restrictions - some specific words can not be used, but it's an amazing opportunity for artists around United States to create Obama Car Art that can be seen by thousands of people commuting daily in the next few months before the election. This is as grass roots as it gets.

If you know any artists that may be interested please send them to Infectious. Or you can grab the banner above and put it up on your blog. Either way - thank you!

Firefox3 PR Backfires?

2008-06-17T11:25:00.000-07:00

After announcing to the world that Firefox 3 will be available at 10am PDT today, June 17th, amidst apparent publicity stunt to generate the record number of downloads per day (has anyone been really keeping a tally?) the web site http://getfirefox.com/ returns the pathetic

Http/1.1 Service Unavailable

Firefox is certainly a decent browser, but it's stability seems to have gone reversely proportional to the self-professed greatness and the size of their publicity stunts.

Safari is now by far the best looking, and fastest browser for Windows, and it has been that way for mac for a while.

Good luck Fifefox 3 (when we see you), but I for one, am not buying it.

Update @ 12:23pm

OK, so finally FF3 is available for download, and I have it installed and running. The first thing is that not all plugins are compatible, and if you are using FireBug for web development, you must install the 1.1 version available here.

Infectious.com launches!

2008-06-11T16:39:00.000-07:00

It finally happened :) I've been working for Infectious.com since about July last year, and it's been a really fun run building a brand new product, from a mere conception to this moment - full public launch.

Of course some people may have caught the glimpse of this from TechCrunch article a few weeks ago. Nevertheless, here it is - Car Art For the Masses - available to you from a spanking new website, designed and developed by the Team Infectious and yours truly.

The idea of course is simple - everyone's car looks the same! OK, so some are dirtier than others, some stink more, all of this is certainly true, but from an aesthetic point of view, if you and your neighbor accidentally buy the same model of a car, you are screwed! Well, not really, and especially not anymore.

Stop wasting time and go get some cool art for your car, make it look like nobody else, even if you drive a stinky dirty '74 Datsun. I'm seriously.

Aptana - Please Don't Suck Any Harder

2008-04-22T22:23:00.000-07:00

I wrote this post after hours of frustration, trying to fix my broken environment, and losing valuable time on a project that was falling behind.

I now realize that the post was overly emotional, and probably not entirely fair to the Aptana team. I'll post updates here as I work through the issues to get my RadRails working again.

Car or Auto Make-Model-Year Database : For Breakfast

2008-03-13T16:42:00.000-07:00

Make Model What?

If you like me were tasked with loading a database of recent car makes/models/years, you would start by looking on the web and seeing if someone else just has it out there, readily available, hopefully for free, but perhaps for a tiny nominal fee.?

If only it was that simple...

I looked and looked, and couldn't find anything that would fit the above requirements. So I thought, who would know about US car models better than Kelly Blue Book? So I went on their site, and sure enough they have a javascript file that lists all known to them makes and models of used cars. Since the file is public, I figured it's not really "evil" if I scrape and parse it for my own benefit. Disagree? Have a better source? Then leave a comment.

Anyway, to cut the long story short, I'm hoping to save a day or so to someone else who may, like me, be looking for this information. The ruby module shown below retrieves and parses the javascript from KBB site into a Ruby data structure of the following form - basically a hash, keyed on make, then on model with list of years as a value:

>> Constants::Auto::DB.keys.sort[0..5]
=> ["AMC", "Acura", "Alfa Romeo", "Audi", "BMW", "Bertone"]
>> Constants::Auto::DB["Subaru"].keys.sort[0..5]
=> ["B9 Tribeca", "Baja", "DL", "Forester", "GL", "GL-10"]
>> Constants::Auto::DB["Audi"]["A4"]
=> ["1999", "2007", "1998", "2006", "2005", "1996", "2004", "2003", "2002", "1997", "2001", "2000"]
>> Constants::Auto::DB["BMW"]["X5"]
=> ["2003", "2002", "2001", "2000", "2005", "2007", "2006", "2004"]

The idea is that you could load the initial hash: @models = KBB::Parser.new.to_hash and then save the output of @models.inspect in your local constants file - hence me using Constants::Auto::DB (I actually have a Rake task for doing this -- let me know if I should post it too). Then you would just re-run this every time you think new car models are added/changed on KBB. Realize, that hitting their site every time you need the data is clearly evil. So use this class to load the data initially, save the result of inspect() call into a ruby file, and use that cached version in your app. Re-run the load every time you want to update your database.

Please let me know if you find this code useful, or if you find a better/cleaner/more comprehensive way of maintaining car make/model/year database.

#
# author: Konstantin Gredeskoul © 2008
# license: public domain
#
require 'net/http'
require 'uri'

module KBB
 MODELS_URL = "http://scripts.kbb.com/kbb/ymmData.axd?VehicleClass=UsedCar"

 class Models
   def initialize(js)   
     @models = {}
     @makes = {}
     n = /ymUsed_\[\d{4}\]\s*=\s*'([^']+)'/
     m = /ymmUsed_\["(\d+)~(\d+)"\]\s*=\s*"([^"]+)"/
     js.split(/\n/).each do |line|
       next if line.strip.blank?
       if matched = n.match(line)
         matched[1].split(/,/).each do |token|
           id, name = token.split('|')
           @makes[id.to_i] = name
         end
       end
      
       if matched = m.match(line)
          year, make_id, models = matched[1], matched[2], matched[3]
          models.split(/,/).each do |t| 
            id, model_name = t.split('|')
            make_name = @makes[make_id.to_i]
            @models[make_name] ||= {}
            @models[make_name][model_name] ||= []
            @models[make_name][model_name] << year
          end
        end
      end
    end
    
    def to_hash
      @models
    end
  end

  class Parser
    def initialize
      @m = Models.new(Net::HTTP.get(URI.parse(MODELS_URL)))
    end
    def to_hash
      @m.to_hash
    end
  end

end

Beaches, Aussies and Christmas

2007-12-04T01:47:00.000-08:00

It's been a little quiet here, as things were a little busy towards the end of November, and now I am away on a short trip to Australia visiting my family, relaxing and sun-bathing on fantastic beaches of Queensland, while sipping overpriced Jameson whiskey :)

Australian Christmas is in full effect, with enormous pines towering over hot beaches fully decorated in typical X-Mas crap, while in the former hippie town Byron Bay pines are dressed with shiny peace signs.

Australians just had an election, where labor party (a weak parallel of US Democratic party) have won -- a welcome change of government after 11 years of Liberals and John Howard (smart, but creepy suck-up to George Bush). The new power has immediately ratified Kyoto protocol in their first week "at the wheel". It's refreshing to see reusable green bags at the supermarkets here, just as they are becoming a norm in San Francisco.

A weird fact discovered today -- capitalism does not mean the demand is satisfied with supply. Beach mats are not heard of in Byron Bay - a beach town with $50 beach towels. There's a few things Australia can learn from third world Thailand, where beach mats were everywhere.

Signing off for now, but stay tuned, as I'm planning a PostgreSQL performance tuning post in a few weeks. This seems like a subject not well covered widely, not sure why.

Domain Registration Mayhem

2007-11-12T10:59:00.000-08:00

I don't think it's a secret that the domain registration business is kinda whack. Just look at all the domain parking companies out there which take "shady business" to the whole new meaning. What I don't understand is the rationale behind some of the rules that govern domain business. For instance...

How does NetworkSolutions Inc gets away with charging $34.99 a year?

Seriously... WTF? I am hosting my domains with 1&1 Hosting, which charges $6.99 a year. This seems like a reasonable price, but here's the main reason the guys like NetworkSolutions are still in business:

You can not transfer your domain 2-3 weeks prior, and two months after the date of your domain renewal.

This dead simple, but annoying rule guarantees that everyone who WANTs to transfer their domain, but is trying to wait till last minute and/or forgets till last minute, CAN NOT move their domain to a more competitive provider. Please give me one good explanation on how this rule protects consumers...

More music is on its way...

2007-11-11T18:51:00.001-08:00

Telene and I have been taking advantage of the crappy weather and doing recording sessions of the tracks for the new album. In addition, this morning the lovely Amber Johnston dropped by our studio and recorded a take or two.. The result - two new awesome songs, with a very different feel. Can't wait to get the album out, but realizing that patience is a virtue.

There are at this moment eight tracks in a near completion stage, for the total of about 45 minutes brand spanking new PolyGroovers.

Sorry, but we don't want to put the tracks up until the whole album is complete, although this one - Spin Cycle for Delicates and Romantics has been available for download for some time now... instrumental only, this is a good massage for the bone behind the ears :)

Oh no, I am Back!

2007-11-10T21:42:00.000-08:00

Yay, blog move is finally complete, and here we are -- with a shiny new dedicated blog domain name, and a Blogger/Google-based account. Blogger has gone a long way since I moved to WordPress from my original blogger account.

My main reason for this renewed love is the flexibility with CSS/HTML template, domain name support and ability to add javascript to your blog is also pretty sweet. In general, it just feels like they trust you more with your own blog here at Blogger. WordPress ends up being ruled by a pretty draconian law, and breeds rather annoyingly hawkish forum users.

Without too much product-bashing, I'm just happy to be done with the move, which unfortunately had to be manual.

Keep an eye out for a set of new posts about drop down menues and REST.

Upgrading to Mac OS-X 10.5 "Leopard"

2007-10-31T16:59:00.000-07:00

Last night I upgraded to Leopard. I was a little frightened, because you know, things don't always go as planned. Especially if you consider that I use my MacBook Pro for music production, in addition to software development in two languages (java and ruby).

So how did it go?

I opted for an "Upgrade" option because the thought of re-installing 200Gb of Native Instruments plug-ins and sample libraries was not very appealing as you might imagine. NI web site suggested not to upgrade until they verify all applications by December, but I decided to bite the bullet and go for it anyway. Someone's gotta do it :)

So far - I must admit, I haven't seen a single hitch. Upgrade took about an hour (even though Installer at first reported it will take 4!!), and things seem to be running smoothly.

Here's a list of applications I have verified to work on my system:

Eclipse IDE, Aptana IDE, IntelliJ IDEA (7.0.1 and 6.0.5), TextMate
Fetch, PostgreSQL, Resin 3
ruby 1.8.6 (my previous installation in /usr/local/ruby-1.8.6), ruby 1.8.6 (installed with Leopard in /usr/bin/ruby)
Photoshop CS, Acrobat Reader, Omni Graffle, MS Office 2004
Audium, Twitterific, iChat
Firefox, Safari (obviously), Opera
QuickTime, iPhoto, iPhotoLibrary, AddresBook, Toast 8
Logic Pro 7, Live Lite 6, Reason 3.0.5
Native Instruments Komplete 4 (Kontakt, Reaktor, Absynth, FM8)

My hardware:

MacBook Pro (Intel) 2.4Ghz with 2Gb of RAM

I'll add more to the list as I am going through and discovering things, but so far I am pretty pleased with how smooth this upgrade was.

What was your experience - especially if you did an upgrade?

Update, November 10th 2007

I've been using Leopard since the upgrade, everything is running smoothly, even my Native Instruments and Waves plugins. All of my Logic projects opened up as they did before. Yay!

I love the Stacks feature, which actually is a really nice way to unclutter your Desktop. I've been long waiting for a clean way to access folders efficiently from the Dock, and this totally hits the bill. I drag folders containing other folders, and it turns into a nice little menu.

Spaces seems cool, but I never remember to use it. Maybe once I get used to it...

Ruby on Rails Hosting: From HostingRails to RailsMachine in a shake of a tail

2007-10-16T16:57:00.000-07:00

Rails hosting is a hot subject, and with everyone asking everyone else about their experience, I thought its only fair to share my own experience, even though it is relatively limited.

Introduction

I started with a shared "professional" hosting plan from HostingRails. It was around $30/month, and included non-root SSH access to a shared server, and additional 150Mb of RAM for total of 200Mb (although that's actually quite misleading, see below). I stayed on this plan for about 3-4 months, and then decided to switch to a virtual hosting plan from RailsMachine - their single server plan, for about $100/month (that includes dedicated 384Mb of RAM and a root access to my virtual server). This post describes the reasons behind switching, and compares pros and cons of each hosting plan. Our needs included the following setup:

One application running in two instances (one production, and one test)
Two-mongrel instances per application (so total of 4 mongrels)
Capistrano based deployment / upgrades / restarts of the servers
Apache 2+
PostgreSQL 8.2
Trac (+ MySQL) served over https
Subversion repository served over https
Subversion post-commit hook linked to a build environment, for continuous integration
Automated remote daily backup of the entire environment

To be fair, the plans described here compare sort of like apples and oranges, so the review has to be taken with a grain of salt. Main differences are, of course, in the fact that one is a shared plan and the other one virtual hosting. People on smaller budget and without tight performance requirements will probably prefer the shared plan, while those with a bit more room in the budget and some UNIX know-how will most likely be much happier with virtual hosting. My hope is that perhaps this review will help someone make the right call for their environment based on their needs, budget and experience.

Detailed Comparison

1. HostingRails

Pros
- Cheap, only $30/mo for 4 mongrel instances (200Mb)
- Easy Trac/SVN install
- Rudimentary backup can be run from web UI
- Friendly support staff
- Timely support, usually within an hour or two
- Lots of info on forums
Cons
- [Update: please see comments section for HostingRails response]. Slow, slow, and slow. We were sharing a server with 4Gb of RAM with another 50+ mongrel instances run by as many users. The server used over 1Gb of swap space and often showed load averages over 30. We were promised numerous upgrades, which did not materialize while we were using the service. Our sites and Trac at times took over 30 seconds to load the first page, probably while the corresponding process was getting un-swapped.
- Have to email support for any changes to top-level apache configuration, such as adding domain aliases. This applies to setting up Trac, svn and mongrel apps.
- Provided backup facility does not backup PostgreSQL/mysql DB, had to write our own script (but forums had a good starting point)
- Our mongrels clearly were not allocated any amount of "reserved" RAM. Looks like buying extra 150Mb in reality means you can run three more mongrel instances, but has nothing to do with how much RAM you actually get, as was witnessed by swap usage stats.
- As we decided to move because of performance issues, HostingRails refused to refund us the unused portion of the yearly fee we paid ahead as we were past first 30 days.
- Locked in their choice of Ruby version, gems, PostgreSQL, MySQL, Trac - can not upgrade these individually, as each is shared among many clients.

Overall, I think that HostingRails has to fix performance issues and not pack as many clients on a single server if they want to be a viable solution for any business. Until they fix performance issues I would NOT recommend this service to anyone. If the performance is fixed, I would recommend this plan to less experienced engineers, who may not know how to use root on a UNIX system, or setup Apache, but have some basic understanding of Rails deployment. I would also recommend this plan on those with a really tight budget.

RailsMachine

RailsMachine provides virtual hosting, meaning you get your own dedicated server with root access. The server, OS and pre-installed software details can be found on RailsMachine website. Because you are sort of expected to use root access, some basic UNIX system administration skills are necessary to take advantage of this plan. However, the flexibility and power that come with this solution by far outweigh the additional $60/month we are paying for our own box. Note that we could pay only $75/month for 256Mb of RAM, but considering the list of applications we wanted to run, I chose to upgrade our plan to at least 384Mb.

Pros
- Much, much, much, faster. When using command line, sometimes you see server "hanging" momentarily while presumably other virtual machines are being processed, but the overall performance improvements are quite dramatic. Pages (trac/mongrel) load within 1-2 seconds consistently.
- Fantastic provided Apache configuration, comes with plenty of dynamically compiled modules, template config files and modular structure. Our single apache server is providing umbrella access to both mongrel apps, SVN over https and Trac over https (using mod_wsgi, not CGI).
- RailsMachine gem allows seamless initial capistrano deployment and subsequent installations
- Apparently each virtual server is backed up automatically (but we do our remote backups anyway)
- CentOS Linux provides convenient yum utility for installing pre-built packages. Sometimes it takes a few minutes to find the right package using "yum search" but once found, installation is a breeze.
- Root access allows installing what you need for your environment, eg - latest PostgreSQL, latest Trac, etc. Was able to install mod_wsgi module to enable much faster Trac access than using traditional CGI access.
- Our entire setup described above takes up 180Mb of RAM, with another 180Mb left for file system caching. Pre-installed MySQL and Apache are built with multi-threading enabled, so the processes manage their shared RAM very effectively. This means we have room to add another 2-3 mongrel instances to the setup without affecting pricing of our plan.
- Responsive, knowledgeable support, although just the nature of the setup means you have to ask a lot LESS questions if you know what you are doing, since you can get most of the things installed yourself without bugging them.
Cons
- [Update: this has been fixed, see comments section] The server came installed with broken permissions on /etc/resolv.conf and /etc/hosts file. This broke host/domain name resolution from the host itself, until the permissions were set to allow read access to the above files.
- Slightly more expensive.
- Occasionally (but mostly rarely), shell access has a latency that appears to be CPU bound by the physical server and not the network.

Overall, I would absolutely recommend RailsMachine to a technically savvy engineers, who want freedom of their own box at a very affordable price. Ability to grow the size of the virtual box dynamically is also great - it's very nice to know that when you need extra 1Gb of RAM, it's there waiting for you (and your wallet).

Install Notes

A quick note about SVN/Apache configuration with authorization turned on. This has been mentioned on many SVN forums, but if you are setting it up, please be aware that mod_dav_svn contains memory leak that makes apache seg-fault during checkouts of a large repository tree (we are using authorization as well as authentication). I had to add the following clause to apache config for SVN directory to resolve to issue: SVNPathAuthz off This is what our SVN configuraiton looks like in case someone is looking for a complete config:

<VirtualHost *:443>
 ServerName svn.YOURDOMAIN.com

 SSLEngine On
 SSLCertificateFile /etc/httpd/sslcert/server.crt
 SSLCertificateKeyFile /etc/httpd/sslcert/server.key

 <Location />
       DAV svn
       SVNPath /var/lib/subversion
       AuthzSVNAccessFile /var/lib/subversion/conf/authz
       SVNPathAuthz off

       AuthType Basic
       AuthName "Authorization Realm"
       AuthUserFile /var/lib/subversion/conf/passwd
       Require valid-user
 </Location>
</VirtualHost>

References

Memory Leak in mod_dav_svn

Closing Notes

Here are the responses I got from HostingRails and RailsMachine - thanks guys...

Hi Konstantin, The issue with broken permissions on hosts/resolv/others has been fixed. It was due to a small (and short-lived) bug in our provisioning system. Thanks for mentioning us! -Rob at Rails Machine

Hi Konstantin, there’s a couple things I’d like to point out that I hope you’d be willing to correct in your review. I understand that you had a bad experience with us and respect your right to voice it.

Running a cluster of 4 Mongrels will only cost $22/mo on our professional plan, and $26, $32/mo on our business and platinum plans, respectively.
On our site we outline that we’re selling (physical + virtual) memory in blocks of 50MB on shared servers — when Mongrels or Static FastCGI instances are fired up they consume (’reserve’) these resources persistently.
Overall, and I suppose you’d have to take my word for it, your server ran into an unfortunate resource situation with a number of users who are now either suspended due to malicious behavior or upgraded to dedicated environments due to extraordinary growth of their apps. Most of our clients are having good hosting experiences with us and we’re working hard to maintain fast and reliable shared servers for Rails deployment at the lowest possible price. All the best, ~William at HostingRails

-- William, thank you for your corrections and I appreciate that it must not be easy to maintain quality service at such a competitive price. I’ve updated the pricing I mention and added a reference to your comment. I am willing to believe that our server may have been exception rather than a rule, and I hope that in the future you have a way to move clients off loaded machines quickly.

On Ruby on Rails with PostgreSQL, and Acts as Paranoid

2007-09-25T16:56:00.000-07:00

Back a few years ago I was researching differences between PostgreSQL and MySQL databases, and chose PostgreSQL because at the time it supported foreign key constraints and many other fantastic SQL extensions that make developer's life a lot easier. Today I am sure MySQL is just as functional as PostgreSQL, and it does appear to be a more popular choice as the Rails DB than MySQL. I still prefer PostgreSQL, it just feels more natural to me coming out of Oracle background, so I am probably biased (who isn't?) Anyway, in the last year and a half or so, I've been writing Rails apps that use PG-based databases and found a few random tricks I'd love to share here. Opinions differ here, but just like accountants like the ancient double entry accounting system, I personally prefer a double validation system - where Rails validates my objects before/after updates, but the database double checks this using proper constraints. Rail's validation system is very robust and extensive, and it is its power. However nobody is protected from human error, and it's pretty easy to forget certain validations, some validations may be performed conditionally, and in general rails validations are at a higher level. I've certainly missed a validation or two in the past myself, or had a bug in my condition. So as a rule, I specifically like to use in all of my migrations:

Foreign key constraints, with "ON DELETE CASCADE" to allow tests to drop/remove fixture data
Check constraints to verify values satisfy certain conditions. Use these to validate finite value columns, such as state/status, gender, inheritance discriminator column, or polymorphic table's type discriminator. Use it to validate pattern based columns, eg if you decide to store SSN as a string of the form XXX-XX-XXXX, a check constraint can ensure it really is.

Foreign Key Constraints

Let's start with foreign key constraints. They are extremely useful in ensuring your relationships work and do not point to non-existing rows. It is even more important if you bypass ActiveRecord and use direct SQL in certain cases to insert data (as this may be the case with some batched or high-volume data loads). I use foreign keys on ALL of my tables, and haven't had much trouble with that. The only issue to watch out for, is the order in which your fixtures are loaded. Basically you'd want to load your independent tables first, such as countries/states - those that do not depend on anything, and then load fixtures that depend on them in the order of dependence.

Please see the following post for an opposite opinion, and note that I don't agree with author's suggestion not to use foreign keys - read first comment, I too was saved on multiple occasions by having them, especially in the development (since that's where you'll be finding most bugs in the first place).

Here's how to add a foreign key constraint in your migration:

create_table :profiles do |t|
   t.column :user_id,    :integer, :null => false
end
execute "ALTER TABLE profiles 
          ADD CONSTRAINT profiles_fk_user_id
          FOREIGN KEY (user_id) 
          REFERENCES users (id) 
          ON DELETE CASCADE"

Notice how all constraints are named. This is very important, because you want to be able to change/update constraints in the future (especially the check constraints - see below). Not giving a constraint a name explicitly forces DB to come up with an auto-generated name, which is not very useful. Any future migration that needs to change this constraint would be at loss as to how to reference it reliably. Using a proper naming convention is also a good idea: table_fk_field is common naming practice for foreign key constraints.

Check Constraints

Check constraints verify that values in a column satisfy provided conditions. They are extremely useful in ensuring you don't get "^%$$$" as your gender value for user Bob, or "Frak!" as a status for your order. Database can be manipulated in many ways, and in my experience unless you protect your columns some weird stuff always ends up in there, whatever the greatest validation framework sits in front of it. In this hypothetical example, we need to ensure that the gender column on our frogs table only allows "M", "F" or "T" as possible gender values (I live in San Francisco, yo!). So in the migration file, specify check constraint similar to how we did this for foreign key constraints:

create_table :frogs do |t|
   t.column :name, :string
   t.column :gender, :string,   :limit => 1, :null => false
end

execute "ALTER TABLE frogs 
          ADD CONSTRAINT frogs_check_gender 
          CHECK (gender IN ('M', 'F', 'T'))"

Please see the following link for more information on PostgreSQL powerful constraints syntax: PostgreSQL Constraints

Partial Indexes, and Acts as Paranoid

If you had to install acts_as_paranoid plugin, you'd notice that it adds deleted_at column which is going to be NULL for all values that are active in your database. Calling destroy will not physically delete the record, but would merely set deleted_at to the current time stamp. By declaring a model as "acts_as_paranoid" it will also add the following to all queries: and (deleted_at is NULL or deleted_at > ?) First of all, the "deleted_at > ?" clause is meant for allowing models to expire in the future, but I haven't met a single person who uses this feature. So the first thing I do when I install this plugin is I modify it's source to remove this comparison. I want my find statements generate the following: and (deleted_at is NULL) Ok, so things are great now, but suddenly I realize that my favorite index on users table and email field, does not really work as well! But of course... I am now querying on both "email = ? and deleted_at is null". So how do you add deleted_at to the index? Here's how:

execute "CREATE INDEX users_idx_email 
            ON users (email, deleted_at) 
            WHERE deleted_at IS NULL"

This uses a cool feature of PostgreSQL called partial index to create an index on subset of values.

Comments and thoughts/suggestions are always welcome! References:

Ruby on Rails: Reducing clutter in actions by placing common code in filters

2007-09-07T16:54:00.000-07:00

This is a tiny but useful tip, that saved many lines of repeated code in my controllers, hence why not share it :)

Do Not Repeat Thyself?

If you've looked at the controller code that's generated by scaffolding, you'll find something like this:

 
  def show
    @user = User.find(params[:id])
  end

  def new
    @user = User.new
  end

  def destroy 
    User.find(params[:id]).destroy
  end

Sure, in this case all we are doing repeatedly is instantiating the user by a potentially available parameter value. What if instead we placed these common fetches in a controller filter, which would simply set an instance variable for us? Hell, we could even handle exceptions (such as invalid ID) in only one place this way! What not to like.

When dealing with a more complicated route that has been defined, this instantiation may become quite a bit more elaborate, and the case for a filter is even more justified.

Consider the case of building a collaboration system where you have projects and individual contributions under that project, as well as a producer of the project. We might want to support all project operations under a URL that looks kind of like this:

/user/kigster/projects/MyVacation/contribution/view/34 with a corresponding route in the config/routes.rb file:

# project management route
map.connect '/user/:username/project/:project_name/contribution/:action/:id',
    :controller     => "contribution"

Based on the route defined, Rails would create params[:username], params[:project_name] and params[:id] for contribution id, in addition to the standard action/controller pair.

Now imagine that the controller we are writing has many actions, such as add, edit, view, list, append, preview, post, comment, etc. All of them could use a handle on the project, it's producer and contribution instances. Ideally - in @project, @producer_user and @contribution respectively.

If we followed the scaffolding example, we'd simply add appropriate lookups at the beginning of each action. But that's a lot of repeated code!

So let's use filters instead, to get our common lookups under control.

class ContributeController < ApplicationController
  before_filter :setup_project

  def home
     # do stuff
  end

  private

  def setup_project
    @producer_user = 
      User.find_by_username(params[:username])
    @project = 
      @producer_user.produced_projects.detect do |p|
      p.name.downcase.eql?(params[:project_name].downcase)
    end
    @contribution = 
        @project.contributions.find_by_id(params[:id])

    return true

  rescue Exception => e
    logger.info "can't show project:", e
    flash[:error] = e.message if e
    redirect_to :controller => "home" and return
  end
end

This is short and elegant, and now every action in this controller (which is not excluded for setup_project filter) will have access to our instance variables. Groovy!

Issues

Of course nothing is free, and in this case we are potentially loosing performance. Perhaps some actions don't need to know the @producer_user. We'll be fetching it all the time, which saves time for us - developers. If performance problems occur because of extra unused fetches the filter can be broken up or optimized. Until then - it sure saves me a headache.

That's it!

Thoughts, comments?

Mac OS-X tips: How to run SSHD on an alternative port

2007-08-27T16:52:00.000-07:00

This tip falls into one of those "I had to spend more than 10 minutes looking for an answer" category, so it's a worthwhile subject for a quick blog post.

Why Run SSH?

Running SSH on Mac OS-X allows you to login to your machine remotely, and also copy file securely via SCP command to and from your Mac OS-X host.

But I am behind my own $50 router/firewall. Can I still connect to my computer from outiside?

Yes. Most off the shelf routers and firewalls will allow you to do two things that are needed for this:

Assign a permanent IP address to your Mac on a local network (see your router documentation for more details)
Create a port forwarding rule on your router. Eg - any request to port 22 on your external IP (provided by your router) can be routed to the specific IP address of your Mac. Exact specifics on this configuration are once again available in your router documentation. Most off the shelf routers support this, including basic Netgear and D-Link.

So let's assume you've set this whole thing up and you are now able to connect from outside to port 22 of your external IP address on your firewall, which is then routed to your Mac port 22. This is great, since now you copy files via SCP from the internet to your computer, connect to it from your work office, etc, etc.

But why would you want to run SSH on alternative port, and not 22?

Simple answer is that port 22 is probably the most obvious port to "probe" from outside. Hackers typically run automated port scans of publicly visible IP addresses, but running it for all 64K ports takes a long time, so typically those scans are just for a small subset of ports. Port 22 is a clear indication of a server system which makes it a likely candidate for an attack. Turn of port 22 and to the outside world your system appears a lot more boring. Boring is good. So I still want to use my SSH connection, but would rather not do it on the default port. More expensive firewalls and routers will allow you to route eg. port 33333 from the outside to port 22 on the inside. But none of the cheap off-the-shelf routers I've seen allow you to do that. They just take input port, and destination IP which means the outside port and inside port must be the same. That's a silly limitation and a small bummer. So to make this work with my cheap a$$ router I have to change the listening port on my Mac. Frankly, this isn't a bad idea either, since whenever you connect your laptop to a public wi-fi network anyone can directly connect to your Mac and probe the open ports. So keeping port 22 open on your Mac is not that great of an idea. So. In order to change the port on Mac OS-X for your SSH daemon, follow the following steps:

Open Terminal and as edit the file /etc/services (as root)
Add a line at the bottom: secret-ssh 43539/tcp # secret SSH port

Edit file /System/Library/LaunchDaemons/ssh.plist and replace the code:

<key>SockServiceName </key>
<string>ssh </string>

with

<key>SockServiceName </key>
<string>secret-ssh </string>

Change your port forwarding configuration on your firewall to route port 43539 instead
Open "Sharing" control panel and ensure that "Remote Login" is checked off (if not uncheck it). Then check it again to start with the new configuration.

Now you should be able to run SSH command to your external IP as follows: ssh -p 43539 ip_address_of_your_server -l username To copy files from remote hosts to your Mac desktop, run the following command:

scp -P 43539 local_file.txt username@ip_address_of_your_server:~/Desktop

Note that "scp" command requires an upper case "P", while regular "ssh" wants a lower case "p" to declare remote server's port. That's it, hope it helps! :)

New laptop and it's a showoff: G5 vs Intel

2007-07-02T16:50:00.000-07:00

As I left my most recent permanent job for a private gig, I had to get myself my own new laptop. Of course the machine had to be a Mac, but which one? A laptop? Desktop? I decided on laptop since it's so much more portable and I do contracting. But then - 17" or 15" and what specs, and should I shell out for a brand new model, or get the older one cheaper? To cut the long story short I went for the top of the line, most recently upgraded model of 17" MacBook Pro, with 2.4Ghz Intel Core Duo processor, 7200RPM 160Gb drive, 2Gb of RAM and a hi-resolution 1920x1200 screen. This machine on Apple store costs $3800 (with tax and apple care), but I managed to save more than a third by buying it from a dude on Craigslist. My first impression - wow! The screen is so incredible, there are so many pixels that a window dragged when onto my older SyncMaster 23" screen looks twice as large! That's because 23" screen has only 1600x1200 resolution, but the laptop packs 1920x1200 in a 17" screen. Even upgrading to 1920 pixel wide screen would still not be adequate since each pixel would be larger than on a macbook. Gotta get used to this zooming effect :) That minor inconvenience aside, I decided to run some basic tests on my older PowerMac Dual 2Hgz G5, with 1.5Gb of RAM and thought it'd be fun to share it here: Test #1: Compile ruby interpreter from scratch:

17" MacBook Pro 2.4Ghz Intel Core Duo:

real    1m3.398s
user    0m45.229s
sys     0m15.373s

PowerMac Dual 2Ghz G5:

real    2m0.422s
user    1m28.266s
sys     0m29.412s

I'll add more tests as I run them, but to see an almost 2x difference between 2 year old PowerMac and a new MacBook is simply amazing. Can't wait to start using the laptop for music! Meanwhile, PowerMac is available for sale for $1000 :) Laterz!

Half moon, Black moon, 14.5%-something moon "Pah Tee"...

2007-04-24T16:46:00.000-07:00

My blog has been silent for a while now for no better reason than me being on a much anticipated and needed vacation. Telene and I managed to sneak out for a whole month of April, and spend a few days in Tokyo doing typical touristy things, before venturing off to our final destination - Thailand.

We haven't really had too much planned out, but one thing we knew is that we wanted rest - beach, sun, water, lots of sleep and massage. As our vacation is nearing the end (and I am writing this from Thailand), I can assure that all of those things we have thus far successfully achieved. And much more.

Most of our trip we ended up spending on the island of Ko Phangan - where the famous monthly full moon party happens on the beach. We are missing the full moon party, but this is not a big deal because every other phase of the moon is celebrated here with just as much fervor and scale, if perhaps by a smaller crowd. Which is actually a good thing, considering that full moon party gets up to 30,000 people (I don't believe these numbers, I'd say it's probably around 10,000 - there's just not a lot of space here for 30,000 people, it's a pretty small island!).

Tonight we are going to check out Half Moon party - it happens in a particular spot in the jungle that's specially outfitted for these parties: there are supposedly two "rooms", cheap massages, multiple bars, even a "chill space" of sorts. There's definitely no shortage of understanding how to professionally throw a party here, but what is lacking is the community feel of a true party scene I am so used to in the Bay Area and Europe. The parties are thrown and organized mostly by local Thais, and attract mostly tourist crowd (although many locals that work at our hotel promise to come too to have fun). The result is that it is a tourist version of rave culture, just as local restaurants serve Italian pizza, they also serve Goa trance - it's mostly the same, but not quite. On the platter, for 300 baht per person, with all amenities and one drink included. But something is also seriously lacking, judging from the three parties we went so far - black moon, the other half moon on Ko Samui, and out-of-moon-phase party in some new location which kind-of tanked, with only 50 people attending at best.

One problem is the attendance - there are just too many "out-of-place" british dudes in shorts that you would never see at any similar event elsewhere. They come to Ko Phangan as the party place to be, which is cheap and you can score (in more ways than one). As the parties happen, they go where the chics go and inevitably end up at the party. You can spot them because they are wearing all white and baseball caps, standing with beer or buckets in the middle of the dance floor and loudly (very loudly) talking over the music about some British soccer crap.

Another problem is the music itself. What could really spice up local party scene is more diversity in genres: we've heard predominantly psy trance here at the parties, some of it pretty good, but also some really outdated. Tonight's party promises house and breaks in the first few hours on the flyer and we are seriously looking forward to that. We've heard some house tunes here in local bars, and with the exception of one particularly good DJ we met from the UK (Dj Silver), locals really don't get house as much as they get psy-trance. It must be the hot climate, and it's indian / goa influences and origin that appeal here, much more than Latin or Caribbean house tunes. It could also be that Thailand happens to be one of the most popular destinations for Israelis, and everyone knows that Israelis pretty much only love goa. But enough on the party subject...

Besides being entertained by really stupid tattoos that belong to British, Scandinavians and Israelis, we indulged in a very comfortable living on a luxury resort with a beach-front pool and a bar, attending to local mini parties in the jungle and on the beach, riding mini dirt bikes around the island and exploring it's attractions (one of my favorites was archery - turns out I can actually shoot with a bow and arrow! My young undying love of Robin Hood has finally grown into something that's not going to pointlessly raise eyebrows in the Castro :). We also did some elephant trekking, snorkeling, socializing, reading and even power yoga (which at air temperatures in mid-30's Celcius is not far from Bikram yoga). If the goal of a good vacation is to slow the time down, then we are definitely on the right track.

There will be plenty more blog entries about stuff we've seen and pictures we took, but for now this will do. Once thing for certain, is that no matter how great the place is we always seem happy to go back home. We just can't help but love San Francisco. And what's there not to love? :)

Adios until the next post, Konstantin from Ko Phangan.

The KORE arrives

2007-02-20T16:45:00.000-08:00

In a quest to bring my soft synths up to date, I took up Native Instruments offer to upgrade my entire NI Complete collection to their latest and greatest, for free - after I buy the KORE - their midi/audio hardware interface and controller. The KORE arrived today, and I played with it a little using my old synths, but I couldn't take full advantage of the KORE without using their preset library (which apparently contains 10,000+ sounds). I couldn't use the presets just yet, because they are all for the upgraded versions of the instruments, which I don't have until the DVDs arrive with KOMPLETE 4 I can't do much with this beautiful piece of gear. Stay tuned for a proper review. Can't wait till it's all here and working.

We been representin'!

2007-01-24T16:43:00.000-08:00

So recently I had a really fun experience working at the Blurb (the company I work for) booth at MacWorld, where the booth was bombarded with non-stop attention. We felt sorry for the booths next to us, some of which had a hard time luring people in, while for us the problem was that of survival - how to you survive five+ hours of talking about your product, with tiny water and bathroom breaks in between, and many eager visitors waiting for your attention. Of course it is awesome to see your company and product be so timely and successful, and so many people excited about the possibilities it offers. As a software engineer you don't often get to talk to the users, witness their pain or glory, you just get to hear about it from a distrgruntled customer service rep who gets all the blame, or a from a marketing person who gets all the glory. So this was certainly refreshing and energizing experience. At some point in the day I was approached by someone with a request to take a quick podcast interview, and as it recently went up live, here is the link where you can listen to yours truly talk about Blurb offering - MacVoices PodCast Interview. If you are interested in self-publishing business, make sure to scan through Blurb's own blog - Blurberati.

Oh Firefox, why aren't thou working?

2007-01-13T16:42:00.000-08:00

Oh Firefox, why aren't thou working?
Once in a while, you stop responding.
My keyboard types, my fingers stomping...
But you're ignoring our bonding.

We've had so many joyful years
Together - sharing laughter, tears
Despise for impotent Explorer
The fear, the weakness and the horror.

And now look at what you've done
You've turned yourself into a pun
A joke on browsers and alike
I can't go on with this last strike.

And so I am leaving for another
Her name's Safari, and she'd rather
Be there, when I need her most.
I am sorry. Our love is lost.

Ruby on Rails: How to load session objects into console

2006-12-06T16:38:00.000-08:00

This took a little bit of poking around, so I decided to post it here.

Basically I was tracking down a bug, which resulted in a corrupt object (cart) in the web session. So I thought, it would be nice to play with this object in the console to see what's up... But how do I get it in there?

There may be a more elegant way to do this but here is how I got it to load:

Look in tmp/sessions and find the most recent file (ls -alrt on UNIX). Let's say the file is called 'tmp/sessions/ruby_sess.8eb9614a7e4e1e3b'
Open console and type:
```
>> session = Marshal.load(File.open('tmp/sessions/ruby_sess.8eb9614a7e4e1e3b'))
=> {"hash"=>{:cart=...
>> cart = session["hash"][:cart]
....
```
In this case I was trying to access a cart object in the session, which was placed in the session with:session[:cart] = Cart.new

That's it!

Wanted: 11.3Tb flash drive to move house to a new location

2006-12-04T16:38:00.000-08:00

Isn't it just great that these days you do not need a truck to move stuff? As long as the stuff you are moving is digital. I was thinking that in a pretty near future each house may include a giant mass storage that holds everything we own which can be digitized. Such are the books, photos, personal documents, bills, music, etc. All of those things are already available in a digital form, but what about furniture, kitchen counters, decorations, paintings, or the super-heavy piano for god's sake? What if... the house of the future is a completely configurable and intelligent hardware device that can reconfigure itself to it's owner's liking and create those items entirely from it's own supply of materials? What if the paintings can be built into walls and show digitized image, books are digital single-page fold-out with downloadable content, furniture can be created from built-in modules... Then - instead of moving your stuff you just have to download your existing house to the new house - and vola! It looks the same! Wouldn't that make moving easier? Well, until the house is so intelligent that it can reconstruct itself to fit my personal furniture and layout needs, the digital move is limited to less extravagant things, such as music, documents, books and media. But that's already lots of really important stuff! Which is why I think that the house of today should come with a built-in storage, that's remotely backed up and fully redundant. A built-in file server, just like a built-in washing machine. That's because just like everyone needs to wash dishes, everyone in this day and age also needs to backup their crap, and there's lots of crap to backup. Go! - the digital movers revolution, just don't forget you can't microwave your hard drive.