Code Elements

SOLR : very impressive lucene based search tool

2009-03-20T17:38:00.000-07:00

Very impressed with Solr and the development community behind it. http://lucene.apache.org/solr/

1.5 days of research, submitted bug and resulting immediate patch and I have a fully funtional search engine for a project at work.

If you've ever thought it would be nice to have your very own google type appliance, I highly recommend you check out solr and the underlying IR engine: lucene.

Passing the Turing Test

2009-03-20T10:23:00.000-07:00

One of the books I am currently reading is "An Introduction to Neural Networks". The introductory chapter covered the Turing Test and the fact that no computer has ever come close to passing it. The context of the chapter was about the limitations of the computer to house and efficiently access a db of human knowledge in order to answer the questions as a human would.

A thought occurred to me. Are humans becoming more computer-like in the way we interact with each other? In this day and age, many of us interact with computers more than we do with humans. I'm doing it write now as I write this. We gain a great deal of our knowledge as a result of direct contact with computer systems and much of our 'human to human' interaction are proxied through computer systems (twitter, IM, email, etc) and we adapt out language on those systems accordingly.

For instance, a human could ask another human:
"Hey Sam, are you attending OSCON this year?"

The equivilent tweet might be:
"@samkeen are you attending #oscon2009"

And some line of computer code might be:
if(samkeen.attending('oscon2009')) {
#take some action;
}

From this you could conclude that the Tweet syntax is bridging the gap between programming code and human dialogue.

If it is true that the way we attain, and share knowledge is becoming more computer-like, will that not someday allow a computer system to pass the Turing Test. Or in that case would it be the human passing the Turing Test?

Trying out bloggers mobile interface

2009-03-18T15:16:00.001-07:00

Adding some message body to see where it ends up.

kindle2 web browser

2009-03-10T14:53:00.000-07:00

Very happy with my Kindle2 so far as a book reader. I enjoy reading tech books and my wife uses it for novels.

I've started playing with the experimental web browser and have found it to be very useful. As expecting it has trouble with graphics but for mobile oriented sites it has proved quite handy, especially since I have access through 3G.

Shown here is m.twitter.com

Winter Coders Social 2008

2008-12-03T11:05:00.000-08:00

This year's Winter Coders Social is set to go. Come join the party this Tuesday, December 9th at Cubespace.

Details: http://calagator.org/events/1250456151

GD imagettftext broken on Leopard/mac ports

2008-11-19T12:21:00.000-08:00

Spent the better part of the morning battling PHP/GD. I needed to overlay text on an image, something I can recall doing in the past without issue. Not this morning though. Everytime the code got to 'imagettftext' it would result in a WSOD. After much troubleshooting and Google'ng I found this mention in the PHP bug db which then pointed to the root of the cause: The macports 'freetype' lib. The solution for fixing that is here.

hope this helps others

Augmenting Web Services with SMS and XMPP

2008-11-13T18:58:00.000-08:00

This talk was centered around adding channels of communication such as SMS and XMPP to existing HTTP web services.
A project called Extapi, is a small web app that manages this coordination between multiple channels and services.
here are the slides for my presentation given on November 14th 2008 at PHP|Works

AT PHP Works

2008-11-13T06:14:00.000-08:00

In Atlanta for the php|works conference.
Listening to the Keynote from Kevin Dangoor (founder of turbo gears). He's discussing successfully fostering open source projects.
Looking forward to meeting new folks with PHP and/or Python backgrounds

Speaking at this years PHPWorks in Atlanta

2008-09-05T15:03:00.000-07:00

Very excited that the talk I submitted to php|works was accepted. I'll be speaking at the Atlanta conference on my continuing work with combining XMPP and SMS with existing web API's

Shizzow's public announcment

2008-08-13T11:08:00.000-07:00

One of my side projects has turned into a full blown startup. Shizzow.com will be officially announced at today's Lunch 2.0 at cube space. It's been a pleasure working with the other fine folks on this project and I'm looking forward to the future.
Thanks to Silicon Florist for all the press

Accessing Trimet Arrival Times with SMS

2008-07-22T17:11:00.000-07:00

I discovered the Portland Trimet API about a month ago and was very inspired by the amount of real-time information available and the ease of accessing it through their web service. One feature of the service is that for a given stop id you can get the arrival times for the buses/trains for that stop. These are not the scheduled times but rather the real-time estimated arrival time based on the known GPS coordinates of the actual vehicles on their routes.
You can get to this data through IVR system by dialing 503-238-RIDE or on the web.

In the mean time, I have also been working with extending existing APIs with additional access such as SMS and XMPP (jabber). So it was natural to take on this trimet service as my first proof of concept.
For SMS access, you simple text "pdxt {stop_id}" to the short code DOTORG (368674)

so for example, for the stop 8334 (Pioneer Square South MAX Station Eastbound), you would text...

"pdxt 8334" to DOTORG (368674)

The result is shown below (with the same request through the web for comparison)

There is slight differences in the times since it is difficult to coordinate the SMS and web request precisely.

In my trials, response times for SMS were in the 5-10 second range which for me is acceptable.

I'll next be working on an XMPP (Jabber) wrapper around the service so you will then be able to access the service with an IM client (which many phones now have). XMPP will give the same mobility benefits of SMS but removes the 155 char limit for the returned text allowing for more information to be included in the returned message.

Once I have completed SMS and XMPP around this service, I plan to refactor the technique so it can be easily applied to other services. I will also document the steps I took to integrate with SMS and XMPP. Look for that on google code soon.

Just Testing the Waters

2008-06-24T11:25:00.000-07:00

I am considering moving my blog here (from the wordpress I host currently)

I like blogger since I will no longer have to do blog software updates, something I never seem to get time for.

I discovered SyntaxHighlighter, so now that requirement is taken care of.

Startupalooza

2008-04-18T09:50:00.000-07:00

Exciting event coming up at the end of March. Startupalooza is gaining steam, building quite the list of product demo’s (including Fyreball, my current employer). The event designed to be Those who have walked the path, sharing their knowledge and experiences with those interested in beginning startups. This is not a typical startup event where the goal is to impress VC’s in the crowd (though I’m sure a few will be lurking).

Event is: Saturday, March 29, 2008, 2:00 PM - 7:00 PM

RSVP at Upcoming

Ignite Portland 2

2008-03-04T15:31:00.000-08:00

Looks much bigger (550+ registered) and better this go around. I went to Ignite 1 and had an excellent time. Can’t wait to see how it goes this time.

Happening Feb 5, 2008 : doors open at 5:15pm at the Bagdad.

Ignite Portland home page

“Portland On Fire” Profiling Local Individuals

2008-01-10T11:02:00.000-08:00

Portland On Fire is a site launched by Raven Zachary on the first day of 2008. It’s goal is to give a quick bio of local folks, showing what they’ve done and what they are currently up to. It is starting with people in the tech and arts sector but the hope is that it expands out to all corners of the Portland talent pool.

Book Review: PHP In Action

2007-12-07T16:37:00.000-08:00

Authors: Dagfinn Reiersol, Marcus Baker, Chris Shiflett
2007 Manning Publications
525 pages

This book is geared toward the intermediate PHP developer who wants to bring in aspects of OOP, Testing and Refactoring to help improve the quality of the code they write. It is split into four parts; Basic Tools and Concepts, Testing and Refactoring, Building the Web Interface, and Databases and Infrastructure.

In addition to PHP, I have decent amount of experience with Java and Java web frameworks such as Struts. So as I worked through this book much of the content was familiar to me but from a Java perspective. It was enlightening to see the authors express these same concepts from a PHP perspective. The fact that many times (not always), the implementation in PHP is more concise and elegant that the Java alternative really shows of the power of a dynamically typed language such as PHP. Also the fact that PHP was bred from the beginning to be a web development language gives it a definate advantage in the web arena.
The authors are honest though, they haven’t simply painted implementing OO, TDD, and Refactoring as completelty painless. For instance in the testing portion they’ve devoted quite a bit of time to showing the difficulties of testing (especially in a Web environment). Such as the need for mock objects and the difficulty in keeping mocks “real enough” so they fail and pass as the real object would. This full disclosure is key for readers to estimate if the extra effort of a concept is worth the benefits for their particual situation.

Overall this is great book for the intended audience. It is not “black and white” about the solutions it proposes. Reasonable alternatives are given and the pros and cons of each are expressed. For those with extensive OO experience, some portions of the book may seem trivial but overall it is still worth a “quick scan” to see the specifics of PHP implementations of general OO concepts.

Book Review: Practical Rails Social Networking Sites

2007-08-01T20:50:00.000-07:00

Title: Practical Rails Social Networking Sites
Author: Alan Bradburne
ISBN: 1590598415
ISBN-13: 9781590598412
421 pages
Published: Jun 2007

So far my web development background has mainly consisted of Java (J2EE) and PHP. I just started playing with Ruby and now Ruby on Rails. I’ve worked through a few of the tutorials linked from www.rubyonrails.org. So essentially I had two or three ‘proof of concept’ sites with models consisting of two to three tables each. So of course at that point I wanted to ramp it up and build a ‘real’ site but that leap from demo site to a production ready, real world site can be a tough one to make. “Practical Rails Social Networking Sites” can certainly help you in that undertaking.

During the span of the book, Alan leads you through building an actual web application with many of the Web 2.0 features you would expect. When you are through with this book, you will have a simple “content management system”, user authentication with groups, RSS feeds, blogging with an API and user created themes, forums, photo gallery with tagging, email and mailing list, XFN support, Google maps and Flickr API integration, and views for mobile devices. So, as you can see, you learn how to build just about everything a social site needs. The author does a good job of using leading edge features of Rails and web development in general to build the components for the site. One that I was very happy to see was REST or “Representational State Transfer”. Rails Restful Routes are used throughout the book and you soon learn to appreciate them as they further simplify the routing of requests from the view, through the controller, and to the model.

For the most part the end of each chapter has a substantial section devoted to testing what was implemented previously in the chapter. I was good to see this emphasis on unit and functional testing as I know what value it brings to the scalability and maintainability of an application. Plus it is so easy to write tests in Rails, there is little excuse not to do so.

After you’ve worked through the examples in the book and have a substantial, working social web app, the final chapter will lead you through the process of deploying your work to a production environment. This chapter is compact but certainly gets you enough info and pointers to get you deployed and into a strategy for scaling and optimizing.

Overall I really liked this book. It gets right down to business and utilizes many of the leading edge Rails best practices to get an impressive amount of work done in very short order, but this is Rails after all

OSCON 2007 Recap

2007-07-27T14:44:00.000-07:00

Just finished up my 2007 ‘OSCON Experience‘ I attended the conference on Tuesday and Wednesday and we had our PDXPHP both in the exhibit hall as last year, but in addition I helped to organize the OSCamp room.

Spent most of my time in the exhibit hall this year. I only made it to one OSCON session + 2-3 OSCamp sessions. Still had a good time though. I was able to catch up with folks I typically only see at OSCON in addition to hanging out with local friends. There were no shortage of after parties this year. It seemed you had your pick of 3-4 parties a night. People were really zombiefied by the end of Thursday (I know I was).

OSL was nice enough to give a demo of the OLPC to the OSCAMP crowd. Getting the device (and Brad from OSL) out of the loud, crowded exhibit hall enabled the OSCamp’s to get a real close look at it and ask all the questions they wanted.

360Flex Comming to Seattle

2007-07-20T10:18:00.000-07:00

Flex seems to be on fire as of late. I really like the theme behind the efforts over at 360Conferences.

For more info on the event, visit www.360flex.com

“The gist is, we’re bringing together the best Flex has to offer, in a community setting where developers can hang out, share ideas, share problems, in general come together and leave with more knowledge and more contacts and friends than when they arrived. Our last event in San Jose was a phenomenal success, this one is shaping up to be better than that.

Additionally we keep the price low, at 360.00 for 3 days, so that even those developers and engineers without an expense account can attend.“

OSCamp 2007

2007-07-20T08:03:00.000-07:00

I sort of just fell into organizing the camp this year but it is really starting to shape up. Didn’t take too much effort and O’Reilly was extremely easy to work with. They were nice enough to kick in for grub, projector rental, and brainstorm / collaboration type office supplies (and those are not cheap when you have to purchase them from the Convention Center $$$)

So what is OSCamp:

OSCAMP is in the spirit of BarCamp and other Unconferences. This clip from BarCamp SF provides a great video summary of the spirit of a BarCamp. This year it will be held on July 25-26 as part of O’Reilly’s OSCon2007.

Be sure to head over to oscamp.org and register (if you have not already registered for OSCON). OSCamp is free and open to the public (After you register

Explanation of Open Source (for the non-geek)

2007-07-05T23:18:00.000-07:00

I gave this presentation to a local Portland Tech Requiting firm. They see more and more demand for applicants with skills in various open source technologies so they wanted a better understanding of its origins and current trends with an emphasis on PHP.

Slides (pdf)

This work is licensed under a
Creative Commons Attribution-Share Alike 3.0 United States License.

Building Administration Interfaces with XUL

2007-07-03T08:54:00.000-07:00

I’m starting to work with mozilla and XUL again. I’ll be posting that work soon, But in the meantime I found this presentation I did back in 2005. [original post] I remember that is when I first stumbled across this strange but useful XmlHTTPRequest object and wondered if it’s use would find its way to the mainstream.

Took a traditional HTML admin interface for a PHP web application and transformed it into a Mozilla XUL interface. Goal is to demonstrate the benefits of using XUL to create Rich Client Application interfaces into your web apps when you have the ability to mandate the browser used (such as an admin interface).

Links and resources:

Working Demo (mozilla browsers only)
Documentation and Source Code (didn’t go too crazy on the docs, but you can see all the source code by viewing the files section of the tree on the left).

Microformats: Path to Sematic Web?

2007-06-07T23:12:00.000-07:00

In the past I have looked into technologies promising to lead us to the promised land of Semantic Web. RDF and OWL are a couple of examples. Though they are powerful and capable formats they are a bit of a chore to come up to speed on and not to easy to implement into something usable and scalable. You see folks using them to solve specific problems in niche and primarily vertical scopes but you don’t seem to see signs of any widespread adoption by the masses.

Then comes along Microformates. From what I’ve read, XFN is what evolved into the first microformat. Today there are many more useful formats. MF’s are sort of a grassroots effort to stitch semantics into the content being published to the web by those who do much of the publishing. Some of the microfomat goals are to solve specific real world problems, start as simple as possible, and reuse existing, widely adopted standards. These concepts make microformats very easy to pick up and start utilizing immediately.

Links I found useful

Tantek Çelik Hcard creator
Firefox Operator add-on (leader of the pack for the FF microformat add-ons)
A PHP microformat lib
A Ruby microformat lib
Get your mf data out to the world
some sites utilizing mf’s (hint: install Operator first)
- cork’d
- Upcoming
- Eventful
- Tech.Yahoo

When Adventurous Coders get Bored

2007-05-19T14:09:00.000-07:00

I convinced a friend to try this, hopefully space/time fabric was unaffected.

One more rule for the universe

…

You may not travel faster than the speed of light
You may not power on a virtual machine in a virtual machine

Amazon Simple Storage Service [S3]

2007-05-06T15:09:00.000-07:00

What is S3??

Simple Storage Service: Is a web service that allows any developer to gain access to highly scalable, very reliable, inexpensive storage space. Your data is replicated to multiple servers at multiple data centers.

How to get started

Go to Amazon’s AWS page, then to the S3 page and sign up (check out the other services while you are there)

Pricing (From Amazon’s site…)

Pricing

…

New Pricing (effective June 1st, 2007)

Storage
$0.15 per GB-Month of storage used

Data Transfer
$0.10 per GB - all data uploaded

$0.18 per GB - first 10 TB / month data downloaded
$0.16 per GB - next 40 TB / month data downloaded
$0.13 per GB - data downloaded / month over 50 TB

Data transferred between Amazon S3 and Amazon EC2 is free of charge

Requests
$0.01 per 1,000 PUT or LIST requests
$0.01 per 10,000 GET and all other requests*
* No charge for delete requests

Storage and bandwidth size includes all file overhead

I looked around the web for similar services (hard to find someone that posts prices), and for 180 gig’s of reduntantly stored data, it was in the $200/month price range.

The same 180Gb on S3 would be

$27 to store for 30 days

$18 to xmit (the entire 180Gb) to S3

In addition to price, with S3, you are in full control of how and when you put and/or get your data.

Only pay for what you actually use

One of the niceties about S3 (and the other Amz web services) is that you pay just for what you use.

Didn’t use that service last month: Pay $0.

This allows you to ‘tinker’ all you want for mere pennies

Amazon S3 - Objects

What is an Object?

Object is the term we use in S3 for the ‘thing’ (file/data) you want to store.
Once an object is stored in S3, it contains the original data (contents of the file), plus a given amount of meta-data (name/value pairs).

You can add your own metadata but some of the standards are ‘Last-Modified‘ and ‘Content-Type‘

A given Object can be from 1byte to 5GBs

Amazon S3 - Buckets

Why Buckets?

Buckets provides a unique namespace for management of objects contained in the bucket

Bucket namespaces are Global across all of S3 (all users of S3. Similar concept as ‘domain names‘)

An S3 account is allowed 100 buckets

Amazon S3 - Keys

Key

A key is the unique identifier for an object within a bucket

Locating an object

Any Object can be located by its [bucket + key] using a RESTful formatted URL

   http://s3.amazonaws.com/foo-products/2006/may/1845.prd

foo-products is the bucket & 2006/may/1845.prd is the Key

S3 - Authentication

Most requests to S3 require authentication, this ensures that you don’t get charged for operations you didn’t authorize, and that nobody else sees your private data.

You can grant various access models (acl) for an Object or an entire Bucket

private
public-read
public-read-write
authenticated-read

To set the ACL, when you PUT the Object to S3, you set a x-amz-acl header. For example…

x-amz-acl: public-read

ACl defaults to private if not set on the PUT

S3: Putting it all Together

How do you speak to S3?

At this point, all interaction is done with the HTTP protocol (the current exception is that you can retrieve objects using http or BitTorrent).

So, creating a program to interact with S3 is just a matter of creating HTTP requests and reading HTTP responses. Something PHP is quite capable of (especially with a little help from PEAR HTTP_Request and Crypt_HMAC)

First get an account and your Keys

Access Key ID: You add this to any requests to S3. Essentially it is your unique identifier that tells S3 a given request is targeted for your account.
Secret Access Key: For requests to S3 for Objects with acl’s that require authentication (i.e. private, authenticated-read), you ’sign’ your request with this secret key.

The following code examples are based on what is in the Amazon S3 developer docs

Create a Bucket

# create bucket request

PUT /[bucket-name] HTTP/1.0
Date: Wed, 08 May 2007 08:45:09 GMT
Authorization: AWS [aws-access-key-id]:[header-signature]
Host: s3.amazonaws.com

# create bucket response

HTTP/1.1 200 OK
x-amz-id-2: VjzdTviQorQtSjcgLshzCZSzN+7CnewvHA+6sNxR3VRcUPyO5fmSmo8bWnIS52qa
x-amz-request-id: 91A8CC60F9FC49E7
Date: Wed, 08 Mar 2006 04:06:15 GMT
Location: /[bucket-name]
Content-Length: 0
Connection: keep-alive
Server: AmazonS3

Put Objects in your Bucket

# put object request

PUT /[bucket-name]/[key-name] HTTP/1.0
Date: Wed, 08 Mar 2006 04:06:16 GMT
Authorization: AWS [aws-access-key-id]:[header-signature]
Host: s3.amazonaws.com
Content-Length: 14
x-amz-meta-title: my title
Content-Type: text/plain

this is a test

# put object response

HTTP/1.1 200 OK
x-amz-id-2: wc15E1LUrjDZhNtT4QZtsbtadnOMKGjw5QTxkRDVO1owwbA6YoiqJJEuKShopufw
x-amz-request-id: 7487CD42C5CA7524
Date: Wed, 08 Mar 2006 04:06:16 GMT
ETag: "54b0c58c7ce9f2a8b551351102ee0938"
Content-Length: 0
Connection: keep-aliveServer: AmazonS3

Retrieve Objects from your bucket

# get object request

GET /[bucket-name]/[key-name] HTTP/1.0
Date: Wed, 08 Mar 2006 04:06:18 GMT
Authorization: AWS [aws-access-key-id]:[header-signature]
Host: s3.amazonaws.com

# get object response

HTTP/1.1 200 OK
x-amz-id-2: FbGpiykb9oJEdJd0bcfwkL6S3lc06X0y7XSeA/GWyRdvlNEZ0irthljxKoeGFfB6
x-amz-request-id: 9298531013923634
Date: Wed, 08 Mar 2006 04:06:18 GMT
Last-Modified: Wed, 08 Mar 2006 04:06:16 GMT
ETag: "54b0c58c7ce9f2a8b551351102ee0938"
x-amz-meta-title: my title
Content-Type: text/plain
Content-Length: 14
Connection: keep-alive
Server: AmazonS3

this is a test

S3 - The PHP way

Implementing an API to S3 with PHP

Prerequisites

You’ll need the PEAR libraries Crypt_HMAC & HTTP_Request (at least things are much easier if you have these)

# sudo pear install Crypt_HMAC
pear.php.net" to update
downloading Crypt_HMAC-1.0.1.tgz ...
Starting to download Crypt_HMAC-1.0.1.tgz (2,149 bytes)
....done: 2,149 bytes
install ok: channel://pear.php.net/Crypt_HMAC-1.0.1

sam$ sudo pear install HTTP_Request
pear.php.net" to update
downloading HTTP_Request-1.4.0.tgz ...
Starting to download HTTP_Request-1.4.0.tgz (15,262 bytes)
.....done: 15,262 bytes
downloading Net_URL-1.0.14.tgz ...
Starting to download Net_URL-1.0.14.tgz (5,173 bytes)
...done: 5,173 bytes
downloading Net_Socket-1.0.7.tgz ...
Starting to download Net_Socket-1.0.7.tgz (5,419 bytes)
...done: 5,419 bytes
install ok: channel://pear.php.net/Net_URL-1.0.14
install ok: channel://pear.php.net/Net_Socket-1.0.7
install ok: channel://pear.php.net/HTTP_Request-1.4.0

Creating the API

At this point all you really need to do is create a function for each needed interaction with S3 (or better yet, a PHP Object with a method for each). So something like…

createBucket()

putObject()

getObject()

getBucketListing()

...

These functions are going to be creating http requests and reading http responses. Sometimes this can be a bit tricky (one missing ‘\n’ and you’re screwed), so leverage what what other have done befor you. The Amazon web services site has some good examples but in particular, I would recommend you look at ‘Test Utility for Amazon S3 in PHP‘ which does a good job of demo’ing most of the S3 functionality using PHP.

I used this code as a starting point to develop a very simple ‘Rsync’ type application for Amazon S3.

Resources

Documentation from Amazon
Example PHP implementation
Who’s Using S3 here and here
My Rsync App (Proof of Concept at this point)