Daily IT Readings

GoalBit - Open Source P2P Streaming

2009-10-05T23:09:00.002+03:00

A group of Uruguay-based P2P researchers recently released an open-source P2P streaming application,GoalBit. It is available for Linux and Windows.

GoalBit is a peer to peer distribution system, capable of distributing high-bandwidth live-content to all network peers preserving its quality. This project follows a bittorrent-like approach where the stream is decomposed into several flows sent by different peers to each client. In order to meassure the peers perceived quality, it is used the recently proposed PSQA (Pseudo-Subjective Quality Assessment) technology.

Backup your Friendfeed with PHP and MongoDB - Part 2

2009-08-11T11:38:00.002+03:00

I just learned that FriendFeed will be acquired by Facebook. So, I will post the backup script, reworked, so you can back up your items.



class FriendFeed {
 
public $mongo;
public $collection;
public $mondo_db_name           = 'ff';
public $mongo_db_collection     = 'items';
public $ff_username             = 'YOUR_FRIENDFEED_USERNAME';

function __construct(){
 //Set up connection to Mongo
 $this->mongo = new Mongo();
 $this->collection = $this->mongo->selectDB( $this->mondo_db_name )->selectCollection( $this->mongo_db_collection );
 $this->collection->ensureIndex( array( "id" => 1 ) );
}

/**
* This function constructs the FF url which reads the user's feed.
* It reads $num messages starting from the $start message.
*/
private function getUrl($start, $num){
 return "http://friendfeed-api.com/v2/feed/".$this->ff_username."?start=$start&num=$num&fof=1&maxcomments=100";
}

/**
* Reads $num messages starting from the $start message.
* It seems like $num=100 is the maximum supported by FF.
*/
public function read($start=0, $num=100){
 $json = file_get_contents($this->getUrl($start,$num));
 return self::conv_obj(json_decode($json));
}

/**
* This function converts the FF json to array.
*/
public static function conv_obj($data){
 if(!is_object($data) && !is_array($data)) return $data;
 if(is_object($data)) $data = get_object_vars($data);
 return array_map(array('FriendFeed','conv_obj'), $data);
}

/**
* This function reads $num messages and inserts them in mongodb
*/
public function update($start=0,$num=100){
 $data = $this->read($start,$num);
 //Process the FF data, updating Mongo with each item
 if (!empty($data))
  foreach ($data['entries'] as $entry){
   //upsert the item in mongo
   $this->collection->update(array("id" => $entry['id']), $entry, true);
  }
}

}

$ff = new FriendFeed();

/*//UNCOMMENT BELOW TO DO BACKUP
$max_num_items = 1600; //change this if you have more items
foreach (range(0,intval($max_num_items/100)) as $i) $ff->update(100*$i);
//*/


/*//UNCOMMENT BELOW TO LIST YOUR 10 LATEST ENTRIES
$cursor = $ff->collection->find()->sort(array( "date" => -1 ))->limit(10);

foreach ($cursor as $value) {
    echo $value['date'].' ('.$value['via']['name'].'): '.html_entity_decode($value['body'])."\n";
}
//*/
?>

Backup your Friendfeed with PHP and MongoDB

2009-08-05T16:47:00.004+03:00

I wanted to backup my FriendFeed (FF), because... well, because it backs up a lot of my social activities. So this is the first attempt. I will use MongoDB because it works naturally with JSON, which suits my needs just fine. I will also use PHP.

What follows is dead simple. First a crude wrapper for accessing FF.


class FriendFeed {
   private $user;

   function __construct($user){
       $this->user=$user;
   }

   /**
    * This function constructs the FF url which reads the user's feed.
    * It reads $num messages starting from the $start message.
    */
   private function getUrl($start, $num){
       return "http://friendfeed-api.com/v2/feed/".$this->user."?start=$start&num=$num";
   }

   /**
    * Reads $num messages starting from the $start message.
    * It seems like $num=100 is the maximum supported by FF.
    */
   public function read($start=1, $num=100){
       $json = file_get_contents($this->getUrl($start,$num));
       return self::conv_obj(json_decode($json));
   }

   /**
    * This function converts the FF json to array.
    */
   public static function conv_obj($data){
       if(!is_object($data) && !is_array($data)) return $data;
       if(is_object($data)) $data = get_object_vars($data);
       return array_map(array('FriendFeed','conv_obj'), $data);
   }
}

This wrapper only reads the FF entries and decodes them to array. Now lets use it to read 100 entries and insert them in MongoDB.

//CONFIGURATION
$mondo_db_name = 'YOUR_MONGO_DB_NAME';
$mongo_db_collection = 'YOUR_MONGO_COLLECTION_NAME';
$ff_username = 'FriendFeed_USERNAME';

//Set up connection to Mongo
$m = new Mongo();
$collection = $m->selectDB( $mondo_db_name )->selectCollection( $mongo_db_collection );
$collection->ensureIndex( array( "id" => 1 ) );

//Read data from FF
$ff = new FriendFeed($ff_username);
$data = $ff->read();

//Process the FF data, updating Mongo with each item
if (!empty($data))
   foreach ($data['entries'] as $entry){
       //output new items
       if (!$collection->findOne(array("id" => $entry['id'])))
           echo $entry['date'].":".$entry['body']."\n";

       //upsert the item in mongo
       $collection->update(array("id" => $entry['id']), $entry, true);
   }


//Output number of items in collection
$collcnt = $collection->count();
echo $collcnt."\n";

Make this run once per hour or once per day and you will have a local copy of your FF stream. Of course, this will not download photos and files. I may add an extension for this in the future, in case I need it.

If you need to cycle through all your local FF items:

$cursor = $collection->find();
$cursor = $cursor->sort( array( "date" => 1 ) );

foreach ($cursor as $id => $value) {
   var_dump( $value );
}

If you need to backup all your items, you need to replace $data = $ff->read(); and the code that follows with something like this:

while ($data = $ff->read($start)) {
   //processing code
   $start+=100;
}

This code (without the while loop) has been tested and works both on Linux and Windows, with PHP 5.3 and MongoDB 0.9.7.

Deioces

2009-03-25T14:42:00.002+02:00

The following story by Herodotus reminds me so much of current politics.

According to Herodotus, the Medes lived independently (autonomon) in scattered villages after they won their freedom from the Assyrians. Ambitious to unite the Medes under his rule, Deioces set about gaining a reputation for honesty. The men in his village grew to trust him and invited him to settle their disputes. As his reputation grew, more and more people submitted their disputes to him until he ﬁnally declared he had had enough and would judge no more lawsuits. His withdrawal plunged the country into lawlessness (anomia) and forced the Medes to make him king. Once in oﬃce, Deioces demanded that his subjects build him a vast palace at Ecbatana. When the palace was complete, Deioces remained inside to keep himself safe from plots and communicated with his people through messengers. He continued to judge lawsuits, but all cases were now submitted to him in writing so that he could keep his distance from the people.

excerpt from "Solon of Athens New Historical and Philological Approaches", Mnemosyne, Supplements, 272, Brill Publishing, 2006

Building a Personal Filtered RSS Reader with Oracle XE and Apatar - Part 2

2009-02-10T10:54:00.002+02:00

This is the second part of my post about building an RSS Reader with Oracle and Apatar. If you have all the prerequisites mentioned in the previous post in place, let's proceed with the database schema. It is actually just one table:

CREATE TABLE "FEEDHUB" (
"POSTDATE" DATE NOT NULL,
"TITLE" VARCHAR2(512 CHAR),
"SCORE" NUMBER, --the feedhub score of the post or null
"URL" VARCHAR2(1000 CHAR) NOT NULL,
"POSTRANK" NUMBER, --the postrank score of the post or null
"AUTO_ID" NUMBER,
CONSTRAINT "FEEDHUB_PK" PRIMARY KEY ("URL")
)

It combines both my Feedhub and Postrank feeds. This is enough to define the Apatar transformations which will load RSS posts into the table automatically.

Next start Apatar, modify this transformation with your feeds and database connection, and you are good to go. Each run will load the new rss entries into the database. You can also create a nice and quick web interface by using Oracle APEX. This is beyond the scope of this post, but it is really easy and should take about 20 minutes.

p.s. You can always achieve the same without installing any software - just use Yahoo Pipes.

The dumbest way to copy a file to a USB stick

2009-01-24T15:34:00.002+02:00

Let me describe my experience from last night where I had to download a 400MB file and copy it to a USB stick. This is one of the dumbest things I've ever done on a computer. Do not repeat it at home.

Step 1. (Mid-morning) I found out that the file I want is 400 MB. I do not want to just go and download it because this will eat all the bandwidth and the other people sharing my Internet connection will be extremely frustrated. I have to shape my connection. Well, I was running Windows and I don't know how to do this on Windows so I fired up Virtualbox where I have a Debian virtual machine (VM). I shape eth0 to 24KB and am good to go.

Step 2. (Late afternoon) I remembered my download - the file is there. It has to go to the USB stick. After two minutes I am cursing Virtualbox for making it so difficult to connect a USB device. After another 3 minutes and 2 restarts of the VM I give up.

Step 3. (One cigarette later) I set up the USB stick as a shared folder in Virtualbox. Then "modprobe vboxfs" and mount it... Now I have access to my stick from within VM. I start copying the file...................... A minute later I understand my mistake, the speed of transfer is 150 KB/s, on average. I curse... in Serbian :)

Step 4. (Evening) It is already dark outside. The file is on the USB stick. Finally. I test it - it opens from Windows. I carefully unmount everything, bag the stick and go.

Step 5. (Next morning, at home) I try to open the USB stick on my laptop. Vista says it wants to format it. I curse Vista.

Step 6. (At the office) My PC at last! Debian, no VMs. I insert the USB stick. Nothing happens. fsck.vfat says it's bad, the partition table is corrupted. I try for an hour to copy (dd) the contents somewhere so I can mount it on the loopback and eventually fdisk in. Nope. I curse my stupidity...

As I said don't try this at home.

Automated Game Design

2009-01-19T13:42:00.001+02:00

A very interesting post in togelius explores the concepts of automated game design. The novel concept here is that it proposes a learning algorithm to be used as a fitness function for the generated games.

Somewhat more technically, our fitness function proceeds in two stages: first it tries to play the game using only random actions. If a random player can win a the game, the ruleset (=the game) is assigned a negative fitness. Otherwise, an evolutionary algorithm is used to try to learn a neural network that plays the game (using the score of the game as fitness function). The fitness of the game then becomes the best fitness found by the "inner" evolutionary algorithm after a certain number of generations.

There are also other interesting resources on this topic like this article, or this older one which explores the automatic generation of dungeons for computer games.

Building a Personal Filtered RSS Reader with Oracle XE and Apatar

2009-01-18T12:55:00.000+02:00

Today I will show you how to build a personal, filtered, web-based RSS reader with free tools and no programming (except simple SQL). I will not go into too many details so that you can have fun experimenting with the tools. Before we start you will need:

A list of the URLs of your favorite feeds - the more the better because we will be filtering the content. High-profile, popular RSS feeds will work better with the filtering I propose. OPML format will be best.
Oracle XE - it's free to use. You might also want to install Oracle SQL Developer - it will make developing databases easier.
Apatar - This is a FOSS ETL tool which we will use to transform the RSS feed items and automatically import them in the database.

In order to have filtering you should use the Feedhub , Postrank or Feedscrub services. In my example I used the first two. Please note that Feedhub and Feedscrub need some "training" in order to work well. After you've registered for these services import your feed list in them and get the URLs of the personalized RSS feeds they provide. You will need those shortly.

In the next post we will create the database and Apatar transformations.

Fun with Oracle and the Dow Jones

2008-12-16T21:23:00.003+02:00

I have written an article describing my explorations in the world of Oracle. I actually started learning Oracle two days before I wrote this article. So if you find that some of the things in it are not done in the best way, you can leave a comment. Before starting you will need of course an Oracle server. You can get the free version - Oracle XE. You might also want to install Oracle SQL Developer. A good knowledge of general SQL is also necessary.

The main question I explore is can we use Oracle to tell us when the financial crisis started?

p.s. Of course, the PostgreSQL fans can try and repeat the steps from the article in PostgreSQL, you will need 8.4dev though in order to use the analytical functions support.

p.s. PostgreSQL fans are actually out of luck, because the implementation of analytical functions does not allow for arbitrary ranges...

Google's newsest patent

2008-09-07T09:46:00.000+03:00

Google has patented a floating platform-mounted computer data center:

A system includes a floating platform-mounted computer data center comprising a plurality of computing units, a sea-based electrical generator in electrical connection with the plurality of computing units, and one or more sea-water cooling units for providing cooling to the plurality of computing units.

Long prime numbers

2008-06-07T18:51:00.001+03:00

1979339333 and 1979339339 are remarkable prime numbers. Do you know why?

Did Amazon Induce Vista's Premature Birth?

2008-02-22T09:31:00.000+02:00

A recent Amazon SEC filing sheds light on the puzzling Sept. 2006 departure of Microsoft Sr. VP Brian Valentine, the Gen. George Patton-like figure charged with pushing Vista developers, who dumped the still not-ready-for-prime-time OS into RC1 status as he bolted for a new gig at Amazon. It seems Valentine had actually signed an Employment Agreement way back in June calling for him to be on board at Amazon on Sept. 11 if he wanted to pick up a $1.7M signing bonus, $150K base salary, another $500K bonus, and 400K shares of Amazon (now worth almost $30M).

Delving into the recent PostgreSQL commits

2008-02-19T09:11:00.000+02:00

After the recent release of PostgreSQL 8.3 activity has now shifted to the future 8.4 release. Tom Lane made an interesting commit last week which removed the restriction that LIMIT/OFFSET can't contain a sub-select. You can now write queries like:

SELECT * FROM test
ORDER BY col1
LIMIT (SELECT testlimit FROM confs WHERE name='test.col1')

Well, you can do this only if you compile PostgreSQL from source, not in 8.3, but I hope it will be included into the 8.3.1 release.

Vendr Neutral SQL Certification

2008-01-28T19:07:00.000+02:00

CIW Database Design Specialist, scheduled to launch in March 2008, is the latest addition to the vendor-neutral, CIW Certification Program.

This new vendor-neutral certification focuses on universal database design principles and SQL. Aimed at database programmers and administrators alike, the exam helps solve the problem of poorly designed databases and validates foundational knowledge of any database, regardless if it’s Oracle, IBM, DB2, MySQL or others.

Flickr to host Library of Congress photos

2008-01-17T13:29:00.001+02:00

The good news is the Library of Congress is putting 3,000 images up at Flickr. The bad news is they're relying on us to tag them all.

Flickr to host Library of Congress photos | Webware : Cool Web apps for everyone

Sun Microsystems Announces Agreement to Acquire MySQL

2008-01-17T11:04:00.001+02:00

Sun Microsystems, Inc. (NASDAQ: JAVA) today announced it has entered into a definitive agreement to acquire MySQL AB, an open source icon and developer of one of the world's fastest growing open source databases for approximately $1 billion in total consideration. The acquisition accelerates Sun's position in enterprise IT to now include the $15 billion database market. Today's announcement reaffirms Sun's position as the leading provider of platforms for the Web economy and its role as the largest commercial open source contributor.

Blogged with Flock

SQL/PSM Manual

2007-11-08T10:40:00.000+02:00

I just noticed that there is a PL/pgPSM manual published. I plan to install and give it a try very, very soon, actually as soon as I install 8.3.

I wonder if all features mentioned in the manual are actually present...

Relational database pioneer says technology is obsolete

2007-09-07T12:56:00.001+03:00

As a researcher at the University of California, Berkeley, in the early 1970s, Michael Stonebraker co-created the Ingres and Postgres technology that underlies many leading relational databases today: Microsoft Corp.'s SQL Server, Sybase Inc.'s Adaptive Server Enterprise, Ingres Corp.'s eponymous product, IBM's Informix, and others. But Stonebraker now argues that relational databases, also known as RDBMSes, are "long in the tooth" and "should be considered legacy technology."