The Data Charmer

OSCON 2008 - MySQL Proxy - from architecture to implementation

2008-07-25T01:31:00.004+02:00

The presentation about MySQL Proxy at OSCON 2008 is over.

Here are the slides.

Thanks to my co-presenter Ronald Bradford and to all the participants. If you have more questions about the session, please use this blog's comments.

Some quirks of circular and row-based replication

2008-07-21T13:30:00.000+02:00

One of the new features introduced by MySQL 5.1 is row-based replication.

Unlike the classic statement-based replication, used in MySQL up to version 5.0, row-based replication transfers the data instead of the statement used to create it.

If you want to have a taste of row-based replication, you can do some experiments with MySQL Sandbox.

First, we create a sandbox of circular replication with MySQL 5.0

./make_replication_sandbox --topology=circular --how_many_nodes=3 \
/path/to/mysql-5.0.51a-YOUR_OS.tar.gz

cd $HOME/sandboxes/rcsandbox_5.0.51
./n1 -e "create table test.t1(i int)"
./n3 -e "insert into test.t1 values (@@server_id)"
./use_all "select * from test.t1"
# server: 1:
i
101
# server: 2:
i
102
# server: 3:
i
103

This is statement-based replication at its best.

A look at the binary log confirms that this is the case.

 $ ./node3/my sqlbinlog node3/data/mysql-bin.000001 |tail
#080720  8:57:59 server id 103  end_log_pos 1958    Query   thread_id=8 exec_time=0 error_code=0
SET TIMESTAMP=1216569479/*!*/;
SET @@session.auto_increment_increment=3, @@session.auto_increment_offset=3/*!*/;
insert into test.t1 values (@@server_id)/*!*/;

It may be desirable to preserve the different variables in every slave, and I know of a few schemes where it is useful, but in most cases a user wants the data on master and slaves to be always the same (think of backup slaves, for instances), and therefore the statement-based replication should be considered faulty in this respect.

Let's do the same experiment with the latest 5.1

./make_replication_sandbox --topology=circular --how_many_nodes=3 \
/path/to/mysql-5.1.26-rc-YOUR_OS.tar.gz

cd $HOME/sandboxes/rcsandbox_5.0.51
./n1 -e "create table test.t1(i int)"
./n3 -e "insert into test.t1 values (@@server_id)"
./use_all "select * from test.t1"
# server: 1:
i
103
# server: 2:
i
103
# server: 3:
i
103

Now, that's different. We can see row-based replication in action. And we can see that the binary log looks a lot different.

 $ ./node3/my sqlbinlog node3/data/mysql-bin.000001 |tail
#080720  9:05:00 server id 103  end_log_pos 3247        Query   thread_id=25    exec_time=0     error_code=0
SET TIMESTAMP=1216569900/*!*/;
SET @@session.auto_increment_increment=3, @@session.auto_increment_offset=3/*!*/;
BEGIN
/*!*/;
# at 3247
# at 3288
#080720  9:05:00 server id 103  end_log_pos 3288        Table_map: `test`.`t1` mapped to number 19
#080720  9:05:00 server id 103  end_log_pos 3322        Write_rows: table id 19 flags: STMT_END_F

BINLOG '
LGKDSBNnAAAAKQAAANgMAAAAABMAAAAAAAAABHRlc3QAAnQxAAEDAAE=
LGKDSBdnAAAAIgAAAPoMAAAQABMAAAAAAAEAAf/+ZwAAAA==
'/*!*/;
# at 3322
#080720  9:05:00 server id 103  end_log_pos 3392        Query   thread_id=25    exec_time=0     error_code=0
SET TIMESTAMP=1216569900/*!*/;
COMMIT

So far, all is as expected (if you have been keeping up with the documentation, of course). The default binlog_format in MySQL 5.1 is mixed, meaning that the server will use statement-based as much as possible, switching to row-based when there is the risk of incorrect replication, like in this case.

Users can change binlog_format at will, thus influencing how replication is performed. However, in the case of circular replication, you must be extra careful, because you don't have control on how the other nodes are behaving.

For example.

$ ./n1 -e "truncate test.t1"
$ ./n3 -e "set binlog_format='statement';insert into test.t1 values (@@server_id)"
$ ./use_all "select * from test.t1"
# server: 1:
i
101
# server: 2:
i
101
# server: 3:
i
103

WTF? Why server 1 and server 3 have inserted their corresponding server ids, and server 2 has not?

Let's do some debugging.

Server 3 inserts @@server_id with statement-based replication, overriding the defaults.
Server 3's slave, server 1, executes the statement. But its default binlog_format is still "row", not "statement".
Server 2, slave of server 1, receives a row-based chunk from the binary log, and therefore inserts the value of its master, because the statement was not passed along.

Using row-based format explicitly would not cause any side effect.


$ ./n1 -e "truncate test.t1"
$ ./n3 -e "set binlog_format='row';insert into test.t1 values (@@server_id)"
$ ./use_all "select * from test.t1"
# server: 1:
i
103
# server: 2:
i
103
# server: 3:
i
103

Looking for really useful uses of row-based replication, we can experience the greatest advantage (compared to statement-based) when inserting the result of an expensive calculation. Let's try. First of all, we force row-based replication on all nodes permanently.

$ ./use_all 'set global binlog_format="row"'

Then we execute an expensive query, and see the results.

$ ./n3 -e "insert into test.t1 select count(*) from information_schema.columns"
$ ./use_all "show global status like 'opened_tables' "
# server: 1:
Variable_name   Value
Opened_tables   17
# server: 2:
Variable_name   Value
Opened_tables   17
# server: 3:
Variable_name   Value
Opened_tables   41

If you know how information_schema works in MySQL, you know that if you want to get statistics on the number of columns, the server has to open all the tables and count them. Server 3 has done exactly that, and the number of opened tables is more than double than its fellows servers.

This is a useful operation to perform when you know that the master is inserting the result of an expensive calculation, and you don't want to delay the slaves.

Switching to row-based replication permanently would be a mistake, though. Consider a simple update like this one:

UPDATE huge_table set some_column = 'some value' where some_other_column = 'X';

This would be relatively inexpensive in statement-based replication. If the table has 200 columns, though, using row-based format your binary log will pass the whole record to the slaves.
Used with care, row-based replication can deliver good results.

Proxy webinar - slides, script, FAQ

2008-07-09T12:29:00.004+02:00

We did it. Designing Scalable Architectures with MySQL Proxy was delivered successfully, with over 150 attendees.
There is a large number of questions that were asked during the session, and you can find them in MySQL Proxy FAQ.
The slides, with the highly entertaining images used by John Loehrer to illustrate his point are also online
Finally, John posted his Connection pooler Lua script in the Forge.
Thanks to John Loehrer for his lively presentation, to Jimmy Guerrero and Rich Taylor for organizing the event, to Jan Kneschke, for answering questions online while we were talking, and to all the attendees, for showing such degree of interest.
And don't forget that there is still a quick poll on the future of MySQL Proxy that you can still vote on.

Shaping the future of MySQL Proxy

2008-07-08T06:30:00.001+02:00

Here is a chance for all users to influence the future development of MySQL Proxy.
If you care about Proxy, you may want to check the current quickpoll in the Dev Zone, and vote for your favorite features.
The developers have a truckload of ideas, of course, but only a finite amount of time. So they can't develop all the features at once. They must start somewhere, and your vote can help them decide which ones should get higher priority.

In the meantime, don't forget the incoming webinar on Designing Scalable Architectures with MySQL Proxy. It's TODAY (tomorrow for some, depending on your time zone), July 8, 2008 at 10am PDT, 1pm EDT, 7pm CEST.

Even faster online backup!

2008-07-07T20:15:00.005+02:00

I discussed my findings with Guilhem Bichot, one of the online backup creators, and he remarks:

You could also try
export MYISAM_BACKUP_NO_INDEX=1
before starting mysqld. It should not backup index (and rebuild them at repair time). Should make a smaller backup and a longer restore.

I am not really looking for a longer restore, but let's give it a try. I restarted the database with the suggested option, and here is what I got:


backup database employees to 'emp2.bkp';
+-----------+
| backup_id |
+-----------+
| 4         | 
+-----------+
1 row in set (3.18 sec)

drop schema employees;
Query OK, 10 rows affected (0.02 sec)

restore from 'emp2.bkp';
+-----------+
| backup_id |
+-----------+
| 5         | 
+-----------+
1 row in set (18.33 sec)

The backup is definitely faster, and the restore is slower than the 11 seconds I got with the previous test. However, it's much faster than the standard restore!
Moreover, the backup file is barely 70 MB. That's an excellent compromise. Perhaps it won't be that fast with a several GB of data, but for now, it looks very good.

Faster online backup with MyISAM driver

2008-07-07T14:30:00.003+02:00

Remember the first test of online backup? I tested the new feature, which was performing quite well, compared to mysqldump. OK. Get ready for a surprise.
I tested the native MyISAM driver from the mysql-6.0-backup tree, and I compared the results with the normal backup.

version	backup time	restore time
standard	25.58	79.11
MyISAM driver	4.15	11.53

Please be amazed!
The difference is also visible from the metadata. The standard version says:

select * from mysql.online_backup\G
*************************** 1. row ***************************
      backup_id: 1
     process_id: 0
     binlog_pos: 0
    binlog_file: NULL
   backup_state: complete
      operation: backup
      error_num: 0
    num_objects: 8
    total_bytes: 69970045
validity_point_time: 2008-07-07 11:39:04
     start_time: 2008-07-07 11:38:39
      stop_time: 2008-07-07 11:39:04
host_or_server_name: localhost
       username: msandbox
    backup_file: emp.bkp
   user_comment:
        command: backup database employees to 'emp.bkp'
        engines: Default, Snapshot

And the native driver version says:

select * from mysql.online_backup\G
*************************** 1. row ***************************
      backup_id: 1
     process_id: 0
     binlog_pos: 0
    binlog_file: NULL
   backup_state: complete
      operation: backup
      error_num: 0
    num_objects: 8
    total_bytes: 335531054
validity_point_time: 2008-07-07 11:32:04
     start_time: 2008-07-07 11:32:00
      stop_time: 2008-07-07 11:32:04
host_or_server_name: localhost
       username: msandbox
    backup_file: emp.bkp
   user_comment:
        command: backup database employees to 'emp.bkp'
        engines: Snapshot, MyISAM

Doesn't it just make you want to rush to download the code at once?
The information about total_bytes is somehow puzzling. The size of the backup file is actually 89 MB for the standard version and 160 MB for the MyISAM driver. Thus, you pay a faster backup and restore with a bigger file. Personally, I can live with that.

As you may suspect, the picture does not have any relationship with backup, but it's cute, isn't it?

MySQL Sandbox 2.0 has been released

2008-07-02T23:30:00.001+02:00

Taking advantage of Launchpad excellent development services, I completed the tasks necessary to release MySQL Sandbox 2.0.
So, you may ask, what's new? Quite a lot, actually:

more intuitive script names
master sandbox directory
faster
supports circular replication
more error checking
includes a test suite
uses a default options file

more intuitive script names

One of the problem in the old sandbox was that the names of the scripts suggested some installation of the sandbox itself. This has been addressed in version 2.0. You now have the very clear names:

 make_sandbox                  creates a single sandbox
make_replication_sandbox      creates a replication sandbox
make_multiple_sandbox         creates sandbox containing many servers
                                of the same version
make_multiple_custom_sandbox  creates a sandbox containing many servers
                                of different versions

master sandbox directory

By default, MySQL Sandbox will create a directory under your home, named "sandboxes", and all sandboxes will be created under that directory. In addition of making your sandboxes more organized, the master directory provides some handy scripts to deal with many sandboxes at once. You can start, stop, or send a SQL command to all the sandboxes with just one command.

faster

The Sandbox had a few "sleep" instructions, to wait for the server being installed to come online. This solution was slowing down fast machines, where the server can start much faster than the standard 3 seconds I added to the sandbox, but was sometimes not enough for slower machines, where the server may take more than 3 seconds to start up, especially with replication. The new sandbox uses a loop coupled with a timeout limit. Thus, quick machines are faster, and slow machines don't fail.

supports circular replication

Now you can create a replication ring.

./make_replication_sandbox --topology=circular --how_many_nodes=5 VERSION
# or
./make_replication_sandbox --master_master VERSION

includes a test suite

If you have resources and MySQL binary tarballs at your disposal, you can stress test MySQL Sandbox (and your box!) with this test suite. For example, assuming that your 5.0.51 and 5.1.25 binaries are in $HOME/opt/mysql

./test/test_sandbox.pl --versions=5.1.51,5.1.25

This will create 22 sandboxes

more error checking

This is a direct consequence of the above item. Having a test suite has made the sandbox much more robust.

uses a default options file

There is also a default option file, where you can keep the options that you need to be executed always.

Try it!

Go get the 2.0 tarball from launchpad. If you find any bugs, there is a very convenient bugs reporting system. Please use it!

OSCON 2008 - Getting in touch with the open source makers

2008-07-02T14:23:00.003+02:00

For some reasons, I have never been able to attend OSCON so far. I have been going to the MySQL Users Conference almost every year in April, and that usually left me without a great desire of crossing the Atlantic again. This year is different. I have crossed the Atlantic already three times, and yet I am about to board a plane once more, heading for the Portland, Oregon.

And for some lucky circumstances, I am also a speaker.
I will join Ronald Bradford on the podium, to speak about (surprise surprise!) MySQL Proxy, from architecture to implementation.
I will have also my hands full in the evenings. There will be plenty of BoFs (including two that I will host) and parties, social gatherings, competitions, which I plan to enjoy for my personal enlightenment, not only to fulfill my duties.
I am looking forward to meet old friends and to make new ones.

Looking back at the first half year

2008-07-01T00:37:00.006+02:00

Half a year has gone, most of it with my new company, Sun Microsystems.Kaj has made a detailed digest of his first half.
I took the DBA approach and queried the Planet MySQL database. I know that this qualifies as cheating, but I could not resist.
I published a total of 111 blog posts since January, with peaks in March and April (the users conference was coming).
I wrot about most everything, but some topics are prominent. Users Conference and MySQL proxy are the winners. All in all, I would say that I have fairly distributed my topics.

Scalable architectures with MySQL Proxy

2008-06-25T22:40:00.001+02:00

MySQL community, mark your calendars!. On July 8th, 2008, there is a Webinar on Designing scalable architectures with MySQL Proxy.
This is not the usual marketing sponsored webinar. Although we love to show off, this is not a "look-how-good-we-are" presentation. This is a community driven event, where a community member, using only MySQL Proxy and some creativity, solved his production problems.

This is a real story of a community member who used open source software to build a customized scalable architecture to suit his purposes. Isn't it a good story?
I won't steal his thunder and tell you in advance what was the problem about. I will introduce the general concepts about Proxy, and then our guest John Loehrer will tell hist story in full colors.
Don't miss this one!

MySQL Sandbox - planning for 2.0

2008-06-20T09:55:00.003+02:00

Taking advantage of Launchpad environment, I started paving the way for version 2.0 of MySQL Sandbox.
I opened a new branch for version 2.0, which currently holds 1.99.1, and I listed the necessary tasks using the built-in blueprints feature.

So now I have a list of tasks to keep track of the development.

implement test suite (have a good prototype - I am getting to it)
replace sleep with check loop (thinking - some analysis required)
create function library (good progress)
script to kill unresponsive servers (good progress)
remote sandbox installer (thinking - hard to implement without being intrusive)
allow url for tarball (thinking - needs external components)
define circular replication (thinking - easily doable)
refactoring for internationalization (thinking - easily doable - long task)

The above features should all go to version 2, although the list is still open. No feature freeze has been called. It's still alpha.
I am unsure about the remote installer. I have implemented such a feature a few years ago, and it turned out to be quite intrusive (the choice was between being intrusive and fragile). I may consider making it a plugin or moving it to a later version. Still thinking.

The development model is open. If you use the Sandbox and want to submit a patch or have comments on its development plans, please contribute!

From Bazaar to Sandbox in 5 moves

2008-06-19T15:30:00.002+02:00

Now that MySQL has switched its development to Bazaar, we can play a bit with the new tools. One thing that I have tried immediately upon adoption of the new RCS is to create a test bench for multiple MySQL versions by combining Bazaar and MySQL Sandbox.

If you are into bleeding edge testing, you may like to know how to get a new MySQL server up and running in a sandbox from a source tree.

First move - make the repository

This is a zero time consuming move, and you will have to do this only once. But this move will save you heaps of time in future. By creating a shared repository, you are telling Bazaar to avoid redundant copies of files (actually Bazaar is much smarter than this, but as a simple explanation it will do).

$ bzr init-repo shared
$ cd shared/

Second move - get the source

Now we're ready to get the first copy of our source. The first branch we get is the current tree, 5.1. Depending on the speed of your network connection, it will take between 20 and 40 minutes.

$ time bzr branch lp:mysql-server/5.1  51

lp stands for launchpad. mysql-server is the project name, and 5.1 is the version. 51 is the directory we are creating on the local disk. You can call it whatever you want. I prefer short names.

This operation may be tricky. The current Bazaar client (1.5) is not verbose. It may actually appear to be doing nothing for very long time. It is actually importing code. Don't be discouraged and bear with me on this. It will pay off in the end. After a while you will see a progress bar and eventually it will display the result.

Branched 2655 revision(s).

real 19m14.110s
user 7m22.883s
sys 1m21.372s

In my laptop, it took less than 20 minutes, but it could be much slower, depending on connection type and intensity of net traffic.

Let's have a look at the size of what we have downloaded. The hidden directory .bzr contains the biggest chunk.

$ du -sh .bzr/ */
498M .bzr/
129M 51/

Now for the juicy part. We want to branch also MySQL 5.0. You may be ready for yet another 20 minutes and 600 more MB of data to be added, but you are in for a good surprise:

$ time bzr branch lp:mysql-server/5.0 50
Branched 2634 revision(s).

real 0m42.205s
user 0m28.100s
sys 0m4.757s

Less than one minute! And what about storage?

$ du -sh .bzr/ */
498M .bzr/
100M 50/
129M 51/

As you see, the new code added just the difference between 5.1 and 5.0, so it did not even had to get it from the net because it mostly got the previous version from the internal history (we'll come back to this point in a later post).
A similar experience awaits you when branching the newest version, 6.0. A bit slower than 5.0 (because it has to get something more from the net), but still quite fast compared to the full tree we got in the first run.

$ time bzr branch lp:mysql-server/6.0 60
Branched 2664 revision(s).

real 4m45.383s
user 1m39.727s
sys 0m20.776s

$ du -sh .bzr/ */
566M .bzr/
100M 50/
129M 51/
146M 60/

Third move - export the code

We have now on disk the code for three versions. We could just go inside each directory and build it, but that would not be clean. So in this move we just export the code to a build directory that we create for this task.

$ mkdir ~/install/build

$ ls
50    51    60

Using a shell loop, we export each tree to a build directory. It takes about 30 seconds for each tree.

$ for R in 50 51 60 ; do echo $R ; cd $R ; bzr export ~/install/build/$R ; cd .. ; done
50
51
60

Fourth move - build the code

We are finally ready to build. Each build can take quite long, depending on your box. In my laptop, it takes about 40 minutes per build.

In the ./BUILD directory in each source tree there are several building scripts. Choose the one that suits your purposes, eventually disabling the parts that you don't need (e.g. cluster).

$ cd ~/install/build

$ for R in 50 51 60 ; do \
  cd $R ; \
  ./BUILD/your_favorite_script && ./scripts/make_binary_distribution ; \
  cd .. ; \
  done

After each build, this loop runs make_binary_distribution, which creates a binary tarball, right what you need for MySQL Sandbox.

Fifth move - sandbox it!

Now, let's use our favorite installation tool to use these new binaries.

Download the sandbox. Make sure that you got at least version 1.21.

$ export SANDBOX_HOME=$HOME/sandboxes
$ cd ~/install/mysql_sandbox-1.21

For each version, you invoke the sandbox installer as

./express_install full/path/to/tarball_name.tar.gz

Here we go

$ ./express_install.pl ~/install/build/50/mysql-5.0.66-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/50/mysql-5.0.66-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/50/5.0.66 \
 --sandbox_directory=msb_5_0_66 \
 --install_version=5.0 \
 --sandbox_port=5066 \
 --no_ver_after_name
  The MySQL Sandbox,  version 1.21 09-Jun-2008
  (C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory                 = /Users/gmax/sandboxes
sandbox_directory              = msb_5_0_66
sandbox_port                   = 5066
datadir_from                   = script
install_version                = 5.0
basedir                        = /Users/gmax/install/build/50/5.0.66
[...]
loading grants
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_5_0_66


$ ./express_install.pl ~/install/build/51/mysql-5.1.26-rc-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/51/mysql-5.1.26-rc-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/51/5.1.26 \
 --sandbox_directory=msb_5_1_26 \
 --install_version=5.1 \
 --sandbox_port=5126 \
 --no_ver_after_name
  The MySQL Sandbox,  version 1.21 09-Jun-2008
  (C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory                 = /Users/gmax/sandboxes
sandbox_directory              = msb_5_1_26
sandbox_port                   = 5126
datadir_from                   = script
install_version                = 5.1
basedir                        = /Users/gmax/install/build/51/5.1.26
my_file                        =
[ ... ]
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_5_1_26

$ ./express_install.pl ~/install/build/60/mysql-6.0.6-alpha-darwin9.3.0-i386.tar.gz
unpacking /Users/gmax/install/build/60/mysql-6.0.6-alpha-darwin9.3.0-i386.tar.gz
Executing ./install.pl --basedir=/Users/gmax/install/build/60/6.0.6 \
 --sandbox_directory=msb_6_0_6 \
 --install_version=6.0 \
 --sandbox_port=6060 \
 --no_ver_after_name
  The MySQL Sandbox,  version 1.21 09-Jun-2008
  (C) 2006,2007,2008 Giuseppe Maxia, MySQL AB
installing with the following parameters:
home_directory                 = /Users/gmax/sandboxes
sandbox_directory              = msb_6_0_6
sandbox_port                   = 6060
datadir_from                   = script
install_version                = 6.0
basedir                        = /Users/gmax/install/build/60/6.0.6
[ ... ]
sandbox server started
installation options saved to current_options.conf.
To repeat this installation with the same options,
use ./install.pl --conf_file=current_options.conf
----------------------------------------
Your sandbox server was installed in /Users/gmax/sandboxes/msb_6_0_6

At the end of the exercise, you have three sandboxes with the three major MySQL versions, ready to use. Enjoy!

Sun Tech day in Cagliari

2008-06-16T16:30:00.003+02:00

The event had an ominous start. Apparently, nobody among the geeks who organized it had paid any close attention to the date to see if it had any significance, and nobody noticed that it was one of the dates when Italian football team was playing a game in the European championship!
I was reminded of a similar case, during FrOSCon 2006, which was held during the World cup. Lenz gave a speech in front of 5 people, all friends and colleagues! So we were afraid that we could face a similar debacle.
Our fears were swept away when the event started, and the room was filled in to capacity by a very dedicated audience, welcomed by professor Giulio Concas, our generous host, who also introduced a project on software quality metrics.

Domenico Minchella, a passionate Solaris ambassador, made a convincing demonstration of ZFS crash recovery. He is never afraid of picking a fight with the audience, and he turns every flamebait into advocacy.

Stefano Sanna presented what he called "the pitiful case of a grown up man who has to use Sun SPOT to justify with his wife why he is still playing with Lego toys!".

The toy performed as expected, and Stefano has more excuses to continue promoting the Sun SPOT.

Also Fabrizio Gianneschi had a toy to play with. In addition to the news of Java technology advancement, he played with a Java powered pen that speaks and replays what you write.

After the last session, where I had to warn the impatient audience that the dolphins from the desk would be distributed only at the very end, Massimiliano Dessì and yours truly performed the by now celebrated "throw them the dolphins" stunt, which was greatly appreciated. With our delight, most of the audience stayed after the session, to enjoy technical chatting and refreshments.

stupid things you can do to kill your business

2008-06-16T10:24:00.004+02:00

In the global information age, you must get things right if you want to survive.

Mindless (mis)information.

My ISP, which has a near monopolistic position in my country, has a support phone line. I called it yesterday, because my DSL was down. The usual routine goes: dial #1 for phone, #2 for data problems. I dial #2, and I get a recorded voice "all operators are busy. You can get support at our web site, www.xxx.it". I just called you because I can't connect, you dumb!

Dumbest e-commerce ever

There is a huge movie theater nearby, which is convenient, but the queue at the ticket counter is uninspiring, especially during week-ends. So I check their online ticket sales. Can you believe this? They charge you an additional EUR 0.5 for online tickets. In my book, online services must be cheaper than manned ones, and this is why the airline and the books industries are moving their business online. Charging more for online tickets means that you did not understand the game, and probably you were cheated by the company you chose for this service. The latter is probably true, because, when I continued the booking procedure, I was asked to fill in my credit card details before being told how much I was charged. No way. I booked at the competitor movie theater, which is a few miles farther, but their e-commerce works as expected.

MySQL Sandbox 1.22 and the joys of developing on launchpad

2008-06-15T12:34:00.007+02:00

I have recently moved MySQL Sandbox to launchpad and Bazaar.

Working with Launchpad is really a joy. It is much more than providing a web interface to Bazaar version control. It also includes provisions for bug reporting and handling, writing blueprints for your roadmap, creating distribution files, and manage your packages if you want to release them in Ubuntu.
Using the above tools, I implemented a couple of new features to MySQL Sandbox, and I started to work on a roadmap for the next version, which is now taking shape.
The latest version (1.22) includes the ability of using a default option file ($HOME/.msandboxrc), a safer usage of the "--force" option, and several bug fixes.

MySQL Sandbox 1.21 - matrioska and launchpad

2008-06-10T00:52:00.004+02:00

MySQL Sandbox has moved to a new home. It is now hosted at Launchpad, a friendly environment where developers can work together.

What's new with MySQL Sandbox? The matrioskas in the image are a clue. Version 1.21 introduces the concept of SANDBOX_HOME, which previously was only a path under which store the sandboxes. With time, when I was using more and more sandboxes, I realized that I wanted to take control of all the sandboxes at once, and issue global commands to all of them.
Starting with this version, the sandbox installer recognizes an environmental variable named $SANDBOX_HOME, and builds the sandboxes under that path.
In addition to grouping all the sandboxes in a place, the installer creates a few scripts that allow you to take control of all the active sandboxes.
Let's say that you have a single sandbox of MySQL 5.0.62, a replication system of 7 5.1.24 servers, and a multiple installation of 5 6.0.5 servers.
Prior to this version, if you wanted to stop each sandbox, you should have to enter each sandbox directory and run "./stop" or "./stop_all". With the sandbox outer container, you just have to run "./stop_all" from one place.
You can also execute the same command in each server in all your sandboxes, by means of the "./use_all" script.
If you want to learn more, on Thursday, June 12, 2008 at 15:00 CEST, I will host a MySQL University session on this very subject.

Which OS for your MySQL server?

2008-06-05T19:01:00.006+02:00

On MySQL developers zone there is a quick poll on which operating system you use.

The combined results of all Linux flavors accounts for 50% of the answers so far. What surprised me is the 6% of mac OSX used as a server. My own experience with macs is just as desktop or laptop. In my very personal view, it feels like a waste to use such a beautiful thing like a Mac as a server. Linux, as Neal Stephenson put it in In The Beginning Was The Command Line is like a tank. Efficient but not appealing, compared to Macs, which are described as stylish cars.
So, if business is a battle, the Linux tank seems your best bet to get out alive when the smoke settles. The funny thing is that Solaris is an even sturdier tank than Linux, and it should deserve more attention. I guess that, being a recent contender in the open source arena, it will have to pay the price of the learning curve. A price that Linux has paid long time ago and now is reaping the fruit.
If you disagree, have your say!

BoF at OSCON

2008-06-02T23:29:00.007+02:00

I have just been informed that in addition to a regular session, I will host two Birds Of a Feather sessions at OSCON 2008.

The call for papers is still open. Any suggestions for more entries?

Firefox 3 is coming

2008-06-02T21:32:00.001+02:00

Firefox has been my steady web companion since version 1.0 (and Mozilla before that). For some time now I wanted to try Firefox 3. I was reluctant to abandon Firefox 2, because of the unavailability of most add-ons. I was especially missing the Google Bar, which I have been using on a regular basis.
However, a few days ago I found a replacement. GoogleBar Lite is almost like the original, minus some feature that I wasn't using anyway.
So I have been using Firefox 3 for a while, and I noticed better performance, and no trouble. The additional features are cool, like the Most Visited pull down menu or the location bar integrated with history and search.
Firefox enthusiasts want to set a download world record. I don't think it's necessary, but certainly will be pleasant to achieve it. I pledged.

Can you see advertising?

2008-06-01T22:50:00.000+02:00

While preparing a presentation for Rome University, I took many snapshots of MySQL web site. I asked for review, and Colin pointed at the advertising that was in most every page.
Now that he mentioned it, yes. I saw the advertising. But when I was working with the live page, taking the screenshots, adjusting them with the Gimp, and inserting them into the presentation, I did not notice them at all.
I am blind to ads.
I must be the worst nightmare for advertisers. I have pop-up blockers and javascript filters in my browser, so don't see many of them, but the remaining ones are like a ink stain on the page. My brain registers the presence of an alien presence, and quickly instructs my senses to ignore it. If someone tells me that there is an ad in a given page, I have to look at the page twice, to put the ad into focus.
Why does this happen?
I guess that, after so much time spent using the web, I am trained to spot where the content is, and I filter off automatically all the distracting bits.
If the ad contains an interesting image, I may register that, and even remember it sometimes, but I have no idea of what they were selling with that picture.

For instance, I saw this image in the Singapore subway. I took a picture because it was cute. I can't remember what was advertised.
Now, if this filtering happens due to my frequent usage of the web, I wonder how many users have developed the same self defense and evolved into a Homo Sapiens Non Publicitarius, a particular breed of human being who has been saturated by ads and can't see them anymore.
Anyone from the same mutation out there?

Piercing the fog

2008-05-28T11:30:00.002+02:00

There have been many surveys on MySQL, but usually not as deep as this one launched by Keith Murphy, editor of the MySQL Magazine and Mark Schoonover.
I have seen some turmoil lately, with talks of forking the code base and organizing a community conference. The community seems restless, and this initiative could be an attempt of counting their ranks to see how much they can dare.
I welcome this initiative of seeing through the fog. If a survey was needed, rather than waiting one to fall from the sky (or from the Sun :-) ), the industrious bloggers have started their own. I will be curious to see the results.
Inside MySQL, we have been talking about creating a phone home feature to do a live count of the user base, but as you can see there is no such thing yet. So this survey could provide the data that the missing phone home feature has failed to bring back. The user base measuring itself: that's an interesting thought.
If you can spare 10 minutes, I recommend taking the survey.

Chaining Proxies

2008-05-27T21:30:00.002+02:00

If you need to combine two scripts with MySQL Proxy you have three choices.

You can manually rewrite the two scripts (good luck!)
you can use the load-multi module to load scripts on demand;
or you can use the proxy chaining technique.

To chain two proxies, you need to start one Proxy with the first Lua script, pointing at the real backend, listening to a non-standard port. Then you start the second Proxy with the second Lua script, having the first proxy as a backend, and listening to the standard proxy port.
It's a difficult and error prone procedure. You would forget about it, unless there were an easy workaround. And indeed you can have such a workaround. Just use the chain_proxy script from MySQL Forge and use it with this simple syntax:

$ chain_proxy first_script.lua second_script.lua

When I made this script, I had the problem of handling the standard output for each proxy instance. The brute force solution is to start each instance in a separate terminal window, but that is less than optimal. It would be almost as much work as doing things manually.
Considering that usually I need the output of a Proxy session only for debugging and that I am a frequent user of GNU screen, I made this arrangement:
The script starts each proxy inside a separate screen. At the end, it gives the list of screens being used, and creates a customized script in /tmp/ to kill the chained proxies and remove the screens.

$ chain_proxy digits.lua loop.lua
proxy started on screen "second_proxy" using script "loop.lua" - pid_file : /tmp/proxy1.pid
proxy started on screen "first_proxy" using script "digits.lua" - pid_file : /tmp/proxy_chain1.pid
stop script is /tmp/proxy_chain_stop
There are screens on:
 21257.first_proxy (Detached)
 21258.second_proxy (Detached)
2 Sockets in /tmp/uscreens/S-gmax.

In this example, the two proxies are started in sequence, and the script gives information on what is going on. The output of "loop.lua" is in the "first_proxy" screen. To see it, I only need to do

$ screen -r first_proxy

When I am finished using the chained Proxy, I type

$ /tmp/proxy_chain_stop
There are screens on:
 21257.first_proxy (Detached)
 21258.second_proxy (Detached)
2 Sockets in /tmp/uscreens/S-gmax.

No Sockets found in /tmp/uscreens/S-gmax.

This script can also be implemented by redirecting the output of each proxy to a different file. YMMV. I like the screen solution better.

I will be speaking about this feature (and more) during my MySQL University session on advanced Lua scripting.

Virtual squares - Taking virtualization to new limits

2008-05-27T13:30:00.002+02:00

During the Italian Free Software Conference in Trento, I attended an amazing presentation on virtual components.
Renzo Davoli, professor at Bologna University and hacker, has stretched the concept of virtualization to new limits. Virtual Square is a project aiming at developing more dynamic implementations of virtual entities, which eventually get separated from the concept of operating system and root privileges.

The coolest aspect of all this project is the virtualization of single elements like a disk drive, a net port, a file system, without root privileges, and with no impact on other users.
Virtualizing single elements makes life easier for demanding users, and more quite for their neighbors, who won't be affected by massive reduction of overall resources as it happen with normal virtualization of operating systems.
Think of the applications: for example, it would be easier to establish dedicated quotas for database and web server users, with better security and easier maintenance and without creating new OS users. MySQL, with its limited interface for user resources, would surely benefit from this system. There is a lot of potential in this idea. Let's hope it is pushed a bit farther than the academic circles.

A look at the Open Source Census

2008-05-22T09:00:00.000+02:00

Yesterday I had a look at the Open Source Census (OSC), a project aiming at counting the installations of open source packages.

It is a collaborative effort. The OSC offers the infrastructure, and it's open to cooperation. It works with a disc scanner, and relies on the principle that the contributions are voluntary and anonymous.
If you want to contribute, you need to register, and you are encouraged to do so anonymously. The OSC is not interested in who you are, but the registration gives you a chance of tracking your results with the rest of the community.
The registration is also necessary to avoid duplicates, and to track your installations over time, should you decide to do that on a regular basis.
If you want to give it a try, the procedure goes like this:

Register (with any fancy name you want);
download the scanner package (it's open source)
scan your computer (or some selected directories), using your unique identifier
look at the results;
send the results to the census (using the same tool).

Bear in mind that there is no way for the OSC to track down who you are, because you are not asked in the first place. So the results are truly anonymous.

Some concerns:

The scan package is open source. However, even the simplest package available the pure Ruby) is more than 6000 lines of code. If you are security conscious, examining the code is not a quick task. You will have to rely on peer review from the community, or do a test scan in a virtual machine to examine what the software does (that's what I did);
If you have a large machine, the scan may take hours. You will have to plan for night scans if you want to contribute seriously.

On the plus side, the process of scanning is open. You can contribute the signatures of your favorite open source tools, to be included in the next scans.
Looking at the results may bring some surprises. I did not know that I had two different versions of MySQL-connector/J in my laptop, or two versions of PostgreSQL connector either! And I wonder how I managed to get 5 (FIVE) different versions of docbook-xml in my laptop, given that I haven't ever asked to install it.
However, the scanner gives you more than the public list of packages found in your box. For your own consumption (it is not sent to the census) it produces a detailed list of where each package was found. So you cam analyze that list and eventually clean up the system.
Other surprising results. I checked how often MySQL was installed, and it turns out to be present on 37% of the scanned boxes. The surprising results, however, is the the distribution of old versions. About 20% of MySQL servers are still using version 4.x. Fascinating!
The project is young, but very promising, in my opinion. There may be problems for adoption from large corporations (security policies will be hard to deal with), but if the community picks up, it may produce good results. Give it a try.

What's in a name?

2008-05-15T17:44:00.003+02:00

What's in a name? That which we call a rose by any other name would smell as sweet.

Good old William had his reasons to say so, but then either he was not concerned about misspellings or he didn't care. (A recent book about the Bard actually points out that all Shakespeare's known signatures are differently spelled, and none of them is spelled Shakespeare!)
The problem with my name is that, for the majority of non-Italians, it does not sound familiar, and consequently it gets misspelled. As a frequent traveler, I have seen every sort of mischief done to my first name.
I don't really understand why, but most English speakers write "GUIseppe" instead of "GIUseppe." Sometimes I was called "Giuseppa" (which is a female name) or "Giuseppi" (which is a non existing plural of my name), or "Jooseppai" (which is how I pronounce it) and several variations with additional or missing vowels.
Let me set the record straight, for anybody who cares.
The right spelling is "G-I-U-SEPPE". To remember the right order, you may use what I call the selfish rule: First I, then U. Got it? :)

Now, let me use this fact to compare the behavior of two companies. When I was hired by MySQL, I braced myself in wait of the misspelling, which did not come! All my official documents, personnel listings, email accounts, company badge, were produced without any mistake. I was impressed. Apparently, MySQL HR people are very attentive to these details. When I was hired by Sun, instead, well, let me say that it was not the same quality. MySQL was an international company, while Sun seems more an US company with international branches. They are learning, though.

I wrote the draft for this post long ago, but I left it aside. Now, since it seems that I am not the only one with this problem, I state my point.