Pons Aelius

Free VMware ESXi and the Hypervisor Wars

2008-07-29T10:45:00.007+01:00

It seems as though VMware ESXi is now free. I hope no one paid for it on Friday! At the moment we're using VMware Server to provide us with virtual machines running various services on Linux, as well as some Windows Server test machines that are running quite well on 8 gigabytes of RAM. All this runs on a CentOS host with the VM images residing on a XFS partition, and with Adaptec Storage Manager nicely monitoring the local RAID array.

While something like ESXi is nice from the point of view that I can worry an awful lot less about the host OS, the trouble with a limited hypervisor only install like ESXi, or even ESX, is that I don't necessarily get all the hardware drivers and support to install it on any machine I might want to and I am severely limited with regards to installing needed software on the host like RAID monitoring. In fact, I can't do it. Only ESX comes with a service console that allows you to manage the host, albeit in a fairly limited and restricted way. I suppose this duplication of effort, and the misplaced notion that the hypervisor is where everything should be, is why some people have had a go at the direction VMware and others like Xen seem to have been heading in. I think this whole separate hypervisor thing is nonsense as well, because when it comes down to it, you need to actually run something on the base hardware that actually makes the hardware.........work. With something like KVM it is tested, compiled and shipped with the kernel it runs within and can run as little or as much as anyone wants. Red Hat have already started. With hardware virtualisation that KVM uses it stops the imperfect need to trap instructions through software or through paravirtualisation, a much better approach, and simply leaves I/O virtualisation. This obviously needs a complete, robust and well tested set of device drivers, and guess who has these? That's right. Linux. Then there is power management, which Xen and VMware hosts simply do not have. Technically, KVM is just the better solution in terms of maintenance, and ultimately, performance.

Ultimately, I believe that this is where things will naturally head. Duplicating effort by forking kernels to retrofit Xen is just expensive and pointless, as is VMware's efforts to come up with their own customised and limited hypervisor-only host platform. Basically, anywhere that Linux can run, KVM can run, and any software you can run on Linux you can run on the host. In terms of reach, economies of scale and the wide range of devices KVM can be taken and run on will ultimately turn it into the clear leader. Lots of companies can also wade in and provide management tools. There are lots of places for KVM to go from where it is now, and is probably why the competition is trying to come up with an argument. While many people believe that the free ESXi is a response to Microsoft and Hyper-V, it is probably equally a play to fight off competition from Xen, and especially future competition from KVM, in their own back yard which are just simply going to be more flexible, run on more hardware and have more places to go. Free virtualisation is just going to be more widespread than a platform very much tied to Windows. Additionally, still charging for ESXi was pointless for VMware since it didn't do very much, and it's there to provide people with an upgrade path to ESX.

Having said all that, KVM is still a work-in-progress. ESXi is freely available, we can install it in no time and if it's any good maybe we'll just pay for ESX? Maybe we'll just sweat the hardware we have now on VMware Server and then just wait for KVM? If you can use virtualisation properly, sweat your hardware and don't buy into hype then the cost savings and the added functionality you can have can be quite large.

Getting apcupsd Working on CentOS, and Probably Elsewhere

2008-04-06T21:36:00.003+01:00

In order to provide proper resilience for any server, it needs to react well in power outages. This means using a UPS (Universal Power Supply) and shutting down gracefully when all battery power has been exhausted, coming back up automatically when power is restored.

In order to do this we need an application that is able to communicate with the UPS, detect that there is a power outage and when the batteries have run down to a certain level, gracefully shut the system down. There is a piece of open source software called apcupsd that does this. Download the RPM for Enterprise Linux 5 from here.

Install via the normal rpm -ivh method. It’s also possible to get apcupsd through the DAG extended repository for CentOS and Red Hat, via RPMforge. As this is a one off package, we’ll just install it manually.

However, if you’re installing on a system that uses LVM, or you have a /usr partition that is separate from / then some further work will be needed. Make sure that your / partition is not on LVM. When you test shut down the machine you will notice that apcupsd does not power off the UPS correctly because it cannot find certain components – namely libnetsnmp.so.10, libz.so.1 and the wall executable. We can fix this by making copies to the /lib64 directory (if you run a 64-bit system):

cp /usr/lib64/libnetsnmp.so.10 /lib64
cp /usr/lib64/libz.so.1.2.3 /lib64
ln -s /lib64/libnetsnmp.so.10.0.1 /lib64/libnetsnmp.so.10
ln -s /lib64/libz.so.1.2.3 /lib64/libz.so.1

In addition, the wall executable cannot be found, as it resides in /usr/bin. If you do a check on the dependencies for wall you will see the following:

ldd /usr/bin/wall
libc.so.6 => /lib64/libc.so.6 (0×0000003092200000)
/lib64/ld-linux-x86-64.so.2 (0×0000555555554000)

This is fine, and shows the dependencies are there in the / directory. All we need to do is make a copy of wall to the root bin directory:

cp /usr/bin/wall /bin/span>

apcupsd will now have all the dependencies it needs when shutting down the machine, and when shutting down the UPS when all but the / partition has been unmounted.

PostgreSQL Versus...........Everyone Else

2008-02-19T14:17:00.004Z

Recently, I have been pushing for us to move our databases for our web-based systems from MySQL to Postgres. This has finally been happening with very little trouble or drama, although many of our systems are now based on Rails and ActiveRecord so the change is transparent.

Things have finally got to a head where the features MySQL are adding are already there in Postgres, MySQL isn't quite as stable as we though it would be for a database system we will leave unattended for months at a time and the advantage of performance that MySQL supposedly once had has all but evaporated. From where I sit, it's becoming much easier for PostgreSQL to work away on getting their many features more efficient than it is for MySQL to add some of the features it lacks reliably. This isn't really a solid reason, but now that MySQL has been taken over by Sun I'm also not sure of their future direction. I've seen first hand how Sun have managed takeovers before, particularly in the case of Cobalt, and it doesn't inspire confidence. Sun always seem to pay over the odds for companies that don't really fit well with them or their current customers and market.

Additionally, I'm also pushing (begging on occasion) for one of our major clients (no names!) to update a system from using Access..............to just about anything else. They're not swallowing this at the moment because the current system works just about OK with the hacks we've put into it (we only keep a few days of the most recent data in the main database and then back up to another) and we get the odd corruption. This is a hangover from how the system was initially developed via another third-party. MTS was going to be used, and this was then dispensed with as they discovered they were trying to store stateful objects in it and the thing died, and then they had all the clients connecting directly to the Access database over a network share as a last resort. Surprisingly, with a few clients this seems to work, although Access 97 seems to be far better at concurrent connections than any later versions. The performance, as you can imagine, is terrible and getting meaningful data out of these databases ,which are kept on different sites, is a nightmare.

I wouldn't care, but this is a financial system! It just shows how temporary hacks that just about work turn into permanent solutions. Postgres would be an absolutely ideal replacement for this, would increase performance markedly, we'd have no corruption trouble, we could introduce proper transactions which would help enormously, we wouldn't need to back up data into another Access database (some of which are now creaking under the stress of their size and historical data), it's far easier to get access to the databases over a leased line with limited bandwidth, it would make support and new systems development far easier for us - and then there's the cost. The rest of the organisation pretty much uses Oracle, and installing and configuring Oracle at all these remote sites would be a really significant cost in terms of licensing and support for the Oracle DBAs.

I've read a lot of arguments about how Postgres is not suitable as a replacement for Oracle, but for all but the really, really, really high end features that pretty much no one uses, it's actually an improvement. I laughed at the comments in that article about editing the pg_hba.conf file being unintuitive (most things in Oracle are unintuitive) and crashing under extreme load. The first time I saw Oracle 9i and 10g start on a Windows box I was shocked. It took a full two or three minutes for it to come up, and that's when there was no data running on it yet. You needed a good gigabyte of memory on the server before it did anything. In contrast, the Windows ports of Postgres and MySQL start up in no time and take up far less resources than any Oracle installation.

I've been really impressed with how much Postgres has improved through the 8.x cycle, and the performance ideas and improvements that has come through such as asynchronous commit, autovacuum is now on by default in 8.3, full text search, lots of procedural language support and a native Windows port that is now truly excellent. These kind of steady improvements increase my confidence in Postgres still further.

Installing and Configuring Fedora Directory Server 1.1

2008-01-21T12:10:00.008Z

Although there is some good documentation on Fedora Directory Server, the route I took to getting it installed was a sharp departure. If you simply install fedora-ds as they say on the installation page then you’ll get a whole lot of unneeded dependencies, such as the console GUI, which aren’t necessary since I’m running a non-GUI server. Firstly, you need to upgrade to CentOS 5.1 by doing a yum upgrade:

yum upgrade

If this pulls in the kernel then remember to configure the divider=10 option we have in this kernel within the /boo/grub/grub.conf file to avoid virtual machines taking up excessive CPU time: http://bugs.centos.org/view.php?id=2189.

Then look at the Enterprise 5 instructions on the FDS project page at http://directory.fedoraproject.org/wiki/Download#Enterprise_Linux_5. We’re going to take a detour from this set of instructions though.

Type the following:

yum install svrcore mozldap perl-Mozilla-LDAP

Set up the system to install Fedora 6 packages:

rpm—import http://download.fedora.redhat.com/pub/fedora/linux/core/6/i386/os/RPM-GPG-KEY-fedora
rpm—import http://download.fedora.redhat.com/pub/fedora/linux/extras/RPM-GPG-KEY-Fedora-Extras

We now need to configure our system to use Sun’s Java, rather than the GCJ used within Red Hat. Read and review the steps on the http://jpackage.org installation page. Download the latest JDK (development kit) Java from http://java.sun.com. Download and install this like so:

wget -O jdk-1_5_0_14-linux-i586-rpm.bin http://192.18.108.147/ECom/EComTicketServlet/BEGIN00F9CA7189084FD8F5239E020E63EC2A/-2147483648/2545881171/1/869090/868934/2545881171/2ts+/westCoastFSEND/jdk-1.5.0_14-oth-JPR/jdk-1.5.0_14-oth-JPR:4/jdk-1_5_0_14-linux-i586-rpm.bin

Install the RPM:

rpm -ivh jdk-1_5_0_14-linux-i586-rpm.bin
Note that the wget -O trick becomes necessary here because of the long filename, which wget doesn't get.

Install the JPackage utilities:

yum install jpackage-utils

Now we need to get the jpackage-compat installation as per the instructions on the JPackage installation page. Grab the matching RPM file from the java-sun-compat download page and get the matching RPM for the JDK you’ve just installed. If you have JDK 1.5 Update 14 then get java-1.5.0-sun-compat-1.5.0.14-1jpp.i586.rpm.

wget ftp://jpackage.hmdc.harvard.edu/JPackage/1.7/generic/RPMS.non-free/java-1.5.0-sun-compat-1.5.0.14-1jpp.i586.rpm

rpm -ivh java-1.5.0-sun-compat-1.5.0.14-1jpp.i586.rpm

We now have JPackage installed and ready for us. We can now go back to the FDS install and install what we need. The above allows the package jss to be installed, or we can use JPackage’s jss should we need to (I didn’t) via adding a yum repository from their download page.

Install admin-util and jss from Fedora Core 6:

rpm -ivh http://download.fedora.redhat.com/pub/fedora/linux/extras/6/i386/adminutil-1.1.5-1.fc6.i386.rpm

rpm -ivh http://download.fedora.redhat.com/pub/fedora/linux/extras/6/i386/jss-4.2.5-1.fc6.i386.rpm

We then need to add the FDS repositories to our system:

cd /etc/yum.repos.d

wget http://directory.fedoraproject.org/sources/idmcommon.repo

wget http://directory.fedoraproject.org/sources/dirsrv.repo

Now, the FDS project page isn’t completely forthcoming about the individual packages needed to get a base directory and administration server going, but they can be found on the FHS packaging page. We need fedora-ds-base, which gives us the base directory server, fedora-ds-admin, which gives us the administration server, fedora-ds-console and fedora-admin-console. The last two provide jars necessary for remote administration.
yum install fedora-ds-base
yum install fedora-ds-admin
yum install fedora-ds-console
yum install fedora-admin-console

Installing VMware Server

2007-11-09T12:00:00.000Z

Firstly, we need to install the gcc compiler so VMware can compile its kernel modules:

yum install gcc

To install VMware, save the RPM from the VMware site in an appropriate location, and run the following to install it:

rpm -ivh VMware-server-1.0.4-56528.i386.rpm

Then run:

/usr/bin/vmware-config.pl

which runs the VMware configuration utility. You’ll see this:

Making sure services for VMware Server are stopped.

Stopping VMware services:
Virtual machine monitor [ OK ]

You must read and accept the End User License Agreement to continue. Press enter to display it.

Do you accept? (yes/no)

Press enter to display the license, then press ‘q’ to quit and then type in yes to accept the license.

In which directory do you want to install the mime type icons?
[/usr/share/icons]

Accept the default and press enter.

The path ”/usr/share/icons” does not exist currently. This program is going to create it, including needed parent directories. Is this what you want? [yes]

Press enter.

What directory contains your desktop menu entry files? These files have a .desktop file extension. [/usr/share/applications]

Press enter.

In which directory do you want to install the application’s icon? [/usr/share/pixmaps]

Press enter.

Trying to find a suitable vmmon module for your running kernel.

None of the pre-built vmmon modules for VMware Server is suitable for your running kernel. Do you want this program to try to build the vmmon module for your system (you need to have a C compiler installed on your system)? [yes]

Press enter. This is where we need GCC to compile the vmware kernel module for our running kernel.

What is the location of the directory of C header files that match your running kernel? [/lib/modules/2.6.18-8.1.14.el5/build/include]

Press enter. This is where the kernel headers and source are to enable compilation.

Extracting the sources of the vmmon module.

Building the vmmon module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmmon-only’
make -C /lib/modules/2.6.18-8.1.14.el5/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make1: Entering directory `/usr/src/kernels/2.6.18-8.1.14.el5-x86_64’
CC [M] /tmp/vmware-config0/vmmon-only/linux/driver.o CC [M] /tmp/vmware-config0/vmmon-only/linux/hostif.o CC [M] /tmp/vmware-config0/vmmon-only/common/cpuid.o CC [M] /tmp/vmware-config0/vmmon-only/common/hash.o CC [M] /tmp/vmware-config0/vmmon-only/common/memtrack.o CC [M] /tmp/vmware-config0/vmmon-only/common/phystrack.o CC [M] /tmp/vmware-config0/vmmon-only/common/task.o CC [M] /tmp/vmware-config0/vmmon-only/common/vmx86.o CC [M] /tmp/vmware-config0/vmmon-only/vmcore/moduleloop.o LD [M] /tmp/vmware-config0/vmmon-only/vmmon.o Building modules, stage 2. MODPOST CC /tmp/vmware-config0/vmmon-only/vmmon.mod.o LD [M] /tmp/vmware-config0/vmmon-only/vmmon.ko
make1: Leaving directory `/usr/src/kernels/2.6.18-8.1.14.el5-x86_64’
cp -f vmmon.ko ./../vmmon.o
make: Leaving directory `/tmp/vmware-config0/vmmon-only’
The module loads perfectly in the running kernel.

This should now have compiled OK.

Do you want networking for your virtual machines? (yes/no/help) [yes]

Press enter for yes. We do want networking in our VMs.

Configuring a bridged network for vmnet0.

Your computer has multiple ethernet network interfaces available: eth0, eth1. Which one do you want to bridge to vmnet0? [eth0]

Because we have two ethernet ports on the server, it will assign virtual interfaces to both of them. Press enter here, since we want vmnet0 to map to eth0.

The following bridged networks have been defined:

. vmnet0 is bridged to eth0

Do you wish to configure another bridged network? (yes/no) [no]

The following bridged networks have been defined:

. vmnet0 is bridged to eth0

Do you wish to configure another bridged network? (yes/no) [no] yes

Type yes, because we want to assign eth1.

Configuring a bridged network for vmnet2.

The following bridged networks have been defined:

. vmnet0 is bridged to eth0
. vmnet2 is bridged to eth1

All your ethernet interfaces are already bridged.

Do you want to be able to use NAT networking in your virtual machines? (yes/no) [yes]

Type no here. This just means that it will create a virtual network sitting behind NAT, which we don’t need.

Do you want to be able to use host-only networking in your virtual machines? [no]

Type yes here, because this creates a virtual internal network that we can put our VMs on, separate from the physical network.

Configuring a host-only network for vmnet1.

Do you want this program to probe for an unused private subnet? (yes/no/help) [yes]

Press enter here. This will just assign an unused network address to the virtual network.

Probing for an unused private subnet (this can take some time)...

The subnet 172.16.120.0/255.255.255.0 appears to be unused.

The following host-only networks have been defined:

. vmnet1 is a host-only network on private subnet 172.16.120.0.

Do you wish to configure another host-only network? (yes/no) [no]

Press enter for no here.

Extracting the sources of the vmnet module.

Building the vmnet module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmnet-only’
make -C /lib/modules/2.6.18-8.1.14.el5/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make¹: Entering directory `/usr/src/kernels/2.6.18-8.1.14.el5-x86_64’ CC [M] /tmp/vmware-config0/vmnet-only/driver.o CC [M] /tmp/vmware-config0/vmnet-only/hub.o CC [M] /tmp/vmware-config0/vmnet-only/userif.o CC [M] /tmp/vmware-config0/vmnet-only/netif.o CC [M] /tmp/vmware-config0/vmnet-only/bridge.o CC [M] /tmp/vmware-config0/vmnet-only/procfs.o CC [M] /tmp/vmware-config0/vmnet-only/smac_compat.o SHIPPED /tmp/vmware-config0/vmnet-only/smac_linux.x86_64.o LD [M] /tmp/vmware-config0/vmnet-only/vmnet.o Building modules, stage 2. MODPOST
WARNING: could not find /tmp/vmware-config0/vmnet-only/.smac_linux.x86_64.o.cmd for /tmp/vmware-config0/vmnet-only/smac_linux.x86_64.o CC /tmp/vmware-config0/vmnet-only/vmnet.mod.o LD [M] /tmp/vmware-config0/vmnet-only/vmnet.ko
make¹: Leaving directory `/usr/src/kernels/2.6.18-8.1.14.el5-x86_64’
cp -f vmnet.ko ./../vmnet.o
make: Leaving directory `/tmp/vmware-config0/vmnet-only’
The module loads perfectly in the running kernel.

Please specify a port for remote console connections to use

There may be a warning here about port 902 already being in use. Press enter, and if there is a warning that port 902 is in use just enter 902 anyway.

In which directory do you want to keep your virtual machine files?
[/var/lib/vmware/Virtual Machines]

Type in the location /vms. This is where our VM partition is.

Please enter your 20-character serial number.

Type XXXXX-XXXXX-XXXXX-XXXXX or ‘Enter’ to cancel:

For this, just copy and past one of the serial numbers in. If you registered, you should have got a bunch e-mailed to you.

Starting VMware services:

Virtual machine monitor [ OK ]

Virtual ethernet [ OK ]

Bridged networking on /dev/vmnet0 [ OK ]

Host-only networking on /dev/vmnet1 (background) [ OK ]

Bridged networking on /dev/vmnet2 [ OK ]

The configuration of VMware Server 1.0.4 build-56528 for Linux for this running kernel completed successfully.

All done.

Satisfying Dependencies With VMware Server

2007-11-09T11:15:00.001Z

Installing VMware Server on a server with no graphical libraries installed is a peculiar experience, because even though no GUI is running VMware depends on a lot of X libraries. CentOS also installs a fair bit by default that it shouldn’t. Running /usr/bin/vmware-config.pl performs a check, and if dependencies aren’t fulfilled then we get this:

The correct version of one or more libraries needed to run VMware Server may be missing. This is the output of ldd /usr/bin/vmware:

linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/libm.so.6 (0×0034a000)
libdl.so.2 => /lib/libdl.so.2 (0×00305000)
libpthread.so.0 => /lib/libpthread.so.0 (0×0038b000)
libX11.so.6 => not found
libXtst.so.6 => not found
libXext.so.6 => not found
libXt.so.6 => not found
libICE.so.6 => /usr/lib/libICE.so.6 (0xf7fdb000)
libSM.so.6 => /usr/lib/libSM.so.6 (0xf7fd2000)#
libXrender.so.1 => not found
libz.so.1 => /usr/lib/libz.so.1 (0×0030b000)
libc.so.6 => /lib/libc.so.6 (0×001c6000)
/lib/ld-linux.so.2 (0×001a9000)

This program cannot tell for sure, but you may need to upgrade libc5 to glibc before you can run VMware Server.

Firstly, do a:

yum remove libXi libXtst libXt libXrender libXext libX11

from the command line. This removes all X libraries pretty much from the system, along with a ton of stuff that CentOS (Red Hat) sees fit to install along with it that we don’t need like Pango and Cairo, and gives us a reasonably clean slate. Next, install the following needed libraries from the command line:

yum install libX11.i386 libXrender.i386 libXt.i386 libXtst.i386 libXi.i386

This is a 64-bit system, but VMware is a compiled 32-bit application, hence the i386 extensions. This should then enable the /usr/bin/vmware-config.pl to successfully run. Additional packages needed are:

yum install xinetd

xinetd is a package that allows servers to register themselves with the system, and listen on ports without the server itself having to be running in the background. xinetd hands off to the server process and it then starts running.

AMD PowerNow and Keeping Time in a VMware Virtual Machine........

2007-09-17T17:46:00.000+01:00

For the past few days I've been scratching my head like anything at a time problem I've been having with running CentOS 5 in a VMware Server virtual machine. Time issues within virtual machines are certainly not uncommon as a bit of Googling should tell you, but rather than gaining time as is usually the case, my test VM has been losing time, and it has been impossible to see a pattern. I've used clocksource=pit, pmtmr, noapic, nolapic and nosmp as kernel options in the VM and nothing has made it any better. VMware tools is installed and working. I did add the following to /etc/vmware/config:

host.cpukHz = 2000000
host.noTSC = TRUE
ptsc.noTSC = TRUE

host.cpukHz is the speed of my processor (a couple of Opterons), which corresponds to 2GHz. This was needed because of the CPU frequency scaling within the motherboard. However, even this didn't make any difference. This was eventually solved by turning off AMD PowerNow in the host machine's BIOS (the real hardware). I took out all the kernel options I had put into the VM grub config file above, and it still worked perfectly. Time is now kept absolutely perfectly within the VMs to the second, with VMware tools installed and functioning.

Keeping the above options in the VMware config file seem to have the effect of keeping the VM's clock a few seconds behind so that the VMware Tools can synchronise properly, and stopping the VM clock from gaining a few seconds. Apparently, VMware tools can do nothing to synchronise the time if the VM's clock is fast.

Software or Hardware RAID Part 3: Conclusion - Hardware RAID Where Possible

2007-06-05T12:36:00.000+01:00

A while ago I agonised over what an awful lot of people seem to agonise over when building and choosing servers and trying to judge the best options for storage, and came to the conclusion at the time that software RAID was going to be the best way because of the hardware we had available and the need for hardware RAID cards to really be in a PCI-X slot. Yes, you can use PCI slots, but really, you're limited by the I/O bandwidth. Like I said at the time, if you can find the right hardware then hardware RAID is preferable - and that's certainly the conclusion I have come to now. If you don't have the right board, or only have fake RAID available to you, please, just use software RAID. You will have pain, I promise you. You may have to forget hotswapping your disks though, because you really need your disk controller to react in the right way when a disk is disconnected and a new one added otherwise bad things will happen.

Well, the motherboard we were going to use just didn't happen for us for various reasons, and so I specifically looked at boards that would accommodate PCI-X slots. The availability of dual processor, server-based motherboards in the UK seems to be surprisingly spartan, with many dual CPU boards given over to ridiculous dual SLI graphics card workstation requirements. Needless to say, I don't need any of that. I eventually plumped for a Supermicro H8DAE after doing some reading around and reading some horror stories regarding an equivalent Gigabyte motherboard and PCI domains. Having got it all up and running now with CentOS, it's a very nice board with nothing more than I need with dual LAN, 4 PCI-X slots, 2 PCI slots, built-in graphics and IPMI support. Other bits and bobs in the new machine include:

The hardware RAID card available to us, at a very reasonable price, was an Adaptec 2420SA four port SATA card. It's a very nice card having got it up and running on Linux. The one thing I was worried about were management tools, because the OS will simply not see or know if a RAID array has a problem - it just sees a disk, remember. The Adaptec Storage Manager is perfectly adequate, with e-mail notifications, and I must commend Adaptec on their improved Linux support. aacraid is directly in the kernel now, and I got the whole thing up and running with no yucky driver installs. Yay! We also have a lot more scope for rearranging our storage and adding larger hard drives later on, without any downtime if we play our cards right. Needless to say, this does not mean you shouldn't have a backup system!
Chenbro server case. Solid little thing that is lockable and has a good SATA cage for RAID purposes.
2 x AMD 146 Opterons. There is no Pacifica virtualisation support with these, or with the motherboard I have, but quite frankly, so what? I'm rather sceptical about the support and quality for this for the foreseeable furture, and VMware will see us right. We already have some existing VMs anyway.
Seasonic S12 600W PSU. A top quality PSU that is absolutely silent and totally cool (I'm talking about its temperature there). You couldn't buy any server with a better PSU than this. Spot on. Just be aware of a caveat with PSUs - make sure you get an EPS-SSI compatible PSU if you need it, otherwise some boards like many Tyans will just not power up. An additional CPU power connector isn't necessarily enough.
2 x Xilence Blade coolers. The ones pictured there seem to have an aluminium block, but mine have a copper one. There's a story behind this :-). When we ordered our motherboard the seller didn't pack the right CPU fitments for a H8DAE and packed the wrong I/O backpanel. They packed those for a H8DAE-2 instead (I got some free SATA cables though ;-)). Our seller refused to send the right stuff out even after it was explained (tossers), and rather than go through the hassle of changing the board I got on to Supermicro support who kindly sent out the right I/O backpanel. I then took the opportunity to get some CPU coolers that were more silent, cooler and better for dust and cleaning than the stock ones I had. I also needed a cooler that had its own motherboard backplate. The flower coolers were too big for the board, so I went for the Xilence coolers. At first I thought they weren't going to fit, so I turned one cooler around one way and the other in the opposite direction. The RAID cage was a little close for comfort to one cooler, but doesn't seem to have made any difference to airflow or cooling having monitored it. Both CPUs maintain an idle temperature of about 40 degrees celsius, with one CPU one or two degrees out, as it always seems to be in dual CPU systems.
2 x 1 gigabyte sticks of EEC registered DDR. Standard stuff. Just make sure you get EEC registered RAM for these boards.
4 x Samsung 200 gigabyte drives. You do get warned by Adaptec that you should use, supposedly, enterprise class disk drives because of the way a normal desktop drive retries on bad sectors. That article does say this can result in data loss, but I'm not entirely sure how, and I'm not sure if this is really relevant or whether it's just a way of selling more expensive hard drives.

All in all, a tidy little machine, and but for the system fan which is necessary considering I have two CPUs and four disks in an enclosed RAID cage in there (albeit with its own fan), it would be almost completely silent. I was quite impressed.

Most of my decisions on what operating system to use, and what flavour of Linux to use, centred around what kind of aacraid support it had and whether I could easily get the Adaptec Storage Manager up and running or not. I mean, if the underlying disk storage is flaky and I can't manage the RAID array and get notifications when something goes wrong, what's the point? Everything else will fall to pieces around it. As it turned out, Adaptec has RPMs built for Red Hat and Suse specifically, but it isn't a whole lot of work to get it up and running on any 32 or 64-bit Linux system. The right dependencies are all there, which is the 32 or 64-bit libXP library as far as I can work out.

So I went for a RHEL (Red Hat Enterprise Linux) compatible system, which for us is CentOS 5. We're not running too much on the base system here, as much of our functionality will be on further virtual machines. CentOS has given me the option of using driver disks from Adaptec and other manufacturers if I so wish (hasn't been necessary as it turns out), packages such as Adaptec Storage Manager I can just install without tinkering, a bit more confidence in 64-bit and 32-bit backwards compatible support, better Xen support if I want it and much more predictable updates to the system. There are some things about CentOS, and hence RHEL, that I don't like but I'll get on to them at another time.

Conclusion: Go for hardware RAID every time, but you must find yourself the right controller. If you can't, you don't have the right hardware and you only have, God forbid, fake RAID then go the software RAID route.

OpenXML as an ISO Standard: What Exactly is the Problem?

2007-05-26T20:48:00.000+01:00

The primary reason why I started to talk about OpenXML more recently (OOXML, Office Open XML - whatever), and make a few comments here and there about it, is simply because I do not understand the rationale behind it in any way shape or form. That's pretty much it in a nutshell. Every comment, article and blog post I have ever read about it (and I've read quite a bit of the actual specification now - heavy going) has basically been trying to justify its existence. It's an awful lot of talk about..........nothing in particular. Some of the reasons have been quite flimsy to say the least, and the main one I've gathered out of it all is this:

OpenXML is about Microsoft documenting their Office formats.

Well, that's absolutely fine. If Microsoft want to document their formats and then submit them to some standards body then that's absolutely fine. No problem. That really is entirely up to them. Quite why Microsoft created another format to do that when it's transparently obvious that OpenXML is simply a translation of the old binary doc format and Microsoft Office features into a pseudo-XML format, I don't know. This is logically the case because Microsoft claims they created OpenXML as being backwards compatible. It isn't backwards compatible in terms of the files actually created, but it supposedly is backwards compatible with the Microsoft Office features and quirks within it because they can't be bothered to do a proper conversion.

Why not just document the binary format instead? Why not just create an entirely new and sensible format, eminently suitable for ISO submission, that referenced no previous versions of Microsoft Office or Windows technology and where older Microsoft Office documents could be simply converted, which is the job of Microsoft Office anyway? The latter question renders the issue of backwards compatibility a pretty void reason to me, but many people avoid that question.

However, the issue of it being an ISO standard is a different matter entirely, simply because any ISO standard must meet the aims and goals of the ISO. The biggest of those aims is about free trade and using standards to effectively communicate internationally, between governments, companies and organisations of various sizes:

http://www.iso.org/iso/en/aboutiso/introduction/index.html#six
http://www.iso.org/iso/en/aboutiso/introduction/index.html#eight

As you can probably imagine, being able to actually implement a standard in full, unambiguously, is of vital importance in an international standard. To create some parity with other industries and ISO standards, imagine just how worthless ISO 9000 would be if it transpired that only one of company could truly implement it - and they sold quality systems. I would imagine that would cause quite a big fricking fuss considering that ISO 9000 has become a requirement for many fields of work over the years. Having to buy a vendor's product to adhere to strictly it properly is not exactly on the agenda. It's expensive enough as it is.

This is why I had a bit of a go at Rick Jellife's blog entry, because in reviewing a proposed ISO standard this is absolutely all that matters. It's very important, no? Ultimately, a standard is about communication, and if the standard miscommunicates in a big, big way then you have a big, big problem - because it's useless. So, being able to implement a standard, and OpenXML should it want to be an ISO standard, is of absolutely vital, primary and fundamental importance. If it can't be proven to be fully implemented beyond one platform and office suite, then given that it is a standard that documents the format of one company's office suite, the ISO has then effectively given them a monopoly. Not good, eh, and not what all the talk about open document formats and the whole issue of ODF was about?

This whole issue is quite apart from any duplication of ODF by OpenXML that many people come up with. These debates usually end with accusations that many ODF supporters want there to be only one ISO office document standard, which would bizarrely create a monopoly for IBM...........somehow. I seem to have missed out several steps there. The IBM monopoly situation is not an issue if ODF is truly and independently implementable, and their own applications stick to it. It is what ODF itself is there to avoid. Yes, IBM could make their own incompatible extensions, but, whatever they are not specified in the standard and no one else is guaranteed to be able to do anything with them. There's simply no good reason for any specific extensions to be referenced absolutely anywhere within ODF - so they're not. You can just do it.

I know Groklaw have had a list of objections up for some time, and I've never seen a Microsoft web page with a point-by-point quote and rebuttal. Many people have tried quite badly to say why these don't matter, including trying to claim that the Microsoft and Office specific extensions and references are not the main part of the specification. Well, the point is, they're in there and the will be used by Microsoft Office, so claiming that no one else needs to use them is rather pointless. If a document cannot be created in Microsoft Office and sent to another system that has another implementation of OpenXML then as an ISO standard it is worthless. I know people have went tit-for-tat over this, so I'll just quote one part of the OpenXML specification and ask a question:

2.15.3.6 autoSpaceLikeWord95 (Emulate Word 95 Full-Width Character Spacing)

This element specifies that applications shall emulate the behavior of a previously existing word processing application (Microsoft Word 95) when determining the spacing between full-width East Asian characters in a document's content.

Yes, I do know this disclaimer is below it:

[Guidance: To faithfully replicate this behavior, applications must imitate the behavior of that application, which involves many possible behaviors and cannot be faithfully placed into narrative for this Office Open XML Standard. If applications wish to match this behavior, they must utilize and duplicate the output of those applications. It is recommended that applications not intentionally replicate this behavior as it was deprecated due to issues with its output, and is maintained only for compatibility with existing documents from that application. end guidance]

I deliberately chose this section of the markup language reference because of the East Asian character reference, in view of its proposed international standard status, but there are many such sections like this within the reference document pertaining to some specific behaviour within a particular version of Word or WordPerfect.

The main, and only really, reason for being, that I've managed to pick up from Microsoft and various people, is that OpenXML exists for backwards compatibility reasons with, as they put it, the billions of Office documents already out there. Presumably, this is why sections such as this exist - for backwards compatibility reasons. Presumably, when older Microsoft Office documents are saved in the shiny new OpenXML format, elements such as this are used to preserve formatting. The only problem is that this behaviour is not specified. Anywhere. I can't implement an OpenXML application that will convert older Microsoft Office binary documents, and worse, I cannot even translate a modern OpenXML file simply because I don't know what this behaviour is. There are a great many sections such as this.

The real problem here is that incompatibilities, quirks, caveats and pitfalls in individual applications are simply being transferred from one format, the old binary doc format, to another with no appreciable benefit to anyone - least of all users. These pitfalls will continue to be inherited in successive versions of the document. The net effect is that anyone implementing OpenXML will have to deal with this inherited behaviour, despite Microsoft's disclaimer that applications should not use it. OpenXML, afterall, was invented for backwards compatibility reasons, remember? It would be interesting to see if Office 2007 still uses many of these so-called deprecated elements.

That's some background as to why it's a problem, but it's all beside the point really. The fact is, until I get some form of detailed specification for the behaviour that I need to handle then I cannot make an independent implementation of OpenXML. So, the question I wanted to ask is can anyone, anywhere, tell me where to find the behaviour I need to implement section 2.15.3.6? This could be in the specification documents, or as a reference. It doesn't really matter. If you read nothing else here, answer this question.

In a concise nutshell, if no one can answer that question then I certainly cannot implement anything within the OpenXML specification for backwards compatibility, OpenXML's reason for being remember, and I simply cannot make an independent implementation of OpenXML for myself and others as a result. This goes absolutely contrary to the ISO's free trade and communication aims and goals for its standards, this being the case. There you have it.

Right-Click Popup Menus on Listviews in Visual Basic

2007-05-25T16:19:00.000+01:00

I ended up going back to doing some Visual Basic (classic that is, not the abomination of VB.Net) the other day for a long-standing client we have, and found myself having to make a popup menu appear on an item in a listview. I can remember managing to do this in some really easy way some time back, but just couldn't remember how. It's just one of those little nuggets of useful information you quickly forget. I looked around on the internet, and there were some pretty convoluted ways of doing this through pretty awful looking Win32 declare functions and constants - none of which worked 100% - but nothing like my original method as I remembered it. So, I checked out the code of another project I'd worked on where I'd done this, and voila. There it was:

Private Sub lvwListView_MouseDown(Index as Integer, Button as Integer, Shift as Integer, X as Single, Y as Single)

Dim lstItem as ListItem

If Button = vbRightButton Then

Set lstItem = lvwListView.HitTest(X, Y)

If Not(lstItem Is Nothing) Then

Set lvwListView.SelectedItem = lstItem
PopupMenu mnuPopup

End If

End If

End Sub

Really easy, no? The only way it could be easier is if the event had the listview reference passed in as a parameter. Let's be honest, it's a fairly obvious thing to want to do. The events in listviews and other list widgets give an X and Y coordinate as to where the user has clicked. We then need to translate that somehow into the listitem that the user had actually clicked on. We do that through HitTest, which returns a relevant listitem reference. The caveat here is that if the user didn't click on a valid listitem (i.e. a listitem that isn't populated) then Nothing will be returned. This needs to be caught otherwise bad things will happen. We check to see if HitTest has returned something and then we can set the selected item to this listitem to help the user and use PopupMenu to display a hidden menu. We could then flesh this out further so that the mnuPopup event knows which listitem has been selected, and then functionality can be added accordingly.

Rick Jelliffe: What is This Guy Actually Talking About?

2007-05-13T15:09:00.000+01:00

Rick Jellife became somewhat famous in the whole Microsoft paying to make OpenXML look better controversy. I mean, I don't have a problem with that per se, but a lot of language in a lot of his blog entries have been so bizarre to say the least that I find it slightly astonishing he's allowed to post and associate himself with O'Reilly. The first sentence of that blog entry goes like this:

My first computer was a Mac Plus. Loved it. My second computer was an AT&T Unix PC running System V. Loved it long time. My third computer was a Sparc running Solaris or SunOS. Loved it. At work I run Linux, Open Office, Firefox, Eclipse, etc. No drama. For the last six years I have been running a little company making Java programs...The only time I use Microsoft products is on my laptop at home...

I worry about that for one reason. Right from the off he's going out of his way to tell us how favourable he is to non-Microsoft systems and applications, and how little he actually uses Microsoft software. Why? Of what relevance does that have to the subject of him potentially getting paid by Microsoft to try and correct some inaccuracies surrounding OpenXML? Already he's going defensive for reasons that are best know to him.

Since then, there have been a number of blog entries that are quite frankly laughably immature at best regarding why OpenXML should be an ISO standard from someone supposedly quite experienced, why the problems many have pointed out are non-issues, and more worryingly, how everything that goes against this thinking is a personal attack. I'm not entirely sure he understands how ridiculous that makes him look, but there you go. He seems to be carefully vetting comments that are posted to his blog now as well.

The latest blog entry is actually an acknowledgement of the many issues people have raised regarding OpenXML, but he's now unwittingly backtracking on a lot of what he's said previously by moving his own personal goalposts on the criteria for OpenXML making it as an ISO standard and whether many peoples' objections fit those criteria:

When, for example, one side says “Open XML normatively refers to MS’ proprietary WMF” and the other side says “Err, where? Not in the Normative Refences sections”

That's playing fast and loose with what a normative reference is. What on Earth is this? "Oh, it's not really in the standard because it's not in the normative sections?"

If it's in the standard then it's in the standard - no amount of 'Oh, it's not in the normative references section' cuts any kind of ice. It's in the format. ODF allows for all sorts of extensions, which could be proprietary, but they are up to someone to add on - they are not specified in the format itself. They're not even informative references either - ODF just tells you there are certain areas where you can store what you like, but as for whether anyone will be able to do anything with them, you're on your own. It's just a ludicrous hair splitting exercise. I don't know what his definition of a normative reference is either:

Broadly speaking, a normative reference specifies a document that must be read to fully understand or implement the subject matter. An exact and precise definition of what is (and is not) a normative reference has proven challenging in practice, as the details and implications can be subtle. Moreover, whether a reference needs to be normative can depend on the context in which a particular RFC is being published in the first place.

http://rfc.net/rfc3967.html

That's about the most reasonable definition I could find.

impartial reviewers will note that there is some kind of concern but that the actual issue raised does is not a problem.

In what way would an impartial reviewer note that, and back that up?

Do you see the difference between saying “Open XML should be banned because it uses EMU” and “Open XML should be improved to allow more than EMU”

Again, it's hair splitting to try and justify the way Microsoft are doing things. One can reasonably ask why EMU is there in the first place considering that it meets no ISO definition and is used nowhere else, and it's a legitimate question if the major software product using OpenXML doesn't use anything else. Why expand it at all if it's essentially useless? Just use units that are correctly defined and recognised and well used. How hard is that? Afterall, how the standard will be used in practice, as well as fair trade concerns, is a legitimate concern as per the ISO's aims and can't be waved away.

I think Open XML is OK in this regard: it allows Word macros, Java, and other scripts, but these are not required and IIRC partitioned.

Therein lies the paradox. If it allows Word macros, in the absence of anything else (particularly what Microsoft Office uses), then it simply doesn't meet this definition. The standard also references an awful lot of technology that is simply not proven to be implementable outside of Microsoft Office and Windows - it is up to Microsoft to prove that. We also have a lot of extremely amateurish and bogus statements within the format it just isn't believable:

This element specifies that applications shall emulate the behavior of a previously existing word processing application (Microsoft Word 95) when determining the spacing between full-width East Asian characters in a document's content....To faithfully replicate this behavior, applications must imitate the behavior of that application, which involves many possible behaviors and cannot be faithfully placed into narrative for this Office Open XML Standard.

OpenXML quite clearly does not meet this definition, either in the standard itself or in the one implementation we have today, so what he's doing here is stirring. "I think OpenXML is OK in this regard" just isn't good enough. Why is OpenXML OK in this regard, because outside of Microsoft Office OpenXML hasn't been implemented, and certainly not in full? In contrast, ODF was implemented for many years before submission for being an ISO standard was done and is continuing to be so. That's just a fact.

In the absense of these kinds of principles, what we have is a line of argument that reduces to “Microsoft is bad, therefore anything they do or make is bad”

No, because that's just a fanboy, activist comment in itself. Let's just stick to what's actually wrong with OpenXML, OK? Quite frankly, he doesn't sound like an impartial reviewer to me.

in this case, abandoning closed, binary formats. Ten years ago, Bill Gates was saying they would be crazy to open up their file formats, now they are doing it.

Could he quit talking about Microsoft for five seconds? Like I said, he doesn't sound like an impartial reviewer, and the above has nothing whatever to do with what's being discussed. What are we all supposed to do? Say "Oh, silly us, Microsoft has definitely turned over a new leaf" and paint over some of the things we see? That's the argument he's using, even if he maybe doesn't realise it.

They're not opening anything. Their open file format still allows for binary blobs to be dropped into it, specified in the standard no less, that are specific to their own software products - Windows and Office. The difference with ODF is that ODF as a format is reasonably reproducible on every platform by any piece of software, and it has proved it for some time now. That statement also proves that Microsoft has a poor track record for this sort of thing, so quite why we think they've changed I can't fathom. Like it or lump it, that's the way it is. The ball is in Microsoft's court - no one elses'.

The future is mix and match.

The future is standards that guarantee future readability of documents with a standard that is proven to be implementable beyond one piece of office software and beyond one platform. Let's not lose sight of that with the stupid 'mix and match' and 'choice' statements Microsoft, and even Rick, come out with. That is the concern of governments and organisations everywhere now, and it is the concern of the ISO through their aims and objectives, certainly in terms of free trade and interchange between different countries, governments, organisations, systems and applications.

The mix and match approach is something Microsoft consistently comes out with - usually with the word choice (http://news.zdnet.co.uk/software/0,1000000121,39287024,00.htm). This is just front-end backwards. The world does not need multiple standards that essentially reproduce each other, but if they're all proven to be fully implementable, OK. The world wants a choice of systems and applications to use and buy that use common interchange formats, not a choice of formats. As it stands, that is not the case - which is why people have been talking about it. It's no coincidence that Microsoft have been the one obstacle consistently in the way of this.

Why does this matter to the ISO? Because it does the reputation of the ISO no good whatsoever if a standard, like OpenXML, became an ISO standard where it transpired that it just wasn't implementable on different platforms by different applications. In effect, what has happened is that the name of the ISO had been used in order to keep a monopoly in place. This is totally contrary to the ISOs aims and objectives.

There's been more bizarre blog entries than this from Rick, and he always seems to come back to a 'personal attack' argument whenever someone queries anything he's written. Quite why the above is so difficult to understand I don't know, and yet, we always seem to come back to the same arguments.

SlimServer 6.5.1 on OpenSuse 10.0

2007-04-16T22:59:00.000+01:00

For quite a while now, I've been using a Squeezebox to share my music in the house from a small server to the Living Room to listen to. It's a great little box to have, and alas, Logitech now seem to have taken them over. Please don't screw it up Logitech (but I know you will).

Having managed to get 6.2.1 installed on my OpenSuse 10.0 server in a roundabout way, as my music collection grew I realised that this version had trouble with albums that were part of a multi-disc set and had their tags set accordingly. As soon as I tried to treat multiple disc albums as a single album, the running SlimServer then helpfully decided to totally screw up many gigabytes' worth of music files and their tags by overwriting them. F$%£&*@ brilliant. Thankfully, well organised person that I am, I have everything backed up at night via a wonderful piece of software called Bacula (more on that soon). Rather than going through all my music files and seeing what had been fecked, I simply did a restore via Bacula, which thankfully worked perfectly.

I then had a long look at what I needed to do to get my multi-disc albums treated properly by the SlimServer. Well, I needed to upgrade to the latest version - 6.5.1 - which has apparently fixed many bugs in this area. Easy, right? Errr, as is often the case with these things, you need to take a deep breath. The previous version I ran, 6.2.1, used SQLite as its back-end database. The new version I wanted to install, 6.5.1, apparently now has MySQL as its default database. I did attempt to get 6.2.1 running with MySQL, but it had some sleep problems with the connection to the database where the SlimServer had complained that the database had suddenly gone away.

To get the ball rolling, I first shut down the running SlimServer and uninstalled it. For me, this was simply a case of renaming /usr/local/slimserver to /usr/local/slimserver.bak to back it up - just in case. I had installed 6.2.1 previously through a tar file, and not via a RPM package. Now download the RPM file from here. Install it through Konqueror and YaST, or if you can be bothered, use rpm at the command line. I can't. Click the 'Install Package With YaST' button and let it do its thing:

Once this is installed, we need to do a bit of tinkering with OpenSuse, because for some bizarre reason the SlimServer RPM is still not Suse friendly - despite a lot of stuff in the changelog about it. The main thing we have to do is to tinker a bit with the startup scripts. In Suse, startup scripts are located in /etc/init.d, but the SlimServer RPM still point the scripts to /etc/rc.d/init.d. The startup script itself is also not Suse friendly. To remedy this download a Suse startup script from here and place this script in /etc/init.d/. I then renamed the shell script to slimserver for no particular reason other than consistency. Make sure that it has execute permissions, otherwise it won't run.

Because I had a previous version of SlimServer installed, I also had a previous slimserver user and group set up - which this RPM installation has promptly went and overwritten. This means the user IDs for the ownership of certain important files are now wrong. We'll have to remedy that. You can of course do this through chown and chgrp, but I'll illustrate doing this through Konqueror. The first important file is the slimserver.conf file in /etc. It is important that the SlimServer is able to write to this file, and since the SlimServer will be running under slimserver, if slimserver can't write to it then we have problems. Open Konqueror and run it as the root user by going to the 'File Manager - Super User mode' in the KMenu. If you can't find it, just do a quick search in the search box. Navigate to /etc and find slimserver.conf. Right-click on it, select properties, select the permissions tab and for ownership, enter slimserver into the user textbox and slimserver into the group textbox. This will allow this file to be written to by the running SlimServer.

Now we come to the database side of things. The SlimServer has its own installation of MySQL, which is version 5, but I already have a MySQL server running on this server. It seemed quite pointless to be running two MySQL instances, so I used this. On OpenSuse 10.0, the MySQL version is 4.1, but SlimServer should run with this fine. Log into Webmin or whatever other MySQL administration tool you use. Create a new database called slimserver, and create a new user called slimserver. When you create this user make sure that you lock down access to it to the localhost (access through the local socket interface), and also create it with no password. This is because of this change in the requirements of a MySQL client. On this Suse system the version of DBD-MySQL does not support the new client authentication, and I just don't have the wherewithall to upgrade it through CPAN or something. Make sure that this slimserver user has full permissions to the slimserver database itself.

Now we'll make some modifications to the /etc/sysconfig/slimserver file. This file tells the SlimServer how to start, and with what settings. My slimserver sysconfig file looks like this:

# Edit this to suit your setup
SLIMSERVER_USER="slimserver"
SLIMSERVER_HOME="/usr/local/slimserver"
SLIMSERVER_CFG="/etc/slimserver.conf"
SLIMSERVER_LOG="/var/log/slimserver/slimserver.log"
SLIMSERVER_CACHE="/var/tmp/slimserver"
SLIMSERVER_ARGS="--daemon --prefsfile=$SLIMSERVER_CFG --logfile=$SLIMSERVER_LOG --cachedir $SLIMSERVER_CACHE"
PID_FILE="/var/run/slimserver.pid"

Check the paths in here, such as the log file and the tmp and cache folders, and make sure that the slimserver user can read and write to them. The best way of doing this is to make sure that the slimserver user owns them. Permissions issues here are often a big reason why the SlimServer can silently fail when you start it up.

Now I was, and we should, be ready to start the slimserver itself. Do a /etc/init.d/slimserver start at the command line and go to http://localhost:9000 or another host in your browser and you should see the SlimServer web interface. If you go into server settings you'll see a music folder in here. This is where you.......specify the folder that holds your music. Do that and then rescan your library.

For you, this might work great. for me it was problematic because of the issue of permissions on my music files. The rescan of the music just silently failed. I had to modify the permissions of my /music folder for the slimserver user. I created a file called acls, and created the following text in it:

user:user1:rwx
user:user2:rwx
user:user3:rwx
user:slimserver:rwx
group::r--
mask::rwx
other::---
default:user:user1:rwx
default:user:user2:rwx
default:user:user3:rwx
default:user:slimserver:rwx
default:group::r--
default:mask::rwx
default:other::---

This is an Access Control List that we are going to apply to our folder. The other users in here just signify other users on the system who need access to these files. These can be deleted. Now, we need to apply this at the command line (recursive ACL permissions is one area where Konqueror does fail us) thus:

setfacl -R --modify-file=acls /music

You may have to issue a setfacl -R -b /music command in order to wipe all the current ACLs from your files and then issue the above command to apply a new set of ACLs.

Now here's the problem. I reasonably thought that SlimServer would only require read permissions to the music files. In fact, not only does it require read and write access, it also required execute permissions as well for the scan to work(??!!). What's going on there? I can't fathom, but that's how it was solved.

Well, provided your albums are tagged properly with disc numbers and have the same album name (not Album Name - (Disc 1) etc.) then SlimServer should now treat all your multi-disc albums as whole albums and play them in the right order. Yay!

Wireless with NetworkManager, KNetworkManager and YaST - or Not

2007-01-05T17:44:00.000Z

I currently have a working Atheros wireless set up on my desktop machine with OpenSuse 10.2, but God alone knows how. It was more down to luck than anything else. I then figured I could do the same for someone else with an exact same set up. Fat chance. The combination of YaST, Network Manager, KNetworkManager and KWallet means that the right hand doesn't know what the left hand is doing most of the time, and this is true of distributions other than Suse as well. Keep in mind that I'm fairly knowledgeable about getting this stuff to run now, and others will most certainly not be. Those using Windows won't be delving into this stuff.

Firstly, it's a case of actually getting the Atheros drivers themselves. I have to download them from here, copy the two RPMs on to a floppy (keep in mind I have to have networking first!) and then install them on to the system. Once I have networking set up, I can get this location installed as a YaST repository. Now that's OK, but nothing is automatic unfortunately. I then have to issue the following commands:

$ su

# modprobe aes
# modprobe wlan_ccmp
# modprobe ath_pci

I then have to ensure that these modules are loaded on boot up, so this is a case of going into YaST and setting the appropriate sysconfig kernel setting. With these modules loaded, I have a fighting chance of getting my wireless card with WPA up and running.

So, I then need to run YaST to set up my network card with NetworkManager support via the Network Card option in YaST. I first select NetworkManager as my network set up method, and then select my wireless card which should now show up. I then set my static IP address and put in my SSID for my wireless network. Note that my SSID is not broadcast:

No, that isn't my real network or key ;-). Click Next, Next, Finish etc. and YaST will then go through the process of starting all the network services. The only problem is, it doesn't, and KNetworkManager is not started in the system tray as it should be. So, I start KNetworkManager and.........nothing happens apart from the fact that the connection window pops up to tell me in a roundabout way that it needs my details. Again:

I then have to manually enter my SSID and key into KNetworkManager itself, I then click connect and the KNetworkManager icon spins, and spins, and spins, and spins, and fails to connect. Brilliant. What could be the problem? Who knows? All I have to go with is the fact that it won't connect, and it repeatedly wants me to re-enter details I've already entered. I've since found that NetworkManager really doesn't like it if you have two network interfaces, say a wireless one and an ethernet one, and just doesn't want to connect your wireless. There is something in the NetworkManager logs about not being able to retrieve a key, which I think is down to a combination of KNetworkManager, KWallet and NetworkManager itself. I've also seen this on Gnome systems as well, so it can't just be the KDE stuff.

I also have an IPW Intel wireless laptop, and this simply doesn't want to know, even if I set the old connection method ifup in YaST. This may or may not be the multiple interfaces thing. Bizarrely, if I manually enter my SSID from the command line with iwconfig, and then my WPA key, and then go into YaST and go through the set up routine again which restarts the interface then it starts working fine.

Wireless is still extremely hit and miss on Linux, especially with NetworkManager which is supposed to make our lives easier. You can't set IP address with it at all (and it has some trouble with static network settings), and it really doesn't like you having ethernet and wireless interfaces activated on your computer at the same time. However, once you get it working the many wireless drivers are pretty good, and stable.

Goodbye Novell, But Not Au Revoir: Part 2?

2006-11-07T23:18:00.000Z

Well, not so long ago I commented that Novell probably had six months to a year to sort itself out or face death. That prediction looks to have been very accurate with the events of the past week.

I did believe that Novell's demise and death would be a slow one, probably over a period of three to four years. Granted, their revenue is falling, too many of their customers are jumping ship to either Microsoft or another Linux supplier and there's no way on Earth that they're going to be able to maintain the overheads that they have. They still believe that they're a multi billion dollar OS company from the 80s and 90s that can afford private jets, and they're not. However, many people still use Netware as a file and print server, and they don't want to replace things like those, and many organisations use Zenworks to deliver their software so there wasn't going to be any particular point at which they would go out of business. It was going to be a gradual decline.

Novell, though, never cease to find ways to accelerate their demise. One just wonders what they're going to get out of this deal. The way they're trying to completely bypass the GPL section 7 patent problem is by saying that this isn't a deal between Novell and Microsoft - even though Novell are paying for those deals. Ho, hum. Novell are effectively giving Microsoft direct access to their customers so that they receive this covenant not to sue. If I were a customer I wouldn't be too happy about receiving such a covenant because it wouldn't inspire me with confidence. For many people this will not reassure them but will make them start wondering what on Earth is wrong, and what else is wrong.

We all wondered just what ideas Hovsepian and some new people in charge would do to improve Novell's software and products, and to make them actually usable and full featured so that people would sit up, take notice and say "Wow, we should be using that!" Sadly, we now know the one idea that Hovsepian has - indemnification. Effectively telling their customers that they are at risk of getting sued, so they've done a deal with Microsoft which they're now trying to sell as competitive advantage. Or something. Perhaps the board and investors now wonder who on Earth they replaced Messman with and who they're employing. Speaking of which, Novell's CTO (Chief Technology Office) Jeffrey Jaffe is always an entertaining read.

I'm loath to do any sort of apparent character assassination on anyone, but considering that Jaffe has replaced good people like Alan Nugent, and particularly Juergen Geck, and considering the money and benefits he's on, one would expect him to be good - no? Jurgen Geck, in particular, was a pivotal thinker and brain at Suse who saw the need for the YaST configuration tool and pushed it forwards, as well as the design of SLES (Suse Linux Enterprise Server). YaST still remains the best universal administration tool on any Linux distribution, along with the graphical tools that are so necessary. By all accounts, Juergen was quite well respected by many people who he worked with externally as well. I don't know much about Alan Nugent, but at least he seemed level-headed and didn't mouth off about sound bites like virtualisation every few seconds.

Jaffe, on the other hand, is another kettle of fish entirely. This guy is a few virtual machines short of a full Xen. I really don't care how much experience he supposedly has, I cannot find a single piece of technology or idea that he's the driving force behind at Novell. Every other post in his blog is entirely devoted to virtualisation, more virtualisation, consolidating servers via virtualisation, the innovation community (whatever that is) and various other things that are essentially the same thing, but worded differently. I hate to break it to JJ, but people have been doing virtualisation for years.......with VMware mostly. As Chief Technology Officer I would expect him to realise that Xen, although useful, hasn't made much headway against VMware simply because Xen cannot run Windows, yet, and it's management tools and overall quality falls short of VMware, particularly in hardware management. Unless that changes, or Novell does something about it, their support of Xen supposedly running Suse Windows under it as part of the Microsoft deal is absolutely useless. The only way Novell seem to be using Xen is as a way to virtualise Netware, since they don't have the resources to run it on real hardware anymore. So, instead of moving to Linux and porting what they need to get Netware based software an tools up and running on SLES, they're either virtualising Netware in Xen or giving people the option of a Netware or Linux kernel to run in OES (Open Enterprise Server). Dead easy and straightforward really!

Some examples of JJ's profound wisdom:

"Our customers have been demanding for quite some time that we develop solutions to make Linux and Windows interoperable."

What, you mean like......Samba.......? That venerable piece of software that every other Linux distribution is using and that Microsoft has never contributed to in its life? He should realise that Microsoft does not interoperate with other software, since that's how they maintain their lock-in.

"As we prepared our first-to-market Linux implementation of XEN, suddenly there were new opportunities. The customer demand for interoperability morphed into a stronger possibility: supporting both SLES and Windows in a virtualized environment. This would help server consolidation in a mixed shop."

Hmmmmmmm. OK. As CTO, for a start, he should realise that Xen doesn't run Windows and there is still some work to be done, mostly hardware-wise, for it to do so. Aside from the fact that people have been doing this with VMware for, oh I don't know.......years.

"Moreover, as we have heard from the CIO of the City of Seattle at the press conference on Thursday"

That wouldn't be the city where Microsoft is based, would it?

"What I would say, however, is their [Microsoft's] strong embrace of Open Source at the “meta” level is significant; in-and-of itself."

It's anybody's guess what this load of claptrap is. Microsoft has never been receptive to open source software of any kind, have tried working out ways to destroy it and deride it and have even tried to come up with their own poor shared source implementation. God only knows what 'at the meta level' is.

"Note that this protection is not limited to the “joint solution”. Any purchaser of SLES and SLED for any usage will not be sued by Microsoft."

Oh cheers. Thanks. I'm going to rush out and buy a copy of SLES now ;-). I don't think Novell realises the message this gives out to customers.

"Virtualization is much broader than an I/T thought – it is the fundamental driver of business"

You literally can do anything with virtualisation. It's really that incredible. It's the elixir of life.

He also liberally splashes the phrase 'open source' here there and everywhere without knowing what it is, how open source projects are different, how the processes within them work and he has seemingly never contributed to an open source project in his life. I thought that might be a given for a CTO. Everyone connected with Novell should get warm fuzzies with such knowledgable people at the helm.

Software or Hardware RAID Part 2

2006-10-08T19:00:00.000+01:00

Well, it seems as though hardware RAID isn't going to happen. The Adaptec card I was going to buy has a PCI-X interface, and I can't find a motherboard with what we want that has PCI-X interfaces. Seriously, can people just not stick to one emerging standard like PCI-Express that most boards use?

So, Linux software RAID will be what we'll be using on the server. I did say that hardware RAID was the preferred option, but if you can't find that right hardware to do it with, don't bother. If it doesn't work properly, you're stuck with it.

On Graphical Configuration Tools

2006-09-04T20:55:00.000+01:00

You know, I never cease to be amazed by the number of Linux enthusiasts and open source proponents - those who tell us that desktop Linux is a reality for many today and that Linux on the server is a done deal incidentally - who cannot comprehend in their wildest dreams that a Linux and open source based operating system needs good quality administration and configuration graphical interfaces.

Oh, of course, if you mention this you'll get all of the standard well worked responses. The first is usually something like "Oh, have you seen YaST? It's a graphical interface, and it's even being ported to other systems like Debian [totally incomplete]". Unfortunately, YaST, as good as some of it is, is simply hugely incomplete when you want to configure an awful lot of the features of Bind or Samba. I've found that YaST seems to screw up my Bind server beyond any simple set up to the point where I have to use Webmin to restore order, and YaST is absolutely woeful in setting up more than just a really basic Samba client. You certainly can't use it to set up a complete Samba server, with a straightforward way to amend many of the settings that many need to in setting up Samba. YaST, just like other tools in other distributions, has no way to set up an LDAP server in a straightforward way that can be used as a central repository, to facilitate single sign on and to use something like Kerberos for central authentication. Does that exist? No, is the short answer. I think we're all supposed to buy eDirectory or something, and then put up with a woeful lack of graphical configuration tools again.

As I've found out, although YaST is probably the best administration tool around for any Linux distribution, the state that it is in after all these years of development leaves it well, well short of what's required. Red Hat's graphical tools are a joke, apart from the Fedora Directory Server's which they didn't write, and Red Hat are the market leader - and they run X and Gnome on their servers...... I can't think why, because there's nothing to show. It now also looks as if even YaST is falling by the wayside in terms of future development. There's now a GTK front-end version, not actually initiated by Novell, and there is absolutely no sign whatsoever that it will be expanded to cover even Novell's own products as an administration tool. Happy Novell users will then go back to the familiar routine of a different web/Java/Mono/*insert flavour of month here* administration tool that requires them to do a fair amount of setting up themselves.

You'll then point this out and then you'll be pointed to Webmin, or SWAT or something else. All very nice, but they're not coherent or integrated in any way, and they're not particularly user friendly. A lot of configuration and set up is still required behind the scenes, such as OpenLDAP for instance, and the error messages you sometimes get out of some of the modules are less than helpful. In terms of having the right packages installed and configured, it's just not integrated with the rest of your system. In short, Webmin is a decent tool for being web based but fairly poor in the grand scheme of things.

We then come on to the last stage. You'll point this out and you'll be told point blank that running a graphical interface and X on a server is a really bad idea (unless you want to try and save resources it's no more or less of a security risk actually), and if you don't have intimate knowledge of the command line, all the command line switches you'll need and several completely different config file formats then you're a nobody who is totally incompetent and shouldn't be administering anything. While I don't doubt that some administrators are less than competent, there seems to be a strange perception that competency is equal to using a command line and editing text files. Ho, hum. Looking at your average Gentoo enthusiast who wants to increase the performance of his system by 150% should tell you this isn't true.

I think this perception of incompetence from using graphical interfaces stems from some of the bad things Microsoft have done with their tools in Windows Server, and the training that many Windows Server administrators undergo. Microsoft do an awful lot of bone headed things with their tools, mostly to make money. A case in point are the new server 'roles' you see in Windows Server. It's clearly an attempt to try and straight jacket your server into particular uses so that you end up buying another server for another particular role, even when possibly you don't need to. Maybe we'll see yet more restrictions in Microsoft software, such as Sharepoint needing to be on a server as a default site, in the future. Microsoft's server functions also tend to be added pretty haphazardly and in a disorganised fashion.

Whatever though, if Linux as a server system wants to expand beyond its current niche as a data centre system and a replacement for Unix into a world of small business and departmental servers (a big opportunity incidentally) then good quality graphical administration and configuration tools are an absolute must. There are no ifs or buts, and no amount of shouting about 'command lines', 'editing config files' and 'running a graphical environment on a server is a bad thing' is going to change that situation. You may want to run a main server with no X server installed, but there should at least be a straightforward option to forward graphical applications to another server or client - perhaps in the install procedure or via an option somewhere. Those abilities are certainly available, but they haven't been made easily configurable in the distributions I've seen.

I suppose one of the reasons why all this hasn't been attempted in the Linux world is because it's hard. It is hard to produce graphical applications for end users, because not only do you have to think about things like usability and how to configure the vast amount of functionality correctly, you also need the right development tools to produce large scale graphical applications of a good enough quality. This is probably why Red Hat, and their usage of GTK, hasn't produced the tools required. I've said it before and I'll say it again, GTK is just simply not good enough for developing graphical applications of a reasonable size and quality.

Additionally, I supose it's the easy way out for many Linux companies like Red Hat and Novell. They can pretend that there is a whole world out there of low hanging fruit - the supposed riches of the replacement of proprietary Unix with Linux - which Microsoft does not play in, so they conveniently won't have to compete with them. Microsoft will continue to improve Windows Server, and improve its command line and performance failings, and Microsoft will consolidate the hold they have over departmental and small business servers (with the help of lock in from the client to things like Exchange) and they will use that as a base to attack what is left of Netware and the Linux and Unix arenas. Of course, this is a situation that can't go on.

A Linux and an open source server should be a fantastic competitor to Windows Server within small businesses and other areas. Certainly, all the software and ingredients are there. The problem is bringing it all together in a straightforward and integrated way, and it seems as if the only company having the slightest go at this is Xandros sadly.