miau.biz

shows what I know

2009-11-08T06:44:00.003Z

ruby 1.9.1 is blazing fast I hear. and it doesn't segfault when I run our build on it, unlike 1.8. it uses native threads. they all acquire the giant interpreter/vm lock. and I finally rediscovered the ruby isn't erlang post I've been talking about.

jruby 1.4.0 is out and buildr 1.3.5. and java5 is dead.

--

cpus

2009-11-07T06:38:00.003Z

cpu benchmarks usually have identical elements across different review sites. on the same site it is certainly important to use the same benchmarks for every cpu but all site have games, video encoding, raytracing and a few synthetic benchmarks.

no one posts benchmarks of compilation and automated test runs. and it's rather expensive to benchmark cpus yourself.

ec2 added the m2.4xlarge which is 2x x5550 and 64gb memory. they count each core as 3.25 ECU (their own generic cpu performance comparison unit) versus 2.5ECU per core on the c1.xlarge which is 2x e5410.

x5550 is a 2.66ghz 95w nehalem with a slight turbo and 2x QPI. i think HT is disabled on ec2.

our 3 minute test run ran in 2 minutes flat over there.

since memory was not an issue in launching multiple jvms for striped_junit, I tried a bunch of different thread counts but they didn't make a difference between 3 and 6. beyond that performance started to deteriorate.

meawhile I was happy to find google's wave protocol using a TimeSource only to have it kicked out of our own code base half a day later. I learned that global state is the ambient monad. and I'd like to see what Misko has to say about Transform.java...

in ~~soviet russia~~ japan you are the worm and the early bird eats you.

Dr. Odersky, would it please be possible to add the following sugar to scalac:

def lol(list: List) = { list.first }

is the same as:

def lol[A](list: List[A]) = { list.first }

--

implicit type conversions

2009-11-03T13:07:00.004Z

the more I think about them the awesomer they are.

compared to ruby's monkey patching they are delightfully unobtrusive and safe.

compared to explicit conversions.

--

I'm tellin' y'all it's double cas

2009-10-16T18:31:00.007+01:00

double compare and swap is the ability to atomically update two independent memory locations if both of them remain unchanged. no mainstream hardware architecture supports this.

x86 has cmpxchg8b for two sequential 32 bit words and cmpxchg16b for two 64 bit words. the sequential variant enables some interesting operations but is generally not very useful.

researchers are busy inventing algorithms that could be implemented if dcas existed. sweet.

simplified, the 'hat trick' lock free deque is a doubly linked list. each end of the list also has a hat. the hat points to the same location as the head of the queue. this location points to null.

when pushing or popping, if both the hat and the null remain unchanged, no one interfered during your operation. you update the hat and the value the hat pointed to previously.

the proposed deque with the dcas on independent pointers also avoids the dereferencing problem mentioned by LWN.

-

privacy and such

2009-09-06T08:25:00.005+01:00

temperature data seemed innocent enough at first thought. no harm in letting people know what the temperature is at my house. I was fascinated when I saw this appear.

the graph shows the temperature measured by two sensors on windows facing the same direction on three consecutive days. at the same time every day a distinct M-shape appears. unless it's a systematic bug in my measurements or code, I guess it has to do with the way the sun shines in through the windows when it passes between two buildings. the drop in the middle is a tree or other obstacle.

this kind of peculiar shape could be used to fingerprint data, determine what latitude the measurements were taken at, what direction the windows are facing, or even identify the windows where the measurements were made.

a wireless sensor network

2009-08-11T10:17:00.012+01:00

update: when not using encryption or any sleep modes I am able to remotely reset the funnelio and upload the firmware automatically. it's awesome.

I jumpered DTR from UARTSB to DIO3 of the UARTSB. (and solder SJ1 on the funnelio) with that simple setup it is possible to use the end_device.pro / coordinator_auto_reset.pro from funnel as is.

--

the 1980s were characterized by the chips that made computers affordable for households. lasers that enabled long range high speed communications in optic fiber defined the 1990s. ubiquituous sensors may be the defining technology of the current decade. mobile phones contain a number of sensors. android and iphone development environments allow easy access to the collected data, and soon even SIM cards will have sensors and be easier to program than some phones.

I went for a simpler sensor network. the network has four nodes that measure temperature and report to a central collector which publishes the data.

I used funnelIO boards, xbee series 1 radios, an uartsb, and ds18b20 temperature sensors.

I chose the original funnelios from sparkfun, over the remixed ones from seeedstudio. the seeedstudio version has a 2.54mm jst plug for the power. they don't sell batteries with those plugs, and neither does anyone else.

there are a number of tutorials on uploading your program wirelessly to an arduino. the tutorials go into much detail but they require you to do exactly what they do with exactly the same hardware and software. there are alternative ways of uploading wirelessly, but they aren't covered. the sparkfun tutorial actually requires you to reprogram your microcontroller. which is difficult in the case of the funnelio.

everything starts with the closing of the solder jumper on the funnelio. the explanation is in the schematic of the funnelio. sj1 connects dio3 of the xbee with the rst pin of the arduino. the arduino is reset when digital input/output number 3 of the xbee is triggered. dio3 of the receiver is configured as DO HIGH (digital out), which is equivalent to holding down the reset button until that pin is triggered. the rest deals with setting up the sending side so that when a request to send(RTS), not the same thing as RST(reset) is sent to the serial on the sending side, that information is passed on to the receiver's reset pin.

if everything works out, you can upload your arduino stuff as if the remote device was connected to your computer with the usb cable. unsurprisingly there are a number of caveats. the xbee must be powered on, it must associate with the network, it cannot be using encryption, the arduino can connect to a single serial port at a time: either xbee or the wired one.

the uartsb actually has 5 pins instead of 6 on the serial port. RTS is not available. RTS is however wired to DIO6, which is right next to DIO3. when using the wired connection for uploading to a funnelio, the DTR can be connected to the RST pin and everything will work. I tried all combinations of connecting DTR and RTS to DIO3 and DIO6 and configuring DIO6 and DIO3 as inputs on the sending side, and configuring the change detect on the receiving side to 8, 0x40, and 0xff. but I could not get the arduino ide to reset the funnelio properly. but!

you can upload your arduino code without autoreset! you just have to make sure that the xbee is powered on, it is associated, encryption is disabled, you press the reset button by hand and you release it at a good time. further, you have to setup DIO3 and change detect so that the default setting of DO HIGH does not hold your reset line high all the time. or avoid closing the solder jumper. the thing to watch out for is the tx and rx lights on the uartsb. when the leds first blink one after the other and then start blinking continuously you're doing good!

I ended up not uploading wirelessly, instead I used the wired interface. all you have to do is unplug the xbee everytime you upload because the funnelio has no switch to choose the serial interface and the xbee has priority over the wires.

802.15.4 or zigbee does not support mesh networks. in this context a mesh networks would be one that allowed multihop routing and self-healing properties. instead there is a coordinator and end devices. my coordinator is the uartsb and the end devices are the sensors.

I wanted to conserve power and to use encryption. I went through a number of different combinations of who should wake up who and whether I wanted to buffer data before sending and so on. it turns out that it is easier for the xbee to wakeup the arduino than the arduino to wakeup the xbee. especially with the way the funnelio's xbee socket is wired. pin 9 is not connected to anything, and it's not trivial to jumper it to the pins of the at168.

I use sleep mode 4, or cyclic sleep - the remote xbees periodically wake up, send a packet to the coordinator, and the coordinator either sends a buffered packet to them or tells them to go back to sleep. after a period of inactivity the xbee goes back to sleep. if a packet is received it is given to the arduino. if the arduino was sleeping, the serial activity wakes it up. the arduino can keep sending on the xbee as long as it doesn't allow the xbee to time out. in case of timeout, the connection is lost, and sent data is not buffered. with cyclic sleep the remotes also become hard to reprogram wirelessly as the xbee radio may be off or unassociated at the time the sender wishes to upload the program. with a short sleep timeout and a long sleep period reconfiguring the xbees with X-CTU also becomes difficult because the xbees will go to sleep and become unavailable. eventually the only way to reprogram them is to unplug the uartsb, shutdown x-ctu, plugin uartsb, start x-ctu, and choose "always update firmware".

The xbee uses 60mA at 3.3v when it is active, 0mA when it sleeping, and the multimeter flashes to about 15mA for a split second when it checks for the packet. the funnelio is an atmega168 running at 8mhz on 3.3v. it uses 6ma when it is calling delay(1000) and 2.5mA when it is sleeping. in this sleep mode wakeup is instantaneous. the batteries I am using are 1000mAh or 1100mAh depending on who you believe. sending a packet back and forth every minute, with timeout at 250ms and temperature measurement taking about 200ms, approximately 200mAseconds are consumed every minute giving an average consumption of 3mA. the batteries should therefore last for about 2 weeks. ofcourse it will take atleast two weeks to find out, unless the batteries run out earlier.

the ds18b20 is a solid state non-thermistor temperature sensor. it uses a proprietary protocol called 1-wire. it has 3 wires, and requires "no external components". that's excluding the 4.7kohm resistor which it requires. the sensor seems to heat up faster than it cools down. I don't know how that's possible. the sensor returns the temperature as a two-byte fixed point reading. it takes 750ms to "convert" the temperature to a 12-bit value, with 4 bits after the decimal point. 3 bits of resolution takes 375ms, 2 bits takes 187,5ms and so forth. I am using the 2 bit resolution, so I get readings in quarter degrees celcius. the accuracy of the sensor is +/- 0.5 degrees and the maximum resolution is 1/16th of a degree. apparently the accuracy discrepancy is fixed over the measurement range, or atleast between consecutive measurements, so there is no noise in consecutive readings. if the measurement changes 1/16th of a degree, the temperature really changed, but it might be between 25.0 and 25.0625 or 24.5 and 24.5625. my four sensors produce results within one degree of each other if placed nearby on a table. the onewire library for arduino doesn't live in version control, and there have been some bugs that have been fixed, but not in the official release. unlike the other examples floating around I didn't use parasite power for the ds18b20. measurement takes the same fixed time, but instead of calling delay(200), I call sensor.read() and wait for it to return something other than zero.

the default mode for xbee is a delightful abstraction: there is no way to tell whether you are connected via radio or if you are at the end of a cable. there are however two problems, and these are addressed by Digi's API mode. in cyclic sleep mode, broadcasting is not supported. the coordinator has to address messages to each remote separately. API mode allows this to be performed without entering command mode. when receiving messages, the identity of the sender is unknown. the data would have to be somehow tagged with the sender identity, and the data might further have to be tagged with some semantic information: for example high byte or low byte of temperature measurement. doing both of these at byte level would require significant overhead. entering command mode to change recipient addresses would also be difficult if there were concurrent incoming messages from remotes, and while in command mode some data might be lost. API mode, whose use is easy enough with the xbee-arduino library, allows data to be grouped into packets which are delivered atomically and with the sender identity automatically present.

I ended up having to set Mac mode to 1. The default mode of zero, which has ACKs and a Digi header got screwy when AES was enabled. Messages would be delivered as double to the remotes, even though only one ACK would be received on the sender side, and worse, after a couple of messages the sender would somehow get clogged and unable to send any more messages or even to reset itself.

I wrote the coordinator code in Ruby with the serialport gem. on Linux and Windows usb-to-serial stuff works out of the box but on OSX a separate driver is required. the driver is included in the arduino download for osx. I send out a message to the remotes every minute or so, they respond with the current temperature and I write it out in json. I rsync the JSON to a server and draw some graphs with HTML5's delightful canvas tag.

updated: the code
--

open source hardware

2009-08-11T10:01:00.006+01:00

I completely slept on this. besides software, hardware is also open source. there are several iniatives. I'll focus on arduino and its derivatives, but that is by no means the only relevant one.

Important for me personally is that you don't need to solder. you may end up soldering, as I did, and find that it can be easy and fun. but it feels good to know that it's totally optional and when I do it, it's because I choose to do it.

arduino is a combination of a microcontroller, which is a tiny computer that runs at 16mhz and has 32kb of ram on it, some standard pins for input and output, a usb-to-serial connector which both powers the device and allows communications with a computer and a development environment which provides high level abstractions for interfacing with devices: reading inputs from temperature sensors, controlling servos and leds.

arduino is aimed at making everything easy and accessible and it works perfectly for that. 99% of the work is still accidental complexity, but otherwise it would be more like 99.999%.

a cool product family for the non-soldery people is electronic bricks from seeedstudio.

communication is often done via serial, aka rs-232. although it is the same protocol as that 9-pin thing you used to connect your mouse to the computer before ps2 came along, these days rs-232 is physically in a usb cable, in 5 or 6 separate pins or even just a single physical pin.

communication speeds are typically around 19200 or 57600 bps. compared to usb2.0's 400mbps and pci-express' 10gbps this is low. low enough to rule out many applications, especially around video or high resolution imaging. but it's fast enough for applications such as lcds, gps, accelerometers, temperature, color sensors, motors, servos. encoding video, audio or images within the constraints of bandwidth, power and processing requires specialized circuitry or an fpga.
--

profilin'

2009-07-16T19:33:00.006+01:00

wicket's Application has Settings which has a PropertiesFactory which has a Map which maps paths to the corresponding Properties. non-existing Properties, which is most of them, are identified with a marker object, so they don't have to be constantly rechecked. Application is a ThreadLocal so it has a semi singletonish global state kind of flavor to it. Most properties don't exist because Wicket by default looks in a lot of places for a component's localization data: by language and country, in properties and xml files, in parent components.

The caching works great in production. In unit tests Application is constantly re-initialized and the cache is flushed all the time. Changing the properties cache to static makes our test suite run twice as fast overall. From 25 minutes to 12 and a half.

update:

here's the code:

public class MyDummyWebApplication extends BaseWicketTester.DummyWebApplication {
private static final Map propertiesCache = new ConcurrentHashMap();

@Override
protected void init() {
optimizePropertyLoading();
}

private void optimizePropertyLoading() {
PropertiesFactory propertiesFactory = (PropertiesFactory) getResourceSettings().getPropertiesFactory();
propertiesFactory.getPropertiesLoaders().remove(1);
Inject.field("propertiesCache").of(propertiesFactory).with(propertiesCache);
}

--

a fast queue

2009-07-15T09:22:00.007+01:00

threads aren't the most natural way to think about concurrency. actor implementations are free to map between actors and threads, but with X cores, it helps to have atleast X-1 threads used by the actors.

the threads need to pass data to the next thread in the chain. this can be slow with memory barriers, blocking, cache trashing. a single thread must do a large amount of work to offset the overhead.

FastForward is a sweet queue that solves exactly this problem. blocking queue + dequeue operations required by each thread can be performed in 30ns on a 2.66ghz opteron. that comes out to roughly 80 cycles. in the example, one pointer's worth of data is passed between the cores. other data can also be passed along, and that data may also qualify for prefetching on the receiving core.
--

transactional memory

2009-06-07T09:24:00.004+01:00

concurrent programming with shared memory is complicated. if there was a way to make such programming easy, shared memory could remain an alternative to message passing in the presence of large amounts of thin cores.

transactional memory aims to automate the selection and acquisition of locks. whether transactional memory can be done efficiently, and adopted incrementally is still uncertain.

there are both software and hardware implementations of transactional memory. at first they seemed to me sequential evolutionary steps but that is not the case. Most HTM implementations operate at cache line granularity and use cache coherence protocols to track and maintain different copies of shared memory areas. most STMs operate at object level and use locks, temporary copies of objects, and additional data structures to keep track of transactions and mutated state. the fundamental problem of HTM is transactions that exceed cache sizes or that last longer than one scheduling quantum. the fundamental problem of STM is overhead. it seems natural to expect a balanced hybrid to offer the best of both worlds.

transactional memory implementations differ in certain key attributes. the choices often depend on whether commits or aborts are expected to be more frequent.

shared state can be modified in place or on a separate copy. in place modification requires a second copy to be created to maintain the original state in case of an abort. in place modification makes commits cheap.

concurrency can be handled optimistically - several threads can enter the same critical section concurrently. if the threads do not conflict the locking would have been overhead. if a conflict is detected all but one thread are aborted.

conflicts can be detected as soon as they occur or only at commit time. late detection saves overhead in the absense of contention. early detection involves more bookkeeping work, but when a conflict occurs it skips the useless computation in the doomed portion of the transaction. late detection can also allow computation on inconsistent state. th results will eventually be discarded but it can result in exceptions that wouldn't occur otherwise.

the semantics of nested transactions can be defined in two ways
for aborts:

a nested transaction aborts the parent transaction.
a nested transaction allows the parent to proceed.

and for commits:

changes committed in a nested transaction become visible to the parent transaction.
changes committed in a nested transaction become visible in global state.

IO and cooperation between transactional and non-transactional software are some of the challenges for the future. The classic chicken-and-egg problem is naturally a key blocker. Intel has a lot of eggs in this basket if they expect anyone to buy 1000 core chips.

yeah

2009-05-23T10:08:00.004+01:00

I bought a pair of pants. Six days later I notice the stitching has come apart. No need for vexation. I go to the store, I show the pants and the receipt to the nice guy behind the counter. He calls the manager just to be sure. The manager decides to come over and look at the pants.

The manager then says the pants are fine, the hole is probably of my own doing, but she's not saying it is, she's merely examining the threading. In the end she tells the cashiers to refund me and she leaves. I get my refund. When the cashier ends with "I'm sorry about your pants, they really should last more than six days," I realize that I'm not mad at the cashiers, I'm mad at the manager.

Flip the tables. You're the cashier. A calm customer comes in for a routine refund of a defective good. Suddenly, your boss butts in, she tells the customer to fuck off, she orders you to finish the transaction, and she retreats to the office.

-

clarksfield turbo

2009-05-21T14:58:00.005+01:00

though clarksfield chips are reported to be clocked at measly 1.6, 1.8, and 2.0ghz, nehalem's turbo boosts them up to 3,2ghz for a lone running thread.

the current top of the line mobile core2 chips x9100 and qx9300 are dualcore @ 3,06ghz and quad core @ 2,53ghz, both with 1066mhz fsb. clarksfield uses dual channel 1333mhz memory, and has DMI instead of the desktop variant's QPI.

conclusion: a 2ghz clarksfield will beat an x9100 in single-thread performance, and a qx9300 in parallel performance. clarksfieldy macbook pros should be out right around the time my summer vacation ends.

how bad is DES?

2009-05-21T13:57:00.003+01:00

DES was kicked out of srs bsns over a decade ago, yet there are no bruteforce des crackers available for home PCs. DES encryption/decryption is not well suited for the instruction set of a general purpose computer. in 1999 distributed.net harnessed the idle cpu time of around 100.000 computers together with EFF's deep crack to break des within 24 hours. the peak of 170 billion decryptions every second on 100.000 computers averages out to 1,7 million decryptions per second per computer.

for computers 10 times as fast now as those of 1999, a large amount of botnet time would be consumed in cracking just a single key. the current GPGPUs aren't particularly well suited for DES cracking either. a single decryption takes a lot of cycles and a lot of power.

FPGAs are optimal for DES. the described implementation is a 37 stage pipeline composed of single cycle steps that changes keys, plaintext, ciphertext and operation mode on every cycle without stalls.

in 2006, german researchers implemented a 9000 euro device with a total power consumption of 600 watts consisting of 120 FPGAs, each capable of 400 million encryptions per second. they eventually raised the clock speed so that an exhaustive search of the DES keyspace takes 12.8 days. DES is futher susceptible to time/memory tradeoff attacks which copacabana performs.

-

how wicket protects you from CSRF

2009-05-19T09:17:00.007+01:00

the beauty of wicket finally became apparent to me when I wrote my first delete operation.

the traditional delete works like this:


miau.biz/delete.php?id=5

instead, wicket sets up a bounded number of continuations that the user can invoke. each continuation corresponds to one of the allowed deletions. if an item isn't allowed to be deleted, that continuation does not exist, and neither does there exist a way to invoke it.

an attacker can however induce a victim to invoke any existing continuation against their will.

say the url to delete id 5 is:


miau.biz/::wicket::invokeContinuation:2

putting that url in an img tag will make the browser load it automatically. oops.

the suggested solution is to include a hard to guess token with each link, and requiring separate login for dangerous operations.

wicket uses null as the name of the default PageMap, and sequential numbering for page id's, page versions and markup id's.

wicket has a builtin fix against csrf. you aren't forced to use their solution however. in fact nextPageId() in Session is protected and not final; I can override that with

private static SecureRandom random = new SecureRandom();


synchronized protected int nextPageId() {
  return random.nextInt(Integer.MAX_VALUE);
}

not every request gets a unique id, but many requests do. and the whole thing is fairly transparent.

Wicket's CryptedUrlWebRequestCodingStrategy has a fallback mechanism to allow you to also use urls that aren't gibberish:

   public RequestParameters decode(final Request request)
  {
      String url = request.decodeURL(request.getURL());
      String decodedQueryParams = decodeURL(url);
      if (decodedQueryParams != null)
      {
          // The difficulty now is that this.defaultStrategy.decode(request)
          // doesn't know the just decoded url which is why must create
          // a fake Request for.
          Request fakeRequest = new DecodedUrlRequest(request, url, decodedQueryParams);
          return defaultStrategy.decode(fakeRequest);
      }

      return defaultStrategy.decode(request);
  }

*cough*

eclipse containers for fun and profit

2009-05-17T09:17:00.003+01:00

given a multi module project in eclipse, where project X depends on project Y,
when I close project Y,
I want project X to depend on a jar of project Y.

eclipse classpath containers are a mechanism for dynamic late binding of classpath elements.

ibm has a tutorial that requires you to register on their site. m2eclipse does the whole thing for maven projects, but also 99 other things, most of which involve running random maven tasks when your trying to write code.

despite all the static methods and integers enumerations, eclipse makes the whole thing easy; including creating your own plugin and publishing an update-site for it.

the heart of the class path container is the is the IClasspathContainer. The getClasspathEntries() method returns whatever is relevant for that situation. in my case it's either a JavaCore.newProjectEntry() or a JavaCore.newLibraryEntry() depending on whether the referenced project is open or closed at the time. For some reason JavaCore.newVariableEntry() comes out as K_SOURCE which doesn't work for a jar. so I use wrapped the resolved path from newVariableEntry() into a newLibraryEntry to make it a K_BINARY. your type is K_APPLICATION, your path is your unique identifier, and stuff after the slash in the path is parameters for your container. in my case the parameters might be the other projects that this project depends on.

then you need a ClasspathContainerInitializer, which will call JavaCore.setClasspathContainer with a new instance of YourContainer on the dependent projects everytime you open or close one the dependee projects. this then sends an F_CLASSPATH_CHANGED signal to those concerned.

you'll know that a project was opened or closed, because you added an IElementChangedListener with JavaCore.addElementChangedListener().

Your initializer is registered by adding it at the extension point org.eclipse.jdt.core.classpathContainerInitializer. never mind the xml, eclipse will generate it for you.

to allow the addition of containers from the build path configuration menu, eclipse requires a classpathContainerPage. either create a dummy one and register it, or add your container to the classpath file by hand.

Finally, I exported my plugin, and then binary imported the same. This is a nonrecoverable operation that replaces the source code of the plugin with the class files.
*cough*

throwing money at it

2009-05-10T20:03:00.004+01:00

sometime in the future apple will get past the awful 1.6.0_07. meanwhile joel on software, did a wonderful piece on using money/ssd drives to speed up their build. I wasn't able to test this myself so I am glad joel did it for me, and confirmed my expectations that compiling/testing is cpu/memory bandwidth bound rather than io bound. our whole build takes up around 200 megabytes of physical space with all its dependencies and the jdk takes another 200 megs. further, a good part of both chunks is probably never loaded. so with enough memory there would be virtually no need for disk io.

what would be interesting beyond this would be to figure out if ssd could help in the overall workloads we use, that is (again): reading email, im'ing, running virtualbox, using an IDE, building, using the browser, profiling, running jetty. all at once. and that could hard to measure unless the difference was so great it would be obvious in actual use.

then I had the chance to work on a brand new windows box with a t5500 (a 1,66ghz core2duo), 2gigs of ram, and corporate antivirus software set to scan all class files, and read michael feather's excellent book working effectively with legacy code.

by the time I highlighted a block of code in eclipse, held down alt and hit the up arrow and nothing happened. 5 seconds elapsed, there was screech from the harddrive for a few seconds and then my method moved up a few lines. I was just smiling from the absurdity.

michael feathers points out the importance of having a fast feedback cycle and his book also alleviated my fears that, while i am waiting for the computer to do something it's natural for my mind to wander off and get frustrated, rather than spending that time subconciously processing the customer's business logic. I've actually found myself trying to optimize my day so that when I have to go to the bathroom, I'll hold out a bit so I can start a build, or get an update from version control before I go. then I realized I was way off putting any mental effort into this.

we now have some good steam going into getting our sprawling build back under control. the experience so far has reminded me of the importance of communication. I just can't guess what others are thinking from just looking at them and watching them act. I really need to make the effort to conciously communicate with people in the same room, the next room, and farther away to find out. and it is often very enlighting to do so.

as an example, it turns out part of the frustration of the slow build was of my own doing. the whole issue could be circumvented not by cleaning up the build, getting faster hardware, optimizing our continuous integration environment, switching to a more branch friendly version control system, but by the simplest solution of all. it all goes away if you just stop running the build.

I also started using a remote git repository on my own server to move scripts around between home and work. I'm expecting it to be even better than posting stuff on my blog and copypasting it at work.

for those on jaunty, who do run the build:

#!/bin/bash
function notify {
 test -x "`which notify-send`" && `which notify-send` -u $1 -c $2 -i $3 "$4" "$5"
}
/usr/bin/time -f "%E real" -o ./time.txt buildr $@
if [ $? -eq 0 ]
then
 notify normal transfer.complete sunny "successful build" "`cat time.txt`"
else
 notify critical transfer.error error  "failed build" "`tail -1 time.txt`"
fi
rm ./time.txt

and this too, because it uses the totally sweet erlang icon in the notifications!

       case regexp:match(ServerName, "compute.amazonaws.com") of
               nomatch -> ets:delete(instances, Nick),
                       os:cmd("/usr/bin/notify-send -i erlang -c device.error -u normal 'joe experienced an error'"),
                       bot:irc_privmsg(Socket, Nick, "some kind of error occurred.");

enjoy.

static analysis

2009-05-08T19:52:00.000+01:00


module FindBugs
  include Extension

  first_time do
    desc 'run findbugs on your project'
    Project.local_task('findbugs')
  end

  before_define do |project|
    project.recursive_task 'findbugs'
    Rake::Task.define_task 'findbugs' do |task|
      fail "you must set FINDBUGS_HOME" unless ENV['FINDBUGS_HOME']
      findbugs_home=ENV['FINDBUGS_HOME']
      Buildr.ant 'findbugs' do |ant|
        ant.taskdef :name=>'findbugs', :classname=>'edu.umd.cs.findbugs.anttask.FindBugsTask', :classpath=>File.join(findbugs_home, "findbugs-ant.jar")
        ant.findbugs :home=>findbugs_home, :outputFile=>project.path_to(:target, "findbugs.xml") do
          ant._class(:location => project.path_to(:target))
          ant.sourcePath(:location => project.path_to(:source, :main, :java))
          ant.sourcePath(:location => project.path_to(:source, :test, :java))
          project.test.dependencies.each do |dependency|
            ant.auxClasspath(:location => dependency)
          end
        end
      end
    end
  end

  after_define do |project|
    findbugs = task('findbugs'=> project.test.compile)
  end

end

class Buildr::Project
  include FindBugs
end

bugs

2009-05-06T17:11:00.003+01:00

so I really kicked myself in the nuts with that TMPDIR stuff.

since I have my m2 repository on the EBS volume, I should really have $TMPDIR on the volume too, instead of in my home directory. and the AMI I was using is geared towards IPv6. and my version of tsocks doesn't play too well with that so. add -Djava.net.preferIPv4Stack=true to java options to get back within the realm of transparent proxying.

I sure am glad I can delete all those broken ebs volumes now.

the missing pieces of the puzzle

2009-05-04T06:57:00.006+01:00

here's the rest. btw, controlling the ebs volumes as shown in the previous post has the added benefit of making changes simple when problems/fixes are discovered.

here's stop.sh which maps an amazon host to an instanceid.

#!/bin/bash
set -e
echo "stopping $1" >> log.txt
ec2-describe-instances | grep $1 | cut -f 2 | ec2-terminate-instances - >> log.txt
echo "completed"

here's the user end script:

#!/bin/sh
if [ ! -e "./buildfile" ]
then
echo "no buildfile in this directory"
exit 1
fi

if [ -z "$EC2_HOST" ]
then
echo "set the EC2_HOST environment variable lol"
exit 2
fi

chmod 600 keys/id_dsa_ec2_buildr
export EC2_HOST_AND_USER="buildr@$EC2_HOST"
export SSH_COMMAND="ssh -i keys/id_dsa_ec2_buildr"
echo "syncing"
rsync -rt -e "$SSH_COMMAND" --delete-excluded --exclude target --exclude .svn --exclude bin --exclude "/*/reports" . $EC2_HOST_AND_USER:teh-build
echo "'n sync.  building"
$SSH_COMMAND -R 10022:a_host_on_your_network:22 $EC2_HOST_AND_USER ./build.sh $@ && echo "build successful"
rsync -rt -e "$SSH_COMMAND" $EC2_HOST_AND_USER:teh-build/reports .
echo "the end"

this is setup.sh on the ebs volume:

#!/bin/bash
useradd -m buildr
dpkg -i /mnt/super/debs/* > /dev/null
rsync -rt /mnt/super/buildr_home/ /home/buildr
chown -R buildr /home/buildr
chmod 700 /home/buildr/.ssh
chmod 600 /home/buildr/.ssh/authorized_keys

this is the directory structure of the ebs:

buildr_home  debs  jdk  jruby  m2

debs contains tsocks and vim-nox .deb files, jdk is openjdk7 b54, m2 is a populated repository, jruby is jruby patched with a different version of jodatime. and finally buildr_home is the prototype home directory for the build user.

here's the directory:

build.sh  scratch  tasks  teh-build  tsocks.conf

scratch is empty, tasks contains some additional tasks that are not needed in the regular build, more specifically stripedjunit and the monkey patch to ant.rb. teh-build is a prepopulated copy of the build directory, so the first rsync doesn't take as long.

here's build.sh:

#!/bin/bash
echo "start"
export EBS_MOUNT=/mnt/super
export LD_PRELOAD=/usr/lib/libtsocks.so
export JAVA_HOME=$EBS_MOUNT/jdk
export M2_REPO=$EBS_MOUNT/m2/repository
export PATH=$JAVA_HOME/bin:$EBS_MOUNT/jruby/bin:/usr/local/bin:/usr/bin:/bin
export BUILDR_PARALLEL=true
export TSOCKS_CONF_FILE=$HOME/tsocks.conf
export TMPDIR=$HOME/scratch
export JAVA_OPTS="-XX:+UseCompressedOops -noverify"
export JAVA_OPTIONS=$JAVA_OPTS
mkdir -p $TMPDIR
cp -r $HOME/tasks teh-build/super
cd teh-build
echo "in directory"
ssh -fN localhost
echo "ssh established."
sed -i "s/svn_revision/99999/" super/buildfile.rb
sed -i "s/\(define \"toplevel\" do\)/\1\ntest.using :stripedjunit/" super/buildfile.rb
echo "start build."
time tsocks buildr $@
export BUILDR_EXIT=$?
echo "that's it"
ssh -O exit localhost
echo "killed ssh"
exit $BUILDR_EXIT

I use a separate temp dir, because jruby is can't move files across devices. I don't copy the .svn files along so I use 99999 for the revision instead.

tsocks.conf, and server_type can be 5 for openssh DynamicForward contrary to what I claimed earlier:

server = 127.0.0.1
# Server type defaults to 4 so we need to specify it as 5 for this one
server_type = 5
# The port defaults to 1080 but I've stated it here for clarity
server_port = 50505

and .ssh config:

Host localhost
User commonuser
DynamicForward 50505
Port 10022
ExitOnForwardFailure yes
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
IdentityFile ~/teh-build/keys/id_dsa_commonuser

here you need to have a host on the local network that has a user that allows logins with a shared key, from version control in our case.

but after all this, the user shouldn't need to do more than, get a hostname from the ircbot, do

export EC2_HOST=the.host.compute.amazonaws.com

and they can run, in the toplevel directory:

super/scripts/ec2-build.sh build_and_report

or any other build command. which reminds of build_and_report, and explains the rsync backwards after the build is finished.

 task :resources do
 #great
 test.using :fail_on_failure=>false
end

def report_failures
failed_tests = projects.map { |project| project.test.failed_tests }.compact.flatten 
report_files = []
projects.each { |project| project.test.failed_tests.each do |failure|
 report_file = File.join(project.test.report_to.to_s, "TEST-#{failure}.txt")
 report_files << report_file
end unless project.test.failed_tests.nil?
}

projects_with_tests = projects.reject { |project| project.test.tests.nil? }.join(", ")

puts "ran tests from #{projects_with_tests}"
unless failed_tests.empty?
 failed_tests.each { |failure| puts "fail: #{failure}" }
 report_files.each do |report|
   FileUtils.cp(report, project('toplevel')._("reports/#{File.basename(report)}"))
 end
 fail "you have #{failed_tests.length} test failures"
end
end

task :build_and_report do
task(:build).invoke
report_failures
end

that is, 1) run all the tests even if there is a failure, but 2) fail the build at the end in case there were failures, 3) report on the failures, 4) put all the failure reports in toplevel/reports/

enjoy!

clouditude

2009-05-03T18:06:00.017+01:00

there's no easy way to share ebs volumes between accounts, and there are no corporate ec2 accounts. and I don't want to share my personal login. and people don't want to do mystifying stuff. so a shell script to set the thing up:

#!/bin/bash
set -e
if [ -z $EC2_PRIVATE_KEY ]
then
 echo "you must set EC2_PRIVATE_KEY" >> log.txt
 exit 4
fi
if [ -z $EC2_CERT ]
then
 echo "you must set EC2_CERT" >> log.txt
 exit 2
fi

export EC2_URL=https://eu-west-1.ec2.amazonaws.com
export EC2_ZONE=eu-west-1b
echo "getting volume" >> log.txt
export EC2_VOLUME=`ec2-describe-volumes |grep $EC2_ZONE | grep available | head -1 | cut -f 2`
if [ -z $EC2_VOLUME ]
then
 echo "no volume available in $EC2_ZONE" >> log.txt
 exit 1
fi
echo "volume selected: $EC2_VOLUME"  >> log.txt
echo "starting instance in $EC2_ZONE" >> log.txt
export EC2_INSTANCE=`ec2-run-instances ami-4acfe73e -t c1.xlarge -g default -g "ssh allowed" -k super-key -z $EC2_ZONE | grep INSTANCE| cut -f 2`
echo "instance started $EC2_INSTANCE" >> log.txt
while [ -z $EC2_HOST_NAME ]
do
 sleep 3
 export EC2_HOST_NAME=`ec2-describe-instances $EC2_INSTANCE | grep INSTANCE | cut -f 4`
 echo "host name is $EC2_HOST_NAME" >> log.txt
done
echo "attaching volume" >> log.txt
ec2-attach-volume $EC2_VOLUME -i $EC2_INSTANCE -d /dev/sdf >> log.txt
sleep 1
echo "running setup script" >> log.txt
ssh -o StrictHostKeyChecking=no -i ~/ec2/super-key.pem root@$EC2_HOST_NAME "mkdir -p /mnt/super; mount /dev/sdf /mnt/super; /mnt/super/setup.sh" 2>> log.txt
echo "great success" >> log.txt
echo $EC2_HOST_NAME

but, I don't want to be running shellscripts all day. so an ircbot


connect(Host, Port) ->
  ets:new(instances, [named_table, public]),
  {ok, Sock} = gen_tcp:connect(Host, Port, [{packet, line}]),
  Ec2Controller = spawn_link(ec2, loop, [Sock]),
  register(ec2_controller, Ec2Controller),

bla bla bla

respond(Nick, Channel, Sock, ["!"|TheRest]) ->
  case TheRest of
          ["hello"] -> irc_privmsg(Sock, Channel, Nick ++ " ! hello");
          ["palkka"] -> irc_privmsg(Sock, Channel, "9,5k nettona");
          ["halp"] -> irc_privmsg(Sock, Channel, "use 'joe ! start' and 'joe ! stop' to control your ec2 server"),
                  irc_privmsg(Sock, Channel, "use 'joe ! check' to see your current status");
          ["check"] -> check_ec2_status(Nick, Sock);
          ["start"] -> io:format("start received~n"), start_ec2_server(Sock, Nick);
          ["stop"] -> stop_ec2_server(Sock, Nick);
          _ -> 0
  end;

and then...

start_ec2_server(Sock, Nick) ->
  Status = ets:lookup(instances, Nick),
  case Status of
          [] -> ets:insert(instances, {Nick, starting}), Ec2Controller = whereis(ec2_controller), Ec2Controller ! {start, Sock, Nick};
          [{Nick,Value}] -> case Value of
                          starting -> irc_privmsg(Sock, Nick, "plz hold I am already processing a request from you.");
                          {running, ServerName} -> irc_privmsg(Sock, Nick, "you already have instance " ++ ServerName)
                  end
  end.

stop_ec2_server(Sock, Nick) ->
  Status = ets:lookup(instances, Nick),
  case Status of
          [] -> irc_privmsg(Sock, Nick, "you dont have a running server.");
          [{Nick,Value}] -> case Value of
                          starting -> irc_privmsg(Sock, Nick, "your server isn't even started yet.");
                          {running, ServerName} -> Ec2Controller = whereis(ec2_controller), Ec2Controller ! {stop, Sock, Nick, ServerName}
                  end
  end.

check_ec2_status(Nick, Sock) ->
  Status = ets:lookup(instances, Nick),
  case Status of
          [] -> irc_privmsg(Sock, Nick, "you dont have a running server.  use 'joe ! start' to start one.");
          [{Nick,Value}] -> case Value of
                          starting -> irc_privmsg(Sock, Nick, "your server is starting, I will notify you when it is available.");
                          {running, ServerName} -> irc_privmsg(Sock, Nick, "you have a running server at: " ++ ServerName), irc_privmsg(Sock, Nick, "you can use 'joe ! stop' when you don't need it anymore.")
          end
end.

yeah. finally:

loop(Sock) ->
      receive
              {start, Sock, Nick} -> io:format("received start~n"),
                      start(Sock, Nick),
                      loop(Sock);
              {stop, Sock, Nick, ServerName} -> io:format("received stop~n"),
                      stop(Sock, Nick, ServerName),
                      loop(Sock)
      end.

stop(Sock, Nick, ServerName) ->
      os:cmd("./stop.sh " ++ ServerName),
      ets:delete(instances, Nick),
      bot:irc_privmsg(Sock, Nick, "your server has been stopped.").

start(Sock, Nick) ->
      io:format("launching instance for ~s~n", [Nick]),
      bot:irc_privmsg(Sock, Nick, "started a server for you.  I will send you the hostname when it is up and running"),
      register_server(Sock, Nick, os:cmd("./launch.sh")).

register_server(Sock, Nick, ServerName) ->
      io:format("server is ready ~s - ~s~n", [Nick, ServerName]),
      case regexp:match(ServerName, "compute.amazonaws.com") of
              nomatch -> ets:delete(instances, Nick),
                      bot:irc_privmsg(Sock, Nick, "some kind of error occurred.");
              _ -> ets:insert(instances, {Nick, {running, ServerName}}),
                      bot:irc_privmsg(Sock, Nick, "your server is " ++ ServerName)
      end.

eclipse and buildr

2009-04-28T19:05:00.002+01:00

the eclipse .classpath is a lot more involved than I originally estimated.

module Buildr
 module Eclipse
   class ClasspathEntryWriter
     alias :old_src_from_absolute_paths :src_from_absolute_paths
     def src_from_absolute_paths absolute_paths, output=nil
       old_src_from_absolute_paths absolute_paths
     end

     def lib libs
     end

     alias :old_relative :relative
     def relative path
       string_path = old_relative path
       return string_path unless string_path=~(/target/)
       @paths_written << "dummy"
       "dummy"
     end
   end
 end
end

task :rename_target_to_bin do
 projects.each do |project|
   if !project.compile.target.nil?
     bin_target=project.compile.target.to_s.sub(/target\/classes/, "bin")
     project.compile.into(bin_target)
   end
 end
end

in brief, I don't want target anywhere in my .classpath file. one of the issues is that compile.target and it's friends have already been invoked and appended into the dependencies before my task gets run. and that also messes with all the uniq's that happen later in there.

maybe I'm halfway done so far. and maybe I can take fold all of the above into a single block later on.

the javac on jdk1.7.0 b54 is really fast compared to 1.6u12. b55 and b56 have a broken headless which doesn't work well for remote builds.

one more spell

2009-04-26T19:00:00.002+01:00


test.using :stripedjunit

class StripedJunit < Buildr::JUnit
  def run(tests, dependencies)
    passed=[]
    max_thrads=4
    if tests.length<100
      max_thrads=1
    end
    threads = (0..(max_thrads-1)).collect { |count|
      Thread.new(count) { |partition_number| teh_tests = tests.select{|test| test.hash%max_thrads==partition_number};  passed << super(teh_tests, dependencies) unless teh_tests.empty? }
    }
    threads.each { |t| t.join }
    passed.flatten
  end
end

Buildr::TestFramework << StripedJunit

you get the idea.

ruby duby duu

2009-04-26T17:34:00.005+01:00

as a french guy pointed out in 2000, ruby's autoload is not thread safe, and neither is require.
there's been some discussion about it. but it doesn't seem to bother normal people.

for people using rake's multitask, autoload fails rather deterministically. two similar tasks, a natural parallelization, will block on the same dependency task, and consequently unblock at exactly the same time, launching at the exact same time. ka-boom.

you could synchronize require yourself with just this monkey patch:

module Kernel
@@lock=Mutex.new
alias :old_require :require
def require requirement
@@lock.synchronize do
 old_require requirement
end
end
end

autoload however is a different beast. autoloads are already done before your buildfile starts to execute, you can't introspect existing autoloads, and if you could, you can't require the autoload stuff eagerly because you might not be in state that allows the specified module to be required. further, since autoload is in variable.c in MRI and somewhere else in JRuby, you'd be fiddling with stuff you're not meant to touch, even ruby wise!

personally I've been using this monkey patch for Antwrap, and then avoid using parallel for the eclipse task:

module Buildr
module Ant
@@lock=Mutex.new
def ant(name, &block)
  options = { :name=>name, :basedir=>Dir.pwd, :declarative=>true }
  options.merge!(:logger=> Logger.new(STDOUT), :loglevel=> Logger::DEBUG) if Buildr.application.options.trace
  Java.load
  ant_project=create_project(options)
  ant_project.tap do |project|
    project.project.getBuildListeners().get(0).
      setMessageOutputLevel((Buildr.application.options.trace && 4) || (verbose && 2) || 0)
    yield project if block_given?
  end
end

def create_project(options)
  @@lock.synchronize do
    Antwrap::AntProject.new(options)
  end
end
end
end

god speed.

and I use this for my eclipse layout:
task :bin_target do
projects.each do |project|
bin_target=project.compile.target.to_s.sub(/target\/classes/, "bin")
project.compile.into(bin_target)
project.test.compile.into(bin_target)
project.resources.filter.into(bin_target)
project.test.resources.filter.into(bin_target)
end
end

task :eclipse => :bin_target
and finally:

Buildr.application.options.rakelib="super/tasks"
mkdir super
mv buildfile super/buildfile.rb
echo "require 'super/buildfile'" > buildfile

heads up display

2009-04-24T07:41:00.004+01:00

cobertura finally came out with 1.9.1 so you don't have to build it yourself or use some alpha maven plugin. hooray!

then some guy released a patch that apparently does what you always wished cobertura would do: tell you which test method and class exercised which line and which branch. combine that with the excellent stuff in last year's gtac keynote.

instrument your application, listen in on wicket component instantiation, record coverage, cross reference the data with your test coverage report. show data about the tests or lack thereof in a hover on each component of your web page. while you're at it, why not add coverage reports of software in m2 repositories, next to the source jar. then you can see which of your dependencies are tested.

how high

2009-04-18T07:34:00.004+01:00

so high that I can kiss the clouds

the story so far:
our project has about 7500 unit tests.
it takes too long to build
refactoring the build requires you to:

run the build
see it in eclipse
see it in idea
repeat

so a catch22 in the current context. to refactor the build into something you can work with, you need a lot of resources, which you don't have. more hardware to the temporary rescue!

faster computers come in two flavors: faster cpus with more memory bandwidth, and more cores. the former is available in roughly 4x the current speed in the W5580, and 2x the current speed in more available iron. 8 cores is the reasonable upper limit of cores discounting HT. compared to the current 2. but already using 2 somewhere else leaves the two at home available for anything else.

I should mention that during the build we'll want to access private m2 repositories as well as some webservices on our private network.

getting clunky desktop computers or noisy server iron in our workspace are not practical options. so we're left with:

our own servers in a datacenter
servers in the customer datacenter
virtual servers in a cloud

the epigraph will tell you what I end up with... at first I was very skeptical about anything off site, for two reasons: 1) VPNs and 2) the overhead of getting the buildable stuff to the server over the network. further, getting dedicated hardware for this and coordinating it between 20 odd developers would be a huge hassle and the requirements might change quickly. if the cloud option worked reasonably, it would be our primary choice. the cloud could further be broken down into a public cloud, and our own cloud based on eucalyptus.

the geographically closest thing amazon ec2 has to our site is eu-west-1, which seems like a long ways away from us. but rsync really works miracles. with rsync -rt --delete --delete-excluded --exclude target --exclude .svn --exclude junit* blahblahblah. rsync between typical svn revisions takes about 3 seconds. and when you --delete-excluded you don't have to run clean on your build either because it will essentially remove the target directories. btw. we had a java package named reports, if you have a package named junit, target or reports, mind this. if you're unlucky your build might pass without a specific package.

rsync should work great also when you're going to make sure stuff works in eclipse and idea, as you can make the changes on one machine quickly, then rsync them to the other to see if they work in the other IDE aswell.

the following voids your warranty. the VPN. given that when you start the remote build, you are inside the private network, the firewall and so on, you don't need to connect your remote server to the vpn, you can instead proxy everything through your own computer.

start an incoming ssh tunnel into your own computer's ssh server:
ssh -R 50505:localhost:22 blahblah.compute.amazonaws.com
then, on the remote host connect to your own computer through the tunnel you just created, and specify a DynamicForward:

ssh -Nf -D 30303 -p 50505 localhost

(Nf goes to the background)
this starts a socks4 proxy on port 30303 of the remote computer (bound to localhost) that proxies to the private network you are on.
now you can use tsocks which hasn't been developed since 2002, but works quite nicely to transparently force your build to use the proxy.

export TSOCKS_CONF_FILE=$HOME/tsocks.conf
tsocks buildr clean build

before logging off you need to kill the background ssh you started:
updated
~~kill -9 `lsof -i @localhost:50505 -t`~~

in .ssh/config:

ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p

and on the commandline:

ssh -p 50505 localhost -O exit

if you're using ProxyCommand and nc, you might need to specify -q 0 or -w 1 in there to avoid leaving a dangling nc, and to fix some other problematic bits.
with tsocks, you need to configure your networks and proxies, especially match the port you opened the proxy on (30303 in this example) and the tsocks configuration, and use "4" as your proxy type, short for socks4.

if more than one person will be using the machine, you need to make sure everyone uses different ports for both the proxy and the return channel (50505 in this example)

you're now ready to choose your cloud servers. there are two interesting ones on ec2, the medium high-cpu, and the xlarge high-cpu. in short they are 2x 2,33ghz e53XX series with 1,7gb ram and a 32bit kernel (and thus jvm) and 8x the same cpu, with 7,5gb ram and a 64bit kernel.

the prices are $0.22 and $0.88 per starting hour, respectively. (so you'll want to start one server when you get to work and shut it down when you leave for the day)

if you can get 4 people on your team to use the same machine, you should not do much worse on the 8way machine than the 2way machine even if everyone starts their build at the same time. the only performance loss would be due to the 64 bit stuff, and in your build compressed oops would likely do nothing to help you as it is not active in interpreted code.

but when the 4 people don't start their build at the same time, you should always do better on the 8 way machine assuming you can get your build to use more than 2 threads for any amount of time. I've gotten our fairly serial build to momentarily use 75% of cpu on an 8way box, so it's using 6 threads at that point.

the total build time difference is 3 vs. 5 minutes of wall time on the 8 and 2 way ec2 boxes. overall, the build uses 2,6 times user+sys vs. real time on the 8way box. but as part of refactoring the build, this could go way up. it's already somewhere south of 1.5x with just a single threaded build as even the c1 compiler now uses a background thread to compile bytecode to native.

last but not least, on ec2 you can use ebs to create your own persistent filesystem on which you can store your m2 repository, your JDKs, your .debs, jruby with the appropriate gems, and a prepopulated build directory.

you can leave everything except the build on your ebs volume. I've made a copy of the initial build onto local storage just to be sure performance wise. this also simplifies things when you have several people using the same server, as they can share everything except the build directory. you might want to sync the build directory back to ebs every now and then, especially if you've made big changes, but just having the bulk of it persisted will save you a couple of minutes at start up, and a few cents each day. with eucalyptus you might be able to something similar for persistent storage.

enjoy your flight.