thatmattbone.com -- blog

wordpress transition imminent

2008-09-22T23:43:00.004-05:00

I'm switching to wordpress. I'm going to try to redirect weblog.thatmattbone.com to thatmattbone.com (which is already up and running) so as not to destroy the RSS feeds of my three readers. If I fail, though, just RSS me at thatmattbone.com now...if you want...or don't.

Sept. 4th

2008-09-06T02:11:00.003-05:00

September Fourth was the two month anniversary of my vegetarian experiment. I'm eating better, in better shape, and I've saved money by eating out less (not to mention the great new dishes I've learned to cook). All told, the vegetarian experiment has been a great time and very pro-veggie. I've really no desire to eat meat, but I am a bit worried about Thanksgiving and Christmas. I think I'll stay vegetarian for the indefinite future.

Bloomington Adventures

2008-08-31T16:07:00.005-05:00

Today I struck out for a ride and got a bit lost. It ended up being 47.4 miles, which is well beyond what I'm comfortably capable of. A map:

Rockport Popcorn Misadventures

Share your bike routes @ Bikely.com

I started out on Rockport which is bike heaven and hell. There are some tremendously steep hills, but most are quite short. The flipside is the descents are terrifying. However, the views cannot be beaten. This may be the road that shows up in Breaking Away. It sure seems like it, and it goes by quite a few quarries.

Riding through the country is great. There are many things to see that I'm not used to. First, I saw a bunch of silly cows standing (and some appeared to be squatting) in a muddy little pond. They had curious looks on their faces. Then I saw a huge rooster run across the road. I stopped a bit later to take a breather and noticed a sign that read "God Bless America and Homemade Pies." mmm....pie.

At one point in the ride I was running short on water, so I stoped at a fire station and they let me fill up my bottles (I figured a fire station has water). The fireman were nice, but gave me strange look when they found out how far I was from home. At my next break, a couple on bikes passed me, and I caught up with them a few miles up the road looking at a map. I stopped to chat and make sure I was going the right direction (I need a map). They had also made a wrong turn, but seemed to a bit more seasoned than I. We parted ways and I stopped one last time at a gas station for some gatorade (I was pretty miserable at this point). A few miles later I was home, writing this incoherent posting. I need a nap.

Update: 2800 feet of elevation change! A bit more, and I'd be ready for the hilly hundred.

erlang-in-lisp user manual now useful

2008-08-18T14:02:00.002-05:00

Most of the Erlang-in-Lisp issues discussed here and on the mailing list this summer have been rolled into the user manual in some form. Get it here in pdf format or here in HTML.

links, timeouts, and refactoring

2008-07-30T10:55:00.002-05:00

First off, I have added some support for timeouts in receive blocks (in erlang this is done with the 'after' block). For now this is only supported on the unix-process implementation, but should be easy to implement. While I know that the timeouts are 'soft' I do wonder how they are implemented in Erlang. For example, does the timeout usually just measure the amount of time spent blocking, or does it include the amount of time spent searching messages already in the process' mailbox (which could be lengthy if the process has accumulated large amounts of messages for whatever reason).

There is also some preliminary support for process linking. More specifically, if one process is linked to another, it behaves like an Erlang system process and is notified via a message if the other linked process experiences an error.

Also there has been some seriuos refactoring. Much like the Closette implementation discussed in The Art of the Metaobject protocol, the implementation of erlang-in-lisp is separated into a thin macro-layer and a heavier layer based on generic functions (thus most behavior can be overridden if necessary). The thin macro layer is where something like muproc compatibility would be implemented. It is essentially the public facing side of erlang-in-lisp. The generic function layer is for those wishing to extend the functionality of erlang-in-lisp by, for example, introducing a new concurrency strategy.

On Monday the erlang-in-lisp operation is moving to Bloomington, Indiana. I'll even be able to write code in my new backyard (or on top of my in-apartment washer and dryer!).

midterm report

2008-07-09T12:07:00.003-05:00

What has been accomplished?
EiL has an implementation of the actors model as illustrated in Erlang. New processes can be spawned, and with the send() and receive() primitives these processes can communicate via message passing. While the spawning of processes is currently limited to a single machine (i.e. not distributed), distribution is possible if processes are spawned by hand on remote machines. Erlang style process registration (mapping a particular process to a name) is supported and is in fact written in EiL using the send() and receive() primitives.

Currently EiL contains two concurrency strategies: unix processes and threads. The decision about which concurrency mechanism to use must be made when EiL is started, and thus the two concurrency strategies cannot be mixed at the moment (some changes to the API should make this possible, and this should be a goal). However, programs written for EiL are unaware of the underlying concurrency mechanism. For example, to run the ping-pong example with unix-process based concurrency we simply:

ERLANG-IN-LISP> (toplevel :process-class 'unix-process)
ERLANG-IN-LISP> (ping-pong 3)
......
ERLANG-IN-LISP> (toplevel-exit)

To use thread-based concurrency we need only change the keyword to the toplevel function:

ERLANG-IN-LISP> (toplevel :process-class 'thread-process)

Thus the goal of pluggable concurrency as described in the original proposal has been accomplished. To a certain extent, so has the goal of pluggable messaging. However, messaging is tightly coupled to the concurrency mechanism (i.e. processes serialize messages and communicate via sockets while threads simply exchange references to the messages in memory). Truly pluggable messaging is best left to each concurrency strategy (as opposed to introducing another layer of abstraction). Thus unix-processes may choose to communicate via secure or unsecure sockets while these options would be unavailable to threads.

What remains to be accomplished?
Much remains to be accomplished. Most important is linked processes. I believe this can be accomplished by relying on the Common Lisp condition system. Processes will maintain a link set as in Erlang, and conditions will be handled and forwarded via the messaging primitives to interested processes. While I have sketched out a plan to this end, no code has yet been written.

True distribution including the remote spawning of processes remains as well. The main sticking point is how to serialize closures and ship them off to a remote system. While we can easily send a function name and the required arguments to a remote process and hope that the remote process knows about that function, this is not ideal (and, in fact spawn() only takes a thunk at the moment, so we'd need to figure out how to serialize an anonymous function to capture these arguments...that or have two separate APIs like spawn() and remote-spawn()). We may also wish to generate bindings for the erl_interface so that EiL nodes and code written in EiL can play nicely in an Erlang cluster.

Finally, our ultimate goal of compiling Erlang to EiL remains. To this end, we should focus on compiling Core Erlang only. It may be helpful to develop 'core EiL' and then build a Lispier macro-based front end for user consumption.

serializing continuations

2008-07-07T15:58:00.002-05:00

Some people in the weblocks community have been thinking about some of the things I've been pondering. No real answers. I need to play with the continuations in arnesi and cl-cont, but if we can't move them between address spaces (either distributed or in different unix processes) it's all for naught.

matching Erlang's receive semantics

2008-07-02T13:59:00.004-05:00

On Sunday and Monday I patched our version of receive. It now matches the Erlang semantics of blocking until a suitable match is found. This works in the unix-process and thread based concurrency models (it required changes to the API, but shows how we should grow the pluggable concurrency API in an organic fashion (and it shows that it actually works)). Receive timeouts are not implemented, though, so that remains.

My biggest worry at the moment is the half-assed serialization strategy I have for shipping off PIDs and how this isn't working for spawning remote processes. Sure I can ship off a function name, arguments, and invoke on a remote server no problem...but this doesn't really match the Lisp semantics (or common sense semantics). I really need some closure serialization ala Termite.

the great CLOS experiment

2008-06-20T14:21:00.003-05:00

I just committed the last touches on some pretty major changes in the erlang-in-lisp epmd branch. I was distracted from my linking/registering work on Tuesday by the fragility of the original design. I had an idea to use CLOS and hoped this would help achieve our goals of pluggable concurrency and pluggable networking as set out in our proposal. I think it worked out; the two branches (main and epmd) currently have the same functionality. However, the epmd branch has an additional layer of abstraction. The concrete implementations of send/spawn/receive can be switched an runtime, and if the appropriate methods are implemented communication should be possible between implementations (currently there is only one implementation).

Next steps are to finally publish my todo list, test distribution (the way things are setup now, it should be working already), and return to my work on registering/linking/spawning remote processes. The CLOS experiment did suck up some considerable time, but I'd still like to finish remote spawning and registration by monday.

One final note, this is really the first time I've used CLOS extensively. My initial attraction to Lisp and other languages with an appreciation for functional style came out of a frustration with the tedium of Java. I guess I closely associated OOP with Java and was never very interested in persuing an OO style in these languages. Working with CLOS and generic methods, however, really is OO done right and appropriately melds FP and OOP. For example, specializing on two variables in method feels very functional (actually it makes me feel like I'm using Haskell...is that a good thing?), but we can still have side-effects within the objects themselves. I'm rambling; I like CLOS.

sname, name, self, node

2008-06-17T01:57:00.003-05:00

One of the things I've always been unclear on is how a server binds to particular address. This became apparent today in my erlang-in-lisp work with iolib and erlang itself. I had been messing around with distributing the ping-pong example last week and things were not going as smoothly as I'd initially hoped. It was all coming down to dns, the hosts file, and binding the server to the wrong address. Today I tried the example in erlang, and my mistake became clear when I read up on the behavior of erlang with regards to 'short names' and 'long names' (it seems erlang is fond of adopting somewhat unusual terminology).

It turns out that some of my thinking last week was unacceptable; I'd been expecting self() to return information that could only be retrieved after a connection to a particular process. This is impossible given the semantics of self(). Furthermore, I'd forgotten about the node() BIF in erlang. By matching the behavior of the sname and name command line parameters in erlang, all has become clear. The user specifies the hostname (or sometimes the IP address) to which the erlang (or erlang-in-lisp) process should be bound. While this may seem a bit tedious at first blush, it really does (along with the cookie system) prevent unsuspecting users from casually opening up erlang processes to the outside world. This will be (has been) duplicated in eil (though there is still a runtime typing problem keeping things from working properly at the moment...hopefully trivial).

Tomorrow I'm planning to swing by Loyola to pick up my 'alumni card'. I hope it allows me into the library (it should) to check out "Unix Network Programming" as I think this reference will be useful in the coming days. I'm planning to tackle distribution and linking processes simultaneously this week. I'm also hoping to be diligent in writing unit tests as I've found myself retyping the same test code at the repl over the last several days.

I've also been wondering about an erlang grammar. Does one exist and could be use it in something like a lisp packrat parser? Off to bed now...I hope this post does not sound like gibberish in the morning :)

ping-pong

2008-06-05T17:39:00.003-05:00

The ping-pong example is working with one caveat. Matches against patterns with just one atom must still be wrapped in a list. This is annoying, but we can fix it.

This first step happened a bit more slowly than I'd like, but I can see where we're going now. We're on our way.

eil: top-level

2008-06-05T13:14:00.002-05:00

One of the things we may want to think about for erlang-in-lisp is a top level. In erlang, one can send and receive messages right at the REPL because it is a process with a mailbox. For us, we have to do some additional bookkeeping (thus the standard REPL is not an erlang-in-lisp process). This should be easy to implement and handy to some.

simple example working

2008-06-03T22:36:00.000-05:00

I have a very simple example (sort of) working in erlang-in-lisp/examples.lisp. I did a quick hack to integrate fare-match into send/receive. The matching really isn't working because the message is being mangled before it is sent over the socket. More tomorrow...

process graphs and trees

2008-06-03T10:08:00.002-05:00

One of the interesting things about processes in Unix is that they are tree structured. This is easily seen with the pstree command; init is the root of all processes. Several things, can, however go wrong with this tree. Defunct or zombie processes show up when a child process dies and the parent is unaware. Likewise, orphan processes occur when the parent dies and the child is still running. Despite the names, neither is a tragedy. Zombie processes are removed from the process table when the parent dies, and orphans become the children of init. Thus, the strict tree structure is maintained.

Compare this to a multithreaded system. Threads are not forked from other threads. They have no explicit parent other than the enclosing process or address space. If we treat threads like objects in a garbage collected OO environment (i.e. we need to have a reference to a thread in order for it to survive), we quickly see they are not tree structured like Unix processes. One thread can create five new threads, pass the references to other threads, and disappear. There is no parent/child relationship, and the newly created threads needn't care about the status of the thread in which they were originally created. The representation of these relationships is not a tree, but rather a graph that evolves over time.

What does this have to do with erlang-in-lisp? The light-weight processes in Erlang are much closer to the graph-nature of multithreaded systems, and with our current approach, we will have to map this graph onto a the Unix process tree.

The simplest approach is to have each process wait for its children to complete before exiting or returning. This should probably be our first approach. However, in larger or long running systems, this could become a burden, with many short-lived processes hanging around and taking up memory (though not cycles) while their longer-lived children run to completion.

make-socket-pair -- remember to flush

2008-06-03T09:29:00.000-05:00

As I suspected, make-socket-pair worked just fine. I discovered this yesterday when I set the input and output buffer sizes to 1. Then, I had the sinking sensation that while I'd already tried flushing the buffer, I had been flushing the wrong side.

I think one of the side items that should come out of erlang-in-lisp is a simple iolib tutorial.

eil: week one report

2008-06-02T10:07:00.000-05:00

Well the first week of SoC has come and gone. I'm disappointed with my progress. I had hoped to have some simple ping/pong examples working by the end of the first week, but I've yet to resolve some issues with iolib and message passing. Realistically, I think I can fix these problems and have the examples working today or tomorrow.

My problems stem from local domain sockets in iolib. I can express what I'd like to do in C with socketpair/fork/read/write (and get it working), and the iolib API should, of course, make this much simpler. But for whatever reason, the streams created by make-socket-pair, don't seem to be connected; when I write to the one side, I cannot read from the other. Late last night I started to suspect this was some buffering issue, and fired up the slime inspector on these objects. I didn't learn much. I also looked closely at the iolib code, and everything seems to be in order. The problem is on my end. My next step is to try using them as binary streams (I've been treating them as character streams) ala the tcp server in philip-jose. Either that or raise hell in #lisp.

I view these hiccups as noise rather than signal, and I still think we're on track for an interesting summer. I have a much more fundamental question about our approach in the next posting.

eil: initial progress

2008-05-25T08:58:00.000-05:00

Though the summer of code does not officially start until tomorrow, I'm pleased to report that we have made some initial progress.

To start, we have common-lisp.net site with a git repository. The site has instructions on how to check out the code. Our (mostly empty) erlang-in-lisp project works with asdf and is easily loaded with slime-load-system. Some dependencies are kept in version control, but most are checked out with the makefile in the deps directory. This makefile is pretty repetitive, and I'm thinking of replacing it with something a little more maintainable (perhaps an scsh script).

I've spent some time thinking about our intial fork-based concurrency approach. Though there are portable libraries for some posix operations like file and directory management, I've been somewhat surprised that there isn't a Common Lisp implementation for all of the posix functionality (though it seems there have been some thoughts in this direction). Still we should be able to rely on the implmentation specific posix packages or roll our own with cffi for now.

Also, there is a little test-misc.lisp file serving as a catch all for miscellaneous tidbits. Right now there are some tests from cl-muproc in there that I think we should get working with our fork framework and IPC messaging by the end of the week.

I still have many questions, though, and I'll be posting a list of them soon.

Google Summer of Code 2008

2008-05-24T20:10:00.000-05:00

I'll be participating in the Google Summer of Code working on the erlang-in-lisp project. I'll be tracking my progress on this blog (so things will get noisier). I had originally created a separate blog, but Google decided it was a spam blog, and I don't really see much point for this kind of separation at the moment.

earthquake!

2008-04-18T15:48:00.000-05:00

I spent a significant portion of my undergraduate career fooling around with the outdated seismology lab in the Loyola Physics department. Though I did eventually connect the seismometers to a computer and was able to record activity, they usually are not running as there is currently no student interested in checking in daily. However, we do turn them on every now and then for a lab that the physics students do. So, we hooked everything up yesterday, and BAM, the biggest earthquake in many years hit Chicago this morning. We captured the quake, and I am positively thrilled. Two traces are below. The times are way off (clock drift), but you can clearly see the quake on the 14:00 line. Also, apparently the after shock was centered in "Bone Gap, IL"! Ha! Amazing.

off to indiana

2008-04-03T20:30:00.000-05:00

A few weeks ago I decided that I'll be off to Indiana University in the fall to start a phd program in computer science. I'm very excited about the program, and I'm also excited about living in Bloomington. It seems like a great little town (that's a bit warmer than Chicago!).

I've already started to study for my free shot at the written qualifiers. One of the suggested books was compared to what? by Gregory Rawlins, a professor at Indiana. Unfortunately the book is out of print (I picked up a used copy), but the first chapter has one of the most insightful introductions to algorithms analysis. It does an excellent job of explaining why we care about an algorithm's running time and how our 'model' (i.e. what operations we are concerned with, what we are trying to optimize for, etc) greatly influences the analysis. I've stared at plenty of proofs and suffered through some big-oh notation, but it's nice to step back and see what the point is. It actually makes algorithms more exciting.

pipe test

2008-01-23T16:07:00.001-06:00

We've created an ETL Yahoo pipe that aggregates the content of members' blogs tagged with 'etl.luc.edu'. What I'm wondering is how quickly the pipe gets updated, and where things get cached along the line...

const id 1 2

2007-12-16T00:21:00.000-06:00

While playing around with the 99 problems in Haskell, I came across this function const that takes two arguments and returns the first without ever evaluating the second. For example:

const 1 2

returns 1. Now that is straightforward enough, but then I noticed that:

const id 1 2

returns 2, and, indeed const id x y always returns y. I was confused. id is the identity function and quite simply returns its argument untouched. After some searching and thinking, I realized how this works. If we consider the example above, it turns out that const id 1 discards the 1 and returns the id function. This function(id) is then evaluated with the remaining argument, 2, and, presto, we get 2 as the answer. Tricky, but not exactly readable. A more human friendly form is probably:

flip const 1 2

A simple description of the flip function is here. Hooray for first class functions and lazy evaluation!

s-expression/xml isomorphism

2007-09-05T11:29:00.000-05:00

After some discussions with Dr. George recently, I was interested in exploring the relationship between xml and s-expressions. This has certainly been done before, but I thought I would try my hand at it. In the end result, I'm able to convert an s-expression such as:


(a :a1 "test" :a2 "test2" "mycdata" (b :a3 "c"))

to xml:


<a a1="test" a2="test2">
mycdata
<b a3="c">
</b>
</a>

I do so with a somewhat strange but more explicit intermediate format:


(:element-name "a" :attributes (("a1" . "test") ("a2" . "test2")) :cdata nil
                   :children (:element-name "b" :attributes (("a3" . "c")) 
                               :cdata "My CDATA" :children nil))

This shows pretty clearly how a single namespace xml document can be represented with an s-expression. What I'm wondering now is if I can go from the intermediate representation to something like YAML or JSON. George and I have been calling these 'tree languages'; I wonder if some sort of overarching equivalence for these 'tree languages' can be shown?

I don't think this has much practical use, but the code is here.

items on my neighbor's roof 2

2007-09-05T11:24:00.001-05:00

From Aug 24th:

The chair, lid and shelf have been removed. This was most likely caused by high winds.

I also have unconfirmed reports of children(!) on the roof.

items on my neighbor's roof 1

2007-08-19T10:42:00.000-05:00

For the past several months I've noticed a bewildering and frequent changing collection of objects on the roof of my neighbor's garage. I will now catalog these.

Inventory:

Trash can lid
Plastic shelf
Chair
Mop
Tire/wheel/pole
Bucket

Changes:

Chair has been positioned upright so as to allow one to sit.