--Mike--

Virginia

2008-07-24T14:00:00.001-05:00

It's time for T-Shirts again. ;-)

The future is Usenet, all over again

2008-07-23T10:52:00.003-05:00

With the rise of Twitter, and the subsequent introduction of Laconica to federate things, I think I'm beginning to see the rhyme of history. Eventually we'll want to replicate everything in Usenet, but just a bit different

Anonymous posting will be prohibited
Tags will replace the hierarchy of groups
Digital Signatures will prevent forgery
All posts will have URIs so we can still link to them.
We still won't be able to markup hypertext. (a pet peeve)
Data will be streamed instead of batch mode.

There were a lot of things to like about Usenet

Push model saved bandwidth
Aggregation was built in
Group hierarchies helped increase signal to noise
It was federated from the start
Binary attachments were supported

So, we'll get some new hybrid which will help us adapt to the contemporary demands of the internet. I believe that a new push infrastructure is on its way. If done right, we could even get rid of Email and the spam problem, but that's story.

DNS -- It's worse than I thought - Technical version

2008-07-15T09:45:00.001-05:00

As most of you know, DNS (Domain Name Service) is the system by which names such as www.apcu.org get translated into numbers that the computer can use to connect to systems on the internet.

There is a very deep and serious flaw in design of DNS which affects virtually everyone, which was recently discovered. The technical details of this flaw are still being kept secret, but it has been disclosed that part of it involves the nature of requests from DNS clients when they do a "recursive" lookup. Here's a link that explains the details: http://www.inetdaemon.com/tutorials/internet/dns/recursive.shtml

It appears that if you know what someone is asking for, you can answer their next question, even if you're not supposed to. This means that you can't trust the answers from your DNS if it happens to be one of the vulnerable ones. (The ones that haven't been patched yet)

The situation is complicated, and made worse by the NAT (Network Address Translation) that we all use to share an internet connection among more than one computer. All of those Linksys, Belkin, Dlink, etc… devices we bought make it easier to guess the next question… if your DNS is behind one of them. This means that anyone with a Windows Domain, or Linux Server who has their own DNS now has to consider moving the DNS back outside the NAT… which isn't a nice prospect.

The DNS clients built into ALL of our PCs need to be updated as well. This means doing Windows Updates for ALL of your PCs

Fortunately, there is a tool to help you test your system (not just your DNS, but the whole chain) to see how you will fare (it's not a guarantee, but a guideline) available online at:

http://www.doxpara.com/

It's on the right side of the screen… conviniently labeled "Check My DNS".

It turns out I'm ok at home because ComCast has patched the servers I use… but I've got a ton of work to do at work.

There will be full public disclosure of the vulnerablity on August 6, 2008. I strongly urge you to use this time to get ALL of your systems tested and patched appropriately before the hackers of the world learn the details.

I'll be spending a lot of time on this… and so will most of you, it's far better to do it now than to have to clean up a mess afterwards. (When you can't trust your DNS to get the patches, etc!)

Exim is a bit nuts

2008-06-23T12:28:00.003-05:00

I'm using Nagios to monitor servers, but was having some trouble getting emails to exit the Ubuntu JeOS server that I had set up to run it under VMware. (Most of my stuff is Windows, and Nagios is a linux program). It turns out that a program called exim is used to send emails, and it's a bit crazy.

All attempts to send email to myself resulted in replies which contained the error:

all relevant MX records point to non-existent hosts

Thanks to this entry on the PkgExim4UserFAQ, I was able to get a clue

A probable cause for this might be that all MX records for the offending domain point to site local or link local IP addresses, which are ignored by the dnslookup router to protect from misconfigured external domains. The default configuration has relaxed checking for domains that the local system is configured to allow relaying to, so adding the offending domain to dc_relay_domains will most probably help. Please note that this entry might be necessary anyway to bypass relay control for the domains in question.
Please note that no domain on the public Internet should have MX records pointing to site local or link local IP addresses, so you might check your externally visible MX records.
If this doesn't help, try analyzing the output of exim -d -bt some.local.part@the.offending.domain.example

Well, I did the requisite test, to find this among the output

ignored host mail.xxxx.com [10.0.0.x]

Ignored host?? Clearly not the same as one that is non-existent. So... the first error was a lie.

All the relevant records pointed correctly to a very much alive and well host, but exim chose to ignore it because it was local.

In order to get around this, you have to follow the suggestion of Marc Haber and tell it that it is going to relay email for your local domain (which sounds like a very bad idea) in order to get it to work.

I don't know why they did it this way, but I'm posting this here to help others figure it out.

XP era ends?

2008-06-16T15:06:00.002-05:00

Larry Dignan over at ZDnet asks:

XP era ends: Will Vista step up?

Larry assumes that Vista is inevitable, and that anyone who questions it is unpatriotic... er... just a complainer. He then goes on to the straw men:

If you’re a Vista complainer you have two options from here:

Move away from Windows completely (users try a new OS and developers jump ship).
Or shut up and go with Vista.

Now, the false choice here asserts that all non-Vista operating systems will suddenly and completely have no value once Microsoft, in their infinite wisdom says so.

There are a lot of reasons to avoid Vista like the plague. People went along with having pre-loaded Windows back when new versions were generally an UP-grade... but Vista breaks far too many things, and causes way too much grief. There is a lot of user hostility being build up over this, causing people to start searching for alternative platforms as an escape route.

Linux seems to be the platform of choice, be it on a Mac or a PC. We can all run XP under VMware if we need to run some application that can't get ported eventually to Linux. Microsoft is quite aware of the place they are putting everyone into, but really has no choice in the matter. They desperately want to keep their OS market share to help drive their Office cash cow, but know that this is ultimately a losing proposition. So they will keep the drama going, probably announcing some form of "extended support" to keep it alive for a while longer to prevent defection to Linux.

The XP era is ending, long live Linux.

166437 and counting

2008-06-08T00:19:00.002-05:00

I've managed to gather all of my photos from the last 11 years in one disk, and I've got 166,437 distinct Jpeg files, plus some movies in AVI and MPG, and a few NEF files. WOW

Mellon Collie

2008-06-06T08:27:00.003-05:00

I got a nice spiffy new hard drive to gather all my photos together on... only to find out that I may have lost a few years worth due to inadequate backups... ugh, I'm feeling a bit of Melancholia.

My main task now is to fight through this, and try to recover what I can. Ugh!

Update: I have all of them in a lower resolution format... good enough for small prints... still working on getting the originals back.

Towards a new database model

2008-06-03T23:23:00.004-05:00

Over the past few days I've been pondering the way databases get used, and I think I have a way to help make things better by shifting things around a bit. I'm going to dive right in to deep territory here...

The current crop of SQL type databases are all batch oriented. They just don't scale well because of this. We need to update the model from one of data that gets visited by the occasional query to one where the queries are always running and updating their results.

If you find yourself requerying a database without having changed the data yourself, you're wasting a huge amount of resources. You really only want to know what's changed, and it's silly to re-examine everything that isn't modified since the last run of the query.

Imagine a table in a database. If you recorded the initial state of the table, and all of the subsequent operations, you could perfectly replicate the state of the table.

If you then wrote code to implement this type of logic, it takes a lot less code to keep valid replicas. You would have a stream of changes instead of a set of facts that constantly needed to be reexamined.

If you ran a query against such a table, then ran the query against the stream of changes, you could have a running output that worked vastly more efficiently, since you only have to examine records that have changed.

I'm not a computer scientist, nor an engineer, but it seems to me that this model should be looked into a bit, and has some promise to save us all some time and effort.

Who knows, it might even save Twitter.

Dead D40... ugh!

2008-05-31T19:07:00.002-05:00

My Nikon D40 just died... it has only taken about 29000 photos since I bought it after Thanksgiving... this really sucks! I've contacted support, and will be sending it in on Monday.

The shutter is fubar... when you turn it on it says

! Error: Press shutter
release button again.

Oh well... time to dig out the Coolpix 8800 and get the batteries charged.

Flow based databases to fix twitter?

2008-05-30T21:50:00.002-05:00

Twitter is a popular web based instant messenger service, which has been having problems with scaling lately. The facts seem to indicate that a traditional Relational DataBase Management System just isn't an appropriate fit in this case.
I believe that this is a perfect case for a new type of database, and perhaps even a completely new framework of programming. I don't have a good name for it, and the ideas are still vague in my head right now, but I'll try to outline what I'm thinking of below.

I would break Twitter up into a series of tables which get distributed and replicated among a cluster of servers. The tables would relate to each other, but not in the strict atomic transaction model, but one of eventual consistency. These tables would be:

Users
Queues
Subscriptions
Content

The real trick would be to treat the tables more like queues or pipes full of data that are appended to with very low random write frequency. The changes would then be aggregated in a channel to make it easy to keep multiple copies for coping with the heavy read access from all of the clients connected at any given time.

The bandwidth external to Twitter is pretty high, because you've got lots of people with many subscribers. The amount of actual non-duplicate data is surprisingly small... and I'm guessing that it's on the order of 3kbytes/second. The real challenge is distributing this 3kbytes in a consistent and reliable manner to all the places it gets copied out.

A flow based database would be able to handle such types of loads by maintaining many local copies and keeping them in eventual consistency by tying them into a channel. This is a place where multicast might be a really good strategy, if not a straight peer-to-peer network.

A flow database could be a straight up normal table in an RDBMS, or it could be something new optimized to the task.

What do you folks think?

Oh my G-d, Scoble killed Twitter...

2008-05-30T16:49:00.002-05:00

So Robert Scoble is to blame for taking out Twitter, the all too popular instant messaging system, because he's just to gosh darned popular.

As if...

As I've stated before, the aggregate flow of all tweets in on the order of 10-20 messages per second, based on peeking at the message sequence numbers. It seems readily apparent that they've chosen the wrong architecture for this.

The tweets themselves should be aggregated with a sequence number, and user sequence number into a stream which should get copied to all of the boxes handling User Interface. Deleting a message would be handled by reposting it to the same queue with no data.

The subscription lists should be another stream.

The user database could be yet another stream.

All of those streams should aggregate out to about 10 kbytes/second. The process of splitting out the work to UI boxes is one of straight forward partitioning of the load, and maintaining a list of tweet sequence numbers for each person to see. The aggregated total of all of the three streams would sit on each UI box so they didn't have to get any of it from across the net.

That's my basic idea for scaling twitter. Comments welcome.

Tim, Noran, Virginia, Bear

2008-05-25T23:48:00.003-05:00

Here's a great photo of Tim, Noran, Virginia and Bear. We took Virginia to the Lincoln Park Zoo in Chicago to celebrate her second birthday. Tim was nice enough to meet us there for the occasion. A fun day was had by all.

Bear got to see some of his cousins.

Greta still remains missing. She was last seen near Niagara falls a few years ago.

Rememberance

2008-05-22T08:55:00.004-05:00

In Flanders fields the poppies blow
Between the crosses, row on row,

That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.

We are the dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved, and were loved, and now we lie
In Flanders fields.

Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high.
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields.
— John McCrae

This is a day to remember those who came before us, and to reflect upon the legacy they left for us.

The time is now, always now, to decide what you can do to make the world a better legacy for our children.

Hezbollah has fiber?

2008-05-20T12:29:00.003-05:00

Today's news of the unexpected is that Hezbollah, the opposition (terrorist?) group in Lebanon has it's own fiber network!

Wow!

Yet, I can't get fiber for a reasonable rate at work or at home in the worlds only remaining SuperPower... hmmmmm.

Found via John Robb.

ACL as punishment?

2008-05-18T22:37:00.002-05:00

Over at Echovar, in the midst of a post summarizing the Internet Identity Workshop:

Chris Saad injected the data portability meme into the flow and suggested personal Access Control Lists, in the form of a “Sharing OK/Not OK” check box on data you give to individuals or companies. It would be interesting to watch Robert Scoble manually configure a complex ACL on his 20,000+ friends (Scoble rushes in where Angels fear to tread).

While it would truly be torture to force a person to manually configure an ACL for 20,000 people, it doesn't have to be that way. One wrong move, and you've lunched everything.

Giving away capabilities on the other hand would be a much easier thing. You have the host environment generate a capabilities token for the piece you wish to delegate access to, then send it through email, or on a web page, or whatever the end user's security policy specifies is the right thing to do.

It would make far more sense to have a system that lets users delegate capabilities to any given part of their information, blog posts, photos, etc. The fact that you start with a model of least privilege means that you start with the most you're willing to give away, and pare down from there. You don't have to worry about giving away the store by mistake.

Yes, Access Control Lists would be punishment, but being able to give away little bits, without fear, is a quite liberating alternative.

I look forward to the future.

QWERTY Considered harmful?

2008-05-17T17:10:00.002-05:00

An interesting observation from Daniel Berger:

Larry Wall’s first rule of computer language design is, “Everybody wants the colon”³. Maybe the problem is that we just don’t have enough symbols on our darned keyboards. The result is that we’re left fighting over the scraps that QWERTY gives us, e.g. the colon. My opinion is that a limited number of usable characters limits our thinking and our expressiveness. (emphasis mine)

In my recent quest to push forward awareness of capabilities, the notion of expressiveness seems to be at the crux of everything. If you don't have a conventional way to express something, it takes a lot of work to come up with something to get your point across.

I believe that rich source code is overdue. The idea first came to me via Chuck Moore's ColorForth, but I think it could be applied in a wider array of places. The ability to simply highlight a section and make it a comment without worrying about syntax would be cool, but I'm sure there are far more powerful uses that would quickly arise, such as the ability to do literate programming, freely mixing source and documentation and content.

The arguments against any new programming technique usually tend towards the fact that pretty much any language can already express any program. These arguments always miss the expressiveness that a new language brings to making it easier to solve a certain class of problem.

Originality

2008-05-16T14:03:00.001-05:00

Quote of the day:

Originality is overrated. Clarity, especially for those of us who have trouble achieving it, is also appreciated.

That was in response to an Megan McArdle's concern her post might not be original enough. It was, and I learned a few things. I liked the CS Lewis quote in the middle.

Almost useful capabilities demo - 0.012

2008-05-15T23:07:00.003-05:00

So, I've done some more programming, and I'm now up to version 0.012. for my Capabilities demo.

The main page at http://127.0.0.1:81 is now the user page, with the protected content. You have to have a capabilities token to edit the data.

The administration page is at http://127.0.0.1:81/admin, which allows you to create and revoke capabilities, and see the current "protected content".

It's all implemented in python, in a single file, just to make it easy to demo.

The slow road to implementation

2008-05-15T16:53:00.003-05:00

There are a lot more choices to make, and details to manage, in the process of programming a web server than I would have expected. I've made a lot of decisions, trying hard not to worry too much about it, to avoid analysis paralysis.

http://warot.com/python/ contains my recent python programs. I have to name them with .txt on the end or the web server tries to run them (and fails).

So far I've managed to get up to webserver008.py, which manages to create random numbers and keep a list of them available. At the rate I'm going, I'll have something usable in a few months, which is better than never. 8)

Should you choose to actually download and run the thing... here's what it does.

In a DOS box (or your command line equivalent)

Welcome to Mike Warot's capability based security demo web server, version 0.008

You can access it at http://127.0.0.1:81
Use control-c to tell it to shut down, which may take up to 10 seconds
started httpserver...

If you then open http://127.0.0.1:81 in a web browser, you'll get a very informative message like this:

this is the default content, not served from a file
Here are the valid tokens:

Now.. for the completely undocumented and poorly written section of code... change the URL to http://127.0.0.81/token, and you'll get something like this:

0451a66530b72a980725745c39992239

Isn't that lovely? If you then go back to the home page at http://127.0.0.1:81 and refresh the page, now you'll see:

this is the default content, not served from a file
Here are the valid tokens:
0451a66530b72a980725745c39992239 [Revoke]

That's a list of all the tokens, with the ability to revoke one of them. That's pretty much the full extent of the power of this demo.

It's a list, with a undocumented, poorly designed and inconsistent UI... but it's a step in the right direction. Oh... and it's licensed with the GPL so you can fork the project. ;-)

I hope to get a reasonable list view with the ability to issue tokens without having to mung the URL in the next day. Code to actually give out capabilities to edit a resource should be next week.

It's slow going, isn't it?

--Mike--

Computer as suspicious package

2008-05-15T12:11:00.003-05:00

It occurs to me that almost none of us could answer the question "did you pack it yourself" in the affirmative if we're using a PC, especially not if we bought it from a vendor who favors crapware.

I haven't actually had total control over the contents of a computer since I built a little box back in the 1980's that watched for a ring signal on a phone line, flipped the relay to pick up the line, used a 4 channel 8 bit A/D converter to sample 4 incoming voltages, then used a speech chip to speak the given voltages (in almost recognizable english) to the caller, twice, then hang up.

I wrote the code, programmed the 2764 EPROM, and it was totally under my control. I packed that piece of hardware... but since then... no way.

If you get a new PC from a good source, you can reasonably trust that the BIOS isn't going to be subversive. Once you load an OS, you've definitely had someone else doing your packing.

If it came loaded (or used)... there's really no way you can truly trust it, you just have to assume it's all going to be ok. Most of the time, it works out that way, or if it is a zombie on a botnet, you don't even know it, which is almost as good for most people.

It's a strange thought... but one I think might provoke some discussion.

Only Communists complain about twitter??

2008-05-13T22:33:00.003-05:00

Why Cliff Gerrish thinks that wanting to fix twitter is the same as communism is beyond me. I'm not part of the Gillmor gang, and I'm annoyed that twitter is broken quite a bit. Does that make me a communist too?

Twitter breaks, a lot... it's broken now, giving me time to write this. It's ok to complain about a broken service. Twitter is a good service, when it works, but it's too valuable to leave to the winds of chance. Thus... replacing twitter with something more reliable is a natural itch.

I guestimate that the aggregate flow through twitter is somewhere around 3kbytes/second when it's at full bore. It can be replaced with a set of machines, with normal code, and normal network hardware. There's no super hardware or non-obvious patentable code buried in it... anyone with enough programming skills, hardware and time could do it.

But... even hinting that we might do this sends Cliff into a 1950's McCarthy era rant about communism... it's just.... odd.

Being able to trap keywords and subscribe to them from the overall stream still only has to content with 3kbytes/second. Again... normal hardware, normal networks, just a bit of distributed software to make it all work.

Even more stuff to choose... ugh!

2008-05-13T16:36:00.002-05:00

The last time I did full time programming, I was using Turbo Pascal 7.0 on a DOS platform. The notion of being able to have multiple users trying to use our little home grown inspection software was just starting to enter reality. I then did a bunch of other stuff.... now programming is coming back into play because of the capabilities demos I want to do.

I'm like a newbie all over again... I've kept up a little bit on the buzzwords, etc... but haven't had to actually implement anything from scratch in more than 12 years. I figured surely in the meanwhile all of this stuff would be sorted out, and there would be a nice standard way to have programs talk to each other across the internet.

So now I know what all of those buzzwords like SOAP, XML-RPC, REST, WDSL and the rest mean... nobody has a nice simple way to do things...

I was hoping to do a nice simple demo of a RESTful capabilities system using Python as a simple standalone app that anyone could just put on their PC (or server). It turns out that there are several things in the way. Here are some of the things I've learned.

Cryptographic random number routines aren't included in python. (Dean Landolt suggests punting the issue and getting on with it... and I agree for the demo)
The library that would do it requires me to be able to re-compile python (using Visual Studio 2003)
REST isn't... the common example of Flickr as a RESTful API isn't.
WDSL is for people who like to write specification specifications, and don't write code.
REST is the choice, except that web browsers don't actually PUT or DELETE, and a lot of people use GET for things with side-effects.
There are a lot of python web toolkits out there, including CherryPy, TurboGears, Web.py, Django, and others.

In spite of all that, here are my design choices to date:

Programming language: Python, because it's cross platform, a known entity, and quite powerful, despite the immutable strings, and comes with a web server library.
Database: None - it's a demo
Random Salt: the built in non-secure RNG from python
Protocol: REST-ish... GET for reading, idempotent operations only, POST for everything else. Rest because there should only be one URL per object, regardless of the compromise about PUT/Delete.

The demo will be of the ability to edit a string. You'll be able to see the string with a straight web page. You'll be able to request a token to edit the string, you'll be able to write the string (provided you have the token) and you'll be able to revoke the token.

I'm hoping that's simple enough for me to get done on a few train trips to/from work.

Random numbers are hard... whoda thunk it?

2008-05-12T16:56:00.003-05:00

It turns out the hard part of doing capabilities on the Internet is the lack of a suitable random number generator... which kinda blindsided me. I'm trying to find an implementation of the ISAAC random number generator that would work in either Python, or active server pages, and haven't been able to find one. It's critical to give out unforgeable tokens, and a cryptographically secure random number generator is the way to go. You can't even think of using the built in random generator, because it's too easy for a determined attacker to guess the next output after a short run of samples.

So, eventually I'll find what I want (or be forced to port it myself)... and then I can get back to the examples... which will generate a token consisting of the object, the capability, and a random number to serve as salt to keep from having it forged.

Wish me luck.

Twitter - Capabilites mashup

2008-05-10T22:15:00.003-05:00

The idea of a distributed replacement for Twitter is floating around... and I've been writing about capabilities recently... what would we get if you merged the ideas?

#1. - Get rid of user accounts on twitter... just hand out the capability to post, which would be different each time it's issued, and individually revoke able. I'd hand them out in an Email, to limit the user base a bit and cut down on spam. You could always store the email address somewhere in a table along with the capability to know who it is if necessary.

#2. - Allow each user to then hand out tokens that would allow a direct message, which they could proxy and/or revoke themselves. This would make it possible for an end user to block someone from making direct messages, without the need for it to happen in the central code. The proxy that does this could be a separate service, and doesn't play a part in the security of the central capability provider code.

#3. - Allow each user to hand out tokens that would allow following them, which like above, they could proxy and revoke themselves. This turns the distributed twitter into an effectively private email system without too much work.

#4. - There's really not much difference between a tweet and a blog post, other than length. There's not much different between a private posting and and email... you could cover all of them this way.

Ok... it's 10:30 and I'm sleep deprived, so this might not be as coherent as it seems at the time... though I hope it is.

Capabilities offer a huge amount of flexibility when doing system design. They make it possible to break apart the logic of a complex application without having to worry about the combinatorial explosion that results from the conventional idea of having every piece of code enforcing a ton of rules.

What do you all think?

--Mike--

Teaching and learning, the circle of life

2008-05-10T20:55:00.003-05:00

I'm glad to see that Dean has made the conceptual leap to understanding capability based security. It's a tricky subject to explain, and I was starting to worry he'd get discouraged... but he's made it... we've got another convert. 8)

Now, the thing to do is to take him at his word, and see exactly what helped him to see the value, and to make it easier to get to there from a world steeped in the Dogma of ACL uber Alles.

The key distinction he makes is that a capability is more than a token. He then presents cases of issuing new capabilities based on old ones, always with less authority that the original. This is a very powerful lever... once you grok it, you'll never forget it.

I've got to spend some of my very limited free time to get some actually capabilities samples up on the net, however that may happen. I've got some knowledge of Delphi, Python, and ASP... one of those should suffice to get something that can issue capabilities and let anyone store a few bits on a server somewhere.

I don't think it's really important to get huge examples working, just enough to squeak by and help others by making the cognitive leap smaller.

Thanks Dean!

You've helped renew my faith that blogging is an effective way to make things better.