Rich's Rants

Facebook Performance

2009-06-03T11:53:00.003-05:00

I've been logging calls to Facebook API servers in my DEV environment for some weeks now.

I'm pretty disappointed by the performance, overall:
Facebook API Call Performance Stats

I realize Facebook has scalability issues beyond my ken, but they also have resources beyond my ken as well...

Facebook doesn't wait over 4 seconds for my server; Surely I shouldn't have to wait 30 seconds for theirs! :-)

In fact, it's kind of pointless in the context of a canvas page for their servers to bother to respond at all after 4 seconds...

User --A--> Facebook server request canvas page --B--> Request to my server --C--> API Call to Facebook server.

If C can detect that we are in the context of a canvas page, and Facebook already knows that B will not wait more than X seconds, they might as well issue an HTTP 408 Request Timeout or something similar after X seconds, because there's no real point in them giving me the info they aren't going to wait for...

I went ahead and created a Feature Request in Facebook's Bugzilla. You can vote for it if you think it will help you.
#5528

svn diff in vim

2009-05-14T13:05:00.002-05:00

In the category of "How did I live without this?!" tools, I'm late to the party with this vim plugin from Brian Shire of Facebook / APC / PHP fame:

If you use svn, git, cvs or similar, this plugin shows you a "diff" in vim.

I don't know how often I quit vi or open another shell just to do a diff.

Now, I don't have to!

Lumni I Likovski, Cook County Assessor's Office

2009-02-14T17:44:00.003-06:00

ADVERTISEMENT: Hire Me

So the other day, I had to go down and file for a home owner's exemption on my house.

I got my little queue ticket #96. They were calling out #32, with 5 or 6 workstations.

I had a half hour, max, to get this done, before I had to be elsewhere.

This did not look good...

Thank God for Lumni I. Likovski, Manager, Taxpayer Exemptions.

He was out in the waiting area, checking for people who didn't need a full-blown computer workstation (such as myself) and doing "triage" to get us the forms we needed, get our questions answered, and get us out of there.

He even cheerfully (and I stress cheerfully) photocopied my Driver's License for me, a requirement to hand in the form.

It's not often in this day and age that you get excellent service/help like this, most especially from government staff.

So I'm dedicating this blog post to LUMNI I LIKOVSKI down that at Cook County. I owe you a beer!

PS
I tried to just send in a comment to his boss using their online form, but it blows up like this, every time I try to submit it:

Database Results Error
Description: The conversion of char data type to smalldatetime data type resulted in an out-of-range smalldatetime value.
Number: -2147217913 (0x80040E07)
Source: Microsoft OLE DB Provider for SQL Server

Maybe they should hire me to re-do their website? :-)

Webgrind ROCKS

2008-07-21T21:28:00.002-05:00

Okay, so I'm a Linux guy through and through, and I'd love nothing more at my day job than to wipe the Doze desktop and throw Gentoo or Debian or FreeBSD or even Fedora on it...

Shoot, I'd even be happy to dual boot the thing.

But my current day job is the kinda job where I can't even surf to my own blog at work, because it's blocked.

So, being stuck with Doze [shudder], I was Oh. My. God. happy to find this gem:

Webgrind

It's kind of like kCacheGrind, only a lot less features and through a browser and dog-slow.

But so what?!

The POINT is, I can actually successfully profile my code with Xdebug for the first time ever on Windows without jumping backwards through flaming hoops!

Now if I could just get JIT debugger client to happen...

G is smart!

2008-05-08T08:41:00.003-05:00

is smart. I mean, is really really smart.
I ask a lot of questions, and often knows the answer.
Of course, sometimes doesn't. Though that often turns out later to be that I just didn't ask the question the right way.
Sometimes gives me a lot of extra information. But that's okay. Sometimes 's extra information teaches me something. And sometimes it's strictly for entertainment value, which is also okay.
And likes multi-color rainbow-y things, which is cool.

Now, I've got just one question for you: Am I talking about Google, or my girlfriend?

Is Google an Artificial Intelligence?

I'm serious here.

Is Google's spidering, indexing, search, and response algorithms, in toto, an Artificial Intelligence?

It doesn't know what I'm supposed to be doing Saturday night, for example.

[Girlfriend interjects] Mother's Day dinner at Indie Cafe

Hmmmm. Now that she's added that, Google does know what I'm supposed to be doing Saturday night!

So clearly Google learns, and learns quickly...

I dunno.

I still think my girlfriend is smarter than Google... But I'm not so sure Google isn't pretty smart.

Solaris vi input mode arrow keys fix

2008-04-17T10:36:00.003-05:00

This blog post is not for you to read.

It's for me.

You see, every once in a awhile, I work for a client/employer who uses Solaris, OpenSolaris, Unix, or some other traditional Unix-like OS.

They have their reasons, and they are good ones, and I'm fine with that.

Alas, I then usually waste about an hour of their time (and they pay me for it) to Google for the hack that makes vi (my editor of choice) actually usable from a terminal program (putty, usually).

To wit, the arrow keys in vi do not work in input mode under these OSes.

This blog post is a reminder for ME to know what the heck to do next time this happens.

So here is what I do:
Open up ~/.exrc and type these things:
set t_ku=[control-v][up-arrow]
set t_kd=[control-v][down-arrow]
set t_kr=[control-v][right-arrow]
set t_kl=[control-v][left-arrow]

Then make a symlink from .exrc to .vimrc
ln -s ~/.exrc ~/.vimrc

Finally, alias vi to vim, since this only works for vim:
alias vi=vim

Bonus Tip:
To get vi/vim to use more than ONE LINE when you start up, use:
TERM=putty screen

I believe "screen" is magical pixie dust that gives you a whole screen of line instead of one line, and the TERM stuff obviously tells screen that you need that screen to go to putty.

If "man screen" had been installed, perhaps I would have a better understanding of the magical pixie dust, but so it goes.

:-)

Adobe Spams

2007-09-13T00:26:00.000-05:00

For the record:

I loathe Flash.

I mean, even on Windows where it kind of sort of works, it sucks up WAY too much RAM, and WAY too much CPU, and crashes far too often.

Not that anything really works well on Windows (other than their Marketing, Legal, and Bullying Departments) but Flash works particularly badly.

On Linux? Forget it.

I gave up installing the damn thing, even on Windows, about a year ago, and you know what?

I have found that there is not one site with any content worth getting that I am missing out, that I can't find elsewhere.

Maybe your experience is different; Maybe you just can't live without uTube.

Fine.

But for me, no more Flash, ever again.

Oh yeah, the real reason I hate Adobe is that they are spammers.

They added me to their mailing list, unsolicited, and sent me commercial junk mail.

So unless you want to support spammers, stop using their products.

It's that simple.

Thank you.

File Upload Progress Meter

2007-07-17T18:24:00.000-05:00

Every week or two, somebody asks how to do a File Upload Progress Meter on the PHP-General mailing list...

Why the BROWSER doesn't provide this feature everybody wants is beyond
my ken.

Surely the browser has some clue how many bytes are uploaded and how
big the file was.

How tricky could it be for Firefox/IE to poke those values into a
couple variables somewhere?

Instead we have a zillion JS hacks by developers generating tons of
traffic back-n-forth to the server to ask it how many bytes it has
received so far...

In fact, why don't the browsers provide a NICE file upload progress meter in the first place, so web designers don't feel the need to re-invent the wheel?

Or even (gasp) some nice hooks involving CSS and Web 2.0 or whatever so web designers can just do this without even needing to know any Javascript?

This certainly would be way more useful than half the crap the browser wars have introduced in the past half decade that we've been needing this feature!

Too Many "Friends"

2007-07-12T23:18:00.001-05:00

Bear with me here, there's a fairly long prologue...

So, here's the thing:

In one of my alter egos, I'm a Talent Buyer at a music venue.

Now, you may not know much about the internal workings of music venues, but, basically, a music venue has about 1000 bands pounding on the doors wanting to play for every available position.

There's nothing we'd love more than for there to be way more fans out there, mind you, so we could actually accommodate about 100 of those artists, instead of just he 1.

Do note that I said 100 of the 1000, not all of them...

90% of everything is still crap.

Anyway, at the email address we've set up specifically for artists to submit a demo, we tend to get a whole lot of friend-link requests to basically all the big social networking sites.

Now, this is going to sound petty, but it's getting pretty dang annoying.

Most of these friend link requests are coming from artists we don't work with.

Many of them are coming from artists we simply can't work with, as they're not the 1 in a thousand.

I daresay a lot of them are coming from the 900 we wouldn't want to work with, even if the the supply/demand went the other way, with way more fans than artists...

Our website pretty clearly asks bands not to add us to their promotional mailing lists. Do we really need to spell it out "and don't send us be-my-friend emails either"?

Anyway, I'm thinking that all these social networking sites should just stop sending out emails on behalf of their users to non-users to invite them to join.

If I want to join your social-networking site, and buy into the whole thing, then fine, I've agreed to get your emails.

But I haven't!

So, really, all these social-networking sites are just thinly-disguised spammers, when you get down to it.

Or maybe there should be some kind of industry-standard minimum proof that the recipient might actually want these dang things, or that the sender actually has an existing relationship.

Knowing an email address is not an existing relationship!

Because it's gotten to the point where a new entrant in the social networking market, to me, just means yet another flurry of invites that I don't really want.

Then I have to spend 20 minutes digging through their site for a place to contact them to say not only to ban that one user from sending me emails, but to ban ALL the users from ever sending me emails.

I don't even want the dang things at my personal address anymore, really, from friends I actually know. It's gotten that bad.

I definitely don't want them from strangers to an email address that was set up for a very specific business purpose.

Am I being too petty?

I don't think so.

We came up with guidelines for the robots, and that seems to have (mostly) worked out.

Can't we come up with guidelines for these social networking / stay in touch sites?

Here are some suggested starting points for guidelines:

If I'm not a member of your site, don't email me more than one invite ever, period.

If I didn't want to join when Lee invited me, I still don't want to join when Fran invites me, okay?

The invite email should provide links including:
ban this user from ever emailing me again
ban all users from ever emailing me
accept invitation
decline invitation, but join site

Perhaps there should also be a "do not social-network-invite" shared database maintained by the larger existing social networking sites, which other social networking sites could pay a reasonable fee on a per email basis to check against, and a person could register with that one place to not get any invites from any social networking site.

Note that the fee should be large enough to make it prohibitive for spammers to just pay up to garner emails, that it should not actually hand out email addresses but return a YES/NO for a submitted address from the social networking site, but be cheap enough that any serious new social networking site would buy in as a matter of course. Maybe there is no such price-point, but at least give it some thought.

I sure don't want to keep contacting every johnny-come-lately social networking site to ask them to put a ban on their users sending me invites. There are too many of them springing up like weeds.

PHP in HTML

2007-07-11T16:24:00.000-05:00

How come this doesn't work:
<button action='<?php mysqli_query($connect,$query)?>'>Click</button>

[sportscaster voice-over]

Joe: Hey Bob, let's look at a slow-motion instant-replay of this common PHP newbie fallacy scenario.
Bob: Sure Joe!
Joe: Okay, so here goes:
  The user requests a HTTP URL document.
  The webserver fires up.
  The webserver finds that it needs PHP to generate the document.
  PHP fires up.
Bob: Wow, look at PHP go! That's fast!
Joe: Yeah, it is fast.
  PHP has generated the document, and spits it out.
Bob: Boy, it's already finished. Hey, it's quit!
Joe: That's right, Bob.
  PHP has FINISHED EXECUTION, and has exited.
  Now watch this!
Bob: Oh boy, I see it coming now...
Joe: Yep, there it is.
  There's some HTML in the browser, trying to execute some PHP code...
Bob: But you can see, PHP has LONG FINISHED and is OUTTA HERE!!!
Joe: That's right, Bob, PHP is simply not around to execute that code.
Bob: So what can you do, Joe?
Joe: Well, if you can live with the browser going back-n-forth to the web-server, with a significant "lag" time...
Bob: Oooh, well, I can see how that might be useful sometimes...
Joe: In those cases, you can use Ajax.
Bob: Anything else?
Joe: Not really. Until you get back to the webserver and PHP, there's just no PHP available. Unless your user is in the extreme minority of uber-PHP-geeks that has installed this EXPERIMENTAL PHP browser plug-in thingie: http://pecl.php.net/package/PHPScript
Bob: Whoa, Joe, I don't think I've ever even heard of anybody who's ever installed that.
Joe: Me neither, though I met Wez Furlong who wrote it, so I have to assume HE has installed it at least once...

[cue to cool Guinness commercial]

PHP Microsoft Excel Reader and Serial Killer Dates

2007-07-07T00:14:00.000-05:00

So, my sister's ex-company of about 6 years that has shut down operations, but still has outstanding loan accounts to collect on...

They have this arcane accounting system and it generates fixed-width field .TXT files, with the wrong dates (with 2-digit year, of course).

The correct dates are in some Excel spreadsheets.

So they wanted a quick hack script to change columns 254 through 259 in some .TXT files based on a simple lookup in some Excel files.

Some developer told them it would take two weeks, and my sister didn't believe that, and they weren't real keen on spending that much $$$.

I told her it was more like a 2 hour job, which might turn into 2 days, when all is said and done.

It actually only took about 8 hours, all told, and most of that was spent in attempting to use various Excel reader PHP scripts that just plain didn't work.

The one up on PHPClasses ends up using parse_url to try to figure out where a local file is, and bombs out. It's a really nifty stream-filter OOP thingie, if it only actually worked...

And what's with all the annoying ads? Maybe less ads would get more traffic would get more revenue... [shrug]

There were several Excel Reader commercial options in the $100 to $200 range, which didn't appeal at all.

There's an Excel Writer in the PEAR Repository, but I couldn't see how to make it read Excel files.

There was some other Excel Reader script, but I forget now why I rejected it. Oh well.

Finally, I found an Excel Reader PEAR package, only it's on Sourceforge instead of in PEAR.

Go figure.

It also had a bug, where it was doing an include of 'OLERead.php' but the file was actually named 'OLERead.inc'

Not quite sure how that passed by a QA process, any QA process but it's trivial to fix.

I did submit a bug report, so hopefully it will get fixed. That is the nice thing about OpenSource.

The example to read a whole sheet in as an array was pretty much all I needed after that quick '.php' -> '.inc' hack.

I didn't even try the reading as a stream thing, since there are only a few thousand accounts.

So I had a nice Excel Reader to do the account number to date lookup.

Yippee.

Then came the joy of Microsoft Excel internal date format...

It's basically a "count" of days from January 1 1900 for the integer part, and a count of seconds for the fractional/decimal part.

Of course, Microsoft aped Lotus 123 and knowingly left in the bug of 1900 being a leap year (it's not)!

So from 0 to 60, the date is "off" by one, and at 60, the bogus date of February 29, 1900 is output. Everything is great from 61 up to a zillion or so where you get to December 31, 9999.

Now, granted, none of these loans date back to the first couple months of 1900, but it's still pretty irksome...

And you'd want a conversion function to be correct and re-usable, rather than something that only works for a limited input set. (Y2K anybody?)

I found a conversion function in C, and ported it to PHP:
http://l-i-e.com/excel_date.phps

I'm not claiming it's the best code ever, and I don't even know what it does, really, as I just changed the variables to have $ in front, and swapped int() function into floor() function.

Slapped in an sprintf instead of returning the individual month/day/year as pass-by-access args, since it seemed easier.

That pretty much sums up the past couple evenings for me.

I occasionally run into a Microsoft devotee who wonders why I hate Microsoft so much.

Really, if this rant doesn't make it clear why I hate Microsoft, I simply cannot hold a rational conversation with you...

It would have been a 2-hour job if MS wasn't so stupid, but it was an 8-hour job because Microsoft is, well, stupid.

Windows PHP Free Development

2007-07-05T22:00:00.000-05:00

Zoe Slattery of IBM has documented exactly how a Windows user would go about hacking PHP source with all free development tools.

http://www-03.ibm.com/developerworks/blogs/page/phpblog

I'm not sure if I should leap with joy or cringe in fear, but now any PHP Windows user could manage to write their own custom extension.

I managed to stumble through it once, thanks to Sara Goleman's fantastic articles up on Zend.com:
Extension Writing Part I: Introduction to PHP and Zend

I wonder how long it will be before Microsoft accidentally on purpose breaks things so you can't do this?

Refund Policy

2007-06-14T15:57:00.000-05:00

So I was setting up a new domain today for a website, and for the first time noticed Hostbaby's Refund Policy at the bottom of their contact page:

REFUND POLICY: Anyone who wants a refund can get a refund for any reason.

Made me laugh out loud.

Now, the thing is, I'm sure some readers are saying "Yeah, right" sarcastically.

But I've worked with these guys long enough to know, that if that's what they say their policy is, that's actually their policy.

REFUND POLICY: Anyone who wants a refund can get a refund for any reason.

You can tell them you want to shut down your site and get a refund because your dog ate your homework, and they'll do it.

Plus they don't even charge you until a full month, so you get your site up before the billing starts.

I love this webhost.

PS They also run CDBaby.com where you can buy a zillion Indie CDs, but no major label junk. :-)

PHP header location redirect refresh

2007-06-13T11:38:00.000-05:00

I haven't posted much, but at php|tek 2007, Chris Shiflett actually said he liked my rant, and that I should post more.

Well, hey, if Chris Shiflett says I oughta do something, I listen.

So, today's rant is about PHP header Location hacks.

First, let's be sure everybody understands what it is:

header("Location: http://example.com");

will re-direct the browser to the URL example.com

Specifically, the PHP tells Apache to issue to the browser a 301 Redirect header to that URL, and then the browser gets that 301 Rediret header and automatically tries to visit that URL.

EDIT
As pointed out on PHP-General, PHP actually sends a 302 Temporarily Moved rather than a 301 Permanently moved. The rest of the rant still applies, as I simply mis-typed 301 for 302 anyway.
/EDIT

Now this seems really cool at first, perhaps because it is really cool, when used for an appropriate problem.

Unfortunately, many PHP scripters are using this as an Idiom or as a Programming Construct with things like:

if (!logged_in()){
header("Location: login.php");
}

Now, this does "work" but there are several problems with it.

First and foremost, the HTTP Specs require a complete URI for the location. And while this fragment of a URI might "work" on most browsers, it will, sooner or later, totally mess you up when the browser mis-interprets this.

Specifically, some versions of IE will do the redirect, but won't POST the original data as a redirect should, with this URI fragment. Use the full URL, and IE actually does the right thing, and redirects with all the POST data intact.

You shouldn't use an incomplete URI just because "it works" any more than you should use non-compliant HTML just because it works.

But let's look at this header("Location: ") in a slow-motion instant replay:
User requests page X
Browser uses dog-slow Internet to contact web server.
Web server receives request, hands it off to PHP
PHP responds over dog-slow Internet with 301 Redirect to login.php.
Browser interprets 301 Redirect, hopefully correctly.
Browser uses dog-slow Internet to contact web server.
Web server receives request for login.php, hands it off to PHP
PHP finally spits out the login form.

Whew.

Maybe you should consider just doing this instead:

if (!logged_in()){
require 'login.php';
exit;
}

And look, ma, no extra round trip through the dog-slow Internet.

Plus, you're not wasting an HTTP connection, which, on a busy server, is a precious resource. Presumably you'd like your website to be popular enough to be busy someday, right?

You're going to end up reading the login.php file in the end anyway. Why would you spend all that time going back and forth to the browser to do it?

And include 'login.php' is no more difficult to read/understand/maintain than the header().

Reserve a header("Location: xxx") for when you really need it: When a document has actually moved and the URL is being retired.

Don't use header("Location: xxx") as a Programming Construct in place of simple if/else and include logic.

PS This same rant applies to the header("Refresh: ") usage as well. Though why you'd want to use that instead of Location: is beyond me, and probably the subject of another rant...

PHP Downloads, Content-Disposition, Content-type, and other arcana.

2006-06-22T14:08:00.001-05:00

Every damn day, some other poor PHP newbie goes off and finds "advice" in Google about how to force browsers to download things, and how to get the filename they want in the "Save As..." prompt box.

Invariably, these newbies are suckered in by Bad Advice^tm from people who clearly do not read the HTTP specs.

Now I'm not claiming to have read all the HTTP specs, much less memorized them, but did fight through this battle in the days of version 3 and 4 browsers, and it still seems to be tripping people up, while my solution has been working for me since nineteen-ninety-mumble.

So I'm gonna let you in on a few little secrets.

Okay, one of the secrets is widely published in the RFCs, and the other is hard-won experience about how the 1995 johnny-come-lately Microsoft made-up Content-disposition header isn't really widely-supported very well, and how to make 100% certain that your downloads always end up prompting the user for the right filename with a crude but effective hack that leaves the brower with no other reasonable choice.

First, if you Google for this topic, you're going to find a lot of people suggesting a lot of different MIME-types to use for the "Content-type" header:


application/octet-stream
application/download
application/force-download
.
.
.

Note that this is to force a download -- If you actually want a browser to display (or attempt to display) some content, the Content-type: should always be the correct Content-type for that type of document. E.g. text/html for HTML, image/gif for a GIF, image/jpeg for a JPEG (image/jpg will not work on some browsers), image/png for a PNG, application/pdf for a PDF, and so on.

If you are ever in doubt about the correct MIME-type for browser display, find a static URL that works, preferably on the corporate site driving that technology, and find out what Content-type: is sent by their site for their sample content.

For a download, application/octet-stream works because it's a part of the HTTP specification, and has been part of the HTTP spec, from the very beginning of HTTP specifications.

The others "work" only because they are made-up content-types, and the browser currently has no idea what to do with them.

Some equally valid Content-type would be:


asdf/asdf
abc/abc
you-can-put/anything-you-want-here
microsoft/sucks
asdfswetrkhkhkvnknsdbknilghwerthiehl/wilerywnfaksvnklgndiglkghadlgha

Only problem is, tomorrow Microsoft can choose, at their discretion, that "application/download", or any of the other made-up MIME-types means "Put it in the My documents directory", because MS knows much better than you that that is what all their users really want, and your download doesn't do what you want any more.

But they cannot change the meaning of application/octet-stream, because that is specifically reserved, in the HTTP specification, for force a download

So, all of the above boils down to the following question:

Do you want to use a MIME type that happens, by sheer coincidence, to not be "taken" yet and will work today, but tomorrow might be re-defined?

Or would you rather use the documented feature that appliction/octet-stream will always work?

For anybody not well-versed in Tech-Talk the correct answer is the latter, and definitely not the former.

So to force the browser to download a document, the only correct solution is:


Content-type: application/octet-stream

Anything else is a game of Russian Roulette.

With Microsoft pulling the trigger.

Now we come to the somewhat more complicated issue of getting the "Save As..." window to provide the filename you like as a default.

Let's assume, for our purposes, that you want this filename:


iwant.xyz

In an ideal world, the "Content-disposition: ... ;filename=iwant.xyz" would work perfectly for this.

... represents some sort of MIME-type, which is largely irrelevant, for the purposes of this article about forced downloads.

Unfortunately, with some versions of some browsers, this Content-disposition: simply will not work.

In fact, no matter what combinations of headers you try to use, there is some minute version, such as x.y.z.37, that doesn't work even though x.y.z.36 and x.y.z.38 do work

This is complicated by the presence or absence, and/or changes to the Content-type: and the "..." part of Content-disposition: that we're ignoring.

To document exactly which versions of which browsers do/don't work for which combination of headers, mime-types and URLs is well beyond the scope of a blog post. Perhaps a Ph.D. Thesis would be more appropriate, if somebody is desparate for a Thesis Topic and has a lot of time to spare.

And for the love of insert diety of choice here do not ask me to tell you which browser/version won't work with your particular solution. I have neither the time nor the inclination to do your browser-testing QA for you. I don't even like doing my own QA, much less yours.

I can only guarantee you that if you test on every minor version of every browser ever released, you will find one that does not work.

But I do have a solution for you:
Provide a URL which the browser cannot possibly mess up.

For example, this URL will mess up some browsers:
http://example.com/download.php?filename=iwant.xyz

This URL, however, the browser, cannot possibly mess up:
http://example.com/download/iwant.xyz
because it's too damn simple to mess up.
K.I.S.S. priniciple is the watchword here.

Don't give the browser any opportunity to screw up. Because if you do, some browser somewhere will screw up.

"But wait!", you say, "I can't do that! I need my PHP script to do the download work"

Yes, you can.

Follow the bouncing ball:

First, 'download' above may look like a directory, but it's not. It's a PHP script. It just doesn't happen to have .php on the end of it.

And iwant.xyz doesn't have to be in any particular location just because it looks like a boring static URL component.

There are a variety of ways to make this work. Apache's mod_rewrite springs to mind for Apache experts, and there are many articles online telling you how to do PHP mod_rewrite.

But, truth to tell, mod_rewrite is a real PITA to mess with. And Apache/PHP has a much easier technique available which I'll detail below.

So, there are two tasks here for this URL:
http://example.com/download/iwant.xyz

The first is to somehow get 'download' to be a PHP script, even without '.php' on the end.
And the second is to somehow make 'iwant.xyz' from the URL available to PHP.

Fortunately, the mechanics of these are very easy tasks.

It seems to be difficult for newbies to wrap their brains around the concepts, but the actual mechanics are trivial, and I'm hoping this How-To will ameliorate the difficulty of the concepts.

Let us begin by assuming your not quite working download.php PHP script looks something like this:


<?php
  //register_globals is off, of course
  $filename = $_GET['filename'];
  
  //Crude cleansing to avoid ../../etc/passwd hacks
  $filename = basename($filename);
  
  //In an ideal world, you would have a specific range of legal values for $filename
  //And your cleansing would test positive only for valid input
  //The following line is far too restrictive in anything but this sample application
  //But it's definitely the Right Way (tm) for THIS sample application
  //Security can't be bought off-the-rack.  It's a custom job like this
  if ($filename != 'iwant.xyz') die("Did you really think I wouldn't add filename validation here?");
  
  //I can virtually guarantee that the next line is not correct.  Fix it.
  //I personally would recommend that it NOT be in your webtree,
  //so Bad Guys (tm) cannot bypass your application and just get it direct.
  //If your webhost does not provide a non-web-tree directory, find a new host.
  //This should be the complete full path to the "real" iwant.xyz file.
  $basename = '/some/path/to/the/real/files/';
  
  //Compose the actual full file path:
  //If you didn't put / at the end of $basename, add it here
  //Or do some fancy footwork to be sure you have the proper number of '/'s you need
  //Or not, as Un*x systems ignore bogus extra '/' in a pathname anyway.
  $fullname = "$basename$filename";
  
  //For larger files, a decent browser will provide a progress meter, if you do this:
  $filesize = filesize($fullname);
  header("Content-length: $filesize");
  
  //As discussed above, the only Documented Feature,
  //sure-fire guaranteed way to force a download every time is:
  header("Content-type: application/octet-stream");
  
  //Now just read the file and spit it out:
  readfile($fullname);
  //For large files, an fopen/fread loop using feof may be more appropriate
?>

Now, instead of the usual 'download.php' you might expect, name this script 'download' without the '.php'

In order to convince Apache that this script really is a PHP script, even without the .php on it, create a file named '.htaccess' in the same directory as 'download' (or a 'higher' directory) and put this in it:


<Files download>
  ForceType application/x-httpd-php
</Files>

The above three lines of magic force Apache to think of 'download' as a PHP script, even though .php is not part of the script name.

This assumes that your webserver has been configured with .htaccess "on" in httpd.conf. If that's not the case, then you would probably want to put those three lines directly in httpd.conf, or in a file that httpd.conf Includes.

If your Apache webserver host provides neither httpd.conf nor .htaccess to you, the I feel truly sorry for you, but cannot help you, other than to suggest finding a better host.

Your URL would then look something like:
http://example.com/download?filename=iwant.xyz

You should go ahead and build this example application now, and then we can move on to our second task of getting rid of the ?filename= part.

Here is a sample application using the above code:
http://l-i-e.com/blogger/download.php?filename=iwant.xyz

Note that you will, depending on your browser make and model, probably be prompted to "Save As..." with the filename 'download.php'

We'll be fixing that in our next task, so just change the name to 'iwant.xyz' by hand when prompted to download.

But if you can find a browser that does not treat that file as a download, I'll send you a Cookie.

Note that you can configure some browsers to just auto-save all downloads in some directory or, blech, on your desktop. That's a user-configuration choice which nothing in the world is going to "fix". Sorry. Educate the user, or live with their freedom of choice, whichever way you want to look at it.

But the browser itself is still treating the output as a 'download' even if it has been [mis-]configured to just dump the file in some random directory.

Now, on to the task of getting the URL to end in /iwant.xyz so that the browser is "fooled" into thinking it's a static URL and it will use 'iwant.xyz' as the default filename for the download window.

As you can see, your script 'download' is going to have some extra stuff at the end of it.

Apache and PHP collaborate to mostly ignore anything extra tacked onto the end of a URL, except for one crucial input they provide:


$_SERVER['PATH_INFO']

This variable is set by Apache/PHP to contain everything after your script name that is in the URL, no matter what is there.

Now, because your real application might need more input than just 'iwant.xyz' I'm going to go above and beyond here, and provide an include file that will give a lot of flexibility.

Here are some URLs the normal way, and some done my recommended way compared side-by-side:

Normal	Recommended
http://example.com/download.php?filename=iwant.xyz	http://example.com/download/iwant.xyz
http://example.com/download?filename=subdirectory/iwant.xyz	http://example.com/download/subdirectory/iwant.xyz
http://example.com/download?page=42&line=20&filename=iwant.xyz	http://example.com/download/page=42/line=20/iwant.xyz

Keep in mind on that last one, that the browser cannot know that you don't have directories named 'page=42' and 'line=20' no matter how odd that may seem for directory names.

Those are perfectly valid directory names, and the browser has to assume that's what you have.

Only you and I will know that 'download' isn't a directory but a PHP script, and those 'extra' bits are really just inputs to this PHP include:


<?php
  //Consider a URL such as:
  //  http://example.com/scriptname/var1=val1/var2=val2/path/to/filename.xyz
  //Transform it into:
  //  $PATH = '/path/to/filename.xyz'
  //  $PATH_VARS['var1'] = 'val1';
  //  $PATH_VARS['var2'] = 'val2';
  $PATH = '';
  $parts = explode('/', $_SERVER['PATH_INFO']);
  foreach ($parts as $part){
    $pieces = explode('=', $part);
    switch(count($pieces)){
      case 1: /* tack it on as part of a pathname */
        //Also ignore the leading '/' of PATH_INFO which turns into an empty '' from explode()
        if ($pieces[0] !== '') $PATH .= "/" . $pieces[0];
      break;
      default: /* Set up something like $_GET only with $PATH_VARS */
        $var = $pieces[0];
        // value might have = within it...
        unset($pieces[0]);
        $val = implode('=', $pieces);
        $PATH_VARS[$var] = $val;
      break;
    }
  }
?>

I've commented the above script heavily, and all it does is transform the PATH_INFO that Apache and PHP provide into a couple convenient variables:
$PATH_VARS will contain any /var=val/ in the URL as $PATH_VARS['var'] = 'val';
$PATH will contain anything else in the path as '/subdir1/subdir2/filename.xyz';

Save that script above as 'pathinfo.inc' and change the top of your 'download' script from $filename = $_GET['filename'] into this:


require 'pathinfo.inc';
$filename = $PATH_VARS['filename'];

Now, you can surf to a URL like this:
http://l-i-e.com/blogger/download/iwant.xyz
and get a download windows with the only reasonable choice for a default filename to "Save As..." that a browser could possibly infer from that static-looking URL: 'iwant.xyz'

It would be nice if this was a Documented Feature or if something like Content-disposition actually worked in all the minor versions of all the browsers ever released.

But, in this case, consider what else a browser could possibly do with the download window it must provide.

If you really think about this, I believe you'll come to the same conclusion I did, many years ago: This is a hack, but a reasonably safe hack, because what else can a browser do with such a simple URL, given that it must prompt the user for a file download (or auto-save the download by user choice) to remain compliant with the HTTP Spec.

Here is the final source for our download script:
http://l-i-e.com/blogger/download.phps

And the pathinfo.inc file:
http://l-i-e.com/blogger/pathinfo.phps

There are a few caveats worth mentioning here:

Unlike $_GET, $PATH_VARS cannot be made into a "super-global" so you'll have to declare it global within your functions/methods.

Actually, technically, you could use PHP's RunKit extension to force $PATH_VARS to be a "super-global" but if you've got RunKit installed on your server, you probably already know everything in this article anyway.

If you don't know what RunKit is, you should just take my word for it that you don't want it installed, but if you need convincing, let me just point out that the purpose of RunKit is to be able to re-define something like:

if ($whatever) echo $something;

so that the 'if' and the 'echo' don't do what you expect them to do anymore.

I.e., RunKit lets you re-define the actual PHP language on-the-fly, for developing a new version of the PHP language.

Let me also point out, in case it's not blatantly obvious, that savvy users can still put any damn thing they want into the URL in attempts to break your script, take over your server, and otherwise cause you much grief.

This URL munging should not be considered primarily as a "Security Measure" though there may, or may not, be some relative increased security in that finding a ? in a URL and then trying variants is probably a very common Bad Guy technique, but cramming more things onto the end of a static URL generally doesn't do anything at all, so most Bad Guys probably don't do a lot of that.

This falls into "Security through Obscurity" though, which are generally very weak security measures, and only useful, if at all, when layered in with other, more robust, Security Measures.

Or, to make a long story short, you must still validate and cleanse any data coming from $PATH and $PATH_VARS, exactly as you would for $_GET.

Required Reading:
http://phpsec.org/

I aleady know that at least one other PHP Developer thinks I'm daft to put /var=val/ into the URL, and that I should just use 'positional' elements.

Unfortunately, I do not find that very convenient, as some of the elements in my scripts are optional, so the URL would end up needing too many '/////' in it and my eyes are too old and worn-out to attempt to count those correctly.

I can also safely predict some bloggers will insist that "Content-disposition:" works just fine in all browsers, or maybe they'll be smart and qualify it as "all modern browsers"

My only possible responses to that are:

You haven't tested enough minor release versions

I believe backwards-compatible legacy support for ancient browsers is important

If you do not like this particular solution, just don't use it.

I happen to believe, based on my experience fixing far too many bug reports from iconoclastic users of niche browsers you may have never even heard of, that it's the only correct solution to browser insanity, paricularly if you use PHP to output dynamic rich media such as Images (GIF, JPEG, PNG), PDF, FDF, Flash/Ming, and so on, as I have done.

The pathinfo.inc file above works wonderfully for a URL such as:
http://example.com/thumbnail/max_width=100/photographer_id=7/artist_id=15/rockstar.jpg

It also works for the PDF URL embedded in an FDF which tend to drive Netscape/IE crazy if you start adding dynamic elements.

This rant was actually referenced by none other than PHP Security Expert Chris Shiflett in The Adobe PDF XSS Vulnerability

Bogus Blog?

2006-05-18T15:47:00.000-05:00

Truthfully, this blog originally got created just so I could add a comment to:

http://phpgirl.blogspot.com/2006/05/chicago-php-user-group-report.html