Evan Reiser's Blog

Halve your site's page sizes in 5 minutes

2009-04-28T14:39:00.002-04:00

This reduces our page sizes by up to 50%+

from django.utils.html import strip_spaces_between_tags
from django.conf import settings

class SpacelessMiddleware(object):
def process_response(self, request, response):
if not settings.DEBUG:
if 'text/html' in response['Content-Type']:
response.content = strip_spaces_between_tags(response.content)
return response

I havent had to worry about SQL recently

2009-03-03T22:31:00.002-05:00

I'm happy django does all the database work for me so i can spend my time building features and writing the same snippets of code all over the application.

howto: group by in django

2009-01-05T00:23:00.003-05:00

Django doesnt officially yet support group by, however the code to acomplish this is actually hidden inside. If you had a model that tracked registrations, which contained a FK to the user that refered the new user such as:

class Registration(models.Model):

...

referer = models.ForeignKey(User,null=True,blank=True)

>>> from registration.models import Registration

>>> query_set = Registration.objects.extra(select={'count': 'count(1)'}, order_by=['-count'],where=['referer_id is not null']).values('count', 'referer')

>>> query_set.query.group_by = ['referer_id']

>>> for row in query_set:

... print row

...

{'count': 26L, 'referer': 6384}

{'count': 4L, 'referer': 6871}

{'count': 3L, 'referer': 9993}

{'count': 2L, 'referer': 7804}

{'count': 2L, 'referer': 7892}

{'count': 2L, 'referer': 9699}

{'count': 1L, 'referer': 6111}

{'count': 1L, 'referer': 7821}

{'count': 1L, 'referer': 5025}

this is as of

Last Changed Rev: 9037

Last Changed Date: 2008-09-15 14:06:02 -0400 (Mon, 15 Sep 2008)

GeoDjango Error when using python2.4

2008-12-04T11:52:00.002-05:00

Anyone getting this error?

  File "/usr/lib/python2.4/site-packages/django/contrib/gis/db/models/__init__.py", line 5, in ?
   from django.contrib.gis.db.models.manager import GeoManager
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/db/models/manager.py", line 2, in ?
   from django.contrib.gis.db.models.query import GeoQuerySet
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/db/models/query.py", line 5, in ?
   from django.contrib.gis.db.backend import SpatialBackend
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/db/backend/__init__.py", line 12, in ?
   from django.contrib.gis.db.backend.postgis import create_spatial_db, get_geo_where_clause, SpatialBackend
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/db/backend/postgis/__init__.py", line 3, in ?
   from django.contrib.gis.db.backend.base import BaseSpatialBackend
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/db/backend/base.py", line 7, in ?
   from django.contrib.gis.geos import GEOSGeometry, GEOSException
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/geos/__init__.py", line 31, in ?
   from django.contrib.gis.geos.base import GEOSGeometry, wkt_regex, hex_regex
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/geos/base.py", line 11, in ?
   from django.contrib.gis.geos.coordseq import GEOSCoordSeq
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/geos/coordseq.py", line 10, in ?
   from django.contrib.gis.geos.prototypes import cs_clone, cs_getdims, cs_getordinate, cs_getsize, cs_setordinate
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/geos/prototypes/__init__.py", line 8, in ?
   from django.contrib.gis.geos.prototypes.coordseq import create_cs, get_cs, \
  File "/usr/lib/python2.4/site-packages/django/contrib/gis/geos/prototypes/coordseq.py", line 67, in ?
   cs_getordinate = cs_operation(lgeos.GEOSCoordSeq_getOrdinate, ordinate=True, get=True)
  File "/usr/lib/python2.4/site-packages/ctypes/__init__.py", line 328, in __getattr__
   func = self.__getitem__(name)
  File "/usr/lib/python2.4/site-packages/ctypes/__init__.py", line 333, in __getitem__
   func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/libgeos_c.so.1: undefined symbol: GEOSCoordSeq_getOrdinate

I found this on our server which was running python 2.4, I've heard from some other people that upgrading to python 2.5 fixed this problem.

New Friend Requests

2008-11-30T09:15:00.001-05:00

The New Friend Requests Page now shows additional information about your potential friend, including your mutual friends

ImportError: cannot import name ImageField

2008-08-16T18:01:00.003-04:00

I came across this error when updating django to the lastest SVN version for gamernook.com.

The fix was to replace

from django.db.models.fields import ImageField

with

from django.db.models.fields.files import ImageField

Easy enough, but took a while to figure out

Building a YouTube Clone (Day7)

2008-02-07T19:57:00.000-05:00

Today i changed the way videos are stored in the database and abstracted the storage of the actual media. Now videos can either be uploaded by users or can be submitted from other video sites such as YouTube. This allows users to discuss and share content even if the video isn't physically on their computer.

Building a YouTube Clone (Day 6)

2008-02-05T19:25:00.000-05:00

Today I finished the user profile pages. User profile pages show basic info about the user. User's can set a featured video to show off on their profile page. User's can now subscribe to various channels/users. The profile page also shows all of the user's videos, as well as their channel subscribers and subscriptions. In the future this page will also showcase the user's playlists, their interests, and their videolog/blog.

User's can also now edit their profiles, and upload a user picture/avatar. Right now the user profile has the same fields as YouTube but this can and will change in the future.

Building a YouTube Clone (Day 5)

2008-02-04T18:05:00.001-05:00

Today I rewrote the video browsing section to allow the user to change the sorting of results, and to change the time period of the search; additionally the user will be able to narrow down their search by category as well. This will affect all result type pages where multiple videos are displayed, such as browsing for videos or viewing which video's you have favorited.

Videos also now keep track of how many times they were commented on and how many times they were marked as favorites.

You can also now search and browse by video tags:

Also whats not shown here is that you can filter down the current videos to a certain video category type as well.

Building a YouTube Clone (Day 4)

2008-01-31T19:16:00.000-05:00

Today I expanding the functionality surrounding videos on the site. Videos can now be rated, favorited, flagged, and commented on. User's can view their videos and their favorite videos.

Video Ratings
Video Time Formatting
Video Favorites
Video Flagging
Video Commenting
Video Comment Rating

Also what few people will appreciate is that the rating for a video are not discrete/quantum values but rather can be a floating point anywhere between 0.0 and 5.0, and the stars will display correctly to represent that rather than Youtube's 1.0, 1.5, 2.0, 2.5 etc

Building a YouTube Clone (Day 3)

2008-01-30T22:44:00.000-05:00

Today I finished integrating the video uploads, browsing and playing, all of this was made easy thanks to Twistage.

Building a YouTube Clone (Day 2)

2008-01-29T21:13:00.001-05:00

Today i redid the site authentication to allow for OpenID, i got most of the code from a great package i found on code.google.com, i'll post the link here later.

I started building out the profile models and views, but i'll have to finish this later.

I've also added a forum and discussion board, pretty much boiler plate stuff here, but its the makings of a Social Network.

Finally i've added in support for "Portable Social Networks" a project by one of pownce's developers that allows users to add info + links to their profiels on other social networks, i can use this so users can show off what other networks they belong to. What's' cool is we can use the data collected here to link profiles across many different networks. This might also be useful in the future for importing contacts from other services.

More tommorow

Building a YouTube Clone (Day 1)

2008-01-28T21:10:00.000-05:00

I'll try to write more here later. I started the project by creating the django project, and setting up the core authentication. I created the base template for the site and did some basic web design with css.

I set up the project in SVN. So i should be ready to go tomorrow

Building a YouTube clone

2008-01-28T17:52:00.000-05:00

Over the next few weeks I'm going to be creating a youtube clone (if you're an investor read: online video community), as part of a new venture I'm working on. Using Twistage for video delivery, I'm going to try to build a video community site by myself in record time.

I'm planning on using django + python to build the site and i'll track my progress here.

CityBombing.com launches

2008-01-16T12:59:00.000-05:00

As some of my friends know, I've spent the last 5 weeks building a photo hosting and community site for graffiti artists. The original idea came from my brother who wanted a mash-up to show graffiti pictures on google maps. I started working on the site in December to learn django (a python web framework).

Thanks to the awesome language and framework, and I was able to build a full fledged social networking site from scratch in less than 2 months. I'll be spending the next week or two adding internationalization support, and re-deploying the application with new graphics as some other photo sharing/tracking community sites.

Special thanks to the django, python, mootools, google maps API, nesh's thumbnail module, as well as my girlfriend and my little brother for their advice, help and support. Also thanks to Dan + Udi for my pesky django + python questions.

Check it out here: www.citybombing.com

Python + PIL: Make Geotagged photos from EXIF Headers

2008-01-08T11:04:00.000-05:00

For my django app, i wanted to take geotagged images and display them on google maps. Geotagged images store GPS data in the EXIF headers of the image, so we'll want to grab this data and extract the lattitude and longitude so we can display where the image was taken in our mapping software.

Using the PIL library we can grab the GPS data out of the image and format it in a way to easily display on a google map. PIL really does all the work for us, all we have to do is convert the Lat/Lng from DDD°, MM', SS.S" format stored in image to DD.DDDDD° format that google uses

Hidden in the Exif data we see this:

34853: {0: (2, 0, 0, 0), 1: 'N', 2: ((40, 1), (37, 1), (279, 10)), 3: 'W', 4: ((74, 1), (8, 1), (373, 10)), 5: 0, 6: (0, 1), 7: ((18, 1), (58, 1), (19, 1)), 9: 'A', 18: 'WGS-84'}

We can then use the following python to extract the data, convert and save it to our photo object

from PIL import Image
from PIL.ExifTags import TAGS

i = Image.open('/pics/DSC07018.JPG')
info = i._getexif()

ret = {}
for tag,value in info.items():
decoded = TAGS.get(tag, tag)
ret[decoded] = value

Nsec = ret['GPSInfo'][2][2][0] / float(ret['GPSInfo'][2][2][1])
Nmin = ret['GPSInfo'][2][1][0] / float(ret['GPSInfo'][2][1][1])
Ndeg = ret['GPSInfo'][2][0][0] / float(ret['GPSInfo'][2][0][1])
Wsec = ret['GPSInfo'][4][2][0] / float(ret['GPSInfo'][4][2][1])
Wmin = ret['GPSInfo'][4][1][0] / float(ret['GPSInfo'][4][1][1])
Wdeg = ret['GPSInfo'][4][0][0] / float(ret['GPSInfo'][4][0][1])

if ret['GPSInfo'][1] == 'N':
Nmult = 1
else:
Nmult = -1

if ret['GPSInfo'][1] == 'E':
Wmult = 1
else:
Wmult = -1

Lat = Nmult * (Ndeg + (Nmin + Nsec/60.0)/60.0)
Lng = Wmult * (Wdeg + (Wmin + Wsec/60.0)/60.0)

Now that you have the Lat and Lng you can save to your model or class to easily display in your web app.

photo= GMapPhoto()
photo.image = i
photo.lattitude = Lat
photo.longitude = Lng
photo.save()

Django Caching

2007-12-18T11:20:00.000-05:00

I was trying to learn more about caching in django, so i had my site output a list of all sql queries that were run at the bottom of the page.

I set the cache time for a certain page to be 5 minutes. However, for some reason, MOST of the time I viewed a page without POSTing back, there were a ton of SQL queries appearing, it seemed like my cache wasn't working at all.

Little did i realize that i had installed django.middleware.cache.CacheMiddleware, which was causing all pages without GET or POST variables to be cached automatically. However, (I think) the cache saves the output of a rendered page, which included the HTML of all the SQL queries being run for the page request. This led me to think all the SQL was actually running until i checked the database.

I guess power is useless unless you know how to use it correctly

Make your website faster in 5 minutes

2007-12-17T15:00:00.000-05:00

The "speed" or "performance" of your website is only based on one thing: how long it takes from when your user clicks until the data gets from your servers to the eyeballs of your user. Some companies can spend hundreds of hours and millions of dollars tuning queries, to shave off 20% of a 5ms query, but at the end of the day all that matters is your application's response time as perceived by the end user.

For most websites I'd say 2% of the time is spent in the database, 20% in the app code, 10-20% in transfer or latency, while the rest is spent or wasted on the front end. If i was going to spend the least amount of time/money to increase the most performance, I would focus on the front end of the application. Specifically, on how the browser requests, caches and renders data.

You're end user shouldn't have to wait around for stuff he doesn't need. Theres no point in hitting the webserver again and again requesting the same exact media files (css/javascript/images). These dont change so its a waste of processing power and bandwidth to server them. In order to prevent the user's browser you have to explain that their local cache shouldn't expire instantly. Set far future expires headers on your static content and you wont have to server it again and again. Doing this is trivial in either Apache of IIS

ExpiresActive On
ExpiresDefault "access plus 1 month"

This is what i have set on my static content server, if you're using the same webserver to serve up dynamic content, you probably want to specify expires header by type instead

ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpg A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType text/css A2592000

You might have just reduced your bandwidth and response time by 20%. Additionally, read up on ETags and why they should be configured correctly. Probably just do this

FileETag none

Obviously, you want to send as little data as possible out of your webserver. Reducing file sizes and using common css/js files are pretty obvious performance techniques. What you might also want to do is compress text files as they are sent form your webserver; modern browsers can decode GZip'ed data at the small expense of processing power. To enable apache webserver to send gzip content use mod_deflate

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript

In my experience this seems to reduce the size of html/js/css files by 60% or more. Obviously this will reduce the transfer time of your content.

Finally, understand how browsers render web pages. Any time JavaScript is run, the browser will block page rendering. This is because JavaScript is able to write out data to the page. If you load all your JavaScript in the , your page won't start rendering on the client until it has all been transfered and loaded on the browser. Additionally you probably want to load all of your CSS as soon as possible; that way you can get required images as soon as possible if needed. A good rule of thumb:

Put your CSS at the top of the page
Put your JavaScript at the bottom of the page

On a side note, Django rocks and mootools rocks

New Site Designs

2007-11-27T03:25:00.000-05:00

Many of my friends seem to think that Social Suggester is my start-up project. Actually its just a demonstration of some of the technology developed by my company Social Sample.

Nonetheless, I've been wanted to re-design the website; despite its deceivingly decent looks, it was really thrown together haphazardly in a weekend with my friend. I've received a wide range of advice about how to improve the site from a wide range of sources including friends, investors and complete strangers.

One common criticism is that the site lacks no personality, and that its hard for users to understand exactly who the target audience is. I'm not exactly sure who the site should be targeted at, but i thought it would be fun to create a much more "gritty", "dirty", "urban" feel for the site, that might scare away adults but might at least add some personality of flavor.

I'm really only an amateur graphic artist, but I do love playing in photoshop. Below are some samples with what i ended up with at the end of tonight. I think it would be pretty cool to post a sideshow from beginning to end of development, since theres been a lot of cool designs that ended up in the trash.

Click for the full screenshot, graffiti courtesy of my little brother

PHP feels like ASP

2007-11-14T16:54:00.000-05:00

My next project I'd like to use a real web application framework so I don't find end up with more of this crap:

Although ASP.NET arguable uses the MVC pattern, my eyes are on Django (if people would stop telling me to use rails)

Facts about Newark Airport

2007-11-09T19:49:00.000-05:00

Airport security does not believe me that ice cream is in fact a solid and therefore does not fall under TSA liquid rules.

Airport security does believe that my vitamin water does not qualify as a travel size container.

They don't notice if you don't take your shoes off.

The stewardess noticed my iphone was still turned on after her warning, i tried to explain to her that i couldn't turn off my phone. She said that if i was unable to turn off the phone that i should remove the battery.

The Fine Art of Software Estimation

2007-11-06T13:13:00.000-05:00

Overheard at Social Sample:

"by the way 25-30 hours really means 60 hours"

"I'm just taking all estimates and doubling them now...even if people have already doubled them, i'm doing it again; blanket policy"

Maybe we need some Evidence Based Estimation

Metallica Fans Don't Support Hilary Clinton

2007-11-02T14:59:00.000-04:00

What happens when you take a giant database full of social data and do some light data-mining and statistics: lots of random facts! (arguable worthless or priceless depending on what you use them for).

I am spending my Friday at Social Sample analyzing social data from supporters of 2008 presidential candidates; data found on online social networks. After coming up with some basic demographics, I searched the user base for music interests with high correlations to the normal population. A similar algorithm to what powers Social Suggester. I found that the artists / bands with the highest correlation to Hillary Clinton supporters were:

"MADONNA"
"TORI AMOS"
"FIONA APPLE"
"BJORK"
"PRINCE"
"DEPECHE MODE"
"GWEN STEFANI"
"PORTISHEAD"
"THE SMITHS"
"ELTON JOHN"
"NO DOUBT"
"KELLY CLARKSON"
"MILES DAVIS"
"THE CURE"
"JUSTIN TIMBERLAKE"

Hmm Interesting. Now what happens if you change the Social Suggester ranking algorithm to find the bands/artists with the lowest correlation between these two data sets?

"ATREYU"
"THE USED"
"MUDVAYNE"
"SLIPKNOT"
"TAKING BACK SUNDAY"
"BREAKING BENJAMIN"
"YELLOWCARD"
"DISTURBED"
"HINDER"
"BLINK 182"
"PANTERA"
"50 CENT"
"STAIND"
"METALLICA"
"SYSTEM OF A DOWN"

Also Interesting. These correlations weren't extremely high, but Hillary Clinton supporters that use online social networks were 4x less likely to be a fan of 50 Cent, and 5x less likely to be a fan of Breaking Benjamin. I'm not going to make any assumptions about what kind of people support Hillary Clinton, all we can say is that there a high correlation between the types of people who enjoy the first set of artists/bands, and a low correlation between the second. If anyone reads this please feel free to share any insight; please let me know if there are any other subcultures you'd like to see info on. I'll leave with some random data about Hillary Clinton supporters, (these are not facts [read: please dont sue me], just the data that our engine spat out based on analysis of the target group). Enjoy.

Interesting differences (all compared to the normal population, sample size =10.5 Million, all statistically significant)

Supporters were no more likely to be Female
Supporters were 10.9x more likely to be gay
Supporters were 1.25x more likely to be married
Supporters were 1.25x more likely to be a parent
Supporters were 1.50x more likely to be in grad school

Supporters were 2.01x more likely to be Agnostic
Supporters were 1.9x more likely to be Jewish
Supporters were 2.50x less likely to be Muslim
Supporters were 1.12x less likely to be Catholic

The same proportion of supporters Smoke cigarettes, but they are 1.1x less likely to drink alcohol

Find other high correlations at Social Suggester, or contact me for something specific. evan at socialsample . com

Ikea

2007-10-18T16:27:00.001-04:00

Ikea and I have a much different idea about what a computer workstation is:

IKEA:Evan:

Drag & Drop in JS

2007-10-05T07:01:00.000-04:00

I realize every new JavaScript toolkit does now but i thought this was pretty cool. Works for FF and IE7. I like being able to assign delegates to onDragOver + onDragOut

<script language="JavaScript" type="text/javascript"><br /></script>

http://neb.net/playground/dragdrop/