It's so, like, social!

Ok, so I know I'm supposed to do the second half of the post about the REST interface, but I got distracted.

You see, I'm on holiday.

On holiday, I like to do something just crazy and interesting. And not too much work ;-)

Anyway, what I've built is a very alpha-ish prototype of a concept for Social Coding.

The original idea for this came to me in September. I posted about it on Google+. Here's the content of the post:

Imagine a web app for social coding.

You get there via a link from somewhere else. At the top of the page in big typeface is the name of a function, the signature (parameters, return type) and a short description.

Then, side by side, are two columns.

On the left, there is a list of unit test implementations. Each is just a slab of code with asserts in it.

On the right there is a list of implementations.

Each unit test and each implementation can be commented on, and can be modded up or down.

Unit tests are listed in two sections; "Working" at the top and and "Broken" at the bottom. The "Broken" section lists tests that are syntactically busted.

Within the sections, the unit tests are listed in order of moderation score.

Implementations are listed in order of the number of unit tests passed. Within the same number, they are listed in moderation order.

You are free to
- Add a new unit test (possibly based on an existing test)- Add a new implementation (possibly based on an existing implementation)- Moderate other tests and implementations- Comment on anything

You can share the page on social networks and so forth with the usual buttons.

The owner of the page is the person who originally defined the function, including description. The owner can modify the definition, can mark any unit tests and implementations as not acceptable.

The function is considered implemented if there is an acceptable implementation that passes all acceptable, working unit tests. An implemented function may still be re-implemented at any time in the future, or become unimplemented by the addition of new tests which it fails.

When writing a test or an implementation, you may define new methods that don't yet exist. All user defined methods behave as links to their own page. Methods that don't have a page have one created for them when first clicked, wiki-like.

You can also define a new function directly, without actually using it in another function.

A method which relies on other methods which are not considered implemented is also not considered implemented.

A unit test which relies on methods which are not considered implemented is part of the "broken" group.

Analogs of this function definition mechanism should be created for object oriented Classes and for html Pages.

If, say, both Javascript and Python are allowed, then you could build a google AppEngine app this way.

Thoughts?

A long chat ensued. A bit of enthusiasm flared up, and then went nowhere, because I did nothing with the idea. Yeah!

Anyway, now I have. I've built a prototype of this idea, in fact, called "Social Code". What a creative name!

It's hosted here: http://socialcode.emlynhrdtest.appspot.com/

The code is in github here: https://github.com/emlynoregan/Social-Code

Basically, it works like this:

• Anyone can log in with a google account.
• You can create functions. Functions are
• A name (purely for descriptive purposes)
• A slab of python code, hopefully containing a function implementation, but can be anything really.
• Another slab of python code, Tests, which should be tests of the function implementation, but again can be anything. When the user hits "Save and Run!", this will be run and will be successful if no errors/exceptions are thrown, fails otherwise.
• A list of past runs, with results & logs.
• Functions can refer to each other
• Say I've separately defined the function "Add"
• If you want to call "Add" in your code, you need to include {{Add}} somewhere in your implementation code, probably in a comment. When you save it, it'll be parsed out, and appear in the "Imports" list.
• You can call anything defined in "Add" in your code. This might include a python function called "Add", but only by convention.

For a really simple example of the way this hangs together, select the function Multiply, see what's going on, then click through to dependencies etc.

Now, anyone can change any of the tests or any of the implementations of any functions, delete any functions, add new ones. That's all so open that the whole thing's only usable as a curiosity at the moment.

In fact my current implementation is really daggy. Not a stick of javascript (or there is, there's an onclick confirm function for the delete button). So mondo page refreshes. Also no design, some css where necessary, inline (gasp!).

And OMG you'll hate editing code in textareas.

Also the "Save and Run!" button just cobbles all the code (tests + implementation + dependencys' implementations) into one big slab o' text and calls "exec" on it. Not terribly sophisticated. It'll need eventually to do stuff like run in an appengine task, probably work with code in a more sophisticated manner (should I be doing something with code objects?) etc. 

I've put the code in a serious sandbox (I think), by not allowing the __builtin__ collection to be in scope. You can't do a python import. So, you can only currently do simple things, intentionally. Opening this up in some way will be necessary, but it'll need some serious discussion, and care I think.

But it gives you something to play with, get a feel for the kind of thing I'm thinking about.

The intention is that, instead of one slab of code for "Implementation" and one for "Tests", we'll have many competing Implementations and many competing Tests. There'll be a concept of the current implementation for any function, which is one that passes all tests, which is voted up by the most users, and probably which is newest, in case of ties. That's the one that other dependent functions will import.

I think potential implementations need to be immutable. If you want to edit an implementation, you actually copy it and edit the copy. Might need a draft/published mechanism to help with this. So you see all the published implementations, and all your own drafts. Published implementations are immutable, drafts are not.

I don't know if/how this applies to tests. 

Also, I think functions should be able to define acceptance tests on other functions. If B depends on A (imports it) then B should be able to define acceptance tests on A. If A changes such that it has new implementation and tests, and passes those tests, but no longer is suitable for B, then B's acceptance tests will fail and flag that B needs fixing.

Of course the current giant-slab-o-tests approach could accommodate this; in the tests for B, just define some tests of A.

Anyway, have a play, read the old google+ post and comments, and come chat on the new google+ post where hopefully people will be yammering about this.

Todo: Rest

First of all, by way of apology for the chirping crickets on this blog, let me say I've been otherwise occupied.

Not busy. I don't have 7 different things to get done by lunch time. I keep my schedule as simple as possible; usually it's get up, get it together, go to work, do cool stuff, come home, do family stuff, crash. Basically my policy is, if I have two things scheduled in a day, that's one too many. My brain just doesn't like that.

But occupied, yep. And that's because I changed jobs, threw out pretty much my entire existing skillset, and started from scratch.

Well, not the whole lot. It's still code. Of course. The world is basically composed of stuff that is code, and stuff that is muttering, and I try to steer clear of muttering. But still, if you're a professional senior C# dev, and have been doing windows coding for over a decade, then it turns out in retrospect to be a thing of some gravity to chuck all that in and jump into the world of LAMP, eschew that and recommend AppEngine, then go hell for leather on rich modern javascript apps on this platform, with all open source tools.

A great and terrible thing.

(and of course some obligatory PHP support is involved, which is pure masochism).

Luckily my new job is awesome, for a little company with brilliant people and the desire to build a really first class dev team, building outstanding stuff. And a healthy hatred for Microsoft (well, you know, we'll integrate with their stuff and all that, but we wouldn't use it ourselves, bleh). The very fact that I can blog about the details of what I've been building commercially, and even release some of it, should say something positive about it.

Also luckily, I'm working with a first rate front end developer, who's all about UX, who is all over javascript and jquery and underscore and backbone and etc etc etc. And that's because I'm mediocre at best at UI dev. I can never take it seriously; making computers easier for the illiterate feels somehow like enabling an alcoholic, and that's just not really in my nature. Figure it out you useless bastards, really.

Anyway, long story stupidly too long.

The crux of it is, I've been trying to learn to do AppEngine well, to do Python well, to understand the crazy (and rather spectacular) world of kludge that is modern javascript programming; more like Escher than Kafka I think, but I'm not sure. Javascript is oddly beautiful actually, and it's lovely to see functional programming burst onto the mainstream(ish) scene finally, as confused as it currently is with OO.

So my mind has been almost entirely occupied with this. I've been frenetic, I've been focused. Not in the "now I'm really paying attention" way, but more in the Vernor Vinge's "A Deepness In The Sky" way. Drooling and a bit broken, looked after by my darling wife.

So just as I'm making some breakthroughs, starting to Get It and feel a bit more comfortable, I got sick for a couple of days. Flu/Cold/Virusy thing. Bedridden, looked after by my darling wife.

All that was on my todo list was to rest.

And, I'm not making this up, what came to my mind was "Todo, rest, hey I could get that todo list sample app and make it talk to AppEngine via a RESTful interface".

 So I built a little REST library, called Sleepy.py, while lying in my sick bed.

As I said, drooling and broken.

---

I've not come at Sleepy cold. Sleepy is actually a rewrite of a rewrite.

What I've been mostly doing commercially in the last few weeks has been to build a decent REST interface for AppEngine. Which of course I shouldn't be doing; just grab one that's already there. The problem is, I haven't been able to find anything serviceable. I looked, believe me, and even tried some out, but they just felt clunky; heavyweight, XML oriented (we want to use JSON), not really welcoming customization, a square peg for a round hole.

I thought we should stick with the existing options, because look how big and complex they are, we don't want to do all that work ourselves! But my good colleague protested, went away one weekend and came back with his own simple REST interface library, which was cleaner, much shorter, and really a bit of a revelation. It needed work, but that opened my eyes to his central observation, which was that this doesn't need to be that hard.

And it doesn't, of course. I mean, what is REST really?

What is REST?

Let me google that for you:

http://en.wikipedia.org/wiki/Representational_state_transfer

That answer sucks

Ok, then what REST means to me, in a practical sense, is an HTTP based API, using the big four requests (GET, PUT, POST, DELETE), which suits heavy javascript clients (ie: talks JSON), and which lets the client side talk to my server side datamodel in some straightforward way.

Now I know that's hopelessly non-purist, but what I'm trying to achieve is this:

- We've got a rough idea for an app to build.
- I build a server side, which is largely data model.
- Someone who is good at it builds a client side javascript app, replete with its own local data model, which must be synced somehow with the server side. That somehow is something very like backbone.js's expectation of a RESTful web api to talk to.
- We'd also like the possibility of other stuff talking to the same api (iPad apps, third party code, martian mind control beams, whatever).
- I must be able to set up a RESTful web api to let the client talk to the server. It needs to give a good balance of simplicity and customizable behaviour, be quick to put in place, and just be clean and easy to work with.

Anyway enough jibber jabber, where's some code?

Yeah yeah ok.

What I decided to do here to demonstrate my approach is to take a simple sample app that demonstrates the use of backbone.js , and make it work in the context of AppEngine. The sample app, by Jérôme Gravel-Niquet, is a todo list which lets the user make a simple but nice to use todo list, and stores it in local storage in the browser (so it really has no back end).

You can check out the sample app here: http://documentcloud.github.com/backbone/#examples

Have a play with it here: http://documentcloud.github.com/backbone/examples/todos/index.html

Starting off

So I started by downloading all the static files for the todo list app, sticking them in a basic appengine app, and popping that in git on github.

The git repo is here: https://github.com/emlynoregan/appenginetodos

The first version that just hosts the static app, still backed by localstorage, is available at the tag v1:
https://github.com/emlynoregan/appenginetodos/tags

You can run that in your local appengine devserver, or upload it to appengine proper, and it'll work. But it's not touching the datastore.

Datamodel

The app is a simple todo list. There's no concept of users, of different lists, nothing. There's just one list, which is a list of Todo items. A todo item is composed of three elements:

• text - string, the text that the user enters describing what is to be done 
• order - int. the numerical order in the list of this item; lower number comes first.
• done - bool, whether the item is done or not

The first thing we need is a datastore model to describe this. This is a simple one:

from google.appengine.ext import db

class ToDo(db.Model):
    text = db.StringProperty()
    order = db.IntegerProperty()
    done = db.BooleanProperty()
    created = db.DateTimeProperty(auto_now_add = True)
    modified = db.DateTimeProperty(auto_now = True)

    def __init__(self, *args, **kwargs):
        db.Model.__init__(self, *args, **kwargs)
        if self.done is None:
            self.done = False

Just one class, ToDo. It's got text, order and done (and I've put a default value on "done" of false in the constructor). I've also added "created" and "modified" fields just for a bit of fun. These are managed by the server (using auto_now_add and auto_now), and wont be exposed to the client.

And now the REST!

Ok, now we want to expose this datamodel class through a REST api. For the ToDo class, we'll use a resource name of /todos. Locally this might be http://localhost:8080/todos. My version on appengine has the url http://todos.emlynhrdtest.appspot.com/todos .

The first thing to think about is the routing (kicking off the right handlers for urls) and basic infrastructure. How will we get /todos to be handled the way we want to handle it, and how would that handling be done in any case?

I figured that since the RESTful web api will be just like a normal web page, but returning JSON, why not use the standard webapp web handler mechanism already available in appengine, and the same routing mechanism? Now we'll want a little help with the routing, because we want to be able to manage /todos, but also calls like /todos/32 (where 32 is a valid id for a model instance) to address particular todo items.

So, you define the routes by providing a list of resourcename / handler pairs, which a helper function then turns into url / handler pairs as required by the wsgi application.

Here's the main.py, showing a standard route for '/', then including the calculated restRoutes.

import webapp2

import htmlui
import restapi

# basic route for bringing up the app
lroutes = [ ('/', htmlui.ToDoHandler) ]

# add api routes, see restapi/__init__.py
lroutes.extend(restapi.restRoutes)

# create the application with these routes
app = webapp2.WSGIApplication(lroutes, debug=True)

restRoutes comes from the restapi module for this app, whose __init__.py looks like this:

from todoresthandler import *
from sleepy import *

restRoutes = [
  ('todos', ToDoRestHandler)
]

restRoutes = Sleepy.FixRoutes(restRoutes)

ToDoRestHandler is a webapp webhandler for the rest api for ToDo. Sleepy.FixRoutes turns restRoutes from a list of (resourcename, handler) pairs to a list of (url, handler) pairs.

    @classmethod
    def FixRoutes(cls, aRoutes, aRouteBase = None):
        """ 
        Modifies routes to allow specification of id
        
        aRoutes should be pairs of resourcename, resource handler.
        
        This is modified to become pairs of route source, resource handler.
        
        aRouteBase is anything you want to prepend all route sources with. 
        
        eg: if you want all your route sources to begin with /api, 

        use aRouteBase="/api" 
        
        Don't include a trailing slash in aRouteBase.
        """ 
        retval = []
        for lroute in aRoutes:
            lfixedRouteSource = '/(%s)(?:/(.*))?' % lroute[0]
            if aRouteBase:
                lfixedRouteSource = aRouteBase + lfixedRouteSource
            
            lfixedRoute = (lfixedRouteSource, lroute[1])
            retval.append(lfixedRoute)
        return retval

We've got the /todos route wired up to ToDoRestHandler. But what's in that? Here it is:

from google.appengine.ext import webapp
from sleepy import Sleepy
from datamodel import ToDo

class ToDoRestHandler(webapp.RequestHandler):
    def get(self, aResource, aResourceArg, *args, **kwargs):
        Sleepy.GetHandler(self, aResource, aResourceArg, *args, **kwargs)

    def put(self, aResource, aResourceArg, *args, **kwargs):
        Sleepy.PutHandler(self, aResource, aResourceArg, *args, **kwargs)

    def post(self, aResource, aResourceArg, *args, **kwargs):
        Sleepy.PostHandler(self, aResource, aResourceArg, *args, **kwargs)
    
    def delete(self, aResource, aResourceArg, *args, **kwargs):
        Sleepy.DeleteHandler(self, aResource, aResourceArg, *args, **kwargs)
    
    def GetModelClass(self):
        return ToDo

What's going on here? Well, I've delegated all the functionality of GET, PUT, POST and DELETE to the Sleepy library. Plus, there's an extra method GetModelClass which tells us (actually Sleepy) which model class we're working with.

So basically you've shown us nothing. What's in this Sleepy?

No I haven't, have I? I'll get onto what's in Sleepy in the next post. For now, if you want to skip ahead, just check out the git repo. Otherwise, you can wait for the next installment, lazy person!