Monday, October 10, 2011

An Embarrassment of Riches


Update: Details on new stuff in Python 2.7 for appengine here: http://code.google.com/appengine/docs/python/python27/newin27.html

AppEngine is all colour and movement at the moment.


  • MySQL compatible db layer, Google Cloud SQL. 
  • Python 2.7 now available as an experimental runtime for all apps. 
  • Cross group (XG) transactions. 
  • Increased limits for all kinds of cool stuff.


First, Cloud SQL is released. Yo Dawg, we heard you like databases, so we put a database in your database so you can query while you query. Or, more clearly, this:


Thursday, October 6, 2011


Google Cloud SQL: Your database in the cloud

Cross-posted from the Google Code Blog

One of App Engine’s most requested features has been a simple way to develop traditional database-driven applications. In response to your feedback, we’re happy to announce the limited preview of Google Cloud SQL. You can now choose to power your App Engine applications with a familiar relational database in a fully-managed cloud environment. This allows you to focus on developing your applications and services, free from the chores of managing, maintaining and administering relational databases. Google Cloud SQL brings many benefits to the App Engine community:
  • No maintenance or administration - we manage the database for you.
  • High reliability and availability - your data is replicated synchronously to multiple data centers. Machine, rack and data center failures are handled automatically to minimize end-user impact.
  • Familiar MySQL database environment with JDBC support (for Java-based App Engine applications) and DB-API support (for Python-based App Engine applications).
  • Comprehensive user interface for administering databases.
  • Simple and powerful integration with Google App Engine.
The service includes database import and export functionality, so you can move your existing MySQL databases to the cloud and use them with App Engine. Cloud SQL is available free of charge for now, and we will publish pricing at least 30 days before charging for it. The service will continue to evolve as we work out the kinks during the preview, but let us know if you’d like to take it for a spin. 


Here's the FAQ: http://code.google.com/apis/sql/faq.html
And the Group: https://groups.google.com/forum/#!forum/google-cloud-sql-discuss 


The signup form implied that we'll be able to mix and match SQL and datastore, which is damned fine.


Then, the prerelease of the new SDK is announced by Ikai Lan:

Hey everyone,

Prerelease SDK 1.5.5 is now available for download! You can get it here:

Python:

Java:

We provide prerelease SDKs as previews for things to come. New features should not work in production yet, and documentation is typically still a work in progress. Release notes are below as well as in the prerelease packages:

Python
===============================
- Python 2.7 is now available as an experimental runtime for all applications
  using the High Replication Datastore. To upload your app to the Python 2.7
  runtime, change the runtime argument in your app.yaml to python27.
- We have released an experimental utility, available in the Admin Console, to
  assist in migrating your application to the High Replication datastore. This
  utility allows you to copy the bulk of your data in the background, while the
  source application is still serving. You then need a brief read-only period to
  migrate your application data while you copy the data that has changed from
  the time the original copy started.
- We have increased the number of files you can upload with your application
  from 3,000 to 10,000.
- We have increased the size limit for a single file uploaded to App Engine from
  10MB to 32MB.
- We have increased the Frontend request deadline from 30 seconds to 60 seconds.
- We have increased the URLFetch maximum deadline from 10 seconds to 60 seconds.
- We have increased the URLFetch Post payload from 1MB to 5MB.
- App Engine now supports Cross Group (XG) transactions with the High
  Replication Datastore, which allow you to perform transactions across
  multiple entity groups.
- We have released an experimental API that can write to Google Storage for
  Developers directly from App Engine.
- We have added a graph to the admin console that displays the number of
  instances for which you will be billed.
- In the XMPP API, get_presence() is deprecated in favor of using the inbound
  presence handlers documented in
- The Task Queue API 'target' parameter now accepts a new value,
  taskqueue.DEFAULT_APP_VERSION, which will send the task to the default
  frontend version, rather than the version or backend where the 'add' method is
  being called.
- In the URLFetch API, make_fetch_call() now returns an RPC object.
- Fixed an issue in the Admin Console where the "Run Now" button did not work
  for tasks with a '-' in the name.
- Fixed an issue where the SDK did not decode Base64 encoded blobs.
- Fixed an issue to provide a better error message when using the Mail API to
  send email to an invalid user address.
- Fixed an issue in the SDK where a skip_files entry caused an ImportError when
  the library was located elsewhere in the PYTHONPATH.
- Fixed an issue in the SDK index viewer where the arrows indicating whether a
  query was ascending or descending were not properly rendered.
- Fixed an issue where httplib did not support the deadline argument for
  URLFetch calls.
- Fixed an issue where you could not schedule a cron job to run every 100
  minutes.
- Fixed an issue in the SDK where failed tasks retried immediately instead of
  waiting for 30 seconds.
- Fixed an issue making it possible to modify request headers using the deferred
  library.

Java
=============
- We have released an experimental utility, available in the Admin Console, to
  assist in migrating your application to the High Replication datastore. This
  utility allows you to copy the bulk of your data in the background, while the
  source application is still serving. You then need to take a short downtime to
  migrate your application data while you copy the data that has changed from
  the time the original copy started.
- We have increased the number of files you can upload with your application to
  from 3,000 to 10,000.
- We have increased the size limit for a single file uploaded to App Engine from
  10MB to 32MB.
- We have increased the Frontend request deadline from 30 seconds to 60 seconds.
- We have increased the URLFetch maximum deadline from 10 seconds to 60 seconds.
- We have increased the URLFetch Post payload from 1MB to 5MB.
- App Engine now supports Cross Group (XG) transactions with the High
  Replication Datastore, which allow you to perform transactions across multiple
  entity groups.
- We have released an experimental API that can write to Google Storage for
  Developers directly from App Engine.
- We have added a graph to the admin console that displays the number of
  instances for which you will be billed.
- In the XMPP API, getPresence() is deprecated in favor of using the inbound
  presence handlers documented in
- Fixed an issue in the Admin Console where the "Run Now" button did not work
  for tasks with a '-' in the name.
- Fixed an issue to provide a better error message when a user tries to parse an
  HttpRequest's input stream more than once in a request.
- Fixed an issue to provide a better error message when using the Mail API to
  send email to an invalid user address.
- Fixed an issue in the SDK where HttpServletRequest.getInputStream().read()
  always returned -1.
- Fixed an issue where you could not schedule a cron job to run every 100
  minutes.



--
Ikai Lan
Developer Programs Engineer, Google App Engine



Posted by at No comments:

The Spiny Norman Test


Update: Results are in, see Go Spiny Norman, Go.

Previously, in The Amazing Story of AppEngine and the Two Orders Of Magnitude, I've written about minimizing the cost of instances in the new AppEngine billing regime. But I think I made a mistake, and I think many people are making the same mistake.

Here's one of the graphs that I showed of instance usage from my appengine app Syyncc:


My posts were largely about trying to drop the blue line down (that's "Total" instances), and I largely ignored the yellow line, "Active" instances.

Now to get that blue line down, I did two things. I first set Max Idle Instances to 1, from Automatic. That is detailed here, and was successful in dropping the blue line down. Next, I changed my app's task behaviour, from kicking off 50 tasks every 2 mins, to smoothing those out, scheduling one every two seconds.

Once I got my billing results, these changes made a huge impact.  But, the numbers were puzzling. Firstly, they were too low (which I just accepted happily, as these numbers represent money in my pocket). Secondly, it appeared that all the benefit was seen based on the first change (Max Idle Instances), with no change from the smoothing out of tasks. That's been bugging me.

And then on the AppEngine list, Gerald Tan made this comment:


The reason why your Frontend Instance hours are lower than you expected is because you assumed that you will be billed for the area under the BLUE line in the Instance graph. It's not. You are being billed for the area under the YELLOW line (Active Instance) PLUS your Max Idle Instance setting. So your Active Instances is hovering at around ~0.72, and I assume you have set your application's Max Idle Instance to 1. Therefore ~1.72 * 24 = ~41.28 Instance Hours

Oh really?? That would match the data, very cool. And why were the numbers so high before I set the Max Idle Instances to 1?

This Post-Preview Pricing FAQ (should have been called a Primer for the alliteration) says some unclear things. We have this:

"Instances are charged for their uptime in addition to a 15-minute startup fee, the startup fee covers what it takes for App Engine to bring up and down the instance. So, if you have an on-demand instance only serving traffic for 5 minutes, you will pay for 5+15 minutes, or $0.08 / 60 * 20 = 2.6 cents. Additionally, if the instance stops and then starts again within a 15 minute window, the startup fee will only be charged once and the instance will be considered "up" for the time that passed. For example, if an on-demand instance is serving traffic for 5 min, is then down for 4 minutes and then serving traffic for 3 more minutes, you will pay for (5+4+3)+15 minutes, or $0.08 / 60 * 27 = 3.6 cents."


On the other hand, this:

Max Idle Instances: Decreasing this value will likely decrease your bill as fewer idle instances will typically be running and we will not charge for any excessive idle instances. In this case the scheduler knob is a suggestion to the scheduler but we will not charge you for excess if the scheduler ignores the suggestion. For instance, if you set Max Idle Instances to 5 and the scheduler leaves 16 instances up for some length of time, you will only be charged for 5 instances.


So, I think this might mean the following:

If you set "Max Idle Instances" to Automatic (the default setting), that means you are letting the scheduler spend your money. It'll keep as many instances running at any time as it thinks you need, and you'll pay for all of them (plus that nasty 15 minute bonus on starting up extras). This means, you pay for the area under the blue line.

If you set "Max Idle Instances" to a specific value, you'll pay for your active instance time plus your "Max Idle Instances" setting, or your Total instance time, whichever is less. ie: you pay for the minimum of (area under yellow line + Max Idle Instances) and (area under blue line).

So setting Max Idle Instances to an actual number is a good idea. The lower you set it, the more it might affect the scheduler's decisions, but still, to minimise cost, set it to a finite number.

Great conjectures. But then, the old lady in my head (oh god she's really in there) says this:


TEST IT!

Ok old lady, I'll test it. razza frazza rackkin testin frazza razza....

---

Ok, so first we need an hypothesis. Put a new line on the graph, a green line, which is the yellow line, raised up by the setting of Max Idle Instances. If Max Idle Instances is 3, it'll look like this:


The pink area is the intersection of the area under the blue line and the area under the green line.

Hypothesis: Ignoring the 15 minute cost for spinning up new instances, the price we pay should be the pink area on the graph. That is, the moment by moment minimum of (total instances) and (active instances + Max Idle Instances). If Max Idle Instances is Automatic, then there is no green line, and we pay for the area under the blue line.

So how do we test that hypothesis?

1 - First test that we pay for the area under the blue line when Max Idle Instances is Automatic.
2 - Next, test that we pay for the pink area when Max Idle Instances is set to something.

To get a good test here, we want to create an instance usage profile where the blue line and the yellow line are disparate. My best guess for how to do this is to create some spiky usage, that should leave too many instances running most of the time.

Enter Spiny Norman!

Spiny Norman is a Worker class, designed to do one thing; cause AppEngine to experience very bursty load.

import logging

from Worker import Worker
from datetime import timedelta
from google.appengine.ext import db

class SpinyNorman(Worker):
    _minutesBetweenSpines = 12
    _spineWidth = 1000000
    _numberOfSpinesRemaining = db.IntegerProperty()
        
    def CreateSpines(cls, aSpineLength, aNumberOfSpines):
        lcount = 0
        while lcount < aSpineLength:
            lnorman = SpinyNorman()
            lnorman._numberOfSpinesRemaining = aNumberOfSpines
            lnorman.enabled = True
            lnorman.put()
            lcount += 1
    CreateSpines = classmethod(CreateSpines)

    def doExecute(self):
        self._numberOfSpinesRemaining -= 1
        #
        lcount = 0
        while lcount < self._spineWidth:
            lcount += 1
        
        logging.debug(lcount)
    
    def doCalculateNextRun(self, aUtcNow, alastDue):
        if self._numberOfSpinesRemaining > 0:
            if alastDue:
                return alastDue + timedelta(minutes=self._minutesBetweenSpines)
            else:
                return aUtcNow + timedelta(minutes=self._minutesBetweenSpines)
        else:
            return None # time to stop
Spiny Norman creates a spiny workload, as follows:

Each "Spine" is a set of tasks running (doExecute()) at the same time. The length of the spine is the number of tasks. The width of the spine (in time) is a measure of how much work the spine will do (how long it'll work for). Spines are set apart from each other in time, which is the minutes between spines. There are a fixed number of spines.

You kick off Spiny Norman by calling SpinyNorman.CreateSpines(spineLength, numberOfSpines) . That creates a number of instances of Spiny Norman equal to the spineLength, and sets the countdown for how many iterations they should continue for (numberOfSpines). _spineWidth is the number of times to sit in a busy loop in doExecute.  _minutesBetweenSpines is used to calculate the next run time in doCalculateNextRun.

I'm using a spine length of 250 (that is, 250 tasks), a spine width of 1,000,000 (enough load to notice some work being done, a few second's worth), 12 minutes between spines and 100 spines total (ie Spiny Norman runs for about 1200 minutes, or 20 hours, total).

I've set up a new AppEngine instance, I've enabled billing, and I've kicked off Spiny Norman to run during his own billing day. I've left Max Idle Instances set to Automatic. We should see a huge difference between the blue and yellow instance lines, and the billing should tell us which one I'm paying for, which will test part 1 of the hypothesis.

In the next post I'll publish the result of this test, and I'll kick off the next test. Stay tuned!
Posted by at No comments:

Saturday, October 1, 2011

The Worker

More Work?

One of the first serious Google AppEngine subjects I've approached recently is the problem of doing work in the background. In my particular case I needed to do some intensive and error prone tasks, then send an email with the results (which is also error prone), on a schedule.

I was going to write some standard job-processing-in-a-loop kind of code, with the loop being processed as a cron job (set up in cron.yaml). That's what Syyncc does. But some bit of my brain kept grumbling about the inelegance of that approach. You're on a platform that wants to do it a different way, says my brain (and who am I to disagree?).

And the cron thing is kind of bad, because it doesn't scale. Let's say I schedule a job every two minutes. It can get through some fixed amount of work (maybe 10 jobs?) before it hits its time limit. It can never do more than that. That's nasty.

People often recommend backends for this kind of work. With them, you stick jobs on a pull queue, and pull them off with the backend. Each backend can process a limited amount of jobs, but you can set them to be automatically created in response to workload, which is cool.

But I'm partial to push queues, what were previously just called Task Queues. At any point in code you can schedule a task to run, which simply comes through as a post to a url in your app:

         taskqueue.add(url='/dosomething', params={'key': key})

It's a bit clunky, because you need to set up a handler for the url, and implement the Post method.

Oh wait, no you don't. Nick Johnson wrote the excellent deferred.defer library, which takes care of the public url and thunking the call from there into a method of your choice. So instead your call can look like this:


from google.appengine.ext import deferred

  def do_something_expensive(a, b, c=None):
      logging.info("Doing something expensive!")
      # Do your work here

  # Somewhere else
  deferred.defer(do_something_expensive, "Hello, world!", 42, c=True)


That's cool, isn't it!

What's also cool about tasks is that you can delay them, either by specifying a countdown or an eta. Using a countdown (number of seconds before execution) is interesting, because you can delay tasks, ie: spread the work out a bit. But using an eta is really fascinating, because it lets you schedule work for specific times. So if you need to schedule an email to go out at midnight, a task with an eta will do that for you, with no real plumbing required on your part. (Can you do this with a pull queue? You may be able to use eta to stop tasks showing up through the lease system before a specified time, I'm not sure about this.)

This is all great for performing scheduled background tasks. Except, what if they fail? Or take a long time to complete? In fact, how can you report on the status of these tasks? Well, you can't. There's no way to go in and find out much about the task through any APIs. Even if there was, you'd probably need custom information suited to the job at hand anyway.

What I need is an object in the datastore that maps to the task. I personally prefer an object oriented approach (ok, I'm an old man set in my ways, yes I know). So, what I'd like is a base object which lets me set up a task, kick it off, record its progress, and lets me see afterwards how it went.

So I created the Worker. The worker is a base class polymodel object, that you can use to do background jobs. You need to override it, and provide it with a job to do (doExecute()) and a method for calculating the next time to run if you want a repeating job (doCalculateNextRun()). You can also provide a specific queue name (override GetQueue()) and you can specify whether or not it should run immediately (override ExecuteImmediately()). If ExecuteImmediately() returns false, then on the first, immediate run it wont call doExecute(), but instead will call doCalculateNextRun() and reschedule itself.

So for instance, if you want to run a background job immediately (say send an email), you make this class:

class SendAnEmailImmediately(Worker)
      

     def doExecute(self):
          logging.info("Sending emails to %s" % lemailStr)
          lmessage = mail.EmailMessage(
                          sender="Anne@example.com",
                          to="Betty@example.com",
                          subject= "Hi Betty",
                          body="I know you love email!"
                          )
          lmessage.send()


       def doCalculateNextRun(self, aUtcNow, alastDue):
           return None # never reschedule

To kick it off, do this:

 
    lsender = SendAnEmailImmediately()
    lsender.status = 0
    lsender.enabled = True
    lsender.put()




And what do you get out of that? Well, not only does the email get sent from a background task, but afterward you'll have a SendAnEmailImmediately object in the datastore, with these properties:

    lastRunSucceeded = db.BooleanProperty()
    lastRunMessage = db.StringProperty()
    lastRunStartTime = db.DateTimeProperty()
    lastRunFinishTime = db.DateTimeProperty()
which give you information on when it ran and how the worker actually went; did it fail? If so, what errors occurred? 

How about a recurring task? Try this one, which sends an email once per hour:

class SendAnEmailEveryHour(Worker) def doExecute(self): logging.info("Sending emails to %s" % lemailStr) lmessage = mail.EmailMessage( sender="Anne@example.com", to="Betty@example.com", subject= "Hi again Betty", body="Are you feeling loved yet?" ) lmessage.send() def doCalculateNextRun(self, aUtcNow, alastDue): if alastDue: lbaseDate = alastDue else: lbaseDate = aUtcNow return lbaseDate + timedelta(minutes=60)

and again, kick it off like this:

    lsender = SendAnEmailEveryHour()
    lsender.status = 0
    lsender.enabled = True
    lsender.put()
Ok, that'll work. However, what if we want a record of each run? Then do it like this instead:

    class SendAnEmailEveryHour2(Worker)
        def doExecute(self):
            lsender = SendAnEmailImmediately()
            lsender.status = 0
            lsender.enabled = True
            lsender.put()

        def doCalculateNextRun(self, aUtcNow, alastDue):
            if alastDue:
                lbaseDate = alastDue
            else:
                lbaseDate = aUtcNow
            return lbaseDate + timedelta(minutes=60)



So now you get a recurring worker kicking off other workers, one per job.



You can see how powerful this is as a simple method of structuring background jobs!



Ok, hold onto your hats, excuse my n00bish python, and get ready for a slab of code. Here's the implementation of Worker:


##################################################################
from google.appengine.ext import db
from google.appengine.ext.db import polymodel
import logging
from datetime import datetime
from datetime import timedelta
from google.appengine.ext import deferred
from lib.pytz.gae import pytz
import uuid



class Worker(polymodel.PolyModel):
    nextDue = db.DateTimeProperty()
    enabled = db.BooleanProperty()
    status = db.IntegerProperty() # 0 = ready, 1 = running, 2 = stopped
    lastRunSucceeded = db.BooleanProperty()
    lastRunMessage = db.StringProperty() # only if
    lastRunStartTime = db.DateTimeProperty()
    lastRunFinishTime = db.DateTimeProperty()
    createTime = db.DateTimeProperty(auto_now_add = True)
    taskid = db.StringProperty()

    # override to change queues
    def GetQueue(self):
        return "default"

    # override to do first run in the future    
    def ExecuteImmediately(self):
        return True
    



    # must override to perform work 


    def doExecute(self):
        raise NotImplementedError
    

    # override to tell us when next to run 




    def doCalculateNextRun(self, aUtcNow, alastDue):
        raise NotImplementedError
    
    def Execute(self, aTaskID, aIsFirstRun, **kwargs):
        try:
            #Don't trust depickled self, go reload self
            #Nick Johnson told me not to do this - needs to be fixed

            self = db.get(self.key())
        except db.NotSavedError, ex:
            self = None
        
        lutcNow = datetime.utcnow()
        
        if not self:
            logging.warning("eek I am gone! (disappears in a puff of logic)")
        elif not aTaskID:
            logging.warning("No aTaskID, skipping")
        elif aTaskID != self.taskid:
            logging.debug("TaskIDs do not match, skipping")
        elif not self.enabled:
            logging.warning("Disabled, skipping")
        elif self.status != 0:
            logging.warning("Wrong status to execute Worker, status = %s, skipping" % (self.status))
        elif self.nextDue and self.nextDue > lutcNow:
            logging.debug("Don't run till %s, reschedule..." % (self.nextDue))
            if (self.nextDue - lutcNow) > timedelta(1):
                # don't reschedule more than a day forward

                lresched = lutcNow + timedelta(1) # add a day
            else:
                lresched = self.nextDue

            lqueue = self.GetQueue()
            
            deferred.defer(
                self.Execute,
                _queue_name=lqueue,
                _eta=lresched,
                aTaskID=self.taskid,
                aIsFirstRun=aIsFirstRun,
            )
        else:
            if aIsFirstRun and not self.nextDue and not self.ExecuteImmediately():
                logging.debug("First run, don't execute")
            else:
                logging.debug("We can execute")
                try:
                    self.status = 1 # running
                    self.lastRunStartTime = datetime.utcnow()
                    self.put()
    
                    logging.debug("Before doExecute()")
                    self.doExecute()
                    logging.debug("After doExecute()")
                    
                    self.status = 0 # ready to run
                    self.lastRunSucceeded = True
                    self.lastRunMessage = None
                except Exception, ex:
                    self.status = 0
                    self.lastRunSucceeded = False
                    self.lastRunMessage = unicode(ex)
                    logging.error(ex)

                self.lastRunFinishTime = datetime.utcnow()

            logging.debug("calculate lnextRun")
            lnextRun = None
            try:
                lutcnow = datetime.utcnow()
                lnextRun = self.doCalculateNextRun(datetime.utcnow(), self.nextDue)
            except Exception, ex:
                logging.error(ex)
            
            if lnextRun:
                logging.debug("got lnextRun, need to reschedule")
                self.nextDue = lnextRun
                self.status = 0
                self.put()
            
                lqueue = self.GetQueue()
                
                if (lnextRun - lutcnow) > timedelta(1):
                    lresched = lutcnow + timedelta(1)
                else:
                    lresched = lnextRun
                    
                if lresched <= lutcnow:
                    # run immediately, no eta provided
                    deferred.defer(
                        self.Execute, 
                        _queue_name=lqueue, 
                        aTaskID=self.taskid, 
                        aIsFirstRun=False
                    )
                else:
                    # schedule future run
                    deferred.defer(
                        self.Execute, 
                        _queue_name=lqueue, 
                        _eta=lresched, 
                        aTaskID=self.taskid, 
                        aIsFirstRun=False
                    )
            else:
                logging.debug("no lnextRun, we are finished.")
                self.status = 2
                self.put()

    # Need to override put to kick off the task if enabled is 
    # set to True
    def put(self, **kwargs):
        lneedPut = True

        # first grab a copy of what's currently stored.
        logging.debug("Entered put, new self = %s" % (self))

        loldself = None
        if self.enabled:
            logging.debug("Need to find out if enabled has been newly set. Load old self from datastore")
    
            try:
                loldself = self.get(self.key())
            except Exception, ex:
                logging.error(ex)
                loldself = None
    
        logging.debug("See if newly enabled has changed")
        if self.enabled and (not loldself or not loldself.enabled):
            logging.debug("Newly enabled. Need to schedule self to run")

            self.taskid = unicode(uuid.uuid4())
            logging.debug("taskid == %s" % (self.taskid))

            logging.debug("Now schedule to run immediately")
    
            #self.nextDue = None
            self.status = 0
            
            logging.debug("Pre-save")
            super(Worker, self).put(**kwargs)
            lneedPut = False
        
            lqueue = self.GetQueue()

            # run immediately                
            logging.debug("call deferred.defer")
            deferred.defer(
                self.Execute, 
                _queue_name=lqueue, 
                aTaskID=self.taskid, 
                aIsFirstRun=True
            )
        else:
            logging.debug("not newly enabled")

        if lneedPut:
            logging.debug("Do the actual put")
            super(Worker, self).put(**kwargs)
##################################################################
A couple of footnotes before I leave you with this:

1: I've used deferred.defer on a class method, which has an issue. Specifically, it has to pickle the whole class, then depickle it when the task is run. That's a little expensive, and it leaves the running task with an old version of the object. So, I have to do this:


self = db.get(self.key())



to replace the passed in version of the object with the object from the datastore.



What would be better would be if Execute were a class method, and I passed the self.key() as a parameter to it, then loaded the full instance using the key on entry to Execute. It's a simple change, but I want to test it before I change it here. I'm sure people will point out all kinds of issue, so I'll wait to change it until then.



2: You'd think I'd have some kind of "Go()" method to kick things off, instead of using an override on put() to detect a change in the "enabled" property. However, I've been specifically using this in the context of a REST api, where I don't want to be calling methods. So, this method of overriding put() has been just the ticket. To complete the implementation I should also do some monkey patching of db.put(), but I haven't needed that yet and it's a minor PITA to do, so it's left to the reader for now. Actually, this approach of overriding put() to do work in a REST context is a paradigm I'll explore in detail in a subsequent post, it goes really well with the rest library appengine-rest-server. 









Posted by



at





No comments:
  











































The Amazing Story of AppEngine and the Two Orders Of Magnitude
















My first shot at blogging about AppEngine was the four part series about the new pricing model and how it'll affect me with my app Syyncc.






I need to do some more followup, because I think I was wrong about how instance pricing works (there's some discussion of that in the comments on the last post I think). If so, the picture is even rosier than I painted it!









Posted by



at





No comments:
  











































Frist






















First post for my professional AppEngine blog. Hi all!



I've been inspired to write this blog based on my work with AppEngine. Until recently, that work was fairly light on, comprising mostly my application Syyncc . However, I recently took on a new role with Ecampus, in which we're pushing forward with new product development in AppEngine. And in beginning that work I've learned some things.



Firstly, I've learned that I barely touched on the capabilities of AppEngine and web development with Syyncc. That app has some interesting back end work going on, but in terms of web apps it's fairly primitive. So, as I now contemplate some serious development in AppEngine (and by "contemplate" I mean "struggle to do"), the gaps in my knowledge are making themselves known.



Secondly, it's becoming clear that AppEngine is deep and wide. This isn't just some toy system, it's got chops. And while it might look a bit like LAMP stack style virtual web hosting, it is much more than that. But, it is also different. It's a serious distributed high level PaaS environment, and to make decent use of it, I'm going to need to understand it on its own terms. So for instance, the HRD is not like a SQL database, it's like HRD.



Thirdly, I'm going to need some frameworks. AppEngine provides amazing features, but they tend to be platform focused, rather than application focused. For professional calibre work, you need libraries, frameworks, to bridge the gap. There do seem to be things around the place, but they're a bit dispersed, many are open source projects in states of gentile decline, and generally there's not much of a big picture. So I'm going to need to do some hard yards pulling a pro framework together myself.



In light of these things, I've decided that I need to put a serious effort into skilling up and creating a codebase of app frameworks. I'll be diving deep into various AppEngine related topics, cutting code, testing hypotheses.



And you know, why not flaunt my ignorance in public? Thus, this blog. I invite you, dear reader, to watch as I stumble my way blindly through this new landscape, skin my knee here and there, and possibly create something useful along the way. And, if like the look of those skinned knees and elbows and want some of your own, then jump in, speak up, and we can plow on painfully together!









Posted by



at





1 comment:
  


































        

      









Subscribe to:
Posts (Atom)