Dedicating Developers to Framework is Scaring

Foreword: The word "framework" is used but could be replaced by "library", "platform", or even "dsl" in the following lines.

Developers should not be dedicated to working on a framework. The framework is not a goal, the goal is its use. Developer working on a framework should also produce applications using the framework. Even better, developers using the framework should have some percents of their time for working on it to serve their use. Bonus: Such developers would then master the framework internals, strenghts and limits.

Working on the framework only would be lead by suppositions, bets and egos. Even answering user developers' needs by dedicated "framework developers" would miss the mark, there is a gap between expectations and solutions. Framework developers would work to improve their own life (which is natural, we are humans) making choices based upon the easiness or complexity of evolving the framework more than its use.

Users of the framework working on its improvement would really tailor the framework to improve their live when using it. Moreover the users would then master the framework its strengths and limitations. Isn't the framework goal to serve the making of end-users solutions ? Thus, choice should be led by improving the work of application developers and not framework dedicated developer.

Corollary of this approach, the framework emerge from its true nature which could be summaries as "How would I ideally like to write this application in order to deliver value to the end user?" An emerging framework sounds "agile", I admit. The concept of "developing a framework upfront, and one for all" sends a chill down my spin, but it looks like common practice.

Frameworks should not be extended because it sounds cool, but only because of usage necessity. Framework should stay light and not trying to achieve all possible goals of a framework developer mind. It should follows Pareto's law and incorporate what is necessary 80% of the time, while being open to easily perform the remaining 20% "by hand". Am I courageous enough to state that it should follow the Single Responsibility Principle? Close to.

I love working on frameworks, but when I forget the true goal, which is serving the delivery of value to end users, the fun making the framework does not lead me to really improve the delivering of this value. Moreover, afterwards it reduces the ability to serve end users, making choices not based on "good solutions" but on solutions that fit the framework.

Having a framework coordinator sounds good to me, but not a framework dictator disconnected from end users / framework use in the field.

Two years...

Without feedback on this channel. [Sic]

The most notable change is that instant feedback is now fully non electronic. Pencil and paper for sketching and drafting. Fast, cheap, human friendly.

Django: One application, multiple instances

A current project requires to be able to have multiple instances of a single web application application running in different process and with segregated databases. In the meantime, the core of the application should not be duplicated.

A solution to this problem using Django is to create 'Shell projects'. Their goal is to import the core application settings, and to overload only the required parts : right now the database file. Each shell project live in its own folder.

  $ django-admin.py startproject core
  # write the application...

All the application is implemented and tested in core. In the instance1 folder we only find the overloading of parts of the core application.

  $ create_shell_application.sh instance1
  $ ls instance1
  __init__.py*        manage.py*        settings.py*
  $ more instance1/settings.py
  from core.settings import *
  DATABASE_NAME = 'instance1.sqlite3'
  # + instance1 specific configuration if needed...

Starting the instance1 application (with ./manage.py runserver) makes it run with the common code, but its own database instance. And one can have as many instances as needed. The core application is to be packaged as an Egg and installed on the target laptops.

Agile Tour 2009 (Lille)

Friday was the Agile Tour 2009 stop in Lille. I attend an excellent workshop session about the Theory of Constraints presented by Pascal Van Cauwenberghe (session material). I realized how much I was not focusing on one action at a time, thinking of the next possible improvement before really applying the one we had just agreed on... definitely worth the attending.

GTAC 2009

Just Spent two great days at GTAC 2009. All the talk were interesting. All the people I met were interesting. And the Google facility in Zurich really looks like a fantastic place to work. It made me remember of a documentary describing the Xerox PARC in the 70s, same spirit.
In addition, while I was not that much exited about Waves, being using it with about 100 other people for two days was really great. Seeing how the group working on the Ruby WebDriver API nicely illustrates what can be achieved with Waves for techies.

More than a year...

...without any feedback.
Lots of questions, lots of projects, and now trying to start a company. Back to the first post, it is all about "Satisfaction Driven Development". Revamping some previous prototypes in order to support the development of satisfying software for end users.

Emacs and TextMate

I really enjoy using TextMate, especially for the support of snippets I use for live coding sessions while teaching. I know that every thing is feasible with Emacs, but did not have time nor energy for implementing such a nice snippet management system (I only used skeletons with dynamic abbrev). Yesterday, I came accross YASnippet and found it providing the same support as TextMate for snippets. Together with pair-mode it is again a great pleasure to be back with emacs full-time.
Thanks a lot pluskid!

Test Driven CherryPy Applications (Part 1)

Context

Model-View-Controller (MVC) pattern eases the development and improves the quality of web applications. Test-Driven Development (TDD) contributes also to the quality of the application and ideally each piece of the application should be testable. While writing unit tests for the controller and model layers is plain Python unit testing, writing tests for the view layer is not as direct.

CherryPy allow us to write plain Python classes for the view layer, but testing these classes functions means comparing expected HTML documents to actual ones. Frameworks such as Twill imply to run the application in order to test its view layer. We are going to use two other strategies for unit testing view classes as any python classes: Using fake template rendering and using BeautifulSoup that eases the querying of HTML documents.

Finally, in the following we are going to stick to the Single Responsibility Principle and reduce the responsibility of the view layer to presentation: selecting the proper template to be used, an relying on the controller to perform the real work.

Faking Template Rendering

The first strategy for testing view classes is to use fake templates. This imply that every view class is written using the following structure: The class explicitly depends upon a function that is used for rendering HTML.

In addition to supporting testing through the fake template rendering, this approach allow you to easily substitute template engines. The following example present three possible implementation of such a rendering function. These are not using any optimization such as using pre-compiled template objects. (To do so a callable object would have to be defined.)

def render_string_template(template, values):
    return string.Template(template).substitute(values)

def render_cheetah_string_template(template, values):
    return Template(template, searchList=[values]).respond()

def render_cheetah_file_template(template, values):
    return Template(file=template, searchList=[values]).respond()

Let's consider the following stories for a simple application.

• The user receive a form if she does not provide her name.
• The user receive a greeting if she has provided her name.

The following tests check that when the greeting() function is called, it uses the proper template with the proper parameter values. The test also checks that the function is exposed as it is required by CherryPy. (An exposed function is mapped to an URL and can be request from the web client.)

class FakeRender:
    def __call__(self, tmpl, values=None):
        self.tmpl = tmpl
        self.values = values


class TestMyView(unittest.TestCase):

    def setUp(self):
        self.render = FakeRender()
        self.view = MyView(render=self.render)

    def test_greeting_is_exposed(self):
        self.assertTrue(self.view.greeting.exposed)

    def test_greeting_returns_a_form_if_no_name_is_provided(self):
        self.view.greeting()
        self.assertEquals('form_tmpl', self.render.tmpl)
        self.assertEquals(None, self.render.values)

    def test_greeting_returns_a_greeting_with_the_provided_name(self):
        self.view.greeting('world')
        self.assertEquals('greeting_tmpl', self.render.tmpl)
        self.assertEquals(dict(name='world'), self.render.values)

The following implementation passes the test.

class MyView:

    def __init__(self, render):
        self.render = render

    @cherrypy.expose
    def greeting(self, name=None):
        if not name:
            self.render('form_tmpl')
        else:
            self.render('greeting_tmpl', dict(name=name))

Remarks

This first solution of decoupling the rendering from the view permits to check that the proper template is used and the expected values are provided to the template. This can be reach without lauching the application server keeping an easy and faster to run test suite. This does not check that the rendered HTML document contains this information which is more related to template testing. It permits to focus on the testing of validation functions and proper call to controllers. Finally, I can let an more-competent-than-I-am person to set up a set of templates in order to have a nice looking web interface.

Instant Feedback: A 10 minutes example.

You want an example to see what Instant Feedback can be? For this purpose, I will use ezWeb which is a small tool I prototyped for experimenting Instant Feedback. It is a Python package for iterative development of web application with a goal to write as less code as possible to get a default version, while sticking to habits when developing Python web applications in order not to be limited afterwards.

Sue needs a system for managing Seminars for knowldge diffusion to students. Sue have identified two main actors: herself as organizer of seminars, and students as attendees. For the moment the system is not required to manage speakers. For the moment, the most important stories are (with their id in parenthesis):

1. A student can list the forthcoming seminars (ListSeminars).
2. A student can view the summary of a seminar (ViewSeminarSummary).
3. A student can register for a seminar (RegisterToSeminar).
4. A student can view the participants of a seminar (ViewSeminarParticipants).

We can define a description of these stories using ezWeb in order to have a skeleton for our web app.

from ezweb.stories.model import *
from ezweb.stories.cherrypy import generate

student = Actor(
name='Student',
usecases=[
  UseCase('View list of seminars'),
  UseCase('View the summary of a seminar'),
  UseCase('View the participants of a seminar'),
  UseCase('Register for a seminar')
]
)

seminars = Application(
name='Seminars',
actors=[student]
)

generate(seminars)

We start with the ListSeminars story using Instant Feedback. Sue says: "There is no information to provide, and for each seminar we want: the name of the seminar, its date, and the room where it takes place."

Using ezWeb we can translate her specification as:

from ezweb.pages.model import *
from ezweb.pages.cheetah.generate import generate

# we need a list
list_of_seminars = List(
name='list_of_seminars',
details='get_details',
content=[
 ListItem('name'),
 ListItem('date'),
 ListItem('room'),
]
)

# the page contains only the list
view_the_list_of_seminars = Page(
name='view_the_list_of_seminars',
css='/css/style.css',
content=[
 Header(title='Seminars'),
 sidebar,
 Main(name='List of seminars', content=[list_of_seminars]),
 Footer(content=[footer])
]
)

seminars = Site(
name='Seminars',
module='seminars',
pages=[
   view_the_list_of_seminars,
]
)

generate(seminars)

Note: A page is split in four parts (only the Main is detailed above):

• Header: name of the application,
• Menu: containing links to the various features,
• Footer: containing recurrent information,
• Main: containing the input / output for a feature.

The generate() function produce templates pages for the application, which are generated in the same folder as the application skeletons (generated from the stories). With this small amount of information we can already present a UI to Sue in order for her to provide feedback.



Sue says that it does not looks like a list, couldn't she see an example of seminars?

The following code excerpt is the one generated by default from the description of scenarios.

class ViewListOfSeminars(object):

  def __init__(self):
      pass

  @cherrypy.expose
  def index(self):
      return bindings.view_the_list_of_seminars()

To provide Sue a more accurate vision, we simply add fake objects in the default code for the feature.

class Seminar(object):
  def __init__(self, name, date, room):
      self.name = name
      self.date = date
      self.room = room

class ViewListOfSeminars(object):

  def __init__(self, ctrl):
      self.ctrl = ctrl

  @cherrypy.expose
  def index(self):
      seminars = [
          Seminar('Agility, XP, Scrum', '2007-05-15 14:00:00', 202),
          Seminar('Java Generics', '2007-05-29 14:00:00', 202),
          Seminar('Python', '2007-05-22 14:00:00', 202),
          ]
      return bindings.view_the_list_of_seminars(seminars)

Then we can show Sue a more realistic list of seminars. This time she just says nice.



On the basis of this small interaction, we now have an acceptance test for the story ListSeminars. On this basis we can write an automated test, and really implement the story: defining the database tables together with the code to extract information from the database (i.e. the true implementation for ViewListOfSeminars.index).

Tools such has ezWeb may be implemented in various languages and environments. The time it took to develop it (about 2 weeks --but only two dozen hours of work-- for a usable version of ezWeb in the context of CherryPy, Cheetah, SQLObject) worth the ROI (return on invest) as we use it in the project for air quality.

Instant Feedback: Is this what you mean?

I like TDD because it gives me feedback. I know if my code pass the expressed specification. Notice the expressed qualifier of specification.

When speaking together, an idea lives at four different levels. First, the idea lives in my mental representation. Second, it try to express as best as I can the idea with my words and the meaning of these words in my mental knowledge. Third, you hear these words with the meanings of these words in your mental knowledge. Finally, the idea lives in your mental representation on the basis of your understanding of the words you just heard.

To make it short: There is a (huge) gap between our mental representation of the same idea. And the less we share common meaning for the same words, the bigger is the gap. Imagine discussing with an end-user who is a medical doctor (GP) in order to build a system to manage her office. The gap is an ocean.

Going back to the TDD loop, the faster I go from the spec to a Running Tested Feature, the faster the end user can confirm my understanding of the idea she expressed. The motivation of Instant Feedback is aimed at reducing the gap and increasing the speed to transcript the end-user request to a piece of software. Its roots are Agility of course but also rapid application developpement and live prototyping.

Instant Feedback can be defined as a 15 minutes exercise: 5 minutes for bluiding the user interface and an empty shell of the function, and then 10 minutes for make it run (even with hard coded data). These 15 minutes are performed together with the end user: We act as a compiler from her words to a program. It requires powerful tools for allowing this rapid prototyping approach. Moreover, the goal is not to throw away this piece of description and software but to use it as a basis for the complete developments.

The integration of Instant Feedback in an agile development cycle can be done as follows:

1. We start by a User story workshop (User Stories Applied, M.Cohn).
2. The end user select the user stories she want to be implemented during this iteration.
3. For each user story we discuss the details and use Instant Feedback when discussing the way the end user will interact with the application and the behavior of the application. Then, we move on to the implementation of the story.

The result of the Instant Feedback exercise can be seen as an acceptance test for the story if you do not wish to stick with its result. But it can also be a basis for the true implementation of a feature.