Python Web Client 0.1.dev1

tl;dr
I built a preliminary version of the Socal Piggies Python Web Client. Take a look at the code and methodology, suggest features/fixes, and otherwise enjoy! There's no live demo because of a gaping security hole.

Intro
First off, I would just like to express my sincere appreciation for everyone who responded to my request for ideas. There were a number of interesting options, but for now, I've decided to build an implementation of the Python Web Client described on the Socal Piggies site. I made this choice because it's a comfortable area for me to work in, it's a tool I can see using, and probably most important, I think I can build a simple version fairly quickly. :)

Design
The first thing I usually do with a new project is retreat to a quiet corner with a notebook and a writing implement. It would be nice to find an electronic way to do this, but so far nothing has come close to what I need in terms of offering a combination or structured and free-form input, along with instant availability.

In this case, I was trying to strip the concept down to the bare essentials. In this case that means:

• One page, with two widgets:
• Create request (Enter a URL)
• Display response (Status Code + Headers)
• Startup script that launches a browser to the service

That last one may seem non-essential, but speaks to my philosophy that "delivery is as important as development". In practice, that means that how a client is introduced to functionality is just as important as how well that functionality works. Make it really easy to start using.

Because this is somewhat of a showcase project, there's a couple of other things I pinned on my design list:

• Testing (unit, functional, system, jsunit)
• Docs (UI + API, published in Sphinx)

While I was putting this together, I also wrote down a whole lot of nice-to-have features for later. You can check them out on the Rally site I am (kind of) using to manage this project. You will need to log in:
alecmunro+public@gmail.com:Experiments

Testing
Because I'm a TDD advocate, let's do that. So first I need to decide what tests I want to have. Since I don't know my code structure at all yet, I'm going to start with system level tests, which I usually define as something that tests the system at the UI level, running very close to how it will run in production:

• Visit site, enter URL, press submit, verify results.

Gist-It for test_ui.py

That pretty much does it, and can be done with something like Selenium, unless I also wanted to test the startup script. Doing so would involve using something like Sikuli(which I do love), to observe the state of the desktop, but that might balloon the scope of this project a bit too much. So we are on to functional testing, in this case defined as testing the API of the web service, in as isolated an environment as we can create. So what are we looking at there?

• Submit URL, verify response
• Probably some variants of this, to test error handling or redirect responses (we are handling those, right?).
• What happens if the URL to retrieve is the URL of the webservice itself? Could we experience some nastiness there?

So both of the test types we have addressed so far require an actual connection to another server. We could use something that's always going to be available, like www.google.com, but we aren't really guaranteed a network connection. So for this, I'll write a small web server that can be set to return whatever you want it to.
Gist-It for echo_server.py

This was actually a bit trickier than I anticipated, due to the need to run the server in a separate thread/process, and this bug. Anyway, here's the API tests:

Gist-It for test_apis.py


Ok, so unit tests now. From our earlier tests, it's become pretty clear that the API will have one view, which accepts the details to construct a request (just a URL to start), submits that request, and returns the status code and headers from the response.

Well, that was all really boring. Maybe the jsunit tests will be more interesting? In practice, I probably won't write these before the code, because it still takes me a while to get into the rhythm of writing tests for javascript. I need a bit of trial-and-error.

• Create Request Widget:
• Enter text and press submit. A call should be made to create the request, and the deferred for that call should be passed to the page.

Gist-It for test_create_request.js

• Display Response Widget:
• Supply it with various responses, and confirm that they display properly. Probably the most interesting bit of testing of the whole lot.

Gist-It for test_display_response.js

Implementation

Ok, so we have our tests, perhaps. Now, on to the implementation. This part is actually really simple.

There's the Python view:
Gist-It for views.py

and the two Javascript widgets:
Gist-It for create_request.js

Gist-It for display_response.js


Feel free to take a look at the GitHub repository for more details (or check it out to run it). 

Documentation
I've left off the documentation for now, both because I wanted to get this up soon, and it's been a while since I started anything with Sphinx, and also because it really doesn't do much yet. I'm also still missing the launch script.

Conclusion
There's lots more work to be done here to make something useful, so I'm taking suggestions. But hopefully this gives you an idea of how you can build a simple and somewhat tested web-app using Pyramid and JQuery. You will notice that it is very testing heavy, probably significantly more than real-world deadlines would allow for. But once you get these tests in place (and a system to run them), they are fairly easy to build on, and can provide a safe container in which to experiment.
For the moment, I'm planning two distinct iterations:

• Add docs and launch script, as well as displaying the response body. That will be 0.1
• Flesh out the request creation ability, to allow settings headers and parameters. Along with hopefully some fixes/refinements, that will be 0.2

Beyond that, development will depend on whether this is interesting to anyone, so let me know.