New Year's Python Meme

As suggested by Tarek, and done by so many others.

1. What’s the coolest Python application, framework or library you have discovered in 2011 ?

Requests. I've dealt with Python's HTTP handling on numerous occasions, and most times found myself constructing an abstraction layer to deal with it. Requests provides this abstraction layer for pretty much all of my use cases. I'll give an honourable mention to Pyramid, because it allowed me to bring together my current use of Pylons and my past experience with Zope.

2. What new programming technique did you learn in 2011 ?

Sorting algorithms ;). On a more serious note, I started actually getting benefit out of Scrum, which is not quite a programming technique but definitely help my programming. On a related note, I learned the benefits of continuous deployment, and some of the techniques one can use to support that.

3. What’s the name of the open source project you contributed the most in 2011 ? What did you do ?

Python Web Client. But I built that myself. I did file an accepted (and quickly fixed) bug in PyDev.

4. What was the Python blog or website you read the most in 2011 ?

Planet Python. Pretty much my go to for Python, though I'm tempted to start reading Python Reddit (I've never used Reddit before).

5. What are the three top things you want to learn in 2012 ?

Chef (and related tools), gevent, Go. I'm also interested in (and more likely to learn) CoffeeScript and Node JS.

6. What are the top software, app or lib you wish someone would write in 2012 ?

• A web service mocking library. I've seen some promising signs in this direction, and I have cause to contribute myself to the effort.
• A self-replicating Python platform, that can deploy itself from device-to-device relatively transparently. I want to be able to say "run code X on instance Y", and not worry about any of the details of preparing Y or moving X to it.

If you want to participate:

• copy-paste the questions and answer to them in your blog
• tweet it with the #2012pythonmeme hashtag

An end to the experiments (for now)

Well, it's been 3 months, and I've gotten a decent number of things written. It took much longer than I was expecting, but I learned a lot of things, and made some good contacts. But most excitingly, I got a job! Last week I accepted a position working for Arc90, most prominently of Readability fame, but they've got plenty of other things going on.

I first heard about them when Kenneth Reitz started working there, and blogged about it. As it turns out, they were looking for someone with test automation experience just about the time I started getting serious about sending my resume around.

I'll be working with the Insight project team, building test automation and reporting tools to ensure confidence in the quality of every release. It's a very different scale from my work at RIM, and I think a very good opportunity for me to succeed or fail entirely on my own merits (I'm going to succeed, btw).

Which brings us to this blog. For the next few months, I don't expect to have much time to commit to writing here. Kenneth has certainly done an admirable job of keeping up blogging (and working on Requests!), so perhaps I will be able to as well. I've done a bit of research on some of the other topics I wanted to write about, and there's a post I think I owe someone for the help they have given me, but Arc90 is going to be my focus.

So this is an end, to this blog as it was. But one of the reasons I chose Arc90 as an employer is their support for blogging and open source work, so I do expect to be back writing before too long. Thanks to everyone who has read and commented on any of this!

Python Web Client 0.2 Release

I'm pleased to announce the 0.2 release of the Python Web Client. It now meets a bare minimum of functionality to be useful. Specifically, it provides a web UI to allow you to put together and submit HTTP requests, specifying request type, URL, parameters and headers.

You can view a demo here.

Oh, and it's hosted on Github here.

What's new in 0.2?

- Added display of response body
- Added option to select request method
- Added field to add parameters
- Added field to add headers

What does the future hold?

Well, there are a lot of potential ideas for where this could go, both in the original SoCal piggies proposal, and on the Rally site where I've been tracking this (log in with alecmunro+public@gmail.com:Experiments). However, I expect to resume my place among the employed soon, and thus far I haven't had any feedback on this. So barring extreme boredom on my part, this is probably the end of the line until someone forks it.

Python Web Client 0.1

The release
I've polished a few rough edges, and I'm now (somewhat) proud to release Python Web Client 0.1. It's still lacking in features, but it's got docs and an easy way to start it, and what's more, its pieces work on non-Windows platforms. I've also handicapped it a bit to allow me to put up a hosted version for people to try out, here.

What's next
Well, with this "release", the foundation of the project should be more or less complete (though I don't have it in continuous integration yet). That should make it pretty safe to add features. I'll see what I can put into it over the next few days and get a 0.2 release out, that will hopefully be more useful. If I people are interested, we might make it to 0.3, at which point I'll try to get it on PyPi (it would be nice to have a project up there).

Sidenote
I'm not sure there's a good solution, but I find all the context-switching that goes along with setting up a new project to be a real drain on productivity.

Python Web Client 0.1.dev1

tl;dr
I built a preliminary version of the Socal Piggies Python Web Client. Take a look at the code and methodology, suggest features/fixes, and otherwise enjoy! There's no live demo because of a gaping security hole.

Intro
First off, I would just like to express my sincere appreciation for everyone who responded to my request for ideas. There were a number of interesting options, but for now, I've decided to build an implementation of the Python Web Client described on the Socal Piggies site. I made this choice because it's a comfortable area for me to work in, it's a tool I can see using, and probably most important, I think I can build a simple version fairly quickly. :)

Design
The first thing I usually do with a new project is retreat to a quiet corner with a notebook and a writing implement. It would be nice to find an electronic way to do this, but so far nothing has come close to what I need in terms of offering a combination or structured and free-form input, along with instant availability.

In this case, I was trying to strip the concept down to the bare essentials. In this case that means:

• One page, with two widgets:
• Create request (Enter a URL)
• Display response (Status Code + Headers)
• Startup script that launches a browser to the service

That last one may seem non-essential, but speaks to my philosophy that "delivery is as important as development". In practice, that means that how a client is introduced to functionality is just as important as how well that functionality works. Make it really easy to start using.

Because this is somewhat of a showcase project, there's a couple of other things I pinned on my design list:

• Testing (unit, functional, system, jsunit)
• Docs (UI + API, published in Sphinx)

While I was putting this together, I also wrote down a whole lot of nice-to-have features for later. You can check them out on the Rally site I am (kind of) using to manage this project. You will need to log in:
alecmunro+public@gmail.com:Experiments

Testing
Because I'm a TDD advocate, let's do that. So first I need to decide what tests I want to have. Since I don't know my code structure at all yet, I'm going to start with system level tests, which I usually define as something that tests the system at the UI level, running very close to how it will run in production:

• Visit site, enter URL, press submit, verify results.

Gist-It for test_ui.py

That pretty much does it, and can be done with something like Selenium, unless I also wanted to test the startup script. Doing so would involve using something like Sikuli(which I do love), to observe the state of the desktop, but that might balloon the scope of this project a bit too much. So we are on to functional testing, in this case defined as testing the API of the web service, in as isolated an environment as we can create. So what are we looking at there?

• Submit URL, verify response
• Probably some variants of this, to test error handling or redirect responses (we are handling those, right?).
• What happens if the URL to retrieve is the URL of the webservice itself? Could we experience some nastiness there?

So both of the test types we have addressed so far require an actual connection to another server. We could use something that's always going to be available, like www.google.com, but we aren't really guaranteed a network connection. So for this, I'll write a small web server that can be set to return whatever you want it to.
Gist-It for echo_server.py

This was actually a bit trickier than I anticipated, due to the need to run the server in a separate thread/process, and this bug. Anyway, here's the API tests:

Gist-It for test_apis.py


Ok, so unit tests now. From our earlier tests, it's become pretty clear that the API will have one view, which accepts the details to construct a request (just a URL to start), submits that request, and returns the status code and headers from the response.

Well, that was all really boring. Maybe the jsunit tests will be more interesting? In practice, I probably won't write these before the code, because it still takes me a while to get into the rhythm of writing tests for javascript. I need a bit of trial-and-error.

• Create Request Widget:
• Enter text and press submit. A call should be made to create the request, and the deferred for that call should be passed to the page.

Gist-It for test_create_request.js

• Display Response Widget:
• Supply it with various responses, and confirm that they display properly. Probably the most interesting bit of testing of the whole lot.

Gist-It for test_display_response.js

Implementation

Ok, so we have our tests, perhaps. Now, on to the implementation. This part is actually really simple.

There's the Python view:
Gist-It for views.py

and the two Javascript widgets:
Gist-It for create_request.js

Gist-It for display_response.js


Feel free to take a look at the GitHub repository for more details (or check it out to run it). 

Documentation
I've left off the documentation for now, both because I wanted to get this up soon, and it's been a while since I started anything with Sphinx, and also because it really doesn't do much yet. I'm also still missing the launch script.

Conclusion
There's lots more work to be done here to make something useful, so I'm taking suggestions. But hopefully this gives you an idea of how you can build a simple and somewhat tested web-app using Pyramid and JQuery. You will notice that it is very testing heavy, probably significantly more than real-world deadlines would allow for. But once you get these tests in place (and a system to run them), they are fairly easy to build on, and can provide a safe container in which to experiment.
For the moment, I'm planning two distinct iterations:

• Add docs and launch script, as well as displaying the response body. That will be 0.1
• Flesh out the request creation ability, to allow settings headers and parameters. Along with hopefully some fixes/refinements, that will be 0.2

Beyond that, development will depend on whether this is interesting to anyone, so let me know.

Sorting out

As someone who almost exclusively uses high-level languages, the idea of writing my own sorting algorithm seems a bit silly. However, I can certainly appreciate the algorithms used for sorting, and it's a neat opportunity to try to document and implement these algorithms using methods I am familiar with.

Preface

These implementations are likely to be extremely naive. If you went CS school, you probably learned better implementations in your first year. There is absolutely no reason to use these in production code anywhere. The built-in python sorting is 10-50 times faster than anything I've produced (I'm actually surprised it's not more). If you've got tips about how to speed these up just by changing the code around, feel free to comment, or you can try forking my repository and sending a pull? request. I can't promise to spend much time on it, because this is pretty well-trod territory, but it would be a bit entertaining to see how close we can get.

Quick Sort

My understanding of quicksort is still developing, but based on what I've read on Wikipedia, it can work something like this:

1. Grab an item in the list, called the "pivot".
2. Create two lists, representing everything greater than the pivot, and everything less then the pivot.
3. Repeat steps 1 and 2 on the two sub-lists, returning the a list comprised of the smaller list items, then the pivot, then the larger list items.

In this way, the entire list should get sorted. It's my understanding there are various optimizations that can be done, such as swapping elements instead of creating new lists. It seems to be one of the "best" all around algorithms, though for certain cases, it's performance can be beaten by others.

Gist-it for quick_sort.py

This was the second fastest list implementation I created, at about 1/12th the performance of the built-in sorting.

Merge Sort
Merge sort relies on splitting the list recursively until you have an ordered hierarchy of l item item lists. Then you compare each list with it's neighbour, and return a combined list of the two merged together. Now you have 2-item sorted lists, which you work through, comparing the left-most element to its neighbour's left-most element, and moving the lowest one to a new list, eventually creating a 4-item list.

Gist-it for merge_sort.py

Heap Sort

Heap sort relies on understanding what a "heap" is, in this context. So let's do that.

A heap is a data structure that is tree based, and follows the rule that all children of a node will have a value that is lower than or equal to the value of that node (for a max-heap, for a min-heap the rule is reversed). In practice, heaps are usually implemented within arrays, with the first or last item of the array being the root of the tree.

Now, why do we do have a heap? What's the advantage? Apparently sorting things. :)

In a heap sort, we first build the heap from the list we've been given. Once we have our heap, we know the item with the highest value is at the root of the heap, which is position 0 in the list representing the heap. So now we swap the first and last elements in that list, and either move the last element from the list, or simply treat that part of the heap as already sorted (which means our heap needs to know to ignore it somehow). Then we take the first item, and trickle it down through the heap until the heap is proper again. This consists of comparing it's value with it's children, and swapping it with the higher one of these, until one of these is not higher.

Gist-it for heap_sort.py

Python apparently does have a built-in heap library, which might be worth benchmarking, as heap was my slowest sort implementation.

Bubble Sort

Bubble sort is fairly trivial.

Iterate through the items, except the last one, if an item is greater than the next one, swap their positions. When you reach the end of the list, do it again, unless there were zero swaps.

Gist-it for bubble_sort.py

Insertion Sort

Insertion sort is another trivial sort. I'm really only including it because I think my (unwitting) choice of an insertion sort in an interview may have cost me a job.

Gist-it for insertion_sort.py

Interestingly enough, for my stupid example of sorting a small list of characters, insertion sort is the fastest implementation I created.

Acknowledgements and Conclusion


Not really much to conclude, except that I now have a better understanding how to write a sorting algorithm, and some of the pros and cons of different algorithms. I certainly can't look at a data set and give you the big O case for a given algorithm.

But one thing I do want to draw special attention to is Wikipedia's pages on this subject. They vary a bit in information density, but they were incredibly informative, and make use of numerous methods to help communicate how each sort works. If you really want to understand this stuff, I would definitely start with these pages:

• Quicksort
• Merge sort
• Heap sort
• Bubble sort
• Insertion sort

Also, my thought to do this post was sparked by another post on Planet Python that mentioned Insertion Sort, where I recognized the algorithm. I'm not entirely sure, but this may be that post.

Let me build you a web app/service!

So, according to my introduction, one of my goals for this project is to build a web app or service to demonstrate my skillset in that area. Unfortunately, it's turns out my idea for a web app has already been done, and is quite well-known (on the plus side, I now understand what 4Chan is). At the moment, I'm drawing blanks about what I should build, so I thought perhaps I could crowdsource that job.

So if you have an idea for a web app or service, but not the time or skills to get it started, or if you would simply like to collaborate with someone on it, post a comment describing your idea.

In general, I'm game for pretty much anything, as I'm confident I can learn whatever I need to. Some general parameters though:

• It's going to be in Python, built on Pyramid, using SQLAlchemy for models
• I will try to strip the idea to it's core, and leave others to make it full-featured
• I will try to make an appealing and usable UI, but again, it will be pretty barebones
• I'm going to blog about it, host it in github and make the code viewable by everyone
• If you want to collaborate:
• I'm open to hosting it elsewhere, as long as it's publicly accessible
• I will be a stickler in design discussion and code reviews

Finally, I will happily put it under any license you want and grant you and copyright or whatnot, as long as it doesn't prevent me from achieving my objectives vis-a-vis skill demonstration.