Software Gardening in practice

Software gardening, as expressed by Hunt/Thomas, Atwood, etc uses the metaphor to deal with how a software project is alive, and is constantly being changed by factors outside of the control of the developers. You can plant a seed, and water it, but you can’t control the sunlight, the bugs, bees, etc. As it grows, you can weed the garden, removing features that are not used but taking up resources, add fertilizer, change the plants to better reflect your needs (why didn’t someone tell me Joe was allergic to cucumbers!?).

I want to expand on this idea, as I believe there are several valuable insights that can be derived from adopting this view. In this post I will explore this concept from three different perspectives: the project itself, a software developer, and a client that wants to get some software developed.

The Unseen Environment

As a particular software environment is like a garden, there are many factors that the garden grows out of. There are obvious factors, like the composition of the soil, and the likely availability of water, and sunlight. But there are also more subtle, “living” factors, that are going to be part of the environment of your garden, yet generally operate independently.

One of the most fascinating things I’ve learned recently is the concept of mycorrhizal networks. These are fungal layers that exist within the soil that serve to break down waste matter and distribute molecules that plants depend on for nutrition. It is generally unseen, but is a huge factor in the ability of plant ecosystems to thrive. Generally much better known, but still somewhat unseen, are the pollinators. Birds, bees, and butterflies allow plants to reproduce, and thus stimulate genetic diversity.

So, above and below, there is an environment that allows plant life to thrive. A similar role is played in software development by the ecosystem that exists around both the software, and the developers.
All developers and software libraries come to a project with a history that is shaped through the absorption of “byproducts” from previous projects. These can take the form of learned code snippets, software structure conventions, and many other things. I would liken this to the way the mycorrhizal network digests the output from various past biological processes and produces raw nutrients to begin new ones.

In a similar way, open/shared source cultures could be compared to pollinators. Moving from project to project with very few barriers, a developer can distribute efficient techniques rapidly, effecting the projects and other developers. This has resulted in the establishment of numerous best practices, to the point where high-profile open source projects are the gold standards for software quality. Another interesting way in which open source diversity is like genetic diversity is in the trend of multiple competing software projects springing up with different attributes tailored to their environments. These can continue for very long periods, sharing “genes” with each other but still continuing as distinct “species”.

In some large organizations, the software ecosystem is distinct enough to effect the software that is produced there, but at this point, almost everyone is dependent on Open Source. It produces specific codebases that underly most every modern project, such as Linux, Apache, OpenSSL, GCC, Boost, Python, etc. But it also establishes the best practices and standards for software development that are used by individual developers, whatever they are working on. Conversely, bug reports come from commercial projects that make Open Source stronger, and developer experiences there continue to inform and refine the state of the art. There are also cases where it is beneficial for commercial development to open source certain pieces of their software, to expose them to that wider diversity, and encourage contributions that will help meet business goals (though like natural evolution, predicting the future of an open source project is far from a science).

Tying that back together, that means that for a developer, every project you work on contributes to this larger community, even when the code itself is closed. Because it changes you, and it changes the conversations you have with others, and how you work to solve problems in the future. As a former user of the Zope Web Application Server, I have watched other users and contributors define the cutting edge of web software (Tarek Ziade, Chris McDonough, Alex Gaynor, and Martijn Faasen, among others).

Garden Development as a Service

As someone who works on software, your value is not the creation and delivery of a product, but rather the cultivation of an environment. Even when you are employed delivering products, there’s no such thing as a final release, just something that is able to meet the needs of the moment. Needs will change, the ecosystem will change, you will change, and new opportunities will arise. How might this change how you approach a single project, and your career?

In terms of approaching a particular project, it means you need to understand the landscape in order to effectively execute on the vision. You need to know what kind of software is already there, and what the state of third party libraries are in that particular domain. You need to plan for the inevitable bugs and maintenance. The actual implementation flows out of these concerns, not the other way around.

From a career standpoint, I should first confess that, well, I’m not a gardener. However the image I have of them is that they don’t get annoyed that all the cool gardens are already cultivated. As long as you have a willingness to work, gardens are going to need cultivation. Likewise, as long as you have a willingness to develop it, there will always be available work developing software. Under this philosophy, nothing ever reaches the point that it cannot be improved or does not require maintenance.

Gardening Agency

When we have a need for some software, and we attempt to consider the effort that will be required, we usually think about the features that we want the software to have. The gardening metaphor suggests this is a very limited way to view this. If we think of the features of a particular garden as “has Roses and Violets, and produces Tomatoes”, we can quickly understand that a good deal more information is required to get a reasonable estimate of the effort that will be necessary.

Is the garden in Hawaii, Alaska, or the Central Park Zoo? What if there’s already a garden there? What's the condition of the soil, or local animal life?

Translating these questions over to software, we can see that the environment a piece of software needs to "live" in can impact the effort required to develop it as much or more than the features it needs to have. For instance, if you are looking to add accounting features for exporting financial data to your partners, the effort could vary tremendously depending on the existence and maturity of the open source software that exists to work with data in the formats they use.

Just as working with a professional landscaper will produce dramatically different results than simply purchasing plants, establishing a relationship with someone who can look at your situation holistically is key to a successful software project. They know the questions that need to be asked, the process for finding the answers, and they know how to build out a development team that will take that reality into account.

Python Web Client 0.1.dev1

tl;dr
I built a preliminary version of the Socal Piggies Python Web Client. Take a look at the code and methodology, suggest features/fixes, and otherwise enjoy! There's no live demo because of a gaping security hole.

Intro
First off, I would just like to express my sincere appreciation for everyone who responded to my request for ideas. There were a number of interesting options, but for now, I've decided to build an implementation of the Python Web Client described on the Socal Piggies site. I made this choice because it's a comfortable area for me to work in, it's a tool I can see using, and probably most important, I think I can build a simple version fairly quickly. :)

Design
The first thing I usually do with a new project is retreat to a quiet corner with a notebook and a writing implement. It would be nice to find an electronic way to do this, but so far nothing has come close to what I need in terms of offering a combination or structured and free-form input, along with instant availability.

In this case, I was trying to strip the concept down to the bare essentials. In this case that means:

• One page, with two widgets:
• Create request (Enter a URL)
• Display response (Status Code + Headers)
• Startup script that launches a browser to the service

That last one may seem non-essential, but speaks to my philosophy that "delivery is as important as development". In practice, that means that how a client is introduced to functionality is just as important as how well that functionality works. Make it really easy to start using.

Because this is somewhat of a showcase project, there's a couple of other things I pinned on my design list:

• Testing (unit, functional, system, jsunit)
• Docs (UI + API, published in Sphinx)

While I was putting this together, I also wrote down a whole lot of nice-to-have features for later. You can check them out on the Rally site I am (kind of) using to manage this project. You will need to log in:
alecmunro+public@gmail.com:Experiments

Testing
Because I'm a TDD advocate, let's do that. So first I need to decide what tests I want to have. Since I don't know my code structure at all yet, I'm going to start with system level tests, which I usually define as something that tests the system at the UI level, running very close to how it will run in production:

• Visit site, enter URL, press submit, verify results.

Gist-It for test_ui.py

That pretty much does it, and can be done with something like Selenium, unless I also wanted to test the startup script. Doing so would involve using something like Sikuli(which I do love), to observe the state of the desktop, but that might balloon the scope of this project a bit too much. So we are on to functional testing, in this case defined as testing the API of the web service, in as isolated an environment as we can create. So what are we looking at there?

• Submit URL, verify response
• Probably some variants of this, to test error handling or redirect responses (we are handling those, right?).
• What happens if the URL to retrieve is the URL of the webservice itself? Could we experience some nastiness there?

So both of the test types we have addressed so far require an actual connection to another server. We could use something that's always going to be available, like www.google.com, but we aren't really guaranteed a network connection. So for this, I'll write a small web server that can be set to return whatever you want it to.
Gist-It for echo_server.py

This was actually a bit trickier than I anticipated, due to the need to run the server in a separate thread/process, and this bug. Anyway, here's the API tests:

Gist-It for test_apis.py


Ok, so unit tests now. From our earlier tests, it's become pretty clear that the API will have one view, which accepts the details to construct a request (just a URL to start), submits that request, and returns the status code and headers from the response.

Well, that was all really boring. Maybe the jsunit tests will be more interesting? In practice, I probably won't write these before the code, because it still takes me a while to get into the rhythm of writing tests for javascript. I need a bit of trial-and-error.

• Create Request Widget:
• Enter text and press submit. A call should be made to create the request, and the deferred for that call should be passed to the page.

Gist-It for test_create_request.js

• Display Response Widget:
• Supply it with various responses, and confirm that they display properly. Probably the most interesting bit of testing of the whole lot.

Gist-It for test_display_response.js

Implementation

Ok, so we have our tests, perhaps. Now, on to the implementation. This part is actually really simple.

There's the Python view:
Gist-It for views.py

and the two Javascript widgets:
Gist-It for create_request.js

Gist-It for display_response.js


Feel free to take a look at the GitHub repository for more details (or check it out to run it). 

Documentation
I've left off the documentation for now, both because I wanted to get this up soon, and it's been a while since I started anything with Sphinx, and also because it really doesn't do much yet. I'm also still missing the launch script.

Conclusion
There's lots more work to be done here to make something useful, so I'm taking suggestions. But hopefully this gives you an idea of how you can build a simple and somewhat tested web-app using Pyramid and JQuery. You will notice that it is very testing heavy, probably significantly more than real-world deadlines would allow for. But once you get these tests in place (and a system to run them), they are fairly easy to build on, and can provide a safe container in which to experiment.
For the moment, I'm planning two distinct iterations:

• Add docs and launch script, as well as displaying the response body. That will be 0.1
• Flesh out the request creation ability, to allow settings headers and parameters. Along with hopefully some fixes/refinements, that will be 0.2

Beyond that, development will depend on whether this is interesting to anyone, so let me know.

Design patterns

Design patterns are a very broad subject, but as far as I can tell, they are primarily common conventions for structuring software. I've never been taught them, and as far as I can tell, I've used nearly all of them, repeatedly. So for an experienced software developer, I don't believe studying design patterns is likely to improve your coding skills much, if at all. However, it could certainly improve your collaboration skills.

Words Matter
If there's one downside to being self-taught in the field of computer programming (there are several), it would be the difficulty of communicating your ideas with those who have a "standard" education. However good my code may be doesn't matter if I spend all my time trying to explain how I'm going to implement something. Formal computer science comes with a whole boatload of terminology, design patterns among them. Being able to say "use the template method pattern to defer processing to subclasses" is much more concise than "in the superclass method, call methods that are overridden in the subclass, but have only stub implementations in the superclass". Even that verbose description is concise compared to what usually comes up with discussing actual project code.

Template Method
The template method pattern is something that eventually comes in handy to anyone who does a fair amount of object oriented design. It's a response to the problem of how to mix specialized behaviour of subclasses with general behaviour of superclasses. The basic idea is that the superclass implementation of a method will contain mostly calls to other methods of the object, which will be mostly abstract in the superclass, and implemented in the subclasses, which won't implement the method being called.
I recently used this on an email reporting system, which used either HTML or text email depending on the job being reported on. There was a fair amount of commonality between both types, in that they dealt with collecting data from the environment and stuffing it into some kind of template (the use of the word template here is unrelated), then formatting that template into an email and sending it along.
The actual email sending was almost the same for both HTML and text, so with just a condition or two we could handle that entirely in the superclass. However, the data that was sent out varied a bit between the two types, as did the emails themselves.

Gist: "Template Method example"

Note that the implementation above is not ideal, because the subclasses do need a certain awareness of the superclass implementation in order to behave properly.

Mediator

The mediator pattern, in my understanding, mostly deals with managing the flow of communication between objects. It puts a layer between your object and it's intended communication target. I use this pattern frequently for building UIs, most of which I build in JQuery, so that's where I use it.

The way I build UIs in JQuery has been a constantly evolving pattern, so this may change, but right now it gives me the flexibility I need. What I usually do is create an object representing the page itself, as well as objects to represent each of the distinct interface areas, which I am going to call widgets (thought that might be too specific of a term in this context). Each of the widgets takes zero or more callback functions, depending on what they do, and the page object is responsible for seeing that these point to the same things. In some cases, they are methods of other widgets (though in typical javascript fashion, they are anonymous functions that call those methods), but more often they are methods of the page object, which accept the communication and see it dispatched to any other widget(s) that are interested. The page object is almost never more than wiring between the widgets.

The following is probably not syntactically correct. It frequently takes a bit of trial and error before I get into the swing of JavaScript OOP.

Gist: "Mediator Pattern example"

Strategy

The strategy pattern deals with abstracting away algorithms. In situations where a different context might require a different algorithm to achieve the desired results, you apply the strategy pattern. This might sounds complex, but in practice, it is as simple as using different validation functions on different data types, as long as you have a common interface to those functions.

Gist: "Strategy Pattern example"

Adapter

The adapter pattern is one that, thanks to Zope 3, I've been familiar with for a long time. An adapter is an object that "wraps" another object to provide a desired interface. Designing a system based on the adapter pattern requires using capital-I Interfaces for establishing object communication. Then, as necessary, adapters are created to allow the available objects to satisfy those interfaces.
To a certain degree, the adapter pattern takes the concept of polymorphism, traditionally understood as being able to use a superclass' interface to interact with instances of subclasses, and expands it. So you can use the specification of an Interface to interact with any objects that are adaptable to that Interface. To use the traditional Bicycle example, Bicycle might extend Vehicle, and have traditional vehicular methods. However, maybe we want to check what color the bike is, or paint it. Vehicle may not have methods for this, but IPaintable might. Then, with a BicyclePainter adapter, we can adapter our Bicycle instance to this interface. (For the record, I've never found the Bicycle example very useful, and I'm not sure if I'm doing you any favours by using it here).

Gist: "Adapter Pattern example"

Conclusion

So there's some design patterns, and their usage. It's by no means comprehensive, and I'm sure many of you will have encountered these in other contexts where my definitions aren't entirely accurate. That's just fine, because the value of these is not in having rigid development patterns, it's having useful ways to elevate the discussion of your development. I've almost never encountered clearly recognizable design patterns in code I inherited, because patterns are theory, and when applied, the real world makes things start to get messy.

References

• Wikipedia: I relied on this for design patterns and their basic descriptions.
• "Adapters" chapter from Grok tutorial