Software Gardening in practice

Software gardening, as expressed by Hunt/Thomas, Atwood, etc uses the metaphor to deal with how a software project is alive, and is constantly being changed by factors outside of the control of the developers. You can plant a seed, and water it, but you can’t control the sunlight, the bugs, bees, etc. As it grows, you can weed the garden, removing features that are not used but taking up resources, add fertilizer, change the plants to better reflect your needs (why didn’t someone tell me Joe was allergic to cucumbers!?).

I want to expand on this idea, as I believe there are several valuable insights that can be derived from adopting this view. In this post I will explore this concept from three different perspectives: the project itself, a software developer, and a client that wants to get some software developed.

The Unseen Environment

As a particular software environment is like a garden, there are many factors that the garden grows out of. There are obvious factors, like the composition of the soil, and the likely availability of water, and sunlight. But there are also more subtle, “living” factors, that are going to be part of the environment of your garden, yet generally operate independently.

One of the most fascinating things I’ve learned recently is the concept of mycorrhizal networks. These are fungal layers that exist within the soil that serve to break down waste matter and distribute molecules that plants depend on for nutrition. It is generally unseen, but is a huge factor in the ability of plant ecosystems to thrive. Generally much better known, but still somewhat unseen, are the pollinators. Birds, bees, and butterflies allow plants to reproduce, and thus stimulate genetic diversity.

So, above and below, there is an environment that allows plant life to thrive. A similar role is played in software development by the ecosystem that exists around both the software, and the developers.
All developers and software libraries come to a project with a history that is shaped through the absorption of “byproducts” from previous projects. These can take the form of learned code snippets, software structure conventions, and many other things. I would liken this to the way the mycorrhizal network digests the output from various past biological processes and produces raw nutrients to begin new ones.

In a similar way, open/shared source cultures could be compared to pollinators. Moving from project to project with very few barriers, a developer can distribute efficient techniques rapidly, effecting the projects and other developers. This has resulted in the establishment of numerous best practices, to the point where high-profile open source projects are the gold standards for software quality. Another interesting way in which open source diversity is like genetic diversity is in the trend of multiple competing software projects springing up with different attributes tailored to their environments. These can continue for very long periods, sharing “genes” with each other but still continuing as distinct “species”.

In some large organizations, the software ecosystem is distinct enough to effect the software that is produced there, but at this point, almost everyone is dependent on Open Source. It produces specific codebases that underly most every modern project, such as Linux, Apache, OpenSSL, GCC, Boost, Python, etc. But it also establishes the best practices and standards for software development that are used by individual developers, whatever they are working on. Conversely, bug reports come from commercial projects that make Open Source stronger, and developer experiences there continue to inform and refine the state of the art. There are also cases where it is beneficial for commercial development to open source certain pieces of their software, to expose them to that wider diversity, and encourage contributions that will help meet business goals (though like natural evolution, predicting the future of an open source project is far from a science).

Tying that back together, that means that for a developer, every project you work on contributes to this larger community, even when the code itself is closed. Because it changes you, and it changes the conversations you have with others, and how you work to solve problems in the future. As a former user of the Zope Web Application Server, I have watched other users and contributors define the cutting edge of web software (Tarek Ziade, Chris McDonough, Alex Gaynor, and Martijn Faasen, among others).

Garden Development as a Service

As someone who works on software, your value is not the creation and delivery of a product, but rather the cultivation of an environment. Even when you are employed delivering products, there’s no such thing as a final release, just something that is able to meet the needs of the moment. Needs will change, the ecosystem will change, you will change, and new opportunities will arise. How might this change how you approach a single project, and your career?

In terms of approaching a particular project, it means you need to understand the landscape in order to effectively execute on the vision. You need to know what kind of software is already there, and what the state of third party libraries are in that particular domain. You need to plan for the inevitable bugs and maintenance. The actual implementation flows out of these concerns, not the other way around.

From a career standpoint, I should first confess that, well, I’m not a gardener. However the image I have of them is that they don’t get annoyed that all the cool gardens are already cultivated. As long as you have a willingness to work, gardens are going to need cultivation. Likewise, as long as you have a willingness to develop it, there will always be available work developing software. Under this philosophy, nothing ever reaches the point that it cannot be improved or does not require maintenance.

Gardening Agency

When we have a need for some software, and we attempt to consider the effort that will be required, we usually think about the features that we want the software to have. The gardening metaphor suggests this is a very limited way to view this. If we think of the features of a particular garden as “has Roses and Violets, and produces Tomatoes”, we can quickly understand that a good deal more information is required to get a reasonable estimate of the effort that will be necessary.

Is the garden in Hawaii, Alaska, or the Central Park Zoo? What if there’s already a garden there? What's the condition of the soil, or local animal life?

Translating these questions over to software, we can see that the environment a piece of software needs to "live" in can impact the effort required to develop it as much or more than the features it needs to have. For instance, if you are looking to add accounting features for exporting financial data to your partners, the effort could vary tremendously depending on the existence and maturity of the open source software that exists to work with data in the formats they use.

Just as working with a professional landscaper will produce dramatically different results than simply purchasing plants, establishing a relationship with someone who can look at your situation holistically is key to a successful software project. They know the questions that need to be asked, the process for finding the answers, and they know how to build out a development team that will take that reality into account.

Introductions

Hi there, I'm Alec Munro. For various reasons, I've been unable to engage online for the last several years. That is no longer true, and I find myself with time, so here I am, engaging (well, I hope so).

So what am I trying to engage with? Well, I'm a software developer, interested in refining the process of software development to produce higher quality software. In practice, my skills and experience tend towards Python and Web Development, with the last 5 years being spent in the realm of test automation. Let's break that down...

When I say I want to produce higher quality software, I'm defining quality as "that which meets or exceeds my customer's expectations". This is a definition I'm stealing from someone, but which feels correct to me. There are two implicit conditions in that definition that go beyond simply writing good code: that you know who your customers are, and that you understand their expectations. So for me, developing quality software is a very holistic process, that encompasses the entire lifecycle of a desired piece of functionality.

I've been developing web sites for 14 years, starting with static HTML and JavaScript, moving through Flash and PHP, and finally "settling" on Python about 8 years ago. Since then, web development has almost never been my primary occupation, but it has almost always been a tool I could use to improve the projects I was working on. In the Python world, I started with Zope 2, moved to Zope 3, then Grok, and most recently Pylons and Pyramid. I'm biased towards building everything as a REST-ish web service, with a UI built in JQuery.

I came to test automation because I saw a need. I saw (seemingly) automatable tasks being performed by manual testers, and I took action to address this. I learned, with great difficulty, that many things I had assumed could be easily automated were actually terrible candidates for automation, often due to the development process that produced them. I learned about "testability" (thanks Misko!), and the role that test automation must play in driving development process improvements. I also gained an appreciation for what I call "proactive transparency", which is the philosophy that much of the information people need is too difficult to find, because those producing it don't know how to publicize it (or don't appreciate the need to do so). In test automation, this applies to results of testing, but I've found it applicable to many other areas.

So that's a rough overview of who I am, and what I can do. But where's the engagement? Well, I find myself without a job, but with a bit of a margin before I need a job. So I thought I would take this time to write on subjects I know about, as well as to study and report on those that I don't. This will hopefully give me the opportunity to broaden my skills (specifically for interviewing purposes, I will admit), while also re-engaging with the development community that I have been absent from for several years. For you, the reader, hopefully some of these topics will interest you, and you will participate in my discussions, to correct me where I err, or weasel further answers out of me when I am unclear.

What kind of topics? Well, I have no formal computer science training, so there's a couple of areas that tend to come up in interviews that I don't think I give satisfactory answers on. These include:

• Design patterns. I know many of their names, and I know I use them often in my work, albeit uncredited. I will research the most common ones, and invent problems where they are applicable, and explain why one over the other.
• Sorting. Quick sort, merge sort, etc. Using Python for web development, I've never been held back by the performance of a sorting operation, but I understand that these algorithms are considered quite foundational, and I know I've suffered in interviews due to the lack of them.

I will also go into some topics that are interesting to me, and play to the strengths I have developed during my career.

• Mock library comparison. I consider the use of mock objects to be an essential (and often neglected) part of almost any software development project. I also think there are too many Python mock object libraries. :) I will discuss this in more detail, and show examples of the use of each library, and try to illustrate the strengths and weaknesses of each.
• Web application development. Because of my recent work history, I have virtually no public code to share with potential interviewers. So, purely as a vanity project, I will create what I consider to be a properly tested and documented web application, and write about the process. I'll try to make a useful application, but I can't promise that.
• RabbitMQ, ZeroMQ, Gevent. I keep running across these things, but I've never had an occasion to do much beyond skim their docs. They seem interesting and well regarded by those who are well regarded by me, so I'll look into them.
• Go (the language, I'll leave the game to my father, at least for now). I've heard being laid off is a great time to learn a new language, and two of the jobs I've seen that would be most interesting mention Go, so I think that will probably be it for me.

To further my ideal of engagement, I will post any code I write for any of these in github (also new to me), and encourage contributions from the community. Hopefully, the community will find enough useful in what I create to do so.