It's yer data! - how Google secured its future, and everyone else's

Dear Google,

This is a love letter and a call to action.

I believe we stand at a place where there is a unique opportunity in managing personal data.

There is a limited range of data types in the universe, and practically speaking, the vast majority of software works with a particularly tiny fraction of them.

People, for example. We know things about them.

Names, pictures of, people known, statements made, etc.

Tons of web applications conceive of these objects. Maybe not all, but probably most have some crossover. For many of the most trafficked apps, this personal data represents a very central currency. But unfortunately, up until now we've more or less been content with each app having it's own currency, that is not recognized elsewhere.

You can change that. You can establish a central, independent bank of data, owned by users and lent to applications in exchange for functionality. The format of the data itself will be defined and evolved by an independent agency of some sort.

There are two core things this will accomplish.

1) It will open up a whole new world of application development free from ties to you, Facebook, Twitter, etc.

2) It will give people back ownership of their data. They will be able to establish and evolve an online identity that carries forward as they change what applications they use.

Both of these have a dramatic impact on Google, as they allow you to do what you do best, building applications that work with large datasets, while at the same time freeing from you concerns that you are monopolizing people's data.

A new application world

When developing a new application, you start with an idea, and then you spend a lot of time defining a data model and the logic required to implement that idea on that data model. If you have any success with your application, you will need to invest further in your data model, fleshing it out, and implementing search, caching, and other optimizations.

In this new world, all you would do is include a library and point it at an existing data model. For the small fraction of data that was unique to your application, you could extend the existing model. For example:

from new_world import Model, Field

BaseUser = Model("https://new_world.org/users/1.0")

class OurUser(BaseUser):
    our_field = Field("our_field", type=String)

That's it. No persistence (though you could set args somewhere to define how to synchronize), no search, no caching. Now you can get to actually building what makes your application great.

Conceivably, you can do it all in Javascript, other than identifying the application uniquely to the data store.

And you can be guaranteed data interoperability with Facebook, Google, etc. So if you make a photo editing app, you can edit photos uploaded with any of those, and they can display the photos that are edited.

Securing our future

People have good reason to be suspicious of Google, Facebook, or any other organization that is able to derive value through the "ownership" of their data. Regardless of the intent of the organization today, history has shown that profit is a very powerful motivator for bad behaviour, and these caches of personal data represent a store of potential profit that we all expect will at some point prove too tempting to avoid abusing.

Providing explicit ownership and license of said data via a third-party won't take away the temptation to abuse the data, but will make it more difficult in a number of ways:

• Clear ownership will make unfair use claims much more cut-and-dried
• A common data format will make it much easier to abandon rogue applications
• Reduced application development overhead will increase the competitive pressure, lowering the chance of a single application monopolizing a market and needing to grow through exploitation of its users data

A gooder, more-productive, Google

By putting people's data back in their hands, and merely borrowing it from them for specific applications, the opportunities for evil are dramatically reduced.

But what I think is even more compelling for Google here is that it will make you more productive. Internally, I believe you already operate similar to how I've described here, but you constantly bump up against limitations imposed by trying not to be evil. Without having to worry about the perceptions of how you are using people's data, what could you accomplish?

Conclusion

Google wants to do no evil. Facebook is perhaps less explicit, but from what I know of its culture, I believe it aspires to be competent enough that there's no need to exploit users data. The future will bring new leadership and changes in culture to both companies, but if they act soon, they can secure their moral aspirations and provide a great gift to the world.

(Interesting aside, Amazon's recently announced Cognito appears to be in some ways a relative of this idea, at least as a developer looking to build things. Check it out.)

Python Web Client 0.2 Release

I'm pleased to announce the 0.2 release of the Python Web Client. It now meets a bare minimum of functionality to be useful. Specifically, it provides a web UI to allow you to put together and submit HTTP requests, specifying request type, URL, parameters and headers.

You can view a demo here.

Oh, and it's hosted on Github here.

What's new in 0.2?

- Added display of response body
- Added option to select request method
- Added field to add parameters
- Added field to add headers

What does the future hold?

Well, there are a lot of potential ideas for where this could go, both in the original SoCal piggies proposal, and on the Rally site where I've been tracking this (log in with alecmunro+public@gmail.com:Experiments). However, I expect to resume my place among the employed soon, and thus far I haven't had any feedback on this. So barring extreme boredom on my part, this is probably the end of the line until someone forks it.

Python Web Client 0.1

The release
I've polished a few rough edges, and I'm now (somewhat) proud to release Python Web Client 0.1. It's still lacking in features, but it's got docs and an easy way to start it, and what's more, its pieces work on non-Windows platforms. I've also handicapped it a bit to allow me to put up a hosted version for people to try out, here.

What's next
Well, with this "release", the foundation of the project should be more or less complete (though I don't have it in continuous integration yet). That should make it pretty safe to add features. I'll see what I can put into it over the next few days and get a 0.2 release out, that will hopefully be more useful. If I people are interested, we might make it to 0.3, at which point I'll try to get it on PyPi (it would be nice to have a project up there).

Sidenote
I'm not sure there's a good solution, but I find all the context-switching that goes along with setting up a new project to be a real drain on productivity.

Python Web Client 0.1.dev1

tl;dr
I built a preliminary version of the Socal Piggies Python Web Client. Take a look at the code and methodology, suggest features/fixes, and otherwise enjoy! There's no live demo because of a gaping security hole.

Intro
First off, I would just like to express my sincere appreciation for everyone who responded to my request for ideas. There were a number of interesting options, but for now, I've decided to build an implementation of the Python Web Client described on the Socal Piggies site. I made this choice because it's a comfortable area for me to work in, it's a tool I can see using, and probably most important, I think I can build a simple version fairly quickly. :)

Design
The first thing I usually do with a new project is retreat to a quiet corner with a notebook and a writing implement. It would be nice to find an electronic way to do this, but so far nothing has come close to what I need in terms of offering a combination or structured and free-form input, along with instant availability.

In this case, I was trying to strip the concept down to the bare essentials. In this case that means:

• One page, with two widgets:
• Create request (Enter a URL)
• Display response (Status Code + Headers)
• Startup script that launches a browser to the service

That last one may seem non-essential, but speaks to my philosophy that "delivery is as important as development". In practice, that means that how a client is introduced to functionality is just as important as how well that functionality works. Make it really easy to start using.

Because this is somewhat of a showcase project, there's a couple of other things I pinned on my design list:

• Testing (unit, functional, system, jsunit)
• Docs (UI + API, published in Sphinx)

While I was putting this together, I also wrote down a whole lot of nice-to-have features for later. You can check them out on the Rally site I am (kind of) using to manage this project. You will need to log in:
alecmunro+public@gmail.com:Experiments

Testing
Because I'm a TDD advocate, let's do that. So first I need to decide what tests I want to have. Since I don't know my code structure at all yet, I'm going to start with system level tests, which I usually define as something that tests the system at the UI level, running very close to how it will run in production:

• Visit site, enter URL, press submit, verify results.

Gist-It for test_ui.py

That pretty much does it, and can be done with something like Selenium, unless I also wanted to test the startup script. Doing so would involve using something like Sikuli(which I do love), to observe the state of the desktop, but that might balloon the scope of this project a bit too much. So we are on to functional testing, in this case defined as testing the API of the web service, in as isolated an environment as we can create. So what are we looking at there?

• Submit URL, verify response
• Probably some variants of this, to test error handling or redirect responses (we are handling those, right?).
• What happens if the URL to retrieve is the URL of the webservice itself? Could we experience some nastiness there?

So both of the test types we have addressed so far require an actual connection to another server. We could use something that's always going to be available, like www.google.com, but we aren't really guaranteed a network connection. So for this, I'll write a small web server that can be set to return whatever you want it to.
Gist-It for echo_server.py

This was actually a bit trickier than I anticipated, due to the need to run the server in a separate thread/process, and this bug. Anyway, here's the API tests:

Gist-It for test_apis.py


Ok, so unit tests now. From our earlier tests, it's become pretty clear that the API will have one view, which accepts the details to construct a request (just a URL to start), submits that request, and returns the status code and headers from the response.

Well, that was all really boring. Maybe the jsunit tests will be more interesting? In practice, I probably won't write these before the code, because it still takes me a while to get into the rhythm of writing tests for javascript. I need a bit of trial-and-error.

• Create Request Widget:
• Enter text and press submit. A call should be made to create the request, and the deferred for that call should be passed to the page.

Gist-It for test_create_request.js

• Display Response Widget:
• Supply it with various responses, and confirm that they display properly. Probably the most interesting bit of testing of the whole lot.

Gist-It for test_display_response.js

Implementation

Ok, so we have our tests, perhaps. Now, on to the implementation. This part is actually really simple.

There's the Python view:
Gist-It for views.py

and the two Javascript widgets:
Gist-It for create_request.js

Gist-It for display_response.js


Feel free to take a look at the GitHub repository for more details (or check it out to run it). 

Documentation
I've left off the documentation for now, both because I wanted to get this up soon, and it's been a while since I started anything with Sphinx, and also because it really doesn't do much yet. I'm also still missing the launch script.

Conclusion
There's lots more work to be done here to make something useful, so I'm taking suggestions. But hopefully this gives you an idea of how you can build a simple and somewhat tested web-app using Pyramid and JQuery. You will notice that it is very testing heavy, probably significantly more than real-world deadlines would allow for. But once you get these tests in place (and a system to run them), they are fairly easy to build on, and can provide a safe container in which to experiment.
For the moment, I'm planning two distinct iterations:

• Add docs and launch script, as well as displaying the response body. That will be 0.1
• Flesh out the request creation ability, to allow settings headers and parameters. Along with hopefully some fixes/refinements, that will be 0.2

Beyond that, development will depend on whether this is interesting to anyone, so let me know.

Let me build you a web app/service!

So, according to my introduction, one of my goals for this project is to build a web app or service to demonstrate my skillset in that area. Unfortunately, it's turns out my idea for a web app has already been done, and is quite well-known (on the plus side, I now understand what 4Chan is). At the moment, I'm drawing blanks about what I should build, so I thought perhaps I could crowdsource that job.

So if you have an idea for a web app or service, but not the time or skills to get it started, or if you would simply like to collaborate with someone on it, post a comment describing your idea.

In general, I'm game for pretty much anything, as I'm confident I can learn whatever I need to. Some general parameters though:

• It's going to be in Python, built on Pyramid, using SQLAlchemy for models
• I will try to strip the idea to it's core, and leave others to make it full-featured
• I will try to make an appealing and usable UI, but again, it will be pretty barebones
• I'm going to blog about it, host it in github and make the code viewable by everyone
• If you want to collaborate:
• I'm open to hosting it elsewhere, as long as it's publicly accessible
• I will be a stickler in design discussion and code reviews

Finally, I will happily put it under any license you want and grant you and copyright or whatnot, as long as it doesn't prevent me from achieving my objectives vis-a-vis skill demonstration.

Introductions

Hi there, I'm Alec Munro. For various reasons, I've been unable to engage online for the last several years. That is no longer true, and I find myself with time, so here I am, engaging (well, I hope so).

So what am I trying to engage with? Well, I'm a software developer, interested in refining the process of software development to produce higher quality software. In practice, my skills and experience tend towards Python and Web Development, with the last 5 years being spent in the realm of test automation. Let's break that down...

When I say I want to produce higher quality software, I'm defining quality as "that which meets or exceeds my customer's expectations". This is a definition I'm stealing from someone, but which feels correct to me. There are two implicit conditions in that definition that go beyond simply writing good code: that you know who your customers are, and that you understand their expectations. So for me, developing quality software is a very holistic process, that encompasses the entire lifecycle of a desired piece of functionality.

I've been developing web sites for 14 years, starting with static HTML and JavaScript, moving through Flash and PHP, and finally "settling" on Python about 8 years ago. Since then, web development has almost never been my primary occupation, but it has almost always been a tool I could use to improve the projects I was working on. In the Python world, I started with Zope 2, moved to Zope 3, then Grok, and most recently Pylons and Pyramid. I'm biased towards building everything as a REST-ish web service, with a UI built in JQuery.

I came to test automation because I saw a need. I saw (seemingly) automatable tasks being performed by manual testers, and I took action to address this. I learned, with great difficulty, that many things I had assumed could be easily automated were actually terrible candidates for automation, often due to the development process that produced them. I learned about "testability" (thanks Misko!), and the role that test automation must play in driving development process improvements. I also gained an appreciation for what I call "proactive transparency", which is the philosophy that much of the information people need is too difficult to find, because those producing it don't know how to publicize it (or don't appreciate the need to do so). In test automation, this applies to results of testing, but I've found it applicable to many other areas.

So that's a rough overview of who I am, and what I can do. But where's the engagement? Well, I find myself without a job, but with a bit of a margin before I need a job. So I thought I would take this time to write on subjects I know about, as well as to study and report on those that I don't. This will hopefully give me the opportunity to broaden my skills (specifically for interviewing purposes, I will admit), while also re-engaging with the development community that I have been absent from for several years. For you, the reader, hopefully some of these topics will interest you, and you will participate in my discussions, to correct me where I err, or weasel further answers out of me when I am unclear.

What kind of topics? Well, I have no formal computer science training, so there's a couple of areas that tend to come up in interviews that I don't think I give satisfactory answers on. These include:

• Design patterns. I know many of their names, and I know I use them often in my work, albeit uncredited. I will research the most common ones, and invent problems where they are applicable, and explain why one over the other.
• Sorting. Quick sort, merge sort, etc. Using Python for web development, I've never been held back by the performance of a sorting operation, but I understand that these algorithms are considered quite foundational, and I know I've suffered in interviews due to the lack of them.

I will also go into some topics that are interesting to me, and play to the strengths I have developed during my career.

• Mock library comparison. I consider the use of mock objects to be an essential (and often neglected) part of almost any software development project. I also think there are too many Python mock object libraries. :) I will discuss this in more detail, and show examples of the use of each library, and try to illustrate the strengths and weaknesses of each.
• Web application development. Because of my recent work history, I have virtually no public code to share with potential interviewers. So, purely as a vanity project, I will create what I consider to be a properly tested and documented web application, and write about the process. I'll try to make a useful application, but I can't promise that.
• RabbitMQ, ZeroMQ, Gevent. I keep running across these things, but I've never had an occasion to do much beyond skim their docs. They seem interesting and well regarded by those who are well regarded by me, so I'll look into them.
• Go (the language, I'll leave the game to my father, at least for now). I've heard being laid off is a great time to learn a new language, and two of the jobs I've seen that would be most interesting mention Go, so I think that will probably be it for me.

To further my ideal of engagement, I will post any code I write for any of these in github (also new to me), and encourage contributions from the community. Hopefully, the community will find enough useful in what I create to do so.