It has been a while since I have been motivated to write a blog entry, mainly because I have been working 50-100 hour weeks for the past few years in the film industry, working on films like Avatar and Christmas Carol. I have a bit more time on my hands because the company I am currently employed at, Imagemovers Digital, is shutting down this year, so work has ramped down. It is natural to reflect a bit when you cross a fork in the road, and two things have been on my mind.
The first thing I am considering is leaving the film industry to work at a startup company of some kind, if you know of a great opportunity, let me know. I prefer to stay in the San Francisco Bay Area, because it is a great place to be, but I am open to moving anywhere in the world for the right opportunity.
The second thing I have been thinking about is Python and quality. For various reasons, recently, I have doing projects in C++, Objective C, Javascript, Python, SQL, Zsh, and I have been dabbling in a bunch of other languages as well, like Haskell and Scala. In my travels with C languages I have encountered a common meme by developers who effectively say, "Python isn't a real language to code professionally in and it doesn't scale". Having programmed professionally in Python for quite some time, I can say this is, of course, untrue, but it is interesting to step back and ask why this perception exists.
One of the things I learned in writing a book, was that it was good to have negative feedback. Even though negative feedback hurt my feelings sometimes, ultimately, some of the best feedback is negative, because it often has a sliver of real truth shoved inside.
Recently someone mentioned to me that I must have been very confident to have written a book on Python, I corrected them and mentioned that I actually felt the opposite. Writing my first book was one of the most insecure times in my life. I constantly felt worried I would make an idiot out of myself. Fortunately, I was able to contain my own insecurities enough to listen to some very wise constructive criticism from technical reviewers like Titus and Shannon.
As a former somewhat over zealous defender of Python, I do feel a twinge of the same type of insecurity about some of the horrible code that is being written in the language, and how this adds to the perception that "Python doesn't scale". The harsh reality about Python is that because it is so easy to read, and easy to use, in the wrong hands, it can become a weapon of mass destruction. I know collectively many of us have this same uneasy feeling every once in a while about Python.
The answer to eliminating bad code in Python is both obvious and difficult. If you hire experienced software developers who care about writing quality code, then you will get quality code. Unfortunately, this doesn't stop companies, and inexperienced developers, from using Python like a "Fat Pill". "Fat Pills" don't work as a software development metaphor, or in real life. Both losing weight and writing quality code require self discipline and hard work.
Of course, this doesn't change the Python perception problem by C++ programmers that Python is a "joke" language. Or the reality, that inexperience developers will continue to write untested spaghetti messes that cause catastrophic failure (Failure, that they often hire people like me to clean up). Perhaps one answer is to force some level of quality control in Python. Maybe we need a Python compiler that generates unit tests, static code analysis and metrics? This is effectively what I do with either Buildbot or Hudson for the projects I work on, but maybe it needs to be built into the language?
Can Python do something to eliminate both the perception of being "unscalable" and the reality of the proliferation of bad Python code? For the first time in my life, to quote the, especially, timely movie Doubt.....
"
[last lines]
Sister Aloysius: Sister James...
Sister James: What is it, Sister?
Sister Aloysius: [crying] I have doubts. I have such doubts.
"
Subscribe to:
Post Comments (Atom)
Posts
13 comments:
Perceptive. As can clearly be seen from various commercial support forums, increasing use of any platform sees more questions being answered by those who do not really have the knowledge to do so. It's difficult to stomp on such misinformation once it gets out.
This is in fact a measure of Python's increasing popularity. The down side is the greater difficulty of getting reliable information out there. This will ultimately work in your favor, since you are a source of sound knowledge.
Haven't those same C++ programmers seen the horrors that come written in C++ code? Many of those horrors are pretty much impossible to perpetrate in python: Crazy code formatting, horrible file structure, poor memory management, cryptic hard to trace crashes (exceptions much easier then segfaults), and complex build systems.
Python obviously doesn't scale if want to write computationally expensive processing algorithms directly in it, but it does scale to large software systems. Google and NASA use it to meld some of their large software projects together, and as you mentioned Python has tools like EpyDoc, Buildbot, UnitTest, and abstract base classes.
I can agree though, maybe folding some more of those into Python by default along with optional type annotations would help.
Hey Noah! If you're looking for a challenge, I think that one of the biggest things that could happen to Python at this point would be new, more and better tools for programmers to help code analysis in a variety of dimensions (from reliable "find definition" or "find all uses" helpers in IDEs to low-overhead profiling and detectors for a variety of concurrency bugs). Wild and crazy idea: maybe something like PyPy could be reworked to help create such tools?
A good python introspection and refactoring library could find use in a lot a places. I am sure the developer(s) of PyDev and several other pythoh IDE's would have a lot to say about that effort.
@Steve,
You make a really good point, that ties into something I have been thinking for a while. I got into Python because I worked in an Ivory Tower at Caltech, and it wasn't about getting a job. Now Python has reached a level of popularity where it is about a job, which is a good thing.
This does mean there are going to uniformed people giving opinions about what Python can actually do, and it also means there are people that will be writing code because they are "forced" to use Python at school or at work. If you take a look at people that go way, way back in Python they take quality seriously, hopefully this new generation of Python programmers pay attention to the old guys.
@Joseph Lisee
Of course, you are correct, C++ has a whole slew of some of the same kind of horrors. Steve brought up a good point that there are people who, quite literally, are not qualified to answer questions about Python.
Truly being an expert in both C++ and Python is a rare thing. I only know of one guy off the top of my head, Bruce Eckel, and he certainly doesn't think Python is a joke language.
The true part of the criticism, which leads to the false assumption that Python doesn't is that Python is easer to use then C++. As a result, the quality of code from an inexperienced developer can be quite bad. As a community, we should figure out a way to get a handle on this. Believe it or not, there is some really, really bad Python code out in the wild.
@Guido
I agree with you. There have been some really interesting advancements in developer tools for OS X environment and it would be nice to port some of the functionality to Python environments.
I have a "home-brew" toolkit that I use on my projects, and I have our continuous integration system run this on every checkin to mercurial or subversion. The first part of the stack is to use nosetests --with-coverage. I then have a plugin architecture that looks at thresholds for various metric tools like: pylint, pymetrics, and clone digger.
For example, with pylint, you can capture an exit code for the sensitivity you are looking to fail at. This in turn, fails the commit.
I think the PyPy idea doesn't seem that crazy at all. Although I admit to being fairly ignorant of PyPy internals. Most good Python developers go through some form of "compilation" while they are coding. This could be as simple as using nosetests --with-coverage, or as involved as what I mentioned above. It would be vary interesting to have these best practices built right into a dev environment. This could help the Python community get in front of bad software development practices, and also just plain be helpful.
I've been using hudson/nose/zc.buildout/coverage to great effect lately.
Perhaps it's time for something like buildout.org (that was you right Noah?) aimed at bringing information related to python software quality together?
Ben
@boothead
No, I didn't write buildot. This was the work of Jim Fulton, who I am a big fan of. He also wrote Bobo, which is a nice, simple web frameworks.
You bring up an interesting idea though. Using buildout to create a Python compiler. Maybe combining that with Pythoscope?
Although I am most known for my Python work, some of my professional consulting work involves software analysis--something that involves looking at real-world code written by others. In that work, I have analyzed code written in just about everything from hand-written assembler to high-level languages.
I have to say, those C++ programmers are right--in terms of creating scalable spaghetti code, nothing even remotely scales like C++. I mean, why would you write something in a few thousand lines of Python that can be more impressively accomplished by 800000 lines of incomprehensible C++?
@noah, sorry I meant I thought were you behind buildout.org the website :-)
>I then have a plugin architecture that looks at thresholds for various metric tools like: pylint, pymetrics, and clone digger.
This is intensely interesting to me - I did the newbie nugget at Baypiggies recently on using clone digger and pygenie to analyse code for low quality spots.
I think the next step in my evolution as a developer is to automate this since I only reach for these sorts of tools when I'm in trouble or consciously in refactor mode. Hudson seems to be the answer for automatically doing this sort of stuff but it feels like it's going to take a chunk of work to get there. I have some articles bookmarked about setting up Hudson and running tests and metrics automatically but I've never deployed a java app and it looks like the recipes for Python will take a little work.
Maybe the old-time pythonistas need to establish the one-way-to-do-it package for CI with a variety of tools already configured to analyse and test a Python code base that could be a de facto standard for python projects...
Can anyone suggest sites with examples of bad python code or even better, bad python style? I'm helping someone teach a beginning Python course and I've been searching for such examples to motivate good style/code without success.
Robert
Post a Comment