Typical Python Development Environment

Disclaimer

There are many other choices in every aspects of Python software development environment. New choices keep emerging in a speed that will always surprise us. It's even hard to define what are the 'typical' choices. Here we just list some examples and we'll try our best to include the 'best' ones that we know.

We respect all the choices.

And this doc will be updated frequently.

Overview

Here you can find some examples of a 'typical' Python development environment. Including:

  • Versions Of Python Language
  • Installing Python Packages
  • Virtual Environment
  • IDE
  • Convention and Code Static Analysis
  • Test Frameworks
  • Automation and Continuous Integration System

Versions of Python Language

You are recommended to read The History of Python.

At this moment Python 2.x and 3.x coexist as popular programming languages.

  • The most widely used Python version is still Python 2.x (x > 5).
    • Google App Engine now uses Python 2.7
  • Python 3.x (released since December 3, 2008) is quite stable now. It broke backward compatibility. That means Python 2.x code would not run with Python 3.x without modification.
    • The difference can be subtle. Same code that is both valid in Python 2.7 and Python 3.x could behave quite differently. E.g. the built-in map function.

Depends on the target user, A good choice now quite often is to write code both work for Python 2.7 and 3.3/4. That usually means you need to configure your Continuous Integration System to use both interpreter.

Alternative Python Implementation

There are several alternative Python implementations of the language interpreter.

  • Python == The main Python implementation == CPython, is written in C. Can be extended with C.
  • Jython compiles into Java byte code, which can then be executed by every Java Virtual Machine implementation.
  • IronPython is the .NET implementation.
  • PyPy is a fast, compliant interpreter of Python 2.7. Its just-in-time compiler brings a significant speed improvement over CPython.
  • Cython != CPython, compiles to C.

Alternative Python Interactive Interpreter

Just type python to enter into standard Python interactive mode.

In []: 
!python

Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

An alternative tool is IPython. It has more advanced features, like syntax highlight, code completion and integration with the OS (shell).

These slides are actually created using IPython notebook.

In the next section, we'll use IPython to show how to install additional Python packages.

Installing Python Packages

The "standard" way of installing Python packages, is using the pip command to install them from the Python Package Index(PyPI). PyPI is a repository of software for the Python programming language.

If you don't have pip already, this is a good guide.

For example, to install IPython in your system globally:

In [4]: 
# Don't do this, it's just a demo.
# You don't always want to install packages globally.
# Read on to know why.
!pip install ipython

Requirement already satisfied (use --upgrade to upgrade): ipython in /Users/terry/.virtualenvs/t1/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): gnureadline in /Users/terry/.virtualenvs/t1/lib/python2.7/site-packages (from ipython)
Cleaning up...

OK, turned out I have ipython on my computer already.

You need to add sudo in front of pip on some operating system (when installing globally).

Virtual Environment

Adding new packages globally to your system will sooner or later become a problem. Packages and their versions might conflict with each other, and the Python version itself might also be a problem. This is especially a problematic when you are working on multiple projects at the same time. The required dependencies are difficult to satisfy if everything is installed globally.

That's why a virtual environment is almost mandatory if you are developing using Python. The most popular one so far is virtualenv. There is also a virtualenvwrapper, which makes working with virtual environments even easier. The link has a comprehensive guide on how to install it.

After installing it, you should be able to simply do:

mkvirtualenv ProjectA-env

to create a new virtual environment, and:

workon ProjectB-env

to switch to another virtual environment.

When working with virtual environments, every pip install you execute will install packages inside the active virtual environment instead of the global location.

IDE

As a dynamic programming language, the IDE (Integrated Development Environment) or text editor for Python cannot do as much as they can for other programming languages like Java. So usually you can just use any text editor. Basically you are looking for three things:

  • syntax highlight
  • Auto indentation
  • (a certain level of) code completion

Feature beyond that are often just nice to have.

Here we use (other than Vim) Eclipse with PyDev plugin (http://pydev.org/).

Installation of Eclipse with PyDev

Install the latest version of Eclipse. From the Help menu select Install New Software:

Add "http://pydev.org/updates" in the "Work with" text box and select PyDev. Then follow the dialog to complete the installation.

As I'm creating these slides, PyDev 3.4 doesn't seem to work with Eclipse Kepler. The solution I found is to uncheck the checkbox "Show only the latest versions of available software." in the above dialog and install PyDev 2.8.2. I hope this bug will be fixed soon.

After the installation you will need to choose the Python Interpreter. If you are using global python environment, just click Auto Config here. If you are using a virtual environment, add New... and find the actual Python executable (for example at ~/.virtualenvs/virtualEnvName/bin/python, but depends on where you created your virtual environment).

Now you can create a new PyDev project.

Think about it

Why does Python get less support than Java from the IDE (or other editor)?

Should Python code completion feature be text-based or context-based?

Convention And Static Analysis

Coding Convention

Python's coding conventions are documented in pep-8.

The easy way to make sure you are following them is to install the pep8 package by:

pip install pep8

Python Enhancement Proposal (PEP) process is the primary mechanism for proposing major new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. Outstanding PEPs are reviewed and commented upon by the Python community and by Van Rossum, the Python project's BDFL._ (from Wikipedia)

Then for code:

In [7]: 
%%file example.py
def long_function_name(
    var_one, var_two, var_three,
    var_four):
    print(var_one)

Overwriting example.py

And then we can run:

In [9]: 
!pep8 example.py

example.py:1:1: E302 expected 2 blank lines, found 0
example.py:3:5: E125 continuation line with same indent as next logical line

Static Analysis

The most commonly used tool for Python code static analysis is pylint (pip install pylint to install it).

In [11]: 
!pylint example.py

************* Module slides.example
C:  4, 0: Final newline missing (missing-final-newline)
C:  1, 0: Missing module docstring (missing-docstring)
W:  2,13: Unused argument 'var_two' (unused-argument)
W:  3, 4: Unused argument 'var_four' (unused-argument)
W:  2,22: Unused argument 'var_three' (unused-argument)

Other static analysis tools

  • Code duplication CloneDigger
  • Software Complexity lizard (same author as this material)

Testing Frameworks

There are many Python test frameworks. Here I just list what I am using:

  • Unit Test the unittest module that comes with Python 2.7 and 3.x.
  • Mocking the mock module that can be installed by pip install mock, it's a standard module since Python 3.3.
  • Test Case Discovery nose, which can be installed by pip install nose.
  • Functional/Acceptance Test RobotFramework. It's implemented in Python and extensible using Python.

Automation and Continuous Integration System

Automation

Unfortunately (or luckily?), there is no Python's equivalent of Java's maven, or Ruby's rake. This is largely because Python modules themselves do not need any external intervention.

It's quite common for Python projects to use Makefile to automate tasks such as the static checkers, unit tests and functional tests, and perhaps deployment.

Some new continuous integration system can do the same job as well.

Continuous Integration System

Jenkins

We are still using Jenkins (yes, we do) as our CI server. And we use ShiningPanda Plugin so that it can use the virtual environment without polluting the whole CI server environment.

Travis CI

We also use a lot of travis-ci in our open source projects. It can be much more easier to separate the dependencies on the environment and it can easily be configured to test against multiple Python versions (including PyPy).

Deployment

What? You have just made an application in Python? Then deployment should not be a problem for you :-). But before you write your own deployments scripts, check out:

Miscellaneous

If you are interested in contributing:Open Sourcing a Python Project the Right Way

If you care about performance, everything is already included in the standard library: Python profiler