Pip Package¶
Ref : https://packaging.python.org/
There have been historically many ways to package and distribute python code. While many of these are still usable, we are focusing here on the python packaging authority way to package python code.
Package Structure¶
This is a generic example inspired by the pure pip package from Pyros dependencies: pyros-setup
Files hierarchy¶
The source file hierarchy is:
.
├── CHANGELOG.rst # A changelog, usually generated from git commit with gitchangelog
├── doc
│ ├── changelog_link.rst # Documentation link to existing changelog
│ ├── conf.py # Sphinx doc configuration
│ ├── mydocs.rst # Documentation for Sphinx
│ └── readme_link.rst # Documentation link to existing readme
├── requirements.txt
├── MANIFEST.in
├── pyros_setup
│ ├── __init__.py
│ └── module1.py
├── README.rst
├── setup.cfg
├── setup.py
└── tox.ini
Dependencies¶
Dependencies are manage with pip, usually installed in a virtual environment.
There are two kinds of dependencies : - The dependencies needed for the installed pacakge to work (in setuptools.setup install_requires parameter) - The dependencies needed for the developer to work with the package (in requirements.txt)
For more detail about these, you should refer to : https://packaging.python.org/requirements/
Package Development Workflow¶
This is a description of the generic python workflow to develop, build, test and release a python based package. We will use pyros-setup project as an example. TODO : travis check these with doctest + running these in isolation in container
Get the Source ie. clone a Repo:
$ git clone https://github.com/asmodehn/pyros-setup.git Cloning into 'pyros-setup'... remote: Counting objects: 718, done. remote: Compressing objects: 100% (64/64), done. remote: Total 718 (delta 29), reused 0 (delta 0), pack-reused 654 Receiving objects: 100% (718/718), 134.89 KiB | 0 bytes/s, done. Resolving deltas: 100% (433/433), done. Checking connectivity... done.
Create a virtual environment and activate it (here using virtualenvwrapper). We also want a virtualenvironment that can use the ROS & system packages:
$ mkvirtualenv pyros-setup --system-site-packages New python executable in /home/alexv/.virtualenvs/pyros-setup/bin/python Installing setuptools, pip, wheel...done.
Install with pip:
$ pip install . Processing /home/alexv/doctest/pyros-setup Collecting six (from pyros-setup==0.1.2) /home/alexv/.virtualenvs/pyros-setup/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning. SNIMissingWarning /home/alexv/.virtualenvs/pyros-setup/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Using cached six-1.10.0-py2.py3-none-any.whl Collecting pyros_config>=0.1.2 (from pyros-setup==0.1.2) Using cached pyros_config-0.1.3-py2-none-any.whl Collecting pytest>=2.5.1 (from pyros-setup==0.1.2) Using cached pytest-2.9.2-py2.py3-none-any.whl Collecting py>=1.4.29 (from pytest>=2.5.1->pyros-setup==0.1.2) Using cached py-1.4.31-py2.py3-none-any.whl Building wheels for collected packages: pyros-setup Running setup.py bdist_wheel for pyros-setup ... done Stored in directory: /home/alexv/.cache/pip/wheels/39/50/33/ab49df5cef0ef2ce4e23dabd0c9ea5d81f9af131c80d4b2523 Successfully built pyros-setup Installing collected packages: six, py, pytest, pyros-config, pyros-setup Successfully installed py-1.4.31 pyros-config-0.1.3 pyros-setup-0.1.2 pytest-2.9.2 six-1.10.0
Run tests in current environment with nose or pytest. Note that to avoid import conflict, and for tests to pass, you might need to move to a directory where the module is not accessible from current working directory ( https://github.com/asmodehn/pyros-setup/issues/27 )
$ pyros_setup --pytest ============================= test session starts ============================== platform linux2 -- Python 2.7.6, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 rootdir: /home/alexv/doctest/pyros-setup, inifile: collected 3 items .. ... =========================== 3 passed in 0.06 seconds ===========================
Run tests on multiple python environments with tox:
$ tox GLOB sdist-make: /home/alexv/doctest/pyros-setup/setup.py py27 create: /home/alexv/doctest/pyros-setup/.tox/py27 py27 inst: /home/alexv/doctest/pyros-setup/.tox/dist/pyros_setup-0.1.2.zip py27 installed: alembic==0.6.2,amqp==1.3.3,ansible==2.1.0.0,anyjson==0.3.3,apt-xapian-index==0.45,argh==0.26.1,args==0.1.0,autopep8==0.9.1,Babel==1.3,backports.ssl-match-hostname==3.5.0.1,beautifulsoup4==4.2.1,billiard==3.3.0.15,binaryornot==0.2.0,blinker==1.3,bloom==0.5.21,bzr==2.1.4,catkin-pkg==0.2.10,catkin-sphinx==0.2.2,catkin-tools==0.4.2,celery==3.1.6,chardet==2.0.1,Cheetah==2.4.4,cl==0.0.3,clint==0.5.1,cobbler==2.4.1,colorama==0.2.5,command-not-found==0.3,configobj==4.7.2,cookiecutter==0.6.4,coverage==3.7.1,debtagshw==0.1,defer==1.0.6,dirspec==13.10,distro-info==0.12,Django==1.6.1,docker-py==1.8.1,dockerpty==0.3.4,docopt==0.6.2,docutils==0.11,dulwich==0.9.4,empy==3.1,enum34==0.9.23,epydoc==3.0.1,fastimport==0.9.2,fig==1.0.1,Flask==0.10.1,futures==2.1.6,gbp==0.6.9,git-remote-helpers==0.1.0,gitchangelog==2.3.0,gunicorn==17.5,html5lib==0.999,httplib2==0.8,importlib==1.0.3,iotop==0.6,ipaddress==1.0.16,itsdangerous==0.22,Jinja2==2.7.2,jsonpickle==0.9.2,keyring==3.5,kitchen==1.1.1,kombu==3.0.7,launchpadlib==1.10.2,lazr.restfulclient==0.13.3,lazr.uri==1.0.3,libvirt-python==1.2.2,lxml==3.3.3,mailer==0.7,Mako==0.9.1,MarkupSafe==0.18,matplotlib==1.3.1,meld3==0.6.10,mercurial==1.4.2,mock==1.0.1,mod-python==3.3.1,MySQL-python==1.2.3,netaddr==0.7.10,netifaces==0.8,nose==1.3.1,numpy==1.8.2,oauth==1.0.1,oauthlib==0.6.1,oneconf==0.3.7.14.4.1,osrf-pycommon==0.1.2,PAM==0.4.2,paramiko==1.10.1,passlib==1.5.3,pathtools==0.1.2,pep8==1.4.6,pexpect==3.1,Pillow==2.3.0,piston-mini-client==0.7.5,pkginfo==1.3.2,pluggy==0.3.1,progressbar==2.3,protobuf==2.5.0,psutil==1.2.1,py==1.4.31,pyasn1==0.1.7,pycrypto==2.6.1,pycups==1.9.66,pycurl==7.19.3,pydot==1.0.28,pyflakes==0.8.1,Pygments==1.6,pygobject==3.12.0,pygpgme==0.3,pygraphviz==1.2,pyinotify==0.9.4,pymongo==2.6.3,PyOpenGL==3.0.2,pyOpenSSL==0.13,pyparsing==2.0.1,pyros-config==0.1.3,pyros-setup==0.1.2,pyserial==2.6,pysmbc==1.0.14.1,pytest==2.9.2,python-apt===0.9.3.5ubuntu2,python-dateutil==1.5,python-debian===0.1.21-nmu2ubuntu2,python-memcached==1.53,pytz==2012rc0,pyxdg==0.25,PyYAML==3.10,pyzmq==14.0.1,redis==2.7.2,reportlab==3.0,requests==2.10.0,requests-toolbelt==0.7.0,roman==2.0.0,ros-buildfarm==1.1.0,rosdep==0.11.5,rosdistro==0.4.7,rospkg==1.0.39,SecretStorage==2.0.0,sessioninstaller==0.0.0,simplejson==3.3.1,six==1.10.0,software-center-aptd-plugins==0.0.0,Sphinx==1.2.2,sphinx-rtd-theme==0.1.7,sphinxcontrib-plantuml==0.6,SQLAlchemy==0.8.4,ssh-import-id==3.21,stevedore==0.14.1,supervisor==3.0b2,termcolor==1.1.0,testfixtures==4.7.0,texttable==0.8.3,tornado==3.1.1,tox==2.3.1,transitions==0.2.7,trollius==2.1,twine==1.8.1,Twisted==13.2.0,Twisted-Conch==13.2.0,Twisted-Core==13.2.0,Twisted-Lore==13.2.0,Twisted-Mail==13.2.0,Twisted-Names==13.2.0,Twisted-News==13.2.0,Twisted-Runner==13.2.0,Twisted-Web==13.2.0,Twisted-Words==13.2.0,urlgrabber==3.9.1,urllib3==1.7.1,vboxapi==1.0,vcstools==0.1.38,virtinst==0.600.4,virtualenv==15.0.2,virtualenv-clone==0.2.4,virtualenvwrapper==4.1.1,VTK==5.8.0,wadllib==1.3.2,watchdog==0.8.3,websocket-client==0.37.0,Werkzeug==0.9.4,wstool==0.1.13,wstools==0.4.3,WTForms==1.0.1,wxPython==2.8.12.1,wxPython-common==2.8.12.1,xdot==0.5,yujin-tools==0.4.24,zenmap==6.40,zope.interface==4.0.5 py27 runtests: PYTHONHASHSEED='473323988' py27 runtests: commands[0] | py.test --pyargs pyros_setup WARNING:test command found but not installed in testenv cmd: /home/alexv/.virtualenvs/pyros-setup/bin/py.test env: /home/alexv/doctest/pyros-setup/.tox/py27 Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting. ================================================================ test session starts ================================================================ platform linux2 -- Python 2.7.6, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 rootdir: /home/alexv/doctest/pyros-setup, inifile: collected 3 items pyros_setup/tests/test_setup.py ... ============================================================= 3 passed in 0.06 seconds ============================================================== ___________________________________ summary ____________________________________ py27: commands succeeded congratulations :)
Build a distribution
Release on Pypi with twine (you can also code a specific detailed workflow in your setup.py)
Continuous Testing Workflow¶
Because no software works until it has been tested, you should configure travis on your repository to run test with each commit and pull request.
Catkin testing can be done with a simple .travis.yml file and a small shell script.
A matrix build can be setup to test behavior in multiple python virtualenvs. Using tox for this is generally a good idea, an example is there <TODO>.