PLAT-2060 Use pip-tools to manage requirements files (take 2)

[jmbowman]

This splits requirements into high-level direct dependency files managed by hand, and fully detailed requirements files generated by pip-compile. The detailed files are all recreated by the "make upgrade" target. Points worth noting:

• There are separate configuration and edx-platform PRs for upgrading setuptools and pip and then eliminating the pre.txt requirements file; those should probably be merged before this.
• pip-tools does not yet support non-editable GitHub URL installations, but I worked around this in a post-processing shell script. There are 3 different open pip-tools PRs to fix this, but all of them have known problems preventing them from being merged. We prefer non-editable installations when possible, because pip won't try to re-install them if the correct version is already installed.
• pip-sync won't work with the output files due to the same issue with non-editable GitHub URLs. We can still use pip install with them as normal, we just don't gain the benefit of automatically uninstalling any packages which were removed from the requirements.
• pip-compile writes some of devstack's global pip configuration data to the generated files; I also handled this in the shell script for now.
• pip-compile currently outputs absolute paths for local editable packages; until this is fixed upstream, I worked around it in the post-processing script also.
• Indirect dependencies resulting from packages installed via GitHub don't have the source indicated in the output files. This is annoying, but better than the situation before (where we often weren't even capturing the fact that we depended on these).
• The file reference removed from openedx.yaml is marked as obsolete in the OEP.
• I removed the version pins in the input files for packages which are already current; this should help us keep them that way.
• None of the existing requirements files still referenced by the configuration repo were removed (although some of them are now empty placeholders to be cleaned up later).

Packages which were deliberately upgraded:

• celery 3.1.18 -> 3.1.25 (minimum requirement for edx-celery-utils)
• click 3.3 -> 6.7 (edx-lint and pip-tools both require >=6)
• cryptography 1.9 -> 2.1.4 (minimum requirement for pyOpenSSL)
• execnet 1.4.1 -> 1.5.0 (indirect dependency of tox, removed it from input requirements)
• fs 2.0.17 -> 2.0.18 (minimum requirement of new fs-s3fs version)
• fs-s3fs 0.1.5 -> 0.1.8 (includes a patch I submitted to allow use with the latest version of six, which we were already using)
• oauthlib 1.0.3 -> 2.0.1 (minimum requirement for django-oauth-toolkit)
• py 1.4.34 -> 1.5.3 (indirect dependency of pytest and tox, removed it from input requirements)
• PyJWT 1.4.0 -> 1.5.2 (minimum requirement for djangorestframework-jwt)
• python-dateutil 2.1 -> 2.4.0 (minimum requirement for Faker, via factory-boy)
• pytz 2016.7 -> 2016.10 (minimum requirement for django-ses)
• requests-oauthlib 0.4.1 -> 0.6.1 (minimum requirement for social-auth-core)
• rules 1.1.1 -> rules 1.3 (claims to fix some Django 1.10 and 1.11 issues)

Packages with recent releases which hadn't been pinned:

• backports.functools-lru-cache 1.4 -> 1.5 (via astroid, pylint)
• cffi 1.11.4 -> 1.11.5 (via cryptography)
• Faker 0.8.10 -> 0.8.12 (via factory-boy)
• httplib2 0.10.3 -> 0.11.1 (via oauth2, zendesk)
• parsel 1.3.1 -> 1.4.0 (via scrapy)
• pbr 3.1.1 -> 4.0.1 (via stevedore)
• python-swiftclient 3.4.0 -> 3.5.0 (via ora2)
• queuelib 1.4.2 -> 1.5.0 (via scrapy)
• text-unidecode 1.1 -> 1.2 (via Faker)
• virtualenv 15.1.0 -> 15.2.0 (via tox)
• voluptuous 0.10.5 -> 0.11.1 (via ora2)
• w3lib 1.18.0 -> 1.19.0 (via scrapy)

[UmanShahzad]

@jmbowman We had a little discussion about this at https://github.com/edx/edx-enterprise/pull/303/files#r180804786 -- can you comment on whether we need to add it to edx-enterprise and erase it from here, or...?

cc @douglashall @itsjeyd

[douglashall]

django-fernet-fields is the edX-recommended package to use for storing encrypted values in the database. We added it to edx-platform requirements, so that future use cases wouldn't have to.

[jmbowman]

My thinking was that because django-fernet-fields is only used by edx-enterprise, it should be listed as a dependency in the install_requires field of its setup.py (which is currently derived from the content of requirements/base.txt). With the new pip-tools configuration in edx-platform, this dependency gets auto-detected by pip-compile and stored in the generated requirements files with a comment like "via edx-enterprise".

My concern about not having it listed as a dependency of edx-enterprise is that we're then counting on a line in edx-platform that says "we need this because edx-enterprise uses it"; if edx-enterprise ever changes such that it needs a different version or doesn't need it at all anymore, somebody has to remember to go change the appropriate lines in edx-platform requirements files. If that information is in the edx-enterprise setup.py, it will get automatically updated in edx-platform when make upgrade is run to update the edx-enterprise verion.

Note that this PR was reverted because of issues building sandboxes from scratch (we have some bizarre stuff in our local package installations), but I'm actively fixing that and hope to have it merged again in a day or two.

[edx-status-bot]

Your PR has finished running tests.

[edx-status-bot]

Your PR has finished running tests.

[edx-status-bot]

Your PR has finished running tests.

[edx-status-bot]

Your PR has finished running tests.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production on Monday, April 16, 2018.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the production environment.

[edx-pipeline-bot]

EdX Release Notice: This PR has been rolled back from the production environment.