S3 Interface and Testing

[MSeal]

From #110 (review) I wanted to keep the issue open for discussing improving our s3 code and testing.

Michel:

Ideally, we would want to use a library to mock boto: either moto (https://github.com/spulec/moto) or placebo (https://github.com/garnaat/placebo) is a good choice. There may be others. Idk. Any preferences?
However, you/we may want to ask yourselves/ourselves whether such module (s3.py) should really exist. It probably makes sense to use a library such as s3fs (https://github.com/dask/s3fs). This is what Pandas uses for S3 and given that this project already has a dependency on Pandas it will not add an "exotic" dependency. Not to say that tests should not be written but if we were to spend time on it we may as well refactor the code and use s3fs.

Matt:

I generally agree with that approach. I'ved used https://github.com/jubos/fake-s3 in the past with a before hook launch -- but it adds ruby as a dependency to tests so I wouldn't recommend it here.

I haven't used s3fs before but your argument sounds solid. There may be a case to be made to add a minimal test here without too much refactor and then go a bigger PR with the swap-over. I'd leave that judgement call to you but I can probably help with s3fs changes/testing later on as well.