Updating a pypi package is not as complicated as it seems, but it was a struggle my first time around.  For my hyperparameter tuning research project, wwu-tinker, we maintain our python API for end users in a  pypi package, so our users can pip install directly instead of cloning our github repository.  The problem is that we have only updated the package once, back in November, and by “we” I mean our team member Max who graduated a few weeks later.

Well, a few months ago, we received a big feature request from Pacific Northwest National Lab asking for the ability to add  hyperparameter configurations and their respective training results to an experiment. The intended effect was to “prime” an optimizer like Bayesian Optimization Algorithm with previous data to attempt to converge on better regions faster.  I won’t go into more detail here though…

Alas, this feature was large requiring updates to both the user facing API and the backend.  The update was so significant, that the existing API barely worked, and I was getting tired of telling people to pip install from wwu-tinker directly on github:

pip install git+https://github.com/hutchresearch/wwu_tinker.git

The first go at trying to update the package was an absolute nightmare, from getting access to our teams pypi project, to trying to find the right documentation, and too many attempts at uploading the project.  I spent hours reading StackOverflow posts, trying different commands, downloading and editing the project over and over… I gave up on all efforts for weeks after that!

Fast forward to a few nights ago, and magically, I found the perfect documentation (THE SOURCE DOCUMENTATION: PACKAGING PYTHON PROJECTS), and got it taken care of in under 20 minutes.  The only thing I changed was spinning up an Ubuntu virtual machine and not being a tired mess.

What did I learn?  Well, for one, just stop after an hour, especially if I’m already tired… and, I must have been that day, because I felt like a complete idiot when I saw how easy it was to do everything on that day.  Second, read the docs all the way through. I looked at the source docs, and I must have not read a single thing except for the commands. The real kicker was simply this command.

twine upload dist/*

This was after I tried to make many of my own commands after following all other prompts.  Overall, I was impressed with the documentation once I actually read it and followed it closely.

We are happy that our users can now simply use the pip install utility.

pip install wwu-tinker

I love software development and technology, and some days can be frustrating, while others are purely rewarding.  And this was one of those moments where I felt like I was smashing my head against the against the screen for hours hoping to absorb some knowledge.  If you have made it this far and have had similar experiences, I would love to hear about it in the comments. Cheers fellow devs!

Updating Pypi Package is (not) an absolute nightmare

Leave a Reply

Your email address will not be published. Required fields are marked *