By Spencer Michaels, William Woodruff, Jeff Braswell, and Cliff Smith
Trail of Bits cares about internet freedom, and one of our most valued partners in pursuit of that goal is the Open Technology Fund (OTF). Our core values involve focusing on high-impact work, including work with a positive social impact. The OTF’s Red Team Lab exists to provide auditing services for the software that protects privacy and ensures open access to the internet, free of censorship. We’re a proud member of the Red Team Lab and have performed numerous engagements on software products that are critical to internet freedom. See what we’ve been up to below.
Security and usability improvements to PyPI
Back in 2019, we partnered with Changeset Consulting and Kabu Creative through the OTF to make security and usability improvements to Warehouse, the codebase that powers the Python Package Index (PyPI). PyPI’s criticality within the Python ecosystem is impossible to overstate: with over 500,000 projects and 750,000 project maintainers as of 2024, PyPI serves over a billion package downloads daily.
Our work on PyPI had four major angles:
- Implementing strong multi-factor authentication (MFA) methods on PyPI, in the forms of TOTP and WebAuthn
- Adding scopeable API tokens to PyPI to allow project maintainers to move away from insecure username/password pairs for package publishing
- Adding audit events to PyPI users and projects so maintainers could review security-sensitive actions performed on their accounts and projects
- Adding accessibility and internationalization enhancements to PyPI’s Web UI, including alignment with the W3C’s Web Content Accessibility Guidelines
Our work was an essential part of PyPI’s modernization efforts, following on the heels of Warehouse’s 2018 public beta. Scoped API tokens and modern MFA methods also made PyPI an early “gold standard” for package index security practices, with other major indices subsequently adding WebAuthn and scopeable API tokens once their security and usability benefits were clear.
All told, these improvements helped raise the security bar for one of the internet’s most critical packaging ecosystems. In doing so, they also demonstrated that indices can make security-enhancing changes without compromising users’ and developers’ ordinary workflows.
Auditing PyPI and its deployment infrastructure
In 2023, we came back to PyPI on the assurance side: in August and September, we audited a medley of codebases tied to PyPI and its deployment infrastructure:
- Warehouse itself, which makes up the bulk of PyPI’s front end and back end
- cabotage, which provides a Heroku-esque deployment substrate for PyPI’s runtime services
readme_renderer, which PyPI uses to safely render arbitrary (package-supplied)READMEfiles into HTML
Our audits of these codebases took place over 10 engineer-weeks and uncovered a total of 29 findings, including some with the potential to disclose otherwise private account states or compromise the integrity of PyPI’s runtime services. We concluded our audit with a fix review, in which we determined that PyPI’s maintainers had satisfactorily patched or otherwise mitigated every finding.
The results of our audits validated PyPI’s development philosophy: a strong emphasis on automated testing, linting, and QA meant that relatively few low-hanging bugs were found and that the majority of findings occurred in parts of the codebase where individual services could interact in unintended ways. We believe this merits consideration in other packaging ecosystems, especially as general interest in supply chain security rises. An ounce of prevention in the form of tests and automated QA is worth a pound of cure at the time of the audit.
You can read our audit report, as well as our accompanying blog post, for more details. PyPI’s administrators have also released a three-part blog post series with an in-depth analysis of each finding: part 1, part 2, and part 3.
OpenArchive’s Save application on iOS and Android
Human rights activists, journalists, and civil society organizations all have a common need to preserve and share media in a way that protects privacy while avoiding data loss and tampering. The OpenArchive Save app provides this diverse group of users with a way to securely upload photos and videos to shared storage providers, optionally using the Tor anonymization network and including cryptographic signatures that authenticate the media files. We recently conducted two code reviews for the iOS and Android versions of the Save app.
Using a threat model that included bad-acting nation states with broad censorship powers, our consultants assessed the Save applications using dynamic testing and code review. OpenArchive worked quickly to improve the security and design of the applications, including performing substantial refactoring, in the months following our engagement. These updates helped defend against social engineering, protect locally stored media and credentials from theft, and ensure safe transmission of data across networks operated by a hostile adversary. We also provided guidance that will help OpenArchive make the best possible use of available cryptographic tools in the future. You can see the publications for each application version in our publications repository: the iOS summary report and the Android summary report.
What the future holds
Knowing the OTF’s vision of “community, collaboration, and curiosity,” we are looking forward to bringing our foundation in fuzzing and continuous testing to future engagements. After all, we often find issues that would be easy to spot early in development with the correct security tooling but that make their way across the software life cycle undetected. In the spirit of collaboration, we’ve gathered what we’ve learned about continuous testing into our new Testing Handbook, which is free for everyone to use.
In addition to effective testing techniques, internet freedom requires reliable software development ecosystems to support open-source development. Our work connected to PyPI has improved the security posture of the Python ecosystem at large, and we welcome opportunities to continue this work in other domains.