Staying safe while using third-party packages in Python

Whenever we are building something with Python, most of the time we rely on third-party packages so that we don't have to reinvent the wheel for some simple things like sending an HTTP request. By doing so we lose control of how safe we are with that decision. If you want to learn how to prevent security holes inside of your application or service, read below :)

Introduction

Before I explain the simple way HOW to prevent security holes that you could have inside of your application, we should first understand WHY this is so important. Let me kick off this article with some interesting stats:

• Cybercrime is up ~600% due to the COVID-19 pandemic
• In 2020, the average time to identify a breach was 207 days
• 43% of cyberattacks target small businesses
• $3.86 million is the global average cost of a data breach

Here comes the scariest one:

• 95% of cybersecurity breaches are a result of human error

In other words - 95% of the above could have been prevented. Unfortunately, human neglect is what leads to these numbers. Most of these could be prevented simply by making use of any sort of vulnerability scanner.

In my article, I will explain in detail how to utilize a simple scanner that will keep your dependencies safe :)

1. Dependency problem

Let's assume you are building an app that will act as currency exchange software (GUI, API, ...). You will need to hook up to some sort of external API to get the exchange rate list that you will rely on to do the currency conventions. Now comes the tricky part - unless you will do EVERYTHING from scratch, you will probably make use of requests (3rd party) package.

This is where things are getting messy. For the requests to work, that 3rd party package ALSO uses other 3rd party packages and installs them inside of your app (docker image, venv, server ...). If you check the setup.py from the requests package you can see exactly which 3rd party packages are needed for requests to work:

requires = [
    'charset_normalizer~=2.0.0; python_version >= "3"',
    'chardet>=3.0.2,<5; python_version < "3"',
    'idna>=2.5,<3; python_version < "3"',
    'idna>=2.5,<4; python_version >= "3"',
    'urllib3>=1.21.1,<1.27',
    'certifi>=2017.4.17'
]

Not that bad, right? Well, the thing is that every one of these can ALSO have sub-packages they need to run themselves.

1.1 Building the complete dependency tree

Ok, let's see all of this in action. Create a new project and create a virtual environment along with it. Activate the venv you just created:

source /path-to-your-venv/bin/activate

Once your venv is activated, create a file (outside of your venv in your project root) called requirements.in and put the following inside of it:

aiohttp
requests

For the fun of it, I also added the aiohttp package. Now let's install pip-tools to compile the dependency tree:

pip install pip-tools

Build the dependency tree by executing the following:

pip-compile -r requirements.in

After it is done, you will notice a new file in your project root called requirements.txt. That's what we were looking for - the complete dependency tree. All of the requirements are there:

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile requirements.in
#
aiohttp==3.8.1
    # via -r requirements.in
aiosignal==1.2.0
    # via aiohttp
async-timeout==4.0.2
    # via aiohttp
attrs==21.4.0
    # via aiohttp
certifi==2021.10.8
    # via requests
charset-normalizer==2.0.11
    # via
    #   aiohttp
    #   requests
frozenlist==1.3.0
    # via
    #   aiohttp
    #   aiosignal
idna==3.3
    # via
    #   requests
    #   yarl
multidict==6.0.2
    # via
    #   aiohttp
    #   yarl
requests==2.27.1
    # via -r requirements.in
urllib3==1.26.8
    # via requests
yarl==1.7.2
    # via aiohttp

Do you see how many there are? And we only included two packages for our application. Imagine how many there are once you are finished creating your application :)

2. Safety

Now that we know WHY we should use a vulnerability scanner, it is time to see HOW to do it.

Safety to the rescue.

Safety (also a 3rd party package!) is a dependency security scanner that checks all of your application dependencies against the Python vulnerability database Safety DB. If there are any vulnerabilities found, safety will output them as a plain text or JSON format. Safety can be used in many ways inside of your project and we will explore most of them.

2.1 Safety as a package

Straight forward way of using safety is to just install it as you would normally do with any other 3rd party package:

pip install safety

After that we can start the scanner by using the following command:

safety check -r requirements.txt

If everything is ok, you will get the following output (trimmed down):

| REPORT
| checked 12 packages, using free DB (updated once a month)
| No known security vulnerabilities found.

2.2 Safety inside of Docker image

Most of the time we will Dockerize our application for the sake of deployment/ease of use/ development/some other reason. Safety can fortunately also be used as a scanner of Docker images. Build your image as you would normally do and add a TAG to it. Afterward, run the command:

docker run -it --rm ${TAG} "/bin/bash -c \"pip install safety && safety check\"

2.3 Safety as a Docker image

If you don't want to install and run safety as a package, you can also make use of the Docker image of safety. That way you will hook up the contents of your requirements.txt and check them against the safety running inside of a Docker image:

cat requirements.txt | docker run -i --rm pyupio/safety safety check --stdin

2.4 Safety inside of CI service

If you are making use of CI services such as GitLab, you can also include safety as a part of your CI pipeline. Add the following as a step inside of any stage that you prefer in your .gitlab-ci.yml:

safety:
  script:
    - pip install safety
    - safety check

Make sure that the image that you use for your pipelines has pip installed

2.5 Safety inside of tox

You can also add safety as a part of your tox configuration:

[tox]
envlist = py39

[testenv]
deps =
    safety
    pytest
commands =
    safety check
    pytest

This way safety will run right before your tests execute.

2.6 Safety as a pre-commit hook

If you are using pre-commit hooks, safety can also be added inside of your .pre-commit-config.yaml:

-   repo: https://github.com/Lucas-C/pre-commit-hooks-safety
    rev: v1.2.4
    hooks:
    -   id: python-safety-dependencies-check
        files: requirements.txt

3. Conclusion

While there are so many ways to keep your application safe regardless of the tech stack it was built in, this time I just wanted to show you why making use of a vulnerability scanner could be essential for your project. It is fairly simple to include one inside of your Python projects and there should be no excuse for not using one in the first place.

Stay safe and happy coding!

Like always, thanks for reading :)