Automatic rsync deployment with Github Workflows

Updated on July 29, 2023
Updated the fact that this blog is now built with Hugo instead of Jekyll

As an indie developer, I used to manually deploy my web applications to my VPS. But the more and more subsystems and apps I tend to build, the more my server get crowded with new applications to deploy and maintain.

As I write those lines, my VPS powers:

By multiplying the number of services, this process became time-consuming, but it was also prone to human error. It was time for me to automate most of those processes

Motivations

Aside the logical time-saving and reliable deployment, what really triggered my shifting to this solution was storage consideration.

Not that I specially want to become a storage Scrooge, but my VPS is only 20G; and I’d prefer to increase this storage only because of business reason, and not because of my manual deployment method.

Manual deployment rely on Git. Git comes with a special folder .git that ends up to contains all the history of the repository. Not relying on git anymore ends-up to save a lot of space, as long as we do not deploy this folder anymore.

Also, the static web generator tools like Jekyll or Hugo are software that will generate the entire website to serve in a subfolder (_site for Jekyll, and public for Hugo). The same goes for the Vue.JS webapp, that will generate a standalone dist folder.

Aside those generated folders, all the “source code” of those websites are also not necessary on the server. It prevents to have assets like images/videos in double, and also save some more space.

Finally, plenty of tools that were installed on the server (Ruby, Hugo, Jekyll and all the dependencies) were installed on the server to perform the generation, and will no longer be of use. Again some space saved on the disk.

As a result, I saved 30% of my total storage by moving to automatic deployment.

Github workflows

GitHub workflows are a powerful tool for developers to automate their workflows and streamline their development process.

A GitHub workflow is a set of automated actions that are triggered based on specific events, such as a push to a repository, a pull request being opened, or a new tag being created.

Workflows are defined in yaml files, and located in the .github/workflows directory right in your repository. It’ll include:

Name
The name of the workflow that will be displayed in the GitHub Actions tab.
On
The event that triggers the workflow, such as a push to a specific branch, a pull request being opened, or a new release being created.
Jobs
The set of actions that should be performed when the workflow is triggered. Each job can include a set of steps, which define the individual actions that should be performed, such as building the code, running tests, or deploying the application.
Environment variables
Optional variables that can be defined for use in the workflow.
Notifications
Optional notifications that can be sent to team members or external services, such as Slack or email, to alert them of specific events in the workflow.

Deployment process

To build and deploy the apps, there are few steps, some optional, to ensure stability and consistency of deployments.

  1. Checkout the code : A must-have. We do need the code to build things
    actions/checkout

  2. Install the required tools : Depending on the app we build: Ruby, Hugo, Node, PHP, Composer…
    ruby/setup-ruby
    peaceiris/actions-hugo
    actions/setup-node
    shivammathur/setup-php

  3. Install the code dependencies : Using the dependency manager, like Composer, Bundler or npm.

  4. Run the test suites when applicable : To make sure we won’t break something, write tests, lot of tests. And run them. Always.

  5. Remove the dev dependencies when applicable : We do not want to deploy some useless dependencies that are only used to run the test suite, when dependencies are shipped with the code (like with composer based app).

  6. Rsync only the result folder : Rsync is powerful because it’ll only send files that have been changed, preserving i/o processing and bandwidth. Of course, we will skip useless folders and files like .git.
    burnett01/rsync-deployments

  7. Run post-deployment scripts when applicable : Some apps need to run database migrations. Others will require an associated daemon process to be restarted.

Dealing with secrets

Rsync-ing requires secrets, like SSH port, SSH private key. Those should never be committed.

Instead, Github provide a secret handling mechanism directly in the repository settings:
Go to your project > Settings > Secrets and variables > Actions

From there, add as many secrets as you may require, like host, port, user, ssh key and deployment folders.

Examples with my workflows

Hugo deployment
Dean’s blog
Jekyll deployment
Affres de Reivax
Symfony deployment
Padlok API
Pet feeder API
Vue.JS deployment
Padlok Share webapp

Closing remarks

Discovering, and using Github workflows; as well as reconfiguring my server to run the newly deployed code, aside with removing the old now useless tools from it took me quite some time. But I’m sure that it’ll work out in the long run.

Haven’t calculated how much, but it reminds me the good old XKCD’s about task automation.

XKCD: Is It Worth the Time?


Don’t miss a thing!

Don't miss any of my indie dev stories, app updates, or upcoming creations!
Stay in the loop and be the first to experience my apps, betas and stories of my indie journey.

Thank you for registering!
You’ll retrieve all of my latest news!

Anti-bot did not work correctly.
Can you try again?

Your email is sadly invalid.
Can you try again?

An error occurred while registering.
Please try again