*This guide is adapted from [A Gamers Grind docs](https://docs.agamersgrind.com/guides/portainer-and-gitops). His guide helped immensely when I first setup gitOps with Portainer.* # Preface Our goal is to leverage GitHub actions (A.K.A gitOps) to provide: - A single point of truth of docker-compose files - Secure DevOps pipeline - Stacks are automatically redeployed when changes to it's compose file are pushed to your repository. - Multi-user collaboration (if needed) - Container image updates via [Renovate](https://github.com/marketplace/renovate) - Convenient maintenance of compose files This guide is specifically for a Komodo core instance **BEHIND** a Cloudflare tunnel that is **not** directly accessible from the internet, however much of what will be included can be implemented without Cloudflare (with some tinkering). A setup like this seems daunting to setup, however the benefits and workflow are well worth the effort. ### Prerequisites *Note:* No paid plans are required. - Cloudflare domain - Cloudflare tunnel - Cloudflare zero-trust (access) setup - Komodo added as an application - CNAME record pointing to CF tunnel - Private GitHub repository (stores docker-compose files) - Working Komodo instance # Laying the groundwork ### Install Renovate in your Repo 1) Create 2 folders in your repo: - `.github` - `docker-compose` 2) In `.github/` create renovate.json5 with the below contents: > [!TIP]- renovate.json5 > ```json5 > { > $schema: 'https://docs.renovatebot.com/renovate-schema.json', > extends: [ > 'config:recommended', > ':disableRateLimiting', > ], > 'docker-compose': { > managerFilePatterns: [ > '/docker-compose/.+\\.ya?ml$/', > ], > }, > "ignorePaths": ["**/docker-compose/komodo/**"] > } > ``` 3) Follow [this link](https://github.com/marketplace/renovate) and install the bot 4) Renovate can run on all of your repos or just the repo for docker-compose files. I suggest limiting it to run on only the repos that are needed. If all goes well, you will now see a new *Issue* in your GitHub repo. This is the dependency dashboard for Renovate. ### GitHub access token 1) Click your account settings > scroll down and click Developer Settings on the left 2) Expand Personnel Access Tokens > Tokens (classic) 3) Generate a new classic token 4) Give it a unique name 5) Tick the box for "Repo" 6) Click generate token at the bottom and **SAVE THE TOKEN IN A SAFE PLACE** ### Komodo Container configuration... - I followed and used the compose and ENV file provided my Komodo docs. - The ENV file needs to have the following options set: - `KOMODO_HOST` - Set to the **external** URL that points to Komodo (through the tunnel), this matches the CNAME record. - For example: `KOMODO_HOST=komo.mydomain.com` - `KOMODO_WEBHOOK_SECRET` - Set a nice password for webhooks to use. In Settings... 1) Providers > New Git account 2) Domain: github.com 3) Username: your username 4) Token: Your personal access token generated on github In Resources > Repos... 1) New repo 2) Give the new repo a name 3) Set your server and builder 4) Account: select the git account you created in providers previously 5) Repo: `{Username}/{Repo_Name}` for example: `MyGitAccount/SelfHostFiles` In Procedures... 1) Create a new procedure 2) Stage 1: 1) Name: Pull Repo 2) Execution: Pull Repo 3) Target: Select the Repo you created in Komodo 3) Stage 2: 1) Name: Deploy Stacks if Changed 2) Execution: Batch Deploy Stack If Changed 3) Target: `*` - I use a template for new stacks which pre-fills some common information such as run directory. If you set the target to `*` the procedure will fail on the template. To fix this I prefix my template name with `_`. The target of my procedure is: `[!_]*` 4) Enable the webhook at the bottom # Create your first compose file Now, let's create our first stack. In this section I will be using [Vikunja](https://vukunja.io) as an example. The only extra legwork we need to do is pin the image version for each service. Renovate will then check these versions in your compose files & then open a PR with the updated image tag. Finding the latest version tag can be a nuisance at times. Create a compose file in `<your git repo>/docker-compose/`: > [!Tip] vikunja.yml > ```yaml > services: > vikunja: > image: vikunja/vikunja:1.0.0-rc1 > user: 1000:1000 > environment: > VIKUNJA_SERVICE_JWTSECRET: secret_string > VIKUNJA_SERVICE_PUBLICURL: your_vikunja_url > # Note the default path is /app/vikunja/vikunja.db. > # This config variable moves it to a different folder so you can use a volume and > # store the database file outside the container so state is persisted even if the container is destroyed. > VIKUNJA_DATABASE_PATH: /db/vikunja.db > volumes: > - files:/app/vikunja/files > - vikunja/db:/db > restart: unless-stopped > ``` You can now check the Renovate dependency dashboard. You should see your new compose file as a detected dependency. *It may take a few minutes for the compose file to appear here* ### Image versioning (DockerHub) Notice the version tag: `vikunja/vikunja:`**1.0.0-rc1** in the above example. **An actual version tag must be used with renovate** To obtain the version tag from an image on docker hub we need to match the tag we want e.g. `latest` to the actual version number. Compare the md5 digests for your system (Most likely amd64). For Vikunja we see the `latest` tag digest for amd64 is `...593f` we can find that same digest under the `1.0.0-rc1` tag. ![[DockerHubVikunjaExample.svg|1000]] ### GitHub Container Registry (ghcr.io) Image Versioning First, go to the repository for the image. Find the "Packages" section on the right and click the image you are looking for. I will be using [gethomepage](https://github.com/gethomepage/homepage). ![[ghcrHomePageExample.svg|1000]] Above, we see the `latest` tag. We would use the tag `v1.4.5` in the compose file. ## Deploy your new stack With your first compose file on github, head into Komodo > Repos > Your repo. Click Pull so that your new file is pulled down to your server. Now, create a new stack: 1) Stacks > new stack 2) Name your stack 3) Mode: files on server **Files Section** - Run Directory: `/etc/komodo/repos/<your_repo>/docker-compose` - File Paths: `<your_compose_file>.yml` - Set environment variables if needed Deploy stack! You should now have your first stack from GitHub working! In the INFO tab for the stack you can see the contents of the compose file that was provided. # Cloudflare Service Auth We need to setup a service auth token for our Cloudflare worker to use: 1) Go to your Cloudflare Zero-trust dashboard 2) Go to Access > Service auth 3) Create a new token 4) Name it something memorable and set the duration 5) Generate new token 6) **SAVE THE PRESENTED STRINGS SOMEWHERE SAFE**, we will need it later. Service auth policy: 1) Access > Policies 2) Add a policy 3) Name the new policy 4) Use `include` with the selector: `service auth token` 5) The value is the name of the service auth token you just created. 6) Make sure you add this policy to the application in Access > Applications > Komodo # Cloudflare Worker Setup Here we will setup the cloudflare worker that will: 1) Receive a webhook from Github 2) Verify webhook secret 3) Add cloudflare access token header 4) Forward the webhook to Komodo through the CF tunnel Head over to your [Cloudflare dashboard](https://dash.cloudflare.com/) and click on "Computer (Workers)" on the left hand side. 1) Create a new worker - choose the "Start with Hello World!" option. 2) Name your worker and deploy it 3) Click on edit code, and use the below script: > [!tip]- worker.js > ```js > export default { > async fetch(request, env, ctx) { > // Only allow POST requests > if (request.method !== 'POST') { > return new Response('Method not allowed', { status: 405 }); > } > > // Verify this is coming from GitHub (optional but recommended) > const userAgent = request.headers.get('User-Agent'); > if (!userAgent || !userAgent.includes('GitHub-Hookshot')) { > return new Response('Unauthorized', { status: 401 }); > } > > try { > // Get the webhook payload > const body = await request.text(); > > // Validate GitHub signature > const githubSignature = request.headers.get('x-hub-signature-256'); > if (!githubSignature) { > console.error('No GitHub signature found'); > return new Response('No signature provided', { status: 401 }); > } > > console.log('GitHub signature:', githubSignature); > console.log('Secret length:', env.GITHUB_WEBHOOK_SECRET ? env.GITHUB_WEBHOOK_SECRET.length : 'undefined'); > console.log('Payload length:', body.length); > > // Verify the signature > const expectedSignature = await generateSignature(body, env.GITHUB_WEBHOOK_SECRET); > console.log('Expected signature:', expectedSignature); > > if (!verifySignature(githubSignature, expectedSignature)) { > console.error('Invalid GitHub signature'); > console.error('GitHub sent:', githubSignature); > console.error('We expected:', expectedSignature); > return new Response('Invalid signature', { status: 401 }); > } > > console.log('Webhook received, forwarding to:', env.KOMODO_WEBHOOK_URL); > > // Create new headers with Cloudflare Access credentials > const headers = new Headers(); > > // Copy original headers (except Host) > for (const [key, value] of request.headers.entries()) { > if (key.toLowerCase() !== 'host') { > headers.set(key, value); > } > } > > // Add Cloudflare Access service token headers > const clientId = await env.CF_ACCESS_CLIENT_ID; > const clientSecret = await env.CF_ACCESS_CLIENT_SECRET; > > headers.set('CF-Access-Client-Id', clientId); > headers.set('CF-Access-Client-Secret', clientSecret); > > console.log('Added access headers, making request...'); > > // Forward to your Komodo instance > const response = await fetch(env.KOMODO_WEBHOOK_URL, { > method: 'POST', > headers: headers, > body: body > }); > > console.log('Komodo response status:', response.status); > console.log('Komodo response headers:', [...response.headers.entries()]); > > const responseText = await response.text(); > console.log('Komodo response body:', responseText); > > // Return the response from Komodo > return new Response(responseText, { > status: response.status, > statusText: response.statusText, > headers: response.headers > }); > > } catch (error) { > console.error('Webhook proxy error:', error); > return new Response('Internal server error', { status: 500 }); > } > } > }; > > // Helper functions for GitHub signature validation > async function generateSignature(body, secret) { > const encoder = new TextEncoder(); > const key = await crypto.subtle.importKey( > 'raw', > encoder.encode(secret), > { name: 'HMAC', hash: 'SHA-256' }, > false, > ['sign'] > ); > > const signature = await crypto.subtle.sign('HMAC', key, encoder.encode(body)); > const hashArray = Array.from(new Uint8Array(signature)); > const hashHex = hashArray.map(b => b.toString(16).padStart(2, '0')).join(''); > return `sha256=${hashHex}`; > } > > function verifySignature(githubSignature, expectedSignature) { > // Use a constant-time comparison to prevent timing attacks > if (githubSignature.length !== expectedSignature.length) { > return false; > } > > let result = 0; > for (let i = 0; i < githubSignature.length; i++) { > result |= githubSignature.charCodeAt(i) ^ expectedSignature.charCodeAt(i); > } > > return result === 0; > } > ``` 4) Deploy the worker ### Set Worker Variables Head back to [Cloudflare dashboard](https://dash.cloudflare.com/) 1) Compute (Workers) > Click the name of your worker 2) Click Settings 3) Click Variables and Secrets, and add the following variables: | Type | Name | Value | | ------ | ------- | ------- | | Secret | CF_ACCESS_CLIENT_ID | String from zero trust service auth token id | | Secret | CF_ACCESS_CLIENT_SECRET | String from zero trust service auth token secret | | Secret | GITHUB_WEBHOOK_SECRET | Matches `KOMODO_WEBHOOK_SECRET` configured in the .env file for the Komodo stack | | Plaintext | KOMODO_WEBHOOK_URL | Matches the public URL of your Komodo instance through the CF tunnel. EX: `komo.mydomain.com` | 4) At the top of the settings page copy the URL for `workers.dev` 5) Redeploy/rebuild worker # GitHub webhook configuration Now, we setup the webhook from GitHub. 1) Head over to your repo on GitHub, and click settings at the top 2) On the left, click "Webhooks" 3) Click add webhook 1) Name it 2) Payload URL: Cloudflare worker (copied from section above) 3) Content type: application/json 4) Secret: Matches `GITHUB_WEBHOOK_SECRET` from Cloudflare worker variables AND `KOMODO_WEBHOOK_SECRET` 5) Enable SSL Verification 6) `just the push event` triggers the webhook. 7) Activate the webhook # Testing your Configuration To test, commit a change such as a comment to an existing compose file. You will see Komodo run the procedure we created before which will pull your repo and re-deploy any changed stacks. If the procedure does not even run, this likely means the webhook did not make it through. To troubleshoot this, check the logs at the following locations: - GitHub webhook recent deliveries - Check the response. You can trigger a re-delivery as well. - Cloudflare worker logs tab - The script that I have in the worker setup is very verbose. Check here for any issues. - View the response # The workflow Here is my workflow! ```mermaid flowchart TD A[I commit a change to a compose file] A1[Renovate finds a new image version] A1 --> B1[Renovate opens a PR with new image tag] B1 --> C1[PR is merged] --> B A --> B[GitHub triggers webhook] B --> C[CF worker verifies secrets, adds headers, forwards to Komodo tunnel] C --> D[CF Access policy allows webhook with valid service auth token] D --> E[Komodo Receives Webhook] E --> F[Komodo pulls reponsitory] F --> G[Komodo re-deploys any changed stacks] ```