Skip to content

Commit 967bd87

Browse files
mbuffambuffa
committed
Publish Blog: Hacking my Linux server at home - Part 2
1 parent 72dd89c commit 967bd87

1 file changed

Lines changed: 317 additions & 0 deletions

File tree

‎content/articles/20260120-blog-server-hacking-part-two.md‎

Lines changed: 317 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,317 @@
1+
+++
2+
title = "Blog: Hacking my Linux server at home - Part 2"
3+
description = "Setting up Gitea, Minikube, a CI Runner and a Cloudflare Tunnel"
4+
date = 2026-01-20
5+
[taxonomies]
6+
tags = ["blog", "server", "sysadmin", "dev-ops", "k8s"]
7+
+++
8+
9+
### See also
10+
* [Part One][0]
11+
12+
## Context
13+
In my previous post, I mentioned setting up a home server, which was at this point not doing a lot of things, aside from having a PostgreSQL database and a SSH server.
14+
15+
In this post, I'll go into details how I managed to get those components up and running:
16+
* A working Gitea (alternative to Github),
17+
* A CI Runner, compatible with Github Actions, for tests and deployments,
18+
* A docker registry to hold my services images,
19+
* A Kubernetes cluster, using Minikube,
20+
* A Cloudflare Tunnel, to expose my services to the World Wide Web,
21+
* And finally, the SystemD configuration files needed to make all of this work (almost) smoothly.
22+
23+
So, let's move on! I'll try not to flood this page with configuration files though, to keep it clean and simple.
24+
25+
## Gitea
26+
Gitea was [straighforward to install][1]. In my case, there was a Arch Linux package ready for me, so I just picked that option. The idea was to have something up and running, with the least possible number of abstractions and dependencies, so I didn't think about running it inside Kubernetes, or a VM. At least, not for now.
27+
28+
Gitea requires a SQL database, so you'd definitely need to provide one, and check that the Postgres database would accept connections for the `gitea` user in the usual `/var/lib/postgres/data/pg_hba.conf` configuration file:
29+
```
30+
# TYPE DATABASE USER ADDRESS METHOD
31+
local giteadb gitea scram-sha-256
32+
```
33+
34+
I believe the package was bundled with the proper SystemD config, so a simple `systemctl enable gitea` was enough to add it to the services SystemD would start on the next reboot.
35+
36+
**One important note though**: In my case, the gitea user was configured with a password reset every six months. So, one morning, I had the sad surprise of realizing something was broken in my setup. You can check if your password has expired by running `chage -l YOUR_USER` and `chage -E -1 YOUR_USER` to disable the password reset.
37+
38+
In my case, since my Gitea user was forbidden to accept TTY login, I decided to disable the password reset. It didn't seem very impactful to do so. And that user would be used only with SSH handshakes to save Git repos on disk (and, yeah, run the CI too).
39+
40+
After this initial setup, I was happy to open my numerous and unfinished Git repositories and run `git remote add self gitea@SERVER:USER/REPO.git`.
41+
When it would be time to push upstream, a simple `git push` and `git push self` (or `gp` and `gp self` with shortcuts) would be enough to send to both my Github account and my Gitea server.
42+
43+
I needed a CI to run tests though, but that was requiring two dependencies. Let's move to the next one.
44+
45+
## The CI Runner
46+
Gitea supports Github Actions, or [Drone CI][2]. I decided to go with the first option, as I already had a few Github 4s working. I wanted backward compatibility, and to be able to deploy from Github in the event that my server would go down.
47+
48+
I think I will eventually migrate, to, maybe, [ArgoCD][3] though, as Github Actions isn't great by itself, and, as I would discover later, Gitea doesn't actually support 100% of Github Actions. Some differences may emerge, regarding services hostnames within a test step, for example.
49+
50+
But anyway, we needed Docker. Just one package to install, and to make sure the service would start on boot, and we were good to go.
51+
52+
The [Act Runner repository][4] is a good place if you want quick notes on how to install and configure it. Downloaded the binary, ran the command to register the runner, and pasted the token I generated inside Gitea, so that the runner would be able to use it.
53+
54+
Remember that I'm using a single machine infrastructure. I may eventually get another Thinkcentre, so that I can declare a new K8s node on it, and have redundancy on my services, but for now, I'm configuring things on only one server, so some security configuration may apply if you decide to setup a swarm of machines.
55+
56+
It was relatively easy to have simple CI tasks such as tests to work. However, I wanted to be also able to deploy both trunk (automatically) and feature branches (manually). This is one part where I wished Github Actions was as good as Gitlab CI, because I wasn't able to DRY my CI with a single "deployment" step with various triggers. I had to declare two separate workflows.
57+
58+
But anyway, we need a Kubernetes cluster, right?
59+
60+
## Minikube
61+
When it comes to having Kubernetes on your calculator, you have a few options, at least three:
62+
* Minikube,
63+
* K3s,
64+
* and MicroK8s.
65+
66+
At least.
67+
68+
To be honest with you there, I quickly watched Canonical's [comparison table][5], saw that there was an Arch package for Minikube, and not for the other ones, and decided to go with it.
69+
70+
Not a professional benchmark, so you would say, but I'll swap the cluster "distribution" the day I lay my hands on another Thinkcentre, as I said, or even buy proper servers.
71+
72+
Anyway, installing Minikube was very easy, but I wrote my own SystemD service since it wasn't bundled with one:
73+
```
74+
[root@tinker makkusu]# cat /etc/systemd/system/minikube.service
75+
[Unit]
76+
Description=Kickoff Minikube Cluster
77+
After=docker.service
78+
79+
[Service]
80+
Type=oneshot
81+
ExecStart=/usr/bin/minikube start --apiserver-ips=ETHERNET_ADDR
82+
RemainAfterExit=true
83+
ExecStop=/usr/bin/minikube stop
84+
StandardOutput=journal
85+
User=makkusu
86+
Group=makkusu
87+
88+
[Install]
89+
WantedBy=multi-user.target
90+
```
91+
92+
The `--apiserver` part is interesting. Since my server does have two interfaces (a wireless, and an ethernet one), i needed to specify one in order to be able to use `kubectl` on my laptop. I believe you may use `0.0.0.0` here.
93+
94+
I also had to make sure it would run after the docker service was started, because I ended up using Docker's registry for images pull. Minikube does come with its own registry, but I wasn't able to make my cluster pull from it.
95+
96+
Also, I decided to make it run under my user, just like the basic official example shows. One would **probably** want to isolate the runtime to a user that isn't in the `sudoers` list though... I'll probably try to setup MicroK8s this way at some point.
97+
98+
Once the cluster is started, you can check that it's running by just using `kubectl get pods` or deploying hello world apps on it.
99+
I wanted to be able to run `kubectl` from my laptop though, and this required generating a certificate on the server, copy it to my laptop alongside the `~/.kube/config` configuration, and adapt the hostname. I won't copy that here for obvious reasons.
100+
101+
Anyway, that's where the "interesting stuff" begins.
102+
103+
## Making the Runner actually deploy
104+
This is where things got interesting, and it required me a bit of trial and error to get it right, or at least working. I'm sure there is some streamlining (and hardening) opportunity at arm's length.
105+
106+
I'll spare you the details about the intermediate steps, but it took a bit of work:
107+
* to get the runner to be able to push Docker images to Minikube's registry,
108+
* to be able to connect to Minikube's API to run kubectl commands in the runner (like `kubectl apply -f`),
109+
* realize pods weren't able to pull images,
110+
* set the runner to push Docker images to Docker's registry instead,
111+
* and have every bit work without breaking another.
112+
113+
And this required a bit of TCP port forwarding. So I needed to add 2 SystemD services using `socat` to route traffic:
114+
* From the Docker IP to my Minikube's registry IP (and port),
115+
* From Any IP to my Minikube API IP (so that kubectl would work in the runner AND on external devices on the network),
116+
117+
This resulted in those two files:
118+
```
119+
# minikube-registry-forward.service
120+
[Unit]
121+
Description=Forward Docker bridge to Minikube registry
122+
After=minikube.service
123+
Requires=minikube.service
124+
125+
[Service]
126+
Type=simple
127+
ExecStart=/usr/bin/socat TCP-LISTEN:5000,bind=DOCKER_IP,fork,reuseaddr TCP:MINIKUBE_IP:5000
128+
Restart=always
129+
RestartSec=5
130+
StandardOutput=journal
131+
StandardError=journal
132+
133+
[Install]
134+
WantedBy=multi-user.target
135+
```
136+
137+
```
138+
# minikube-api-forward.service
139+
[Unit]
140+
Description=Forward Docker bridge to Minikube API server
141+
After=minikube.service docker.service
142+
Requires=minikube.service docker.service
143+
144+
[Service]
145+
Type=simple
146+
ExecStart=/usr/bin/socat TCP-LISTEN:8443,fork,bind=0.0.0.0,reuseaddr TCP:MINIKUBE_IP:8443
147+
Restart=always
148+
RestartSec=5
149+
StandardOutput=journal
150+
StandardError=journal
151+
152+
# Security hardening
153+
NoNewPrivileges=true
154+
PrivateTmp=true
155+
156+
[Install]
157+
WantedBy=multi-user.target
158+
```
159+
160+
Now that I see it posted, it seems ridiculously straightforward. But I did loose a bit of hair when trying to figure out why my CI would work one day, and would stop working the next one, after I fixed `kubectl` on my laptop.
161+
162+
I previously mentioned that my k8s pods weren't able to pull images initially. I honestly do not remember what the message was, and maybe I was just missinbg a port forward, or some authorization, but I ended up using Docker's registry anyway.
163+
164+
### One more port forward
165+
One more port forward needed to route HTTPS traffic to the cluster's ingress!
166+
```
167+
# ingress-https-forward.service
168+
[Unit]
169+
Description=Forward HTTPS to Minikube Ingress NodePort
170+
After=network.target
171+
After=minikube.service
172+
173+
[Service]
174+
Type=simple
175+
ExecStartPre=/bin/bash -c 'until nc -z MINIKUBE_API_IP 32655; do sleep 5; done'
176+
ExecStart=/usr/bin/socat TCP-LISTEN:443,bind=0.0.0.0,fork,reuseaddr TCP:MINIKUBE_API_IP:32655
177+
Restart=always
178+
179+
[Install]
180+
WantedBy=multi-user.target
181+
```
182+
183+
This is an interesting one. You can see that I'm starting the service after the network interface and minikube are started. But that doesn't mean they would be ready to use.
184+
185+
Hence the `ExecStartPre=/bin/bash -c 'until nc -z MINIKUBE_API_IP 32655; do sleep 5; done'` line, to delay the service start until Minikube is actually ready to welcome traffic.
186+
187+
`32655` is my Minikube port for HTTPS.
188+
189+
### One last thing
190+
I needed a TLS certificate, stored as a Kubernetes secret. Of course, that secret would stay on the server. I would only reference it in my k8s files in Git.
191+
I would eventually evaluate modern SecretOps alternatives. I only used SOPS in the past.
192+
193+
I'll obviously skip all the TLS generation part, which ends up with a `.pem` and a `.key`. Those are common but quite tedious to generate.
194+
195+
Things were not over though. At this point, I was able to access my server over my local network with a simple `/etc/hosts` entry, but I wanted to be able to access my pods over the Internet.
196+
197+
Gitea would stay local network only.
198+
199+
I remembered about DynDNS a lifetime ago. I wanted to have something similar setup, and to avoid needing a static IP address from my ISP. So I decided to go with Cloudflare Tunnels.
200+
201+
## Opening to the Internet
202+
I already had a cool domain name. Now it was about time to use it.
203+
204+
The plan was:
205+
* To create the tunnel, and run `cloudflared` on the server,
206+
* To modify some DNS entries to redirect to Cloudflare's Tunnel,
207+
* And to make sure traffic was effectively hitting the Tunnel, the Ingress, and eventually, the Pods
208+
209+
You can say I saw that Cloudflare "Gateway Error" gray page for a few hours.
210+
211+
Setup was relatively easy to follow. You can basically choose to create Tunnels from their dashboard, or yourself with the CLI, which I believe they call "Self-Managed Tunnels".
212+
213+
But I had forgotten to remove the `/etc/hosts` entry at some point. So I was quite a bi confused with the debugging.
214+
215+
The setup went basically like this:
216+
```sh
217+
pacman -Ss cloudflare
218+
yay -S cloudflared
219+
cloudflared tunnel login # Open a Oauth2 page and gets a token
220+
cloudflared tunnel create suto-tunnel # Yeah, that's the name of my app
221+
cloudflared tunnel route dns suto-tunnel suto.MY_SECRET_DOMAIN.io
222+
sudo cloudflared service install # To create the SystemD files
223+
sudo systemctl enable cloudflared
224+
```
225+
226+
`sudo cloudflared service install` created a couple of services, mainly `cloudflared.service` and a timer update service which was generic enough and that I didn't modify at all.
227+
228+
However, my `cloudflared.service` file ended up looking like this:
229+
```
230+
# cloudflared.service
231+
[Unit]
232+
Description=cloudflared
233+
After=network-online.target
234+
After=ingress-https-forward.service
235+
Wants=network-online.target
236+
237+
[Service]
238+
TimeoutStartSec=15
239+
Type=notify
240+
ExecStart=/usr/bin/cloudflared --loglevel debug --config /etc/cloudflared/config.yml tunnel run
241+
Restart=on-failure
242+
RestartSec=5s
243+
244+
[Install]
245+
WantedBy=multi-user.target
246+
```
247+
248+
One trivial detail: Using `create suto-tunnel` and `route dns` cloudflared subcommands would update your user's``~/.cloudflared/config.yaml`. So you have to make sure your services point to the right config files, if you decided to put those into `/etc/cloudflared`, of course.
249+
250+
I also thought that the service journal wasn't verbose at all, hence the `--loglevel debug`. I was getting crazy, not knowing if my external requests were actually hitting the tunnel, or if my DNS entries weren't properly set up or refreshed.
251+
252+
I still had issues with TLS though. So CF's default configuration ended up like this for me:
253+
```
254+
tunnel: TUNNEL_ID
255+
credentials-file: /home/makkusu/.cloudflared/TUNNEL_ID.json
256+
257+
ingress:
258+
- hostname: suto.MY_SECRET_DOMAIN.io
259+
service: https://localhost:443
260+
originRequest:
261+
noTLSVerify: true
262+
- service: http_status:404
263+
```
264+
265+
I basically had to end the TLS encryption to my tunnel. My browser wouldn't accept my self-signed certificate otherwise, because of my own self-declared authority. Cloudflare's certificate would take the relay here.
266+
267+
I don't believe this is a big deal, though.
268+
269+
## Moment of Truth
270+
One thing I like to do after finishing a setup just relies on pressing the power button.
271+
272+
You don't know if you did everything right if you kept your session alive, but forgot to enable or restart services. The only way to know is to pull the plug.
273+
274+
Yeah. This step took me a few times.
275+
276+
I think I still got an issue to this day though, because every now and then, trying to access my server would result in the Cloudflare gateway error page, and I don't think this is on them. I probably still have a race condition to fix, or to make sure the tunnel runs when all of its dependencies are ready, not just started.
277+
278+
## Finishing Touches
279+
Setting up a server is nice, but it's even nicer to keep track of its configuration and ADRs, so I just created a repository for my server, similar to infamous `dotfiles`. It's just securely stored in a Proton Drive :)
280+
281+
I do have a `k8s` folder in a few repositories, which contain app's YAML manifests. I decided to go with Kustomize for those, because I believe it's the simplest and the cleanest way to deploy small apps on k8s, because it uses plain YAML to generate other YAML, while Helm relies on templating and substitutions. It's like replacing Wordpress with a static site generator, in a way.
282+
283+
And eventually, I'll probably have a "generic" CI for my server itself, for shared services, such as PostgreSQL databases with specific versions, and some observability services too :)
284+
285+
Helm can be very handy though, with its existing collection of charts. But I would keep it at bay for my own apps.
286+
287+
It's also worth mentioning that I used Claude Code from time to time, especially for generating a self-signed certificate. This definitely made this step less tedious. And since I stored my server's general info (such as IP address, the services it runs, etc) as documentation, I was able to leverage Claude in interesting ways. This post is purely from a human, though.
288+
289+
## Wrap-up
290+
So, this was quite a big chunk. What's next?
291+
292+
Well.
293+
294+
I'm still undecided which one in particular is next though, as this post is pretty up-to-date with my server's current state.
295+
296+
What was this guy doing all of this for again?
297+
Oh yeah. Dev server, and training.
298+
299+
Well, next time, I'll probably cover one of those topics:
300+
* Setting up ArgoCD (GitOps),
301+
* Setting up Grafana LGTM for Observability, Logs, Tracing (exciting!),
302+
* Setting up SOPS for secrets management,
303+
* Adding a new machine as a cluster node (ok, this one is definitely not next, I need to buy the hardware) (and swapping Minikube with MicroK8s?),
304+
* Server hardening? Since I do not have that many services running or ports open yet, that one may be short.
305+
306+
Yeah. We'll see, we'll see.
307+
308+
Anyway, see you next time.
309+
310+
P.S.: I didn't include any copy of my app's kubernetes YAML files. Those are pretty standard and would have cluttered the body of this post for little added value.
311+
312+
[0]: /articles/20251110-blog-server-hacking-part-one
313+
[1]: https://docs.gitea.com/category/installation
314+
[2]: https://www.drone.io/
315+
[3]: https://argoproj.github.io/cd
316+
[4]: https://gitea.com/gitea/act_runner
317+
[5]: https://canonical.com/microk8s/compare

0 commit comments

Comments
 (0)