Autodiscover New Services in Repository Without Needing to Update doco-cd.yaml

[epoch-philosophy]

Hi! There's one feature/roadblock that is keeping me from migrating over to doco-cd! That's the ability to add new services/compose files in your directory without having to manually add them to doco-cd.yaml. This one is a huge bummer to me.

For example, in FluxCD in Kubernetes, you simply add new files to your repository (in this case of doco-cd, it would be a new docker-compose file!) and FluxCD will simply deploy them. To my knowledge this will not work in doco-cd. You have to manually add a new service to the doco-cd.yaml file which adds another layer of configuration to deploy containers. This is the one thing that bugs me about Portainer and Komodo. It's just another layer of configuration and work to do when deploying a stack.

If we can simply throw new docker-compose files into your repository, say: apps/service/compose.yaml and point doco-cd to scan the apps/ subdirectory without having to add new services in doco-cd.yaml, that would be an absolute game changer and would make this THE automation program for docker. Nothing would come close to it.

I think for now, because you need to manually add each new service file, there's not a huge reason for people to switch from Komodo to doco-cd unless you are resource starved! (I will add that the integration of SOPS and Swarm is a big plus!)

Anyhow, thank you for the work on this project. It has a ton of potential, but man, this feature would simply put docker git ops in near parallel functionality to Kubernetes. This would change everything.

(If doco-cd is capable of this and I'm an idiot please let me know. I couldn't get it to work this way.)

[kimdre]

Good idea, I didn't think about something like this yet. Doing it this way will however require the users to always set the project name in the compose file: https://docs.docker.com/compose/how-tos/project-name/
I'm not sure if Swarm Mode also supports this.

[epoch-philosophy]

Glad you think it's a good idea. I would be so stoked if implemented. This has been the one thing I've missed about using FluxCD.

Hmm, that would seem like an okay tradeoff if that allowed the feature to work. I am not sure if Swarm supports compose names, either. Would there be no way for doco-cd to scan any service/container name under "services" in the docker-compose file?

[mirisbowring]

Or maybe "compile" the stack name from subdir?

Like apps/nextcloud/compose.yaml would be the nextcloud stack?

[epoch-philosophy]

That's also a good idea. I am no software developer, so hopefully this is possible!

[kimdre]

I'm working on the feature: #766

[epoch-philosophy]

Very awesome! Big thanks.

[kimdre]

I created a pre-release you can test. Please send me some feedback if this what you had in mind.
The tag is 0.50.0-rc.1
See the release notes for more info: https://github.com/kimdre/doco-cd/releases/tag/v0.50.0-rc.1

[epoch-philosophy]

Sure. Let me spin up a fresh VM this evening and I'll test it with Docker and Docker Swarm!

[epoch-philosophy]

Commenting back to confirm it seems to be working well on my end!

[kimdre]

@epoch-philosophy Currently doco-cd always auto-deletes missing apps from the Docker host. I could add some sync-options (similiar to what argo-cd has), so you can for example disable the auto-delete.

Edit:
I addes some options to the auto_discovery features, see https://github.com/kimdre/doco-cd/releases/tag/v0.50.0-rc.2

[epoch-philosophy]

Very cool! I think adding options there would be a great idea.

That said, not sure if I quite comprehend the following option:

auto_discover_opts: 
  delete: (bool)

If I move the subdirectories outside the auto_discover directory, wouldn't it be deleted automatically?

[kimdre]

Yes that is the default behaviour. You can disable it if you want to keep the deployments for whatever reason.

[gdechene]

Hello,
Thanks for the project!
How to handle extra env variables with the autodiscovery? I installed infinisal for my secrets

[kimdre]

The same way as before. Use a Prefix in your variables if you need to separate between your services.

[kimdre]

The new release v0.50.0 is now available and contains this feature. :)

[epoch-philosophy]

Looks like we got the feature! Thanks a ton!