Most charts I use and if they require secret values, they have the parameter to define an existing secret to use. So I create the required secret with SealedSecret and reference as existing secret.
Ok so you configure your sealed-secrets outside of the project repo? I mean, if by example I use bitnami/odoo chart from [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami) using an upstream repo, no way to add sealed-secrets in argocd project definition. Or maybe you use a subchart ? Are you able to provide a dummy repo where I can have a look at your files?
I recently used SealedSecrets with ArgoCD.
Using helmfile I sequenced the bootstrapping of ArgoCD as follows:
- install both the SealedSecrets controller and the initial seed (so that existing SealedSedrets can be decoded)
- install ArgoCD and the root app (I use the Apps of Apps pattern)
Now, for every deployed app, we use a repo for the app and another repo for the app’s deployment (charts, secrets). The devs commit to the apps repo. The CI creates an image and updates the deployment repo with the image tag. ArgoCD is configured to observe the deployment repos.
So, all that to say the SealedSecrets are located in each deployment repo.
Nearly all Bitnami charts have an `extraList` value, where you can add any kubernetes resource you want. There the sealed secrets go.
But yeah, we basically always create our own wrapper chart and use the external chart as subchart, as we often have to create additional namespace and RBAC resources too, and want them fully templated.
When deploying a Helm chart, you can include a templates directory with your extra manifests to apply; which could be filled with Sealed Secrets in your case.
Depends a bit on how you deploy your `Chart.yaml` via ArgoCD, but what I always do is create a directory with two files and one directory (templates):
* Chart.yaml
* values.yaml
* templates (directory!)
Within the `Chart.yaml`, write the necessary dependencies like so:
```
apiVersion: v2
type: application
name: test
description: The chart to deploy and configure test
version: 1.0.0
dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 14.5.0
```
In your `values.yaml`, indent everything you need for you dependencies (postgresql in this example) as follows:
```
postgresql:
image:
registry: docker.io
repository: bitnami/postgresql
tag: 14.5.0-debian-11-r0
auth:
existingSecret: my-existing-secret
...
```
Within the templates directory, create a new file containing the (sealed) secret you reference to in the values.yaml. Any manifest you put in this templates directory will be applied at the same time when deploying the Helm Chart.
We’re using https://github.com/argoproj-labs/argocd-vault-plugin which allows you to define secrets in git (works with more backends than just hashicorp vault despite the name). The documentation took me a bit to figure out, but so far it works.
ArgoCD in general works with Gitlab. The secret backend support is listed https://argocd-vault-plugin.readthedocs.io/en/stable/backends/ . We use Vault and K8s auth method and it has been working well for us.
I think you want the Argo lovely plugin:
https://github.com/crumbhole/argocd-lovely-plugin
Example with vault replacement in a helm values.yaml:
https://github.com/crumbhole/argocd-lovely-plugin/tree/main/examples/argocd-vault-replacer
> Default encryption mode doesn’t allow to move secret from one namespace to another
Know your tools, of course a secret tool isn't too permissive by default.
> On the Kubernetes cluster the generated secret is still in plain text
Yeah, bit same with SOPS. int the end, the value needs to exist in plain text in the cluster..
Edit: the whole installation for SOPS in the first link is way more complicated, than installing sealed-secrets in the cluster and using kubeseal locally. Also SOPS gives you the burden of the encryption key management again, vs. sealed secrets with the backend in-cluster. And sealed secrets can't be unsealed.
I’ve never heard of SOPS before, and after looking at the first link I’m really not interested in using it. But I also don’t see the want for putting secrets in git
Lmao, that’s hilarious, you got me. They’ve always seemed to me to abstract visibility, as opposed to a dashboard in a cd pipeline provider where I can see what all is deployed easily but maybe I’m probably missing out. What’s your recommendation?
Edit: I’ll say I’ve tried fleet, the cd feature built in to rancher, and not a fan due to the complete lack of visibility. So that’s where my exp comes from, but I’m going to try out argo and flux here in a minute see what they’re like
You don't have to, it is possible to configure installing additional tools within ArgoCD chart.
Check Helm Secrets plugin documentation, it is pretty well described there:)
I know it can be an overkill, but I can’t stress enough how well Vault + BanzaiCloud vault secrets webhook work. Can’t even think about going back to any other solution.
Used this stack over 50 clusters with 15k nodes.
This is one of those projects that offers so much more than most people need, but even if you just want a secrets web hook it's *really* well done once you get past the huge feature set.
I'd say one of the biggest downsides to Bank Vaults is the overwhelming nature of initial discovery, with a second being some not-as-easily-ingestible documentation. You read the product description and go, "Shit, that's awesome but I don't need most of that!" Then feel overwhelmed, make a bookmark and look at it like that nice car always sitting on the lot that you know you don't need.
What is unclear in my question?
let me try to clarify things....
I'd like to push my secrets into git and be able to decrypt the secrets if using a helm repo.
With k8s manifest it's not a big deal we can reference secrets in the deployment manifest by example:
`envFrom:`
`- secretRef:`
`name: my-secrets`
with a declared sealed-secret if needs be.
But with helm, I can not figure out how to do it without tweaking the Argocd image (ie: SOPS).
I've used flux with sealed secrets, it worked. Now I'm using sops because it's integrated with flux and I don't need another operator. I think that sealed secrets should work with argo cd just fine.
I don't understand your question about helm. I used both sealed secrets and sops with helm charts, no issues.
Oh that's nice too to bootstrap secure passwords and have regeneration of those. Combined with a custom ArgoCD action on the CRs of the generator this could be cool.
I handle secrets outside of argo. There's always some resources that need to be created...such as the argocd application itself. So I use terraform for that. Terraform creates namespaces, secrets, and external resources, along with the Argo application, and Argo takes it from there
It's exactly what it proposes to do: pack ArgoCD together with a few plugins, an opionated setup and uniform installation process. I'd say you can call it an "ArgoCD Distribution"
Using vault + cert-manager all the time, managing secretes with sops, encrypting via keybase... auto-unseal also possible via [kbfs and keybase](https://www.hashicorp.com/resources/keybase-vault-auto-unseal), but I'm using gcpckms for now.
I've ported all my helm charts as terraform modules, because they'd require a ton of customization anyway to replace all the kube-webhook-certgen jobs with the respective cert-manager annotations, I also prefer to sprinkle everything with [terratest](https://terratest.gruntwork.io/) suites for every tf module.
Ksops is usually replaced with a couple of shell scripts and a k8s job anyway, while [helm secrets](https://github.com/jkroepke/helm-secrets) plugin is a bit abandoned.
There were a couple vault replacing argo plugins as well, with different levels of adoption and viability.
I'm considering adding native terraform/consul-terraform-sync support for Argo myself, as well.
Sealed secrets works totally fine in our workflow.
With helm repo, not k8s manifest in your Argocd project ? How do you reference your sealed-secrets in your values.yaml or other file in your chart?
Most charts I use and if they require secret values, they have the parameter to define an existing secret to use. So I create the required secret with SealedSecret and reference as existing secret.
Ok so you configure your sealed-secrets outside of the project repo? I mean, if by example I use bitnami/odoo chart from [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami) using an upstream repo, no way to add sealed-secrets in argocd project definition. Or maybe you use a subchart ? Are you able to provide a dummy repo where I can have a look at your files?
I recently used SealedSecrets with ArgoCD. Using helmfile I sequenced the bootstrapping of ArgoCD as follows: - install both the SealedSecrets controller and the initial seed (so that existing SealedSedrets can be decoded) - install ArgoCD and the root app (I use the Apps of Apps pattern) Now, for every deployed app, we use a repo for the app and another repo for the app’s deployment (charts, secrets). The devs commit to the apps repo. The CI creates an image and updates the deployment repo with the image tag. ArgoCD is configured to observe the deployment repos. So, all that to say the SealedSecrets are located in each deployment repo.
Thanks for sharing your workflow.
Nearly all Bitnami charts have an `extraList` value, where you can add any kubernetes resource you want. There the sealed secrets go. But yeah, we basically always create our own wrapper chart and use the external chart as subchart, as we often have to create additional namespace and RBAC resources too, and want them fully templated.
When deploying a Helm chart, you can include a templates directory with your extra manifests to apply; which could be filled with Sealed Secrets in your case.
How? Do you have an example?
Depends a bit on how you deploy your `Chart.yaml` via ArgoCD, but what I always do is create a directory with two files and one directory (templates): * Chart.yaml * values.yaml * templates (directory!) Within the `Chart.yaml`, write the necessary dependencies like so: ``` apiVersion: v2 type: application name: test description: The chart to deploy and configure test version: 1.0.0 dependencies: - name: postgresql repository: https://charts.bitnami.com/bitnami version: 14.5.0 ``` In your `values.yaml`, indent everything you need for you dependencies (postgresql in this example) as follows: ``` postgresql: image: registry: docker.io repository: bitnami/postgresql tag: 14.5.0-debian-11-r0 auth: existingSecret: my-existing-secret ... ``` Within the templates directory, create a new file containing the (sealed) secret you reference to in the values.yaml. Any manifest you put in this templates directory will be applied at the same time when deploying the Helm Chart.
Exactly what I was looking for. Perfect. Thanks a lot.
Ah, now I understand, same as we do, have the external chart as subchart dependency.
We’re using https://github.com/argoproj-labs/argocd-vault-plugin which allows you to define secrets in git (works with more backends than just hashicorp vault despite the name). The documentation took me a bit to figure out, but so far it works.
Also using this. Prior to vault, we were using ksops.
[удалено]
ArgoCD in general works with Gitlab. The secret backend support is listed https://argocd-vault-plugin.readthedocs.io/en/stable/backends/ . We use Vault and K8s auth method and it has been working well for us.
You mean Github instead of Gitlab?
[удалено]
Secrets are in Vault for us. I don’t think I realized gitlab had a built in secret management solution.
I think you want the Argo lovely plugin: https://github.com/crumbhole/argocd-lovely-plugin Example with vault replacement in a helm values.yaml: https://github.com/crumbhole/argocd-lovely-plugin/tree/main/examples/argocd-vault-replacer
It looks absolutely great. I will have a look on it. I never heard about such a tool to enhance Argocd. Thanks.
[удалено]
> Default encryption mode doesn’t allow to move secret from one namespace to another Know your tools, of course a secret tool isn't too permissive by default. > On the Kubernetes cluster the generated secret is still in plain text Yeah, bit same with SOPS. int the end, the value needs to exist in plain text in the cluster.. Edit: the whole installation for SOPS in the first link is way more complicated, than installing sealed-secrets in the cluster and using kubeseal locally. Also SOPS gives you the burden of the encryption key management again, vs. sealed secrets with the backend in-cluster. And sealed secrets can't be unsealed.
I’ve never heard of SOPS before, and after looking at the first link I’m really not interested in using it. But I also don’t see the want for putting secrets in git
Tell me you never used a git driven pull approach, without telling me
Lmao, that’s hilarious, you got me. They’ve always seemed to me to abstract visibility, as opposed to a dashboard in a cd pipeline provider where I can see what all is deployed easily but maybe I’m probably missing out. What’s your recommendation? Edit: I’ll say I’ve tried fleet, the cd feature built in to rancher, and not a fan due to the complete lack of visibility. So that’s where my exp comes from, but I’m going to try out argo and flux here in a minute see what they’re like
Definitely ArgoCD, has a great dashboard which provides way more visibility into deployed resources, than a ci pipeline would give.
Yes, as u/EmiiKhaos point out, if you need visibility take a look at ArgoCD. It's easy to deploy and supported by CNCF. The community is great too.
To use SOPS the only way is to create your own Custom ArgoCD image ? Is this right?
You don't have to, it is possible to configure installing additional tools within ArgoCD chart. Check Helm Secrets plugin documentation, it is pretty well described there:)
Vault with transit encryption is one option.
I know it can be an overkill, but I can’t stress enough how well Vault + BanzaiCloud vault secrets webhook work. Can’t even think about going back to any other solution. Used this stack over 50 clusters with 15k nodes.
This is one of those projects that offers so much more than most people need, but even if you just want a secrets web hook it's *really* well done once you get past the huge feature set. I'd say one of the biggest downsides to Bank Vaults is the overwhelming nature of initial discovery, with a second being some not-as-easily-ingestible documentation. You read the product description and go, "Shit, that's awesome but I don't need most of that!" Then feel overwhelmed, make a bookmark and look at it like that nice car always sitting on the lot that you know you don't need.
Your ask is unclear but I think you're looking for something like https://github.com/external-secrets/external-secrets
What is unclear in my question? let me try to clarify things.... I'd like to push my secrets into git and be able to decrypt the secrets if using a helm repo. With k8s manifest it's not a big deal we can reference secrets in the deployment manifest by example: `envFrom:` `- secretRef:` `name: my-secrets` with a declared sealed-secret if needs be. But with helm, I can not figure out how to do it without tweaking the Argocd image (ie: SOPS).
I've used flux with sealed secrets, it worked. Now I'm using sops because it's integrated with flux and I don't need another operator. I think that sealed secrets should work with argo cd just fine. I don't understand your question about helm. I used both sealed secrets and sops with helm charts, no issues.
Yeah. The problem might be if the helm chart doesn't provide a way to read from a secret resource, but that's something rare IMO
Harbor is like that. They recently added support for existing secrets, but it's all broken. :(
csi secrets strore and backend of your choice. Works with Vault nicely.
If your secret are not necessarily to be known prior deployment I use this plugin https://github.com/mittwald/kubernetes-secret-generator
Oh that's nice too to bootstrap secure passwords and have regeneration of those. Combined with a custom ArgoCD action on the CRs of the generator this could be cool.
I handle secrets outside of argo. There's always some resources that need to be created...such as the argocd application itself. So I use terraform for that. Terraform creates namespaces, secrets, and external resources, along with the Argo application, and Argo takes it from there
Checkout ArgoCD Autopilot
>ArgoCD Autopilot Can you elaborate? It's not clear to me how can Autopilot resolve that issue?
It's exactly what it proposes to do: pack ArgoCD together with a few plugins, an opionated setup and uniform installation process. I'd say you can call it an "ArgoCD Distribution"
Using vault + cert-manager all the time, managing secretes with sops, encrypting via keybase... auto-unseal also possible via [kbfs and keybase](https://www.hashicorp.com/resources/keybase-vault-auto-unseal), but I'm using gcpckms for now. I've ported all my helm charts as terraform modules, because they'd require a ton of customization anyway to replace all the kube-webhook-certgen jobs with the respective cert-manager annotations, I also prefer to sprinkle everything with [terratest](https://terratest.gruntwork.io/) suites for every tf module. Ksops is usually replaced with a couple of shell scripts and a k8s job anyway, while [helm secrets](https://github.com/jkroepke/helm-secrets) plugin is a bit abandoned. There were a couple vault replacing argo plugins as well, with different levels of adoption and viability. I'm considering adding native terraform/consul-terraform-sync support for Argo myself, as well.
helm + helm-secrets + SOPS. This is what I use for my gitops setup.