Sync conflict between multiclusterhub-operator and opp-policy ArgoCD Application

[lpanza]

There is a sync conflict between these two objects since the multiclusterhub-operator tries to add the "acm" ConsolePlugin while the "opp-policy" Application in ArgoCD tries to leave just the "odf-console" one

Here is the ArgoCD Application's Manifest for reference:

---
apiVersion: operator.openshift.io/v1
kind: Console
metadata:
  labels:
    argocd.argoproj.io/instance: opp-policy
  name: cluster
spec:
  plugins:
    - odf-console

Here are the operator logs:

1.6750967510576084e+09	INFO	Reconciling MultiClusterHub	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967510667846e+09	INFO	controller_multiclusterhub	Found image overrides from environment variables set by operand image prefix
1.6750967516626077e+09	INFO	Reconciling MultiClusterHub	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517205005e+09	INFO	Ensuring namespace: multicluster-engine	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517205572e+09	INFO	Ensuring operator group exists in ns: multicluster-engine	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517205713e+09	INFO	Ensuring OLM multicluster-engine/multicluster-engine subscription	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.675096751740915e+09	INFO	Lets preserve that namespace!	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757106e+09	INFO	MCH Current Version: '2.6.3', Desired Version: '2.6.3'	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757683e+09	INFO	Checking if current version is 2.6	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757752e+09	INFO	Checking if desired version is 2.6	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757795e+09	INFO	isACM26x=true, isUpgrade=false	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757983e+09	INFO	MCE current version '2.1.4', MCE desired version '2.1.4'	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517758012e+09	INFO	Checking MCE current version	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.675096751775806e+09	INFO	MultiClusterEngine version '2.1.4' does not need to be updated	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.675096751775812e+09	INFO	using trust bundle configmap open-cluster-management/trusted-ca-bundle	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967526264307e+09	INFO	Dynamic plugins are supported. Adding ACM plugin to console	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967526264775e+09	INFO	Ready to add plugin	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967526500366e+09	INFO	Added acm consoleplugin to console	{"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}

[mbaldessari]

@day0hero @ipbabble Can you guys take a look?

[mbaldessari]

Thanks for the report @lpanza . For now this has been somewhat worked around via 0e24e92
I suspect that once we upgrade to ArgoCD 2.5 we can just use server side apply for this plugin:

diff --git a/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml b/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml
index ccbfb14..a98cc29 100644
--- a/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml
+++ b/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml
@@ -11,6 +11,8 @@ apiVersion: operator.openshift.io/v1
 kind: Console
 metadata:
   name: cluster
+  annotations:
+    argocd.argoproj.io/sync-options: ServerSideApply=true
 spec:
   plugins:
   - odf-console

I am leaving this open just to track the above.