Greatly improved Unknowns handling

Author: Patrick J. McNerthney

README.md

@@ -56,17 +56,18 @@ spec:
   package: ghcr.io/fortra/function-pythonic:v0.0.3
 ```
 
-## Managed Resource Dependencies
+## Composed Resource Dependencies
 
-function-pythonic automatically handles dependencies between managed resources.
+function-pythonic automatically handles dependencies between composed resources.
 
 Just compose everything as if it is immediately created and the framework will delay
 the creation of any resources which depend on other resources which do not exist yet.
 In other words, it accomplishes what [function-sequencer](https://github.com/crossplane-contrib/function-sequencer)
 provides, but it automatically detects the dependencies.
 
-If a resource has been composed and a dependency no longer exists due to some unexpected
-condition, the observed value for that field will automatically be used.
+If a resource has been created and a dependency no longer exists due to some unexpected
+condition, the composition will be terminated or the observed value for that field will
+be used, depending on the `unknownsFatal` settings.
 
 Take the following example:
 ```yaml
@@ -83,9 +84,13 @@ subnet.spec.forProvider.cidrBlock = '10.0.0.0/20'
 If the Subnet does not yet exist, the framework will detect if the vpcId set
 in the Subnet is unknown, and will delay the creation of the subnet.
 
-Once the Subnet has been created, if for some mysterious reason the vpcId passed
-to the Subnet is unknown, the framework will automatically use the vpcId in the
-observed Subnet.
+Once the Subnet has been created, if for some unexpected reason the vpcId passed
+to the Subnet is unknown, the framework will detect it and either terminate
+the Composite composition or use the vpcId in the observed Subnet. The default
+action taken is to fast fail by terminating the composition. This can be
+overridden for all composed resource by setting the Composite `self.unknownsFatal` field
+to False, or at the individual composed resource level by setting the
+`Resource.unknownsFatal` field to False.
 
 ## Pythonic access of Protobuf Messages
 
@@ -186,37 +191,39 @@ The BaseComposite also provides access to the following Crossplane Function leve
 | self.response | Low level direct access to the RunFunctionResponse message |
 | self.logger | Python logger to log messages to the running function stdout |
 | self.ttl | Get or set the response TTL, in seconds |
-| self.autoReady | Perform auto ready processing after the compose method returns, default True |
 | self.credentials | The request credentials |
 | self.context | The response context, initialized from the request context |
 | self.environment | The response environment, initialized from the request context environment |
 | self.requireds | Request and read additional local Kubernetes resources |
-| self.resources | Define and process managed resources |
+| self.resources | Define and process composed resources |
 | self.results | Returned results on the Composite and optionally on the Claim |
+| self.unknownsFatal | Terminate the composition if already created resources are assigned unknown values, default True |
+| self.autoReady | Perform auto ready processing after the compose method returns, default True |
 
-### Managed Resources
+### Composed Resources
 
-Creating and accessing managed resources is performed using the `BaseComposite.resources` field.
-`BaseComposite.resources` is a dictionary of the managed resources whose key is the composition
+Creating and accessing composed resources is performed using the `BaseComposite.resources` field.
+`BaseComposite.resources` is a dictionary of the composed resources whose key is the composition
 resource name. The value returned when getting a resource from BaseComposite is the following
 Resource class:
 
 | Field | Description |
 | ----- | ----------- |
 | Resource(apiVersion,kind,namespace,name) | Reset the resource and set the optional parameters |
-| Resource.name | The composition resource name of the managed resource |
-| Resource.observed | Low level direct access to the observed managed resource |
-| Resource.desired | Low level direct access to the desired managed resource |
-| Resource.apiVersion | The managed resource apiVersion |
-| Resource.kind | The managed resource kind |
-| Resource.externalName | The managed resource external name |
-| Resource.metadata | The managed resource desired metadata |
+| Resource.name | The composition resource name of the composed resource |
+| Resource.observed | Low level direct access to the observed composed resource |
+| Resource.desired | Low level direct access to the desired composed resource |
+| Resource.apiVersion | The composed resource apiVersion |
+| Resource.kind | The composed resource kind |
+| Resource.externalName | The composed resource external name |
+| Resource.metadata | The composed resource desired metadata |
 | Resource.spec | The resource spec |
 | Resource.data | The resource data |
 | Resource.status | The resource status |
 | Resource.conditions | The resource conditions |
 | Resource.connection | The resource connection details |
 | Resource.ready | The resource ready state |
+| Resource.unknownsFatal | Terminate the composition if this resource has been created and is assigned unknown values, default is Composite.unknownsFatal |
 
 ### Required Resources (AKA Extra Resources)
 

crossplane/pythonic/composite.py

@@ -10,8 +10,8 @@
 
 class BaseComposite:
     def __init__(self, request, response, logger):
-        self.request = protobuf.Message(None, None, request.DESCRIPTOR, request, 'Function Request')
-        self.response = protobuf.Message(None, None, response.DESCRIPTOR, response)
+        self.request = protobuf.Message(None, 'request', request.DESCRIPTOR, request, 'Function Request')
+        self.response = protobuf.Message(None, 'response', response.DESCRIPTOR, response)
         self.logger = logger
         self.autoReady = True
         self.credentials = Credentials(self.request)
@@ -20,6 +20,7 @@ def __init__(self, request, response, logger):
         self.requireds = Requireds(self)
         self.resources = Resources(self)
         self.results = Results(self.response)
+        self.unknownsFatal = True
 
         observed = self.request.observed.composite
         desired = self.response.desired.composite
@@ -134,6 +135,7 @@ def __init__(self, composite, name):
         self.desired = desired.resource
         self.conditions = Conditions(observed)
         self.connection = Connection(observed)
+        self.unknownsFatal = None
 
     def __call__(self, apiVersion=_notset, kind=_notset, namespace=_notset, name=_notset):
         self.desired()
@@ -333,22 +335,38 @@ class Results:
     def __init__(self, response):
         self._results = response.results
 
-    def __call__(self, message=_notset, fatal=_notset, warning=_notset, reaoson=_notset, claim=_notset):
-        result = Result(self._results.add())
-        if fatal != _notset:
-            result.fatal = fatal
-        elif warning != _notset:
-            result.warning = warning
-        if message != _notset:
-            result.message = message
+    def info(self, message, reason=_notset, claim=_notset):
+        result = Result(self._results.append())
+        result.info = True
+        result.message = message
+        if reason != _notset:
+            result.reason = reason
+        if claim != _notset:
+            result.claim = claim
+        return result
+
+    def warning(self, message, reason=_notset, claim=_notset):
+        result = Result(self._results.append())
+        result.warning = True
+        result.message = message
+        if reason != _notset:
+            result.reason = reason
+        if claim != _notset:
+            result.claim = claim
+        return result
+
+    def fatal(self, message, reason=_notset, claim=_notset):
+        result = Result(self._results.append())
+        result.fatal = True
+        result.message = message
         if reason != _notset:
             result.reason = reason
         if claim != _notset:
             result.claim = claim
         return result
 
     def __bool__(self):
-        return len(self._results) > 0
+        return len(self) > 0
 
     def __len__(self):
         len(self._results)
@@ -364,35 +382,47 @@ def __iter__(self):
 
 
 class Result:
-    def __init(self, result=None):
+    def __init__(self, result=None):
         self._result = result
 
     def __bool__(self):
         return self._result is not None
 
     @property
-    def fatal(self):
-        return bool(self) and self._result == fnv1.Severity.SEVERITY_FATAL
+    def info(self):
+        return bool(self) and self._result.severity == fnv1.Severity.SEVERITY_NORMAL
 
-    @fatal.setter
-    def fatal(self, fatal):
+    @info.setter
+    def info(self, info):
         if bool(self):
-            if fatal:
-                self._result = fnv1.Severity.SEVERITY_FATAL
+            if info:
+                self._result.severity = fnv1.Severity.SEVERITY_NORMAL
             else:
-                self._result = fnv1.Severity.SEVERITY_NORMAL
+                self._result.severity = fnv1.Severity.SEVERITY_UNSPECIFIED
 
     @property
     def warning(self):
-        return bool(self) and self._result == fnv1.Severity.SEVERITY_WARNING
+        return bool(self) and self._result.severity == fnv1.Severity.SEVERITY_WARNING
 
     @warning.setter
     def warning(self, warning):
         if bool(self):
             if warning:
-                self._result = fnv1.Severity.SEVERITY_WARNING
+                self._result.severity = fnv1.Severity.SEVERITY_WARNING
+            else:
+                self._result.severity = fnv1.Severity.SEVERITY_NORMAL
+
+    @property
+    def fatal(self):
+        return bool(self) and self._result.severity == fnv1.Severity.SEVERITY_FATAL
+
+    @fatal.setter
+    def fatal(self, fatal):
+        if bool(self):
+            if fatal:
+                self._result.severity = fnv1.Severity.SEVERITY_FATAL
             else:
-                self._result = fnv1.Severity.SEVERITY_NORMAL
+                self._result.severity = fnv1.Severity.SEVERITY_NORMAL
 
     @property
     def message(self):
@@ -565,22 +595,15 @@ def _find_condition(self, create=False):
             for condition in self._conditions._response.conditions:
                 if condition.type == self.type:
                     return condition
-        for observed in self._conditions._observed.resource.status.conditions:
-            if observed.type == self.type:
-                break
-        else:
-            observed = None
         if not create:
-            return observed
+            for observed in self._conditions._observed.resource.status.conditions:
+                if observed.type == self.type:
+                    return observed
+            return None
         if self._conditions._response is None:
             raise ValueError('Condition is read only')
         condition = fnv1.Condition()
         condition.type = self.type
-        if observed:
-            condition.status = observed.status
-            condition.reason = observed.reason
-            condition.message = observed.message
-            condition.target = observed.target
         return self._conditions._response.conditions.append(condition)
 
 

crossplane/pythonic/function.py

@@ -26,8 +26,9 @@
 class FunctionRunner(grpcv1.FunctionRunnerService):
     """A FunctionRunner handles gRPC RunFunctionRequests."""
 
-    def __init__(self):
+    def __init__(self, debug=False):
         """Create a new FunctionRunner."""
+        self.debug = debug
         self.logger = crossplane.function.logging.get_logger()
         self.clazzes = {}
 
@@ -117,12 +118,57 @@ async def RunFunction(
             logger.exception('Compose exception')
             return response
 
-        for name, resource in [entry for entry in composite.resources]:
-            if resource.desired._hasUnknowns:
+        unknownResources = []
+        warningResources = []
+        fatalResources = []
+        for name, resource in sorted(entry for entry in composite.resources):
+            unknowns = resource.desired._getUnknowns
+            if unknowns:
+                unknownResources.append(name)
+                warning = False
+                fatal = False
+                if resource.observed:
+                    warningResources.append(name)
+                    warning = True
+                    if resource.unknownsFatal:
+                        fatalResources.append(name)
+                        fatal = True
+                    elif resource.unknownsFatal is None and composite.unknownsFatal:
+                        fatalResources.append(name)
+                        fatal = True
+                if self.debug:
+                    for destination, source in sorted(unknowns.items()):
+                        destination = self._trimFullName('response', 'desired', destination)
+                        source = self._trimFullName('request', 'observed', source)
+                        if fatal:
+                            logger.error('Observed unknown assignment', destination=destination, source=source)
+                        elif warning:
+                            logger.warning('Observed unknown assignment', destination=destination, source=source)
+                        else:
+                            logger.debug('New unknown assignment', destination=destination, source=source)
                 if resource.observed:
                     resource.desired._patchUnknowns(resource.observed)
                 else:
                     del composite.resources[name]
+        if fatalResources:
+            if not self.debug:
+                logger.error('Observed Resources with unknown assignments', resources=fatalResources)
+            message = f"Observed Resources with unknown assignments: {', '.join(fatalResources)}"
+            composite.conditions.Unknowns(False, 'FatalUnknowns', message)
+            composite.results.fatal(message, 'FatalUnknowns')
+            return response
+        if warningResources:
+            if not self.debug:
+                logger.warning('Observed Resources with unknown assignments', resources=fatalResources)
+            message = f"Observed Resources with unknown assignments: {', '.join(warningResources)}"
+            composite.conditions.Unknowns(False, 'ObservedUnknowns', message)
+            composite.results.warning(message, 'ObservedUnknowns')
+        elif unknownResources:
+            if not self.debug:
+                logger.info('New Resources with unknown assignments', resources=unknownResources)
+            message = f"New Resources with unknown assignments: {', '.join(unknownResources)}"
+            composite.conditions.Unknowns(False, 'NewUnknowns', message)
+            composite.results.info(message, 'NewUnknowns')
 
         if composite.autoReady:
             for name, resource in composite.resources:
@@ -133,6 +179,16 @@ async def RunFunction(
         logger.debug('Returning')
         return response
 
+    def _trimFullName(self, message, state, name):
+        name = name.split('.')
+        ix = 0
+        for value in (message, state, 'resources', None, 'resource'):
+            if value and ix < len(value) and name[ix] == value:
+                del name[ix]
+            else:
+                ix += 1
+        return '.'.join(name)
+
 
 class Module:
     pass

crossplane/pythonic/main.py

@@ -66,7 +66,7 @@ def main():
 
     logging.configure(logging.Level.DEBUG if args.debug else logging.Level.INFO)
     runtime.serve(
-        function.FunctionRunner(),
+        function.FunctionRunner(args.debug),
         args.address,
         creds=runtime.load_credentials(args.tls_certs_dir),
         insecure=args.insecure,