[prefect-aws] ECS Worker - excessive task definition re-registration when multiple containers are used

[bdalpe]

Bug summary

Problem

When using custom job configurations with multiple container definitions, the Prefect ECS worker re-registers task definitions even when they are identical. This occurs when the essential field is not explicitly set on container definitions.

Root Cause

AWS ECS sets essential to True for containers if it's not specified at Task Definition registration time.

When comparing task definitions:

1. The generated task definition from Prefect may not have essential set on all containers
2. The task definition retrieved from AWS has essential values set by AWS defaults
3. The comparison logic in _task_definitions_equal() wasn't normalizing the essential field correctly
4. This mismatch causes the comparison to fail, leading to unnecessary re-registration

Additional Issue

essential: False is removed from container definitions

The _drop_empty_keys_from_dict function removes boolean False values from task definitions, causing essential: False to be stripped from container definitions before registration with AWS. AWS then defaults essential to True, changing the intended behavior.

Version info

prefect-aws<=0.7.2

Additional context

Example Configuration

This issue affects customers using custom job configurations like:

task_definition:
  containerDefinitions:
    - name: "prefect"
      image: "{{ image }}"
    - name: "datadog-agent"
      image: "public.ecr.aws/datadog/agent:latest"