Our Kubernetes platform test suite creates namespaces with their corresponding contents, then deletes everything during cleanup. We noticed a strange problem: namespace deletion would sometimes get stuck indefinitely. The root cause was surprising — we had to clear the contents before deleting the namespace! We also learned that getting stuck isn’t the only issue that can occur if we don’t do this.

This was counterintuitive because Kubernetes automatically deletes a namespace’s contents when you delete the namespace itself. So why does the order matter? Here’s what can go wrong if you delete the namespace first (or simultaneously with its contents):

  1. When you initiate namespace deletion, it enters the “Terminating” phase. During this phase, no new resources can be created inside the namespace. The namespace only disappears once all its contents are gone.

  2. When an existing resource has a finalizer, its controller may need to create new resources as part of the finalization process. However, it can’t do this if the namespace is already in the Terminating phase.

    Note: Finalizers are markers that tell Kubernetes to perform certain cleanup operations before actually deleting a resource. They’re commonly used to clean up external dependencies.

    Depending on the resource type, this can result in unintended behavior, such as lost information, or finalization getting stuck indefinitely.

  3. Since one or more resources can’t complete finalization, the namespace itself can’t finish terminating!

Example: how information gets lost

When deleting an Ingress, the ingress controller may need to create Events. If there’s an external problem (like a cloud API returning an error), you’d expect to see error events on the Ingress resource. However, you won’t — the controller can’t create Events in a terminating namespace, so this diagnostic information is lost.

Example: how deletion gets stuck

Some resources’ finalization processes involve creating new resources. For example, when deleting a Crossplane resource, Crossplane may create a ProviderConfigUsage resource for accounting purposes. It can’t do this in a terminating namespace, causing finalization to hang indefinitely.

Example: how permission problems occur

Our Kubernetes platform automatically creates RoleBindings in new namespaces to grant permissions to relevant groups. Some end-to-end tests rely on these permissions to delete resources during cleanup. However, if the RoleBindings are deleted too early (as part of namespace deletion), the tests’ subsequent resource deletion requests will fail with permission errors.

Fixing a stuck namespace

If you’re already stuck with a namespace that won’t delete, the only way out is to manually remove finalizers:

kubectl patch <resource> <name> -n <namespace> -p '{"metadata":{"finalizers":[]}}' --type=merge

Warning: This bypasses cleanup logic and may leave orphaned resources in external systems. Prevention is always better than the cure.

Best practice: delete contents first

To avoid this issue entirely, always delete namespace contents before deleting the namespace itself!