[multiple] Add OOMKill log

[danpawlik]

It might happen that there would be an OOMKill of the container.
We would like to be aware about that issue.

[averdagu]

Looks good to me. Did we found any occurrence of this?

[danpawlik]

@averdagu in some cases, we see that tempest tests can not pass, where we wondering what can be root cause of it. Maybe it is oomkill, maybe bad health check? Something to discover. We just want to more aware whats going on.

[averdagu]

/lgtm

[openshift-ci]

New changes are detected. LGTM label has been removed.

[fmount]

I'm not against the idea of being aware of any OOMKill that might happen in the cluster, but I think just getting the events is not enough to have a clear idea of the nodes status.
Instead considering to stores the output of oc describe nodes that contains more data about Conditions and Capacity and can drive through a resolution of the problem.

[fmount]

--all-namespaces in a large environment might take a bit, and I'm not sure you need to check which Pod is oomikilled all over the cluster.
Did you try in a DC or DZ environment?
Perhaps you can consider running the regular openshift must gather along with the openstack one if you're looking for this kind of data. [1]

[1] e.g.

oc adm must-gather \
   --image-stream=openshift/must-gather \
   --image=quay.io/openstack-k8s-operators/openstack-must-gather:latest

If you get OOMKill all over the cluster I expect must-gather does not work and you've probably lost access to the OCP API at this point.

[danpawlik]

How in quick way, I would be able to get information that there was an OOMKill without downloading 1GB logs and make grep? Does must gather provide a dedicated file with OOMKIll log?

[danpawlik]

@fmount I can add the --field-selector type=Warning - why not. I see it is suggested by AI, where Stackoverflow have just simplier way :)

You can get the json and use jq to filter the "Reason" you're interested to, but to me that kind of post- processing should happen later, as an additional task, on the gathered file.

still, I need to download a file, use jq or other tool, grep instead of take a look into a dedicated file. Such way we can consider if OOMKill would be often - something to decide in the future.

Instead considering to stores the output of oc describe nodes that contains more data about Conditions and Capacity and can drive through a resolution of the problem.

So far, we just want to see OOMKill. We can add other info in other PR if needed.

[fmount]

@fmount I can add the --field-selector type=Warning - why not. I see it is suggested by AI, where Stackoverflow have just simplier way :)

What? I don't think the point is AI vs stackoverflow vs #anythingelse. the point is that sorting by time doesn't make sense to me. So please stop with that.

You can get the json and use jq to filter the "Reason" you're interested to, but to me that kind of post- processing should happen later, as an additional task, on the gathered file.

still, I need to download a file, use jq or other tool, grep instead of take a look into a dedicated file. Such way we can consider if OOMKill would be often - something to decide in the future.

-o json | jq is what I meant.

Instead considering to stores the output of oc describe nodes that contains more data about Conditions and Capacity and can drive through a resolution of the problem.

So far, we just want to see OOMKill. We can add other info in other PR if needed.

Still I don't think that a query to all-namespaces is the right way. If your etcd, or apiserver or any other kubernetes infra pod gets oomkilled, these tasks are useless.

Feel free to move forward w/ this, but I'm -1 with the idea, believe I'm an AI or not.

[danpawlik]

@fmount I can add the --field-selector type=Warning - why not. I see it is suggested by AI, where Stackoverflow have just simplier way :)

What? I don't think the point is AI vs stackoverflow vs #anythingelse. the point is that sorting by time doesn't make sense to me. So please stop with that.

You can get the json and use jq to filter the "Reason" you're interested to, but to me that kind of post- processing should happen later, as an additional task, on the gathered file.

still, I need to download a file, use jq or other tool, grep instead of take a look into a dedicated file. Such way we can consider if OOMKill would be often - something to decide in the future.

-o json | jq is what I meant.

Instead considering to stores the output of oc describe nodes that contains more data about Conditions and Capacity and can drive through a resolution of the problem.

So far, we just want to see OOMKill. We can add other info in other PR if needed.

Still I don't think that a query to all-namespaces is the right way. If your etcd, or apiserver or any other kubernetes infra pod gets oomkilled, these tasks are useless.

Feel free to move forward w/ this, but I'm -1 with the idea, believe I'm an AI or not.

so you just want to collect openstack namespace OOMKill instead of all namespaces, so I need to create new task in other place where I will collect those info. Make sense

[danpawlik]

we have an agreement with @fmount that collecting cluster logs/state info should be in other place than os_must_gather role or at least separate task file.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign evallesp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment