Trainer is always in waiting state

[PankajJain5889]

I just started using this after initial setup done right I am able to see agent output and NPc's but the trainer is always in waiting state

[harbarex]

@PankajJain5889 did you find a solution for this?

[PankajJain5889]

Couldn't find a viable solution but will look in it this weekend, also what version of python and tensorflow-gpu are you using

[harbarex]

Try installing the latest version of cudnn. And change the value of TRAINER_MEMORY_ALLOCATION to 0.4. I am using the following configuration, and the code worked for me:

tensorflow-gpu: 1.15.0
keras: 2.2.4
cuda: 10.0.1
cudnn: 7.6.4
Python: 3.7.5

[PankajJain5889]

Can you tell me what graphic card are you working with I am just unsure that 10.0.1 Cuda would work with mine cause as per my online research 9.0 is stable, please suggest

[harbarex]

I am using the NVIDIA Quadro P4000 - 8GB Memory

[FHof]

I've noticed that the ConsoleStats hides a python keras Exception and found a way to avoid it: #9