Make the PushJobAgent compatible with more CEs (HTCondor? SSH?)

[aldbr]

So that it can be reused for HPCs with no ARC endpoint.

[fstagni]

Who is requesting this?

[aldbr]

@hmiyake would potentially reuse the PushJobAgent using ssh if I remember correctly.

[hmiyake]

Sorry for late reply.

We have several use cases for pilot submission to non-grid CE.

One is local HTCondor site running no HTCondorCE, then we needed to deploy DIRAC to submit pilots. That site does not open ssh for massive job execution.

The other is sites having a local batch system and opening ssh. Currently we submit pilots via SSHComputingElement.

For both use cases, those sites have no special restriction in outbound connectivity. I've read
https://dirac.diracgrid.org/en/integration/AdministratorGuide/Resources/supercomputers.html#no-outbound-connectivity
and wonder if we can use PushJobAgent in our use cases... Or is there any advantage in scalability compared with, say, SSHComputingElement?

As a background of our story, during DUW I consulted about HTCondor site, for the case that we migrate to DiracX and cannot deploy external DiracX on the site.

[fstagni]

SSHComputingElement (and in general the usual "pull" pilot mode) is more scalable than PushJobAgent, which is a solution to be used mostly if the worker nodes are not open to outside network.

[aldbr]

That site does not open ssh for massive job execution.

Out of curiosity, could you clarify this point please?
Do you mean that the site is protected through a VPN and you cannot connect through SSH via DIRAC?

[hmiyake]

Thank you for clarification!

[hmiyake]

Regarding as the specific site, it opens ssh port for maintenance but the site admin did not like the port is heavily accessed, at that time.

So we did not use SSHComputingElement but LocalComputingElement.

[aldbr]

I am closing that issue because:

• it's not the PushJobAgent that would need to be adapted, it's the CEs (would need to support jobs with their inputs/outputs)
• we won't do that in DIRAC but of course it will be taken into account in DiracX

As a side note: PushJobAgent in v9 should be much more scalable because it's stateless but the job workflows need to work without any external connectivity to be used.