Add `atlasopenmagic` DID finder by ponyisi · Pull Request #1292 · ssl-hep/ServiceX

ponyisi · 2026-01-31T06:39:37Z

Adds an atlasopenmagic DID finder to the backend (will need a frontend update for proper support in the client). Gives the list of URLs reported by the atlasopenmagic package for a given release/dataset[/skim] combination.

for more information, see https://pre-commit.ci

codecov · 2026-01-31T06:39:52Z

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.13%. Comparing base (e7748f1) to head (bb8eedb).
⚠️ Report is 16 commits behind head on develop.

Files with missing lines	Patch %	Lines
...sopenmagic/src/did_finder_atlasopenmagic/celery.py	93.54%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1292      +/-   ##
===========================================
+ Coverage    87.07%   87.13%   +0.05%     
===========================================
  Files           96       97       +1     
  Lines         3389     3420      +31     
  Branches       399      406       +7     
===========================================
+ Hits          2951     2980      +29     
- Misses         367      368       +1     
- Partials        71       72       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

BenGalewsky · 2026-02-03T16:08:17Z

When I deploy this I get:

/opt/servicex/.venv/lib/python3.14/site-packages/billiard/pool.py:1855: SyntaxWarning: 'return' in a 'finally' block
  return self._send_ack(
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:349: SyntaxWarning: 'return' in a 'finally' block
  return f
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:351: SyntaxWarning: 'return' in a 'finally' block
  return self._get_more_data(ov, maxsize)
/opt/servicex/.venv/lib/python3.14/site-packages/celery/concurrency/asynpool.py:400: SyntaxWarning: 'return' in a 'finally' block
  return remove(fd)
Traceback (most recent call last):
  File "/opt/servicex/.venv/bin/celery", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
             ~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/celery.py", line 236, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/base.py", line 135, in caller
    return f(ctx, *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/worker.py", line 348, in worker
    worker = app.Worker(
        hostname=hostname, pool_cls=pool_cls, loglevel=loglevel,
    ...<4 lines>...
        quiet=ctx.obj.quiet,
        **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 98, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 131, in setup_instance
    self.on_init_blueprint()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 116, in on_init_blueprint
    self._custom_logging = self.setup_logging()
                           ~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 180, in setup_logging
    return self.app.log.setup(
           ~~~~~~~~~~~~~~~~~~^
        self.loglevel, self.logfile,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        redirect_stdouts=False, colorize=colorize, hostname=self.hostname,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 63, in setup
    handled = self.setup_logging_subsystem(
        loglevel, logfile, colorize=colorize, hostname=hostname,
    )
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 118, in setup_logging_subsystem
    get_multiprocessing_logger(),
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/utils/log.py", line 270, in get_multiprocessing_logger
    return util.get_logger()
           ~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/billiard/util.py", line 131, in get_logger
    logging._acquireLock()
    ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'logging' has no attribute '_acquireLock'

BenGalewsky · 2026-02-03T16:09:15Z

I did update the deployment template to apply resource limit and request settings as we've done to all other pods

BenGalewsky · 2026-02-03T16:56:43Z

Can you provide suggestions on how to test this?

MattShirley · 2026-02-03T18:41:14Z

+    enabled: true
+    image: sslhep/servicex-did-finder-atlasopenmagic
+    pullPolicy: Always
+    tag: develop


This tag does not currently exist: https://hub.docker.com/r/sslhep/servicex-did-finder-atlasopenmagic/tags

It will once PR is accepted

MattShirley · 2026-02-03T18:50:55Z

When I deploy this I get:

/opt/servicex/.venv/lib/python3.14/site-packages/billiard/pool.py:1855: SyntaxWarning: 'return' in a 'finally' block
  return self._send_ack(
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:349: SyntaxWarning: 'return' in a 'finally' block
  return f
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:351: SyntaxWarning: 'return' in a 'finally' block
  return self._get_more_data(ov, maxsize)
/opt/servicex/.venv/lib/python3.14/site-packages/celery/concurrency/asynpool.py:400: SyntaxWarning: 'return' in a 'finally' block
  return remove(fd)
Traceback (most recent call last):
  File "/opt/servicex/.venv/bin/celery", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
             ~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/celery.py", line 236, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/base.py", line 135, in caller
    return f(ctx, *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/worker.py", line 348, in worker
    worker = app.Worker(
        hostname=hostname, pool_cls=pool_cls, loglevel=loglevel,
    ...<4 lines>...
        quiet=ctx.obj.quiet,
        **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 98, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 131, in setup_instance
    self.on_init_blueprint()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 116, in on_init_blueprint
    self._custom_logging = self.setup_logging()
                           ~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 180, in setup_logging
    return self.app.log.setup(
           ~~~~~~~~~~~~~~~~~~^
        self.loglevel, self.logfile,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        redirect_stdouts=False, colorize=colorize, hostname=self.hostname,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 63, in setup
    handled = self.setup_logging_subsystem(
        loglevel, logfile, colorize=colorize, hostname=hostname,
    )
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 118, in setup_logging_subsystem
    get_multiprocessing_logger(),
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/utils/log.py", line 270, in get_multiprocessing_logger
    return util.get_logger()
           ~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/billiard/util.py", line 131, in get_logger
    logging._acquireLock()
    ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'logging' has no attribute '_acquireLock'

I also get this error when setting the image tag to atlasopenmagic.

ponyisi · 2026-02-09T00:54:53Z

@BenGalewsky @MattShirley logging issues should be fixed by latest push

ponyisi · 2026-03-10T03:07:25Z

@MattShirley @BenGalewsky reminder on this one

BenGalewsky

Would be nice to know how to test this....

ponyisi and others added 2 commits January 31, 2026 00:37

Add atlasopenmagic DID finder

872efbc

[pre-commit.ci] auto fixes from pre-commit.com hooks

f5c48de

for more information, see https://pre-commit.ci

ponyisi added 4 commits January 31, 2026 00:49

CI python versions

e3b7649

CI try again

4a6d7b4

Poetry update

e0781e3

Update versions

059f5f4

ponyisi mentioned this pull request Jan 31, 2026

ATLAS OpenMagic DID Finder #1281

Closed

ponyisi requested review from ArturU043, BenGalewsky, MattShirley and gordonwatts January 31, 2026 07:28

Add resource settings to atlas open magic did finder deployment

170d1aa

MattShirley reviewed Feb 3, 2026

View reviewed changes

Comment thread .github/workflows/ci_servicex.yaml

Update version of billiard to fix logging issues

9e4c491

Merge branch 'develop' into atlasopenmagic

bb8eedb

ponyisi requested a review from MattShirley March 6, 2026 18:57

BenGalewsky approved these changes Mar 10, 2026

View reviewed changes

ponyisi merged commit 662473e into develop Mar 10, 2026
84 checks passed

ponyisi deleted the atlasopenmagic branch March 10, 2026 22:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `atlasopenmagic` DID finder#1292

Add `atlasopenmagic` DID finder#1292
ponyisi merged 9 commits intodevelopfrom
atlasopenmagic

ponyisi commented Jan 31, 2026 •

edited by BenGalewsky

Loading

Uh oh!

codecov Bot commented Jan 31, 2026 •

edited

Loading

Uh oh!

BenGalewsky commented Feb 3, 2026

Uh oh!

BenGalewsky commented Feb 3, 2026

Uh oh!

BenGalewsky commented Feb 3, 2026

Uh oh!

MattShirley Feb 3, 2026

Uh oh!

ponyisi Feb 9, 2026

Uh oh!

Uh oh!

MattShirley commented Feb 3, 2026 •

edited

Loading

Uh oh!

ponyisi commented Feb 9, 2026

Uh oh!

ponyisi commented Mar 10, 2026

Uh oh!

BenGalewsky left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ponyisi commented Jan 31, 2026 • edited by BenGalewsky Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

BenGalewsky commented Feb 3, 2026

Uh oh!

BenGalewsky commented Feb 3, 2026

Uh oh!

BenGalewsky commented Feb 3, 2026

Uh oh!

MattShirley Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

ponyisi Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MattShirley commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ponyisi commented Feb 9, 2026

Uh oh!

ponyisi commented Mar 10, 2026

Uh oh!

BenGalewsky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ponyisi commented Jan 31, 2026 •

edited by BenGalewsky

Loading

codecov Bot commented Jan 31, 2026 •

edited

Loading

MattShirley commented Feb 3, 2026 •

edited

Loading