Skip to content

Weaver tests are flaky on PyPy #5176

@xrmx

Description

@xrmx

They are skipped for now but would be nice to have them work reliably if possible. I guess we also have slow installing times due to missing wheels since failing runs took more than 10 minutes.

-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=53135): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=53135): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 63%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_and_check_raises_on_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
ERROR    opentelemetry.test.weaver_live_check:weaver_live_check.py:318 WeaverLiveCheck did not start: WeaverLiveCheck process exited unexpectedly (code -6), logs: 
E0424 15:12:55.913446    7686 ev_epoll1_linux.cc:371] (event_engine) Epoll1Poller:0x1d0db120 encountered epoll_wait error: Bad file descriptor

FAILED                                                                   [ 72%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_no_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=54733): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=54733): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 81%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_with_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=39173): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=39173): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 90%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_report_span_statistics 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=34619): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=34619): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [100%]

=================================== FAILURES ===================================
_________ TestSDKInitLiveCheck.test_end_and_check_raises_on_violations _________

self = <HTTPConnection(host='localhost', port=38821) at 0x1c237280>

    def _new_conn(self) -> socket.socket:
        """Establish a socket connection and set nodelay settings on it.
    
        :return: New socket connection.
        """
        try:
>           sock = connection.create_connection(
                (self._dns_host, self.port),
                self.timeout,
                source_address=self.source_address,
                socket_options=self.socket_options,
            )

.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/urllib3/connection.py:204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/urllib3/util/connection.py:85: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

address = ('localhost', 38821), timeout = 5, source_address = None
socket_options = [(6, 1, 1)]

.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/requests/adapters.py:678: ConnectionError

The above exception was the direct cause of the following exception:

self = <tests.test_weaver_live_check.TestSDKInitLiveCheck testMethod=test_end_and_check_raises_on_violations>

    def test_end_and_check_raises_on_violations(self):
        """end_and_check() raises LiveCheckError with the report attached."""
>       with WeaverLiveCheck(
            registry=_REGISTRY_DIR, policies_dir=_TESTDATA_DIR
        ) as weaver:

tests/opentelemetry-test-utils/tests/test_weaver_live_check.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:299: in __enter__
    return self.start()
tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:314: in start
    self._wait_for_ready()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <opentelemetry.test.weaver_live_check.WeaverLiveCheck object at 0x000000001cb5dc90>

    def _wait_for_ready(self) -> None:
        retry = Retry(
            total=10,
            backoff_factor=1,
            backoff_max=1,
            # Any non-2xx response from /health means weaver isn't ready yet.
            status_forcelist=list(range(300, 600)),
            raise_on_status=True,
            allowed_methods=["GET"],
        )
        session = Session()
        session.mount("http://", HTTPAdapter(max_retries=retry))
        try:
            session.get(
                f"http://localhost:{self._admin_port}/health", timeout=5
            )
        except Exception as exc:  # pylint: disable=broad-except
            if self._process is not None and self._process.poll() is not None:
>               raise RuntimeError(
                    f"WeaverLiveCheck process exited unexpectedly (code {self._process.returncode})"
                ) from exc
E               RuntimeError: WeaverLiveCheck process exited unexpectedly (code -6)

tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:342: RuntimeError
------------------------------ Captured log call -------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
ERROR    opentelemetry.test.weaver_live_check:weaver_live_check.py:318 WeaverLiveCheck did not start: WeaverLiveCheck process exited unexpectedly (code -6), logs: 
E0424 15:12:55.913446    7686 ev_epoll1_linux.cc:371] (event_engine) Epoll1Poller:0x1d0db120 encountered epoll_wait error: Bad file descriptor
=========================== short test summary info ============================
FAILED tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_and_check_raises_on_violations
======================== 1 failed, 10 passed in 16.23s =========================
pypy3-test-opentelemetry-test-utils: exit 1 (17.22 seconds) /home/runner/work/opentelemetry-python/opentelemetry-python> pytest /home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/tests -ra pid=7639
  pypy3-test-opentelemetry-test-utils: FAIL code 1 (779.65=setup[762.43]+cmd[17.22] seconds)
  evaluation failed :( (781.37 seconds)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions