GH-143493: Conform to spec for generator expressions while supporting virtual iterators

[markshannon]

• Moves the GET_ITER instruction into the generator function preamble. This means the the iterable is converted into an iterator during generator creation, as documented, but keeps it in the same code object allowing optimization.

• Issue: Behavior of generator expressions doesn't match the docs when passed a non-iterable #143493

[serhiy-storchaka]

Sorry, the code was changed so much since I worked with it, that I can no longer make qualified review. I can understand particular changes, but I cannot catch if something was missed. The original idea looks reasonably.

BTW, what happens when __iter__() raises StopIteration?

Is similar change needed for asynchronous generators?

[markshannon]

BTW, what happens when __iter__() raises StopIteration?

The exception is raised before the generator is created and the traceback shows the iterable as the location of the exception.

[markshannon]

Is similar change needed for asynchronous generators?

I don't think asynchronous generators were ever changed, so they still have the old behavior from 3.13 and earlier.

[DinoV]

Other than the one nit looks good to me

[DinoV]

nit: Is it worth renaming these to something like iter_state like it is in codegen_comprehension?

[markshannon]

It isn't it state, it is its location. Maybe I'll rename IterStackState instead.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 macOS 3.x (tier-2) has failed when building commit ae53da5.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/725/builds/12565) and take a look at the build logs.
4. Check if the failure is related to this commit (ae53da5) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/725/builds/12565

Failed tests:

• test_urllib2net

Failed subtests:

• test_call_count_thread_safe - test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_call_count_thread_safe
• test_preauth_data_to_tls_server - test.test_ssl.TestPreHandshakeClose.test_preauth_data_to_tls_server

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_ssl.py", line 5476, in test_preauth_data_to_tls_server
    self.assertIn("before TLS handshake with data", wrap_error.args[1])
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'before TLS handshake with data' not found in '[SSL: RECORD_LAYER_FAILURE] record layer failure (_ssl.c:1096)'


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_unittest/testmock/testthreadingmock.py", line 219, in test_call_count_thread_safe
    self.assertEqual(m.call_count, LOOPS * THREADS)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 998 != 1000