Merge branch 'main' into fix-127727

Author: sepehr-rs

Doc/c-api/threads.rst

@@ -736,10 +736,10 @@ Low-level APIs
 .. c:function:: PyObject* PyThreadState_GetDict()
 
    Return a dictionary in which extensions can store thread-specific state
-   information.  Each extension should use a unique key to use to store state in
+   information.  Each extension should use a unique key to store a state in
    the dictionary.  It is okay to call this function when no :term:`thread state`
    is :term:`attached <attached thread state>`. If this function returns
-   ``NULL``, no exception has been raised and the caller should assume no
+   ``NULL`` and no exception has been raised, then the caller should assume no
    thread state is attached.
 
 

Doc/howto/free-threading-python.rst

@@ -165,3 +165,132 @@ to false.  If the flag is true then the :class:`warnings.catch_warnings`
 context manager uses a context variable for warning filters.  If the flag is
 false then :class:`~warnings.catch_warnings` modifies the global filters list,
 which is not thread-safe.  See the :mod:`warnings` module for more details.
+
+
+Increased memory usage
+----------------------
+
+The free-threaded build will typically use more memory compared to the default
+build.  There are multiple reasons for this, mostly due to design decisions.
+
+
+All interned strings are immortal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For modern Python versions (since version 2.3), interning a string (e.g. with
+:func:`sys.intern`) does not cause it to become immortal.  Instead, if the last
+reference to that string disappears, it will be removed from the interned
+string table.  This is not the case for the free-threaded build and any interned
+string will become immortal, surviving until interpreter shutdown.
+
+
+Non-GC objects have a larger object header
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The free-threaded build uses a different :c:type:`PyObject` structure.  Instead
+of having the GC related information allocated before the :c:type:`PyObject`
+structure, like in the default build, the GC related info is part of the normal
+object header.  For example, on the AMD64 platform, ``None`` uses 32 bytes on
+the free-threaded build vs 16 bytes for the default build.  GC objects (such as
+dicts and lists) are the same size for both builds since the free-threaded
+build does not use additional space for the GC info.
+
+
+QSBR can delay freeing of memory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to safely implement lock-free data structures, a safe memory
+reclamation (SMR) scheme is used, known as quiescent state-based reclamation
+(QSBR).  This means that the memory backing data structures allowing lock-free
+access will use QSBR, which defers the free operation, rather than immediately
+freeing the memory.  Two examples of these data structures are the list object
+and the dictionary keys object.  See ``InternalDocs/qsbr.md`` in the CPython
+source tree for more details on how QSBR is implemented.  Running
+:func:`gc.collect` should cause all memory being held by QSBR to be actually
+freed.  Note that even when QSBR frees the memory, the underlying memory
+allocator may not immediately return that memory to the OS and so the resident
+set size (RSS) of the process might not decrease.
+
+
+mimalloc allocator vs pymalloc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default build will normally use the "pymalloc" memory allocator for small
+allocations (512 bytes or smaller).  The free-threaded build does not use
+pymalloc and allocates all Python objects using the "mimalloc" allocator.  The
+pymalloc allocator has the following properties that help keep memory usage
+low: small per-allocated-block overhead, effective memory fragmentation
+prevention, and quick return of free memory to the operating system.  The
+mimalloc allocator does quite well in these respects as well but can have some
+more overhead.
+
+In the free-threaded build, mimalloc manages memory in a number of separate
+heaps (currently four).  For example, all GC supporting objects are allocated
+from their own heap.  Using separate heaps means that free memory in one heap
+cannot be used for an allocation that uses another heap.  Also, some heaps are
+configured to use QSBR (quiescent-state based reclamation) when freeing the
+memory that backs up the heap (known as "pages" in mimalloc terminology).  The
+use of QSBR creates a delay between all memory blocks for a page being freed
+and the memory page being released, either for new allocations or back to the
+OS.
+
+The mimalloc allocator also defers returning freed memory back to the OS.  You
+can reduce that delay by setting the environment variable
+:envvar:`!MIMALLOC_PURGE_DELAY` to ``0``.  Note that this will likely reduce
+the performance of the allocator.
+
+
+Free-threaded reference counting can cause objects to live longer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the default build, when an object's reference count reaches zero, it is
+normally deallocated.  The free-threaded build uses "biased reference
+counting", with a fast-path for objects "owned" by the current thread and a
+slow path for other objects.  See :pep:`703` for additional details.  Any time
+an object's reference count ends up in a "queued" state, deallocation can be
+deferred.  The queued state is cleared from the "eval breaker" section of the
+bytecode evaluator.
+
+The free-threaded build also allows a different mode of reference counting,
+known as "deferred reference counting".  This mode is enabled by setting a flag
+on a per-object basis.  Deferred reference counting is enabled for the
+following types:
+
+* module objects
+* module top-level functions
+* class methods defined in the class scope
+* descriptor objects
+* thread-local objects, created by :class:`threading.local`
+
+When deferred reference counting is enabled, references from Python function
+stacks are not added to the reference count.  This scheme reduces the overhead
+of reference counting, especially for objects used from multiple threads.
+Because the stack references are not counted, objects with deferred reference
+counting are not immediately freed when their internal reference count goes to
+zero.  Instead, they are examined by the next GC run and, if no stack
+references to them are found, they are freed.  This means these objects are
+freed by the GC and not when their reference count goes to zero, as is typical.
+
+
+Per-thread reference counting can delay freeing objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To avoid contention on the reference count fields of frequently shared
+objects, the free-threaded build also uses "per-thread reference counting"
+for a few selected object types.  Rather than updating a single shared
+reference count, each thread maintains its own local reference count array,
+indexed by a unique id assigned to the object.  The true reference count is
+only computed by summing the per-thread counts when the object's local
+count drops to zero.  Per-thread reference counting is currently used for:
+
+* heap type objects (classes created in Python)
+* code objects
+* the ``__dict__`` of module objects
+
+Because the per-thread counts must be merged back to the object before it
+can be deallocated, objects using per-thread reference counting are
+typically freed later than they would be in the default build.  In
+particular, such an object is usually not freed until the thread that
+referenced it reaches a safe point (for example, in the "eval breaker"
+section of the bytecode evaluator) or exits.  Running :func:`gc.collect`
+will merge the per-thread counts and allow these objects to be freed.

Doc/library/ctypes.rst

@@ -14,6 +14,14 @@ used to wrap these libraries in pure Python.
 
 .. include:: ../includes/optional-module.rst
 
+.. warning::
+
+   :mod:`!ctypes` provides low-level access to native libraries and the
+   process's memory, bypassing Python's safety mechanisms and allowing
+   execution of arbitrary native code.
+   Incorrect use can corrupt data and objects, reveal sensitive information,
+   cause crashes, or otherwise compromise the running process.
+
 
 .. _ctypes-ctypes-tutorial:
 
@@ -198,10 +206,8 @@ argument values::
    OSError: exception: access violation reading 0x00000020
    >>>
 
-There are, however, enough ways to crash Python with :mod:`!ctypes`, so you
-should be careful anyway.  The :mod:`faulthandler` module can be helpful in
-debugging crashes (e.g. from segmentation faults produced by erroneous C library
-calls).
+The :mod:`faulthandler` module can help debug crashes,
+such as segmentation faults produced by erroneous C library calls.
 
 ``None``, integers, bytes objects and (unicode) strings are the only native
 Python objects that can directly be used as parameters in these function calls.

Doc/library/enum.rst

@@ -222,7 +222,7 @@ Data types
       names of the members in *cls*::
 
         >>> dir(Color)
-        ['BLUE', 'GREEN', 'RED', '__class__', '__contains__', '__doc__', '__getitem__', '__init_subclass__', '__iter__', '__len__', '__members__', '__module__', '__name__', '__qualname__']
+        ['BLUE', 'GREEN', 'RED', '__class__', '__contains__', '__doc__', '__getitem__', '__init_subclass__', '__iter__', '__len__', '__members__', '__module__', '__name__', '__qualname__', '_generate_next_value_', '_missing_']
 
    .. method:: EnumType.__getitem__(cls, name)
 
@@ -355,7 +355,7 @@ Data types
          ...         print(f'today is {cls(dt.date.today().isoweekday()).name}')
          ...
          >>> dir(Weekday.SATURDAY)
-         ['__class__', '__doc__', '__eq__', '__hash__', '__module__', 'name', 'today', 'value']
+         ['__class__', '__doc__', '__eq__', '__hash__', '__module__', '_add_alias_', '_add_value_alias_', '_generate_next_value_', '_missing_', 'name', 'today', 'value']
 
    .. method:: Enum._generate_next_value_(name, start, count, last_values)
 

Doc/library/inspect.rst

@@ -1614,10 +1614,11 @@ properties, will be invoked and :meth:`~object.__getattr__` and
 may be called.
 
 For cases where you want passive introspection, like documentation tools, this
-can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr`
+can be inconvenient. :func:`getattr_static` has a similar signature as :func:`getattr`
 but avoids executing code when it fetches attributes.
 
-.. function:: getattr_static(obj, attr, default=None)
+.. function:: getattr_static(obj, attr)
+              getattr_static(obj, attr, default)
 
    Retrieve attributes without triggering dynamic lookup via the
    descriptor protocol, :meth:`~object.__getattr__`

Doc/library/lzma.rst

@@ -356,12 +356,26 @@ options. Valid filter IDs are as follows:
 
 * Branch-Call-Jump (BCJ) filters:
 
-  * :const:`FILTER_X86`
-  * :const:`FILTER_IA64`
-  * :const:`FILTER_ARM`
-  * :const:`FILTER_ARMTHUMB`
-  * :const:`FILTER_POWERPC`
-  * :const:`FILTER_SPARC`
+  * :const:`!FILTER_X86`
+  * :const:`!FILTER_IA64`
+  * :const:`!FILTER_ARM`
+  * :const:`!FILTER_ARMTHUMB`
+  * :const:`!FILTER_POWERPC`
+  * :const:`!FILTER_SPARC`
+
+    The above work on all lzma runtime library versions.
+
+  * :const:`!FILTER_ARM64`
+
+    Only works if the lzma version is 5.4.0 or later.
+
+    .. versionadded:: next
+
+  * :const:`!FILTER_RISCV`
+
+    Only works if the lzma version is 5.6.0 or later.
+
+    .. versionadded:: next
 
 A filter chain can consist of up to 4 filters, and cannot be empty. The last
 filter in the chain must be a compression filter, and any other filters must be

Doc/library/pydoc.rst

@@ -68,6 +68,11 @@ will start a HTTP server on port 1234, allowing you to browse the
 documentation at ``http://localhost:1234/`` in your preferred web browser.
 Specifying ``0`` as the port number will select an arbitrary unused port.
 
+.. warning::
+
+   The :mod:`!pydoc` HTTP server is intended for local use during
+   development and is not suitable for production use.
+
 :program:`python -m pydoc -n <hostname>` will start the server listening at the given
 hostname.  By default the hostname is 'localhost' but if you want the server to
 be reached from other machines, you may want to change the host name that the

Doc/reference/compound_stmts.rst

@@ -620,7 +620,7 @@ The match statement is used for pattern matching.  Syntax:
    match_stmt: 'match' `subject_expr` ":" NEWLINE INDENT `case_block`+ DEDENT
    subject_expr: `flexible_expression` "," [`flexible_expression_list` [',']]
                : | `assignment_expression`
-   case_block: 'case' `patterns` [`guard`] ":" `!block`
+   case_block: 'case' `patterns` [`guard`] ":" `suite`
 
 .. note::
    This section uses single quotes to denote

Doc/reference/datamodel.rst

@@ -496,7 +496,7 @@ subscript notation ``a[k]`` selects the item indexed by ``k`` from the mapping
 :keyword:`del` statements. The built-in function :func:`len` returns the number
 of items in a mapping.
 
-There is currently a single intrinsic mapping type:
+There are two intrinsic mapping types:
 
 
 Dictionaries
@@ -535,6 +535,20 @@ module.
    an implementation detail at that time rather than a language guarantee.
 
 
+Frozen dictionaries
+^^^^^^^^^^^^^^^^^^^
+
+.. index:: pair: object; frozendict
+
+These represent an immutable dictionary.  They are created by the built-in
+:func:`frozendict` constructor.  A frozendict is :term:`hashable` if all of
+its keys and values are hashable, in which case it can be used as an element
+of a set, or as a key in another mapping.  :class:`!frozendict` is not a
+subclass of :class:`dict`; it inherits directly from :class:`object`.
+
+.. versionadded:: 3.15
+
+
 Callable types
 --------------