Tracking Issue for pointer_bytes_offsets

[Gankra]

Feature gates:

#![feature(pointer_byte_offsets)]

#![feature(const_pointer_byte_offsets)]

This is a tracking issue for the pointer_byte_offsets raw pointer conveniences like ptr.byte_add(offset)

Public API

impl<T: ?Sized> *const T {
    // feature gates `pointer_byte_offsets` and `const_pointer_byte_offsets`
    pub const unsafe fn byte_offset(self, count: isize) -> Self;
    pub const unsafe fn byte_add(self, count: usize) -> Self;
    pub const unsafe fn byte_sub(self, count: usize) -> Self;

    pub const fn wrapping_byte_offset(self, count: isize) -> Self;
    pub const fn wrapping_byte_add(self, count: usize) -> Self;
    pub const fn wrapping_byte_sub(self, count: usize) -> Self;

    pub const unsafe fn byte_offset_from<U: ?Sized>(self, origin: *const U) -> isize;
}

// ... and the same for` *mut T`

Steps / History

• Implementation: Add convenience byte offset/check align functions to pointers #95643
• Making byte_offset_from more generic: Make pointer::byte_offset_from more generic #103489
• Final comment period (FCP): Tracking Issue for pointer_bytes_offsets #96283 (comment)
• Stabilization PR: Stabilize [const_]pointer_byte_offsets #116205

Unresolved Questions

• Should these operations actually accomodate DSTs? This seems deeply semantically dubious/broken but idk, you can plausibly use them right if you are extremely careful and understand the implications of stacked borrows for slices/projections.

[Gankra]

To clarify the open question's concerns a bit: if you make a ref-slice and then cast it to a raw-slice, offsetting the raw pointer will not change the slice's len, but you can't actually turn the offset slice back into a ref, because (at least under the strictest rules) the original ref-slice has provenance over only its exact range, and so derived pointers cannot access outside the range. Creating the offset ref-slice is trying to assert provenance over additional bytes you don't have permission to access.

Similarly, trait-object-refs will make you sad if you do that.

This can only be soundly used (meaningfully) if you from_raw_parts the raw-DST from a raw pointer with provenance over the whole "array" you want to stride through.

So like, in general you need to write something like (pseudocode of the general pattern):

let array = [...];

// Steal metadata from an example or make the metadata from whole cloth
let dst = &array[something];
let metadata = get_metadata(dst);

// Now build a raw DST from a pointer to the whole array
let raw_array_ptr = array.as_ptr();
let raw_dst = ptr::from_raw_parts(raw_array_ptr, metadata);

// Yay this actually can be strided through the whole array now
let next_dst = raw_dst.byte_add(offset_to_next_dst);

(CC @RalfJung since this is an interesting stacked borrows example, no need to reply if there's nothing more to say.)

[WaffleLapkin]

@Gankra actually, I think allowing these methods take pointers to DST unlocks a lot of abilities for custom DSTs. In particular byte_sub is required to implement Arc::from_raw (as it accepts a possibly fat pointer to T that needs to be offseted back to ArcInner<T>).

[WaffleLapkin]

I want to propose to stabilize these methods! (can someone open an fcp?)

Stabilization report

Implementation History

• Initially these methods were implemented in Add convenience byte offset/check align functions to pointers #95643
• The only unresolved question raised since then is whatever to allow calling these methods on pointers to unsized types. I propose to resolve this to "yes, we should actually", this will unlock a lot of options for working with unsized types (see expitience report for an example).

API Summary

impl *const T {
    pub const unsafe fn byte_offset(self, count: isize) -> Self;
    pub const unsafe fn byte_add(self, count: usize) -> Self;
    pub const unsafe fn byte_sub(self, count: usize) -> Self;

    pub const fn wrapping_byte_offset(self, count: isize) -> Self;
    pub const fn wrapping_byte_add(self, count: usize) -> Self;
    pub const fn wrapping_byte_sub(self, count: usize) -> Self;

    pub const unsafe fn byte_offset_from<U: ?Sized>(self, origin: *const U) -> isize;
}

// ... and the same for` *mut T`

Experience Report

#99113, #100819 and #100030 refactored some std code into using these methods simplifying the code.

These methods allow custom Arc implementations to work with unsized types just as well as std::arc::Arc, namely they allow to offset a pointer by <size of arc metadata> while keeping pointer metadata, which is required for example for Arc::from_raw.

This also allows to polyfil with_metadata_of by using wrapping_byte_offset, which is again very useful when working with DSTs.

[scottmcm]

Pondering: does byte_offset_from really need its argument to be a *const T? Arguably it doesn't care what type you pass it, so it could be *const U instead (or *const impl Sized, I suppose).

That would allow things like ptr::addr_of!(x.b).byte_offset_from(ptr::addr_of!(x)) to compile without needing an arbitrary choice of cast.

EDIT: this was updated in #103489