The definition of fragment specifier is inconsistent

[xmh0511]

Consider this example:

macro_rules! test{
    ($id:ident, $e:expr)=>{
        $id.$e
    }
}
fn main() {
  let a = (1,);
  test!(a,0);
}

The compiler reports an error

error: unexpected token: `expr` metavariable
 --> src/main.rs:3:13
  |
3 |         $id.$e
  |             ^^
...
8 |   test!(a,0);
  |   ---------- in this macro invocation
  |
  = note: this error originates in the macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

IIUC, the reason can be interpreted by the syntax

Valid fragment specifiers are:

• expr: an Expression

$e is defined as an expr fragment specifier, which can match LiteralExpression; however, the valid syntax of Tuple indexing expressions is defined as

TupleIndexingExpression → Expression . TUPLE_INDEX

The component following . is TUPLE_INDEX in which an Expression cannot appear. So, this is the reason for the error, which is ruled by syntax.

However, see the definition of path

path: a TypePath

This is a fixed version by #2248

A TypePath is

TypePath → ::? TypePathSegment ( :: TypePathSegment )*

Instead, a path in an expression should be

PathExpression →
PathInExpression
| QualifiedPathInExpression

The path that appears in an expression doesn't expect the syntax TypePath, that is, the following code should have had a similar error as the first example:

macro_rules! test2{
    ($id:ident,$p:path)=>{
        $id = $p;
    }
}
mod S{
    pub const I:i32 = 0;
}
fn main() {
  let mut i = 0;
  test2!(i,S::I);
}

However, this code is ok.

[ehuss]

If I had to guess, I would say the conversion of a type path to an expression path is done here.

@fmease would you concur? I can't honestly claim to remember why I used the phrase "TypePath style path" back in 2018, but I want to say that this is the reason. It accepts TypePath, but when expanded it can be translated to other path kinds, like PathExpression and SimplePath.

[xmh0511]

If I had to guess, I would say the conversion of a type path to an expression path is done here.

@fmease would you concur? I can't honestly claim to remember why I used the phrase "TypePath style path" back in 2018, but I want to say that this is the reason. It accepts TypePath, but when expanded it can be translated to other path kinds, like PathExpression and SimplePath.

Maybe we want to express it in this way:

TypePath style path means that the sequence of tokens matches TypePath

which can be briefly said that it has the form of TypePath. However, we cannot say a TypePath, because it's a syntax that is limited to appearing in a specific grammar.

[fmease]

Hmm, I see. AFAICT that page doesn't precisely define match as in "fragment-specifier matches a Rust syntax fragment of the kind specified". A few paragraphs above it does briefly talk about "opaque ASTs" wrt. matching behavior but it seems this section isn't sufficient.

Right now I think I wouldn't want to go back to the "Rule-style" phrasing (at least not without properly defining it somewhere and using it all other relevant places) since there's a lot more to it, transcending path / TypePath.

Going back to

fragment-specifier matches a Rust syntax fragment of the kind specified

I think it's ambiguous as to whether it means "structural" or "nominal" match. Well, we know (from rustc's impl and as alluded to by the "opaque AST" section) it's a nominal match with several exceptions and nuances IIUC.

Now, since it's a nominal match that begs the question what the relationship is between all syntax fragment kinds. For example, we know item and expr are compatible with stmt; block and literal with expr. Would we need to explicitly specify that fact in the Reference or is that implied by the fact that the corresponding grammar rules (e.g., Statement for stmt) also generate/accept token sequences produced/accepted by the other corresp. grammar rule (e.g., Item for item)? Wouldn't that be structural matching though which we know it isn't?

An "interesting" case (similar to TypePath vs. pat, expr, …) is for example: expr is intentionally compatible with pat (!) as long as the not-so-opaque fragment contains a maybe-negated literal (structurally):

// OK!

macro_rules! test {
    ($x:expr) => {
        let /*pat ctxt!*/ $x = (loop {}) else { loop {} };
    }
}

fn scope() { test!(0); }

// OK!
macro_rules! m0 { ($x:expr) => { m1! { $x } } }
macro_rules! m1 { ($x:pat) => {} }
m0! { "" }

Furthermore, either syntax fragments branded with tt are compatible with all other kinds or they don't get "branded" in the first place; I don't know the specifics. Finally, injected identifiers and lifetimes usually aren't treated as opaque syntax fragments but as transparent ones; I'm relatively sure that that's user observable but I'm not an expert on that matter.

I'm really starting to ramble now.

Anyway, I think a consequence of that would be the following: Fragment kind lifetime (LIFETIME_TOKEN) can be used in places where LIFETIME_OR_LABEL is expected:

// OK!

macro_rules! check {
    ($l:lifetime) => { $l: loop { break $l; } };
}

fn scope() { check!('a); }

Moving towards the early expansion phase, since PR rust-lang/rust#146961 you can use $e:expr in places where ConfigurationPredicate is expected (as long as it reparses as a configuration predicate // is structurally a configuration predicate) despite it not allowing arbitrary expressions in its grammar.

[fmease]

So in summary path/TypePath being "malleable"/reinterpretable isn't a one-off, there are lots of similar cases due to most syntax fragments being treated as opaque and there being various "ad hoc" "conversion" rules in rustc (which is fragile and buggy; see for example issue rust-lang/rust#143221). IINM so far this isn't documented in an exhaustive and principled manner in the Reference.

Maybe I've missed the mark somewhere and the examples I gave aren't comparable to the path case.

[fmease]

Of course, theoretically we could precisely describe these opaque syntax fragment rules by mimicking rustc in the way it deals with metavars internally. However, that'd be rather janky I feel like.

That would entail updating the Reference grammar like this perhaps:

...

Expression ->
      ExpressionWithoutBlock
    | ExpressionWithBlock
    | `⟦expr` Expression `⟧`
    | `⟦literal` `-`? LiteralExpression `⟧`
    | `⟦block` BlockExpressionNoAttributes `⟧`
    | `⟦path` TypePath `⟧`;

...

where ⟦ and ⟧ would by definition represent tokens that are not part of the normal user-writable alphabet. They would form the basis of branded invisible delimiters. So a metavariable $x of kind block matching token sequence {} would be transcribed as ⟦block {}⟧.

Instead, a path in an expression should be

PathExpression →
    PathInExpression
   | QualifiedPathInExpression

They should not, they are still TypePaths. Consider:

// OK

macro_rules! m { ($p:path) => { $p } }

fn main() {
    let _ = m!(Option<i32>::None);
            // ~~~~~~~~~~~~~~~~~ a valid TypePath
            //                   not a valid PathInExpression (those require turbofish)
}