Normalization of call kinds

[ynfle]

As of now, the proposed representation for Nim proc/func/template/macro calls is to normalize them all to regular calls. For example:

`+` 1, 2 => `+`(1, 2)
1 + 2    => `+`(1, 2)
-1       => `-`(1)

Is this the desired result?
Certain call styles are more idiomatic

[reneegyllensvaan]

For operators

Will any part of the representation be shown to students? IME idiomatic nim tends to be all in infix style, call form for operators is generally rare (other than the occasional a.foo.`$`.bar) and showing them feels like it can build bad habits.

Even if we don't show any part of it to students, it still feels nice to have them be infix.

Since most solutions are going to use only infix anyway, should we even normalize a~b <==> `~`(a,b)?

func twoFer*(name = "you"): string =
  `&`(`&`("One for ",
          name), 
      ", one for me.")

func twoFer*(name = "you"): string =
  "One for " & name & ", one for me."

These are two very different solutions imo, and I feel like the first one (conceivably written by a homesick Lisper) deserves to be flagged for feedback if we ever get something like it. Again, if is the operative word, from experience every solution outside a blue moon will look more like the latter.

Is there a technical limitation preventing just normalizing them to infix form? Are there tradeoffs (e.g. lots of parentheses to be sure, parser should handle that though)? If not, I say absolutely normalize to infix.

For function calls

Function calls in general can be normalized to foo(a,b), I'd say. UFCS is a concept in itself and currently I just mentor it manually if it looks like people haven't discovered it themselves.

If we can rely on UFCS being shown somewhere else, then I don't see any issue with normalizing everything to foo(a,b). I'd also prefer it over a.foo(b) because we can just ignore the 0-ary and unary cases.

[ynfle]

The purpose of the representer is only for mentors to assign feedback, they are never shown to the students.
If we'd want to give feedback to students that use infix operators as prefix in general, then we shouldn't normalize them. (See this slack discussion for reference)

Regarding function calls, until the compiler checks the left and right side of a.b, it is not known whether b is a proc, field etc. Therefore, the only way to know, for sure, that b is a call node (proc, template, macro or fund) would be if it is using the call (a.b()) syntax with or without arguments, or the command (seq.add 1, 2, 3) syntax with arguments.
Therefore, if a dot expression is used with the call or command syntax, it can be normalized to a regular call and just be turned off for the UFCS concept exercise. This just means that it will not be evident if the student actually wrote a.b(1).c(2).d(3).e(4).f(5) or f(e(d(c(b(a, 1), 2), 3), 4), 5). I personally think we'd want to know that advise against it.
@reneegyllensvaan, what would you do in this situation? Do you think that information is valuable?

[reneegyllensvaan]

I'd say don't normalize infix operator calls to prefix then, again the set of solutions that'll actually use prefix form is really small, and the fraction of those in turn that we won't want to give specific feedback related to prefixes is vanishingly small.

Regarding function calls, until the compiler checks the left and right side of a.b, it is not known whether b is a proc, field etc. Therefore, the only way to know, for sure, that b is a call node (proc, template, macro or fund) would be if it is using the call (a.b()) syntax with or without arguments, or the command (seq.add 1, 2, 3) syntax with arguments.
Therefore, if a dot expression is used with the call or command syntax, it can be normalized to a regular call and just be turned off for the UFCS concept exercise.
@reneegyllensvaan, what would you do in this situation?

So I don't do a lot of metaprogramming (it's just not the part of nim I find interesting) but if I had this problem I'd probably just change my macro to operate on the typed AST. Skimmed the representer source and I guess there are no macros because it's all done manually on an external file via parseStmt. Since it all happens statically you could pass the parse tree to a macro that takes a typed argument and get Call nodes for any form of function call, something like this:

import macros

const code = """
func foo(a: int): int =
    a+a

echo(10.foo)
"""

proc normalize(input: NimNode): NimNode =
  input

macro codeAsUntypedAst(): untyped =
  parseStmt(code)

macro formatNormalizedTypedTree(t: typed): string =
  let normalized = t.normalize
  result = newStrLitNode normalized.repr

const typedFormattedTree = formatNormalizedTypedTree(codeAsUntypedAst())

echo typedFormattedTree

#=>
# func foo(a: int): int =
#   result = a + a
#
# echo([foo(10)])

But the reductions that happen on the way to becoming a typed AST are too significant, like in this case it's desugaring varargs for echo, so it's not really a solution and there isn't really a solution without pulling in the compiler toolchain.

That was mostly a sidebar, let's ignore the tech side for a moment.

I thought about this, and I don't really mind not normalizing some call forms. If we have 4 (non-block) unary call forms:

1) a.foo
2) a.foo()
3) foo(a)
4) foo a

Then as long as we normalize 2->3 (i.e. flatten (Call (DotExpr a foo)) into (DotExpr a foo)) and 4->3 ((Command foo a) into (Call foo a)), we should be good in terms of false non-matches. Those differences are also not relevant for mentoring purposes, call form vs. dot form sort of are (mentees usually discover them by themselves, e.g. via documentation, but sometimes you'll see some very nested calls that'd be a lot cleaner as pipelined). All the other arities should be fine if they just follow the unary cases.

Therefore, if a dot expression is used with the call or command syntax, it can be normalized to a regular call and just be turned off for the UFCS concept exercise. This just means that it will not be evident if the student actually wrote a.b(1).c(2).d(3).e(4).f(5) or f(e(d(c(b(a, 1), 2), 3), 4), 5). I personally think we'd want to know that advise against it.

@ynfle I think this is what I'm trying to convey, I think we're on the same page. Keeping dot form representations as they are (and just removing parentheses for unary dot syntax calls) seems like a reasonable minimum verifiable prototype for this, if the queue starts filling up with duplicates we can take another look at this with more info.

The TL;DR of what I'd wager is the maximum level of normalization without sacrificing mentoring granularity is

1) a.foo
2) a.foo()
3) foo(a)
4) foo a

5) a ~ b
6) a.`~` b
7) a.`~`(b)
8) `~`(a)
9) `~` a

Gets transformed according to

• 2->1
• 4->3
• Keep all other forms (1, 3, 5) intact
• Treat 6..9 like their equivalents in 1..4

I have some personal biases, e.g. I have a hard personal rule against copy-pasting anything other than extra-credit type stuff, so I'm much more in favor of granular differentiation between solutions than spending less time on each one. This looks good and sensible though, excited to see where it leads! ✨

[ynfle]

Converting AST to typed

The typed AST is significantly more complex than the untyped AST

With this code:

import strformat

proc twoFer(name = "you): string =
  fmt"One for {name}, one for me."

The untyped AST is

StmtList
  ImportStmt
    Ident "strformat"
  ProcDef
    Postfix
      Ident "*"
      Ident "twoFer"
    Empty
    Empty
    FormalParams
      Ident "string"
      IdentDefs
        Ident "name"
        Empty
        StrLit "you"
    Empty
    Empty
    StmtList
      CallStrLit
        Ident "fmt"
        RStrLit "One for {name}, one for me."

versus the typed AST

StmtList
  ImportStmt
    Sym "strformat"
  ProcDef
    Sym "twoFer"
    Empty
    Empty
    FormalParams
      Sym "string"
      IdentDefs
        Ident "name"
        Empty
        StrLit "you"
    Empty
    Empty
    Asgn
      Sym "result"
      StmtListExpr
        VarSection
          IdentDefs
            Sym "fmtRes"
            Empty
            Call
              Sym "newStringOfCap"
              HiddenStdConv
                Empty
                IntLit 37
        Call
          Sym "add"
          Sym "fmtRes"
          StrLit "One for "
        Call
          Sym "formatValue"
          HiddenAddr
            Sym "fmtRes"
          Sym "name"
          StrLit ""
        Call
          Sym "add"
          Sym "fmtRes"
          StrLit ", one for me."
        Sym "fmtRes"
    Sym "result"

The Ast is a lot more complicated and macros are expanded. Both of these factors make it hard to work on and not representative of the students code.

Normalizing Calls

For 1) or 2) above, which is the more idiomatic way in nim? Other than that 👍

[reneegyllensvaan]

For 1) or 2) above, which is the more idiomatic way in nim? Other than that 👍

They're interchangeable imo, I usually omit them, so does the stdlib from what I've read (and a quick grep just now), so I'd say 1), but neither is discouraged.